Datasets

加载文本数据

本指南将向您展示如何加载文本数据集。要了解如何加载任意类型的数据集，请参阅通用加载指南。

文本文件是存储数据集最常用的文件类型之一。默认情况下，🤗 Datasets 会逐行采样文本文件来构建数据集。

>>> from datasets import load_dataset
>>> dataset = load_dataset("text", data_files={"train": ["my_text_1.txt", "my_text_2.txt"], "test": "my_test_file.txt"})

# Load from a directory
>>> dataset = load_dataset("text", data_dir="path/to/text/dataset")

要按段落甚至整个文档对文本文件进行采样，请使用 sample_by 参数：

# Sample by paragraph
>>> dataset = load_dataset("text", data_files={"train": "my_train_file.txt", "test": "my_test_file.txt"}, sample_by="paragraph")

# Sample by document
>>> dataset = load_dataset("text", data_files={"train": "my_train_file.txt", "test": "my_test_file.txt"}, sample_by="document")

您也可以使用 grep 模式来加载特定文件：

>>> from datasets import load_dataset
>>> c4_subset = load_dataset("allenai/c4", data_files="en/c4-train.0000*-of-01024.json.gz")

要通过 HTTP 加载远程文本文件，请传入 URL：

>>> dataset = load_dataset("text", data_files="https://huggingface.co/datasets/hf-internal-testing/dataset_with_data_files/resolve/main/data/train.txt")

To load XML data you can use the “xml” loader, which is equivalent to “text” with sample_by=“document”:

>>> from datasets import load_dataset
>>> dataset = load_dataset("xml", data_files={"train": ["my_xml_1.xml", "my_xml_2.xml"], "test": "my_xml_file.xml"})

# Load from a directory
>>> dataset = load_dataset("xml", data_dir="path/to/xml/dataset")

Update on GitHub