The from hugging Face provides a very efficient way to load and process NLP datasets from raw files or in-memory data. These NLP datasets have been shared by different research and practitioner communities across the world. Datasets library You can also load various evaluation metrics used to check the performance of on numerous tasks. NLP models If you are working in and want an NLP dataset for your next project, I recommend you use this library from Hugging Face. Natural Language Processing You can use this library with other popular frameworks in machine learning, such as , , , and You will learn more in the examples below. Numpy Pandas Pytorch TensorFlow. The NLP datasets are available on different tasks such as Text-classification Question-Answering Sequence-modelling Machine translation, Automatic speech-recognition Sentiment Analysis Speech Processing Name entity Recognition Part of Speech Tagging others You can find the NLP datasets in more than 186 languages. All these datasets can also be browsed on the and can be viewed and explored online with the . HuggingFace Hub Datasets viewer In this article, you will learn how to download, load, configure and use NLP datasets from the hugging face datasets library. Let's get started. How to Install Datasets Library Installation is easy and takes only a few minutes. You can use pip as follows: pip install datasets Another option for installation is using conda as follows. conda install -c huggingface -c conda-forge datasets List of Datasets To view the list of different available datasets from the library, you can use the function from the library. list_datasets() datasets list_datasets, load_dataset pprint pprint

datasets_list = list_datasets() 
pprint(datasets_list,compact= ) from import from import True Currently, the datasets library has a total of 1182 datasets that can be used to create different NLP solutions. List of Dataset with Details You can also view a list of the datasets with details by adding an argument called equal to in the list_datases() function as follows. with_details True datasets_list = list_datasets(with_details= ) 
pprint(datasets_list) True This will show a list of datasets with their id(name), descriptions, and files (if downloaded in your local machine). Load Dataset To load the dataset from the library, you need to pass the file name on the function. load_dataset() The load_dataset function will do the following. Download and import in the library the file processing script from the Hugging Face GitHub repo. Run the file script to download the dataset Return the dataset as asked by the user. By default, it returns the entire dataset dataset = load_dataset( , ) 'ethos' 'binary' In the above example, I downloaded the dataset from hugging face. ethos "ETHOS: onlinE haTe speecH detectiOn dataSet. This repository contains a dataset for hate speech detection on social media platforms, called Ethos. There are two variations of the dataset:"- HuggingFace's page Each dataset can have several configurations that define the sub-part of the dataset you can select. For example, the ethos dataset has two configurations. Note: binary version multi-label version Let’s load the binary version we have downloaded. print(dataset) Split the Train and Validation Sets Sometimes you don't want to download the entire dataset, you can use the split parameter to specify if you want to download the train, validation, or test set from the dataset. ethos_train = load_dataset( , ,split= )

ethos_validation = load_dataset( , ,split= ) 'ethos' 'binary' 'train' 'ethos' 'binary' 'validation' This will save the training set in the variable and the validation set in variable. ethos_train ethos_validation Not all datasets have the train, validation, and test set. Some of them can contain only the train set. So you need to read more about the dataset you want to download from the . Note: hugging face dataset page Load Dataset from Local File You can use the library to load your local dataset from the local machine. You can load datasets that have the following format. CSV files JSON files Text files (read as a line-by-line dataset), Pandas pickled dataframe To load the local file you need to define the format of your dataset (example "CSV") and the path to the local file. dataset = load_dataset( , data_files= ) 'csv' 'my_file.csv' You can similarly instantiate a Dataset object from a pandas DataFrame as follows: datasets Dataset pandas pd 

df = pd.DataFrame({ : [ , , ]})
df_dataset = Dataset.from_pandas(df)

print(df_dataset) from import import as "a" 1 2 3 Download Mode When you load already downloaded data from the cache directory, you can control how the function handles it by setting its download_mode parameter. load_datasets() The parameter options are as follows. (default) - This will reuse both raw downloads and the prepared dataset if they exist in the cache directory. "reuse_dataset_if_exists" This will use the cached dataset. "reuse_cache_if_exists"- - This will redownload the dataset. "force_redownload" dataset = load_dataset( , , download_mode= ) 'ethos' 'binary' "force_redownload" In the above example, we redownload the ethos dataset with binary configuration. Setting The Format You can set the format of the dataset instance by using the function that takes as arguments. set_format() an optional string that defines the type of the objects that should be returned by : type: datasets.Dataset.getitem() (default): return python objects, None/'python' return PyTorch tensors, 'torch'/'pytorch'/'pt': : return Tensorflow tensors, 'tensorflow'/'tf' return JAX arrays, 'jax': : return Numpy arrays, 'numpy'/'np' : return Pandas DataFrames 'pandas'/'pd' ethos_train.set_format(type= , columns=[ , ]) 'pandas' 'comment' 'label' In the above example, we set the format type as . "pandas" Final Thoughts on NLP Datasets from Huggingface In this article, you have learned how to download datasets from hugging face datasets library, split into train and validation sets, change the format of the dataset, and more. We did not cover all the functions available from the datasets library. Check the following resources if you are looking to go deeper. Datasets: https://huggingface.co/datasets Datasets documentation: https://huggingface.co/docs/datasets/ NLP Metrics: https://huggingface.co/docs/datasets/loading_metrics.html If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in Data Science? in the footer below. Subscribe to our newsletter

Twitter

NLP Tutorial: Topic Modeling in Python with BerTopic

NLP Datasets from HuggingFace: How to Access and Train Them

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10 Best African Language Datasets for Data Science Projects

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

10 Best African Language Datasets for Data Science Projects

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps