Hugging Face Datasets Guide

Training Datasets & Benchmark Datasets (December 2025)

A comprehensive reference for datasets available on Hugging Face for pre-training, fine-tuning, and evaluating language models.

Table of Contents

1. Pre-Training Datasets

2. Fine-Tuning & Instruction Datasets

3. Benchmark Datasets - General Knowledge

4. Benchmark Datasets - Coding

5. Benchmark Datasets - Reasoning & Math

6. Benchmark Datasets - Multimodal

7. Specialized Domain Datasets

Pre-Training Datasets

Large-scale datasets for training foundation models from scratch.

Fine-Tuning & Instruction Datasets

Datasets for instruction-tuning and alignment of pre-trained models.

Benchmark Datasets - General Knowledge

Datasets for evaluating general knowledge, reasoning, and language understanding.

Benchmark Datasets - Coding

Datasets for evaluating code generation and programming capabilities.

Benchmark Datasets - Reasoning & Math

Datasets for evaluating mathematical and logical reasoning capabilities.

Benchmark Datasets - Multimodal

Datasets for evaluating vision-language and multimodal capabilities.

Specialized Domain Datasets

Domain-specific datasets for training and evaluation.

Medical & Healthcare

Law & Compliance

Multilingual & Translation

Computer Science & Technical

Retrieval & Search

Conversational & Chat

Safety & Alignment

Dataset Selection Guide

For Pre-Training

For Fine-Tuning

For Benchmarking

Important Notes

Data Contamination

• GPQA: Gated access to prevent contamination

• LiveCodeBench: Continuously updated to avoid memorization

• FineWeb-Edu: Only 1% benchmark overlap (lowest among major datasets)

• SWE-bench: Real GitHub issues, hard to contaminate

Benchmark Saturation

• MMLU: Becoming saturated (frontier models >85%)

• HumanEval: Only 164 problems (1-2% = 1-2 problems)

• Solution: Use MMLU-Pro, HumanEval+, GPQA for harder tests

License Considerations

• ODC-By: Open Data Commons, very permissive

• Apache 2.0: Permissive open source

• MIT: Very permissive

• CC-BY: Attribution required

• CC-BY-NC: Non-commercial only

• Always check individual dataset licenses before commercial use

Dataset Quality Issues

• DROP: Removed from some leaderboards due to normalization issues (Dec 2023)

• MMLU variants: Different implementations give different scores (Eleuther, HELM, original)

• Book3 in Pile: Removed due to copyright concerns

Evaluation Best Practices

1. Use multiple benchmarks (single benchmarks misleading)

2. Include domain-specific tests for your use case

3. Check for contamination in training data

4. Use held-out test sets when possible

5. Consider both automated metrics and human evaluation

6. Pay attention to few-shot vs zero-shot settings

Accessing Datasets

Using Datasets Library (Python)

​x
from datasets import load_dataset

# Load full dataset
dataset = load_dataset("HuggingFaceFW/fineweb", "default")

# Load specific split
dataset = load_dataset("cais/mmlu", "all", split="test")

# Load with streaming (for large datasets)
dataset = load_dataset("HuggingFaceFW/fineweb", streaming=True)

# Load specific configuration
dataset = load_dataset("HuggingFaceFW/fineweb", "sample-10BT")

Using Hugging Face Hub CLI

xxxxxxxxxx
# Download dataset
huggingface-cli download HuggingFaceFW/fineweb --repo-type dataset

# Login for gated datasets
huggingface-cli login

Recent Dataset Releases (2025)

Community Resources

• Hugging Face Datasets: https://huggingface.co/datasets

• Dataset Viewer: Built into Hugging Face (preview before downloading)

• Papers with Code: https://paperswithcode.com/datasets

• Leaderboards: https://huggingface.co/spaces (search "leaderboard")

• Documentation: https://huggingface.co/docs/datasets

Last Updated: December 20, 2025

Note: Dataset availability and links may change. Always verify on Hugging Face Hub. Benchmark scores and difficulty ratings are approximate and may vary by model and evaluation setup.

Sources: Hugging Face Dataset Hub, official dataset papers, benchmark leaderboards, community documentation (December 2025)