LLMs and VLMs on Hugging Face + AI2 OLMO Models

Reference Guide - December 2024/January 2025

This document provides a comprehensive overview of Language Models (LLMs) and Vision-Language Models (VLMs) available on Hugging Face, along with the fully open OLMO models from the Allen Institute for AI (AI2).

Language Models (LLMs)

Tiny Models (≤3B parameters)

Small Models (4B-10B parameters)

Medium Models (11B-20B parameters)

Large Models (21B-50B parameters)

Very Large Models (>50B parameters)

Vision-Language Models (VLMs)

Tiny VLMs (≤3B parameters)

Medium VLMs (4B-20B parameters)

Large VLMs (>20B parameters)

Quantization Types Explained

Standard Precision Formats

• FP16 (Float16): 16-bit floating point - standard precision (2 bytes per parameter)

• BF16 (BFloat16): Brain Float 16 - optimized for neural networks, same range as FP32 (2 bytes per parameter)

• FP32: 32-bit floating point - full precision (4 bytes per parameter)

Quantization Methods

• 4-bit (QAT+LoRA): Quantization-Aware Training with Low-Rank Adaptation

  • Simulates quantization effects during training

  • Uses LoRA adapters to maintain quality

  • Achieves ~75% size reduction

• 4-bit (SpinQuant): Advanced post-training quantization

  • Optimized rotation-based quantization

  • Can be applied to fine-tuned models

  • Balances accuracy and performance

• 4-bit (AWQ): Activation-aware Weight Quantization

  • Protects salient weights based on activation patterns

  • Better accuracy retention than naive quantization

• 4-bit (GPTQ): Generalized Post-Training Quantization

  • Layer-wise quantization with error compensation

  • Popular for large model compression

• 8-bit: 8-bit integer quantization (1 byte per parameter)

  • ~50% size reduction

  • Minimal accuracy loss

Mixture of Experts (MoE)

• Only a subset of model parameters are active for each token

• Much smaller memory footprint during inference

• Example: DeepSeek V3 has 671B total parameters but only 37B active

Hybrid Mamba-Transformer Architecture

• Combines Mamba-2 (State Space Model) with Transformer attention

  • Mamba-2 layers: Linear time complexity O(n), constant memory per token

  • Transformer attention layers: Strategic placement for in-context learning

  • Typical ratio: 1 attention layer per 7-8 Mamba layers

• Key Advantages:

  • 3-6× faster inference than pure Transformers (especially long outputs)

  • Constant memory during generation (no growing KV cache)

  • Linear scaling with sequence length vs quadratic

  • Matches or exceeds Transformer accuracy

• Architecture Components (Nemotron example):

  • Mamba-2 layers: Efficient sequential processing

  • Attention layers: Handle copying and in-context learning

  • MLP layers: Standard feed-forward computation

  • Optional MoE: Sparse expert activation (Nemotron-3-Nano)

• Example: Nemotron-H-8B

  • 24 Mamba-2 layers + 4 attention layers + 24 MLP layers

  • 92% Mamba, 8% attention by layer count

  • Achieves +2.65 points over pure Transformer baseline

  • 3× faster inference, same training data

NEMOTRON Models - Hybrid Architecture Pioneer

The NVIDIA Nemotron family represents the first production-grade hybrid Mamba-Transformer models at scale:

What Makes NEMOTRON Unique

1. Hybrid Mamba-Transformer Architecture

   • Replaces 90%+ of attention layers with Mamba-2 (State Space Model)

   • Linear time complexity instead of quadratic

   • Constant memory per token during generation

   • 3-6× faster inference than pure Transformers

2. Multiple Model Families:

   • Nemotron-H: Pure hybrid baseline models (8B, 47B, 56B)

   • Nemotron Nano 2: Reasoning models with ON/OFF modes (9B, 12B)

   • Nemotron 3 Nano: Hybrid + MoE architecture (30B total, 3B active)

3. Reasoning Capabilities:

   • Configurable reasoning mode (ON/OFF via chat template)

   • Explicit chain-of-thought generation when enabled

   • Competitive with or exceeding pure Transformer baselines

4. Extreme Context Support:

   • Nemotron Nano 2: 128K tokens

   • Nemotron 3 Nano: 1M tokens (1 million native support)

   • Nemotron-H-47B: 1M tokens in FP4 on RTX 5090

5. Training Innovations:

   • FP8 pre-training at scale (56B model on 20T tokens)

   • Efficient compression (56B → 47B with 63B tokens)

   • Warmup-Stable-Decay (WSD) learning rate schedules

Nemotron Model Variants

Nemotron-H (Hybrid Base)

• Pure hybrid Mamba-Transformer without MoE

• Apples-to-apples comparison with Transformers (same training data)

• Research-focused baseline models

• 8B, 47B (compressed), 56B (flagship)

Nemotron Nano 2 (Reasoning)

• Configurable reasoning with ON/OFF modes

• 6× throughput improvement over Qwen3-8B

• 128K context window

• 9B (compressed from 12B), 12B base

Nemotron 3 Nano (MoE + Hybrid)

• Combines hybrid architecture with Mixture-of-Experts

• 30B total parameters, only 3B active per token

• 1M token native context window

• 128 experts with top-6 routing

• 23 Mamba-2 + 23 MoE + 6 attention layers

License & Availability

• License: Apache 2.0 / NVIDIA Open Model License

• Commercial Use: Permitted

• Open Resources:

  • Model weights on Hugging Face

  • Training recipes in NeMo Framework

  • Majority of pre-training data released (6.6T tokens)

  • Evaluation code and benchmarks

Performance Highlights

• Nemotron-H-8B: +2.65 points average over Transformer baseline

• Nemotron Nano 2: 6× throughput vs Qwen3-8B on reasoning tasks

• Nemotron 3 Nano: 3.3× throughput vs Qwen3-30B, matches GPT-OSS-20B

• Nemotron-H-56B: Matches Llama-3.1-70B/Qwen-2.5-72B with 3× speed

OLMO Models - Unique Features

The OLMO (Open Language Model) family from AI2 represents the gold standard for truly open AI:

What Makes OLMO "Fully Open"

1. Complete Model Flow: Not just weights, but the entire development pipeline

   • Pre-training data mixtures (fully documented and downloadable)

   • Training code and recipes

   • All intermediate checkpoints

   • Evaluation frameworks (OLMES)

   • Post-training recipes (Tülu 3)

2. OlmoTrace Integration:

   • Trace model outputs back to specific training data

   • Understand why models generate specific responses

   • Available in AI2 Playground

3. License: Apache 2.0 for all components

   • True open source

   • Commercial use permitted

   • No restrictions on model outputs

4. Reproducibility:

   • Everything needed to reproduce from scratch

   • Published training efficiency metrics

   • 2.5x more efficient than comparable models (e.g., vs Llama 3.1)

OLMO Model Variants

Base Models

• Pre-trained foundation models

• Ready for fine-tuning on specific tasks

• 65K token context window (16x larger than OLMo 2)

Instruct Models

• Instruction-tuned for dialog and tool use

• Multi-turn conversation capabilities

• Direct drop-in replacements for assistant applications

Think Models

• Reasoning-enhanced with explicit chain-of-thought

• Strong performance on math, coding, and logic tasks

• Shows intermediate reasoning steps

RL Zero (Experimental)

• Reinforcement learning from scratch

• Research into training dynamics

• Experimental release for community exploration

Size Estimation Formula

To estimate model storage requirements:

Float16/BF16 size (GB) ≈ Parameters × 2 bytes
4-bit quantized size (GB) ≈ Parameters × 0.5 bytes
8-bit quantized size (GB) ≈ Parameters × 1 byte

Examples:

• 7B model in FP16: 7 × 2 = ~14 GB

• 7B model in 4-bit: 7 × 0.5 = ~3.5 GB

• 72B model in FP16: 72 × 2 = ~144 GB

• 72B model in 4-bit: 72 × 0.5 = ~36 GB

Note: Actual sizes may vary due to:

• Embedding layers

• Additional model components

• Framework overhead

• Attention cache requirements during inference

Hardware Requirements

Consumer GPUs (Inference)

• 4GB-8GB VRAM: 1B-3B models (quantized)

• 12GB-16GB VRAM: 7B models (quantized), 3B models (full precision)

• 24GB VRAM: 7B-13B models (full precision), 30B models (quantized)

• 40GB-48GB VRAM: 30B-70B models (quantized)

• 80GB+ VRAM: 70B+ models (full precision)

Typical Inference Hardware

• Mobile/Edge: 1B-3B quantized models

• Consumer Desktop (RTX 3090/4090): Up to 7B full precision, 13B quantized

• Workstation (A5000, A6000): Up to 32B quantized, 13B full precision

• Data Center (A100, H100): 70B+ models, multiple GPUs for 400B+ models

Access and Deployment

Hugging Face Hub

All models listed are available at: https://huggingface.co/

Popular Model Repositories:

• Meta: meta-llama/

• Qwen: Qwen/

• AI2: allenai/

• NVIDIA: nvidia/

• Mistral: mistralai/

• Google: google/

AI2 Resources

• Playground: https://playground.allenai.org/

• Model Downloads: https://allenai.org/olmo

• Documentation: Available with each model release

• Hugging Face Collection: https://huggingface.co/allenai

NVIDIA Nemotron Resources

• Developer Portal: https://developer.nvidia.com/nemotron

• Hugging Face Models: https://huggingface.co/nvidia

• NeMo Framework: https://github.com/NVIDIA/NeMo

• Technical Reports: https://research.nvidia.com/labs/adlr/nemotronh/

• Pre-training Datasets: https://huggingface.co/collections/nvidia/nemotron-pretraining-datasets

Deployment Tools

• Transformers: Standard HuggingFace library

• vLLM: High-throughput LLM serving

• TGI (Text Generation Inference): HuggingFace inference server

• Ollama: Local model running (Mac/Linux/Windows)

• LlamaCPP: CPU-optimized inference

• ExecuTorch: Mobile deployment (iOS/Android)

Evaluation Leaderboards

LLM Leaderboards

• Open LLM Leaderboard: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

• LMSys Chatbot Arena: https://chat.lmsys.org/

• OLMES (OLMo Evaluation): 20 benchmark suite for core capabilities

VLM Leaderboards

• Open VLM Leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard

• Vision Arena: https://huggingface.co/spaces/WildVision/vision-arena

Key Benchmarks

• MMLU: Massive Multitask Language Understanding (knowledge)

• HumanEval: Code generation

• MATH: Mathematical reasoning

• GSM8K: Grade school math

• BBH: Big Bench Hard (reasoning)

• GPQA: Graduate-level questions

• IFEval: Instruction following

Recent Developments (2024-2025)

Notable Trends

1. Efficiency Focus: Smaller models (1B-7B) achieving near-parity with larger predecessors

2. Hybrid Architectures: Mamba-Transformer models achieving 3-6× speedups with equal/better accuracy

3. Long Context: Models extending to 128K-1M token contexts

4. Multimodal Integration: VLMs becoming standard, not specialized

5. True Openness: More fully open models (OLMO, Qwen, Nemotron) vs just open weights

6. Reasoning Models: Explicit chain-of-thought becoming standard feature

7. Edge Deployment: Aggressive quantization enabling mobile/IoT deployment

Major Releases

• Llama 3.2 (Sept 2024): First Llama with 1B/3B models and vision capabilities

• OLMo 2 (Nov 2024): Outperforming larger models with full openness

• OLMo 3 (Nov 2024): First fully open 32B reasoning model with OlmoTrace

• Nemotron-H (March 2024): First large-scale hybrid Mamba-Transformer (up to 56B)

• Nemotron Nano 2 (Aug 2024): Configurable reasoning modes, 6× throughput

• Nemotron 3 Nano (Jan 2025): Hybrid MoE with 1M context window

• Qwen2.5-VL (Jan 2025): Hour-long video understanding, computer use

• DeepSeek V3 (Dec 2024): Efficient 671B MoE architecture

• SmolVLM (2025): 2B VLM for edge deployment

License Information

Common Licenses

• Apache 2.0: Fully open, commercial use (OLMO, Qwen)

• Llama Community License: Restrictive at scale, commercial use allowed under threshold

• MIT: Permissive open source

• CC BY-NC: Non-commercial use only

Always check the specific model card for license details before commercial deployment.

Additional Resources

Documentation

• Hugging Face Docs: https://huggingface.co/docs

• Transformers: https://huggingface.co/docs/transformers

• PEFT (LoRA, QLoRA): https://huggingface.co/docs/peft

• TRL (Training): https://huggingface.co/docs/trl

Communities

• Hugging Face Forums: https://discuss.huggingface.co/

• AI2 Discord: Via allenai.org

• r/LocalLLaMA: Reddit community for local deployment

Papers and Technical Reports

• OLMo Technical Reports: Available at allenai.org/olmo

• Qwen Papers: Available with model releases

• Llama Papers: ai.meta.com

Document Information

Last Updated: January 3, 2025
Data Sources: Hugging Face Hub, AI2 official releases, NVIDIA Nemotron documentation, model cards, technical reports
Maintained By: Community reference - verify current specs before deployment

Note: Model capabilities and sizes are rapidly evolving. This guide now includes NVIDIA's hybrid Mamba-Transformer models which represent a significant architectural advancement. Always check the official model card on Hugging Face or the provider's website for the most current information.

This document is provided as a reference guide. Model availability, performance characteristics, and specifications may change. Always verify against official sources before making deployment decisions.