Graphics Processing Unit (GPU)

Imagine you need to add up 10,000 numbers. A regular computer processor (CPU) is like a genius mathematician who can only do one calculation at a time. They're incredibly smart and fast at complex problems, but they have to work through the 10,000 numbers one by one. A GPU is like 10,000 simple calculators working together. Each calculator isn't as smart as the CPU, but because they all work at the same time, they finish the job in a fraction of the time. AI models are essentially massive mathematical operations (matrix multiplications) that need to be done millions of times. GPUs, originally designed to render video game graphics (which also requires thousands of parallel calculations), turned out to be perfect for AI. This accidental synergy is why NVIDIA, a gaming graphics card company, became the most valuable chip company in the world.

The Simple Version

Detailed Explanation

GPUs revolutionized AI by providing the parallel compute power needed to train deep neural networks. While CPUs excel at sequential, complex tasks, GPUs excel at simple, repetitive tasks done in parallel. Why GPUs Work for AI: Parallel Architecture: Modern GPUs have thousands of small, efficient cores (e.g., NVIDIA H100 has 16,896 CUDA cores) AI operations (matrix multiplications) are inherently parallelizable GPUs can process entire batches of data simultaneously Memory Hierarchy: High-bandwidth memory (HBM) provides fast access to large datasets VRAM (Video RAM) stores model weights and intermediate activations Modern GPUs have 40-80GB of VRAM (A100: 80GB, H100: 80GB, B200: 192GB) Specialized Instructions: Tensor Cores (NVIDIA) accelerate matrix operations critical for deep learning Mixed-precision training (FP16, BF16, INT8) doubles or quadruples throughput Generations of AI GPUs: NVIDIA (Market Leader): V100 (2017): First GPU with Tensor Cores, enabled modern deep learning A100 (2020): 3x performance over V100, 80GB VRAM, workhorse of GPT-3 training H100 (2023): 3x over A100, Transformer Engine, powered GPT-4 and modern LLMs B200 (2024): 2.5x over H100, 192GB VRAM, current frontier Blackwell Ultra (2025-2026): Next-generation architecture Competitors: AMD (MI300X, MI325X): Growing market share, competitive performance, more affordable Google TPU (Tensor Processing Unit): Custom ASIC for AI, used internally and in Google Cloud Intel (Gaudi, Arc): Enterprise-focused, competitive pricing AWS Trainium/Inferentia: Amazon's custom AI chips for cloud customers GPU vs. TPU vs. CPU: Hardware — Best For — Strengths — Weaknesses CPU — General computing, small models — Versatile, handles complex logic — Slow for AI workloads GPU — Training & inference, LLMs — Massive parallelism, mature ecosystem — Expensive, power-hungry TPU — Large-scale training (Google) — Optimized for TensorFlow, cost-effective at scale — Less flexible, Google Cloud only Key GPU Metrics for AI: TFLOPS: Trillions of floating-point operations per second (raw compute) VRAM: Video RAM capacity (determines max model size) Memory Bandwidth: GB/s (critical for large model inference) Interconnect: NVLink, InfiniBand (for multi-GPU scaling)

Code Example

# Checking GPU availability and specs in PyTorch
import torch

# 1. Check if CUDA (NVIDIA GPU) is available
if torch.cuda.is_available():
    print(f"✅ GPU is available!")
    print(f"GPU Name: {torch.cuda.get_device_name(0)}")
    print(f"GPU Count: {torch.cuda.device_count()}")
    
    # 2. Get detailed GPU memory info
    gpu_memory = torch.cuda.get_device_properties(0)
    print(f"Total Memory: {gpu_memory.total_memory / 1e9:.2f} GB")
    print(f"Compute Capability: {gpu_memory.major}.{gpu_memory.minor}")
    
    # 3. Move a model to GPU for training/inference
    model = torch.nn.Linear(1000, 1000).to('cuda')
    
    # 4. Move data to GPU
    input_tensor = torch.randn(32, 1000).to('cuda')  # Batch of 32
    
    # 5. Run inference on GPU (1000x faster than CPU for large batches)
    output = model(input_tensor)
    
    # 6. Monitor GPU memory usage
    print(f"Memory Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
    print(f"Memory Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
    
else:
    print("❌ No GPU available. Using CPU (much slower for AI workloads).")
    # Fallback to CPU
    device = torch.device('cpu')

# Example: Comparing CPU vs GPU performance
import time

# CPU inference
model_cpu = torch.nn.Linear(10000, 10000).to('cpu')
input_cpu = torch.randn(1000, 10000).to('cpu')

start = time.time()
for _ in range(100):
    _ = model_cpu(input_cpu)
cpu_time = time.time() - start

# GPU inference (if available)
if torch.cuda.is_available():
    model_gpu = model_cpu.to('cuda')
    input_gpu = input_cpu.to('cuda')
    
    start = time.time()
    for _ in range(100):
        _ = model_gpu(input_gpu)
    gpu_time = time.time() - start
    
    print(f"\nPerformance Comparison:")
    print(f"CPU Time: {cpu_time:.2f}s")
    print(f"GPU Time: {gpu_time:.2f}s")
    print(f"Speedup: {cpu_time / gpu_time:.1f}x faster on GPU")
    # Typical output: GPU is 50-200x faster for large matrix operations

Key Characteristics

Massively Parallel: Thousands of cores working simultaneously
Memory-Intensive: Requires large VRAM for model weights and activations
Power-Hungry: Modern GPUs consume 300-700W each
Ecosystem-Dependent: NVIDIA's CUDA dominates; alternatives face software challenges
Rapidly Evolving: New generations every 1-2 years with significant performance gains

Why It Matters

GPUs are the critical infrastructure bottleneck for enterprise AI: Why GPUs Matter: Training Cost: Training GPT-4 required ~25,000 A100 GPUs for 90 days (~$100M+) Inference Cost: Serving LLMs at scale requires GPU clusters Supply Chain: GPU shortages have delayed AI projects across industries Competitive Advantage: Access to GPUs determines who can build frontier AI Enterprise GPU Strategies: Cloud GPUs (Pay-as-you-go): AWS, Azure, GCP: Rent A100/H100 instances by the hour Pros: No upfront capital, instant scaling Cons: Expensive at scale, availability issues Cost: $2-5/hour per A100, $5-10/hour per H100 On-Premises GPUs (Capital Investment): Buy GPU clusters (DGX systems, custom builds) Pros: Lower long-term cost, full control, data sovereignty Cons: High upfront cost ($150K-$2M+), maintenance, cooling Best for: High-volume inference, regulated industries Specialized AI Clouds: CoreWeave, Lambda Labs: GPU-focused cloud providers Pros: Better GPU availability, optimized for AI workloads Cons: Less mature than hyperscalers GPU Cost Optimization: Spot Instances: 60-80% cheaper (but can be interrupted) Quantization: INT8/INT4 reduces VRAM requirements Multi-Instance GPU (MIG): Partition one GPU for multiple workloads Serverless GPU: Pay only for inference time (Modal, Replicate) ROI Considerations: Training: One-time cost, amortized over model lifetime Inference: Ongoing operational cost, directly impacts margins Optimization: Proper GPU utilization can reduce costs by 3-5x

Real-World Analogy

A restaurant kitchen. A CPU is like a single master chef — brilliant at complex recipes but can only cook one dish at a time. A GPU is like a kitchen with 10,000 line cooks, each capable of chopping one vegetable. For a simple task (chopping 10,000 onions), the GPU kitchen finishes in seconds while the CPU chef takes hours. But for a complex, multi-step recipe requiring judgment and timing, the CPU chef might be more efficient. AI workloads are like chopping onions — massively parallel, perfectly suited for GPUs.

Common Misconceptions

Myth: More GPUs always mean faster training.
Reality: Multi-GPU training requires careful optimization (data parallelism, model parallelism). Poorly optimized multi-GPU setups can be slower than a single GPU due to communication overhead.
Myth: Any GPU works for AI.
Reality: AI workloads require GPUs with sufficient VRAM (at least 16GB for modern LLMs) and Tensor Cores. Gaming GPUs work but are less efficient than data center GPUs (A100, H100).
Myth: GPUs are only for training, not inference.
Reality: GPUs are critical for both training and inference. While CPUs can handle small models, serving LLMs at scale requires GPUs for acceptable latency and throughput.
Myth: NVIDIA is the only option for AI GPUs.
Reality: While NVIDIA dominates (80%+ market share), AMD MI300X, Google TPUs, and AWS Trainium are viable alternatives, especially for cost-sensitive workloads. The software ecosystem (CUDA) is NVIDIA's main advantage.

Graphics Processing Unit (GPU)

The Simple Version

Detailed Explanation

Code Example

Key Characteristics

Why It Matters

Real-World Analogy

Common Misconceptions

Related Terms

Sources & Further Reading

Graphics Processing Unit (GPU): Direct Definition

Technical Definition

Why It Matters