Skip to main content

Activation Function

Imagine you're voting on whether to go to a party. Each friend gives you a reason (input), and you weight how important each reason is. But you don't just add up the weighted reasons — you apply a decision rule: "If the total score is above 7, I'll go. Otherwise, I won't." That decision rule is like an activation function. Without it, the neural network would just be a series of linear equations (addition and multiplication), which can only learn straight-line relationships. Activation functions introduce the "decision rules" that let the network learn complex, non-linear patterns. Common activation functions include: ReLU: "If positive, keep it. If negative, make it zero." Sigmoid: "Squish the output between 0 and 1." Tanh: "Squish the output between -1 and 1."

The Simple Version

Imagine you're voting on whether to go to a party. Each friend gives you a reason (input), and you weight how important each reason is. But you don't just add up the weighted reasons — you apply a decision rule: "If the total score is above 7, I'll go. Otherwise, I won't." That decision rule is like an activation function. Without it, the neural network would just be a series of linear equations (addition and multiplication), which can only learn straight-line relationships. Activation functions introduce the "decision rules" that let the network learn complex, non-linear patterns. Common activation functions include: ReLU: "If positive, keep it. If negative, make it zero." Sigmoid: "Squish the output between 0 and 1." Tanh: "Squish the output between -1 and 1."

Detailed Explanation

Activation functions are applied after each linear transformation (weights × inputs + bias) in a neural network. They determine whether a neuron should "fire" (activate) based on its input. Why Non-Linearity Matters: Without activation functions, a neural network with multiple layers is mathematically equivalent to a single-layer network. No matter how many layers you add, the network can only learn linear relationships. Activation functions break this limitation, enabling the network to approximate any function (Universal Approximation Theorem). Common Activation Functions: ReLU (Rectified Linear Unit): Most widely used activation function Simple and computationally efficient Problem: "Dying ReLU" — neurons can get stuck outputting 0 Used in: Most CNNs, feedforward networks GELU (Gaussian Error Linear Unit): Smooth approximation of ReLU Better gradient flow than ReLU Used in: Transformers (BERT, GPT, Llama) SiLU / Swish: Self-gated activation function Smooth, non-monotonic Used in: EfficientNet, some modern architectures Sigmoid: Outputs values between 0 and 1 Problem: Vanishing gradients for large inputs Used in: Binary classification output layers, LSTM gates Tanh (Hyperbolic Tangent): Outputs values between -1 and 1 Zero-centered (better than sigmoid for hidden layers) Problem: Still suffers from vanishing gradients Used in: RNNs, LSTM gates Leaky ReLU: Fixes dying ReLU problem Small slope for negative inputs Used in: When ReLU dying is a concern Softmax: Converts logits to probability distribution Outputs sum to 1 Used in: Multi-class classification output layers Choosing Activation Functions: Use Case — Recommended Activation Hidden layers (CNNs, feedforward) — ReLU or GELU Transformers — GELU Binary classification output — Sigmoid Multi-class classification output — Softmax RNNs / LSTMs — Tanh (hidden), Sigmoid (gates) When ReLU dying is a problem — Leaky ReLU or GELU

Code Example

# Common activation functions in PyTorch
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

# Create sample input
x = torch.linspace(-5, 5, 100)

# 1. ReLU
relu = nn.ReLU()
y_relu = relu(x)

# 2. GELU
gelu = nn.GELU()
y_gelu = gelu(x)

# 3. Sigmoid
sigmoid = nn.Sigmoid()
y_sigmoid = sigmoid(x)

# 4. Tanh
tanh = nn.Tanh()
y_tanh = tanh(x)

# 5. Leaky ReLU
leaky_relu = nn.LeakyReLU(negative_slope=0.1)
y_leaky = leaky_relu(x)

# Neural network with activation functions
class SimpleNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.activation = nn.GELU()  # Activation after linear layer
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 1)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)  # Non-linearity introduced here
        x = self.fc2(x)
        x = self.activation(x)
        x = self.fc3(x)
        return x  # No activation on output (for regression)

# For classification, use sigmoid or softmax on output
class ClassificationNetwork(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.activation = nn.ReLU()
        self.fc2 = nn.Linear(64, num_classes)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)
        x = self.fc2(x)
        x = torch.softmax(x, dim=1)  # Softmax for multi-class
        return x

print("Activation functions demonstrated successfully")

Key Characteristics

  • Non-Linearity: Enables learning of complex patterns
  • Differentiability: Must be differentiable for backpropagation
  • Range: Determines output range (e.g., ReLU: [0,∞), Sigmoid: [0,1])
  • Computational Cost: Varies by function (ReLU is cheap, GELU is more expensive)
  • Gradient Flow: Affects how well gradients propagate during training

Why It Matters

While activation functions are a technical detail, understanding them helps interpret model behavior and training dynamics: Why It Matters: Model Performance: Choice of activation affects learning capability Training Stability: Poor activation choice can cause vanishing/exploding gradients Compute Efficiency: Simpler activations (ReLU) are faster than complex ones (GELU) Architecture Decisions: Different architectures use different activations (Transformers use GELU) Enterprise Implications: Model Selection: Understand what activations models use (affects performance) Custom Models: Choose appropriate activations for your architecture Debugging: Activation-related issues (dying ReLU, vanishing gradients) can be diagnosed and fixed Optimization: Activation choice impacts inference speed

Real-World Analogy

A bouncer at a club. The bouncer decides who gets in based on their input (appearance, ID, etc.). Different bouncers have different rules: ReLU bouncer: "If you look over 21, you're in. Otherwise, you're out." Sigmoid bouncer: "I'll give you a probability of getting in, between 0% and 100%." Tanh bouncer: "I'll rate you from -100% to +100%." The bouncer's rule (activation function) determines how inputs are transformed into outputs.

Common Misconceptions

  • Myth: ReLU is always the best activation function.
  • Reality: ReLU works well for many tasks, but Transformers use GELU for better performance. The best activation depends on the architecture and task.
  • Myth: Activation functions don't matter much.
  • Reality: Activation functions are critical for learning. Poor choices can lead to vanishing gradients, dying neurons, or inability to learn complex patterns.
  • Myth: More complex activation functions are always better.
  • Reality: Simple activations (ReLU) are often sufficient and computationally efficient. Complex activations (GELU) provide marginal gains at higher computational cost.
  • Myth: You need different activations for every layer.
  • Reality: Most networks use the same activation throughout (e.g., ReLU for all hidden layers). Only the output layer typically uses a different activation (softmax for classification).

Related Terms

Sources & Further Reading