Activation Function Comparator

Compare sigmoid, ReLU, tanh, and other activation functions side by side

Overview

The Activation Function Comparator lets you visualize and compare popular neural network activation functions. Examine their shapes, derivatives, output ranges, and understand when to use sigmoid, tanh, ReLU, Leaky ReLU, ELU, Swish, or GELU.

Tips

  1. Compare derivatives alongside the functions to understand gradient flow - notice how sigmoid/tanh derivatives vanish at extremes while ReLU maintains constant gradient
  2. Look at the behavior near zero where most activations differ - ReLU has a sharp corner, while Swish and GELU are smooth
  3. Check the output ranges - sigmoid (0,1) for probabilities, tanh (-1,1) for centered outputs, ReLU [0,∞) for unbounded positive values
  4. Use ReLU as your default for hidden layers and only switch to alternatives (Leaky ReLU, GELU) when you encounter specific problems like dying neurons
  5. Match output layer activation to your problem - sigmoid for binary classification, softmax for multi-class, linear for regression