Neural Network

How a neural network works

Strip away the metaphors and a neural network is layered matrix multiplications. A single layer does:

output = activation(input × weights + bias)

Where input is a vector, weights is a matrix, bias is a vector, and activation is a non-linear function applied element-wise. Stack many such layers and you get a "deep" neural network. The weights are the parameters that get trained; the inputs flow forward through the stack.

Without the non-linearity, stacking layers is mathematically pointless — multiple matrix multiplications collapse into a single matrix. The activation function (ReLU, sigmoid, tanh, GELU) introduces non-linearity that lets the network represent complex functions.

Layers and activations

Activation	Function	Used in	Notes
ReLU	max(0, x)	Hidden layers (default)	Fast, no vanishing gradient on positive side
Sigmoid	1 / (1 + e⁻ˣ)	Binary output, gates in LSTMs	Saturates → vanishing gradient in deep nets
tanh	(eˣ − e⁻ˣ) / (eˣ + e⁻ˣ)	Hidden layers (older networks)	Like sigmoid but centered at 0
GELU	x × Φ(x)	Transformers (BERT, GPT)	Smooth, slightly outperforms ReLU
Softmax	eˣⁱ / Σ eˣʲ	Multi-class output layer	Outputs sum to 1; interpret as probabilities
Linear (none)	x	Regression output, attention scores	For continuous outputs

ReLU has been the default since 2012. Variants (Leaky ReLU, GELU, Swish) outperform plain ReLU by small amounts in some benchmarks; the field has mostly settled on GELU for transformers and ReLU for everything else.

Training — gradient descent + backpropagation

Training adjusts the weights to minimize a loss function. The loss measures how wrong the network is on a given example (e.g., cross-entropy for classification, mean squared error for regression). The procedure:

1. Forward pass. Run input through the network, compute output.
2. Compute loss. Compare output to ground truth.
3. Backward pass. Compute gradient of loss with respect to every weight using the chain rule (this is backpropagation).
4. Update weights. Take a small step in the direction that reduces loss: w ← w − learning_rate × gradient.
5. Repeat over batches of training examples until the loss stops decreasing.

The "small step" is what makes it gradient descent. Take steps too large and you overshoot; too small and training never finishes. Adaptive optimizers (Adam, AdamW) automatically adjust the step size per-parameter based on the gradient's history.

The universal approximation theorem

A theorem from 1989 (Cybenko; Hornik et al.) states that a neural network with a single hidden layer of sufficient width can approximate any continuous function on a compact domain to arbitrary accuracy. Why isn't every network just one wide layer, then?

Because "sufficient width" can mean exponentially many neurons. Deeper networks are exponentially more parameter-efficient at representing certain functions — they exploit hierarchical structure (low-level features compose into mid-level features into high-level concepts). Empirically, depth wins decisively for vision, language, and audio.

Architecture types

Type	Best for	Distinguishing feature	Examples
Multi-layer perceptron (MLP)	Tabular data	Fully connected layers	Generic baseline
Convolutional (CNN)	Images	Translation-invariant filters	ResNet, VGG, EfficientNet
Recurrent (RNN, LSTM, GRU)	Sequences (older)	State carried token-to-token	seq2seq, language models pre-2017
Transformer	Sequences (modern)	Self-attention over all positions	BERT, GPT, T5, Llama
Graph neural network (GNN)	Graph-structured data	Message passing along edges	Drug discovery, social networks
Diffusion model	Image generation	Iterative denoising	DALL-E 2/3, Stable Diffusion

JavaScript: minimal MLP from scratch

function relu(x)        { return Math.max(0, x); }
function reluDeriv(x)   { return x > 0 ? 1 : 0; }
function softmax(arr)   {
  const max = Math.max(...arr);
  const exp = arr.map(v => Math.exp(v - max));
  const sum = exp.reduce((a, b) => a + b);
  return exp.map(v => v / sum);
}

class Layer {
  constructor(nIn, nOut) {
    // Xavier initialization — keeps activations from exploding
    const scale = Math.sqrt(2 / nIn);
    this.W = Array.from({ length: nIn }, () =>
      Array.from({ length: nOut }, () => (Math.random() - 0.5) * 2 * scale)
    );
    this.b = new Array(nOut).fill(0);
  }

  forward(x) {
    this.input = x;
    const out = new Array(this.b.length).fill(0);
    for (let i = 0; i < this.b.length; i++) {
      let sum = this.b[i];
      for (let j = 0; j < x.length; j++) sum += x[j] * this.W[j][i];
      out[i] = relu(sum);
    }
    this.output = out;
    return out;
  }

  // Backward pass — compute gradients and update weights
  backward(dOut, lr) {
    const dIn = new Array(this.input.length).fill(0);
    for (let i = 0; i < this.b.length; i++) {
      const dPreAct = dOut[i] * reluDeriv(this.output[i]);
      this.b[i] -= lr * dPreAct;
      for (let j = 0; j < this.input.length; j++) {
        dIn[j] += dPreAct * this.W[j][i];
        this.W[j][i] -= lr * dPreAct * this.input[j];
      }
    }
    return dIn;
  }
}

class MLP {
  constructor(sizes) {
    this.layers = [];
    for (let i = 0; i < sizes.length - 1; i++) {
      this.layers.push(new Layer(sizes[i], sizes[i+1]));
    }
  }
  forward(x) { return this.layers.reduce((a, l) => l.forward(a), x); }
  // Cross-entropy loss + backward — left as exercise
}

const net = new MLP([784, 128, 64, 10]); // MNIST-shaped
// Then training loop: forward → compute loss → backward → repeat

Production deployments use frameworks (PyTorch, JAX, TensorFlow) that handle GPU parallelism, automatic differentiation, distributed training, and a thousand performance-critical details. Implementing from scratch is purely educational.

Modern scale

Model	Year	Parameters	Trained on
LeNet-5	1998	60K	MNIST handwritten digits
AlexNet	2012	60M	ImageNet (1.2M images)
BERT-Large	2018	340M	3.3B words (Wikipedia + BookCorpus)
GPT-3	2020	175B	~500B words
GPT-4	2023	~1.8T (estimated)	Multi-modal corpus
Llama 3 70B	2024	70B	15T tokens

The trend has been compute-driven — when more compute became available (GPUs from 2012, larger clusters since 2017), the field built bigger models. Scaling laws (Kaplan et al., 2020) showed model size, dataset size, and compute scale together; insufficient on any axis caps performance.

Common pitfalls

• Vanishing gradients in deep networks. Sigmoid and tanh activations have derivatives ≤ 1; through 50 layers the gradient becomes 0.5⁵⁰ ≈ 10⁻¹⁵. Use ReLU, batch normalization, residual connections (ResNet's contribution) — these enable training networks 100+ layers deep.
• Exploding gradients. The opposite — gradients grow without bound through layers, weights diverge. Solutions: gradient clipping, careful weight initialization, smaller learning rate.
• Wrong learning rate. Too high → loss oscillates or diverges. Too low → training takes forever. Practical default — Adam optimizer with lr=3e-4 to 1e-3, learning rate scheduling (cosine decay or warmup-then-decay).
• Data leakage. Test set accidentally contains training examples (or close variants). Network learns to memorize specific examples rather than generalizing — performs perfectly on test, fails in production. De-duplicate carefully; split by user/time, not random shuffle, when applicable.
• Confusing training loss with test loss. Training loss can go to 0 by memorization; test loss reveals actual generalization. Always monitor both; stop training when test loss starts rising even if training loss keeps falling.
• Imbalanced classes silently breaking everything. 99% of fraud examples are non-fraud → network predicts "non-fraud" always, gets 99% accuracy, useless. Use class weights, focal loss, or appropriate metrics (precision/recall, F1, AUC).