mej7.comHow AI Works
EN SHQIP
Beginner Intermediate Advanced Mechanistic Interpretability
ADVANCED TRACK

Past the visuals — into the real mechanics

Backpropagation, attention as code, KV-cache, MoE routing, RLHF, quantization, and interpretability — with interactive diagrams at simple, technical, and research depth.

Think of this track as opening the hood: you still get pictures, but each picture is tied to an equation or a few lines of real training code.
BACKPROP

Chain rule on a tiny graph

Training adjusts weights by flowing loss gradients backward through every multiply and activation — exactly what autograd records during the forward pass.

loss.backward()  # ∂L/∂W accumulates via chain rule
ATTENTION

Self-attention, line by line

Step through the same operations PyTorch-style frameworks implement inside each transformer block.


          

          

            
            
          

          

        

      

    


    

      

        

          
MATH

          
Scaled dot-product scores

          
Each row is a token asking how much it should read from every position — softmax turns scores into weights.

        

        

          

        

      

    


    

      

        

          
KV CACHE

          
Why caching keys & values matters

          
During generation, past tokens' K and V don't change — recomputing them every step would waste compute. KV-cache stores them so each new token only runs one layer pass.

        

        

          
          
          
Compute saved (illustrative)0%

          
KV memory (per layer, schematic)16× d_model

          
        

      

    


    

      

        

          
IO

          
FlashAttention intuition

          
Attention is memory-bound on long sequences. Tiled kernels fuse softmax with matmul so SRAM holds working blocks instead of materializing the full N×N matrix.

        

        

          
        

      

    


    

      

        

          
MOE

          
Mixture-of-Experts routing

          
Only a few expert FFNs fire per token. A router softmax picks top-k experts — huge parameter count, moderate compute.

        

        

          
          
          

        

      

    


    

      

        

          
QUANT

          
Low-bit weights

          
INT8/INT4 shrinks memory and bandwidth; the curve gets stair-steps as precision drops.

        

        

          
          
          
Memory vs FP16

          
Typical error (illustrative)

          
        

      

    


    

      

        

          
ALIGNMENT

          
From base model to helpful assistant

          
Training often stacks: pretrain on text → fine-tune on instructions → reward model scores answers → RLHF nudges the policy toward human-preferred responses.

        

        

          

          

          
        

      

    


    

      

        

          
PEFT

          
LoRA: low-rank adapters

          
Freeze W, train small matrices A·B so ΔW ≈ BA — millions of trainable params instead of billions.

        

        

          

            W (frozen)+B @ A→W′
          

          
h = W @ x + (B @ A) @ x   # rank r ≪ d

        

      

    


    

      

        

          
INTERP

          
Circuits & mechanistic hints

          
Advanced interpretability means causal tests: patch activations, ablate heads, measure logits — not just heatmaps.

        

        

          
The Mech Interp track has the playground — attention map, value mixing, residual stream, and 3D embeddings. Open it after you understand attention math here.

          
        

      

    


    

      

        

          
Need the visual lab?

          
Tokens, embeddings, NN lab, and generation demos live in the intermediate track.