Concept 1 / 1
Key Concepts to Memorize
2 main research topics:
1. Learning mathematical models for dynamics from data
2. Mathematical analysis of Transformers
Dynamical system (formal definition):
- A pair (X, f) where X is a vector space (state space) and f: X → X is a function
- State space X = all possible states the system can be in
- Function f = maps current state to next state
- Problem: learning from noisy data alone → predictions must NOT escape to infinity
Absorbing sets / Trapping regions:
- A subset of the state space that every model simulation will eventually enter and never leave
- Fantuzzi's lab learns polynomial velocity vectors with guaranteed absorbing sets
- Ensures physical plausibility of learned dynamical models
Transformers — mathematical view:
- GPT = Generative Pre-trained Transformer
- A transformer is a function mapping finite subsets of ℝᵈ → finite subsets of ℝᵈ
- Built by composing 3 types of layers:
1. Feed-Forward layers (multi-layer perceptrons)
2. Self-Attention layers — the key feature of transformers
3. Normalization layers
- Parameters learned during training process
How transformers work geometrically:
- Transformer blocks "move points in space"
- Tokens cluster together as they pass through layers
- Clustering = "context" in language models (words with similar meaning cluster)