Transformer

A form of Generative Models that exist to take an input token, and generate a distribution of output tokens that can be repeatedly sampled to continue generation.

Structure

1. Text Embeddings
2. Positional Encoding

Concepts

• Anthropic Definition of Transformer
• BPE
• Look Up Table Matrix
• Transformer Block
• Residual Stream
• Causal Attention
• Autoregressive
• Layer Normalization