Gen AI Fundamentals โ€บ Transformers & Attention

๐Ÿค– What's Inside an LLM โ€” Transformers & Attention

Tokens โ†’ embeddings โ†’ attention โ†’ next token, on repeat

BIG IDEA An LLM is a next-token predictor

Strip away the mystique and a large language model does one thing: given a sequence of tokens, it predicts the probability of the next token. It picks one, appends it, and predicts again โ€” over and over. That loop is called autoregressive generation. Everything impressive an LLM does โ€” answering, coding, reasoning โ€” emerges from doing this next-token guess extremely well, at massive scale.

๐Ÿ”‘ The architecture that made this work well is the Transformer (2017, "Attention Is All You Need"). The GPT in ChatGPT literally stands for Generative Pre-trained Transformer. Claude, Gemini, and Llama are all Transformers too.

DIAGRAM The pipeline, end to end

flowchart TB T["Input text"] --> TK["1 ยท Tokenize
text โ†’ tokens"] TK --> EM["2 ยท Embed
each token โ†’ a vector"] EM --> PE["+ position info"] PE --> BLK subgraph BLK["3 ยท Transformer blocks (stacked ร—N)"] direction TB AT["Self-attention
(tokens look at each other)"] --> FF["Feed-forward
(think about each token)"] end BLK --> LG["4 ยท Logits
a score for every possible next token"] LG --> SM["5 ยท Softmax + sample
pick the next token"] SM --> OUT["Next token"] OUT -.append & repeat.-> TK classDef hi fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class AT,FF,SM hi;
  1. Tokenize โ€” split text into tokens (see tokens).
  2. Embed โ€” turn each token into a vector of numbers that captures meaning, plus its position in the sequence.
  3. Transformer blocks โ€” stacked layers (dozens to 100+) that repeatedly mix information between tokens (attention) and process each token (feed-forward).
  4. Logits โ€” the final layer outputs a score for every token in the vocabulary.
  5. Sample โ€” convert scores to probabilities (softmax) and pick one โ€” how "random" depends on temperature. Append it and go again.

ATTENTION The key trick: tokens look at each other

Attention is what lets the model understand context. When processing a word, the model attends to the other words that matter for its meaning โ€” no matter how far away they are. Take:

"The trophy didn't fit in the suitcase because it was too big."

To resolve "it," the model attends back to "trophy" (not "suitcase"). Attention computes, for every token, how much each other token should influence it, and blends them accordingly. Doing this across many layers and many parallel "attention heads" (each learning different relationships โ€” grammar, references, topic) is what builds rich understanding.

This is why the context window matters and why long prompts get pricey: every token attends to every other token, so cost grows quadratically with sequence length. Longer context = a lot more attention math.

SCALE Why "just predicting the next token" is so powerful

To predict the next token well across the entire internet, a model is forced to learn grammar, facts, translation, code, and reasoning patterns โ€” because all of those help it guess better. Push three dials up together โ€” more parameters (the learned weights, the "B" in "70B"), more data, more compute โ€” and new capabilities appear. That's the "scaling" story behind modern LLMs.

โœ… What the Transformer gave us

  • Handles long-range context via attention
  • Processes a sequence in parallel (fast to train)
  • Scales smoothly with size & data

โš ๏ธ The costs

  • Attention is quadratic in length โ†’ long context is expensive
  • Huge models need huge compute to train & run
  • It predicts plausible text โ€” hence hallucination

TAKEAWAYS

Remember: an LLM generates one token at a time, feeding its own output back in โ€” that's why output is the slow, costly part.
Remember: attention is how the model uses context; it's also why doubling your prompt length more-than-doubles the work.
Remember: the model has no database of facts โ€” knowledge is baked into weights as statistical patterns, which is why it can be confidently wrong (ground it with RAG).

GLOSSARY

TermIn one sentence
TransformerThe neural-network architecture behind modern LLMs, built around attention.
Self-attentionEach token weighs how much every other token should influence it.
Attention headOne of many parallel attention mechanisms, each learning a different kind of relationship.
AutoregressiveGenerating one token at a time, each conditioned on all previous ones.
Parameters / weightsThe billions of learned numbers that store what the model "knows."
Logits โ†’ softmaxRaw next-token scores turned into probabilities to sample from.
EmbeddingA vector representing a token's meaning (also used in RAG).
โžก๏ธ Next: how these weights are created and then used โ€” Training vs Inference.