How Generative AI Works: From Training Data to Output

The whole process at a glance

Every Generative AI system, whether it writes text, draws images or produces code, follows the same broad shape. A model is trained once, at great expense, on a large collection of examples. After that, the trained model is used many times to generate new output from prompts. Most of the confusion about Generative AI comes from mixing up these two phases: training is where the model learns, and generation is where it applies what it learned without learning anything new from your conversation.

The table below follows a text model from raw data to the reply you read. The rest of this guide explains each row, first in plain terms for beginners, then in the detail intermediate learners need to reason about behaviour, and finally in the depth advanced learners need to build and operate real systems.

Beginner: a model learns patterns, not a list of answers

Think about how you learned to finish the sentence “Twinkle, twinkle, little…”. Nobody gave you a rule; you heard the pattern often enough that the next word feels obvious. A generative model learns in a similar, though entirely mathematical, way. It is shown a piece of text with the next word hidden, it guesses, its guess is compared with the real word, and its internal numbers are nudged so that the right word becomes a little more likely next time. Repeat that trillions of times across a huge range of text and the model becomes very good at continuing text in a sensible way.

Those internal numbers are called parameters or weights. A large language model has billions of them. They are not a database of sentences: you cannot open a model and find a stored copy of a particular web page. Instead, the parameters encode statistical patterns: which words tend to follow which, how grammar works, how a recipe is usually laid out, how Python code is structured, and a great deal of factual association along the way.

This explains two things beginners notice straight away. First, a model can write something new that never appeared in its training data, because it combines patterns rather than copying. Second, it can state something false with complete confidence, because it produces what is likely given the patterns, not what has been checked to be true.

• Training data: the examples a model learns from, such as books, websites, code and image–caption pairs.
• Parameters (weights): the adjustable numbers that store what the model has learned.
• Training: repeatedly comparing the model’s guesses with real data and adjusting the parameters to reduce the error.
• Inference or generation: using the finished model to produce output; the parameters do not change while you chat with it.

Beginner: tokens and next-token prediction

Text models do not read letters or whole words. They read tokens, which are common words or pieces of words. “Generative” might be one token or split into “Gener” and “ative”, depending on the model’s tokenizer. As a rough guide, a token in English is often three or four characters, and languages such as Hindi or Telugu frequently need more tokens for the same meaning, which matters because cost and length limits are counted in tokens.

When you send a prompt, the model calculates a probability for every token in its vocabulary of being the next one. The table shows illustrative numbers for the prompt “Hyderabad is the capital of”. One token is chosen, appended to the text, and the model runs again on the slightly longer text to predict the token after that. A long answer is simply this loop repeated hundreds of times. The model never plans the whole reply in advance in the way a person drafts an essay; everything it writes is produced one token at a time.

The example also shows how training data shapes answers. Hyderabad was the shared capital of Andhra Pradesh and Telangana until June 2024, so text written before then often pairs it with Andhra Pradesh. A model trained mostly on older text may still give that answer some probability. This is why a model’s knowledge has a cut-off date and why recent facts should come from a source you provide.

Intermediate: embeddings, attention and the transformer

Inside the model, each token is converted into an embedding: a list of hundreds or thousands of numbers that places the token in a space where related meanings sit close together. The embedding for “bank” starts out the same in every sentence, so the model needs a way to tell a river bank from a savings bank. That is the job of attention.

Attention, introduced as the core of the transformer architecture, lets every token look at the other tokens in the context and decide how much each one matters. In “I deposited cash at the bank”, the vector for “bank” pulls in information from “deposited” and “cash”. Technically, each token produces a query, a key and a value. Comparing one token’s query with every key gives relevance scores; these are turned into weights that add up to one, and the token’s new representation is the weighted mix of the values. A language model also applies a mask so that each position can attend only to earlier tokens, because it must predict the future without seeing it.

A transformer stacks many such layers, each with several attention heads running in parallel and a small feed-forward network after them. Early layers tend to capture local patterns such as word pieces and grammar; later layers build more abstract features. After the final layer, the vector at the last position is converted into scores for every token in the vocabulary, which become the probabilities you saw in the previous section.

The code below implements scaled dot-product attention with NumPy for four tokens. Real models learn the projection matrices during training and run this across many heads and layers, but the arithmetic is the same.

import numpy as np

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)  # subtract the max for numerical stability
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V):
    """Scaled dot-product attention, as described in the transformer paper."""
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)       # relevance of every token to every other token
    weights = softmax(scores, axis=-1)  # each row now sums to 1
    return weights @ V                  # a weighted blend of the value vectors

rng = np.random.default_rng(0)
x = rng.normal(size=(4, 8))            # 4 tokens, each an 8-number embedding
Wq, Wk, Wv = (rng.normal(size=(8, 8)) for _ in range(3))
out = attention(x @ Wq, x @ Wk, x @ Wv)
print(out.shape)                       # (4, 8): one updated vector per token

Sources: Vaswani et al.: Attention Is All You Need. Checked 23 October 2026.

Intermediate: from a text predictor to a helpful assistant

A model that has only been pretrained is a powerful autocomplete. Ask it a question and it may continue with three more questions, because that is a plausible continuation of a list of questions on a web page. The assistant you use in a chat app has been through further training stages that shape how it responds.

Each stage changes behaviour, not just knowledge. That is why two models trained on similar data can feel very different to use, and why a provider can release an updated version of a model that answers the same prompt differently. When you build on a model, record the exact version you tested.

• Fine-tuning teaches behaviour and format well, but it is an unreliable way to add facts that change; retrieval is usually better for those.
• Safety behaviour, such as declining harmful requests, is learned in these later stages and can be imperfect, so applications still need their own controls.

Intermediate: how the next token is actually chosen

The model produces probabilities; a separate decoding step decides which token to use. Always taking the most likely token, called greedy decoding, gives repeatable output but can be flat or get stuck repeating itself. Sampling picks a token at random in proportion to its probability, which gives more varied and natural text.

Two settings control sampling. Temperature divides the model’s raw scores before they become probabilities: a value below one sharpens the distribution towards the top tokens, and a value above one flattens it so unlikely tokens are chosen more often. Top-p, or nucleus sampling, keeps only the smallest group of top tokens whose probabilities add up to p and samples from those, which cuts off the long tail of unlikely tokens. Use a low temperature for extraction, classification and code; use a higher one for brainstorming and creative drafts.

The function below shows both settings. Running the same prompt several times with sampling switched on is the quickest way to see why a model can give different answers to the same question, and why tests for Generative AI applications should run each case more than once.

import numpy as np

def sample_next(logits, temperature=1.0, top_p=1.0, rng=None):
    """Choose the index of the next token from the model's raw scores (logits)."""
    if temperature == 0:
        return int(np.argmax(logits))              # greedy: always the top token
    rng = rng or np.random.default_rng()
    z = np.asarray(logits, dtype=float) / temperature
    probs = np.exp(z - z.max())
    probs /= probs.sum()
    order = np.argsort(probs)[::-1]               # most likely first
    before = np.cumsum(probs[order]) - probs[order]
    kept = order[before < top_p]                  # the nucleus: top tokens up to top_p
    p = probs[kept] / probs[kept].sum()
    return int(rng.choice(kept, p=p))

logits = [4.0, 1.5, 0.3, -1.0]
print(sample_next(logits, temperature=0))         # always 0
print(sample_next(logits, temperature=1.2, top_p=0.9))

How image generation works: diffusion, GANs and VAEs

Most current image generators use diffusion. During training, real images have random noise added to them in many small steps until nothing recognisable is left, and a neural network learns to predict the noise that was added at each step. To generate, the process runs in reverse: start from pure noise and remove a little predicted noise at a time until an image emerges. A text encoder turns your prompt into vectors that steer every denoising step, so the image moves towards something that matches the description.

Many systems run this process in a compressed latent space rather than on full-resolution pixels, which makes it much faster, and then decode the result into an image. More denoising steps usually improve detail at the cost of time. The same idea extends to audio and video, where the model must also keep output consistent over time.

Two older families are still worth knowing. A generative adversarial network (GAN) trains a generator to produce images and a discriminator to tell them from real ones, each improving against the other; GANs can be fast but are harder to train. A variational autoencoder (VAE) learns to compress data into a smooth latent space and decode it back; VAEs are often used as the compression step inside latent diffusion systems.

Sources: Ho et al.: Denoising Diffusion Probabilistic Models; Goodfellow et al.: Generative Adversarial Networks; Kingma and Welling: Auto-Encoding Variational Bayes. Checked 23 October 2026.

Advanced: context windows, latency and cost

The context window is the maximum number of tokens the model can consider at once, covering the system instructions, the conversation so far, any documents you include and the reply being generated. Anything outside it has no influence. In a long conversation, an application has to decide what to keep, summarise or drop, and that choice changes the answers.

Generation has two phases with different performance. In the prefill phase, the model processes the whole prompt at once, which can be done in parallel and largely determines the time to the first token. In the decode phase, it produces output one token at a time, which sets the speed at which text streams out. To avoid recomputing attention for earlier tokens at every step, inference engines keep their keys and values in a KV cache; this is fast but uses a lot of memory, which is one reason long contexts are expensive.

Hosted APIs usually charge per input and output token, and many price output tokens higher than input tokens. Attention cost grows with context length, and models are often less reliable at using information buried in the middle of a very long prompt. In practice, a short, well-chosen context usually beats pasting everything in: it is cheaper, faster and more accurate.

• Measure time to first token and tokens per second separately; they have different causes and different fixes.
• Estimate cost per completed task, including retries and failed attempts, rather than cost per request.
• Reuse stable prompt prefixes where your provider supports caching, and keep variable content at the end.
• Test with realistic context lengths, because behaviour on a short demo prompt does not predict behaviour on a full document.

Why generative models get things wrong

Once you understand next-token prediction, the common failures stop being mysterious. A model is optimised to produce plausible continuations. When it lacks the information, the most plausible continuation of a confident question is often a confident answer, so it may invent a statistic, a citation or an API function. This is usually called hallucination, and lowering the temperature reduces randomness but does not make the model know something it does not.

Other failures have equally clear causes. Knowledge stops at the training cut-off. Patterns in the training data, including social biases and outdated practices, reappear in output. Tokenization makes some tasks, such as counting letters or doing exact arithmetic on long numbers, harder than they look. And because instructions and data arrive through the same channel, text inside a document can sometimes be read as an instruction.

Advanced: grounding with retrieval and tools

Real applications rarely send a user’s question straight to a model. Retrieval-augmented generation (RAG) first searches a document collection for passages relevant to the question, usually by comparing embeddings, and places the best passages in the prompt with an instruction to answer only from them and cite them. The model’s language ability is used to read and summarise, while the facts come from documents you control and can update without retraining.

Tool use extends the same idea. The model is given descriptions of functions it may call, such as a search, a database query or a calculator. Instead of answering directly, it can produce a structured request to call a tool; the application runs the tool and feeds the result back, and the model continues. An agent is this loop repeated until the task is done. In every case the model still only generates text; the application decides what is actually executed, which is where permissions, validation and approval belong.

When grounded systems fail, the cause is often retrieval rather than the model: the right passage was never found, so the model had nothing correct to work from. Measure retrieval and answer quality separately so you know which part to fix.

• Ask for citations to retrieved passages and check that each cited passage supports the claim.
• Include unanswerable questions in your tests; a good system says it does not know.
• Validate structured output, such as JSON, in code before acting on it.
• Log the retrieved passages and tool calls for every answer so that failures can be traced.

Sources: Lewis et al.: Retrieval-Augmented Generation; Anthropic: prompt engineering overview. Checked 23 October 2026.

Advanced: risks that follow from how the technology works

Several risks are specific to Generative AI because of the mechanics described above. Prompt injection exploits the shared channel for instructions and data: a web page or email can contain text that tries to redirect the model. Sensitive information can leak if it is placed in prompts, logs or training data without controls. Excessive agency arises when a model is allowed to take actions, such as sending messages or changing records, with too little oversight.

Treat retrieved content and tool output as untrusted, give each tool the narrowest permissions it needs, keep secrets out of prompts, require human approval for consequential actions and monitor outputs in production. Risk-management frameworks help teams decide which risks to accept, which to reduce, and how to keep checking as models and data change.

Sources: OWASP: Top 10 for LLM Applications; NIST: AI Risk Management Framework. Checked 23 October 2026.

What to practise at each level

Understanding how Generative AI works is most useful when you test it yourself. Each level below has a short exercise that turns the ideas in this guide into something you have observed.

Common questions

Continue learning

• Generative AI Training in Hyderabad
• Types of Generative AI
• Generative AI models explained
• Generative AI roadmap
• Generative AI chatbots explained