Type a prompt into DALL-E, Midjourney, or Stable Diffusion and an image appears, seemingly conjured from text. The results range from impressive to unsettling to absurd, often in the same session. But how does a text prompt actually become pixels? The underlying process is counterintuitive and worth understanding — both because it's genuinely interesting and because it explains a lot about why these systems behave the way they do.

The answer, in almost all major image generation systems today, is diffusion.

The Counterintuitive Starting Point

Here's the central idea, which still surprises people when they first hear it: AI image generation starts with random noise.

Not with a blank canvas. Not with a rough sketch. Pure, random, static — like a TV with no signal. And then, step by step, the noise is refined into an image. Each step removes a small amount of noise, guided by the prompt, nudging the random pixels gradually toward something coherent.

The visualization below shows this process. Watch the noise resolve into structure as the steps progress.

Each frame is one denoising step. The model learns to remove noise in a way that's steered by the text prompt — in this case, toward a night scene with mountains and a moon.

What the Model Actually Learned

Training a diffusion model involves a clever trick. Take millions of real images. Gradually add noise to each one — a little at first, then more, until each image is completely unrecognizable static. Then train a neural network to reverse that process: given a noisy version of an image and how much noise was added, predict what the original looked like.

Do this with enough images and enough compute, and the network learns something remarkable: it develops a deep understanding of what images look like. It knows that sky pixels tend to be blue and in the upper half. That edges have consistent orientations. That faces have specific geometric relationships between features. That fur has a particular texture. None of this was programmed — it was learned by repeatedly solving the "denoise this" puzzle across hundreds of millions of training examples.

Once the network can denoise, you can run it in reverse. Start from pure noise and ask: "What noise would I remove if the underlying image were a photograph of a golden retriever in a field?" The network's answer, applied repeatedly over 20–50 steps, converges to an image that actually looks like a golden retriever in a field.

How Text Gets Involved

The diffusion process I described so far produces images, but random ones — there's nothing steering it toward "a golden retriever in a field" rather than "a skyscraper at night." Text conditioning is what makes prompts work.

This is where language models and image models meet. The text prompt is first converted into a numerical representation by an encoder — essentially a compressed mathematical description of the semantic content of your words. That representation is fed into the denoising network at each step, influencing which direction the denoising goes. The network is trained to denoise toward the content described in the text.

In practice, this means the network has learned to associate "sunset" with warm orange and pink tones in the upper portion of an image, "forest" with green texture, "photorealistic portrait" with the specific pixel patterns of high-quality photography. The prompt doesn't directly write pixels — it steers the statistical process that does.

The Role of Randomness

Because you start from random noise, the same prompt generates a different image every time. The random starting point is called the seed, and it determines the specific path the denoising takes through the space of possible images. Photographers and designers who use these tools extensively learn to save seeds for images they like, so they can regenerate the same composition with small variations.

There's also a parameter called guidance scale (or cfg scale in Stable Diffusion) that controls how strictly the model follows the prompt versus how much creative freedom it takes. Low guidance: more varied and sometimes surprising outputs, occasionally drifting from the prompt. High guidance: very literal interpretation of the prompt, but sometimes over-saturated and strange. Most practical prompting lives in the middle range.

Why the Same Models — Different Results?

DALL-E (OpenAI), Midjourney (independent), and Stable Diffusion (Stability AI, open-source) all use diffusion, but they look distinctly different. Why?

  • Training data. Each model was trained on a different dataset. Midjourney is trained heavily on artistic and photographic content, which is why it produces such consistently aesthetic results. Stable Diffusion was trained on LAION-5B, a broad scrape of the internet. The "style" of a model is largely a function of what it saw during training.
  • Architectural variations. Models differ in how they implement the denoising network, how they process text, how many steps they use, and dozens of other choices. DALL-E 3 uses a particularly good text encoder (inherited from GPT-4) which is why it handles complex, detailed prompts better than earlier systems.
  • Fine-tuning and filtering. Every commercial model has been fine-tuned on curated examples and filtered to avoid certain types of output. This shapes the aesthetic range and the refusal behavior of each system.

Latent Diffusion: Why It's Fast Enough to Use

One practical challenge: running diffusion directly on full-resolution pixel grids is enormously expensive. A 512×512 image has 786,432 pixels, and you'd need to run the denoising network across all of them at every step. That's prohibitively slow.

The insight behind latent diffusion (the architecture Stable Diffusion uses, and a key innovation from the Munich-based CompVis lab) is to run the diffusion process in a compressed latent space rather than in pixel space. A separate encoder compresses the image down to a much smaller representation (a latent), the diffusion process runs in that compressed space, and a decoder expands the result back to pixels at the end. This makes the process roughly 10x faster with minimal quality loss. It's why Stable Diffusion can run on consumer-grade hardware while producing results that were science fiction a few years ago.

The Broader Picture

Diffusion models are now the dominant approach not just for images but for audio generation (music, voice), video generation (Sora uses a diffusion-inspired architecture), and even some protein structure prediction. The core idea — learn to denoise, then generate by controlled denoising — has proven remarkably general.

The results sometimes feel like magic, and the implementation details are genuinely sophisticated. But the fundamental logic is traceable and understandable. A network learned what images look like. You start from noise. You denoise, guided by your words. Fifty steps later, you have an image.

The part that remains genuinely mysterious — to researchers as much as to anyone — is why that process sometimes produces images of uncanny quality and coherence, and sometimes produces the famously cursed AI hands or melted text. The model is doing statistics at scale, not reasoning about what an image "should" look like. The impressive cases and the failure cases are both outputs of the same process, and the line between them isn't always predictable in advance.