The context window is how much text an AI can consider at once — your prompt, the conversation, and any documents. It's why chats 'forget', and why long context eats VRAM. Explained simply.
An AI's context window is the maximum amount of text it can consider at one time — everything it's 'looking at' to generate a response: your current prompt, the earlier conversation, and any documents or code you've pasted in. It's measured in tokens (roughly ¾ of a word each), and every model has a limit — common ones are 8K, 32K, or 128K tokens. The context window is why a long chat eventually starts to 'forget' earlier messages (they fall outside the window), and, for local AI, it's why a big context uses more VRAM. Here's how it works, in plain English.
Why chats 'forget' — and why context costs VRAM
Two practical consequences flow from the context window. First, 'forgetting': an AI has no memory beyond its context window, so once a conversation grows past the limit, the oldest messages drop out and the model genuinely can't see them anymore — it's not being forgetful, it literally no longer has that text in view. That's why very long chats lose track of things you said at the start. Second, and crucial for running AI locally: a bigger context window uses more VRAM. Beyond the model's weights, the AI keeps a 'KV cache' — a running memory of the tokens in context — and that cache grows with the length of your context. So a model that just fits your GPU at a short context can run out of memory on a very long one. This is why our VRAM guidance says to leave headroom: the model weights are the baseline, and context is the variable that eats into whatever VRAM is left. It's also why the VRAM calculator asks about context — it changes the real requirement.
The context window is everything the AI can 'see' at once — and on local hardware, more of it costs VRAM. · Unsplash
How much context do you actually need?
Match the context window to what you're doing, because bigger isn't automatically better — it costs VRAM and can slow things down. For normal chat and quick questions, a modest context (8K or so) is plenty; you're not going to fill it. For working with long documents — analyzing a contract, chatting with your files via RAG, or feeding in a big codebase — you want a larger window (32K, 128K, or more) so the whole thing fits in view. The trade-off on local hardware is real: if you crank the context length on a model that barely fits your VRAM, it may run out of memory or slow down, so it's often better to run a slightly smaller model with a comfortable context than a bigger one starved for context room. A useful mental model: the model's intelligence comes from its size, but its attention span comes from the context window — and on your own GPU, you're paying VRAM for both. Set the context to what your task needs, and no more.
Quick answers
What is a context window in AI?
A context window is the maximum amount of text an AI model can consider at one time — including your current prompt, the earlier conversation, and any documents or code you've pasted in. It's measured in tokens (each roughly three-quarters of a word), and every model has a limit, commonly 8K, 32K, or 128K tokens. The context window defines what the AI can 'see' when generating a response; anything outside it is invisible to the model, which is why very long conversations start to lose track of earlier messages.
Why does AI 'forget' earlier parts of a conversation?
Because of the context window. An AI has no memory beyond the text currently in its context window, so once a conversation grows past the token limit, the oldest messages fall out of the window and the model literally can no longer see them. It's not forgetfulness in a human sense — the text is simply no longer in view. This is why long chats lose track of things said at the start. Some tools work around it by summarizing older messages or using retrieval (RAG) to pull relevant past content back into context.
Does context window size affect VRAM on local AI?
Yes, significantly. Beyond the model's weights, running an AI keeps a 'KV cache' — a running memory of the tokens in context — and that cache grows with the context length. So a larger context window uses more VRAM. A model that just fits your GPU at a short context can run out of memory at a very long one. That's why you should leave VRAM headroom and match context length to your actual need: use a big window only when working with long documents, and a modest one for regular chat.
The context window is the AI's attention span — everything it can see at once, measured in tokens, and paid for in VRAM on local hardware. Set it to your task, leave headroom, and use the VRAM calculator which factors context in. For working with long documents, that's what local RAG is for. Related: how much VRAM you need.
