Context window — why AI forgets what you just said

"I just told you that." If you have ever said this to an AI, you have met the context window. AI does not have a bad memory; it never remembers in the first place. Here is why this happens.

---

Background — not memory, but re-reading

Let's clear up the biggest misconception first. AI does not remember the conversation.

When we ask a third question, the system sends the first question, the first answer, the second question, the second answer, and the third question all over again to the model. Each time, the model reads the entire transcript like someone seeing it for the first time and writes an answer.

So the reason AI feels like it 'remembers' is that it re-reads the whole transcript on every turn. And the size of the container that holds this transcript is the context window .

Anthropic's official documentation clearly distinguishes this from training data and calls it 'working memory' . The knowledge the model learned in training and the content it can reference in this particular conversation are entirely different layers.

---

What goes into the container

What matters here is that the container holds more than the conversation. According to Anthropic's documentation, the following take up the context window.

• The system prompt (instructions the service planted in advance)
• All of the conversation so far
• Attached images, PDFs, and documents
• Tool definitions(the list of functions the AI can use, itself)
• Tool execution results(such as page content fetched by web search)
• The AI's thinking process — depending on the model, thinking blocks from earlier turns remain and keep taking up space
• And the answer currently being generated

This is why a conversation gets dramatically heavier when 'all I did was upload one PDF'. As we saw in Part 1, a single paper PDF is roughly 120,000 tokens.

---

Key data — context windows as of August 2026

Model	Context window	Max output	Long-context surcharge
Claude Opus 5 / Sonnet 5	1 million tokens	128K tokens	None (standard price across the full range)
Claude Haiku 4.5	200K tokens	—	—
GPT-5.6 (Sol/Terra/Luna)	1.05 million tokens	128K tokens	Above 272K: input 2x, output 1.5x
Gemini 3.1 Pro	2 million tokens	—	Above 200K: input 2x, output 1.5x

(Sources: Anthropic official Context windows docs, checked August 2026 / OpenAI GPT-5.6 specs / Google Gemini pricing docs)

? Interpretation: The easy-to-miss part of the table is the rightmost column. A bigger window does not mean the same price. OpenAI and Google raise the unit price itself beyond a certain threshold, while Anthropic charges the standard price across the entire 1-million-token range. For work that handles long documents whole, this difference decides the bill.

Note that chat services have separate policies. Per Anthropic's help center, on paid plans Opus 5 and Sonnet 5 use a 1-million-token window, while Opus 4.8/4.7/4.6 and Sonnet 4.6 use 500K. It's worth remembering that API specs and app specs can differ.

---

In-depth analysis

There are three different reasons for forgetting

'The AI forgot' looks like a single phenomenon, but the causes split into three. The countermeasures differ for each.

① Overflow — old conversation gets pushed out

The simplest case. When the container is full, something has to go. On the API, input exceeding the window returns an error outright, but according to Anthropic's documentation, chat interfaces like claude.ai may manage the window on a first-in, first-out (FIFO) basis. The oldest parts of the conversation quietly get pushed out.

If the AI forgot your initial instructions at the end of a long conversation, this is probably not a performance problem; that content most likely went physically outside the window.

② Context rot — it blurs before it overflows

This is the trickier one. Anthropic states in its official documentation that 'more context is not automatically better; as token count grows, accuracy and recall degrade' , and calls this context rot. It is a rare case of a vendor writing the limits of its own spec directly into its documentation.

In other words, having a 1-million-token window does not mean filling it with 1 million tokens is optimal.

③ Lost in the middle — it reads differently by position

The paper 'Lost in the Middle' by Liu et al. (TACL, 2024) showed that models do not read long inputs evenly. Performance traces a U-shaped curve. The very beginning and the very end are read well; the middle blurs. A drop in accuracy of more than 30% when the relevant information sat in the middle was confirmed repeatedly across multiple model families.

The practical takeaway is simple. Put important instructions at the very beginning or the very end of the prompt. Bury them in the middle and the odds they go unread go up.

The window on the spec sheet and the effective window are different

There is a caution here. The advertised context length and the length the model actually uses reliably are separate things.

NVIDIA's RULER benchmark showed that models claiming 32K or more often have an effective context length below the stated value. Chroma's 'Context Rot' report, published in July 2025, covered 18 major models and found that even on simple tasks performance degrades as input grows, and unevenly across models.

⚠️ There are secondary sources that give specific numbers such as 'the effective context is in the 200K–400K range', but these are measurements on particular benchmarks at particular times, not official announcements. The direction (longer is worse) is trustworthy, but it is safer not to quote the multiples or thresholds as is.

Korean users have a narrower window

The conclusion of Part 1 continues here. Korean uses more tokens than English to hold the same content.

Per Anthropic's documentation, in English one token is about four characters. Korean is split more finely, so feeding in the same amount of document fills tokens faster. If we take the Korean/English ratio on new-generation tokenizers as 1.4x, then a 1-million-token window effectively becomes a window of roughly 700K tokens for a Korean user.

⚠️ This 700K figure is a simple conversion from the 1.4x assumption; the actual ratio varies with the nature of the text. If you need an exact value, the sure way is to run the document through a token-counting API.

Cached tokens still take up space

In Part 1, we said prompt caching can cut the cost of repeated input by 90%. There is an easy misunderstanding here.

Anthropic's documentation nails it down. A cached prefix still takes up its place in the context window. Caching changes how much you pay for those tokens, not whether they occupy space.

Price and capacity are separate matters. That is the key to understanding the context window.

---

Implications

First, a perceived gap emerges in long-document work. The more a job deals with long Korean text, such as contract review, case law analysis, or writing business plans, the faster the window fills and the earlier you enter the context rot zone. The overseas rule of thumb that 'a document this size is handled in one go' cannot be applied as is.

Second, context design becomes more important than prompt design. We are in a phase where choosing what to put in (searching and summarizing to include only what's needed) affects results more than polishing how you ask. The industry calls this 'context engineering'.

Third, the practical tips are surprisingly simple.

• Start a new conversation when the topic changes (the most effective)
• Place important instructions at the very beginning or the very end
• Don't upload irrelevant files
• For long tasks, periodically ask 'summarize what we have so far' to create a digest and continue in a new window

---

Outlook and variables to watch

• Positive variable: A compaction feature, in which the server automatically summarizes the earlier part and continues when a conversation exceeds the window, is being introduced (Anthropic offers it in beta for models from 4.6 onward). Some models also have 'context awareness', recognizing their remaining token budget on their own.
• Risk variable: The bigger the window, the more the cost and latency grow along with it. And enlarging the window alone does not solve context rot.
• Checkpoints: ① Whether long-context surcharges apply ② whether the model is one that keeps earlier thinking blocks in the window ③ the compaction and summarization policy of the service you use.

---

Conclusion

1. AI does not remember. It re-reads the entire transcript every time. The size limit of that transcript is the context window.
2. There are three reasons for forgetting. It got pushed out of the window, it blurred as things got long, or it happened to sit in the middle. Each cause has a different remedy.
3. A big window is not a reason to fill it. Even the vendor's official documentation says 'more is not better'.

[References]

• Anthropic, Context windows (official docs, checked August 2026): https://platform.claude.com/docs/en/build-with-claude/context-windows
• Anthropic, Pricing (long-context tier pricing): https://platform.claude.com/docs/en/about-claude/pricing
• Anthropic Help Center, context windows on paid plans: https://support.claude.com/en/articles/8606394-how-large-is-the-context-window-on-paid-claude-plans
• Anthropic, Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• OpenAI, GPT-5.6 announcement: https://openai.com/index/gpt-5-6/
• Google Cloud, Generative AI pricing (long-context surcharge): https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024) — related summary: https://arxiv.org/pdf/2603.26707
• Chroma Research, "Context Rot" report — related summary: https://medium.com/@trmquang3103/context-rot-what-a-2-am-night-taught-me-that-benchmarks-never-did-dfda7df72f40
• RULER benchmark reference (effective long-context length): https://arxiv.org/pdf/2505.10570