The new documentation from Anthropic pretends that Opus 5 handles well the 1M context window: “Claude Opus 5 has a 1M token context window as both the default and the maximum, and its instruction following, tool calling, and reasoning stay consistent throughout the window”
Personally I have a system that save the current status (task/session progress) to disk and clears the context window at about 32%. Then the work can resume with a fresh context window. Can I leave the context window utilization grow larger than 20-30%
For opus 4.x the evidence was that a context window of 1M is useless. Is there new evidence for Opus 5?
I don’t count the tokens. If its going to be a lengthy affair, such as design code and document then I discuss the design with one session and get it to document it.
Then I get another session to code it. It also makes a user manual or a as built document.
If it was a substantial bit of code then I get a third one to review the code and the final document.
It isn’t just the token count, different sessions have different approaches, sometimes, they are quite different.
As an example, I’m working on long path support in Finalbuilder - sounds simple but it’s a few million lines of code, with thousands of instances where we access file paths etc.
I got a claude to review the code and do some long path probing, and save a report.
Then I fed that into a planning session and saved the plan to a file - reviewed and edited it (took me a few hours, plan was extensive) - then I fed the plan into a new session.
I find I get better results that way. Today I will kick off a new session to review the code again in case it missed or broke anything. I will of course also be actually testing the application.
Sometimes it’s better to do it one way, sometimes – the other: long memory helps when working on large projects and if you restart the session, the chances are the agent would have to re-read everything again anyway = no savings.
OUT tokens cost 5 times the IN tokens, but long sessions mean you are sending 1000 times more IN = your IN costs 200 times the OUT.
On the other hand, I’m using my own AI client (BrainStorm) and MCP tools, all written completely in Delphi (which gives me trust in what it’s going to do) and on top of everything else it has memory tools, file IO tools which it also uses to record steps, plans, designs, actions and tricks (apart from doing actual development work) in files, then I use almost full 1M window, but auto-compact it after ~2Mbytes (~2/3 of 1M tokens) using the same agent, and run all this against a local DeepSeek-V4-Flash-0731 model, so I do not have to count tokens. This becomes expensive when using either ChatGPT or Claude, but they do it better than DeepSeek (they are much bigger).
Restarting session helps, when you have multiple unrelated questions/tasks, even against the same codebase: it may need to read different chunks then, and overlapping memories in the same session would make it bigger than it needs to be.
Opus 4.x: your “1M is useless” was right, and Anthropic’s own reports prove it.
At 1M tokens, Opus 4.7 scored 32.2% on MRCR (a test that asks for one exact earlier answer in a long conversation).
On GraphWalks (a search through a graph that fills the window), Opus 4.8 fell from 85.9 at 256K to 68.1 at 1M.
So, if you fill your context at max about 1/3 of the information you put in the context will be ignored. That’s a very scary number! You are working on pure hallucinations.
Anthropic (conveniently) decided to go the “trust me bro”-way and removed that context-rot test when they evaluated Opus 5.
I am running my own experiments. They don’t look good at all. I think "reasoning stay consistent throughout the window” is yet another inflation from Anthropic.
I will post the results.
For those that let the context window fill (mine stops at 320K) please ask Claude to extract some statistics to see how many times a request from you failed, relative to the context window load.
I haven’t seen figures for performance based on context length in recent times, but I do know I have runs that go for 12+ hours. Admittedly half of that is probably running unit tests, but if it has a good plan - lots of steps to do, it should be able to go a long time. When I get some time I will update my Claude Code stats program to show what the lengths of my sessions have been - both token wise and time wise.
I haven’t seen figures for performance based on context length in recent times
Context Arena shows A improvement, though really small (only 3-4 percent). Definitively not “consistent throughout the window” as Anthropic states.
The conclusions I draw today are here.
The first table shows specifically the difference between Opus 4.8 and 5. Sorry, for other LLMs or models you’ll have to extract the data from Context Arena.
My system saves the task status to disk and self-compacts (or clear) when the context usage exceeds a threshold. In time I got greedy and I slowly increased the threshold until I reach 320K. Looks like I have to put it back to 128 or 200K.
PS: Today I wasted all my credits re-testing the system
PS: Every time I look at that table I get really scared! Fill the context over 64K and is like you are playing the lottery!
