Understanding Large Language Model Context Limits

We do not think about token budgets until something breaks - the app stops working, a sentence gets cut off midway, etc. Fun fact: token budgets are more of a system design problem :) Token budgets matter because they are the edge condition for everything an LLM does. A model is capped by its context window. Go beyond it, and you get failures. Stay too far under it, and you are making your system less intelligent. Token budgets are a system design problem. What fills a context window is not random, and hence it does not need to be treated as random. It is system prompts, conversation history, retrieved documents, tool call results, and reasoning chains. Every one of these is an architectural decision. If your RAG pipeline dumps 10 unranked chunks into the context, that is a retrieval design problem. If your conversation history grows unbounded across turns, that is a state management problem. A prompt instruction does not fix either. System design gives you levers - chunking strategy, context window allocation, history summarization, tiered retrieval, parallelization, etc. These decisions compound, and you need a rock-solid design and data flow to build a robust AI system. By the way, a well-designed system does more with a 32k context window than a poorly designed one with 200k - and at a fraction of the cost. Models also behave differently near their context limit. Attention degrades, key details get dropped, and output quality falls non-linearly. You cannot prompt your way around that. You design around it. The prompt is the last 5% - the system is the other 95%.

Large context windows are now becoming a major part of model marketing. 1 million tokens. 2 million tokens. But the important question is not: “How much context can the model technically accept?” The better question is: “How much context can the model use reliably?” Those are very different things. Even when models advertise very large context windows, serious benchmarks show that reasoning quality often starts degrading much earlier — frequently somewhere around the 100K–200K token range, depending on the task. The evidence is becoming fairly consistent. Chroma’s Context Rot study tested 18 frontier models and found that every model degraded as input length increased, even on relatively simple retrieval tasks. The NoLiMa benchmark from LMU Munich and Adobe, accepted at ICML 2025, removed easy keyword-matching shortcuts and showed that 11 of 13 models dropped below 50% of their baseline accuracy at just 32K tokens. In code-heavy workloads, the degradation can be even sharper. You also pay more above 200K tokens because the model now has to process more information. Why does this happen? Three forces compound. 1. Attention dilution - As the context gets larger, the model has to distribute attention across more tokens. Specific facts become harder to retrieve reliably 2. Lost-in-the-middle behavior - Models tend to attend more strongly to the beginning and end of the input, and less reliably to information buried in the middle. 3. Distractor interference - Irrelevant but semantically similar content can actively mislead the model. This matters because real enterprise context is rarely clean. It contains duplicate documents, stale decisions, old chat history, similar tickets, outdated specs, partial tool outputs, and contradictory references. This is why context window should not be treated as memory. It is better understood as working surface area. And like any working surface, it becomes less useful when it is cluttered. A more practical concept is Maximum Effective Context Window — the amount of context a model can use with acceptable reliability for a given task. That number is usually much smaller than the advertised maximum. For high-stakes workflows — legal review, regulated document intelligence, production code agents, financial analysis, enterprise search — the answer is not simply to use a bigger window. The answer is better context engineering: - targeted retrieval - hard relevance filtering - structured chunking - reranking - pruning of stale context - separation of memory from working context - task-specific context assembly before each call A dense 100K-token context with the right information will usually outperform a diluted 1M-token context filled with chat history, logs, tool outputs, and loosely related documents. 1M tokens is a ceiling, not a destination. #EnterpriseAI #AITransformation #Trainingledtransformation

The interview is for a Generative AI Engineer role at Cohere. Interviewer: "Your client complains that the LLM keeps losing track of earlier details in a long chat. What's happening?" You: "That's a classic context window problem. Every LLM has a fixed memory limit - say 8k, 32k, or 200k tokens. Once that's exceeded, earlier tokens get dropped or compressed, and the model literally forgets." Interviewer: "So you just buy a bigger model?" You: "You can, but that's like using a megaphone when you need a microphone. A larger context window costs more, runs slower, and doesn't always reason better." Interviewer: "Then how do you manage long-term memory?" You: 1. Summarization memory - periodically condense earlier chat segments into concise summaries. 2. Vector memory - store older context as embeddings; retrieve only the relevant pieces later. 3. Hybrid memory - combine summaries for continuity and retrieval for precision. Interviewer: "So you’re basically simulating memory?" You: "Yep. LLMs are stateless by design. You build memory on top of them - a retrieval layer that acts like long-term memory. Otherwise, your chatbot becomes a goldfish." Interviewer: "And how do you know if the memory strategy works?" You: "When the system recalls context correctly without bloating cost or latency. If a user says, 'Remind me what I told you last week,' and it answers from stored embeddings - that’s memory done right." Interviewer: "So context management isn’t a model issue - it’s an architecture issue?" You: "Exactly. Most think 'context length' equals intelligence. But true intelligence is recall with relevance - not recall with redundancy." #ai #genai #llms #rag #memory

🚨 The biggest misunderstanding about LLM limits I see every week Today someone asked advice to analyze a 500,000-character file. They thought: “Easy, I’ll just convert the PDF into .txt and paste it into the model.” Except… that’s not how large-context models work. What actually happened was: 1) The model accepted the file. 2) It looked like it processed the whole thing. 3) It even responded confidently that the analysis was done But when the user asked what it really did, it finally admitted: It only analysed ~30% of the text The rest never even made it into memory. And honestly? This happens all the time. Why this happens: GPT-5-class models can handle ~272k tokens for input (≈ 200k words) ~128k tokens for output A 500k-character document → far beyond that limit. So the model quietly samples, truncates, or drops earlier context as it processes. This isn’t an error but an intrinsic limitation of the model. A limitation by design, even: Imagine ChatGPT's 800 million weekly users uploading huge documents on OpenAI servers all at once... ...not even all data centers on Earth would be enough. But most people don’t realize it. ⚠️ The hidden risk When context goes over the limit: -The model won’t throw an error -It won’t warn you -It will reply with confidence anyway And you’re left assuming it processed everything correctly Which is exactly how bad analysis, missed insights, and false certainty happen. ✔️ What to do instead If you’re working with very large documents: -Chunk the text intentionally -Use multi-pass or hierarchical summaries -Feed sections in controlled sequences -Or use external retrieval rather than raw uploads In other words: If the file is bigger than the model’s brain, upgrade the workflow, not the file format. Final thoughts AI can be useful for certain tasks, but it’s not magic. And it’s definitely not reading half-million-character documents in one go. Know your tools. Know their limits. And don’t let confidence trick you into thinking you got a full analysis when you only got 30%. ---- Follow me Chiara Gallese, Ph.D. for an honest analysis of AI limitations and risks

Mid-conversation with Claude yesterday, I got this message: "Compacting our conversation so we can keep chatting. This takes about 1-2 minutes." At 62% capacity, I watched it reorganize its thoughts. And I realized: most AI users have no idea this is happening. Here's what you need to understand. Context windows are AI's working memory. It's the total text the model can "see" at once—your prompts, its responses, uploaded documents, everything. Claude offers 200,000 tokens (roughly 150,000 words) for paid users. Sounds massive until you're deep into a complex project. When you hit that ceiling, something has to give. Claude's approach: Auto-compaction kicks in around 95% capacity. Earlier messages get summarized, keeping what the AI thinks matters most. Your full history is preserved—you can scroll back—but the AI's "working memory" gets compressed. Each compression cycle loses granularity. Manus AI takes a different path. Rather than compacting in place, it externalizes memory to the file system—creating todo.md files to maintain focus, saving intermediate results externally, spinning up sub-agents with their own context windows for discrete tasks. When context fills up, it uses "recoverable compression"—dropping content but keeping URLs and file paths so it can retrieve information later if needed. Neither is perfect. Both involve tradeoffs. The takeaway: Context limits are real constraints on how much complexity AI can handle in a single session. If your team uses AI for research, strategy, or extended projects, you need to understand this. Three practical tips: → Checkpoint manually at 70% rather than waiting for auto-compaction at 95%. You control what's preserved. (This only works in Claude Code using the /compact switch) → Summarize at natural breakpoints. Ask the AI to capture key decisions before moving on. You may manually bring this across to another chat or ask it to save it to memory. → For complex projects, externalize documentation (E.g use Projects in Claude and ChatGPT). Don't rely solely on conversation memory. As context windows expand—Claude's testing 1 million tokens for some API users—this will matter less. But for now, understanding your AI's memory limits is the difference between productive collaboration and frustrating repetition.

Up until now, much of domain specific knowledge injection to LLMs has answered the question: "How do we get the right context INTO the window?", but with the success of coding agents and recursive language models, that question has changed to: "How do we let the model NAVIGATE context itself?" Large language models have a limited context window, or maximum amount of tokens that can be input as its entire context. This is a hard constraint resulting from the transformer architecture itself, and while modern models have pushed context windows into the hundreds of thousands (even millions) of tokens, more context doesn't always mean better results. Research has shown that model performance actually degrades as input length increases, a phenomenon known as context rot, where models struggle to reliably use information buried deep in long sequences, especially when surrounded by similar but irrelevant content. The solution up until now has been Retrieval Augmented Generation (RAG), chunking and embedding documents into vector databases, then retrieving the most relevant pieces via semantic similarity. This works, but it frames context management purely as a search problem, and scaling it starts to feel more like building a search engine than an AI system. What coding agents like Claude Code, Cursor, and Codex stumbled into was a different approach entirely: give the LLM a terminal and let it explore. Filesystem-based context navigation lets models directly explore, preview, and selectively load content using tools they already understand. Instead of engineering a pipeline to deliver the right context, the model finds it itself. Recursive Language Models (RLMs) formalize this further, with a slight distinction: in a coding agent, opening a file or running a tool dumps results back into the context window. RLMs instead store the prompt and all sub-call results as variables in a code environment, only interacting with them programmatically. Recursion happens during code execution, meaning the model can spawn arbitrarily many sub-LLM calls without polluting its own context, orchestrating understanding of 10M+ tokens without ever having to look at all of it at once. This gives us two differently motived options: RAG gives you fast, narrow retrieval great for latency-sensitive apps like chatbots. RLM-style frameworks trade speed for deeper exploration, better suited when thorough analysis matters more than response time. To learn more about context rot, how coding agents changed context delivery, and how recursive language models are formalizing it all, check out my latest video here: https://lnkd.in/ehszSKV7

MIT researchers found a way around context limitations. (Here's why it changes everything) For months, everyone accepted this idea: LLMs can only handle around 100,000 tokens. End of story. MIT researchers just broke that assumption. They did not make the context window bigger. They changed how models use context. Their new approach, Recursive Language Models, can work with millions of tokens - up to 100x more than what was considered realistic. Accuracy stays strong. Costs are comparable or even go down. So what changed? LLMs use LLMs recursively. Instead of pushing huge documents into the model, they treat the document like a system the model can query. The model does not read everything at once. The document lives outside the context window as a variable that the model can inspect with code. Think about how you use Google or a book. You do not memorize everything. You search for what you need. Same idea here. Why this matters: - Context limits were shaping how we built AI tools. - We summarized data. - We filtered information before sending it to models. All of that was just a temporary fix. Now models can work with full, messy, real-world data. What developers can do now: - Work with massive codebases. - Scan years of git history. - Query huge documentation sets. - Build tools that use data that was impossible to handle before. This points to something deeper. What we call “hard limits” in tech are often just design choices. MIT didn’t remove a limit. They changed how the problem is framed. And that shift is what creates real breakthroughs.

🧠 What “Memory” Really Means for Large Language Models Ever notice how we keep comparing LLMs to human brains? The reality is LLMs are not brains and the cognitive architecture is still - and might always be - significantly different. The hardware constraints and capabilities are quite different between humans and computer systems. Human memory 1. Sensory memory (milliseconds of raw sight/sound) 2. Working memory (what you can hold in mind right now) 3. Semantic long-term memory (facts & concepts) 4. Episodic long-term memory (your life events) LLM analogues 1. Tokeniser buffer - Vanishes the moment text is chunked—irrelevant in practice. 2. Context window - Fixed-size RAM. When it’s full, older tokens fall off a cliff. 3. Model weights - Billions of frozen parameters—an immutable encyclopedia. 4. ❌ Not in the base model - Needs external vector DBs, caches, or online fine-tuning. Key takeaway: Today’s LLM is basically a fact-based semantic engine with a short-term scratch-pad. It remembers Paris is the capital of France, but forgets everything you told it five minutes ago once the context window scrolls past. We've performed some neat tricks with RAG and context engineering but the basic cognitive deficiencies of lack of effective long term memory still is there. ⸻ Why it matters 1. Product design: If you need continuity across sessions—customer profiles, project history, personal preferences—you must bolt on an external memory layer. There are some good tools out there but this requires advanced engineering to be done right, it's time consuming. 2. Safety & accuracy: Stale facts stay frozen until you retrain or fine-tune. Real-time knowledge requires retrieval-augmented generation (RAG) or streaming updates. Also requires aggressive pruning of dead code and bad data. 3. Cost/performance: Throwing more tokens at the context window scales O(n²). Smarter retrieval beats blind stuffing. 4. Research frontier: Adaptive weights, parameter-efficient “write-backs,” and unified memory architectures will blur the line between working and long-term memory in the next gen. ⸻ 🛠 Build like a brain—that actually forgets, forgetting what should be forgotten and keeping the important stuff. Done right, you get AI systems that learn from every interaction instead of Memento. Using anything like this currently? What's worked, what hasn't? #AI #LLM #MemoryArchitecture #ContextEngineering

All new models come with a larger Context Window...but do you know what it is? Here's my quick guide: The Definition - Context window = amount of text an AI model can process at once - Larger windows allow AI to handle more information simultaneously - For instance, if the context window is 1024 tokens, the model can utilize up to 1024 tokens of prior text to understand and generate a response. Why It Matters - Enhanced Understanding: Larger context windows allow the model to retain more information from the ongoing conversation or document, leading to more coherent and contextual responses. - Complex Tasks: With a bigger context, models can tackle more complex tasks like long-form document analysis, multi-turn conversations, or summarizing lengthy articles without losing track. Reduced Fragmentation: A larger context window reduces the need to break down input into smaller chunks, leading to more natural and uninterrupted interactions. What to Expect - More Insightful Outputs: As AI models continue to evolve, expect richer and more insightful outputs, especially in applications like content generation, chatbots, and customer support. - Increased Productivity: Businesses leveraging these models can achieve higher productivity by allowing AI to handle more sophisticated tasks with less human intervention. Alternative to Large Context Windows: 1. Chunking: Breaks large text into smaller chunks, processing them independently. - Pros: Memory efficient, scalable. - Cons: Risk of losing context, complex to stitch results together. 2. RAG: Retrieves relevant information from external sources during generation. - Pros: Accesses vast knowledge, improves accuracy, works with smaller context windows. - Cons: Complex to set up, potential latency, depends on data quality. Things to Be Careful With: - Context Loss: Whether chunking or using RAG, losing the overall context is a risk. Ensuring that each chunk or retrieved information is relevant and seamlessly integrated is crucial. - Latency: Larger context windows and RAG systems can increase processing time, affecting real-time applications like chatbots or live interactions. - Memory and Computational Overhead: Larger context windows demand more memory and computational power, which can be a limitation for some systems. - Complexity of Implementation: Both alternatives, especially RAG, require a more complex setup, including retrieval systems and databases. This can increase the cost and time needed for development. - Data Relevance: In RAG, the quality of the generated output is highly dependent on the relevance and accuracy of the retrieved data. Ensuring the retrieval system is well-tuned, and the knowledge base is up-to-date is essential. Choose the right approach based on your specific use case!