Innovations in Context Length for Llms

Exciting breakthrough in long-context language models! Microsoft researchers have developed a novel bootstrapping approach that extends LLM context lengths to an impressive 1M tokens while maintaining strong performance. >> Key Innovation The team introduces a clever self-improving workflow that leverages a model's existing short-context capabilities to handle much longer contexts. Rather than relying on scarce natural long-form data, they synthesize diverse training examples through: 1. Instruction generation using short-context LLMs 2. Document retrieval with E5-mistral-7b 3. Recursive query-focused summarization 4. Response generation >> Technical Details Their SelfLong-8B-1M model achieves remarkable results: - Near-perfect performance on needle-in-haystack tasks at 1M tokens - Superior scores on the RULER benchmark compared to other open-source models - Progressive training strategy with RoPE base frequency quadrupling at each stage - Efficient training using RingAttention for distributed processing - Implementation of PoSE-style training for hardware constraints - Utilizes vLLM for inference optimization >> Impact This work demonstrates that existing LLMs can be effectively extended far beyond their original context windows through careful engineering and clever data synthesis. The method requires only readily available open-source components, making it highly accessible to the research community. The researchers have validated their approach across multiple model sizes (1B, 3B, 8B parameters) and even pushed to 4M tokens in experimental settings.

LLM memory is missing something fundamental. During pre-training, Llama 70B compresses the entire internet into 140GB of model weights. But just putting Steve Jobs’ Wikipedia page into the context window creates an 80GB key-value cache. If we want models that can efficiently reason over millions of tokens of context, we cannot simply dump everything into a context window. We need to continue training models at test-time, using long-context as training data to compress massive amounts of information directly into the model weights. Incredibly excited to share work led by my student Arnuv Tandon, in partnership with NVIDIA AI, that has been over a year in the making: End-to-End Test-Time Training for Long Context. As the title suggests, we continue training language models at test-time using the same next-token prediction objective as pre-training — allowing our model to scale with context length like full attention without maintaining a key and value for every token in the sequence. With linear complexity, our method is 2.7x faster than full attention at 128K tokens while achieving better performance. We believe test-time training is the key to unlocking a future with long-horizon agents, robots with human-like memory, and truly personal AI with your own model weights. Read the full paper: https://lnkd.in/g3f2BFcx Read the NVIDIA blog post: https://lnkd.in/gnWvS3Uk

91.3% accuracy vs 0%. Same model. Same task. The only difference: treating your prompt as code instead of text. Recursive Language Models (RLMs) from MIT have completely changed how I think about handling long context in LLMs. Instead of cramming everything into the context window, RLMs treat your prompt as part of the 𝘦𝘯𝘷𝘪𝘳𝘰𝘯𝘮𝘦𝘯𝘵 that the model can programmatically explore. 𝗧𝗵𝗲 𝗖𝗼𝗿𝗲 𝗜𝗻𝘀𝗶𝗴𝗵𝘁 Once you hit the context limit in an LLM, you're done. But LLMs are trained for code as well, right? Why not use their coding skills for more than just coding? 1. Load your prompt as a 𝘷𝘢𝘳𝘪𝘢𝘣𝘭𝘦 in a REPL programming environment 2. Give the model tools to peek into, decompose, and recursively process parts of that variable 3. Let the model write 𝘤𝘰𝘥𝘦 that calls itself on programmatic slices of the input This enables the model to handle prompts that are literally 100x longer than its context window. The 𝗿𝗲𝗰𝘂𝗿𝘀𝗶𝘃𝗲 element is the key insight here - the LLM can call itself (or a smaller subagent) for smaller tasks, allowing it to batch and concatenate results to answer complex questions. 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 I tested it in Python (via DSPy), input the full alice in wonderland book and asked it to give a sentiment analysis of the openings of each chapter. The LLM: 1. Explored the prompt (book) to see how the chapter headings were formatted 2. Implemented regex to split the full string into chunks before/after each chapter heading 3. Invoked the LLM sub-agent on each paragraph to analyse the sentiment Even if the full prompt can't fit into history, LLMs have notoriously suffered from context rot. This approach enabled each task to be separately analysed by the sub-agent, each having no knowledge of the greater task. 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 • RLMs successfully process inputs up to 𝘁𝘄𝗼 𝗼𝗿𝗱𝗲𝗿𝘀 𝗼𝗳 𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲 beyond model context windows • On BrowseComp-Plus (6-11M tokens), RLM(GPT-5) achieved 91.3% accuracy vs 0% for the base model RLMs aren't perfect. The inference cost has high variance - median costs are comparable to base models, but some trajectories explode to 3x+ the cost due to long recursive chains. I also found, as the authors note in the appendix, that the models continue analysing well past when they had already found an answer. My hunch is that each LLM invocation always wants to do 𝘴𝘰𝘮𝘦𝘵𝘩𝘪𝘯𝘨, even if that something has already been done. It always wants to check its answer. Because of how they're trained, LLMs never just say "Okay, done!". The paper demonstrates that with better training (especially on-policy rollouts at scale), native RLMs could become far more efficient than current implementations suggest. I'll be extremely excited if this becomes a core part of model training, building custom models that excel at managing their prompt with code. Read the paper: https://lnkd.in/eq_xUJvJ

Everyone’s flexing about 1M context like it’s the moon landing. At WRITER we’re aiming for 500M+ token windows for Palmyra LLM — and no, we’re not planning to brute-force it with more FlashAttention and RAM. Some crazy (but possibly real) ideas we’re playing with: - Fractal Attention Networks Instead of a flat attention pattern, recurse into token clusters at multiple resolutions. Global patterns? Use coarse grain. Local nuance? Zoom in. Like Stable Diffusion, but for attention. - Learned Virtual Memory Tokens don’t live in RAM. They get paged in and out of a learned SSD-like buffer. LRU meets gradient flow. Think Hugging Face meets operating system design. - Satellite Experts Spin off remote tiny models trained on fixed corpora (like your CRM history). The core LLM consults them in-place with latency-aware calls. Each expert handles its own namespace of long-term memory. - Entropy-Guided Compression Not all tokens are born equal. Compress context based on information gain — low-entropy sequences get collapsed dynamically. The model gets the gist, not the full log dump. - Recurrent Slot Replay with Alignment Feedback A separate module continuously summarizes and aligns long-past context into reusable “slots” the LLM can query via natural language. Basically: a compressed memory loop that talks back. - Semantic Decay Functions Let token salience fade over time, unless periodically refreshed by relevance. Like a memory that forgets until reminded — built-in forgetting as a feature. - Instructional Time-Bending Instead of storing 100M tokens raw, let the model simulate re-reading instructions. Prompt the model to act like it just read something, without needing to load it. Memory as roleplay. We’re not shipping all this tomorrow, but Palmyra’s long-context roadmap is getting… philosophical. If you’re also trying to bend time and space for LLM memory, let’s jam

Excited to share Memento: our recent work on teaching LLMs how to manage their context. This is a joint effort with an incredible team at AI Frontiers, a boutique AI lab within Microsoft Research. In this work, we built OpenMementos, data that can teach models to compress their own chain-of-thought during generation. Without the use of an external summarizer, restarts, or orchestration. The mementified model learns to segment its reasoning into blocks, compress each into a dense memento, and reason forward from that alone. What we found: - You can teach context management as a meta skill with standard SFT on ~30K examples. The model learns when to end a block, what to compress, and how to continue from the compression. - Peak KV cache drops 2–3× and serving throughput nearly doubles, with small accuracy gaps that shrink with scale and close with RL. - The most surprising finding: masked tokens don't fully disappear. Their information leaks forward through the KV cache representations of the mementos, forming an implicit second channel without which accuracy drops by 15 percentage points. We believe context management should be a learned capability, not something orchestrated around the model. Memento is a first step, and there's a long way to go: better training data, stronger RL, and agent applications are next. Everything is open sourced under MIT License: - OpenMementos: https://lnkd.in/gg6eab-y (228K annotated reasoning traces built on OpenThoughts-v3) - Data generation pipeline & vLLM patch: https://lnkd.in/gvJPdvAm - Our paper: https://lnkd.in/gqj57rJC Joint work with Vasilis Kontonis, Yuchen Zeng, Shivam Garg, Lingjiao Chen, Hao Tang, Ziyan Wang, Ahmed Awadallah, John Langford, and Eric Horvitz.

Up until now, much of domain specific knowledge injection to LLMs has answered the question: "How do we get the right context INTO the window?", but with the success of coding agents and recursive language models, that question has changed to: "How do we let the model NAVIGATE context itself?" Large language models have a limited context window, or maximum amount of tokens that can be input as its entire context. This is a hard constraint resulting from the transformer architecture itself, and while modern models have pushed context windows into the hundreds of thousands (even millions) of tokens, more context doesn't always mean better results. Research has shown that model performance actually degrades as input length increases, a phenomenon known as context rot, where models struggle to reliably use information buried deep in long sequences, especially when surrounded by similar but irrelevant content. The solution up until now has been Retrieval Augmented Generation (RAG), chunking and embedding documents into vector databases, then retrieving the most relevant pieces via semantic similarity. This works, but it frames context management purely as a search problem, and scaling it starts to feel more like building a search engine than an AI system. What coding agents like Claude Code, Cursor, and Codex stumbled into was a different approach entirely: give the LLM a terminal and let it explore. Filesystem-based context navigation lets models directly explore, preview, and selectively load content using tools they already understand. Instead of engineering a pipeline to deliver the right context, the model finds it itself. Recursive Language Models (RLMs) formalize this further, with a slight distinction: in a coding agent, opening a file or running a tool dumps results back into the context window. RLMs instead store the prompt and all sub-call results as variables in a code environment, only interacting with them programmatically. Recursion happens during code execution, meaning the model can spawn arbitrarily many sub-LLM calls without polluting its own context, orchestrating understanding of 10M+ tokens without ever having to look at all of it at once. This gives us two differently motived options: RAG gives you fast, narrow retrieval great for latency-sensitive apps like chatbots. RLM-style frameworks trade speed for deeper exploration, better suited when thorough analysis matters more than response time. To learn more about context rot, how coding agents changed context delivery, and how recursive language models are formalizing it all, check out my latest video here: https://lnkd.in/ehszSKV7

MIT researchers found a way around context limitations. (Here's why it changes everything) For months, everyone accepted this idea: LLMs can only handle around 100,000 tokens. End of story. MIT researchers just broke that assumption. They did not make the context window bigger. They changed how models use context. Their new approach, Recursive Language Models, can work with millions of tokens - up to 100x more than what was considered realistic. Accuracy stays strong. Costs are comparable or even go down. So what changed? LLMs use LLMs recursively. Instead of pushing huge documents into the model, they treat the document like a system the model can query. The model does not read everything at once. The document lives outside the context window as a variable that the model can inspect with code. Think about how you use Google or a book. You do not memorize everything. You search for what you need. Same idea here. Why this matters: - Context limits were shaping how we built AI tools. - We summarized data. - We filtered information before sending it to models. All of that was just a temporary fix. Now models can work with full, messy, real-world data. What developers can do now: - Work with massive codebases. - Scan years of git history. - Query huge documentation sets. - Build tools that use data that was impossible to handle before. This points to something deeper. What we call “hard limits” in tech are often just design choices. MIT didn’t remove a limit. They changed how the problem is framed. And that shift is what creates real breakthroughs.

𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 (𝗔𝗖𝗘) 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 The new paper, “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models” (Stanford, Berkeley, SambaNova Systems, Oct 2025), introduces ACE, a framework that redefines how large language models (LLMs) adapt and improve after training. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 ACE proposes that instead of periodically rewriting prompts or fine-tuning weights, language models should evolve their contexts, treating them as “evolving playbooks” that grow richer over time. Contexts capture strategies, heuristics, and insights from prior executions, allowing continuous self-improvement. 𝗛𝗼𝘄 𝗔𝗖𝗘 𝘄𝗼𝗿𝗸𝘀 ACE organizes LLM self-adaptation into three agentic roles: 🛠️ Generator – performs tasks and produces reasoning traces 🔎 Reflector – extracts lessons from successes and failures 📚 Curator – integrates these lessons into structured, incremental delta updates This modular workflow prevents two chronic issues in adaptive prompting: 🗜️ Brevity bias, where optimizers compress useful detail into overly short prompts 🧠 Context collapse, where iterative rewrites erase accumulated knowledge Through its incremental delta update and grow-and-refine processes, ACE keeps contextual knowledge comprehensive yet efficient, maintaining interpretability and scalability. 𝗤𝘂𝗮𝗻𝘁𝗶𝘁𝗮𝘁𝗶𝘃𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀 Across benchmarks, ACE significantly boosts both accuracy and efficiency: 🏆 +10.6% improvement on the AppWorld agent benchmark 🏦 +8.6% improvement on financial reasoning tasks (FiNER and Formula) 🏎️ 86.9% lower adaptation latency and over 80% cost reduction versus prior adaptive methods 🏁 ACE even matches or beats GPT‑4.1–powered agents on AppWorld using a smaller open-source model (DeepSeek‑V3.1). 𝗪𝗵𝘆 𝗶𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 ACE reframes LLM evolution as a continuous context optimization problem. It shows that longer, structured, and interpretable contexts, not compressed summaries, drive robustness, efficiency, and autonomy in modern AI agents. ACE points the way toward self-improving, label-free, cost-efficient AI systems, where knowledge grows dynamically, not statically.

Context Engineering (CE) is an organic evolution of Prompt Engineering (PE). As we explore different agentic architectures, it’s becoming increasingly important to define how business context, taxonomies, and even user preferences are shared across agents. A common approach today is to pass this information through one context window, and with ever-expanding context sizes, this feels plausible. Yet for complex business workflows, bundling everything into a single context soon becomes infeasible. Managing context itself becomes a core challenge. We also know that context window size matters only up to a point, as knowledge retention decays even within range. This degradation varies by persona, but data scientists often feel it most acutely. They juggle massive data dictionaries, mappings, and business taxonomies. All of these form the context needed for LLMs to generate accurate insights. Emerging techniques like RAG (with chunking and vector databases), memory management (short-term vs. long-term), compression, and time- or state-based sliding contexts are all part of this new discipline. There will surely be more approaches coming in. As agentic systems mature, mastering these methods will be essential. Context “engineering” is the next prompt “engineering”. #ExperienceFromTheField #WrittenByHuman

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗶𝘀𝗻’𝘁 𝗷𝘂𝘀𝘁 𝗸𝗶𝗻𝗴, 𝗶𝘁’𝘀 𝘁𝗵𝗲 𝗲𝗻𝘁𝗶𝗿𝗲 𝗸𝗶𝗻𝗴𝗱𝗼𝗺. After diving deep into context engineering for agentic AI, one insight keeps hitting me:- we’ve been thinking about prompts all wrong. The real game isn’t crafting the perfect prompt anymore. It’s about curating the perfect 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗯𝘂𝗱𝗴𝗲𝘁. Think about it:- LLMs are like us, limited working memory, diminishing returns as information piles up. Every token you feed an agent depletes its ability to focus on what actually matters. 𝗧𝗵𝗲 𝗯𝗿𝗲𝗮𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵? Progressive disclosure. Instead of front-loading everything, let agents explore just-in-time. Give them lightweight identifiers, not full data dumps. Let them navigate their environment like humans do, with bookmarks, not encyclopedias. 𝗧𝗵𝗿𝗲𝗲 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵𝗲𝘀 𝗜’𝗺 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗶𝗻𝗴 𝘄𝗶𝘁𝗵: → Compaction:- Summarize, compress, reinitiate. → Structured note-taking:- Persistent memory outside context windows. → Sub-agent architectures:- Specialists returning distilled insights. The pattern is clear:- 𝘁𝗿𝗲𝗮𝘁 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗮𝘀 𝗽𝗿𝗲𝗰𝗶𝗼𝘂𝘀 𝗮𝗻𝗱 𝗳𝗶𝗻𝗶𝘁𝗲, 𝗻𝗼𝘁 𝗶𝗻𝗳𝗶𝗻𝗶𝘁𝗲 𝗮𝗻𝗱 𝗳𝗿𝗲𝗲. What context engineering strategies are you seeing work in production? Would love to hear what’s actually moving the needle for your agents. Check out this one from Anthropic! #AI #AIAgents #LLM #ContextEngineering #PromptEngineering #MachineLearning #ArtificialIntelligence #AgenticAI #BuildingWithAI #TechInnovation #AIEngineering #DevTools #Claude #Anthropic