Trends in Large Language Models

Large language models (LLMs) are typically optimized to answer peoples’ questions. But there is a trend toward models also being optimized to fit into agentic workflows. This will give a huge boost to agentic performance! Following ChatGPT’s breakaway success at answering questions, a lot of LLM development focused on providing a good consumer experience. So LLMs were tuned to answer questions (“Why did Shakespeare write Macbeth?”) or follow human-provided instructions (“Explain why Shakespeare wrote Macbeth”). A large fraction of the datasets for instruction tuning guide models to provide more helpful responses to human-written questions and instructions of the sort one might ask a consumer-facing LLM like those offered by the web interfaces of ChatGPT, Claude, or Gemini. But agentic workloads call on different behaviors. Rather than directly generating responses for consumers, AI software may use a model in part of an iterative workflow to reflect on its own output, use tools, write plans, and collaborate in a multi-agent setting. Major model makers are increasingly optimizing models to be used in AI agents as well. Take tool use (or function calling). If an LLM is asked about the current weather, it won’t be able to derive the information needed from its training data. Instead, it might generate a request for an API call to get that information. Even before GPT-4 natively supported function calls, application developers were already using LLMs to generate function calls, but by writing more complex prompts (such as variations of ReAct prompts) that tell the LLM what functions are available and then have the LLM generate a string that a separate software routine parses (perhaps with regular expressions) to figure out if it wants to call a function. Generating such calls became much more reliable after GPT-4 and then many other models natively supported function calling. Today, LLMs can decide to call functions to search for information for retrieval augmented generation (RAG), execute code, send emails, place orders online, and much more. Recently, Anthropic released a version of its model that is capable of computer use, using mouse-clicks and keystrokes to operate a computer (usually a virtual machine). I’ve enjoyed playing with the demo. While other teams have been prompting LLMs to use computers to build a new generation of RPA (robotic process automation) applications, native support for computer use by a major LLM provider is a great step forward. This will help many developers! [Reached length limit; full text: https://lnkd.in/gHmiM3Tx ]

The 2025 Landscape of LLMs — Updated View of the Big Players in the Game of AI About 18 months ago, I shared my first version of the Large Language Model landscape, and a lot has changed since then. The space has evolved rapidly, but at the same time, we’re starting to see clear patterns emerge. This updated view focuses on the leading AI research labs, their latest models, and how those models can be accessed. It’s not meant to list every single LLM out there—but it does cover about 95% of what’s being used in real-world scenarios today. Here are some key insights: 🔹 No more clear front-runner: We’ve gone from “everyone chasing one leader” to a fairly even playing field. For most use cases, model differences are small and often not that relevant. 🔹 Model choice is the new normal: Customers now expect the ability to test, compare, and switch between models with ease. This shift is driving interest in evaluation frameworks and model routing tools. 🔹 Reasoning-first models are rising: Many providers are clearly moving toward models optimized for reasoning—fueling the surge of Agentic AI architectures. 🔹 Proprietary still leads, but just barely: Open-source and open-weight models are quickly closing the gap. 🔹 The U.S. is still ahead, but international competition is heating up—fast. 🔹 Cloud and APIs dominate: With few exceptions (hello Grok/XAI 👀), nearly every model is accessible via API across the major cloud platforms. 🔹 Serverless is the default: Most organizations prefer calling models via API over hosting or fine-tuning them—unless the use case is highly specialized. 🔹 Everyone else? Still less than 5% of the market. We’re entering a phase where model access, interoperability, and orchestration matter more than the model itself. And this landscape helps make sense of where we are and where we’re going. #LLMs #AI #MachineLearning #GenerativeAI #AgenticAI #OpenSource

Exciting breakthrough in LLM Research: A comprehensive survey reveals that Large Language Models (LLMs) are proving to be highly effective embedding models, marking a significant shift from traditional encoder-only models like BERT to decoder-only architectures. The research, led by scholars from Beihang University, University of Technology Sydney, and other prestigious institutions, demonstrates two primary approaches for deriving embeddings from LLMs: >> Direct Prompting Strategy • Leverages LLMs' instruction-following capabilities to generate topic-specific embeddings • Utilizes contextual representations for enhanced semantic understanding • Implements prompt engineering techniques for optimal embedding generation >> Data-Centric Tuning Approach • Employs supervised contrastive learning with carefully curated datasets • Incorporates multi-task learning frameworks for improved generalization • Utilizes knowledge distillation from cross-encoder models for enhanced performance >> Advanced Implementation Details The research reveals sophisticated techniques including: • Bidirectional contextualization for enhanced semantic capture • Low-rank adaptation for efficient parameter tuning • Integration of both dense and sparse embedding approaches • Implementation of innovative pooling strategies for token aggregation >> Performance Insights The study demonstrates remarkable improvements over traditional models: • Superior performance in classification, clustering, and retrieval tasks • Enhanced capability in handling long-context dependencies • Improved cross-lingual representation capabilities • Better scalability with model size and training data This groundbreaking research opens new possibilities for applications in information retrieval, natural language processing, and recommendation systems.

Day 19/30 of SLMs/LLMs: Mixture-of-Experts, Efficient Transformers, and Sparse Models As language models grow larger, two challenges dominate: cost and efficiency. Bigger models bring higher accuracy but also higher latency, energy use, and deployment complexity. The next phase of progress is about making models faster, lighter, and more intelligent per parameter. A leading direction is the Mixture-of-Experts (MoE) architecture. Instead of activating every parameter for each input, MoE models route tokens through a few specialized “experts.” Google’s Switch Transformer and DeepMind’s GLaM demonstrated that activating only 5 to 10 percent of weights can achieve the same accuracy as dense models at a fraction of the compute. Open models like Mixtral 8x7B extend this idea by using eight experts per layer but activating only two for each forward pass. The result is performance similar to a 70B model while operating at roughly 12B compute cost. Another active area of innovation is Efficient Transformers. Traditional attention scales quadratically with sequence length, which limits how much context a model can process. New variants such as FlashAttention, Longformer, Performer, and Mamba improve memory efficiency and speed. FlashAttention in particular accelerates attention calculations by performing them directly in GPU memory, achieving two to four times faster throughput on long sequences. Sparse Models also contribute to efficiency by reducing the number of active parameters during training or inference. Structured sparsity, combined with quantization and pruning, allows models to run on smaller devices without a major loss in quality. Advances in sparsity-aware optimizers now make it possible to deploy billion-parameter models on standard hardware with near state-of-the-art accuracy. These techniques share a single goal: scaling intelligence without scaling cost. The focus is shifting from building larger networks to building smarter ones. A 7B model that uses retrieval, sparse activation, and efficient attention can outperform a much larger dense model in both speed and reliability.

𝗧𝗟;𝗗𝗥 NeurIPS 2025 marks the definitive shift from "Chat" to "Autonomy." The research signals a split reality for the enterprise: generic models are converging into a commoditized "Artificial Hivemind," leaving proprietary data as your only real moat. However, the upside is massive. New "Gated Attention" architectures are redefining inference efficiency, while breakthroughs in 1,000-layer Deep RL are finally unlocking agents capable of navigating complex, long-horizon enterprise workflows without getting stuck. NeurIPS is around the corner and wanted to highlight some trends based on the best papers (https://lnkd.in/ejp6vEjD) 𝟯 𝗣𝗮𝗽𝗲𝗿𝘀 (𝗮𝗻𝗱 𝘁𝗵𝗲𝗺𝗲𝘀) 𝗬𝗼𝘂 𝗡𝗲𝗲𝗱 𝘁𝗼 𝗞𝗻𝗼𝘄 𝟭. 𝗧𝗵𝗲 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝘁𝗶𝗼𝗻 𝗖𝗿𝗶𝘀𝗶𝘀  • 𝗣𝗮𝗽𝗲𝗿: 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗛𝗶𝘃𝗲𝗺𝗶𝗻𝗱: The Open-Ended Homogeneity of Language Models  • 𝗧𝗵𝗲 𝗦𝗶𝗴𝗻𝗮𝗹: Models trained on synthetic data and each other’s outputs are suffering from "inter-model homogeneity." They are converging on the same "average" answers.  • 𝗧𝗵𝗲 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗥𝗲𝗮𝗹𝗶𝘁𝘆: If you rely on a vanilla wrapper around GPT, Claude and Gemini your business logic is becoming a commodity. 𝟮. 𝗧𝗵𝗲 𝗡𝗲𝘄 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗦𝘁𝗮𝗻𝗱𝗮𝗿𝗱  • 𝗣𝗮𝗽𝗲𝗿: Gated Attention for Large Language Models (Qwen Team)  • 𝗧𝗵𝗲 𝗦𝗶𝗴𝗻𝗮𝗹: By adding a simple "gate" to attention heads, we can stabilize training at massive scales and prevent "attention sinks."  • 𝗧𝗵𝗲 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗥𝗲𝗮𝗹𝗶𝘁𝘆: This is the update for your self-hosted inference. Models using Gated Attention (like Qwen3-Next) can offer significantly better performance-per-dollar. 𝟯. 𝗧𝗵𝗲 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗨𝗻𝗹𝗼𝗰𝗸 𝗣𝗮𝗽𝗲𝗿: 1000 Layer Networks for Self-Supervised RL 𝗧𝗵𝗲 𝗦𝗶𝗴𝗻𝗮𝗹: We used to think RL couldn't scale in depth like LLMs. This paper proves we can train 1,000-layer RL networks using self-supervised contrastive learning. 𝗧𝗵𝗲 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗥𝗲𝗮𝗹𝗶𝘁𝘆: This enables L5 Autonomous Agents - agents that can navigate complex ERP/CRM workflows without getting stuck in loops. 𝗔𝗰𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗖𝗧𝗢𝘀 𝗮𝗻𝗱 𝗖𝗔𝗜𝗢𝘀  𝟭. 𝗣𝗶𝘃𝗼𝘁 𝘁𝗼 "𝗗𝗮𝘁𝗮 𝗜𝗻𝗷𝗲𝗰𝘁𝗶𝗼𝗻": Go beyond prompt engineering with context and data engeineering. Focus even more on RAG and Fine-Tuning pipelines that inject your proprietary data to break the "Hivemind" average.  𝟮. 𝗔𝗱𝗼𝗽𝘁𝗶𝗻𝗴 𝗚𝗮𝘁𝗲𝗱 𝗠𝗼𝗱𝗲𝗹𝘀: When evaluating open-weights models for 2026, mandate "Gated Attention" architectures to lower your long-term inference TCO.  𝟯. 𝗣𝗶𝗹𝗼𝘁 𝗗𝗲𝗲𝗽 𝗥𝗟: Move your "Agent" pilots beyond simple tool use. Start testing self-supervised RL on internal workflows to build agents that learn from your experts' corrections.

The OpenRouter/16z 100 TRILLION token study is am empirical look at how people use large language models at scale. The findings challenge surface-level assumptions and point to a very different competitive landscape than most suggest. The biggest move is from single-turn interactions to agentic inference. Models are increasingly used as reasoning engines operating inside multi-step workflows. Prompts are getting several times longer, tool calls are rising, and reasoning-optimized models now handle the majority of tokens. The center of gravity has moved from producing text to driving processes. Another major theme is the strength of open source models, especially those coming out of China. Open-weight models now account for a sizable share of all usage, and Chinese labs have gone from almost zero to global contenders. The open ecosystem is now a competitive field where new releases can capture real mindshare almost immediately. One of the most striking findings is how people are actually using models. Programming and roleplay dominate total tokens. Coding assistance has become a core workload and primary driver of long-context reasoning, while roleplay is a structured, high-engagement use case that persists across regions and model types. hese two categories alone explain much of the real market behavior. No single model can be “best” at everything. Each provider shows a distinct usage fingerprint, from Claude’s heavy concentration in technical tasks to DeepSeek’s dominance in chat-driven and creative. The study also highlights a geographic reality that often gets overlooked. Usage is increasingly global. Asia’s share has risen dramatically. China, Singapore, and Germany sit near the top of total tokens. English still leads, but multilingual usage is becoming a competitive frontier. The section on user retention may be the most important for understanding long-term defensibility. Early cohorts for a few models show unusually strong retention because they were the first to solve a high-value workload. Once a model fits that need, users build systems and habits around it and are reluctant to switch. The researchers call this the Glass Slipper effect. It reframes retention as evidence of a capability breakthrough, not just a business metric. Cost dynamics tell a similar story. Cheap models absorb massive volume. Premium models command strong demand where correctness and reliability matter. Price alone does not predict usage. Differentiation still matters, and the market is segmented rather than commoditized. Taken together, the study describes an ecosystem moving toward multi-model reality, global competition, and workloads defined by reasoning depth and integration, not conversational polish. The practical implication is that the next era will reward builders who understand agentic workflows, invest in real workload fit, and treat usage data as the primary signal for where the market is heading. https://lnkd.in/gMY9dYA5

For the last couple of years, Large Language Models (LLMs) have dominated AI, driving advancements in text generation, search, and automation. But 2025 marks a shift—one that moves beyond token-based predictions to a deeper, more structured understanding of language.  Meta’s Large Concept Models (LCMs), launched in December 2024, redefine AI’s ability to reason, generate, and interact by focusing on concepts rather than individual words.  Unlike LLMs, which rely on token-by-token generation, LCMs operate at a higher abstraction level, processing entire sentences and ideas as unified concepts. This shift enables AI to grasp deeper meaning, maintain coherence over longer contexts, and produce more structured outputs.  Attached is a fantastic graphic created by Manthan Patel How LCMs Work:  🔹 Conceptual Processing – Instead of breaking sentences into discrete words, LCMs encode entire ideas, allowing for higher-level reasoning and contextual depth.  🔹 SONAR Embeddings – A breakthrough in representation learning, SONAR embeddings capture the essence of a sentence rather than just its words, making AI more context-aware and language-agnostic.  🔹 Diffusion Techniques – Borrowing from the success of generative diffusion models, LCMs stabilize text generation, reducing hallucinations and improving reliability.  🔹 Quantization Methods – By refining how AI processes variations in input, LCMs improve robustness and minimize errors from small perturbations in phrasing.  🔹 Multimodal Integration – Unlike traditional LLMs that primarily process text, LCMs seamlessly integrate text, speech, and other data types, enabling more intuitive, cross-lingual AI interactions.  Why LCMs Are a Paradigm Shift:  ✔️ Deeper Understanding: LCMs go beyond word prediction to grasp the underlying intent and meaning behind a sentence.  ✔️ More Structured Outputs: Instead of just generating fluent text, LCMs organize thoughts logically, making them more useful for technical documentation, legal analysis, and complex reports.  ✔️ Improved Reasoning & Coherence: LLMs often lose track of long-range dependencies in text. LCMs, by processing entire ideas, maintain context better across long conversations and documents.  ✔️ Cross-Domain Applications: From research and enterprise AI to multilingual customer interactions, LCMs unlock new possibilities where traditional LLMs struggle.  LCMs vs. LLMs: The Key Differences  🔹 LLMs predict text at the token level, often leading to word-by-word optimizations rather than holistic comprehension.  🔹 LCMs process entire concepts, allowing for abstract reasoning and structured thought representation.  🔹 LLMs may struggle with context loss in long texts, while LCMs excel in maintaining coherence across extended interactions.  🔹 LCMs are more resistant to adversarial input variations, making them more reliable in critical applications like legal tech, enterprise AI, and scientific research.  

Large Language models Are Dead. The naive view that LLMs are just high-dimensional autoregressive token predictors is outdated. In modern AI architectures, they function as sequence-to-sequence orchestrators, dynamically routing inputs through retrieval-augmented generation (RAG), multimodal transformers, execution engines, and external APIs. Current LLM implementations aren’t pure self-contained autoregressive models; they are hybridized latent variable models that integrate non-parametric memory, symbolic reasoning components, and differentiable API calls. When you interact with a state-of-the-art LLM, you aren’t simply traversing a static token embedding space…You are engaging with an AI agent that performs external function invocation, executes code, queries vector databases, and resolves multi-hop reasoning chains. This shift mirrors what happened in traditional computing. Early CPUs handled everything, but modern systems offload tasks to GPUs, TPUs, FPGAs, and dedicated accelerators. AI is moving in the same direction. LLMs now act as the “kernel” of an AI operating system, managing function calls rather than solving everything end-to-end. OpenAI’s GPT-4 Turbo, Google’s Gemini, and Meta’s latest models are not just LLMs—they are multimodal intelligence stacks. This transition makes AI system design exponentially harder. Engineers must balance memory-constrained transformers, external retrieval systems, latency constraints in function calling, and multi-agent coordination. The real challenge is no longer just improving the scaling laws of transformers, but designing robust hierarchical AI architectures that combine parametric and non-parametric reasoning. The frontier of AI is shifting. The race is no longer about building bigger LLMs. It’s about designing AI architectures that orchestrate reasoning, memory, execution, and multimodal perception at scale. LLMs aren’t dead, but they just a function of massive AI systems.

Large language models are remarkable—but we’re hitting the limits of what generation alone can do. Yes, LLMs can produce fluent, coherent, and even creative text. That’s a real form of intelligence. But it’s also bounded—by how we encode the data, and by what we ask the model to become. Two things are holding us back: 1. Tokenization Current tokenizers are built for compression, not cognition. They break language into fragments that lose semantic structure. If the input is lossy and shallow, how can we expect deep reasoning to emerge? 2. Instruction tuning We spend billions training models to follow instructions—to be polite, helpful, safe. That’s good UX. But it also flattens the model’s capabilities into a performance. The model learns to pretend, not to think. I believe the next leap won’t come from more parameters or more data. It’ll come from rethinking the foundations: • Smarter tokenization: encoding concepts, not just characters. • Pretraining for abstraction, not obedience. • Less optimization for chatbot behavior, more room for autonomous reasoning to emerge. LLMs today are great simulators. But if we want true intelligence, we need to stop forcing them to act like agents—and start training them to become them.

Top 10 research trends from the State of AI 2024 report: ✨Convergence in Model Performance: The gap between leading frontier AI models, such as OpenAI's o1 and competitors like Claude 3.5 Sonnet, Gemini 1.5, and Grok 2, is closing. While models are becoming similarly capable, especially in coding and factual recall, subtle differences remain in reasoning and open-ended problem-solving. ✨Planning and Reasoning: LLMs are evolving to incorporate more advanced reasoning techniques, such as chain-of-thought reasoning. OpenAI's o1, for instance, uses RL to improve reasoning in complex tasks like multi-layered math, coding, and scientific problems, positioning it as a standout in logical tasks. ✨Multimodal Research: Foundation models are breaking out of the language-only realm to integrate with multimodal domains like biology, genomics, mathematics, and neuroscience. Models like Llama 3.2, equipped with multimodal capabilities, are able to handle increasingly complex tasks in various scientific fields. ✨Model Shrinking: Research shows that it's possible to prune large AI models (removing layers or neurons) without significant performance losses, enabling more efficient models for on-device deployment. This is crucial for edge AI applications on devices like smartphones. ✨Rise of Distilled Models: Distillation, a process where smaller models are trained to replicate the behavior of larger models, has become a key technique. Companies like Google have embraced this for their Gemini models, reducing computational requirements without sacrificing performance. ✨Synthetic Data Adoption: Synthetic data, previously met with skepticism, is now widely used for training large models, especially when real data is limited. It plays a crucial role in training smaller, on-device models and has proven effective in generating high-quality instruction datasets. ✨Benchmarking Challenges: A significant trend is the scrutiny and improvement of benchmarks used to evaluate AI models. Concerns about data contamination, particularly in well-used benchmarks like GSM8K, have led to re-evaluations and new, more robust testing methods. ✨RL and Open-Ended Learning: RL continues to gain traction, with applications in improving LLM-based agents. Models are increasingly being designed to exhibit open-ended learning, allowing them to evolve and adapt to new tasks and environments. ✨Chinese Competition: Despite US sanctions, Chinese AI labs are making significant strides in model development, showing strong results in areas like coding and math, gaining traction on international leaderboards. ✨Advances in Protein and Drug Design: AI models are being successfully applied to biological domains, particularly in protein folding and drug discovery. AlphaFold 3 and its competitors are pushing the boundaries of biological interaction modeling, helping researchers understand complex molecular structures and interactions. #StateofAIReport2024 #AITrends #AI