The transformer architecture from the 2017 "Attention Is All You Need" paper is not what's running inside modern LLMs. Llama, Mistral, and GPT-era models all use a modified version with 4 key upgrades. Here's what changed and why it matters: 1. RMS Norm instead of Layer Norm Layer Norm did 4 operations — calculate mean, center around zero, calculate variance, normalize. Researchers found that centering around zero wasn't actually doing anything useful. What matters is just keeping numbers in bounds. So they removed it. RMS Norm does 2 operations instead of 4. Since normalization runs after every single layer in a deep model, this compounds into a significant efficiency gain. They also moved normalization to run before each layer rather than after — giving the model cleaner inputs from step one. 2. SwiGLU instead of ReLU ReLU has a hard cutoff at zero — negative values get zeroed out completely. The problem is small negative values often carry meaningful information. SwiGLU introduces a learned gating mechanism — one path decides what information matters, another carries the raw signal, and they're multiplied together. The gate is trained alongside the model, so it gets progressively better at filtering signal from noise. Every major frontier model uses this today. 3. RoPE instead of Sinusoidal Positional Embeddings Original positional embeddings were purely absolute — each token only knew its own position number. This meant the same phrase appearing at different positions in a sequence was treated as completely different information, forcing the model to re-learn it each time. RoPE encodes relative distance instead. By rotating Query and Key vectors based on position, the angle between any two tokens reflects how far apart they are — regardless of where they appear in absolute terms. This is why the clock analogy works: 12 and 2 have a 60 degree gap. 7 and 9 have the same 60 degree gap. Relative distance preserved. 4. Group Query Attention instead of Multi-Head Attention Standard multi-head attention stores separate Key and Value vectors for every head — memory scales linearly with heads and sequence length. Multi-Query Attention tried collapsing all heads to share one K and V pair — too much compression, performance degraded. Group Query Attention found the middle ground: group heads in pairs, each pair shares one K and V. Memory reduced by half. Performance nearly identical to full multi-head. This is the standard in production-scale models today. These four changes — faster normalization, smarter activation, relative position encoding, and memory-efficient attention — are what made the transformer actually viable at scale. #MachineLearning #DeepLearning #ArtificialIntelligence #LLM #Transformers #AIEngineering #GenerativeAI #BuildInPublic #LearnInPublic #DataScience
More Relevant Posts
-
Kimi K3 Architecture Figure 𝗕𝗲𝗰𝗼𝗺𝗲 𝗮 𝗠𝗮𝘀𝘁𝗲𝗿 𝗮𝘁 𝗔𝗜 𝗶𝗻 𝗷𝘂𝘀𝘁 𝟯 𝗗𝗮𝘆𝘀 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗕𝗲𝘀𝘁 𝗔𝗜 𝗟𝗲𝗮𝗱𝗲𝗿𝘀 (1 Free Ticket) Hear directly from the best AI Leaders and Industry Experts, come with your toughest questions. Learn from their real-world experience, practical insights, and get the opportunity to interact with them live. 𝗦𝗶𝗴𝗻 𝘂𝗽 𝗻𝗼𝘄 → https://lnkd.in/gUR-jZ2N Original post: __________ The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. Repost and share with your network to help them Credit to Sebastian Raschka, PhD. Follow him for more. Don’t just study these resources. Come learn directly from these people building them and become a Master. 1 free ticket for you 𝗦𝗶𝗴𝗻 𝘂𝗽 𝗻𝗼𝘄 → https://lnkd.in/gUR-jZ2N #ai #future #education #Creativity #jobs
-
-
Poolside's architecture decision on Laguna S 2.1 encodes a specific engineering bet: a 118B-parameter MoE with only 8B active parameters per token keeps the full expert set resident in memory while routing through roughly 6.8% of weights per step, which means the hardware floor is set by total parameters, not active ones. From the Poolside technical release: "All 118B parameters remain resident in memory, but only approximately 8B route through the network per step." ⚙️ That residency requirement is the constraint that determines deployment viability before any benchmark number enters the conversation. At 4-bit quantization (NVFP4 or INT4), the weights occupy approximately 59 GB, which fits within the 128 GB unified memory of a single DGX Spark. FP8 pushes to roughly 118 GB, still within a single Spark or H200. BF16 requires approximately 236 GB, meaning two linked Sparks or a multi-GPU datacenter node. Poolside reports day-one support for vLLM, SGLang, and Ollama, with hosted access through OpenRouter priced at $0.10/$0.20/$0.01 per million input/output/cache-read tokens at the full 1M context window. On the benchmark side, the article reports 70.2% on Terminal-Bench 2.1 with thinking enabled and 78.5% on SWE-Bench Multilingual, topping the published table outright among open disclosed-size models. The more telling number is DeepSWE v1.1: Laguna S 2.1 at 40.4% against DeepSeek-V4-Pro-Max at 9.0%, with roughly one-sixth the active parameters. The gain from disabling to enabling max thinking on DeepSWE alone is 16.5% to 40.4%, at a cost of approximately 249k completion tokens versus 99k without thinking. That token multiplier matters for cost-per-call economics on any workload where thinking is always-on. What the performance table leaves unresolved is where the score comes from at the trajectory level. Poolside published three unedited runs at trajectories.poolside.ai, including a 181-step browser engine construction and a harness optimization that reduced memory allocation by roughly 71%, but the benchmark aggregate does not decompose by task category, repository size, or language distribution, which makes it difficult to characterize where the model's agentic reasoning is actually load-bearing versus where a smaller or cheaper model would have been sufficient. #AI #MachineLearning #AgenticAI #LLMs #AIEngineering https://lnkd.in/epB9GF7c
-
-
The underlying architectural changes in KIMI are foundational & it being #openWeights with 3 Trillion parameters is more than Amazing.
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
Looking at architectures like Kimi K3, I'm reminded of the evolution of DevOps and cloud computing. As cloud platforms matured, the focus shifted from simply provisioning infrastructure to automation, orchestration, and efficient resource utilization. LLM architectures seem to be following a similar path. Instead of simply building larger models, the emphasis is increasingly on architectural innovations—such as Mixture of Experts (MoE), LatentMoE, Multi-Head Latent Attention, Delta Attention, and NoPE—that improve inference efficiency, reduce memory usage, and lower operational costs. For enterprise AI, these engineering trade-offs may become just as important as model quality. Thanks to Sebastian Raschka, PhD for the detailed architecture breakdown and technical insights. It's a great resource for anyone interested in how modern LLM architectures are evolving. #KimiK3 #LLM #GenerativeAI #AIEngineering #PlatformEngineering #DistributedSystems #MLOps #EnterpriseAI
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
A good quick read item explaining how new Kimi model deals with 1M token window with such a low-cost computing. It seems their delta attention layers (now inherently delves with temporal ordering implicitly rather than any explicit injection. So, it is plausible that fixed-size state with KDA. Eventually, inference gets cheaper due to reduced KV-cache requirement, though multi-head attention layers seem needing to grow KV cache a bit, but the ratio 3:1 is okay.
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
Spent the morning digging into the Kimi K3 technical report. As someone heads-down building a vertical LLM for investments & sustainability, releases like this are both humbling and clarifying. K3 is now the largest open-weight model at 2.8T params. But what struck me isn't the scale, it's the philosophy of making every FLOP count: → LatentMoE and Latent Attention replacing standard components → NoPE everywhere - no positional embeddings at all → Native multimodal out of the box In financial AI, you can't burn GPU cycles for marginal gains. You parse filings, extract ESG signals from PDFs, and reason across tables and text at a cost enterprises will actually deploy. The lesson for vertical builders: architecture innovation beats raw scale. The efficiency tricks that make 2.8T viable are the same ones that let a 70B domain-tuned model punch two weight classes above its size. What architectural shift are you betting on right now? OpenAI Anthropic DeepSeek Amazon Web Services (AWS) #KimiK3 #VerticalAI #Finance #Sustainability #Investment #LLM #OpenWeight #Vaia
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
I like the way Sebastian Raschka, PhD shares the architecture of every new model that comes in the market. Through this, we get to know the trade-offs of the architecture of the models.
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
Curious about Delta Attention in Kimi-k3: would replacing or replicating positional embeddings with a conv layer be an effective design choice?
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
-
Founder & Tech Lead | AI/ML | Cheminformatics Software | Computational Chemistry/Biology | NMR
Open-weight, natively multimodal LLM with a 1M-token context window, competing with frontier commercial models in reasoning and coding. Its API pricing, however, is comparable to Claude and Codex.
ML/AI research engineer. Author of Build a Large Language Model From Scratch (amzn.to/4fqvn0D) and Ahead of AI (magazine.sebastianraschka.com), on how LLMs work and the latest developments in the field.
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) 2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention. 3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details). 4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. 5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. 6. Kimi K3 now also has native multimodal support, which is great! There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
-
Explore content categories
- Career
- Productivity
- Finance
- Soft Skills & Emotional Intelligence
- Project Management
- Education
- Technology
- Leadership
- Ecommerce
- User Experience
- Recruitment & HR
- Customer Experience
- Real Estate
- Marketing
- Sales
- Retail & Merchandising
- Science
- Supply Chain Management
- Future Of Work
- Consulting
- Writing
- Economics
- Artificial Intelligence
- Employee Experience
- Workplace Trends
- Fundraising
- Networking
- Corporate Social Responsibility
- Negotiation
- Communication
- Engineering
- Hospitality & Tourism
- Business Strategy
- Change Management
- Organizational Culture
- Design
- Innovation
- Event Planning
- Training & Development