Zyphra

Excited to have Zyphra's Beren Millidge Rishi Iyer Robert Washbourne at AMD’s annual Advancing AI conference speaking about how Zyphra built the first MoE Diffusion LLM on AMD. Registration still open:  https://lnkd.in/eVu4QJv8

Today we're releasing ZUNA1.1, a substantially more flexible version of ZUNA1, our open-source 380M-parameter foundation model for EEG channel reconstruction, denoising, and upsampling.  ZUNA1.1 is open-source, Apache 2.0, and available for free to try in your browser on the Zyphra Cloud in our EEG Playground. Real EEG is rarely clean, fixed-length, or densely instrumented. ZUNA1.1 is built to solve these problems. Two changes make it broadly applicable to messy real-world EEG data, much more so than our previous ZUNA1 model. First, it can process variable-length recordings from 0.5 to 30 seconds, rather than only fixed 5-second windows. Second, it was trained on a much richer set of reconstruction tasks, so it can operate across a far wider range of real-world conditions. ZUNA1.1 remains channel-agnostic, conditioning on true electrode positions to generalize across any montage without retraining from four-electrode consumer headbands to 256-channel research caps. We used quality-aware preprocessing to grow the training corpus to roughly 3.5 million channel-hours of public EEG. Reconstruction quality exceeds ZUNA1 on average across our benchmarks, while ZUNA1.1 is also capable of reconstructing across a far broader range of realistic conditions. The weights remain fully open under the Apache 2.0 license, so teams can self-host. ZUNA1.1 is also available via our Zyphra Cloud: you can now try ZUNA 1.1 instantly in your browser on cloud.zyphra.com, no GPU required. Organizations and researchers interested in collaborating on future versions for specific needs should reach out to Krithik Puthalath, Beren Millidge, Paul White Great work by the Zyphra BCI team Chris Warner, Ph.D. , Jonas Mago, Jon Huml Learn more: Technical blog:  https://lnkd.in/gCe8e8-F Hugging Face:  https://lnkd.in/g3kQ26Ss GitHub:  github.com/Zyphra/zuna PIP Install Package - [pip install zuna]

Today we're releasing new research from Zyphra Research, and it marks our first step into continual learning. Continual learning is the ability of a model to keep learning and improving after its initial training, taking in new facts, domains, and languages over time. Many in the field see it as one of the most important open problems on the path to AGI through recursive self improvement (RSI). One of the principal obstacles to continual learning is a phenomenon called loss of plasticity. As a network trains for longer and longer, it gradually loses its ability to learn anything new. This is different from forgetting old information. The model simply becomes worse at absorbing new information at all. We studied this in GPT-style language models by training them on a long, repeating sequence of languages, then periodically testing how quickly they could pick up a brand new language they had never seen. Across every model size we tested, the same pattern emerged. The models improved at first, due to transfer learning, but then steadily got worse at learning new material – demonstrating loss of plasticity. Larger models held out longer, but none of them avoided the decline, and our analysis shows that adding parameters buys less and less delay each time. Scale alone cannot save us from plasticity loss. We also found that this is not only a problem for systems that switch between tasks, like in the classical plasticity literature. Even when we trained on a single fixed mixture of data with no task changes at all, just as we do in standard pretraining, plasticity still eroded over long training horizons. Our results suggest loss of plasticity may quietly affect many models trained today, not just specialized continual-learning systems. We looked inside the networks for a clear cause, measuring several signals that change as models lose plasticity. None of them fully explains the effect, and we do not yet have a single fix. We believe finding one is an important direction for the field, and this work is our first step toward it. Read the technical paper: arxiv.org/abs/2606.24752 Read the blog: https://lnkd.in/gDSUfyCh Zyphra Research focuses on novel architectures, intelligence efficiency, and silicon performance. Advances in learning algorithms are foundational to all three, and to the future of efficient, open superintelligence. Great work by Juan Fernando Hernandez, Tomás Figliolia, Beren Millidge, and Krithik Puthalath

Today we're releasing new research from Zyphra Research, Norm-AGnostic (NAG) residual networks, a new architecture for building deeper and more efficient AI models. Modern language models often do the same amount of work on every token. Inside the model, a long stack of layers repeatedly adds updates to a shared running state: the residual stream. But as this stream passes through more layers, its norm keeps growing. Later layers therefore write into a much larger accumulated state than earlier ones, so updates of the same size have a smaller relative effect on the representation. As a result, deeper layers can become progressively less influential while still costing the same FLOPs, memory, parameters, and training time. Much of a model’s depth goes underused. NAG fixes this by separating the representation a model is building from the raw size of the residual stream. Layers operate on the normalized direction while norm is carried separately, allowing each layer to make a controlled, comparable contribution regardless of depth. In our experiments, the benefit grows as models get deeper, and the internal state no longer balloons the way it usually does. The same design unlocks a natural form of adaptive computation. Because NAG can estimate in advance how much a layer will rotate a token’s residual state, it can skip layers that would barely move the representation. Skipping layers has usually been treated as an inference-time efficiency trick, often at the cost of quality. NAG makes that tradeoff much smaller, allowing depth skipping to be used during pretraining itself. At around 20–25% sparsity, the saved computation can be reinvested into training on more data, producing a model that remains just as capable while being cheaper to run. To our knowledge, this is the first strong case for using depth sparsity as a pretraining-time scaling axis. These are early, smaller-scale studies, but the benefits grow with depth, which points toward training far deeper and more efficient models than are practical today. Longer term, NAG points toward architectures that behave less like fixed-depth feedforward stacks and more like differentiable computers in residual space: systems that allocate compute dynamically, run variable-depth internal computation, and scale inference without forcing every intermediate step through natural language. Zyphra Research focuses on novel architectures, intelligence efficiency, and silicon performance. Architectures that make every layer count are foundational to all three, and to the future of efficient, open superintelligence. Read the technical report: arxiv.org/abs/2606.16112 Read the technical blog: https://lnkd.in/gfVzHAux Great work by Tomás Figliolia, Beren Millidge, Krithik Puthalath, Paul White

Today we're releasing ZONOS2, our next-generation real-time text-to-speech model with high-fidelity voice cloning. ZONOS2 is the most expressive open-source TTS model, while also being the first open-source MoE TTS. Available for free for a limited time on Zyphra Cloud running on AMD. Real-time speech synthesis has historically forced a choice between quality and speed. ZONOS2 mitigates that tradeoff and achieves both. Compared to leading proprietary and open-source TTS models, it performs especially well at speaker similarity and prosody, the qualities that determine whether a voice actually sounds human. ZONOS2 is a sparse Mixture-of-Experts model with 8 billion total parameters and 900 million active. This coupled with scaling our training data and our architecture improvements results in fewer hallucinations and more natural speech across a far wider range of voices and languages. The ZONOS2 weights are also downloaded under Apache 2.0, with inference code available for self-hosting. Try it on Zyphra Cloud: cloud.zyphra.com Check out the blog: zyphra.com/our-work/zonos Model weights on Hugging Face: https://lnkd.in/g8MpP_A9 Inference code on GitHub: github.com/Zyphra/ZONOS2

Zyphra Research continues to explore architecture innovations beyond standard Transformers. Today we’re releasing Zamba2-VL, our second series of vision-language models, extending our prior Zamba2 work on hybrid SSM-Transformer architectures into the visual domain. Zamba2-VL is released at three scales: 1.2B, 2.7B, and 7B parameters. It is competitive with the leading open Transformer-based vision-language models of comparable scale across image understanding, reasoning, OCR, grounding, and counting benchmarks, while delivering roughly an order of magnitude faster time-to-first-token at every scale. Prior SSM-based vision-language models were either distilled from existing Transformer models or built on weaker open SSM bases. Zamba2-VL demonstrates that the inference-efficiency advantages of hybrid state-space LLMs carry cleanly into the multimodal setting. Pure SSMs struggle to recall specific details from earlier in a long input. The hybrid design keeps that ability by mixing in a small number of attention layers where it matters. We are releasing Zamba2-VL as a research artifact for the open community.  Zamba2-VL is released under Apache 2.0 with weights freely available on Hugging Face. Read the blog: https://lnkd.in/giZqTUSK Read the technical report:  arxiv.org/abs/2606.00390 Model weights on Hugging Face: https://lnkd.in/gRiYxRvz Inference code: https://lnkd.in/guGNFn9v

At Zyphra Research, we co-design models and kernels across heterogeneous silicon evaluating various accelerators by workload fit, cost, and engineering effort. In collaboration with Amazon Web Services (AWS), Zyphra Research evaluated AWS Trainium/Inferentia and AWS Neuron stack for its potential use in workloads where communication and memory movement dominate cost, including decode-bounded inference, long-context attention, and MoE-style models. To test this regime, we implemented Domino-style tensor-parallel communication overlap within the AWS Neuron inference stack and benchmarked Llama 3-8B across varying NeuronCore counts on Inferentia2. Our results improve aggregate output throughput and reduce both time-to-first-token and time per output token, with the clearest gains appearing at higher tensor-parallel widths where collective communication becomes a larger portion of the critical path. Our results demonstrate Zyphra’s ability to optimize across the Trainium/Inferentia stack and highlights the systems work required to unlock performance across multiple hardware ecosystems. For further details, check out the blog:  https://lnkd.in/g6gVq3_X

Today we’re releasing new research from Zyphra Research. Modern AI is built on backpropagation, the algorithm behind nearly every neural network trained today. It's powerful, but it has a known limitation: it requires a separate "backward" network to teach the model, which is part of why training frontier AI is so resource-intensive. Our team has extended a learning algorithm, Equilibrium Propagation, to a broader class of systems that includes biophysical neuron models. This is a step toward learning methods that do not depend on explicit backward passes and may enable AI systems beyond today’s GPU-centric hardware stack. Zyphra Research focuses on novel architectures, intelligence efficiency, and silicon performance. Advances in learning algorithms are foundational to all three, and to the future of efficient, open superintelligence. Read the full paper:  arxiv.org/abs/2605.21568 Read the technical blog: https://lnkd.in/g7RA344C

Today we're publishing end-to-end benchmarks for Zyphra Inference, running on AMD Instinct™ MI355X GPUs. These are production numbers from a single node of our cloud, serving three leading open models: Kimi K2.6, GLM 5.1, and DeepSeek V3.2. They represent the first phase of our full-stack optimization effort spanning kernels, parallelism, communication, quantization, and speculative decoding. A note on how we approached this. Fair inference benchmarking is genuinely difficult, so we've been explicit about our methodology and its limits. These are single-node results without live traffic, measured on vLLM against standard datasets (ShareGPT and LMSYS-Chat-1M), at a fixed point in time. We'd rather show grounded numbers our customers can trust than headline figures that don't survive scrutiny. The MI355X has a greater memory capacity compared to the B200  at 288GB of HBM3E per GPU vs 180GB for the B200. This enables holding more KV and prefix cache, supporting longer context per replica, and requiring less parallelism to serve a model. That translates to larger models, longer contexts, and more concurrent users per node. Models at this scale require multi-GPU parallelism to serve, which puts intra-node fabric performance at the center of inference efficiency.  Zyphra Research has developed two novel parallelism and communication schemes specifically for this: Tree Attention, which reorganizes long-context attention around AMD's fabric, and Tensor-Sequence Parallelism, which keeps the full model-parallel group on high-bandwidth intra-node links. Combined with kernel tuning and ROCm-tuned speculative decoding, these are the foundation of the performance we deliver on AMD. This is the first round of optimization, not the last. Upcoming work extends to other models like DeepSeek V4 (1.6T-parameter models, 1M-token context), training-informed quantization, and diffusion-based speculative decoding. Zyphra Inference is available now at cloud.zyphra.com. Built in partnership with AMD. Read more about our full methodology and benchmarking here: https://lnkd.in/dvXdyggy

Today we're releasing ZAYA1-8B-Diffusion-Preview, our first diffusion language model, and the first diffusion language model trained on AMD . Almost every LLM in production today is autoregressive, generating text one token at a time. This is fundamentally bottlenecked by memory bandwidth. Each new token requires reloading the model's working memory (the KV-cache), leaving the GPU sitting idle or underutilized. As inference workloads scale and reasoning models generate longer outputs, this bottleneck has become one of the largest cost and latency drivers in deploying LLMs. Diffusion language models take a different approach. Rather than generating one token at a time, they draft entire blocks of tokens in parallel and then verify them in parallel. Because the model produces many tokens in a single forward pass (using the same KV-cache), the GPU stays compute-bound and operates closer to peak utilization. We built ZAYA for diffusion from the start, co-designing the architecture around AMD hardware. Our FLOP-efficient CCA attention variant lets us diffuse more tokens at once before becoming compute-bound, dramatically increasing inference speed. Training a diffusion LLM from scratch is challenging, and few proven recipes exist. With ZAYA1-8B-Diffusion-Preview, we take a different approach: building on the TiDAR recipe, we convert our existing autoregressive ZAYA1-8B model into a discrete diffusion model through additional mid-training. This lets us reuse our existing pretraining stack while gaining the inference benefits of diffusion. The result is the first MoE diffusion model converted from an autoregressive LLM. The preview model: - Diffuses blocks of 16 tokens simultaneously, achieving up to 4.6x decoding speedup with a lossless sampler and up to 7.7x with our new logit-mixing sampler - Shows minimal performance degradation versus the autoregressive base, with gains on several evaluation benchmarks - Was trained and converted on AMD, with our compute-efficient CCA attention variant and the larger VRAM capacity of AMD hardware making diffusion training practical. Beyond raw inference speed, diffusion enables more expressive generation than left-to-right autoregression and substantially reduces the cost of on-policy rollouts. This translates to more reinforcement learning and test-time compute scaling for the same compute budget, areas where diffusion's advantages compound over time. ZAYA1-8B-Diffusion-Preview is an early step. We plan to post-train and release a full diffusion model soon. Read more: https://lnkd.in/gsrQCFc3