Scaling Large Language Models from GPT-1 to GPT-3

📍 Day 31 of #100DaysOfAI 𝐇𝐨𝐰 𝐀𝐫𝐞 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 𝐑𝐞𝐚𝐥𝐥𝐲 𝐁𝐮𝐢𝐥𝐭? Many assume models like ChatGPT = “more parameters + more GPUs.” But after attending 𝐘𝐚𝐧𝐧 𝐃𝐮𝐛𝐨𝐢𝐬' 𝐠𝐮𝐞𝐬𝐭 𝐥𝐞𝐜𝐭𝐮𝐫𝐞 𝐚𝐭 𝐒𝐭𝐚𝐧𝐟𝐨𝐫𝐝 𝐂𝐒229 (𝐀𝐮𝐠 2024), I learned it's far deeper. Here’s a distilled breakdown: ♦ 𝐏𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 ≠ 𝐈𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐅𝐨𝐥𝐥𝐨𝐰𝐢𝐧𝐠 Pretraining teaches LLMs to predict the next word, not to help users or answer questions. Core building blocks: • 𝐀𝐮𝐭𝐨𝐫𝐞𝐠𝐫𝐞𝐬𝐬𝐢𝐯𝐞 𝐦𝐨𝐝𝐞𝐥𝐢𝐧𝐠 using the chain rule • 𝐂𝐫𝐨𝐬𝐬-𝐞𝐧𝐭𝐫𝐨𝐩𝐲 loss over massive token sequences • 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧 (e.g., BPE) — subtle, foundational --- ♦ 𝐃𝐚𝐭𝐚 > 𝐏𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 The quality of an LLM depends on the quality of its training data. Modern datasets include: • Billions of pages from Common Crawl • Heuristically filtered for PII, spam, and duplication • Domain-specific weighting (code, books, forums) --- ♦ 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐋𝐚𝐰𝐬 𝐆𝐨𝐯𝐞𝐫𝐧 𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 LLM design isn’t guesswork anymore. We now predict performance by balancing: • Data volume • Model size • Compute power 𝐂𝐡𝐢𝐧𝐜𝐡𝐢𝐥𝐥𝐚 𝐋𝐚𝐰: → ~20 tokens/parameter for training → ~150 tokens/parameter for inference  𝘊𝘩𝘪𝘯𝘤𝘩𝘪𝘭𝘭𝘢 𝘓𝘢𝘸 𝘴𝘩𝘰𝘸𝘴 𝘵𝘩𝘢𝘵 𝘵𝘳𝘢𝘪𝘯𝘪𝘯𝘨 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘤𝘺 𝘮𝘢𝘵𝘵𝘦𝘳𝘴 𝘮𝘰𝘳𝘦 𝘵𝘩𝘢𝘯 𝘴𝘪𝘻𝘦 𝘢𝘭𝘰𝘯𝘦. 𝘛𝘰 𝘮𝘢𝘹𝘪𝘮𝘪𝘻𝘦 𝘱𝘦𝘳𝘧𝘰𝘳𝘮𝘢𝘯𝘤𝘦, 𝘵𝘳𝘢𝘪𝘯 𝘸𝘪𝘵𝘩 ~20 𝘵𝘰𝘬𝘦𝘯𝘴 𝘱𝘦𝘳 𝘱𝘢𝘳𝘢𝘮𝘦𝘵𝘦𝘳, 𝘢𝘯𝘥 𝘦𝘹𝘱𝘦𝘤𝘵 ~150 𝘵𝘰𝘬𝘦𝘯𝘴 𝘱𝘦𝘳 𝘱𝘢𝘳𝘢𝘮𝘦𝘵𝘦𝘳 𝘵𝘰 𝘣𝘦 𝘶𝘴𝘦𝘥 𝘥𝘶𝘳𝘪𝘯𝘨 𝘳𝘦𝘢𝘭-𝘸𝘰𝘳𝘭𝘥 𝘪𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦. Example Let’s say you’re designing a model with 10B parameters. • 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 𝐧𝐞𝐞𝐝𝐞𝐝: 10B × 20 tokens = 200B tokens • 𝐓𝐨 𝐮𝐧𝐥𝐨𝐜𝐤 𝐟𝐮𝐥𝐥 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Expect to process/use 10B × 150 tokens = 1.5T tokens over various downstream tasks --- ♦ 𝐏𝐨𝐬𝐭-𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 = 𝐈𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 Pretraining alone doesn’t make helpful models. Post-training does: • SFT (Supervised Fine-Tuning): mimic high-quality responses • RLHF: align outputs to human preferences • DPO: Direct Preference Optimization — a simpler alternative to PPO with faster deployment --- ♦  𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐈𝐬 𝐇𝐚𝐫𝐝𝐞𝐫 𝐓𝐡𝐚𝐧 𝐈𝐭 𝐋𝐨𝐨𝐤𝐬 Perplexity works for pretraining — not for aligned models. Modern evaluation: • Chatbot Arena: Head-to-head human preferences • LLM-as-a-Judge: e.g., GPT-4 rating output quality But both come with biases — verbosity, prompt sensitivity, etc. --- I turned this lecture into a clear, visual blog for learners at any level: Read the blog ( https://lnkd.in/dZZp8UPs ) #AIWithAkanksha #LLMs #GenAI #StanfordCS229 #RLHF #DPO #Tokenization #ChinchillaLaw #MachineLearning #AIResearch 🗓️ 1st May 2025

If you’re an AI engineer, understanding how LLMs are trained and aligned is essential for building high-performance, reliable AI systems. Most large language models follow a 3-step training procedure: Step 1: Pretraining → Goal: Learn general-purpose language representations. → Method: Self-supervised learning on massive unlabeled text corpora (e.g., next-token prediction). → Output: A pretrained LLM, rich in linguistic and factual knowledge but not grounded in human preferences. → Cost: Extremely high (billions of tokens, trillions of FLOPs). → Pretraining is still centralized within a few labs due to the scale required (e.g., Meta, Google DeepMind, OpenAI), but open-weight models like LLaMA 4, DeepSeek V3, and Qwen 3 are making this more accessible. Step 2: Finetuning (Two Common Approaches) → 2a: Full-Parameter Finetuning - Updates all weights of the pretrained model. - Requires significant GPU memory and compute. - Best for scenarios where the model needs deep adaptation to a new domain or task. - Used for: Instruction-following, multilingual adaptation, industry-specific models. - Cons: Expensive, storage-heavy. → 2b: Parameter-Efficient Finetuning (PEFT) - Only a small subset of parameters is added and updated (e.g., via LoRA, Adapters, or IA³). - Base model remains frozen. - Much cheaper, ideal for rapid iteration and deployment. - Multi-LoRA architectures (e.g., used in Fireworks AI, Hugging Face PEFT) allow hosting multiple finetuned adapters on the same base model, drastically reducing cost and latency for serving. Step 3: Alignment (Usually via RLHF) Pretrained and task-tuned models can still produce unsafe or incoherent outputs. Alignment ensures they follow human intent. Alignment via RLHF (Reinforcement Learning from Human Feedback) involves: → Step 1: Supervised Fine-Tuning (SFT) - Human labelers craft ideal responses to prompts. - Model is fine-tuned on this dataset to mimic helpful behavior. - Limitation: Costly and not scalable alone. → Step 2: Reward Modeling (RM) - Humans rank multiple model outputs per prompt. - A reward model is trained to predict human preferences. - This provides a scalable, learnable signal of what “good” looks like. → Step 3: Reinforcement Learning (e.g., PPO, DPO) - The LLM is trained using the reward model’s feedback. - Algorithms like Proximal Policy Optimization (PPO) or newer Direct Preference Optimization (DPO) are used to iteratively improve model behavior. - DPO is gaining popularity over PPO for being simpler and more stable without needing sampled trajectories. Key Takeaways: → Pretraining = general knowledge (expensive) → Finetuning = domain or task adaptation (customize cheaply via PEFT) → Alignment = make it safe, helpful, and human-aligned (still labor-intensive but improving) Save the visual reference, and follow me (Aishwarya Srinivasan) for more no-fluff AI insights ❤️ PS: Visual inspiration: Sebastian Raschka, PhD

At some point, LLMs stop just getting better - and start getting weirdly smarter. This idea - called emergent abilities - is not just AI hype. It is one of the most fascinating shifts happening as we scale language models. Researchers from Google and Stanford recently dug into this - and the findings are wild. 𝐋𝐞𝐭’𝐬 𝐮𝐧𝐩𝐚𝐜𝐤 𝐢𝐭: 𝟏. 𝐒𝐤𝐢𝐥𝐥𝐬 𝐣𝐮𝐬𝐭… 𝐚𝐩𝐩𝐞𝐚𝐫. Models do not slowly learn tasks like arithmetic or coding. They fail. Fail. Fail. Then suddenly - at a certain size - they nail it. 𝟐. 𝐘𝐨𝐮 𝐜𝐚𝐧𝐧𝐨𝐭 𝐩𝐫𝐞𝐝𝐢𝐜𝐭 𝐢𝐭. Smaller models give you zero signal these abilities are coming. No curve. Just a cliff. 𝟑. 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐬 𝐛𝐫𝐞𝐚𝐤. Most evaluation metrics expect gradual improvement. But emergent skills show nonlinear jumps. We are measuring the wrong things in the wrong way. 𝟒. 𝐈𝐭 𝐢𝐬 𝐧𝐨𝐭 𝐚𝐛𝐨𝐮𝐭 𝐭𝐡𝐞 𝐦𝐨𝐝𝐞𝐥 𝐭𝐲𝐩𝐞. GPT-3, PaLM, Chinchilla, Gopher - they all show this. What triggers it? Scale. Not architecture. 𝟓. 𝐖𝐡𝐲 𝐢𝐭 𝐦𝐚𝐭𝐭𝐞𝐫𝐬: • You might be using a model that has hidden capabilities - just not prompted correctly. • Evaluation needs a rethink. • Safety, trust, and alignment take on new complexity when abilities show up unannounced. We are not just scaling performance anymore. We are crossing thresholds into new behaviour. And that changes everything - from how we build, to how we prompt, to how we think about what is possible. Link to paper: https://lnkd.in/edyATvFB Have you seen these jumps in your own work with LLMs? Drop your stories below - I am curious.

GPT-5 just launched. But it didn’t come out of nowhere. Every version before it taught us something about what AI can and can’t do. Here’s the journey so far - from simple text prediction to real-time multimodal agents — and what it means for developers, product teams, and enterprises building with AI: GPT-5 (Aug 2025) Finally feels like an assistant, not a tool. With 1 million-token context and mini to pro versions, it’s powering long memory agents that can summarize entire corpuses, auto-document legacy code, or write RFPs end-to-end. Some teams are already using GPT-5 to replace human QA on test automation. Yet to see how effective it will be. GPT-4.5 (2025) Quietly one of the most accurate releases. Marked the shift to real-time use: uploading files, running web searches, and doing live data extraction. Still only on ChatGPT Pro, but was the “stealth upgrade” everyone noticed without a major launch. GPT-4o and GPT-4o Mini (2024) Omni means one model for all inputs — text, images, audio, and even video. With faster response times and stronger multilingual skills, these versions powered the first practical voice agents that didn’t sound robotic. Great fit for customer service and language learning tools. GPT-4.1 (2025) Highly structured thinker. Better at following instructions, writing clean code, and responding in exact formats. This model won favor in dev tools, especially where reliability matters — like generating SQL queries or step-by-step code edits. GPT-4 (2023) A major milestone. First time models could handle visual inputs and pass complex exams. Scored in the top 10% of the simulated bar exam. Used in everything from legal assistants to document parsing. GPT-3.5 and 3.5 Turbo (2022) What most people think ChatGPT is. Turbo charged response time and cost-efficiency. Enabled the boom in customer-facing chatbots and knowledge bases. Still used in many apps behind the scenes today. GPT-3 (2020) The breakout star. 175B parameters and a massive leap from previous models. Powered the first wave of AI writing tools and showed us what “few-shot learning” could look like. GPT-2 (2019) Impressive at the time but hit or miss. Could generate paragraphs that sounded human, but often lost the thread. Its release sparked debate about the risks of synthetic text. GPT-1 (2018) Where it all began. 117M parameters, trained on BookCorpus. Could answer simple questions but was mostly academic. So what now? With GPT-5, we’re entering the age of true long-memory assistants and AI-native workflows. The real challenge is no longer: “Can this model do it?” It’s: “How do we redesign our teams, tools, and systems to let it?” #AI #GPT5 #OpenAI #EngineeringLeadership #GenAI #FutureOfWork #Productivity #AIagents #MetaShift