Intuit Open Source’s Post

We reviewed the same document twice in the same session. Scores dropped 30%+. No document changes. That's not noise. That's a broken measurement instrument. I've been running a structured case study on LLM reliability as quality gates — using three frontier models as independent evaluators across multiple adversarial debate rounds. The core finding is uncomfortable: LLMs are not unreliable because they're bad models. They're unreliable because consistency is a systemic property, not an individual one. Humans are consistent because rules are externally enforced — traffic law, employment contracts, professional standards. The enforcement structure exists independently of any individual's judgment. LLMs have no equivalent. No external enforcement. No consequences. No persistent commitment to a prior position. Every evaluation is context-only, consequence-free, stateless. The practical implication for engineering teams: → Don't gate merges on LLM numeric scores. They're not reproducible — even within the same session. → Use binary blocking findings (present/absent) instead. → LLM review = finding generator. Deterministic infrastructure = the actual gate. The paper also documents an honest methodological problem: the debate that illustrates the thesis was conducted using the very instruments the thesis defines as unreliable. We don't resolve the circularity — we disclose it. The load-bearing evidence is the rating drift data, which requires no LLM judgment to interpret. The fix isn't better prompting or better models. It's an external enforcement layer. Full case study published on Notion 👇 https://lnkd.in/eD3hzPym Are you using LLM scores as merge gates? What's your stopping condition? #LLM #AIEngineering #SoftwareQuality #MLOps #AIGovernance #EngineeringLeadership

Most LLM benchmarks measure one thing “Did the model get the answer right?” That’s not enough anymore After running large-scale evaluations I kept seeing the same pattern A model could be: correct… but statistically misleading efficient… but unreliable cheap… but impossible to maintain accurate… but terrible at adapting to real environments So I stopped thinking about benchmarks as a single score Instead, I started evaluating models across 4 independent dimensions: Correctness → did it produce the right answer? Code Quality → is the solution maintainable and production-ready? Efficiency → how much compute, latency, and iteration did it require? Statistical Validity → are the conclusions actually reliable? That last one is the dimension most benchmarks completely ignore And honestly, it changes the rankings more than people expect Because in real workflows, “correct” doesn’t always mean “trustworthy.” The future of LLM evaluation probably isn’t a leaderboard It’s a failure profile

Where the Constraint Came From I didn’t set out to write a constraint paper; the constraint paper found me. I set out to understand why evaluating AI systems felt structurally different from evaluating the software that came before them. Not harder in a tooling sense, but different in a more fundamental way, as if the thing being evaluated wasn’t fully present at the moment of evaluation. That intuition stayed with me for a while before I tried to formalize it. This research began in adjacent—and still practical—territory. I worked through observability theory, software architecture, and control systems: disciplines that approach related problems from different angles. Each was genuinely adjacent, yet none quite named what I was trying to name. The question I kept returning to wasn’t whether internal state could be reconstructed; it was what evaluation can legitimately claim at the moment a participant encounters a system’s output. Those are different questions. Eventually the path led me to Shannon. I came to Shannon the same way I came to Nyquist twenty years earlier—for engineering reasons, not theoretical ones. I spent years working with telecommunications systems: TDM, voice codecs. Understanding how G.711 encoded voice at an 8 kHz sampling rate meant understanding Nyquist’s theorem directly. Understanding why G.729 could encode the same voice with a fraction of the bandwidth—despite using a completely different mechanism—meant understanding that the boundary condition Nyquist identified survived a fundamental change in technology. The principle held even when everything else changed. Shannon formalized that condition mathematically. His equivocation—the precise measure of what cannot be recovered from what crossed a channel boundary—is the closest prior statement to what the Interaction Boundary Constraint is saying. The distinction is equally precise: Shannon measures what is lost in transmission; the constraint characterizes what was never transmissible in the first place. Nyquist and Shannon were colleagues at Bell Laboratories. Nyquist arrived in 1917; Shannon in 1941. Shannon’s 1948 paper cites Nyquist directly—not as a passing reference, but as foundational to his framework. Two people in the same institution, twenty years apart, building explicitly on each other. I came to both of them through applied engineering decades later, from an entirely different direction. The influence was already there long before I had the language for what I was trying to describe.

Two systems. Same model. Completely different outcomes.⁣ The difference? 𝐂𝐨𝐧𝐭𝐞𝐱𝐭.⁣ ⁣ In LLM systems, performance is not just about the model. It is about what the model sees.⁣ ⁣ Two setups can use the same model and produce very different results.⁣ ⁣ The difference is usually context.⁣ ⁣ Instead of swapping models, 𝐭𝐫𝐲 𝐜𝐡𝐚𝐧𝐠𝐢𝐧𝐠:⁣ → How much information you pass?⁣ → How it is structured?⁣ → What gets included or excluded?⁣ → The order in which it appears.⁣ ⁣ Small changes here can shift:⁣ → Accuracy⁣ → Hallucination⁣ → Cost⁣ ⁣ 𝐅𝐨𝐫 𝐞𝐱𝐚𝐦𝐩𝐥𝐞:⁣ A system using fewer, more relevant chunks often outperforms one that sends everything.⁣ ⁣ More context does not mean better answers. It often means more noise.⁣ ⁣ The key is to 𝐭𝐫𝐞𝐚𝐭 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 𝐥𝐢𝐤𝐞 𝐚 𝐯𝐚𝐫𝐢𝐚𝐛𝐥𝐞.⁣ Test it.⁣ Measure it.⁣ Refine it.⁣ And do not rely on one metric.⁣ ⁣ Accuracy alone is not enough.⁣ Look at:⁣ → Correctness⁣ → Hallucination⁣ → Latency⁣ → Cost⁣ ⁣ Because the best system is not just accurate. It is efficient and reliable.⁣ ⁣ Most teams chase better models. But the real leverage is in better context design.⁣ ⁣ If you are not testing context, you are guessing.⁣ ⁣ What part of your context pipeline would you test first?⁣ ⁣ #ContextEngineering #RAG #LLM

Most senior engineers debug by intuition. Mathematically, that's insane.🤯 Here's what I mean: In a system with 1,048,576 possible failure points, binary search finds the broken one in 20 tests. Twenty. Not two hundred. But almost nobody debugs this way. We chase hunches. We re-check what we already checked. We "feel" where the bug is. @Steve Litt calls the alternative Divide and Conquer: 1. Map the system as a block diagram in your head 2. Pick a test that splits the remaining search space in half 3. Rule out one side 4. Repeat That's it. No genius required. Just discipline. The trap: picking the test that "feels likely" instead of the one that splits cleanest. Likely ≠ informative. A test that confirms a 90% hunch tells you almost nothing when it passes. A test that splits the space 50/50 always cuts your work in half. Next time you're 2 hours into a bug, stop. Ask one question: "What single test would rule out the most ground right now?" Then run that one. Not the easy one. Not the obvious one. The one that halves the unknown.

Everyone is turning simple logic into an LLM call. That’s not innovation — it’s architectural decay. I’ve seen this pattern creep into production systems. Replacing deterministic code with a call to Anthropic (or any LLM) for trivial logic looks clever in a demo. At scale, it’s expensive laziness. Clear position: LLMs are for ambiguity. Not for boolean checks. What breaks? • 𝗖𝗼𝘀𝘁 – You’re paying inference pricing for something the CPU can solve in microseconds. Multiply that by traffic. • 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 – Network + model latency added to hot paths. Now your API feels slow for no reason. • 𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆 – You’ve introduced external dependency risk into deterministic logic. For founders: this quietly increases burn without improving product value. For product teams: you’re coupling core logic to a third-party API. For scaling companies: this becomes architectural debt that’s painful to unwind. Serious engineering teams separate: Deterministic logic → code Ambiguity & language reasoning → LLM layer Clear boundaries. Clear contracts. If your system needs AI to decide what str[0] is doing, your problem isn’t intelligence, it’s discipline. #AIEngineering #SystemDesign #LLMArchitecture #StartupTech #EngineeringLeadership

A recurring inefficiency I see in LLM systems: They spend most of their budget solving the same problem repeatedly. A few days back, I was reviewing a cost chart from a friend’s LLM-based system. - Nothing looked obviously broken. - No spikes. No outages. No bad prompts. But the cost kept climbing. When we dug deeper, the issue was simple: - The system had no notion of intent reuse. Consider a simple support flow: - How do I upgrade my plan - I want to switch to premium - Can I move from free to paid Different phrasing. Same underlying intent. Yet many production systems treat these as independent queries and invoke the model each time. At scale, this becomes a structural cost issue, not a model issue. The framing here matters. This is not just a caching problem. It is a combination of: - semantic deduplication - freshness management - and controlled reuse The critical shift is upstream of generation. Before calling the model, the system should be able to answer: Have we already solved this intent with sufficient confidence That introduces a different set of design concerns: - How do you represent intent reliably - What similarity threshold is safe for reuse - When does semantic proximity become operational risk - How do you degrade gracefully when confidence is ambiguous Reuse, in particular, is where most implementations break. Upgrade and cancel may sit close in embedding space. But they diverge completely in execution paths. So reuse cannot be binary. It needs a decision layer: - High confidence -> reuse - Medium confidence -> adapt with a cheaper model - Low confidence -> generate fresh Equally important is freshness. Not all responses have the same half-life. - Feature explanations can be relatively stable - Billing, pricing, and policy responses are time-sensitive Without explicit expiry and metadata, caching introduces correctness risk. A practical architecture tends to converge to a layered flow: - Normalize and classify the query - Resolve intent via embeddings - Evaluate similarity and confidence - Validate freshness constraints - Decide between reuse, adaptation, or regeneration - Persist response with metadata for future decisions The broader point: - Most teams invest heavily in prompt quality and model selection. - Fewer invest in the decision system that determines whether generation is required in the first place. - In mature systems, that decision layer is where a significant portion of cost efficiency and reliability is won. #SystemDesign #LLM #AIArchitecture #DistributedSystems #ScalableSystems #EngineeringLeadership #SoftwareArchitecture

Most teams think LLM evaluation = logging. It’s not. In production, we hit two walls. First: visibility. You can’t debug what you can’t see. So we added full traces — prompts, context, tool calls, versions. That solved visibility. Then scale broke us. At thousands of calls a day, no one reads the logs. And unread logs are useless. What actually worked: Heuristics to flag obvious failures Statistical monitoring to catch drift LLM-as-judge for scalable scoring Humans only on escalation Logging is not evaluating. If you don’t know what the model saw, you’re guessing. If you try to read everything, you’ll lose. Evaluation is deciding what matters — at scale. https://lnkd.in/eWyHiTyG

Over the past few weeks, the first four‑part series on Verification‑as‑a‑Service sparked far more discussion than we expected. Regulators wrote in about sovereignty. Scholars questioned semantic limits. Compliance teams asked about auditability. Strategy heads wanted to know what this becomes in five years — and in fifteen. The questions kept coming, especially after the final essay in the series — “The Infrastructure of Trust: From Engine to Ecosystem” — went live on Medium (link below). So we’re doing something different next. Starting this week, Sai Sumanth and I are releasing a new 4‑part series — not essays, but conversations. Direct, structured, and grounded in the questions readers actually asked. Part 1 begins with the two groups whose authority any architecture must respect: the regulators who set the rules, and the scholars who interpret them. We go deep into: how an engine respects multiple jurisdictional authorities without flattening them, how it handles evolving standards without overwriting history, how it reads contracts structurally rather than semantically, and how it absorbs scholarly feedback without collapsing legitimate ikhtilaf into a synthetic consensus. If the first series laid out the architecture, this one explores the people, the authority, and the limits that shape it. Stay tuned — Part 1 drops shortly.

With the addition of 'E is for Empirical', my CRESS Principles for Context Engineering start to form a unified theory that has fundamental implications for AI-assisted software development workflows. Input contexts need to present an accurate, up-to-date picture of observed reality whenever possible, enforced by a feedback loop that requires generated output to pass through quality gates that catch any output that doesn't satisfy our intent. A key implication of this is that contexts be short-lived - I suspect much shorter than many of you might be thinking. See comments for a link to the full blog post.