AI System Design: From Idea to Production - Apoorva Joshi, MongoDB

TL;DR

• 1

  Specs are the new code: In the AI coding era, the hard part is defining product requirements, system design, and evaluation criteria, not writing code itself, and even Anthropic and OpenAI leaders emphasize this.

• 2

  Start with a solution-agnostic business problem: A good business problem states who the users are, the current state, quantifies pain points, and doesn't prescribe what the system should be.

• 3

  Identify business constraints early: Regulatory requirements, data residency rules, approved vendors, and cases requiring human review all shape downstream architectural decisions.

• 4

  Design the simplest system first: The most common mistake is over-engineering before knowing what's actually failing, so start simple, evaluate, and iterate.

• 5

  Guardrails are essential for LLM systems: Unlike traditional software, LLMs are probabilistic and can produce unexpected or harmful outputs, requiring input/output guardrails and domain-specific metrics.

• 6

  Optimization is non-negotiable for production: Accuracy, cost, latency, and reliability constraints require techniques like prompt engineering, reranking, semantic caching, and structured outputs before shipping.