Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog

TL;DR

• 1

  PostHog wants observability to end at a PR, not a dashboard: Joshua Snyder says the current flow from signal to dashboard to issue to PR is painfully slow, and the goal is to turn product events directly into fixes that are ready in GitHub.

• 2

  Naive embeddings failed because they grouped by format instead of meaning: Off-the-shelf embeddings clustered all errors together and all Slack messages together, so PostHog switched to generating LLM queries from each signal and matching those queries instead.

• 3

  The pipeline starts with safety filtering because some signals are attacker-controlled: Public-facing inputs like website errors can be poisoned, so an LLM classifier drops malicious or suspicious signals before anything else happens.

• 4

  Research agents get better when grounded in company context: Their Claude Agent SDK setup runs in Modal sandboxes with access to an MCP server, codebase context, and external tools like Linear and Notion, which improved problem summaries and reviewer selection via Git blame.

• 5

  Actionability is the gating function that prevents noisy PRs: Specific signals like error-tracking issues are often fixable, while vaguer inputs from Slack or session replay usually need more evidence or human input before an agent should write code.

• 6

  'Tokens are free' was a useful mindset during experimentation: Snyder says they initially worried too much about cost, but running agents often exposed recurring patterns that could later be compressed into cheaper one-shot LLM calls or trained models.