Make your LLM app a Domain Expert: How to Build an Expert System — Christopher Lovejoy, Anterior

Abstract

Vertical AI is a multi-trillion-dollar opportunity. But you can’t build a domain-expert application simply by grabbing the latest LLMs off-the-shelf: you need a system for codifying latent insights from domain experts and using that to drive development of your application.

In this talk, we’ll describe the system we’ve built at Anterior which has enabled us to achieve SOTA clinical reasoning and serve health insurance providers covering 50 million American lives. We’ll share:

• how and why to encode domain-specific failure modes as an ontology
• a practical system for converting domain expertise into quantifiable eval metrics
• how we structure work and collaboration between our clinicians, engineer and PMs
• our eval-driven AI iteration process and how this can be adapted to any industry