689: Observing LLMs in Production to Automatically Catch Issues

<p>Arize&apos;s Amber Roberts and Xander Song join Jon Krohn this week, sharing invaluable insights into ML Observability, drift detection, retraining strategies, and the crucial task of ensuring fairness and ethical considerations in AI development.<br/><br/>This episode is brought to you by <a href='https://posit.co/'>Posit</a>, the open-source data science company, by <a href='https://go.aws/3zWS0au'>AWS Inferentia</a>, and by <a href='https://superdatascience.com/anaconda'>Anaconda</a>, the world&apos;s most popular Python distribution. Interested in sponsoring a SuperDataScience Podcast episode? Visit <a href='https://jonkrohn.com/podcast'>JonKrohn.com/podcast</a> for sponsorship information.<br/><br/>In this episode you will learn:<br/>• What is ML Observability [05:07]<br/>• What is Drift [08:18]<br/>• The different kinds of model drift [15:31]<br/>• How frequently production models should be retrained? [25:15]<br/>• Arize&apos;s open-source product, Phoenix [30:49]<br/>• How ML Observability relates to discovering model biases [50:30]<br/>• Arize case studies [57:13]<br/>• What is a developer advocate [1:04:51]<br/><br/>Additional materials: <a href='https://www.superdatascience.com/689'>www.superdatascience.com/689</a></p>