AI Application Observability & LLMOps

AI governance skill demand is up 150% year-over-year, yet most AI applications ship with zero observability. There's a documented 37% lab-vs-production performance gap and 50x cost variation for similar accuracy in agent deployments — problems that only surface with proper instrumentation. This full-day workshop teaches participants to instrument, monitor, evaluate, and govern LLM applications using the converging observability stack: LangSmith for application tracing and evaluation (now with @decorator-style auto-trace, ~5 min setup), W&B Weave for full ML-lifecycle coverage, and RAGAS for automated RAG quality metrics. Participants build a complete observability pipeline from auto-tracing through custom evaluation suites to automated quality gates.

For developers and ML engineers building LLM applications. Comfort with Python and basic LLM API experience required. Pairs naturally as a follow-on to the AI Agents, Multi-Agent, or Production RAG workshops. Covers: tracing architectures, LangSmith setup and analysis, W&B Weave integration, custom evaluation metrics, evaluation-driven development, RAGAS for RAG quality, cost tracking, quality gates for CI/CD, and red-teaming with promptfoo/garak.

About Brent Laster

Hi, I'm Brent Laster - a global trainer and book author, experienced corporate technology developer and leader, and founder and president of Tech Skills Transformations LLC. I've been working with and presenting at NFJS events for many years now and it is always exciting and interesting.

Through my decades in programming and management,I've always tried to make time to learn and develop both technical and leadership skills and share them with others Regardless of the topic or technology, my belief is that there is no substitute for the excitement and sense of potential that come from providing others with the knowledge they need to help them accomplish their goals.

In my spare time, I hang out with my wife Anne-Marie, 4 children and 2 small dogs in Cary, North Carolina where I design and conduct trainings and write books. You can find me on LinkedIn (linkedin.com/in/brentlaster), Twitter (@brentclaster) or through my company's website at www.getskillsnow.com.

More About Brent »