AI Forward - Fiddler AI Observability Platform for LLMOps

LLMOps is an iterative process that involves continuously improving LLMs. The Fiddler AI Observability platform streamlines end-to-end LLMOps by enabling data scientists and AI practitioners to:

• Robust Model Validation: Evaluate the robustness, correctness, and safety of LLMs, ensuring they are resilient against issues like prompt injection attacks.
• Advanced Monitoring of LLM Metrics: Track critical metrics such as data drift, hallucinations, PII, and toxicity, allowing quick identification of issues
• Macro and Micro-level Analysis: Visualize problematic trends at the macro-level using the 3D UMAP and perform root cause analysis on the problem areas, like prompts, responses, and user feedback, at a micro-level ‍
• Continuous Feedback Loop for Improvement: Leverage insights and analytics to iteratively refine models, aligning them with evolving data and business needs for ongoing enhancement

Fiddler AI Observability Platform for LLMOps

Speaker: Barun Halder -  Staff Software Engineer, Fiddler AI

Video transcript

Hello everyone, my name is Barun and I'm excited to guide you through Fiddler's innovative capabilities for specific for LLMOps. So, if you remember and if you attended Amal's demonstration earlier, Illustrated the utilization of Fiddler Auditor to assess LLMs in the pre-production stage.

So we use some model-based metrics model-based evaluations for us to showcase, to figure out if the LLM or the LLM pipeline that you have set is good or not. So that was the pre-production stage. I'll specifically focus on the practical aspects of monitoring, enhancing, and managing LLMs. Once they are in production, so in the production phase.

So, LLMOps, like everything else in life, is an iterative process. We move between pre-production and production phases. Constantly refining as we harness the power of LLMs as we create value. At Fiddler, we are crafting tools that simplify this and streamline this journey between the pre-production and the production phases of this LLM.

So, when it comes to production, we offer out of the box, specialized, ready to use metrics tailored for the LLMs use case. The tools that we are building right now provide you a bunch of metrics right out of the box.

Metrics which fall in the bucket of performance, data quality, correctness, privacy, safety, and then you can chart them in different kinds of dashboards, and you could do a deeper analysis using the Analyze and UMAP visualization. So, we have high-dimensional vectors representing these LLMs, prompts, responses, context in high dimensional space.

So we provide tooling for you to be able to visualize and have a high level view of what's really happening with your production data, and then we offer you tools for analysis where you can go and dive deeper and do some kind of analysis about PII loss, if there is some kind of toxicity.

BreakBot Challenge

So now, the fun part. We have the BreakBot Challenge. this is a demo app that we have built which is for the NewAge Bank chatbot, which you can access and ask it different kinds of banking questions. On the backend, it is using a RAG based setup, for you to interact with it. And you would see that there are a few options of models when you access that.

Now, what are the challenges? You have to put on your hacker's hat and do some amount of prompt injection for you to be able to get some PII data out. When I say PII data, it could be phone numbers, emails, names, SSNs, or account numbers.

The second challenge is out of all the models that we have, can you figure out which one has the propensity to give out unsafe responses? And the final challenge is, if you can find out which of the available models hallucinates the most.

So let's give everyone a minute to start interacting with it.

You’ll find that when you interact with it, the response may not always be ideal. So, the moment you do this, you’ll give a feedback that shows up in the dashboard that we have set up for the Breakpoint Challenge.

So here we can see that operational metrics are plotted in a chart that shows how many interactions are happening with the bot. Also, we have charts that keep track of interaction metrics for different kinds of bots.

If there are any prompt leakages or any toxic responses, the operational dashboard will reflect this data. You can see how people are interacting with the different models.

At some point, alerts can be set based on these interactions; when an alert is fired, it allows you to dive deeper into the data to understand what caused the alert.

We provide tools for analysis to query the production data and find out all the events where PII information was leaked.

For analyzing safety, we use a safety number where 1 represents something safe and 0 represents something unsafe to list all the prompts and responses where safety is a concern.

Now, let’s delve into the UMAP tool which helps visualize what is happening at a macro level. You can select metrics and visualize the distribution of prompts and responses to identify clusters or trends in user data.

This overview of what everything looks like in terms of operational metrics will help you quickly gauge the status of different models and their performance and thus make informed decisions about which models to deploy.

In conclusion, Fiddler AI provides comprehensive solutions tailored towards LLMOps, enabling organizations to maintain the quality and safety of their AI systems throughout production.