The Missing Link in Generative AI: Model Monitoring | Fiddler AI Blog

Generative AI based models and applications are being rapidly adopted across industries to augment human capacity on a wide range of tasks. Looking at the furious pace of the Large Language Model (LLM) rollout, it’s evident companies are feeling the pressure to adopt generative AI or risk getting disrupted and left behind. However, organizations face hurdles in deploying generative AI at scale given the nascent state of available AI tooling.

Timeline of LLM launches
In this blog, we’ll explore the key risks of today’s generative AI stack, focusing on the need for model monitoring, explainability, and bias detection when building and deploying generative AI applications. We then discuss why model monitoring is the missing link for completing the generative AI tech stack, and peek into the future of generative AI.

The Generative AI stack

In the emerging generative AI stack, companies are building applications by developing their own models, invoking third party generative AI via APIs, or leveraging open source models that have been fine-tuned to their needs. In all these cases, the three major cloud platforms typically power the underlying AI capabilities. Andreessen Horowitz recently noted that the emerging generative AI tech stack includes the following:

Generative AI-powered applications (e.g., Jasper and Copilot)
1. Two paradigms of foundation models
2. Closed-source proprietary models (e.g., GPT-4)
Open-source models (e.g., Stable Diffusion)
Model hubs to share and host foundation models

Preliminary generative AI tech stack
There are however several risks and concerns with generative AI — inaccuracies, costs, lack of interpretability, bias and discrimination, privacy implications, lack of model robustness, fake and misleading content, and copyright implications, to name just a few. Questions therefore remain on how to safely deploy this technology at scale — and that’s where considerations around model monitoring, explainability, bias detection, and privacy protection are of paramount importance; organizations must ensure that these models are continuously monitored, the usage of the generative AI apps is tracked, and users understand the reasoning behind these models.

Monitoring - The Key to Enterprise Adoption of Generative AI

Model monitoring involves the ongoing evaluation of the performance of a generative AI model. This includes tracking model performance over time, identifying and diagnosing issues, and making necessary adjustments to improve performance. For example, when generative AI models are leveraged for applications such as helping financial advisors search wealth management content or detecting fraudulent accounts on community platforms, it’s crucial to monitor the performance of the underlying models and detect issues in a timely fashion, considering the significant financial stakes involved. Likewise, when we deploy generative AI-powered applications with huge social impact, such as tutoring students and serving as a visual assistant for people who are blind or have low vision, we need to monitor the models to make sure that they remain reliable and trustworthy over time.

Correctness

Generative AI models are plagued by the accuracy of their content, which impacts all modalities of data. LLMs came under full public scrutiny as Google and Microsoft geared up to launch their AI-assisted experiences. Google Bard was incorrect in a widely viewed ad, while Microsoft Bing made a lot of basic errors. Midjourney, Stable Diffusion, and DALL-E 2 all have flawed generation of human fingers and teeth. Monitoring how accurate these outputs are from the end user helps to keep track of predictions that can be used to fine-tune the model with additional data or switch to a different model.

Performance

Model performance degrades over time, in a process known as model drift, which results in models losing their predictive power. In contrast to predictive AI models, measuring the performance of generative AI models is harder since the notion of “correct” response(s) is often ill-defined. Even in cases where model performance can be measured, it may not be possible to do so in (near) real-time. Instead, we can monitor changes in the distribution of inputs over time and treat such changes as indicators of performance degradation since the model may not perform as expected under such distribution shifts.

Cost

OpenAI, Cohere, and Anthropic have enabled easy access to their generative AI models via APIs. However, costs can add up quickly. For example, 750K words of generated text costs $30 on OpenAI’s GPT-4. Not only do teams need to stay on top of these expenses, but also assess which AI-as-an-API service provider is giving them the better bang for the buck. Tracking costs and performance metrics gives a better grasp on ROI as well as cost savings.

Prompt Monitoring

Prompts are the most common way end users interact with generative AI models. Users typically cycle through multiple iterations of prompts to refine the output to their needs before reaching the final prompt. This iterative process is a hurdle and has even spawned a new growing field of Prompt Engineering. In addition to prompts having insufficient information to generate the desired output, some prompts might give subpar results. These instances need to be captured from user feedback and collated to understand when the model quality is frustrating users so this can be used to fine-tune or change the model.

Latency

Generative AI models, especially LLMs like GPT-3 with 175B parameters, can require intense compute to run inferences. For example, Stable Diffusion inference benchmarking shows a latency of well over 5 secs for 512 X 512 resolution images, even on state-of-the-art GPUs. Once we take network delays also into account, roundtrip latencies for each API call can grow quickly. As customers typically expect to engage with generative AI applications in real-time, anything over a few seconds can hurt user experience. Latency tracking can be an early warning system to avoid model availability issues from impacting business metrics.

Safety Monitoring

Self-supervised training on a large corpora of information leads to the model inadvertently learning unsafe content and then sharing it with users. OpenAI, for one, has dedicated resources to put in safety guardrails for its models and has even shared a System Card that outlines all the safety challenges that were explored. Therefore, models might generate content that might not be safe, whether prompted or unprompted. Inaccuracies can have serious consequences, especially for critical use cases that could lead to potential harm, such as incorrect or misleading medical information and encouraging self-harm. Safety must be closely monitored based on user feedback to the model’s objectionable outputs or on the user's objectionable inputs, so that these models can be replaced or additional constraints can be imposed, if necessary.

A/B Testing

As new versions of generative AI models are released, application teams should evaluate the effectiveness of them before transitioning their business completely to the new version. Performance, tone, prompt engineering, and quality may vary between versions, so customers should run A/B tests and evaluate shadow traffic to ensure they’re using the correct version. Tracking metrics across different versions gives teams information and context to make this decision.

Explainable AI Meets Generative AI

Model bias and output transparency are lingering concerns for all ML models and are especially exacerbated with large data and complex generative AI models. After the initial furor about a lack of information sources for the answers being provided by LLMs, Bing’s recent update to its chat language model often cites its sources to be more transparent with users.

Explainability, the degree to which a human can understand the decision-making process of an ML model, is particularly important in applications where the model's output has significant consequences, such as medical diagnoses or lending decisions. However, explainable AI for these complicated model architectures is still a topic of active research.

Generative AI models can also incorporate biases from the large corpora of data they’re trained on. As these datasets are often heavily skewed towards a small number of ethnicities, cultures, demographic groups, and languages, the resulting generative AI model could be biased and produce inaccurate results for other cultures. Such biases can show up in blatant or subtle ways. Therefore, it’s crucial to identify and mitigate any biases that may be present in a generative AI model before deploying it.

Completing the Generative AI Stack

The generative AI stack, like the MLOps stack, therefore needs to have model monitoring to monitor, understand, and safeguard deployment of these models. Model monitoring connects to AI Applications, Model Hubs, or hosted models to continuously monitor their inputs and outputs to gain insights from metrics and detect potential biases in the data being fed to these models.

Future of Generative AI

Generative AI is still in its early stages. If the rapid advances over the past two years are any indicator, this year is shaping up to be even bigger for generative AI, as it goes multi-modal. Accelerated adoption of generative AI can, however, only happen with the maturation of tooling. The generative AI Ops or LLMOps workflow needs to advance in training, tuning, deploying, monitoring, and explaining so that fine-tuning, deployment, and inference challenges are addressed. With the right tools in place, 2023 will kick off the industrialization of generative AI and set the pace for its future adoption.

The Missing Link in Generative AI

Table of Contents