preventing model decay

Preventing Model Decay: Tecton + Fiddler for ML Drift Detection

Tecton and Fiddler Integration Guide

Sergio Ferragut
Principal Developer Advocate
Karen He
Principal Product Marketing Manager
Danny Brock
VP of Customer Success

Why Feature Drift Detection is Important for ML Performance

Your models are only as good as the data you’re feeding them. Your data can change over time, shifting away from your training data before you even know it, or it could even be broken upstream of your model. So, how do you make sure that the online feature values used for inference are still aligned with the features used to train your models?

When feature distributions change significantly compared to your training baseline, three critical problems emerge:

Model Performance Stability: Models are typically trained on historical data. When the properties of the input features change significantly, the model might not generalize well to new data, leading to performance degradation.
Early Warning System: Detecting drift in features can act as an early warning mechanism. Before noticeable deterioration in predictive performance occurs, teams can be alerted to the changes in data distributions and take proactive measures.
Data Integrity and Reliability: Proactive monitoring for feature drift helps ensure that the incoming data remains consistent with the data used in training. This is critical in regulated industries or high-stakes environments where decision-making relies on accurate predictions.

The Solution for Implementing Feature Drift Detection

We've built a powerful solution that combines two best-in-class platforms into a comprehensive ML monitoring solution:

Tecton

A feature platform that uses consistent feature definitions for offline training and online serving, which removes the initial risk of training/serving misalignment. It manages the data pipelines and continuously keeps features fresh and ready to serve for inference.

Fiddler

An AI Observability and Security platform for monitoring, analyzing, and improving ML model performance in production. The platform monitors 30 out-of-the-box ML metrics and custom metrics. Based on the ML use case, it tracks drift — both in model output and in the features supplied as inputs.

Together, Tecton’s feature platform and Fiddler’s AI Observability and Security capabilities help enterprises create training datasets, serve features online for inference, and send those same feature values and predictions to Fiddler for logging and monitoring.

Tecton + Fiddler Integration Data Flow

When training data is used to build a model, it can be uploaded to Fiddler to create the pre-production baseline that establishes expected feature and prediction distributions. When a model makes predictions using features from Tecton, the event identifier, the resulting prediction, and input features are submitted to Fiddler as an event or batch upload.

Setting Up a Fraud Detection Feature Service in Tecton

For model training, we start with some labeled event data. In this fraud detection example, the label is is_fraud:

Tecton uses feature views to transform raw streaming, batch, and real-time data into features for training and serving models online.

@stream_feature_view(
    source=transactions_stream,
    entities=[user],
    mode="pandas",
    timestamp_field="timestamp",
   features = [\
       Aggregate( name="sum_amount_10min", function="sum",\
                  input_column=Field("amount", Float64),\
           time_window=timedelta(minutes=10)),\
       Aggregate( name="sum_amount_last_24h",function="sum",\
                  input_column=Field("amount", Float64),\
                  time_window=timedelta(hours=24)),\
       Aggregate( name = "amount_mean_30d",function="mean",\
                  input_column=Field("amount", Float64),\
    time_window=timedelta(days=30)),\
       Aggregate( name = "amount_count_30d",function="count",\
    input_column=Field("amount", Float64),\
    time_window=timedelta(days=30)),\
       Aggregate( name = "amount_stddev_30d",function="stddev_samp",\
                  input_column=Field("amount", Float64),\
                  time_window=timedelta(days=30)),\
   ],
)
def user_txn_recent_activity(transactions_stream):
    return transactions_stream[["user_id", "timestamp", "amount"]]

A Realtime Feature

We’ll also need a way to identify outliers when a new transaction comes in, so we’ll use a real-time feature view to calculate a z-score:

request_ds = RequestSource(
name = "request_ds",
schema = [Field("amount", Float64)]
)

request_time_features = RealtimeFeatureView(
   name="request_time_features",
   sources=[request_ds, user_txn_recent_activity],
   features = [\
       Calculation(\
           name="amount_zscore_30d",\
           expr="""
( request_ds.amount -\
user_txn_recent_activity.amount_mean_30d\
)\
/ user_txn_recent_activity.amount_stddev_30d\
""",
       ),\
   ],
)

Finally, we’ll roll both of these feature views into a single feature service:

from tecton import FeatureService

fraud_detection_feature_service = FeatureService(
    name="fraud_detection_feature_service",
features = [ user_txn_recent_activity, request_time_features]
)

tecton apply

The `tecton apply` command publishes the data sources, feature views, and feature services defined in the Python modules into the platform:

> tecton apply

Using workspace "fraud_detection" on cluster https://fintech.tecton.ai
✅ Imported 1 Python module from the feature repository
⚠️  Running Tests: No tests found.
✅ Collecting local feature declarations
✅ Performing server-side feature validation: Initializing.

Pre-Production Training

Data scientists use the feature service method get_features_for_events to retrieve time-consistent training data from a notebook.

training_data = fraud_detection_feature_service.get_features_for_events(training_events)

Registering and Baselining Our Model in Fiddler

Next, we should register our model with Fiddler and create a baseline for the model training data and outputs:

project = fdl.Project.get_or_create( name="tecton_integration" )

model_spec = fdl.ModelSpec(
   inputs = input_columns,              # list of model input features
   outputs = ['predicted_fraud'],       # model prediction
   targets = ['fraud_outcome'],         # truth label
   metadata = ['user_id', 'timestamp'],
)

fdl_model = fdl.Model.from_data(
   name = "transaction_fraud_monitoring",
   version = "v1.0",
   project_id = project.id,
   source = training_result_data.sample(100),
   spec = model_spec,
   task = fdl.ModelTask.BINARY_CLASSIFICATION,
   task_params = fdl.ModelTaskParams(target_class_order=['Not Fraud', 'Fraud']),
   event_id_col = 'transaction_id',
   event_ts_col = 'timestamp',
)

fdl_model.create()

Additional example code ...

Payoff: Proactive ML Monitoring in Action

We did all this work so that we can leverage Fiddler to give us feedback on model performance and early alerts on our feature data pipeline. This feedback can then be used to adjust feature engineering in Tecton and/or retrain the model.