Production ML Pipelines: From Notebook to Serving

In my first attempt at production machine learning, the model worked beautifully in the notebook. It had an AUC of 0.81 on a holdout set, feature importance plots that made sense, and a clean confusion matrix. Two weeks after I deployed it, the business side asked why we were suddenly approving almost every repeat applicant. I checked the logs, traced the predictions, and eventually found a single feature, days_since_last_application, that had been computed with max(application_date) from the training set as the anchor. In serving, that anchor was the current timestamp. The feature shifted its distribution by months, and the model happily made decisions based on something it had never seen during training.

That incident is the whole argument for production ML pipelines in one story. A notebook is a great place to explore, but it is a terrible place to leave the model. While building the Aura Finance pipeline, I learned that the valuable artifacts are not weights and checkpoints, but the contracts, feature definitions, and validation gates around them. This post describes how we took the repo from a collection of Final_v2_real.ipynb files to a reproducible pipeline that serves models in production, and what I would do differently next time.

The Ownership Revolution

The first step was changing what we considered the "model" to be. For us, Aura Finance's model is not model.onnx or a pickled gradient boosting object. The model is the whole path from raw events to a served probability, including the feature transformations, the validation splits, the label definition, and the monitoring thresholds. If any of those pieces is ambiguous, the model is not production ready.

We encoded that path as a versioned pipeline module. The notebook was removed from the repo. In its place we had a pipeline.py that describes a run as configuration plus an ordered set of step functions.

from dataclasses import dataclass
from datetime import date
from pathlib import Path

@dataclass(frozen=True)
class PipelineConfig:
    raw_table: str = "warehouse.raw_applications"
    feature_version: int = 2
    model_version: str = "credit_risk_v2"
    train_start: date = date(2023, 1, 1)
    train_end: date = date(2024, 6, 30)
    validation_end: date = date(2024, 9, 30)

def run(cfg: PipelineConfig) -> Path:
    run_dir = Path("runs") / f"{cfg.model_version}-{cfg.feature_version}"
    raw = load_raw(cfg.raw_table, cfg.train_start, cfg.validation_end)
    features = build_features(raw, cfg.feature_version)
    train_df, valid_df = temporal_split(features, cfg.train_end)
    metrics = train_and_evaluate(train_df, valid_df)
    write_metrics(run_dir, metrics)
    promote_if_improved(run_dir, min_auc_lift=0.005)
    return run_dir

Every function reads from a clear source and writes to a path. There is no kernel state, no "which cell did I run last?", and no df lingering in memory from an earlier experiment. The pipeline object is the unit we test, deploy, and roll back.

Why Notebooks Fail at the Boundary

The notebook is not the enemy of exploration; it is the enemy of the boundary between training and serving. A notebook cell that computes a feature can work because a variable from a previous cell happens to exist. But the serving code has no notebook, no hidden state, and no chance to run cells in the order a human did on a good day.

Our hard-won lesson: do not try to parse notebooks. I considered tools that execute notebooks top-to-bottom and export functions, but every one of them inherits the notebook's global state model. The elegant fix is structural, not syntactic: move the logic into modules with explicit function signatures and force all data flow through return values. If it cannot run in a clean Python process with only the arguments you pass, it cannot run in production.

| Property | Notebook code | Pipeline module | |---|---|---| | State | implicit kernel globals | explicit function arguments | | Re-run | depends on cell order | deterministic run id | | Testing | nearly impossible | unit and contract tests | | Serving path | reimplemented from memory | same code reused | | Artifact | saved cells | registry + bytes |

This table is not academic. The repeated bugs we found in production came from a developer writing a feature in the notebook, verifying it interactively, and then writing a separate implementation for the API server. Two implementations, one data drift, and nobody knew who was wrong until the approvals went sideways.

The Pipeline as a Contract

Once we had modules, we treated the pipeline as a contract between raw data and reproducible artifacts. A run produces a directory with a versioned model, feature metadata, evaluation metrics, and the exact schema of every table it read or wrote. That run directory is the unit of reproducibility.

The contract has three layers:

1. Input contract: raw tables must satisfy row counts and null-rate checks before the pipeline runs.
2. Transform contract: features must have expected types, ranges, and timestamp monotonicity.
3. Output contract: the model artifact must be readable by the serving engine and produce consistent predictions on a golden batch.

We encoded these checks as SQL and Python assertions, and we ran them in CI on every pull request. For example, a single validation query catches bad source data before it burns a training run.

-- Run in CI before training. Any row that violates this fails the build.
SELECT
  COUNT(*) FILTER (WHERE loan_amount <= 0) AS negative_amounts,
  COUNT(*) FILTER (WHERE application_date > ingestion_ts) AS future_dates,
  COUNT(*) FILTER (WHERE customer_id IS NULL) AS missing_customer_ids
FROM warehouse.raw_applications;

This is not a subtle model tuning knob; it is a gate. If the warehouse starts receiving malformed rows, the pipeline stops instead of silently learning from garbage.

Feature Store: Copying Data Is a Tax

The hardest production problem was feature consistency. We kept writing SQL for training features and different Python dictionaries for serving features. The feature store was the answer, but we chose a deliberately thin one.

Offline features

The offline side is a set of versioned feature definitions that operate on a DataFrame and return new columns. Each feature version is immutable; when the feature logic changes, it gets a new version, not a silent edit. That lets backtests use old versions and lets the serving path load a specific version.

Online materialization

The online side is a Redis hash per entity. After a training run, the same feature definitions are applied to recent raw data and materialized into Redis so the serving API can read them with a single HGETALL.

from __future__ import annotations

import redis
import pandas as pd
from dataclasses import dataclass

@dataclass
class FeatureSpec:
    version: int
    func: callable

class FeatureStore:
    def __init__(self, online: redis.Redis):
        self._online = online

    def compute_offline(self, specs: list[FeatureSpec], df: pd.DataFrame) -> pd.DataFrame:
        out = df.copy()
        for spec in specs:
            col = f"f{spec.version}_{spec.func.__name__}"
            out[col] = spec.func(out)
        return out

    def materialize(self, entity_id_col: str, df: pd.DataFrame) -> None:
        with self._online.pipeline() as pipe:
            for row in df.to_dict("records"):
                key = f"feat:{row[entity_id_col]}"
                payload = {k: row[k] for k in row if k != entity_id_col}
                pipe.hset(key, mapping=payload)
            pipe.execute()

    def get(self, entity_id: str) -> dict:
        return self._online.hgetall(f"feat:{entity_id}")

The important choice is that the feature functions are not duplicated. There is no training SQL and serving SQL. There is one definition, one version, and one function that is used by both the offline computation and the online materialization job. When someone wants to change a feature, they create a new version and run a backfill. We avoid the tax of copying feature logic into a second language.

Training, Validation, and the Leakage Audit

For a lending model, the target is whether a loan defaults within 30 days. That means labels are not available at application time. They become available 30 days later. If our validation set contains applications from the last 29 days, those labels are not yet observed, and any clever imputation is leakage by another name.

We made this explicit in the temporal split function.

from datetime import timedelta
import pandas as pd

def temporal_split(
    df: pd.DataFrame,
    feature_cutoff: pd.Timestamp,
    label_cutoff: pd.Timestamp,
    horizon_days: int = 30,
):
    """Split on two clocks: when features exist and when labels exist.

    A row's label is knowable only after application_date + horizon_days.
    Training rows must be fully observed before feature_cutoff and have
    labels available by label_cutoff.
    """
    df = df.copy()
    label_available = df["application_date"] + timedelta(days=horizon_days)

    train = df[
        (df["application_date"] <= feature_cutoff - timedelta(days=horizon_days))
        & (label_available <= label_cutoff)
    ]
    valid = df[
        (df["application_date"] > feature_cutoff - timedelta(days=horizon_days))
        & (df["application_date"] <= feature_cutoff)
        & (label_available <= label_cutoff)
    ]
    return train, valid

We also ran a leakage audit on all candidate features. A feature column is allowed to contain information about the customer and the current application, but not about the future. Our audit script searched for features with a correlation to the target suspiciously higher than their simulation on shuffled labels. It found exactly one feature that had to be removed because it was derived from the repayment behavior of the same loan, a classic label leakage disguised as a behavioral score.

The Model Registry and the Binary We Actually Serve

It is tempting to skip a model registry when you have a small team. We almost did. Then we deployed a model, and three days later someone asked "which model is the API actually running?" Nobody could answer with certainty.

We built a minimal registry on PostgreSQL and S3. The registry stores the model name, an immutable version, the pipeline run id, the feature version, the evaluation metrics, and a byte-identical artifact path. Promoting a model is not uploading a file; it is inserting a row that flips the "active" flag. Every prediction response includes the model version in the header, so we can trace a decision back to its exact training run.

A model registry matters because the artifact is not the model. The artifact plus all the feature transformations that feed it is the model. If the registry records only the artifact, it is a file server with extra steps.

Serving: Latency, Batching, and the Cold Start Tax

Our serving API is a stateless FastAPI service that loads the model artifact from S3 at startup, reads features from Redis, and returns a probability. It sounds simple, but the first version was slow because each request computed features inside the API process from the raw warehouse. A single feature cache miss turned a 40 ms request into a 900 ms request.

The solution was to separate the feature path from the model path. The API never touches the warehouse. It reads precomputed features from Redis. If a feature is missing, the request returns a neutral score and marks it for analysis; it does not attempt to compute features on the fly.

We considered three serving patterns before settling on the synchronous API.

| Mode | Latency | Cost | Best use | |---|---|---|---| | Synchronous REST | p99 under 250 ms | medium | interactive decisions | | Batch scoring | minutes to hours | low | monthly portfolio review | | Streaming | sub-second with backpressure | high | real-time risk alerts |

For Aura Finance, the synchronous REST path was the only one that met the partner bank's response time. We added a small batch of five requests per prediction loop to the API, and p99 latency went from 1.4 seconds to 230 ms. Batching is not always needed, but when each model call is cheap and the HTTP overhead dominates, a small batch is the cheapest performance win you will ever get.

Monitoring: Drift Is Not a Hypothesis

After deployment, the first thing we did was set up dashboards for feature distributions. That was the wrong frame. Drift is not a hypothesis to be proven with a pretty chart; it is an operational risk to be detected by a gate. We monitor three signals:

1. Input drift: KS or PSI on each feature compared to the training set.
2. Prediction drift: daily average prediction vs. expected range.
3. Outcome drift: actual default rate vs. predicted default rate, after a 30-day lag.

If any signal crosses its threshold, the service does not fail closed. It emits an alert, logs the model version, and gives the risk team a decision: continue, shadow-deploy a newer model, or roll back to the previous active version. The monitoring system is part of the pipeline, not a bolt-on afterthought.

The Orchestrator Trap

We started with an Airflow DAG that triggered a training job once a week. It worked, but it invited the orchestration trap: putting business logic inside the orchestrator. Eventually we had Python code in Airflow operators that was hard to run locally and hard to test. That is the opposite of the pipeline module we had built.

We moved the orchestration to a thin scheduler that simply launches a containerized pipeline, passes a PipelineConfig, and records the exit status. The pipeline code is not aware of the orchestrator. The same command that runs on our laptop runs in CI and runs in the production scheduler. The only difference is the environment variables.

The lesson is that a scheduler should schedule, not think. If a step needs retries, backfills, or conditional logic, put that logic in the pipeline module where it can be tested. The orchestrator should be a clean entrypoint, not a programming language.

What We Would Do Differently

If I were building Aura Finance again, these are the decisions I would make on day one.

• I would version features before the first model. Retroactive versioning is painful because every historical run becomes ambiguous.
• I would write the serving contract first. A JSON schema of the request and response shape, plus a golden sample, prevents a whole class of integration bugs.
• I would add data quality gates at the source, not at the feature store. Fixing bad data after it enters a feature is more expensive than rejecting it at ingestion.
• I would avoid the word "dashboard" for the first month. A single timely alert is more useful than a 20-tile dashboard nobody watches.
• I would keep the model as a statistical artifact, not a rule engine. When business people asked us to hard-code a policy, we had to resist; a policy encoded in a model becomes unobservable.

Conclusion

The path from notebook to serving is not about adopting a specific tool. It is about accepting that production machine learning is an engineering discipline with contracts, tests, and operational ownership. The notebook was great for discovery, but it had no state boundaries, no versioning, and no explicit contract between the data science world and the serving world. The Aura Finance pipeline replaced that ambiguity with a small set of deliberate boundaries.

What surprised me most was how little of the work was about model algorithms. The hard parts were feature consistency, temporal validation, and the discipline of knowing exactly which version of the code produced a given prediction. Every piece of value we added came from making those boundaries explicit.

If you are in the middle of a notebook-to-production migration, start with the smallest boundary you can defend: move your feature logic into a module, version it, and call it from both training and serving. That one change will buy more reliability than any model tuning.

This is exactly the journey we went through, and the code is open source. View the project on GitHub to see the pipeline, feature definitions, and serving code in one place.