Key Takeaway
By the end of this blueprint you will have a feature store architecture using Feast that serves consistent features to both training pipelines and production inference, with batch and streaming ingestion, point-in-time correct feature retrieval, schema versioning, and freshness monitoring.
Prerequisites
- Python 3.11+ with familiarity with Pandas and data transformation patterns
- PostgreSQL for the offline feature store and registry
- Redis for the online feature store (low-latency serving)
- Apache Kafka or a managed equivalent for streaming ingestion (optional)
- Understanding of ML training workflows and feature engineering concepts
The Training-Serving Skew Problem
Training-serving skew is the most common and hardest-to-debug cause of model quality degradation in production. It happens when the features computed during training differ from those computed during inference — different code paths, different data sources, or different timing. A feature store solves this by providing a single feature computation layer that both training and inference consume. The same transformation code runs in both contexts, eliminating the skew by construction.
Training-serving skew is not limited to traditional ML. LLM-based applications that use dynamic context (user history, recent activity, account metadata) as prompt inputs suffer from the same problem. If the context assembly code differs between your development environment and production, your prompts are effectively different despite the template being the same.
Architecture Overview
The architecture has two ingestion paths — a batch pipeline for historical data and a streaming pipeline for real-time events — that converge in a feature computation layer. Computed features are written to both an offline store optimized for bulk reads during training and an online store optimized for low-latency point lookups during inference. A feature registry tracks schemas, ownership, lineage, and freshness SLAs for every registered feature.
Feature Definition with Feast
Feast is the most widely adopted open-source feature store. Features are defined as code — Python files that declare feature views (groups of related features), their data sources, and their entity keys. This code-as-configuration approach means feature definitions live in version control alongside your model code, and changes go through the same review process as any other code change.
"""Feature definitions using Feast."""
from datetime import timedelta
from feast import (
Entity,
FeatureView,
Field,
FileSource,
PushSource,
ValueType,
)
from feast.types import Float32, Int64, String
# ---- Entities ----
user = Entity(
name="user_id",
value_type=ValueType.STRING,
description="Unique user identifier",
)
# ---- Data Sources ----
# Batch source: daily aggregated user activity
user_activity_source = FileSource(
path="data/user_activity.parquet",
timestamp_field="event_date",
created_timestamp_column="created_at",
)
# Streaming source: real-time events via push API
user_realtime_source = PushSource(
name="user_realtime_events",
batch_source=user_activity_source,
)
# ---- Feature Views ----
user_activity_features = FeatureView(
name="user_activity",
entities=[user],
ttl=timedelta(days=7),
schema=[
Field(name="sessions_7d", dtype=Int64),
Field(name="avg_session_duration_sec", dtype=Float32),
Field(name="messages_sent_7d", dtype=Int64),
Field(name="tools_used_7d", dtype=Int64),
Field(name="preferred_model", dtype=String),
Field(name="total_tokens_7d", dtype=Int64),
],
source=user_activity_source,
online=True, # Materialize to online store
tags={
"team": "platform",
"freshness_sla": "daily",
"version": "2",
},
)
user_realtime_features = FeatureView(
name="user_realtime",
entities=[user],
ttl=timedelta(hours=1),
schema=[
Field(name="current_session_messages", dtype=Int64),
Field(name="current_session_tokens", dtype=Int64),
Field(name="last_active_minutes_ago", dtype=Float32),
],
source=user_realtime_source,
online=True,
tags={
"team": "platform",
"freshness_sla": "realtime",
"version": "1",
},
)Batch and Streaming Ingestion
Batch ingestion runs on a schedule (daily or hourly) and computes aggregate features from your data warehouse. Streaming ingestion processes events in real-time via a push API, updating the online store within seconds. The two paths complement each other: batch features provide rich historical aggregations (7-day averages, lifetime totals), while streaming features provide recency signals (current session activity, time since last action). Both write to the same online store so the serving layer presents a unified feature vector.
"""Batch feature computation and materialization."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import pandas as pd
from feast import FeatureStore
store = FeatureStore(repo_path="feature_repo/")
def compute_user_activity_features(
raw_events_path: str,
output_path: str,
) -> pd.DataFrame:
"""Compute 7-day user activity aggregations.
Args:
raw_events_path: Path to raw event data (Parquet).
output_path: Where to write computed features.
Returns:
DataFrame with one row per user.
"""
events = pd.read_parquet(raw_events_path)
cutoff = datetime.now(timezone.utc) - timedelta(days=7)
recent = events[events["event_date"] >= cutoff]
features = (
recent.groupby("user_id")
.agg(
sessions_7d=("session_id", "nunique"),
avg_session_duration_sec=("duration_sec", "mean"),
messages_sent_7d=("message_count", "sum"),
tools_used_7d=("tool_invocations", "sum"),
preferred_model=("model", lambda x: x.mode().iloc[0] if len(x) > 0 else "unknown"),
total_tokens_7d=("tokens", "sum"),
)
.reset_index()
)
features["event_date"] = datetime.now(timezone.utc)
features["created_at"] = datetime.now(timezone.utc)
features.to_parquet(output_path)
return features
def materialize_features():
"""Materialize batch features to the online store."""
store.materialize(
start_date=datetime.now(timezone.utc) - timedelta(days=7),
end_date=datetime.now(timezone.utc),
)Point-in-Time Correct Retrieval
Point-in-time correctness means that when you retrieve features for training, you get the feature values as they existed at the time of each training example — not the current values. Without this, your training data contains information from the future (data leakage), and your model learns patterns that do not exist at inference time. Feast handles this automatically: when you call get_historical_features with a DataFrame of entity keys and timestamps, it joins features using the correct temporal semantics.
Feature Store Comparison
| Feature | Feast (OSS) | Tecton | Databricks Feature Store |
|---|---|---|---|
| Deployment | Self-managed | Managed SaaS | Databricks-integrated |
| Streaming ingestion | Push API | Native Spark/Flink | Delta Live Tables |
| Online serving latency | 1-5ms (Redis) | Sub-1ms | 5-10ms |
| Point-in-time joins | Built-in | Built-in | Built-in |
| Cost | Infrastructure only | Per-feature pricing | Databricks DBUs |
| Best for | Teams wanting control | Enterprise with SLA needs | Databricks-native stacks |
Feature Freshness Monitoring
Stale features are a silent quality killer. If your batch pipeline fails silently and features stop updating, the online store serves increasingly outdated values. Monitor feature freshness by tracking the timestamp of the latest materialized value for each feature view. Alert when any feature exceeds its freshness SLA (e.g., daily features older than 26 hours, real-time features older than 5 minutes).
Schema changes require coordination between the feature store, the training pipeline, and the serving application. Use schema versioning: add new fields as new versions, keep old versions active during transition, and only deprecate old versions after all consumers have migrated. Breaking schema changes (removing or renaming fields) require a migration plan.
Feature Definitions
Ingestion
Operations
Version History
1.0.0 · 2026-03-01
- • Initial publication with Feast feature store setup
- • Batch and streaming ingestion pipeline patterns
- • Point-in-time correct retrieval for training datasets
- • Feature store comparison: Feast vs Tecton vs Databricks
- • Freshness monitoring and schema versioning guidance