Event sourcing from static data using Kafka

August 6, 2025 · Baoss

Event Sourcing from Static Data in 2026: the BAOSS IT consulting solution without Distributed Schedulers

In August 2020 we published on this blog an approach to event sourcing from static data using Kafka. The problem was clear: events do not “wake up” on their own. For example, a promotion with an expiration_date does not automatically generate a deactivation event when the date arrives. The solution back then involved KTable joins and foreign keys in Kafka 2.4 (KIP-213). At BAOSS IT consulting we keep solving this problem with event-as-source-of-truth architectures.

Event sourcing from static data using Kafka, explained by BAOSS IT consulting

If you want to go further, we recommend reading AI agents: is your company exposed to the shock of disruption? how to control what your company is already deploying.

Five years later, the landscape has changed radically. At BAOSS we have accompanied 12 enterprise clients (banking, retail, telco, logistics) in migrating legacy event-driven architectures to cloud-native platforms with Kafka 3.7+, Flink SQL and Iceberg as the analytical storage layer. However, the temporal expiration problem has not gone away. In fact, it has grown more complex with event versioning, schema evolution and the arrival of autonomous AI agents that consume and produce events in real time.

The Real Problem 2026-2027: Time, Consistency and AI Agents

Today, a typical event-driven platform at our clients handles:

  • 50-200M events/day on core topics (orders, prices, inventory, payments)
  • 15-40 services consuming/producing with versioned Avro/Protobuf contracts (Schema Registry)
  • 3-5 autonomous AI agents (LangGraph, CrewAI, AutoGen) running multi-step workflows over event streams
  • Requirements of eventual consistency < 500ms and exactly-once on critical flows
  • Compliance with DORA, NIS2, GDPR with full traceability of automated decisions

The classic expiration_date case for prices/promotions still exists, but it now coexists with:

  • Dynamic SLAs computed by ML models (GPT-4o / Claude 4 via function calling) that re-evaluate priorities every minute
  • Regulatory retention policies that require selective event replay by correlation_id for up to 7 years
  • RAG + MCP agents that query knowledge graphs in Neo4j/GraphDB to enrich events with historical context
  • Distributed feature flags (LaunchDarkly, Unleash) that change consumer behavior without redeploying

Traditional distributed schedulers (Quartz, Spring Batch, Airflow, even Walmart’s Big Ben) introduce temporal coupling, single points of failure and unpredictable latency. Therefore, in 2025 the answer is not “more schedulers” —it is native stream processing with temporal semantics.

BAOSS Reference Architecture 2026: Flink SQL + Iceberg + Kafka

Our current approach to event sourcing from static data with temporal attributes combines three layers:

  • Ingestion layer: Kafka 3.7+ (KRaft mode) with topic tiered storage to S3/GCS. Producers use transactional.id for end-to-end exactly-once.
  • Temporal processing layer: Flink SQL 1.19+ with watermarks, TTL state and temporal table joins. This is where we solve expiration without schedulers.
  • Analytical / replay layer: Apache Iceberg on object storage with time travel, partition evolution and row-level deletes for GDPR.

The Pattern: Temporal Table Join in Flink SQL

Instead of KTable foreign key joins (limited to primary keys and with no time semantics), we use Flink’s temporal table joins. A temporal table is a time-versioned view: each key has multiple versions with valid_from / valid_to. In addition, the join is resolved automatically according to the watermark of the incoming event.

-- Temporal price table definition (versioned by expiration_date)
CREATE TABLE prices_temporal (
    price_id STRING,
    product_id STRING,
    amount DECIMAL(10,2),
    currency STRING,
    status STRING, -- 'ACTIVE', 'EXPIRED', 'SUPERSEDED'
    valid_from TIMESTAMP(3),
    valid_to TIMESTAMP(3),
    WATERMARK FOR valid_from AS valid_from - INTERVAL '5' SECOND,
    PRIMARY KEY (price_id, valid_from) NOT ENFORCED
) WITH (
    'connector' = 'kafka',
    'topic' = 'prices-cdc',
    'properties.bootstrap.servers' = 'kafka:9092',
    'scan.startup.mode' = 'earliest-offset',
    'format' = 'avro-confluent',
    'avro-confluent.schema-registry.url' = 'http://schema-registry:8081'
);

-- Incoming orders stream (business events)
CREATE TABLE orders_stream (
    order_id STRING,
    product_id STRING,
    quantity INT,
    order_ts TIMESTAMP(3),
    WATERMARK FOR order_ts AS order_ts - INTERVAL '10' SECOND
) WITH (
    'connector' = 'kafka',
    'topic' = 'orders',
    'properties.bootstrap.servers' = 'kafka:9092',
    'format' = 'avro-confluent'
);

-- Temporal join: each order gets the price IN FORCE at its order_ts
SELECT
    o.order_id,
    o.product_id,
    o.quantity,
    p.amount AS unit_price,
    p.currency,
    (o.quantity * p.amount) AS total_amount,
    o.order_ts
FROM orders_stream o
LEFT JOIN prices_temporal FOR SYSTEM_TIME AS OF o.order_ts p
    ON o.product_id = p.product_id
WHERE p.status = 'ACTIVE';

Result: zero schedulers. Expiration is intrinsic to the data model. When the watermark passes valid_to, the version expires automatically from the state backend (RocksDB incremental checkpoints every 30s). Consequently, Flink emits a changelog to the output topic —downstream consumers receive the “deactivation” as just another event.

Real BAOSS Case (Anonymized): Multinational Retail — 47M Events/Day

Context: Retail client with 3,200 stores, e-commerce and marketplace. Legacy platform with overnight Spring Batch jobs to expire promotions (4h window, frequent failures, manual replay). Migration to event-driven in 2026-2027.

Pre-Migration Metrics (Q4 2024)

  • Promotion expiration latency: 2-6 hours (overnight batch)
  • Monthly incidents from incorrect prices: 12-18 (€280K-420K/month impact)
  • Historical replay time (audit): 4-8 hours (DB dump + restore)
  • Batch infrastructure cost (EC2 r6g.4xlarge x 12): €18,500/month

BAOSS Implementation (Q1-Q2 2025)

  • CDC migration (Debezium 2.5+) → Kafka → Flink SQL (Kubernetes, K8s Operator)
  • Temporal table modeling for 14 domains (prices, promos, stock, SLA, loyalty)
  • AI agents integration: 3 LangGraph agents consuming enriched events for dynamic pricing and stock allocation
  • Iceberg layer (Tabular/Trino) for audit time travel and the ML feature store
  • Observability: OpenTelemetry + Tempo + Mimir + Grafana (end-to-end P99 latency < 200ms)

Results Q3 2025 (6 months in production)

  • Expiration latency: < 500ms (watermark-driven, deterministic)
  • Incorrect price incidents: 0 in 6 months (vs 12-18/month)
  • Selective audit replay: < 3 min (Iceberg time travel + partition pruning)
  • Streaming infrastructure cost (EKS + MSK + S3): €11,200/month (-39% vs batch)
  • ROI 3.4x in 6 months (direct savings + avoided sanctions + recovered revenue)
  • New domain deployments: 2 days vs 3 weeks (Flink SQL templates + GitOps CI/CD)

The client’s internal team (12 engineers) now operates the platform autonomously. BAOSS contributed architecture, coaching and knowledge transfer —not “body shopping”.

Second Case: Banking — DORA/NIS2 Compliance with Certified Event Replay

A European bank (assets >€200B) needed to prove to the regulator full traceability of automated credit decisions (ML models + business rules). It also required point-in-time replay by application_id for up to 7 years.

  • Challenge: Events across 38 topics, evolved Avro schemas (150+ versions), PII in payloads.
  • Solution: Flink SQL temporal joins to reconstruct state at any event_ts + Iceberg row-level deletes for the right to be forgotten (GDPR Art. 17) without breaking time travel.
  • AI agents: 2 CrewAI agents (automated audit + regulatory narrative generation) consume curated events via MCP (Model Context Protocol) from Iceberg.
  • Key metric: DORA audit Q2 2025 completed in 4 days vs the estimated 6 weeks. Zero critical findings.

Tools 2026-2027 in Our Stack

CategoryToolRole in the BAOSS architecture
Ingestion & streamingKafka 3.7+ (KRaft), Amazon MSKCore topics with transactional.id for exactly-once; tiered storage to S3/GCS
CDCDebezium 2.5+Capturing changes from legacy databases into Kafka without batch windows
Data contractsSchema Registry (Avro / Protobuf)Versioned schema evolution for 15-40 consumer services
Temporal processingFlink SQL 1.19+Temporal table joins, watermarks and TTL state (RocksDB) — expiration without schedulers
Analytical storageApache Iceberg + TrinoTime travel, partition evolution and row-level deletes for replay and GDPR
OrchestrationKubernetes + GitOps (CI/CD)Deploying Flink jobs in 2 days vs 3 weeks
AI agentsLangGraph, CrewAI, AutoGen; GPT-4o / Claude 4Multi-step workflows over streams; function calling and RAG + MCP
Context / graphsNeo4j / GraphDBKnowledge graphs to enrich events with history
Feature flagsLaunchDarkly, UnleashChanging consumer behavior without redeploying
ObservabilityOpenTelemetry + Tempo + Mimir + GrafanaEnd-to-end traceability, P99 latency < 200 ms

Conclusion: Time Is Data, not a Scheduler

Five years after our first post on event sourcing with static data, the lesson still holds and has grown louder: expiration, consistency and regulatory compliance must not be delegated to overnight batch jobs. By modeling time as part of the data model itself —temporal tables in Flink, time travel in Iceberg— we eliminate temporal coupling, single points of failure and unpredictable latency. We also open the door for autonomous AI agents to operate on deterministic, auditable events.

Going from 12-18 monthly incidents from incorrect prices to zero in 6 months, with a 3.4x ROI and audit replay in minutes instead of hours, is not magic. On the contrary, it is native stream architecture with temporal semantics. Moreover, the same applies to banking (DORA/NIS2), telco or logistics.

At BAOSS we do not deliver “body shopping”: we design the architecture, transfer the knowledge and leave your team operating the platform autonomously. Want to assess your current event-driven platform and find out where schedulers are costing you money and incidents? If so, let’s talk — we prepare a 2-week assessment with a concrete migration plan.

Prefer to talk to us directly? Contact the BAOSS team and tell us about your challenge.