Event Sourcing from Static Data Using Kafka

August 6, 2025 · Baoss

Event Sourcing with Static Data in 2026: the BAOSS IT consulting solution without Distributed Schedulers

In August 2020 we published on this blog an approach to event sourcing from static data using Kafka. The problem was clear: events do not “wake up” on their own. For example, a promotion with an expiration_date does not automatically generate a deactivation event when the date arrives. The solution back then involved KTable joins and foreign keys in Kafka 2.4 (KIP-213). At BAOSS IT consulting we are still solving this problem with event-as-source-of-truth architectures.

Event sourcing from static data using Kafka, explained by BAOSS IT consulting

If you want to go further, we recommend reading AI agents: is your company exposed to the disruption shock? how to control what your company is already deploying.

Five years later, the landscape has changed radically. At BAOSS we have accompanied 12 enterprise clients (banking, retail, telco, logistics) in migrating legacy event-driven architectures to cloud-native platforms with Kafka 3.7+, Flink SQL and Iceberg as the analytical storage layer. However, the temporal expiration problem has not disappeared. In fact, it has become more complex with event versioning, schema evolution and the arrival of autonomous AI agents that consume and produce events in real time.

The Real 2026-2027 Problem: Time, Consistency and AI Agents

Today, a typical event-driven platform at our clients handles:

  • 50-200M events/day on core topics (orders, prices, inventory, payments)
  • 15-40 services consuming/producing with versioned Avro/Protobuf contracts (Schema Registry)
  • 3-5 autonomous AI agents (LangGraph, CrewAI, AutoGen) running multi-step workflows over event streams
  • Requirements of eventual consistency < 500ms and exactly-once on critical flows
  • DORA, NIS2, GDPR compliance with full traceability of automated decisions

The classic expiration_date case in pricing/promotions still exists, but it now coexists with:

  • Dynamic SLAs computed by ML models (GPT-4o / Claude 4 via function calling) that re-evaluate priorities every minute
  • Regulatory retention policies requiring selective event replay by correlation_id for up to 7 years
  • RAG + MCP agents querying knowledge graphs in Neo4j/GraphDB to enrich events with historical context
  • Distributed feature flags (LaunchDarkly, Unleash) that change consumer behavior without redeploying

Traditional distributed schedulers (Quartz, Spring Batch, Airflow, even Walmart’s Big Ben) introduce temporal coupling, single points of failure and unpredictable latency. Therefore, in 2025 the answer is not “more schedulers” —it is native stream processing with temporal semantics.

BAOSS 2026 Reference Architecture: Flink SQL + Iceberg + Kafka

Our current approach to event sourcing from static data with temporal attributes combines three layers:

  • Ingestion layer: Kafka 3.7+ (KRaft mode) with topic tiered storage to S3/GCS. Producers use transactional.id for end-to-end exactly-once.
  • Temporal processing layer: Flink SQL 1.19+ with watermarks, TTL state and temporal table joins. This is where we solve expiration without schedulers.
  • Analytics / replay layer: Apache Iceberg on object storage with time travel, partition evolution and row-level deletes for GDPR.

The Pattern: Temporal Table Join in Flink SQL

Instead of KTable foreign key joins (limited to primary keys and with no time semantics), we use Flink’s temporal table joins. A temporal table is a versioned-in-time view: each key has multiple versions with valid_from / valid_to. Moreover, the join resolves automatically according to the watermark of the incoming event.

-- Definition of the temporal prices table (versioned by expiration_date)
CREATE TABLE prices_temporal (
    price_id STRING,
    product_id STRING,
    amount DECIMAL(10,2),
    currency STRING,
    status STRING, -- 'ACTIVE', 'EXPIRED', 'SUPERSEDED'
    valid_from TIMESTAMP(3),
    valid_to TIMESTAMP(3),
    WATERMARK FOR valid_from AS valid_from - INTERVAL '5' SECOND,
    PRIMARY KEY (price_id, valid_from) NOT ENFORCED
) WITH (
    'connector' = 'kafka',
    'topic' = 'prices-cdc',
    'properties.bootstrap.servers' = 'kafka:9092',
    'scan.startup.mode' = 'earliest-offset',
    'format' = 'avro-confluent',
    'avro-confluent.schema-registry.url' = 'http://schema-registry:8081'
);

-- Incoming orders stream (business events)
CREATE TABLE orders_stream (
    order_id STRING,
    product_id STRING,
    quantity INT,
    order_ts TIMESTAMP(3),
    WATERMARK FOR order_ts AS order_ts - INTERVAL '10' SECOND
) WITH (
    'connector' = 'kafka',
    'topic' = 'orders',
    'properties.bootstrap.servers' = 'kafka:9092',
    'format' = 'avro-confluent'
);

-- Temporal join: each order gets the price VALID at its order_ts
SELECT
    o.order_id,
    o.product_id,
    o.quantity,
    p.amount AS unit_price,
    p.currency,
    (o.quantity * p.amount) AS total_amount,
    o.order_ts
FROM orders_stream o
LEFT JOIN prices_temporal FOR SYSTEM_TIME AS OF o.order_ts p
    ON o.product_id = p.product_id
WHERE p.status = 'ACTIVE';

Result: zero schedulers. Expiration is intrinsic to the data model. When the watermark passes valid_to, the version expires automatically from the state backend (incremental RocksDB checkpoints every 30s). Consequently, Flink emits a changelog to the output topic —downstream consumers receive the “deactivation” as just another event.

Real BAOSS Case (Anonymized): Multinational Retail — 47M Events/Day

Context: Retail client with 3,200 stores, e-commerce and marketplace. Legacy platform with a nightly Spring Batch job to expire promotions (4h window, frequent failures, manual replay). Migration to event-driven in 2026-2027.

Pre-Migration Metrics (Q4 2024)

  • Promotion expiration latency: 2-6 hours (nightly batch)
  • Monthly incidents from incorrect prices: 12-18 (€280K-420K/month impact)
  • Historical replay time (audit): 4-8 hours (DB dump + restore)
  • Batch infrastructure cost (EC2 r6g.4xlarge x 12): €18,500/month

BAOSS Implementation (Q1-Q2 2025)

  • CDC migration (Debezium 2.5+) → Kafka → Flink SQL (Kubernetes, K8s Operator)
  • Temporal table modeling for 14 domains (pricing, promos, stock, SLA, loyalty)
  • AI agent integration: 3 LangGraph agents consuming enriched events for dynamic pricing and stock allocation
  • Iceberg layer (Tabular/Trino) for audit time travel and the ML feature store
  • Observability: OpenTelemetry + Tempo + Mimir + Grafana (end-to-end P99 latency < 200ms)

Q3 2025 Results (6 months in production)

  • Expiration latency: < 500ms (watermark-driven, deterministic)
  • Incorrect price incidents: 0 in 6 months (vs 12-18/month)
  • Selective audit replay: < 3 min (Iceberg time travel + partition pruning)
  • Streaming infrastructure cost (EKS + MSK + S3): €11,200/month (-39% vs batch)
  • ROI 3.4x in 6 months (direct savings + avoided sanctions + recovered revenue)
  • New domain deployments: 2 days vs 3 weeks (Flink SQL templates + GitOps CI/CD)

The client’s internal team (12 engineers) now operates the platform autonomously. BAOSS provided architecture, coaching and knowledge transfer —not “body shopping”.

Second Case: Banking — DORA/NIS2 Compliance with Certified Event Replay

A European bank (assets >€200B) needed to demonstrate to the regulator full traceability of automated credit decisions (ML models + business rules). It also required point-in-time replay by application_id for up to 7 years.

  • Challenge: Events across 38 topics, evolved Avro schemas (150+ versions), PII in payloads.
  • Solution: Flink SQL temporal joins to reconstruct state at any event_ts + Iceberg row-level deletes for the right to be forgotten (GDPR Art. 17) without breaking time travel.
  • AI agents: 2 CrewAI agents (automated audit + regulatory narrative generation) consume curated events via MCP (Model Context Protocol) from Iceberg.
  • Key metric: The Q2 2025 DORA audit completed in 4 days vs the estimated 6 weeks. Zero critical findings.

2026-2027 Tools in Our Stack

CategoryToolRole in the BAOSS architecture
Ingestion & streamingKafka 3.7+ (KRaft), Amazon MSKCore topics with transactional.id for exactly-once; tiered storage to S3/GCS
CDCDebezium 2.5+Capture of changes from legacy databases into Kafka with no batch windows
Data contractsSchema Registry (Avro / Protobuf)Versioned schema evolution for 15-40 consuming services
Temporal processingFlink SQL 1.19+Temporal table joins, watermarks and TTL state (RocksDB) — expiration without schedulers
Analytical storageApache Iceberg + TrinoTime travel, partition evolution and row-level deletes for replay and GDPR
OrchestrationKubernetes + GitOps (CI/CD)Flink job deployment in 2 days vs 3 weeks
AI agentsLangGraph, CrewAI, AutoGen; GPT-4o / Claude 4Multi-step workflows over streams; function calling and RAG + MCP
Context / graphsNeo4j / GraphDBKnowledge graphs to enrich events with history
Feature flagsLaunchDarkly, UnleashConsumer behavior changes without redeploying
ObservabilityOpenTelemetry + Tempo + Mimir + GrafanaEnd-to-end traceability, P99 latency < 200 ms

Conclusion: Time Is Data, not a Scheduler

Five years after our first post on event sourcing with static data, the lesson holds and is amplified: expiration, consistency and regulatory compliance must not be delegated to nightly batch jobs. By modeling time as part of the data model itself —temporal tables in Flink, time travel in Iceberg— we eliminate temporal coupling, single points of failure and unpredictable latency. Likewise, we open the door for autonomous AI agents to operate on deterministic, auditable events.

Going from 12-18 monthly incidents due to incorrect prices to zero in 6 months, with a 3.4x ROI and audit replay in minutes instead of hours, is not magic. On the contrary, it is native stream architecture with temporal semantics. What’s more, the same applies to banking (DORA/NIS2), telco or logistics.

At BAOSS we do not deliver “body shopping”: we design the architecture, transfer the knowledge and leave your team operating the platform autonomously. Do you want to assess your current event-driven platform and find out where schedulers are costing you money and incidents? If so, let’s talk — we prepare a 2-week assessment with a concrete migration plan.

Prefer to talk to us directly? Contact the BAOSS team and tell us about your challenge.