Apache Spark 3.5+ in 2025: Real cases, metrics and an LLM-ready architecture
In 2016 we wrote that Spark had “settled as a technological pillar”. In 2025, that phrase falls short. Spark is now the control plane where data engineering, classic ML and generative AI converge. The challenge is no longer “processing big data”. It is now serving features to a LangGraph agent in under 50ms, indexing 50TB of PDFs for RAG without blowing the GPU budget and governing everything under Unity Catalog or OpenLineage. Moreover, at BAOSS IT consulting we see it in every deployment. The challenge is no longer processing data, but serving it to agents in production.
For example, at BAOSS we have guided three clients —a European retailer, a tier-1 bank, an industrial manufacturer— through that transition. There is no conference theory here. There are deployed architectures, production metrics and lessons learned (anonymized, with real numbers).
The 2025-2026 problem BAOSS IT consulting solves: the data stack has fragmented (and gotten expensive)
Now then, companies come to us with the same pattern:
- First, a “modern” data lake (Iceberg/Delta/Hudi): but with slow SQL queries due to a lack of Z-Order / liquid clustering.
- Then, batch pipelines on Airflow + Spark: taking 6h; the business wants freshness under 15 min.
- Also, LLM experiments in notebooks (LangChain, LlamaIndex, CrewAI) that don’t scale to production: OOM on the driver, single-threaded embeddings, no prompt versioning.
- Likewise, runaway cloud cost: fixed 24/7 clusters, idle GPU instances, shuffle spill to disk.
- Finally, broken governance: lineage broken between raw table and feature store; unmasked PII in embeddings.
Therefore, the answer is not “buy Databricks/Fabric/EMR Serverless and pray”. It is redesigning the architecture on Spark 3.5+ leveraging:
- First, Spark Connect: lightweight clients (VS Code, remote PySpark, JDBC/ODBC) with no local driver.
- Then, Structured Streaming + stateful processing: session windows, exact-once deduplication, “as-of” joins with Delta tables.
- Also, Pandas API on Spark / PySpark 3.5: 1:1 migration of legacy pandas code to distributed.
- Likewise, English SDK / AI Functions: `df.select(ai(“summarize this complaint in 3 bullets”))`. This calls GPT-4o / Claude 4 Sonnet via native batch inference.
- Finally, native GPU scheduling: (NVIDIA RAPIDS plugin + vLLM / Ollama on workers): embedded inference without moving data.
Case 1 — Retail: Real-time personalization under 50 ms P99
Context: 120M events/day (clicks, add-to-cart, purchases) on Kafka. The stack combines a ranking model (LightGBM) + LLM re-ranking (Llama 3.1 8B LoRA) to “reason” over offers. SLA: end-to-end under 50 ms from event to recommendation in the app.
Deployed architecture
- First, Ingest: Kafka → Spark Structured Streaming (10 min watermark, RocksDB state off-heap).
- Online feature store: Redis Cluster (7d TTL) populated by a micro-batch every 30s from Delta Lake (CDF).
- Hybrid inference:
- LightGBM (CPU) → broadcast variable, scoring in mapPartitions (vectorized Pandas UDF).
- LLM re-ranking → vLLM on GPU workers (A10G 24GB) invoked via gRPC from mapPartitions; dynamic batching of 32 reqs.
- Serving: Spark Connect endpoint + gRPC gateway (Go) exposed to Kubernetes (KServe).
Hard metrics (3 months in prod)
| KPI | Before (nightly batch + heuristics) | After (streaming + LLM) |
|---|---|---|
| Event→reco P99 latency | ~4 h (batch) | 42 ms |
| CTR uplift | baseline | +18 % |
| Infra cost / month | €42k (fixed EMR + SageMaker endpoints) | €25k (EMR Serverless spot + GPU autoscaling 0→12) |
| New model deployment time | 2 weeks (manual CI/CD) | 4 h (MLflow 2.14 + GitOps ArgoCD) |
Technical key: stateful streaming + broadcast model + colocated vLLM avoids network hops. Also, the 40 % cost reduction came from shutting down fixed clusters. The key was using EMR Serverless + Spot Fleet (max 20 % on-demand). We also applied aggressive dynamic allocation (60s idle scale-down).
Case 2 — Banking: Corporate RAG at scale (50 TB, 12M docs)
Challenge: Index the legal/contracts archive (PDFs, OCR scans, emails) for internal assistants (Claude 4 Opus + RAG). However, the legacy pipeline (Python + single-threaded LlamaIndex) took 14 days for a full re-index. Goal: <strong>< 8 h and full governance (lineage, PII, prompt versioning).
Spark-native pipeline
- Ingest: Autoloader (cloudFiles) → Delta Lake bronze (binary + metadata).
- Then, distributed parsing:
pandas_udfwith marker/pdfplumber (CPU) + Tesseract GPU for scans. Output: clean markdown + bounding boxes. - Also, semantic chunking: LangGraph agent (recursive splitter + LLM boundary detection) run as mapInPandas (batch of 512 docs/worker).
- Likewise, embeddings: vLLM + BGE-M3 (1024-d) on GPU workers;
foreachPartition→ batch upserts of 1k to Milvus/Zilliz (HNSW, SQ8). - Finally, governance:
- Unity Catalog:
silver.chunkstable (automatic lineage from bronze). - OpenLineage + Marquez: prompt→chunk→embedding traceability.
- PII detection (Presidio) in silver; masking policy in the Catalog.
- Unity Catalog:
Results
- First: 50 TB full re-index: 6.3 h (vs 14 days) — 53x speedup.
- Then: indexing cost: €0.18 / GB (Spot GPU g5.2xlarge, 90 % utilization).
- Also, incremental freshness via CDC (Change Data Feed) → 15 min micro-batch. Thus, new chunks land in the vector DB in under 3 min.
- Finally, adoption: 1.2k daily active users. RAG P99 latency is 1.2 s (retrieval 300 ms + LLM 900 ms).
The leap was not “using Spark”, but modeling the RAG pipeline as declarative ETL (Delta Live Tables / DLT). We added quality expectations (chunk_size, embedding_norm, pii_free) that block deployment if they fail. In short, when those pipelines feed autonomous agents, the risk of AI agents demands control from the design stage.
Case 3 — Industrial: Predictive maintenance + GenAI reports (3.2x ROI in 6 months)
Plant: 4.5k sensors (vibration, temperature, pressure) @ 10 Hz → 3.8 TB/day. Goal: detect anomalies and generate a natural-language technical report for the shift operator.
Unified “Lakehouse + AI” architecture
- First, Edge → Kafka → Iceberg (on-prem MinIO): Spark Structured Streaming (exactly-once, S3-compatible checkpoint).
- Then, feature engineering: Also: time windows (1h, 24h), FFT and rolling statistics. Everything is written in Pandas API on Spark (direct migration from data science notebooks).
- Then, classic ML: Isolation Forest + XGBoost (Spark MLlib) trained on a CPU cluster (r6gd.16xlarge). MLflow model registry + Model Serving via Spark UDF (nightly batch scoring).
- Also, GenAI report: Alert → LangGraph agent, which retrieves historical context and PDF manuals indexed in Case 2. Then → GPT-4o mini (batch inference via the
ai()function). The PDF/HTML is then served on the internal portal. - Finally, human-in-the-loop: The operator validates/corrects → feedback loop to Delta (
gold.operator_feedbacktable) → automated weekly retraining (DLT pipeline). - Governance of this automated layer follows the lesson from the Hugging Face case for your business. That is: explicit permissions, audit and reversibility.