Starting a Data Project in 2026: From the Avalanche to Autonomous AI Agents

In June 2015 we published on this very blog an X-ray of two classic scenarios. On one hand, the “data avalanche” that drowns teams. On the other, the “departmental silo” where Marketing and Technology speak different languages. Ten years later, the diagnosis has not changed as much as the tooling. What has mutated is the speed. For example, according to IDC, the global datasphere will exceed 181 zettabytes in 2026 (up from 64 ZB in 2020). In addition, Gartner estimates that 75% of companies will have operationalized generative AI by the end of 2026. The challenge is no longer storage, but deciding which piece of data triggers an autonomous agent at 3 a.m. while your team sleeps. This guide to starting a Big Data project in 2025 summarizes the 4-milestone methodology we use today.

starting a Big Data project in 2025: 4-milestone methodology

The real problem in 2026-2027: noise, latency and trust

Today organizations do not suffer from a lack of data. In fact, they suffer from an excess of signals without semantics. A retail client with 12,000 SKUs and 300 stores confessed to us: “We have dashboards for everything, but nobody knows why the average ticket drops on Tuesdays”. Consequently, the bottleneck has shifted from storage to meaning:

Semantic fragmentation: data catalogs disconnected from the business language (ERP, CRM, IoT, logs, PDFs, support tickets).
Decision-data latency: the “question → SQL → dashboard → meeting → action” cycle averages 11 days in mid-sized companies (BAOSS survey 2024, n=47).
Trust deficit: only 23% of executives “fully” trust the outputs of their internal models (MIT Sloan 2024).

However, the intuition we mentioned in 2015 remains the starting point. Today, moreover, it materializes in structured prompts that trigger RAG (Retrieval-Augmented Generation) flows over versioned knowledge graphs, not in spreadsheets shared by email.

Reference architecture 2026: from the Data Lake to Knowledge Graph + Agents

At BAOSS we have standardized a pattern we call Semantic Data Mesh. It is a layer of business ontologies (defined with stakeholders, not just IT) that exposes versioned data products via REST/GraphQL APIs and MCP (Model Context Protocol). On top of that layer run specialized agents orchestrated with LangGraph or CrewAI. Each one has its own system prompt, tools and guardrails.

Typical production stack (Q1 2025)

Ingestion/Streaming: Kafka / Redpanda + Debezium CDC + Apache Iceberg (ACID tables on S3/MinIO).
Semantic catalog: DataHub + OWL/SHACL ontologies synced with GitOps (ArgoCD).
LLM engine: GPT-4o / Claude 4 (via API) + local Llama-3.1-70B-Instruct models served with vLLM or Ollama for sensitive data (PII, financial).
Advanced RAG: semantic chunking + reranking (Cohere Rerank 3.5) + graph-RAG over Neo4j/FalkorDB.
Agent orchestration: LangGraph (stateful, cycles) / CrewAI (role-based) / AutoGen (conversational multi-agent).
Observability: LangSmith / Arize Phoenix + OpenTelemetry traces end-to-end.
Continuous evaluation: versioned eval sets (RAGAS, DeepEval) in CI/CD (GitHub Actions / GitLab CI).

For example, the key is that there is no “one model to rule them all”. For instance, a Text-to-SQL agent uses a small fine-tuned model (e.g. sqlcoder-7b on vLLM). An executive summary agent leans on GPT-4o. And a third one for regulatory validation runs locally with Ollama. The router (LangGraph) decides which to invoke based on intent, cost and SLA latency.

BAOSS Case #1: Omnichannel retail — from 11 days to 4 hours (anonymized)

Context: 300 stores, e-commerce, app, 12,000 SKUs, 1.2 M loyal customers. In addition, the data team has 6 people (2 DE, 2 DA, 1 ML, 1 lead).

Pain point: Questions like “why is conversion dropping in the sports category in Madrid?” required 3 manual joins. Then, validation with the store and a weekly meeting, so action arrived at day 11.

Intervention (8 weeks):

Weeks 1-2: first, domain-driven design workshops with Marketing, Operations and Finance → shared ontology (Product, Store, Customer, Ticket, Promo, Stock).
Weeks 3-4: next, Iceberg tables + DataHub + data contracts (Pydantic) versioned in Git. Also, CDC deployment from Oracle ERP + PostgreSQL e-commerce + Mongo app logs.
Weeks 5-6: then, Graph-RAG over FalkorDB (entities + temporal relationships). InsightHunter agent (LangGraph): receives a natural-language question → decomposes it into SQL sub-queries + graph traversals → synthesizes an answer with source citations + confidence level.
Weeks 7-8: finally, eval set of 200 real historical questions → RAGAS faithfulness 0.89 / answer_relevancy 0.92. Deployment in Slack + Teams via bot with human-in-the-loop for final validation.

For example, metrics at 6 months:

Time-to-insight: 11 days → 3.7 hours (median, p95 < 8 h).
Adoption: 84% of business users active weekly (vs 12% for the previous dashboards).
ROI: 3.2x in 6 months (estimated savings of €420k/year in meetings + stock/price decisions).
LLM cost: €0.018/question (70% local vLLM / 30% GPT-4o mix).

BAOSS Case #2: Mid-sized bank — autonomous regulatory compliance (anonymized)

Context: 450 employees, 180k customers and MiFID II + EBA Guidelines + LOPD regulation. In addition, the compliance team has 4 analysts + 1 IT person.

Pain point: manual review of 2,300 communications/month (emails, chats, recordings) generates 40% false positives. It also causes 3 weeks of delay and sanction risk.

Solution (10 weeks): first, a multimodal pipeline (Whisper-large-v3 → diarization → semantic chunking) + ComplianceGuard agent (CrewAI: 3 roles — Extractor, Validator, Redactor). Main model: Llama-3.1-70B-Instruct on vLLM (on-prem A100 80GB GPU) for zero data egress.

Specifically, results at 4 months:

False positives: 40% → 6% (precision 0.94 / recall 0.91).
Review time/analyst: 45 min → 8 min (only validating real findings).
Coverage: 100% of communications, versus 30% sampling before.
Inference cost: €0.003/communication (own infrastructure, amortized in 14 months).

BAOSS methodology for starting a Big Data project in 2025: 4 milestones, 0 “big bang”

The lesson from 2015 still holds: don’t try to boil the ocean. However, today the “ocean” has AI currents that change every quarter. Our pragmatic roadmap:

Week 1-2: Discovery & Ontology Sprint. 2 workshops of 3 h with business owners + data owners. Output: agreed business dictionary (Google Sheet → YAML → OWL), list of 15-25 prioritized competency questions (MoSCoW).
Week 3-6: First Data Product + Pilot Agent. One bounded domain (e.g. “Store stock + Promo + Ticket”), CDC to Iceberg, minimal RAG graph, 1 LangGraph agent exposed in Slack/Teams. Definition of Done: RAGAS > 0.85 on a real eval set + positive feedback from 3 key users.
Week 7-12: Horizontal scaling + Governance. Replicate the pattern to 2-3 more domains. CI/CD with semantic regression tests. Access policies (OPA/Rego) over data APIs. FinOps: cost dashboards per agent/domain.
Month 4-6: Progressive autonomy. Finally, introduce planner agents that chain sub-agents (LangGraph Supervisor). North-star metric: % of questions resolved without human intervention (target > 70% in mature domains).

In addition, every milestone delivers measurable value. If the pilot fails (rare when the ontology is born from the business), the cost is < €25k and 6 weeks — not 18 months and €400k of a traditional Data Lake.

2026 traps that didn’t exist in 2015 (and how to dodge them)

Prompt sprawl: 200 loose prompts in Notion/Confluence → version them in Git, treat prompts as code (tests, PR, rollback).
Eval theater: public benchmark metrics (MMLU, GSM8K) ≠ your domain. First, build eval sets with 100-200 real cases before touching the model.
Naïve RAG: fixed 512-token chunking + cosine similarity → hallucinations on tables/time series. Instead, use semantic chunking + graph-RAG + reranking.
LLM vendor lock-in: multi-model architecture (router + local fallback) from day 1. This way, MCP makes swapping easy.
Shadow AI: Marketing contracts SaaS with embedded AI without going through IT. That is why an approved AI services catalog + a single gateway (Kong/Traefik + auth/rate-limit/observability plugins) is advisable.

Indicative 2026 budget (mid-range company, 200-1,500 employees)

Item	Range (€/year)	Notes
Data infrastructure (Iceberg, Kafka, catalog, CI/CD)	35,000 – 90,000	Cloud (AWS/GCP/Azure) or on-prem (K8s + MinIO)
LLM APIs (GPT-4o, Claude 4, embeddings)	12,000 – 40,000	Volume-dependent; semantic caching cuts 60-70%
BAOSS team (lead + 2 consultants, 6 months)	120,000 – 180,000	Includes knowledge transfer, docs, governance
Estimated TOTAL	212,000 – 370,000	Average payback 5-7 months in proven AI cases

Conclusion: Big Data isn’t “started”, it’s “activated”

In 2015 the goal was to “get started with Big Data”: build infrastructure and wait for insights. However, in 2026-2027 the goal is to activate decisions. You don’t need “more data” — you need a semantic layer + agents that decide with governance and business metrics (not technical ones) from day 1.

At this point, the 2015 question still stands. That said, the focus changes: it is no longer “Is my company using Big Data?” but rather “Are your autonomous agents making better decisions than your team, 24/7, with traceability and auditable ROI?”. If the answer is no, the time to start is now — but with the right architecture, not with more dashboards.

Want to go from 11 days of decision-data latency to 4 hours (or less) with governed AI agents? Talk to our consulting team.

Passionate about Big Data, AI and analytics? Join the BAOSS team and grow with projects that matter in banking, retail, healthcare and the public sector, alongside partners such as IBM, Red Hat and Google. We are looking for data, AI, cloud, development and cybersecurity profiles. Work with us and send us your CV.

Prefer to talk to us directly? Contact the BAOSS team and tell us about your challenge.

How Do I Start a Big Data Project? (Updated 2026)