TLDR Data 2026-07-09

HubSpot scaled its Vector-as-a-Service platform from an early Qdrant proof of concept into a production semantic-search layer with 20B+ vectors, 200+ indexes, 140+ clusters, and usage across 38+ teams. To make that manageable, it moved from manual Helm-based operations to Kubernetes operators that automate cluster creation, scaling, shard balancing, and replication recovery, cutting spin-up from hours to minutes and reducing operational load.

Apache Iceberg v3's Variant type solves the JSON-in-a-string problem by storing semi-structured data in a compact binary format and shredding common fields into real Parquet columns. That preserves flexibility for evolving schemas while enabling typed reads, column pruning, and file/row-group statistics. Shredding trades write time for read speed. It delivers an open-standard path to faster analytics on telemetry, events, and API payloads without constant schema migrations.

Apache DataFusion can run billion-edge graph algorithms on a laptop-class memory budget by pushing graph processing into disk-backed bulk scans, sort-merge joins, aggregations, and spill-aware execution. PageRank on Graphalytics graph500-26 with 1.05B edges computes using 5GB RAM in ~30 minutes for 15 iterations, and computes weakly connected components on twitter_mpi with 1.96B edges using 10GB RAM.

AI agents can help across data engineering, but production work needs a correctness layer: deterministic tooling that validates SQL, schemas, lineage, query equivalence, and blast radius instead of relying on a model's confidence. The article argues for clear project structure, modularity, declarative configs, and purpose-built data agents for serious workflows.

A framework for building reliable agents across three layers: the model, the agent harness that runs tool loops, context management, guardrails, memory, observability, and retries, and the harness configuration that gives the agent the right use-case context, tools, permissions, and human-review thresholds.

AI-generated data pipelines tend to run fine but be silently wrong since models are non-deterministic and blind to the actual data. Adhere to fundamentals like parameterization and idempotency, let the agent inspect real schemas via database MCPs, define the output as an upfront contract validated before publishing (Write-Audit-Publish), and package it all as reusable skill files.

Same approved inputs, different output next run? Then you don't have production control. Run 15 yes/no checks across reproducibility, reviewability, and rollback, and see exactly where your releases can drift. Score and a prioritized fix list on screen in under 10 minutes. Run the 15 checks →

Apache Ossie, formerly the Open Semantic Interchange effort, is now an Apache Incubating project. The spec models datasets, fields, metrics, dimensions, and relationships so teams can keep definitions consistent across tools while giving agents governed business context.

Apache Hudi brings native vector search capabilities to the lakehouse, allowing efficient semantic search and RAG applications directly on large-scale Hudi tables without needing a separate vector database, supporting vector columns, indexing strategies (e.g., HNSW), hybrid search (vector + filters), and tight integration with Hudi's table services and query engines.

Malloyyo is a lightweight MCP server and web app that lets AI tools query published data models, run analyses, and return shareable results. It includes a CLI for developing and publishing models, built-in DuckDB support for local files, and a browser UI for reviewing, editing, re-running, and sharing queries.

linger.ms is a key Kafka producer setting that controls how long to wait before sending a batch, directly affecting throughput, latency, and efficiency. Higher values enable larger, more efficient batches and higher throughput at the cost of increased latency, while low values prioritize low latency but create many small batches and higher overhead.

Grammar for Data Engineering proposes treating data pipelines as a formal language with defined grammar, semantics, and composability. This makes workflows more declarative, reusable, testable, and maintainable compared to ad-hoc scripts or rigid DAGs.

A strong semantic layer gives AI a governed map of business meaning, so it can turn user intent into reliable queries instead of plausible but wrong SQL. The curated model is the product, not the agent.

LLM, RAG, and eval skills are becoming table stakes, much like SQL.

ClickHouse's RayTracer is a full path tracer written entirely in ClickHouse SQL that renders PNG images, procedural terrain, CSG geometry, shadows, reflections, and even animated text without UDFs or external code.