TLDR Data 2026-05-14

Netflix introduced Data Projects to replace brittle ACLs and human-owned workflow identities across millions of tables and thousands of jobs. Projects group tables, workflows, secrets, and assets under durable team-owned app identities, with scoped roles and tokens to reduce permission churn.

Thousands of tiny Parquet files on S3 can break Spark reads with UnknownHostException, even when networking is fine, by overwhelming DNS, S3 LIST/GET calls, and driver/task metadata. Spark partition tuning can help stabilize reads, but the real fix is compaction and table formats like Delta Lake or Iceberg.

Enterprise AI agents fail in long workflows because they reset, lose context, and rely on bloated prompts or flat vector search. Microsoft's memory architecture uses consolidation, forgetting, and delayed maturation to keep high-value events, reaching 97.2% retention precision and stabilizing around 400 to 500 memories.

Meta migrated its massive data ingestion system from legacy customer-owned pipelines to a simpler self-managed service using a phased Shadow → Reverse Shadow → Cleanup lifecycle, row count and checksum checks, automated promotion tooling, custom debugging infrastructure, and rollback mechanisms to prevent bad CDC data propagation.

dbt's growth has created tension between its practitioner-led roots and enterprise ambitions. dbt must better protect community trust, improve dbt Core, strengthen integrations, fix developer ergonomics, and make dbt Cloud feel like a real IDE. The risk is not adoption, but alienating the users who made dbt valuable.

A 334-response April 2026 pulse survey shows data modeling pain is overwhelmingly organizational, not tooling: 28.1% want training, 24.6% clearer requirements, 21.6% more time, 21.0% dedicated ownership, and only 4.8% better tools. Modeling is often owned by whoever builds pipelines (42.5%), while only 19.2% have a dedicated modeler or architect, and 68.3% refactor only occasionally or rarely. Teams with enforced standards are about 5x more likely to say their models hold up.

Lakehouse query engines often struggle because the statistical metadata they need to plan queries, skip irrelevant data, size joins, and handle skew is optional, inconsistent, or missing across formats like Iceberg, Delta Lake, and Parquet. Without reliable stats, engines are forced to guess, leading to bad query plans, wasted reads, higher costs, memory issues, and slow or failed queries.

Kafka Queues (Share Groups) shine when consumer processing involves delays or external I/O that causes Head-Of-Line Blocking. By allowing more consumer instances than partitions, share groups enable linear scaling of throughput (tested up to 8x with 32 instances) with no noticeable per-instance overhead, making them very effective for I/O-bound workloads.

Quack is a new client-server protocol that lets separate DuckDB instances communicate over HTTP instead of only running in-process. It uses a request/response model with custom application/duckdb serialization, default token-based auth, localhost binding, and no SSL by default for local use, while supporting remote connections through standard HTTP infrastructure.

Views are just stored rewrite rules (macros) that get expanded at query time. They behave like tables for simple cases, but create hidden complexity through nested spirals, fragile dependencies on attribute numbers, painful schema changes, and limited writability, often leading to the classic advice: “use them, but don't treat them like tables.”

Agentic search models are emerging to orchestrate the full retrieval workflow, replacing today's brittle stack of embeddings, rerankers, query classifiers, and BM25 with thinner backend primitives. Unlike frontier LLMs that handle the “80% case,” models trained specifically for search can encode domain-specific intent and the “last 20%” of retrieval nuances, improving relevance in narrow contexts like e-commerce or job search. Early examples such as SID-1 and Waldo emphasize smaller size and lower latency.

Many data projects fail not because of technical issues, but because engineers jump straight into building without properly understanding the stakeholders' real needs and processes. Instead, start by asking the stakeholder to walk through their current workflow, create a one-sentence Definition of Done, ship an ugly MVP, and iterate on it to turn vague requests into shipped, adopted work while dramatically reducing wasted effort.

DuckDB is no longer just an embedded analytics database.

A reusable AWS ETL pattern is built with ECS Fargate, DuckDB, ECR, EventBridge, CloudWatch, Secrets Manager, and Terraform, targeting single-node jobs rather than Spark-heavy Glue.