TLDR Data 2026-10-12
LinkedIn Gives Agents Context π, Claude Dashboards π, Salesforce Spark Optimization β‘
Your agents sound confident. Cube makes them correct (Sponsor)
As agents get more autonomous, they act on your numbers without waiting for a person. An LLM pointed at raw tables re-derives your metric logic on every prompt. Whatever it guesses becomes your source of truth. π€¦ββοΈ
Cube is the agentic analytics and BI platform, built on a semantic layer. Your definitions and permissions are enforced deterministically β in dashboards, embedded apps, and Claude, ChatGPT or Cursor over MCP.
π Brex chose Cube over dbt Semantic Layer and LookML. Budget answers that took two days now take 2 seconds, for 35,000+ customers.
π Start free β free-forever tier with Chat and MCP β
What Defines The Streamhouse Architecture? (11 minute read)
Apache Fluss' βStreamhouseβ concept proposes one maintained logical table for streaming, serving, and analytics instead of separate synchronized systems. The key idea is to unify fresh stream data and historical lakehouse data behind shared metadata and tiering, so teams stop rebuilding state in multiple places. This can cut duplicate ingestion while still allowing specialized caches or replicas for specific access patterns.
Inside LinkedIn's data context layer: curating trustworthy context for humans and agents (16 minute read)
LinkedIn built a shared layer that tells people and AI agents which datasets to trust, which filters and joins are required, and what each row means. It combines profiles that an LLM writes from schemas, query logs, and code with notes that teams write and review in Git like code. After LinkedIn's analytics agent was connected to it, the share of acceptable answers in one test rose from 25.9% to 79.5%, and median time to answer fell from 10.7 to 4.7 minutes.
Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day (7 minute read)
Salesforce runs about 8 million Spark jobs a day and found some workloads left executors idle more than 90% of the time. Sizing targeted workloads to measured usage reduced weekly compute costs by 31%, while runtime, failures, and retries remained within acceptable levels.
From Activity to Intent: Generating User Journeys with LLMs (14 minute read)
Pinterest replaces a fragile keyword-clustering pipeline with a fine-tuned 4B LLM that converts a user's activity history into ranked, localized βjourneysβ (active goals). Teacher-model distillation makes it affordable at scale. The system improves notification engagement while applying strict output validation, safety filters, and feedback loops.
Claude Dashboards and the shift from chart builder to chart specifier (14 minute read)
Anthropic's new Claude Dashboards beta can build a dashboard without making a single chart manually. You can trace every number back to its source, but there is no row-level security, refresh, or audit log yet. The advice to BI developers: spend less time on tool tricks and more on data modelling, SQL, and design.
Five PostgreSQL queries that did more work than I asked for (4 minute read)
Five PostgreSQL benchmarks show how unnecessary deduplication, CTE materialization, random sorting, extra column retrieval, and unindexed expressions can dramatically slow queries, with small SQL changes or a matching index delivering major speedups.
Your Ontology Records Permission. It Does Not Resolve Authority (11 minute read)
Enterprise agents can enforce permissions yet make bad decisions when formal records omit the people whose judgment shapes outcomes. The Authority Resolution Framework separates documented rights, system permissions, recorded approvals, informal influence, and an agent's mandate, then treats disagreement among them as a governance signal. Audit authority gaps before automation turns organizational blind spots into confident actions.
Data pipelines and lineage for AI agents on AWS (Sponsor)
Your AI agent's accuracy comes down to the data pipeline behind it. Data Pipelines and Lineage is a guide to engineering that pipeline: source-to-response lineage, data quality checks, and pipeline observability.
Read the guide in AWS Marketplace.
Leviathan: Deep memory for agents over large datasets (GitHub Repo)
Leviathan is a Rust tool that gives AI agents compact, cited search results from large datasets using SQLite full-text indexing, with optional persistent memory and CLI, MCP, or remote access.
I'm now convinced single-node DuckDB is the best engine at any realistic scale (Reddit Thread)
A Databricks user found that Spark, Trino, and DuckDB all spent most of their time waiting on S3, with CPUs about 90% idle. After running four times as many read threads as CPU cores, one DuckDB machine beat Trino on the same hardware by 1.7 to 2.1 times on 10 TB queries. Against a Trino cluster with 5 to 13 nodes, it was 2 to 3 times slower but about half the cost.
Agent Mode in the DuckDB CLI (11 minute read)
The DuckDB 2.0 command line now detects when an AI agent is using it. It then switches to compact output, flags cut-off results, stops runaway queries, and returns errors as JSON. Across 132 TPC-H runs, agents read back about 60% fewer tokens and still got every answer right.
OntologyβDriven Observability: Building the E2E Knowledge Graph at Netflix Scale (51 minute video)
Netflix is using an ontology-driven knowledge graph to make observability data, incident chatter, and service topology queryable as one system. This delivers faster automated triage and root-cause analysis at large scale, with AI helping turn messy human signals into structured facts. The key takeaway: better incident response may come from modeling operational meaning, not just collecting more telemetry.
Postgres relation files, filed under base/ (7 minute read)
Postgres stores each relation as physical files identified by tablespace, database, and relfilenode, with extra forks and TOAST files appearing as needed. Logical objects and physical files are loosely coupled: operations like VACUUM FULL or REINDEX can rewrite storage and swap relfilenodes while the table's identity stays the same. Large relations also continue transparently across segmented files.
Curated deep dives, tools and trends in big data, data science and data engineering π
Join 590,000 readers for
one daily email