TLDR Data 2026-11-19
MongoDB Buys Firebolt 🍃, Playing GTA 2 via Airflow 🎮, LinkedIn Rebuilds Data Lineage 🧵
Together With
Toptal launches Toptal AI: one partner for all your AI needs (Sponsor)
Toptal AI is a new full-stack AI services company that develops, evaluates, and deploys frontier-grade AI using open-weight and proprietary models, while organizations retain full ownership and control over their AI systems and data.
With Toptal AI, organizations can:
- Develop and fine-tune open-weight, domain-specific models tailored to their data and business.
- Deploy production-ready agents that operate across existing workflows while keeping humans in control.
- Commercialize their data and AI capabilities to create new revenue streams.
Learn more about Toptal AI.
De-identifying text at scale: How Airbnb protects personal data in AI training (11 minute read)
Training teams need realistic text data without letting PII leak into shared ML workflows. Airbnb packaged open-source Presidio, originally developed by Microsoft, as an in-process pipeline step, added domain-specific recognizers and multilingual evaluation, and then tagged de-identified datasets through its model registry. The useful pattern is turning privacy policy into testable, reusable data controls rather than asking every team to improvise.
MySQL tables in four open source Iceberg catalogs, read by Trino and DuckDB (5 minute read)
Can MySQL changes land in Iceberg without a bespoke warehouse pipeline? DBTrail followed the binlog, exported initial and incremental snapshots into four open-source REST catalogs, then checked row counts and aggregates through Trino and DuckDB against MySQL. The tests surfaced real interoperability gaps around credential vending and equality deletes—not just a happy-path compatibility claim.
Agentic data engineering is here! Project Osmos: What, Why, and How? (22 minute read)
Microsoft Fabric's Project Osmos is an AI agent that plans, builds, runs, and validates Spark data pipelines from a written specification. In a test using Vienna's public transit timetables, it built a working pipeline, but independent checks caught a daylight saving bug that the agent's own tests missed.
Rebuilding Data Lineage at LinkedIn: From Storage to Compute (15 minute read)
LinkedIn rebuilt its data lineage platform to capture dependencies directly from running jobs, giving engineers a more precise view of where data comes from and what depends on it. The resulting graph covers about 6 million datasets and helps teams trace incidents, assess the impact of changes, and safely retire unused pipelines.
Are we Finally Entering the Golden Age of Data Science? (6 minute read)
AI and modern data tools are making it faster to build pipelines and answer routine business questions. This gives data scientists more time to ask questions that matter, design experiments, and use AI agents to test what actually drives business outcomes.
Data Contracts (32 minute read)
Data contracts apply API design to datasets, describing their contents and guarantees so that people and AI agents can work with data on their own. A quality rule, such as less than 1% empty names, becomes an assertion that tooling can pass or fail. Published test results can then show how often each promise was kept.
One Connected Plan for Your IPO Decisions (Sponsor)
IPO decisions rarely live in one lane. When equity, liquidity, investing, and planning are connected, you can see tradeoffs clearly and act with purpose.
J.P. Morgan Private Bank brings a coordinated approach that adapts as priorities change. Bring your questions—we'll help you think about what comes next.
Connect with J.P. Morgan Private Bank
pgvectorfast (GitHub Repo)
pgvectorfast is a drop-in replacement for pgvector that uses a faster buffer pool to speed up vector search and analytical queries in PostgreSQL.
MongoDB Acquires Firebolt (4 minute read)
MongoDB has acquired Firebolt and will run its analytics engine alongside the database, kept in sync automatically so there is no pipeline to build. MongoDB pitches it as removing the data-movement tax, since agents can drive thousands of concurrent queries and need sub-second answers on fresh data. It will join search, vector search, and Voyage AI models in the platform.
Open Source DocumentDB 1.0 Reaches General Availability (3 minute read)
DocumentDB 1.0 brings an MIT-licensed, MongoDB-compatible document store to PostgreSQL through an extension and protocol gateway, letting existing drivers and queries work with a familiar relational foundation. The GA release covers PostgreSQL 15-18, adds a Kubernetes operator, and focuses on stability across indexing, aggregation, and geospatial queries.
Airflow plays GTA2 (GitHub Repo)
This Apache Airflow 3.4 demo uses a local AI model to drive a car through a custom delivery mission in Grand Theft Auto 2. Each move is one iteration of an Airflow task loop, and a dashcam plugin lets you watch the run live or replay it later.
How one SurrealQL query scans 10 billion rows a second (9 minute read)
SurrealDB says its Earth database scanned more than 10 billion rows per second in a launch benchmark by splitting one query across about 1,000 Cloudflare Workers. Each worker reads a different slice of the same database snapshot, and SurrealDB checks and combines the results into one answer.
Securing Enterprise Data Agents with Policy as Code: A Preflight Gate for Multi-Domain NL2SQL (12 minute read)
Shared service accounts can bypass the user-level controls a data agent is meant to enforce. This prototype parses generated SQL into an AST, resolves every table and column dependency, evaluates CEL policies, then prepares approved row filters before execution. Database roles remain the final guardrail. The takeaway: authorize the query's actual dependencies, not just its visible output.
Curated deep dives, tools and trends in big data, data science and data engineering 📊
Join 630,000 readers for
one daily email