Data ToolsDatabaseSQLData EngineeringEmbeddedOpen Source

An Embedded SQL Engine For Files That Is Honest About Its Speed

SlothDB is an experimental embedded SQL engine in C++20 that queries Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly in-process, reaching 476 GitHub stars from data engineers who want a DuckDB-style tool for ad hoc file querying, and its issues expose the gap between a benchmark demo and a dependable engine: reading Parquet with zstd compression fails, Avro queries return invalid results, Python wheels ship without the required native library, and a pointed top-voted issue argues the speed claims are exaggerated because the planner pattern-matches known benchmark query shapes rather than generalizing. People want fast, embeddable SQL over the messy file formats they actually have. The wedge is an embedded SQL engine whose format support is correct and whose performance is honest on real, non-benchmark queries.

Overall

Problem Statement

A data engineer reaches for an embedded SQL engine to query the Parquet, Avro, and Excel files they already have, but Parquet files with zstd compression fail to read, Avro queries return invalid responses, the Python wheel is missing its native library, and the headline speed turns out to come from a planner that pattern-matches known benchmark queries rather than generalizing. The promise of fast in-process SQL over any file is compelling, but an engine that fails on common compression and formats and is fast only on benchmark shapes cannot be trusted with real pipelines.

The Idea

An embedded, in-process SQL engine that correctly queries real-world file formats like Parquet, Avro, and Excel with honest, generalizable performance instead of benchmark-tuned shortcuts.

Why Now

DuckDB proved that in-process SQL over files is how analysts and apps want to work in 2026, and SlothDB's traction shows appetite for alternatives, but its zstd Parquet failures, broken Avro reads, missing native libraries, and a community calling out benchmark-shaped optimizations show that correctness across formats and honest general-case performance, not flashy ClickBench numbers, are what stand between a demo engine and one teams embed in production.

Target User

Data engineers and app developers needing in-process SQL over file formats

Target Market

Embedded analytical SQL engines

MVP Scope

Correct reads for Parquet with zstd and other common codecs.
Reliable Avro and Excel query results.
Python wheels bundling all native dependencies.
A planner with honest, generalizable performance.
No distributed or cloud features at MVP.

Validation Plan

1Pass a correctness suite across all advertised formats and codecs.
2Benchmark on non-ClickBench queries with honest numbers.
3Confirm Python wheels install and run with no missing libraries.
4Reach an adoption milestone among data engineers in a quarter.
5Interview 15 engineers on what they need beyond DuckDB.

Product Workflow

1Developer embeds the engine in-process in their app or pipeline.
2It queries Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel directly.
3Compressed formats like zstd Parquet read correctly.
4The planner generalizes across query shapes, not just benchmarks.
5Distributed wheels include all required native libraries.

Score Rationale

Market pull

Demand50

Pain54

Pay intent50

Timing58

Friction

Competition40

Build difficulty64

Acquisition difficulty54

Risk58

Conviction

Novelty48

Confidence60

Monetization

Open-source engine with a commercial license for embedding in proprietary products, plus paid enterprise support and connectors for niche formats, mirroring how embedded-database vendors monetize. The value metric is queries served correctly in-process; the buyer is the software vendor or data team that embeds the engine and will pay for support, format coverage, and a commercial license once correctness is proven.

Strategic Upside

A correct, honestly-benchmarked embedded engine can win developers wary of DuckDB lock-in or needing specific format support, and grow into commercial embedding and support; correctness and credible performance are the moat, because data engineers test engines hard and an engine that fails on real formats or inflates benchmarks loses their trust permanently.

Acquisition Channels

•r/dataengineering and r/Database.
•Comparison content versus DuckDB and DataFusion.
•Hacker News Show HN with honest benchmarks.
•Data-engineering newsletters and Slack communities.
•Conference talks on embedded analytics.

Competitors

Risks

DuckDB is dominant, fast, and extremely well supported.
Building a correct SQL engine is a multi-year effort.
Credibility damage from benchmark claims is hard to undo.
Format edge cases are endless and costly to cover.
Monetizing open-source embedded engines is difficult.

Counter Evidence

DuckDB already queries most of these formats correctly and fast.
DataFusion offers an embeddable Rust engine with momentum.
Most teams have no reason to leave DuckDB.

The full brief is free to read

Create a free account to unlock the complete build-ready brief for “An Embedded SQL Engine For Files That Is Honest About Its Speed”, including:

Problem statement & the core idea
Why now & target user / market
MVP scope & feature boundaries
Step-by-step validation plan
Score rationale across 11 dimensions
Monetization model & pricing angle
Competitors with links
Acquisition channels & go-to-market
Risks & counter-evidence

More Data Tools opportunities

Data Tools

Resource Consumption Tracker and Cost Allocation Engine for Fivetran

Buyer reviews for Fivetran consistently highlight cost management gap friction, specifically: MAR-based pricing is opaque, can't predict costs when source schemas change. A ; No way to set per-connector cost budgets or pause syncs when spending thresholds. This pain is concentrated among Data team leads managing ELT pipeline budgets with unpredictable volumes and creates demand for a focused tool that resolves the gap without requiring a platform switch. The Data Tools category has matured enough that users have committed to Fivetran as infrastructure, making adjacent tooling more viable than platform replacement.

View opportunity Data Tools

Automated QA and Configuration Validator for dbt Workflows

Buyer reviews for dbt consistently highlight testing gap friction, specifically: Data testing beyond basic schema tests requires custom macros. No built-in anoma; Test coverage reporting doesn't exist natively. Can't see which columns lack tes. This pain is concentrated among Analytics engineers managing data transformation quality in production and creates demand for a focused tool that resolves the gap without requiring a platform switch. The Data Tools category has matured enough that users have committed to dbt as infrastructure, making adjacent tooling more viable than platform replacement.

View opportunity Data Tools

Data Migration Toolkit and Platform Transition Planner for Stitch Data Users

Buyer reviews for Stitch Data consistently highlight migration difficulty friction, specifically: Since Talend acquired Stitch, development has stalled. Connectors break and don'; Need to migrate off Stitch but evaluating Fivetran, Airbyte, and Meltano is a 3-. This pain is concentrated among Data engineers moving off Stitch after Talend acquisition uncertainty and creates demand for a focused tool that resolves the gap without requiring a platform switch. The Data Tools category has matured enough that users have committed to Stitch Data as infrastructure, making adjacent tooling more viable than platform replacement.

View opportunity Data Tools

AI Database Query Optimization Advisor

Slow database queries degrade application performance but most developers lack DBA expertise to optimize them. An AI query advisor that analyzes slow queries, suggests indexes, and recommends rewrites could bring DBA-level optimization to every team.

View opportunity Data Tools

Self-Updating Client Report Generator for Digital Marketing Agencies

Preswald enables building data apps and dashboards, but agencies have a more specific pain: client reports that must be rebuilt every week with fresh data. A self-updating report generator that pulls data from Google Analytics, ad platforms, and SEO tools, formats it in a client-ready template, and sends it on schedule would eliminate 5-10 hours of weekly agency busywork.

View opportunity Data Tools

Unified Batch and Streaming Data Pipeline with Python API

Data engineers maintain separate codebases for batch and streaming pipelines. A unified Python framework that runs the same transformation logic in both batch and real-time modes could eliminate pipeline duplication and reduce maintenance burden by 50%.

View opportunity