Run Your Own Benchmarks

Everyone says you should. BenchBox makes it easy to do. Use as CLI, Python library, or MCP for AI assistants. No Docker. No compilers. Just pip install.

CLI, Library, or MCP 
# CLI - Quick benchmarking
benchbox run --platform duckdb --benchmark tpch

# MCP - AI assistant integration
"Run TPC-H on DuckDB" # Claude executes via MCP

# Library - Deep Python integration
from benchbox import TPCH
Star on GitHub ⭐ v0.1.1 Python 3.10+ No Docker Required
BenchBox - Database Benchmarking Toolkit

What Makes It Simple

🚀

Python-Only Core

Minimal requirements. Python-only core with optional extras for cloud platforms. `pip install` and go.
📊

18 Benchmarks

TPC standards, academic (SSB, Join Order), industry (ClickBench, NYC Taxi), SQL primitives, and experimental workloads.

🔄

SQL & DataFrame

SQL databases (DuckDB, ClickHouse, cloud platforms) and DataFrame libraries (Polars, Pandas).
⚡

One Benchmark, Run Anywhere

Automatic SQL translation. Same benchmark queries run on local and cloud databases.
📈

Generate Data at Any Scale

10MB to 100TB! No need to download pre-generated files.
🤖

AI-Powered with MCP

MCP server for Claude Code and AI assistants. Run benchmarks with natural language.

Benchmarks by Category

TPC Standards

Official industry standards for comparing databases

TPC-H

Standard data warehouse benchmark with 22 business intelligence queries.

8 tables • 22 queries
BenchBox TPC-H Documentation →
TPC.org Standard ↗

TPC-DS

Complex decision support benchmark modeling multi-dimensional analysis.

25 tables • 99 queries
BenchBox TPC-DS Documentation →
TPC.org Standard ↗

TPC-DI

Data integration benchmark testing ETL operations and data quality.

15 tables • ETL workflow
BenchBox TPC-DI Documentation →
TPC.org Standard ↗

Academic Benchmarks

Research benchmarks from academia

Star Schema Benchmark

Simplified data warehouse testing basic OLAP functionality.

4 tables • 13 queries
BenchBox SSB Documentation →
Original Paper ↗

AMPLab

Big data benchmark focusing on machine learning and analytics workloads.

3 tables • 4 queries
BenchBox AMPLab Documentation →
Original Site ↗

Join Order Benchmark

Query optimizer testing with complex multi-table joins from IMDB data.

21 tables • 113 queries
BenchBox Join Order Documentation →
Original Paper ↗

Industry Benchmarks

Real-world benchmarks from practitioners

ClickBench

Analytics benchmark with real-world web analytics queries.

1 table • 43 queries
BenchBox ClickBench Documentation →
Official Site ↗

H2ODB

Data science benchmark testing aggregation and join performance.

2 tables • 5 query groups
BenchBox H2ODB Documentation →
Official Site ↗

NYC Taxi

Real-world transportation analytics with geospatial and temporal queries.

2 tables • 25 queries
BenchBox NYC Taxi Documentation →
NYC TLC Data ↗

TSBS DevOps

Time-series benchmark for infrastructure monitoring and observability.

Time-series • 18 queries
BenchBox TSBS Documentation →
Original Repo ↗

CoffeeShop

Small analytical benchmark modeling a coffee shop with sales and inventory.

3 tables • 10 queries
BenchBox CoffeeShop Documentation →
Original Post ↗

BenchBox Primitives

Fundamental database operation testing

Read Primitives

Read operation testing across 26 categories.

TPC-H schema • 109 queries
BenchBox Primitives Documentation →

Write Primitives

Comprehensive write operation testing including bulk loads and merges.

TPC-H schema • 113 operations
BenchBox Write Primitives Documentation →

Transaction Primitives

ACID transaction testing with isolation levels, commits, and rollbacks.

TPC-H schema • 8 operations
BenchBox Transaction Primitives Documentation →

BenchBox Experimental

Experimental benchmarks for specialized testing

TPC-Havoc

Query optimizer stress testing with 220 syntax variants of TPC-H queries.

TPC-H schema • 220 variants
BenchBox TPC-Havoc Documentation →

TPC-H Skew

Test optimizer behavior on non-uniform data distributions (Zipfian, exponential).

TPC-H schema • Skewed data
BenchBox TPC-H Skew Documentation →

TPC-DS-OBT

TPC-DS queries on a single denormalized "One Big Table" schema.

1 wide table • 99 queries
BenchBox TPC-DS-OBT Documentation →

Data Vault

TPC-H adapted for Data Vault 2.0 modeling (Hub-Link-Satellite pattern).

21 tables • 22 queries
BenchBox Data Vault Documentation →

Supported Platforms

Single-Node Analytics Engines

In-process and local columnar databases

DuckDB

In-process analytical database. Built-in, no setup required.

Built-in • OLAP

ClickHouse

Column-oriented OLAP database for real-time analytics.

Local/Cloud • OLAP

DataFusion

Apache Arrow-native query engine for in-memory analytics.

In-process • Arrow

Row-Based Databases

Traditional relational databases

PostgreSQL

The world's most advanced open source relational database.

OLTP/OLAP • Relational

SQLite

Embedded transactional database for testing and CI/CD.

Built-in • OLTP

Cloud Data Platforms

Enterprise cloud data warehouses and lakehouses

Snowflake

Multi-cloud data warehouse with elastic compute.

Cloud • Warehouse

Databricks

Data Intelligence Platform with lakehouse architecture.

Cloud • Lakehouse

BigQuery

Google Cloud serverless data warehouse.

GCP • Serverless

ClickHouse Cloud

Managed ClickHouse with serverless and dedicated options.

Cloud • OLAP

Redshift

AWS cloud data warehouse (serverless or provisioned).

AWS • Warehouse

Amazon Athena

AWS serverless query service for S3 data lakes.

AWS • Serverless

Azure Synapse

Microsoft Azure analytics service.

Azure • Analytics

Microsoft Fabric

Unified analytics platform with OneLake.

Azure • Unified

Firebolt

Cloud data warehouse optimized for analytics.

Cloud • OLAP

MotherDuck

Serverless cloud DuckDB with hybrid local/cloud execution.

Cloud • DuckDB

Starburst

Enterprise Trino with managed cloud offering.

Cloud • Federation

Open Source Platforms

Distributed query engines for federated data

Trino

Distributed SQL query engine for federated data.

Distributed • Federation

PrestoDB

Distributed SQL query engine (Meta's fork).

Distributed • Federation

Apache Spark SQL

Distributed SQL engine for large-scale data processing.

Distributed • Spark

Managed Spark Services

Cloud-managed Spark for lakehouse and data lake analytics

Onehouse Quanton

Serverless Spark with Hudi, Iceberg, and Delta Lake support.

Serverless • Multi-format

AWS Glue

Serverless data integration service with Spark ETL.

AWS • ETL

Amazon EMR Serverless

Serverless Spark with automatic scaling.

AWS • Serverless

Athena for Apache Spark

Interactive Spark notebooks on AWS Athena.

AWS • Interactive

GCP Dataproc

Managed Spark and Hadoop clusters on Google Cloud.

GCP • Clusters

GCP Dataproc Serverless

Serverless Spark with no cluster management.

GCP • Serverless

Microsoft Fabric Spark

SaaS Spark with OneLake integration.

Azure • SaaS

Azure Synapse Spark

Enterprise Spark pools with ADLS Gen2.

Azure • Enterprise

Snowpark Connect

PySpark API compatibility on Snowflake.

Snowflake • PySpark API

Time Series Databases

Optimized for time-stamped data

TimescaleDB

Time-series database built on PostgreSQL.

Time-series • PostgreSQL

InfluxDB

Time-series database for metrics and events.

Time-series • IoT

DataFrame Platforms

Native DataFrame APIs instead of SQL

Polars

Fast Rust-based DataFrame library with lazy evaluation.

polars-df • Expression API

Pandas

Reference Python DataFrame implementation.

pandas-df • Pandas API

PySpark DataFrame

Apache Spark DataFrame API for distributed computing.

pyspark-df • Distributed

DataFusion DataFrame

Arrow-native DataFrame with lazy evaluation.

datafusion-df • Arrow

Modin

Distributed Pandas replacement (Ray/Dask backend).

modin-df • Pandas API

Dask

Parallel computing library with DataFrame API.

dask-df • Distributed

cuDF (RAPIDS)

GPU-accelerated DataFrame library from NVIDIA.

cudf-df • GPU

Open Table Formats

Convert benchmark data to modern columnar formats for optimized storage, ACID transactions, and time travel capabilities.

File Formats

Columnar storage for fast analytics

Apache Parquet

Industry-standard columnar format with universal platform support.

Universal • 3-5x compression

Vortex

High-performance columnar format optimized for analytical workloads.

DuckDB/DataFusion • Best compression

Table Formats

ACID transactions, time travel, and schema evolution

Delta Lake

Open table format with ACID transactions and time travel on Parquet.

Databricks • Time Travel

Apache Iceberg

Modern table format with hidden partitioning and schema evolution.

Multi-engine • Enterprise

Apache Hudi

Table format optimized for incremental processing and record-level updates.

Streaming • Upserts

DuckLake

DuckDB's native table format with ACID transactions and optimal performance.

DuckDB Native • Time Travel
Convert benchmark data to any format 
# Convert TPC-H data to different formats
benchbox convert --input ./tpch_sf1 --format parquet
benchbox convert --input ./tpch_sf1 --format delta
benchbox convert --input ./tpch_sf1 --format iceberg
benchbox convert --input ./tpch_sf1 --format ducklake

AI Assistant Integration

BenchBox includes an MCP (Model Context Protocol) server for Claude Code and other AI assistants. Run benchmarks with natural language instead of memorizing CLI flags.

Quick Setup

Add to Claude Code 
claude mcp add benchbox -- uv run python -m benchbox.mcp

What You Can Do

Natural language commands for benchmarking workflows

Discover

"What benchmarks are available?"

"Which platforms support TPC-DS?"

Explore 18 benchmarks and 38 platforms without reading documentation.

Execute

"Run TPC-H on DuckDB at scale 0.1"

"Compare Polars and Pandas on SSB"

Run benchmarks without memorizing CLI syntax or options.

Analyze

"Which queries were slowest?"

"Compare results from my last two runs"

Get AI-powered analysis of performance patterns and regressions.

Full MCP Documentation →

Get Started

1. Install

uv add benchbox

2. Run as CLI

# Quick TPC-H benchmark on DuckDB
benchbox run --platform duckdb --benchmark tpch --scale 0.1

# Preview on cloud before spending credits
benchbox run --platform databricks --benchmark tpch --dry-run ./preview

3. Or Use as Library

from benchbox import TPCH

# Initialize and generate benchmark data
tpch = TPCH(scale_factor=0.1)
data_files = tpch.generate_data()

# Get schema and queries
create_sql = tpch.get_create_tables_sql()
query = tpch.get_query(1)  # Q1: Pricing Summary Report

Requirements

  • Python 3.10 or higher
  • No external dependencies for data generation
  • Optional: database drivers (duckdb, sqlite3, etc.)