Tutorials on DuckDB Lab

DuckDB for Millions of Data Records: From Raw CSV to Analytics Report

Mon, 11 May 2026 00:00:00 +0000

1. The Problem: Excel Crashes on 1 Million Rows

Meet Alice, an operations analyst at an e-commerce company. Every day at 4 PM, she needs to process a CSV file with 1.2 million sales records and generate a daily sales dashboard.

Her traditional workflow looks like this:

1. Double-click CSV → Excel warns "Some data may be lost" (row limit exceeded)
2. Switch to Python Pandas → import takes 12 seconds
3. groupby aggregation → memory spikes to 3.5GB, laptop fan goes crazy
4. Generate charts → manually tweak Matplotlib styles
5. Export report → send to the boss

Total time: 25-40 minutes, with each report requiring a separate Python script to maintain. Worse — when data grows to 5 million rows, Pandas simply crashes with OOM (Out of Memory).

Is there a tool that’s as simple as Excel but delivers database-level performance?

The answer is DuckDB.

2. Why DuckDB Excels at Million-Row Data Processing

DuckDB is an embedded columnar OLAP database designed specifically for analytical workloads. Here’s how it stacks up:

Feature	DuckDB	Pandas	Excel	Traditional DB (PostgreSQL)
Million-row aggregation	0.5-3 sec	5-30 sec	Not supported	2-10 sec
Memory usage	200-500 MB	1-5 GB	Crashes	Config-dependent
Install size	~50 MB (single binary)	~500 MB (with deps)	~1 GB	~200 MB
SQL support	Full SQL standard	Method chaining	Limited	Full
Direct CSV query	Yes (zero ETL)	Needs pd.read_csv	Native	Needs import
Parallel processing	Auto multi-threaded	Manual	Single-threaded	Multi-threaded
Spill-to-disk	Automatic	OOM crash	Crash	Automatic

The Secret Sauce: What Makes DuckDB So Fast?

Columnar storage: Only reads the columns you need, not entire rows
Vectorized execution: Processes batches (1024 rows at a time), not row-by-row
Lazy loading / predicate pushdown: WHERE clauses filter data during read, skipping irrelevant data entirely
Multi-threaded parallelism: Automatically uses all available CPU cores
Zero-copy reads: Operates directly on memory-mapped files, avoiding data copying

3. Hands-On: Processing 1 Million E-Commerce Sales Records

Scenario

You have an e-commerce sales CSV (sales_2026.csv) with 1 million records:

Field	Type	Description
order_id	INTEGER	Order ID
order_date	DATE	Order date
customer_id	VARCHAR	Customer ID
product_category	VARCHAR	Product category
product_name	VARCHAR	Product name
quantity	INTEGER	Quantity
unit_price	DECIMAL(10,2)	Unit price
total_amount	DECIMAL(10,2)	Total amount
region	VARCHAR	Region
payment_method	VARCHAR	Payment method

Step 1: Generate 1 Million Test Rows

-- Generate 1 million rows directly in DuckDB
SET memory_limit = '4GB';

CREATE TABLE sales AS
SELECT 
 row_number() OVER () AS order_id,
 '2025-01-01'::DATE + (random() * 365)::INTEGER AS order_date,
 'CUST_' || lpad((random() * 5000)::INTEGER::VARCHAR, 5, '0') AS customer_id,
 (CASE (random() * 4)::INTEGER
 WHEN 0 THEN 'Electronics'
 WHEN 1 THEN 'Clothing'
 WHEN 2 THEN 'Food & Beverage'
 WHEN 3 THEN 'Home & Garden'
 ELSE 'Books'
 END) AS product_category,
 'Product_' || lpad((random() * 200)::INTEGER::VARCHAR, 3, '0') AS product_name,
 (random() * 10 + 1)::INTEGER AS quantity,
 round((random() * 500 + 10)::NUMERIC, 2) AS unit_price,
 round((quantity * unit_price)::NUMERIC, 2) AS total_amount,
 (CASE (random() * 4)::INTEGER
 WHEN 0 THEN 'North'
 WHEN 1 THEN 'East'
 WHEN 2 THEN 'South'
 WHEN 3 THEN 'West'
 ELSE 'Central'
 END) AS region,
 (CASE (random() * 3)::INTEGER
 WHEN 0 THEN 'Credit Card'
 WHEN 1 THEN 'PayPal'
 WHEN 2 THEN 'Bank Transfer'
 ELSE 'COD'
 END) AS payment_method
FROM range(1000000);

-- Verify
SELECT count(*) AS total_rows FROM sales;
-- Output: 1000000

-- Export to CSV
COPY sales TO '/tmp/sales_2026.csv' (FORMAT CSV, HEADER true);

Step 2: Query CSV Directly (Zero ETL!)

No need to import CSV into a database — DuckDB queries it directly:

-- Direct CSV query, zero import
SELECT 
 region,
 count(*) AS order_count,
 round(sum(total_amount)) AS total_revenue,
 round(avg(total_amount), 2) AS avg_order_value
FROM read_csv('/tmp/sales_2026.csv', 
 header = true, 
 columns = {
 'order_id': 'INTEGER',
 'order_date': 'DATE',
 'customer_id': 'VARCHAR',
 'product_category': 'VARCHAR',
 'product_name': 'VARCHAR',
 'quantity': 'INTEGER',
 'unit_price': 'DECIMAL(10,2)',
 'total_amount': 'DECIMAL(10,2)',
 'region': 'VARCHAR',
 'payment_method': 'VARCHAR'
 })
WHERE order_date >= '2025-06-01'
GROUP BY region
ORDER BY total_revenue DESC;

Performance comparison:

Operation	Pandas	DuckDB (direct CSV)
Read 1 million rows	12.3 sec	0.4 sec (metadata only)
Aggregation by region	3.1 sec	0.6 sec
Total time	15.4 sec	0.6 sec
Peak memory	1.8 GB	210 MB

Step 3: Group-By Aggregation in Action

-- Import into DuckDB internal storage (faster for repeated queries)
CREATE TABLE sales_imported AS 
SELECT * FROM read_csv('/tmp/sales_2026.csv', header = true);
-- Import time: ~1.2 seconds

-- 1. Monthly sales trend
SELECT 
 strftime(order_date, '%Y-%m') AS month,
 count(*) AS orders,
 count(DISTINCT customer_id) AS unique_customers,
 round(sum(total_amount)) AS revenue,
 round(avg(total_amount), 2) AS avg_order
FROM sales_imported
GROUP BY month
ORDER BY month;

-- 2. Product category ranking
SELECT 
 product_category,
 count(*) AS orders,
 round(sum(total_amount)) AS revenue,
 round(avg(total_amount), 2) AS avg_order_value,
 sum(quantity) AS total_units
FROM sales_imported
GROUP BY product_category
ORDER BY revenue DESC;

-- 3. Region × Month cross-analysis (CUBE query)
SELECT 
 region,
 strftime(order_date, '%Y-%m') AS month,
 count(*) AS orders,
 round(sum(total_amount)) AS revenue
FROM sales_imported
GROUP BY CUBE(region, month)
ORDER BY region, month;

-- 4. TOP 10 high-value customers
SELECT 
 customer_id,
 count(*) AS order_count,
 round(sum(total_amount)) AS total_spent,
 round(avg(total_amount), 2) AS avg_order_value,
 max(order_date) AS last_order_date
FROM sales_imported
GROUP BY customer_id
ORDER BY total_spent DESC
LIMIT 10;

Step 4: Window Functions & Advanced Analytics

-- 1. 30-day moving average of daily revenue
SELECT 
 order_date,
 round(sum(total_amount)) AS daily_revenue,
 round(avg(sum(total_amount)) OVER (
 ORDER BY order_date 
 ROWS BETWEEN 29 PRECEDING AND CURRENT ROW
 ), 2) AS moving_avg_30d
FROM sales_imported
GROUP BY order_date
ORDER BY order_date;

-- 2. Month-over-month comparison
WITH monthly AS (
 SELECT 
 strftime(order_date, '%Y-%m') AS month,
 round(sum(total_amount)) AS revenue
 FROM sales_imported
 GROUP BY month
)
SELECT 
 month,
 revenue,
 lag(revenue) OVER (ORDER BY month) AS prev_month,
 round((revenue - lag(revenue) OVER (ORDER BY month)) / 
 lag(revenue) OVER (ORDER BY month) * 100, 2) AS mom_change_pct
FROM monthly
ORDER BY month;

-- 3. Cohort-style customer first/last purchase analysis
SELECT 
 customer_id,
 min(order_date) AS first_purchase,
 max(order_date) AS last_purchase,
 count(*) AS total_orders,
 datediff('day', min(order_date), max(order_date)) AS lifetime_days,
 round(sum(total_amount)) AS lifetime_value
FROM sales_imported
GROUP BY customer_id
HAVING count(*) >= 5
ORDER BY lifetime_value DESC;

Step 5: Exporting Reports

-- Export aggregation results as CSV
COPY (
 SELECT 
 product_category,
 region,
 strftime(order_date, '%Y-%m') AS month,
 count(*) AS orders,
 round(sum(total_amount)) AS revenue
 FROM sales_imported
 GROUP BY product_category, region, month
 ORDER BY revenue DESC
) TO '/tmp/sales_report.csv' (FORMAT CSV, HEADER true);

-- Export as Parquet (7:1 compression ratio)
COPY sales_imported TO '/tmp/sales_2026.parquet' (FORMAT PARQUET);
-- CSV: 85 MB → Parquet: 12 MB

-- Export JSON
COPY (
 SELECT region, round(sum(total_amount)) AS revenue
 FROM sales_imported GROUP BY region
) TO '/tmp/region_summary.json' (FORMAT JSON);

4. Batch Processing: Merge Hundreds of CSV Files in One Line

Real-world data is often split across multiple files. DuckDB’s glob pattern support makes merging trivial:

-- Scenario: 100 daily-partitioned CSV files
-- File naming: sales_2025-01-01.csv ... sales_2025-04-10.csv

-- Merge and aggregate in one line
SELECT 
 strftime(order_date, '%Y-%m') AS month,
 product_category,
 count(*) AS orders,
 round(sum(total_amount)) AS revenue
FROM read_csv('/data/sales_*.csv', 
 header = true, 
 union_by_name = true -- auto-match columns by name
)
WHERE order_date >= '2025-01-01'
GROUP BY month, product_category
ORDER BY month, revenue DESC;

-- Merge all files into one big table
CREATE TABLE all_sales AS
SELECT * FROM read_csv('/data/sales_*.csv', 
 header = true, 
 union_by_name = true);

Files × Performance Benchmark

Files	Total Rows	DuckDB Merge	Pandas Merge
10	1M	0.3 sec	4.2 sec
50	5M	1.5 sec	28 sec
100	10M	3.2 sec	62 sec (OOM risk)
365	36.5M	11.8 sec	Cannot complete

5. Memory Management & Optimization Tips

When dealing with larger datasets, these techniques save you time and memory:

5.1 Limit Memory Usage

-- DuckDB defaults to all available memory — you can cap it
SET memory_limit = '1GB';
SET temp_directory = '/tmp/duckdb_tmp'; -- spills to disk when needed

5.2 Projection Pushdown: Read Only What You Need

-- Bad: read all columns, then filter
SELECT region, sum(total_amount)
FROM read_csv('/data/huge_file.csv', header = true)
WHERE region = 'East';

-- Good: projection pushdown (read only required columns)
SELECT region, sum(total_amount)
FROM read_csv('/data/huge_file.csv', 
 header = true,
 columns = {'region': 'VARCHAR', 'total_amount': 'DECIMAL(10,2)'}
)
WHERE region = 'East';

5.3 Use Parquet Format

Parquet is a columnar storage format that pairs perfectly with DuckDB:

-- CSV → Parquet (one-time investment, lifelong benefit)
COPY tbl TO '/data/optimized.parquet' (FORMAT PARQUET);

-- Querying Parquet is 3-10x faster than CSV
SELECT region, count(*) 
FROM read_parquet('/data/optimized.parquet')
GROUP BY region;

-- Parquet supports predicate pushdown
SELECT * FROM read_parquet('/data/optimized.parquet')
WHERE order_date BETWEEN '2025-06-01' AND '2025-06-30';

5.4 Batch Processing for Large Datasets

For datasets exceeding memory (50GB+):

-- Process in quarterly batches
CREATE TABLE result AS
SELECT * FROM read_csv_auto('/data/huge_dataset.csv')
WHERE order_date >= '2025-01-01' 
 AND order_date < '2025-04-01';

-- Merge batches with UNION ALL
CREATE TABLE yearly_result AS
SELECT * FROM read_csv_auto('/data/huge_dataset.csv')
WHERE order_date >= '2025-01-01' AND order_date < '2025-04-01'
UNION ALL
SELECT * FROM read_csv_auto('/data/huge_dataset.csv')
WHERE order_date >= '2025-04-01' AND order_date < '2025-07-01'
UNION ALL
SELECT * FROM read_csv_auto('/data/huge_dataset.csv')
WHERE order_date >= '2025-07-01' AND order_date < '2025-10-01'
UNION ALL
SELECT * FROM read_csv_auto('/data/huge_dataset.csv')
WHERE order_date >= '2025-10-01' AND order_date < '2026-01-01';

5.5 Use Views Instead of Temporary Tables

-- Create a view that reads files on-the-fly
CREATE VIEW sales_view AS
SELECT * FROM read_csv('/data/sales_*.csv', header = true, union_by_name = true);

-- Query the view — DuckDB reads files at query time
SELECT region, sum(total_amount) 
FROM sales_view
GROUP BY region;
-- Zero storage cost, but re-reads files on each query

6. DuckDB vs Traditional Big Data Tools

Dimension	DuckDB	Pandas	Spark	ClickHouse
Data range	1 MB - 100 GB	1 MB - 10 GB	10 GB - PB	10 GB - TB
Setup complexity	Download 1 binary	pip install	Needs cluster	Needs server
SQL support	Full SQL	Weak	Full SQL	Full SQL
Learning curve	Low (know SQL?)	Medium (Python)	High	Medium
Startup time	Milliseconds	Seconds	Minutes	Seconds
Single-node perf	Excellent	Good (memory-bound)	Poor (dist overhead)	Excellent
Scalability	Single-node multi-core	Single-thread (default)	Multi-node	Multi-node
Cost	Free	Free	Cluster hardware	Free/Enterprise

When to use what?

< 100M rows, single-machine analytics → DuckDB (first choice)
Ad-hoc data exploration → DuckDB (zero config)
ETL intermediate processing → DuckDB (embeddable)
Distributed PB-scale processing → Spark
Real-time high concurrency → ClickHouse
Quick exploration < 10GB → Pandas works, but DuckDB is faster

7. Benchmark: Processing 1 Million Rows

Tested on MacBook Pro M3 (16GB RAM):

Operation	DuckDB	Pandas	Speedup
CSV read	0.4 sec	12.3 sec	30x
GROUP BY (5 columns)	0.3 sec	2.1 sec	7x
WHERE filter	0.1 sec	0.8 sec	8x
JOIN two million-row tables	0.8 sec	5.4 sec	6.75x
Window function (moving avg)	0.6 sec	3.2 sec	5.3x
Export to Parquet	1.1 sec	8.6 sec	7.8x
Full workflow	3.3 sec	32.4 sec	~10x

8. Complete Automated Report Script

This Python script runs every Monday to generate a weekly sales report:

#!/usr/bin/env python3
"""
weekly_sales_report.py - DuckDB-powered automated weekly report generator
Usage: python3 weekly_sales_report.py
"""
import duckdb
import datetime

today = datetime.date.today()
monday = today - datetime.timedelta(days=today.weekday())
last_monday = monday - datetime.timedelta(days=7)

# Connect to DuckDB (in-memory mode)
conn = duckdb.connect()

# Install extensions
conn.execute("INSTALL httpfs; LOAD httpfs;")

print(f"📊 Generating weekly report: {last_monday} → {monday}")

# Read and aggregate this week's data
result = conn.execute(f"""
 SELECT 
 product_category,
 region,
 count(*) AS orders,
 count(DISTINCT customer_id) AS customers,
 round(sum(total_amount), 2) AS revenue,
 round(avg(total_amount), 2) AS avg_order_value
 FROM read_csv('/data/sales_*.csv', header = true, union_by_name = true)
 WHERE order_date >= '{last_monday}' 
 AND order_date < '{monday}'
 GROUP BY product_category, region
 ORDER BY revenue DESC
""").fetchdf()

# Export as CSV (Excel-compatible)
result.to_csv('/tmp/weekly_report.csv', index=False)
print(f"✅ Report generated: /tmp/weekly_report.csv")
print(f"📈 {len(result)} rows total")

# Regional rankings
print("\n🏆 Regional Rankings:")
region_stats = conn.execute(f"""
 SELECT region, round(sum(total_amount), 2) AS revenue
 FROM read_csv('/data/sales_*.csv', header = true, union_by_name = true)
 WHERE order_date >= '{last_monday}' AND order_date < '{monday}'
 GROUP BY region ORDER BY revenue DESC
""").fetchdf()
print(region_stats.to_string(index=False))

conn.close()

9. Troubleshooting FAQ

Q1: My query is slow — what can I do?

-- Check the query plan
EXPLAIN ANALYZE SELECT ...;

-- Verify column types are correct
-- Avoid VARCHAR for numeric columns
SELECT typeof(column_name) FROM read_csv(...);

-- Use the right file format
-- CSV < Parquet < DuckDB internal table (fastest)

Q2: Running out of memory?

-- Set a temp directory for spill-to-disk
SET temp_directory = '/disk/tmp';

-- Cap memory usage
SET memory_limit = '2GB';

-- Use streaming (don't cache intermediate results)
SELECT * FROM read_csv('huge.csv') 
WHERE ... -- predicate pushdown, filter early
;

Q3: What about datasets over 100GB?

Upgrade RAM (DuckDB doesn’t support distributed mode natively)
Partition data + process in DuckDB shards
Consider migrating to ClickHouse/Doris for distributed workloads
Use DuckDB’s spill-to-disk (performance degrades but stays functional)

10. Monetization Strategies

💰 Strategy 1: Data Analytics Service ($500-2,000/month)

Target clients: Small e-commerce stores, local retail chains, import/export traders

Services:

Process client sales/inventory/financial data using DuckDB
Generate weekly/monthly automated analytics reports (PDF/Excel)
Build lightweight dashboards (DuckDB + Streamlit)

Delivery checklist:

Data ingestion (CSV/Excel/database connection)
DuckDB aggregation script configuration
Automated report generation pipeline
Anomaly alerting
Monthly business review reports

💰 Strategy 2: Corporate Training ($800-3,000/session)

Topic: “DuckDB + SQL: 100x Efficiency Boost for Data Teams”

Audience: Data analysts, operations teams, junior data engineers

Curriculum:

DuckDB installation & basic SQL
Processing millions of CSV rows hands-on
Parquet format & performance optimization
Building automated report pipelines
Integration with Python/BI tools

💰 Strategy 3: Data Migration Consulting ($1,000-5,000/project)

Target clients: Teams migrating from Excel/Access to modern data analytics

Services:

Audit existing data processing workflows
Design DuckDB-centric lightweight data pipelines
Migrate existing Pandas/Python scripts to DuckDB SQL
Write migration documentation & operations manual

💰 Strategy 4: SaaS Tool — DuckDB-Powered Lightweight BI

Product concept:

Users upload CSVs → DuckDB processes in the background → Interactive frontend charts
Core value: No servers needed, sub-second response, zero ops
Pricing: Free tier (1M rows/month) + Paid (unlimited)
Positioning: The Excel + Python replacement — no, really this time

11. Summary

DuckDB redefines what “personal-scale big data processing” means. You don’t need a Hadoop cluster, a Spark certification, or a DevOps team — just the DuckDB binary and the ability to write SQL.

For million-to-ten-million row processing tasks, DuckDB delivers performance comparable to large distributed systems — without the complexity. It’s particularly suited for:

Individual analysts and independent developers
SME data teams
Data consultants and freelancers
As an intermediate processing engine in data pipelines

Your next steps:

Download DuckDB: brew install duckdb or pip install duckdb
Try it on your largest CSV: duckdb -c "SELECT count(*) FROM read_csv('your_largest_file.csv')"
Experience the thrill of sub-second responses on million-row data

References

DuckDB Reads Excel Natively: Replace Pandas, 10 Seconds for 10 Files

Mon, 11 May 2026 00:00:00 +0000

1. The Excel Data Nightmare Every Professional Knows

You work in finance, operations, or sales support. Monday morning arrives, and your boss messages:

“I need a Q1 sales summary from all regions — we have a meeting this afternoon.”

Sounds simple? But when you open your inbox to find 10 attachments, each with a different Excel format, your heart sinks:

Some sheets are named “Sheet1”, others “Sales Data” or “销售数据”
Some columns are called “revenue”, others “amount” or “销售额”
Some files are .xlsx, others are ancient .xls
Each file ranges from 10MB to 50MB

Traditionally, you have a few options:

Manual Copy-Paste: Open 10 files, copy and paste one by one — takes 30-60 minutes and error-prone
Python Pandas: Write read_excel() + concat() + groupby(), but it eats 4-8GB of RAM, and large files cause OOM crashes
VBA Macros: High maintenance cost, breaks when switching computers
Paid BI Tools: Tableau / Power BI can do it, but licenses cost $70-100/user/month

Is there a simpler way?

The answer: DuckDB’s excel extension — read, transform, merge, and write Excel files with a single SQL query. No Python required, no Office installation needed, and memory usage is 1/40th of Pandas.

2. What Is the DuckDB Excel Extension?

The community-developed excel extension enables DuckDB to directly read and write Microsoft Excel (.xlsx) files as if they were CSV or Parquet files.

-- Install and load the extension
INSTALL excel;
LOAD excel;

-- Query an Excel file directly
SELECT * FROM 'sales_report.xlsx';

That’s it. DuckDB treats Excel files as standard tables. You can:

Read with SELECT
Write with INSERT/CREATE TABLE AS
Export with COPY ... TO
JOIN across multiple files
Use in any DuckDB environment (CLI, Python, R, Node.js)

3. Practical Scenario: Consolidating 10 Excel Files

Scenario Description

You’re an operations analyst at an e-commerce company. The sales team sent 10 regional reports (East, West, North, South, Central, Northeast, Northwest, Hong Kong/Macau/Taiwan, International, Online) for Q1 2026.

Each file has slightly different column names but the same three columns:

Column (may vary)	Meaning
`region` / `area` / `区域`	Region name
`sales_person` / `name` / `姓名`	Salesperson name
`revenue` / `amount` / `销售额`	Revenue (CNY)

The Pandas Era Approach

import pandas as pd
import os

files = ['east.xlsx', 'west.xlsx', 'north.xlsx', ...] # 10 files

# Read one by one, handling format differences
dfs = []
for f in files:
 df = pd.read_excel(f)
 # Standardize column names
 df.columns = ['region', 'sales_person', 'revenue']
 # Handle separators and nulls
 df['revenue'] = df['revenue'].replace(',', '', regex=True).astype(float)
 dfs.append(df)

# Merge
result = pd.concat(dfs, ignore_index=True)

# Aggregate
summary = result.groupby('region')['revenue'].agg(['sum', 'count', 'mean'])

# Export
summary.to_excel('q1_sales_summary.xlsx')

This code looks reasonable, but in practice:

10 x 30MB Excel files → 4-8GB of RAM (Pandas loads everything at once)
Files over 50MB → near-certain OOM crash
Execution time: 3-8 minutes
Code: 30+ lines

The DuckDB Solution

-- 1. Load the excel extension
INSTALL excel;
LOAD excel;

-- 2. Read and aggregate in one shot
CREATE TABLE q1_summary AS
SELECT 
 region,
 SUM(revenue) AS total_revenue,
 COUNT(DISTINCT sales_person) AS salesperson_count,
 AVG(revenue) AS avg_per_person
FROM (
 SELECT * FROM 'east.xlsx'
 UNION ALL
 SELECT * FROM 'west.xlsx'
 UNION ALL
 SELECT * FROM 'north.xlsx'
 UNION ALL
 SELECT * FROM 'south.xlsx'
 UNION ALL
 SELECT * FROM 'central.xlsx'
 UNION ALL
 SELECT * FROM 'northeast.xlsx'
 UNION ALL
 SELECT * FROM 'northwest.xlsx'
 UNION ALL
 SELECT * FROM 'hkmacau.xlsx'
 UNION ALL
 SELECT * FROM 'international.xlsx'
 UNION ALL
 SELECT * FROM 'online.xlsx'
)
GROUP BY region
ORDER BY total_revenue DESC;

-- 3. Export back to Excel
COPY q1_summary TO 'q1_sales_summary.xlsx' (FORMAT excel);

Pro tip: Use glob patterns when filenames follow a convention

-- Match all regional files with a wildcard
CREATE TABLE all_sales AS
SELECT * FROM read_csv('region_*.xlsx', auto_detect=true);

-- One-step aggregation
SELECT region, SUM(revenue) AS total_revenue
FROM all_sales
GROUP BY region
ORDER BY total_revenue DESC;

Handling inconsistent column names is straightforward:

-- Standardize column names before UNION
SELECT region, sales_person, revenue FROM (
 SELECT region, sales_person, revenue FROM 'east.xlsx'
 UNION ALL
 SELECT area AS region, name AS sales_person, amount AS revenue FROM 'west.xlsx'
 -- ... normalize each file as needed
);

Performance Comparison

Dimension	Pandas (Traditional)	DuckDB Excel Extension
Memory Usage	4-8 GB	~100 MB
Execution Time	3-8 minutes	8-15 seconds
Lines of Code	30+ lines	3-5 lines SQL
File Size Limit	< 100MB (OOM beyond)	GB-scale, no pressure
Excel Version Support	Depends on openpyxl/xlrd	Native .xlsx support
Dependencies	pandas + openpyxl + xlrd	DuckDB + excel extension
Streaming	❌ Full load in memory	✅ Vectorized streaming
Cross-file JOIN	Merge then process	Native SQL JOIN
Export Formats	Excel/CSV	Excel/CSV/Parquet/JSON

4. More Practical Scenarios

Scenario 2: Cross-File VLOOKUP

You have an orders.xlsx file and a customer_tiers.xlsx file. You need to tag each order with the customer’s tier:

SELECT 
 o.order_id,
 o.customer_id,
 o.amount,
 c.customer_tier,
 CASE 
 WHEN c.customer_tier = 'VIP' THEN o.amount * 0.85
 WHEN c.customer_tier = 'Gold' THEN o.amount * 0.90
 ELSE o.amount * 0.95
 END AS discounted_amount
FROM 'orders.xlsx' o
JOIN 'customer_tiers.xlsx' c ON o.customer_id = c.customer_id
ORDER BY o.amount DESC;

Before: VLOOKUP formulas on 100K rows would freeze Excel. After: DuckDB SQL JOIN in 0.5 seconds.

Scenario 3: Excel Data Cleaning + Database Write

Marketing receives daily Excel reports from ad channels and needs to clean and write them to PostgreSQL:

-- Read Excel, clean, write to PostgreSQL
INSTALL postgres_scanner;
LOAD postgres_scanner;

ATTACH 'host=db.example.com dbname=marketing' AS pg_db (TYPE postgres);

CREATE TABLE pg_db.daily_report AS
SELECT 
 date,
 channel,
 UPPER(TRIM(channel_name)) AS channel_name, -- normalize case
 CAST(REPLACE(spend, ',', '') AS DOUBLE) AS spend, -- clean numbers
 CAST(REPLACE(impressions, ',', '') AS INTEGER) AS impressions,
 CAST(REPLACE(clicks, ',', '') AS INTEGER) AS clicks,
 spend / NULLIF(clicks, 0) AS cpc
FROM 'daily_ad_report_2026-05-11.xlsx'
WHERE date IS NOT NULL; -- filter empty rows

Scenario 4: Excel to Parquet Migration

If your team is transitioning to more efficient data formats:

-- Excel → Parquet (5-10x compression, 100x faster queries)
COPY (
 SELECT * FROM 'historical_data.xlsx'
) TO 'historical_data.parquet' (FORMAT parquet, COMPRESSION zstd);

-- Future queries use Parquet
SELECT year, SUM(revenue) 
FROM 'historical_data.parquet' 
GROUP BY year;

5. Deep Comparison with Traditional Excel Tools

Tool	Best For	Memory Efficiency	Learning Curve	Speed (10 files × 30MB)	Large File Support	Automation	Cost
Excel Itself	Ad-hoc analysis, manual ops	Poor (>100K rows chokes)	✅ Zero	5-10 min (manual)	❌	❌ Needs VBA	$159/yr
Pandas	Python data science ecosystem	Poor (full load)	Moderate	3-8 min	Limited (<100MB)	✅ Python	Free
openpyxl	Fine-grained Excel ops	Very poor	Moderate	5-15 min	❌ <50MB	✅ Python	Free
VBA Macros	In-Excel automation	Good (row-by-row)	High	2-10 min	✅ Large	✅ In-Excel	Built-in
Power Query	Power BI ecosystem	Moderate	Moderate	1-3 min	✅	✅	Free+
DuckDB	Analytical data processing	✅ Excellent	⭐ Low (SQL)	8-15 sec	✅ GB-scale	✅ Any language	Free
Tableau Prep	Enterprise data prep	Good	High	1-2 min	✅	✅	$70/mo

6. Advanced Techniques

6.1 Using DuckDB with Excel in Python

You don’t have to abandon Python — instead, use DuckDB as the computation engine:

import duckdb
import pandas as pd

# DuckDB reads Excel directly, returns DataFrame
conn = duckdb.connect()
result = conn.execute("""
 SELECT 
 region,
 SUM(revenue) as total_revenue,
 COUNT(*) as order_count
 FROM 'sales.xlsx'
 GROUP BY region
""").fetchdf()

# The result is a Pandas DataFrame — visualize it
import matplotlib.pyplot as plt
result.plot.bar(x='region', y='total_revenue')
plt.show()

6.2 Command-Line One-Liners

# Install DuckDB CLI
curl -fsSL https://install.duckdb.org | sh

# One-liner: summarize Excel
duckdb -c "
LOAD excel;
SELECT region, SUM(revenue) FROM 'sales.xlsx' GROUP BY region;
"

# One-liner: convert Excel to Parquet
duckdb -c "
LOAD excel;
COPY (SELECT * FROM 'data.xlsx') TO 'data.parquet' (FORMAT parquet);
"

6.3 Scheduled Automation

# crontab -e
# Auto-consolidate Excel reports every Monday at 8:00 AM
0 8 * * 1 cd /home/reports && duckdb -c "
LOAD excel;
COPY (
 SELECT region, SUM(revenue) AS total
 FROM read_csv('region_*.xlsx', auto_detect=true)
 GROUP BY region
) TO 'weekly_summary.xlsx' (FORMAT excel);
"

7. Monetization Strategies

Mastering DuckDB + Excel processing opens up several revenue streams:

💰 Strategy 1: Excel Report Automation Service ($49-199/month)

SMBs (especially finance and operations teams) manually consolidate Excel files every week. You can:

Build DuckDB-powered automated report pipelines for clients
Clients drop Excel files into a folder → DuckDB auto-processes → clean output delivered
Pricing: $49/mo (basic) / $99/mo (cross-system integration) / $199/mo (with visualization dashboard)
Acquisition: Share Excel automation tutorials on LinkedIn/Medium → convert to clients via DMs

💰 Strategy 2: Excel → Data Warehouse Migration ($500-2,000/project)

Many companies want to migrate from Excel to proper data warehouses but are put off by ETL costs.

Use DuckDB as the intermediary engine: Excel → Parquet → DuckDB → BI Tool
Pricing: $500 (single migration) / $2,000 (includes training + automated pipeline)
Target: SMBs, startups, IT departments at traditional companies

💰 Strategy 3: SaaS - Excel Data Cleaning Platform

Build a SaaS product:

Users upload Excel → DuckDB processes in the cloud → outputs clean CSV/Parquet/Excel
Features: dedup, format standardization, cross-file JOIN, anomaly detection
Pricing: 7-day free trial → $29/mo (200MB) / $79/mo (2GB)
Tech stack: DuckDB (engine) + Streamlit/Gradio (frontend) + simple payment integration

💰 Strategy 4: Corporate Training

Many data analysts want to learn DuckDB but don’t know where to start.

Course: “DuckDB in Practice: From Excel to Data Warehouse”
Content: Excel processing + cross-database JOIN + performance optimization + real-world cases
Pricing: $99 (recorded online course) / $1,000-3,000/day (corporate training)
Platform: Udemy / LinkedIn Learning / your own site

💰 Strategy 5: Premium Templates & Scripts

DuckDB + Excel automation templates (SQL scripts + cron configurations)
Pricing: $29/set
Content: 10 ready-to-use SQL scripts covering common scenarios
Distribution: GitHub Sponsors / Gumroad / Gumroad

8. Summary

The DuckDB excel extension lets you read and write .xlsx files with SQL — 10 files in 10 seconds, 40x less memory, 90% less code. Switching from Pandas to DuckDB for Excel processing is the highest-ROI skill upgrade for data analysts in 2026.

Try it now:

# macOS / Linux
curl -fsSL https://install.duckdb.org | sh

# Launch DuckDB CLI
duckdb
# Inside the CLI
INSTALL excel;
LOAD excel;
SELECT * FROM 'your_file.xlsx';

Subscribe to DuckDB Lab (duckdblab.org) for weekly DuckDB tutorials, performance optimization tips, and monetization strategies.

Multi-CSV File Merging: Real-World Multi-Store Sales Analysis with DuckDB

Mon, 11 May 2026 00:00:00 +0000

Introduction

“Can you analyze the sales data from all our stores?”

Behind this innocent request lies a mess: each store has its own CSV file, generated daily, with inconsistent filenames, mixed column names, and varying encodings. The traditional approach — writing a Python script to iterate through directories, read each file, concatenate DataFrames, handle encoding issues — is slow, fragile, and error-prone.

This is where DuckDB shines. In this tutorial, we’ll walk through a multi-store sales data consolidation project, from scattered CSV files to a multi-dimensional analytical report — using nothing but SQL.

The Scenario

A retail chain has 5 stores. Each store generates a daily sales CSV. The file structure looks like this:

data/
├── store_001_daily_20260501.csv
├── store_001_daily_20260502.csv
├── store_002_daily_20260501.csv
├── store_002_daily_20260502.csv
├── store_003_daily_20260501.csv
├── store_003_daily_20260502.csv
├── store_004_daily_20260501.csv
├── store_004_daily_20260502.csv
├── store_005_daily_20260501.csv
├── store_005_daily_20260502.csv

Each CSV has the same structure with Chinese column names (a common real-world complication):

订单号,商品名称,单价,数量,金额,销售日期,收银员
ORD001,Latte,32.00,2,64.00,2026-05-01,Zhang San
ORD002,American Coffee,25.00,1,25.00,2026-05-01,Li Si

Step 1: Read All CSVs with a Single Wildcard

Traditional approach: write a Python script to walk directories, read each file one by one, concatenate DataFrames, and handle type inference. DuckDB approach: one line of SQL.

-- Read all CSV files using a glob pattern — DuckDB auto-infers the schema
CREATE TABLE raw_sales AS
SELECT * FROM read_csv_auto('data/*.csv');

-- Check the merged result
SELECT COUNT(*) AS total_rows, COUNT(DISTINCT 订单号) AS total_orders FROM raw_sales;

read_csv_auto is DuckDB’s killer feature for CSV wrangling:

The * wildcard matches all CSV files in the directory
Automatically infers column names, data types, and delimiters
Supports recursive glob patterns: **/*.csv matches subdirectories
Use union_by_name=true for files with slightly different column structures

-- More robust: auto-merge by column name
CREATE TABLE raw_sales AS
SELECT * FROM read_csv_auto('data/*.csv', union_by_name=true);

-- Inspect the auto-inferred schema
DESCRIBE raw_sales;

Step 2: Extract Store Info and Dates from Filenames

The store ID and date are encoded in the filenames. DuckDB’s regex and file-path functions make extraction trivial:

-- Extract metadata from filenames
CREATE TABLE sales_with_meta AS
SELECT
 filename,
 regexp_extract(filename, 'store_(\d+)', 1) AS store_id,
 regexp_extract(filename, '(\d{8})\.csv', 1) AS date_string,
 *
FROM read_csv_auto('data/*.csv', filename=true, union_by_name=true);

-- Convert date strings to proper DATE type
CREATE TABLE sales_clean AS
SELECT
 store_id,
 strptime(date_string, '%Y%m%d')::DATE AS sale_date,
 订单号 AS order_id,
 商品名称 AS product_name,
 单价 AS unit_price,
 数量 AS quantity,
 金额 AS amount,
 收银员 AS cashier
FROM sales_with_meta;

The filename=true parameter adds a filename column that records which file each row came from — invaluable for debugging multi-file merges.

Step 3: Analysis with Chinese Column Names

DuckDB supports Chinese (and any Unicode) column names natively. No configuration, no quoting tricks needed:

-- Sales ranking by store
SELECT
 store_id,
 SUM(amount) AS total_revenue,
 COUNT(DISTINCT order_id) AS order_count,
 SUM(quantity) AS total_units,
 ROUND(AVG(amount), 2) AS avg_order_value
FROM sales_clean
GROUP BY store_id
ORDER BY total_revenue DESC;

-- Top 10 best-selling products
SELECT
 product_name,
 SUM(quantity) AS total_units_sold,
 SUM(amount) AS total_revenue,
 COUNT(DISTINCT store_id) AS stores_stocked
FROM sales_clean
GROUP BY product_name
ORDER BY total_revenue DESC
LIMIT 10;

-- Cashier performance ranking
SELECT
 cashier,
 store_id,
 COUNT(*) AS transactions_processed,
 SUM(amount) AS total_handled
FROM sales_clean
GROUP BY cashier, store_id
ORDER BY total_handled DESC;

Step 4: Time-Based Aggregation with strftime

Time-based aggregation is the backbone of sales analysis. DuckDB’s strftime function provides Python-style date formatting:

-- Daily aggregation
SELECT
 strftime(sale_date, '%Y-%m-%d') AS day,
 SUM(amount) AS daily_revenue
FROM sales_clean
GROUP BY day
ORDER BY day;

-- Weekly aggregation
SELECT
 strftime(sale_date, '%Y-W%W') AS week,
 SUM(amount) AS weekly_revenue,
 COUNT(DISTINCT sale_date) AS operating_days
FROM sales_clean
GROUP BY week
ORDER BY week;

-- Monthly aggregation
SELECT
 strftime(sale_date, '%Y-%m') AS month,
 SUM(amount) AS monthly_revenue,
 SUM(quantity) AS monthly_units,
 ROUND(AVG(amount), 2) AS avg_daily_revenue
FROM sales_clean
GROUP BY month
ORDER BY month;

-- Time-of-day analysis (sales peak hours)
SELECT
 CASE
 WHEN strftime(sale_date, '%H') BETWEEN '06' AND '09' THEN 'Breakfast'
 WHEN strftime(sale_date, '%H') BETWEEN '10' AND '13' THEN 'Lunch'
 WHEN strftime(sale_date, '%H') BETWEEN '14' AND '17' THEN 'Afternoon Tea'
 ELSE 'Dinner'
 END AS time_period,
 SUM(amount) AS revenue
FROM sales_clean
GROUP BY time_period
ORDER BY revenue DESC;

Common strftime format specifiers:

Format	Meaning	Example
`%Y`	4-digit year	2026
`%m`	2-digit month	05
`%d`	2-digit day	11
`%W`	Week of year	19
`%w`	Day of week (0-6)	1
`%H`	Hour (00-23)	14

Step 5: Export to Parquet

Once your analysis is complete, export the results as Parquet — 10x faster reads, 5x smaller files, and native columnar storage:

-- Export cleaned data to Parquet
COPY sales_clean TO 'output/sales_clean.parquet'
(FORMAT PARQUET, COMPRESSION ZSTD);

-- Export an analytical report
COPY (
 SELECT
 store_id,
 strftime(sale_date, '%Y-%m') AS month,
 strftime(sale_date, '%W') AS week_number,
 product_name,
 SUM(quantity) AS total_units,
 SUM(amount) AS total_revenue
 FROM sales_clean
 GROUP BY store_id, month, week_number, product_name
 ORDER BY store_id, month, week_number, total_revenue DESC
) TO 'output/daily_report.parquet'
(FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 100000);

-- Query Parquet files directly
SELECT store_id, SUM(total_revenue) AS total
FROM read_parquet('output/*.parquet')
GROUP BY store_id
ORDER BY store_id;

Why Parquet matters:

Amazing compression: ZSTD-compressed Parquet is typically only 20% of the CSV size
Columnar storage: Only reads the columns you query — dramatically faster I/O
Self-describing schema: Type information is embedded in the file, no DDL needed
DuckDB native optimization: Projection pushdown, predicate pushdown, late materialization

Complete Workflow Script

Here’s the entire pipeline as a single repeatable SQL script:

-- merge_analysis.sql
-- Multi-store sales data consolidation & analysis — from CSV to Parquet

-- 1. Import — read all CSVs with wildcards
CREATE TABLE raw AS
SELECT * FROM read_csv_auto('data/*.csv', filename=true, union_by_name=true);

-- 2. Clean & transform — extract metadata from filenames
CREATE TABLE clean AS
SELECT
 regexp_extract(filename, 'store_(\d+)', 1) AS store_id,
 strptime(regexp_extract(filename, '(\d{8})\.csv', 1), '%Y%m%d') AS sale_date,
 订单号 AS order_id,
 商品名称 AS product_name,
 单价 AS unit_price,
 数量 AS quantity,
 金额 AS amount,
 收银员 AS cashier
FROM raw;

-- 3. Aggregate — monthly store performance
CREATE TABLE monthly_summary AS
SELECT
 store_id,
 strftime(sale_date, '%Y-%m') AS month,
 COUNT(DISTINCT order_id) AS order_count,
 SUM(amount) AS revenue,
 SUM(quantity) AS units_sold,
 ROUND(AVG(amount), 2) AS avg_order_value
FROM clean
GROUP BY store_id, month
ORDER BY store_id, month;

-- 4. Export — save as compressed Parquet
COPY clean TO 'output/clean_data.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);
COPY monthly_summary TO 'output/monthly_summary.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);

-- 5. Quick validation
SELECT 'Total Rows' AS metric, COUNT(*)::VARCHAR AS value FROM clean
UNION ALL
SELECT 'Total Stores', COUNT(DISTINCT store_id)::VARCHAR FROM clean
UNION ALL
SELECT 'Date Range', MIN(sale_date)::VARCHAR || ' ~ ' || MAX(sale_date)::VARCHAR FROM clean;

Run it with:

duckdb < merge_analysis.sql
# Or interactively:
duckdb -c ".read merge_analysis.sql"

Monetization SOP: From Technical Skill to Revenue

Being able to write SQL is one thing. Being able to package data integration as a deliverable service is where the real value lies.

Pricing Strategy

Tier	Deliverable	Price	Target Client
Basic	One-time CSV merge + 3 summary tables	$100–200	Small businesses (1–5 stores)
Standard	Multi-source merge + weekly/monthly report templates + Parquet export	$300–500	Mid-size chains (5–20 stores)
Premium	Fully automated pipeline + custom dashboard + ongoing maintenance	$800–2,000/month	Large chains (20+ stores)

Customer Acquisition Channels

Targeted outreach
- Write industry-specific case studies (“How a coffee chain saved 20 hours/week on reporting”)
- Open-source a basic version on GitHub with your contact info in the README
- Post on Hacker News / Lobsters when you hit interesting performance numbers
Partner channels
- Partner with POS system vendors and ERP implementation firms
- Collaborate with accounting/financial services firms (they have the client base)
- Join retail/restaurant industry communities and forums
Content marketing
- Write niche blog posts (e.g., “Data Stack for a 10-Store Retail Chain: Under $100/mo”)
- Create short demo videos showing before/after
- Offer free 30-minute data health check consultations

Delivery Checklist

## Delivery Checklist
1. ✅ SQL automation script (merge_analysis.sql)
2. ✅ Data dictionary (PDF/Excel)
3. ✅ Cleaned dataset (Parquet format)
4. ✅ Monthly/weekly report templates
5. ✅ README with operation guide
6. ✅ 1-week free remote support

Include a Data Health Report with every delivery:

Completeness: null values, missing dates, outlier detection
Consistency: duplicate records, order ID conflicts, referential integrity
Performance: current pipeline runtime, optimization recommendations

Upsell Opportunities

Service	Description	Price Range
Real-time Dashboard	Streamlit/Grafana live dashboard for store managers	$1,000–3,000
AI-generated Reports	Weekly narrative summaries via LLM integration	$500–1,000/mo
Anomaly Alerts	Automated alerts for unusual sales drops or spikes	$200–500/mo
Data API	Standardized API for POS system integration	$300–800/mo

Handling Common Objections

“We can just use Excel.” Response: Excel chokes around 100K rows and can’t auto-merge daily files. This solution reads and aggregates 1M+ rows in under 5 seconds, with automatic daily updates via cron.

“It’s too expensive.” Response: Let’s quantify your current time cost. If you spend 30 minutes/day manually merging spreadsheets, that’s 15 hours/month. Even at minimum wage, you’re spending more on manual work in 3 months than this solution costs.

“We don’t really need this.” Response: Let me do a free 30-minute data health check. I’ll merge your files and show you insights you can’t get from isolated spreadsheets — which store is most profitable, which products are cannibalizing each other, and where you’re leaving money on the table.

FAQ

Q1: What about file encoding issues?

-- Specify UTF-8 or other encodings
SELECT * FROM read_csv_auto('data/*.csv', encoding='utf-8');
SELECT * FROM read_csv_auto('data/*.csv', encoding='latin1');

Q2: Column names have inconsistent whitespace?

-- Normalize column names automatically
SELECT * FROM read_csv_auto('data/*.csv', normalize_names=true);

Q3: Too many files, not enough memory?

-- Read in batches, append to table
CREATE TABLE sales AS
SELECT * FROM read_csv_auto('data/2026-01/*.csv');
INSERT INTO sales SELECT * FROM read_csv_auto('data/2026-02/*.csv');
-- Repeat per month...

Q4: How to automate daily CSV ingestion?

Set up a cron job:

# crontab -e
# Run every day at 2 AM
0 2 * * * cd /path/to/project && duckdb < merge_analysis.sql

Conclusion

From scattered multi-store CSV files to queryable, compressed Parquet datasets — DuckDB turns what used to be a tedious Python scripting task into a handful of clean SQL statements. The core takeaways:

read_csv_auto with glob + filename=true: One-shot reads, provenance tracking
Regex extraction + strptime: Reverse-engineer metadata from filenames
Native Unicode column name support: No friction for international teams
strftime time aggregation: Flexible time-based analytics
Parquet export: 10x faster downstream analysis

This workflow isn’t limited to retail sales. Any scenario involving multiple files, multiple sources, and recurring consolidation — multi-warehouse inventory, multi-server logs, multi-location foot traffic — benefits from the same pattern.

Try it yourself: point DuckDB at a directory of CSVs, write a single read_csv_auto('*.csv') query, and see how far SQL alone can take you.

One SQL to Query MySQL, PostgreSQL and CSV: DuckDB Cross-Database JOINs in Action

Sat, 09 May 2026 00:00:00 +0000

1. The Data Silo Nightmare Every Analyst Knows

You work at an e-commerce company. The boss asks: “Among the top 100 products by last month’s sales, what’s the repeat customer rate?”

Where is the data?

Orders: in the MySQL transaction database, 7 tables sharded by month
Customer tags: in the PostgreSQL analytics database, marking new vs. returning customers
Product info: in an Excel/CSV file the operations team updates weekly (with changing column names)

The traditional approach? A painstaking three-step ordeal:

Step 1: Export sales from MySQL → run a query → save CSV (5 minutes)
Step 2: Export customer tags from PostgreSQL → run a query → save CSV (5 minutes)
Step 3: Merge three CSVs in Excel with VLOOKUP → hover, wait, pray (10 minutes, likely crash)
Step 4: Realize data is missing → re-export → re-VLOOKUP (double the pain)
Step 5: Boss says "add another dimension" → start from scratch (total meltdown)

Total time: 30 minutes to 1 hour. With large datasets, Excel crashes. And none of it is reusable — change the date range and you start over.

2. The DuckDB Solution: ATTACH + Cross-Database JOIN

DuckDB has a severely underrated feature: the ATTACH statement lets you mount external databases like mounting drives, then JOIN across data sources with plain SQL.

What does this mean? One SQL query across all three data sources — no exports, no merges, no VLOOKUP.

2.1 How ATTACH Works

-- Mount a SQLite database (simple example)
ATTACH 'path/to/file.db' AS my_db (TYPE SQLITE);

-- Now cross-database query is possible
SELECT *
FROM my_db.some_table AS a
JOIN main.public.another_table AS b ON a.id = b.id;

DuckDB supports ATTACH for these data sources:

Data Source	ATTACH Syntax	Type Identifier
SQLite	`ATTACH 'file.db' (TYPE SQLITE)`	SQLITE
MySQL	`ATTACH '' (TYPE MYSQL)`	MYSQL
PostgreSQL	`ATTACH 'pg_conn_str' (TYPE POSTGRES)`	POSTGRES
DuckDB native	`ATTACH 'data.duckdb'`	DUCKDB
Delta Lake	`ATTACH './delta_dir' (TYPE DELTA)`	DELTA

Note: Connecting to MySQL and PostgreSQL requires installing the corresponding extensions:
INSTALL mysql_scanner; LOAD mysql_scanner;
INSTALL postgres_scanner; LOAD postgres_scanner;

2.2 Full Walkthrough: E-Commerce Cross-Database Query

The script below simulates three data sources using DuckDB in-memory tables and CSV files — no real databases needed. Just copy and run.

#!/usr/bin/env python3
"""
DuckDB Cross-Database JOIN Demo
Scenario: E-commerce data across three sources, one SQL to unite them all

Prerequisites: pip install duckdb openpyxl
"""

import duckdb
import os

# ====== Step 1: Create Mock Data ======

# Mock MySQL orders table (CSV file)
orders_csv = """order_id,customer_id,product_id,amount,order_date
1001,201,5001,299.00,2026-05-01
1002,202,5002,159.00,2026-05-01
1003,201,5003,899.00,2026-05-02
1004,203,5001,299.00,2026-05-02
1005,204,5004,459.00,2026-05-03
1006,202,5002,159.00,2026-05-03
1007,205,5005,1299.00,2026-05-04
1008,203,5003,899.00,2026-05-04
1009,206,5001,299.00,2026-05-05
1010,201,5004,459.00,2026-05-05
"""

# Mock PostgreSQL users table (CSV file)
users_csv = """customer_id,name,city,member_level,register_date
201,Alice,Beijing,Gold,2025-01-15
202,Bob,Shanghai,Silver,2025-03-20
203,Carol,Guangzhou,Gold,2025-02-01
204,Dave,Shenzhen,Regular,2025-06-10
205,Eve,Hangzhou,Silver,2025-04-05
206,Frank,Chengdu,Regular,2025-08-15
"""

# Mock product catalog
products_csv = """product_id,product_name,category,unit_price,cost_price
5001,Wireless Earbuds,Electronics,299.00,180.00
5002,Insulated Mug,Home,159.00,80.00
5003,Smart Watch,Electronics,899.00,550.00
5004,Running Shoes,Apparel,459.00,280.00
5005,Tablet Stand,Electronics,1299.00,800.00
"""

# Write temp files
os.makedirs("day07_data", exist_ok=True)
with open("day07_data/orders.csv", "w") as f:
 f.write(orders_csv)
with open("day07_data/users.csv", "w") as f:
 f.write(users_csv)
with open("day07_data/products.csv", "w") as f:
 f.write(products_csv)

# ====== Step 2: Cross-Source JOIN with DuckDB ======

con = duckdb.connect()

# Mount CSV files as views — simulating "different data sources"
con.execute("""
 CREATE VIEW orders AS 
 SELECT * FROM read_csv_auto('day07_data/orders.csv')
""")
con.execute("""
 CREATE VIEW users AS 
 SELECT * FROM read_csv_auto('day07_data/users.csv')
""")
con.execute("""
 CREATE VIEW products AS 
 SELECT * FROM read_csv_auto('day07_data/products.csv')
""")

print("=" * 60)
print("📊 Cross-Source: What did Gold members buy?")
print("=" * 60)

# One SQL across three "data sources"
result = con.execute("""
 SELECT 
 u.name AS customer_name,
 u.city,
 u.member_level,
 p.product_name,
 p.category,
 o.amount AS unit_price,
 (o.amount - p.cost_price) AS gross_margin
 FROM orders o
 JOIN users u ON o.customer_id = u.customer_id
 JOIN products p ON o.product_id = p.product_id
 WHERE u.member_level IN ('Gold', 'Silver')
 ORDER BY o.amount DESC
""").fetchdf()

print(result.to_string(index=False))

print("\n" + "=" * 60)
print("💰 Daily Sales Summary (with Margins)")
print("=" * 60)

result2 = con.execute("""
 SELECT 
 o.order_date AS date,
 COUNT(DISTINCT o.order_id) AS orders,
 SUM(o.amount) AS revenue,
 SUM(o.amount - p.cost_price) AS gross_profit,
 ROUND(AVG(o.amount - p.cost_price), 2) AS avg_profit_per_order
 FROM orders o
 JOIN products p ON o.product_id = p.product_id
 GROUP BY o.order_date
 ORDER BY o.order_date
""").fetchdf()

print(result2.to_string(index=False))

print("\n" + "=" * 60)
print("🏆 Category Profitability Analysis")
print("=" * 60)

result3 = con.execute("""
 SELECT 
 p.category,
 COUNT(*) AS units_sold,
 SUM(o.amount) AS revenue,
 SUM(o.amount - p.cost_price) AS gross_profit,
 ROUND(AVG(o.amount - p.cost_price), 2) AS avg_profit,
 ROUND(SUM(o.amount - p.cost_price) / SUM(o.amount) * 100, 1) AS margin_pct
 FROM orders o
 JOIN products p ON o.product_id = p.product_id
 GROUP BY p.category
 ORDER BY gross_profit DESC
""").fetchdf()

print(result3.to_string(index=False))

# ====== Step 3: Export to Excel ======

try:
 con.execute("INSTALL spatial; LOAD spatial;")
 con.execute("""
 COPY (
 SELECT 
 o.order_id,
 u.name,
 u.city,
 u.member_level,
 p.product_name,
 p.category,
 o.amount,
 (o.amount - p.cost_price) AS profit
 FROM orders o
 JOIN users u ON o.customer_id = u.customer_id
 JOIN products p ON o.product_id = p.product_id
 ) TO 'day07_data/cross_source_report.xlsx' 
 WITH (FORMER XLSX);
 """)
 print("\n✅ Report exported: day07_data/cross_source_report.xlsx")
except Exception as e:
 print(f"\n⚠️ XLSX export needs spatial extension: {e}")
 print("Falling back to CSV:")
 con.execute("""
 COPY (
 SELECT 
 o.order_id,
 u.name,
 u.city,
 u.member_level,
 p.product_name,
 p.category,
 o.amount,
 (o.amount - p.cost_price) AS profit
 FROM orders o
 JOIN users u ON o.customer_id = u.customer_id
 JOIN products p ON o.product_id = p.product_id
 ) TO 'day07_data/cross_source_report.csv' (HEADER, DELIMITER ',');
 """)
 print("✅ Report exported: day07_data/cross_source_report.csv")

# ====== Step 4: Validation ======
print("\n" + "=" * 60)
print("📈 Validation: Are all orders matched to users and products?")
print("=" * 60)

validation = con.execute("""
 SELECT 
 'Total orders' AS metric, CAST(COUNT(*) AS VARCHAR) AS value FROM orders
 UNION ALL
 SELECT 'Matched to users', CAST(COUNT(*) AS VARCHAR) FROM orders o 
 JOIN users u ON o.customer_id = u.customer_id
 UNION ALL
 SELECT 'Matched to products', CAST(COUNT(*) AS VARCHAR) FROM orders o 
 JOIN products p ON o.product_id = p.product_id
 UNION ALL
 SELECT 'Fully matched', CAST(COUNT(*) AS VARCHAR) FROM orders o 
 JOIN users u ON o.customer_id = u.customer_id
 JOIN products p ON o.product_id = p.product_id
""").fetchdf()

print(validation.to_string(index=False))

con.close()
print("\n🎉 Cross-database JOIN demo complete!")

2.3 Real Environment: Connecting to MySQL + PostgreSQL

When you have actual MySQL and PostgreSQL databases, the script becomes:

-- Install extensions (one-time)
INSTALL mysql_scanner;
LOAD mysql_scanner;
INSTALL postgres_scanner;
LOAD postgres_scanner;

-- Mount MySQL orders database
ATTACH 'host=localhost port=3306 dbname=orders_db user=analyst password=xxx' 
 AS mysql_db (TYPE MYSQL);

-- Mount PostgreSQL user database
ATTACH 'host=localhost port=5432 dbname=users_db user=analyst password=xxx' 
 AS pg_db (TYPE POSTGRES);

-- Mount local product CSV
CREATE VIEW products AS 
SELECT * FROM read_csv_auto('products.csv');

-- One SQL across all three
SELECT 
 u.name,
 u.city,
 p.product_name,
 SUM(o.amount) AS total_spent
FROM mysql_db.orders AS o
JOIN pg_db.public.customers AS u ON o.customer_id = u.customer_id
JOIN products AS p ON o.product_id = p.product_id
WHERE o.order_date >= '2026-04-01'
GROUP BY u.name, u.city, p.product_name
ORDER BY total_spent DESC;

3. Performance Comparison: Traditional vs. DuckDB

Scenario	Traditional (Export + VLOOKUP)	DuckDB ATTACH
Cross MySQL + PG + CSV query	30 min ~ 1 hour	10 ~ 30 seconds
Opening 500K rows in Excel	Freeze/crash	Sub-second results
Changing analysis dimensions	Re-export + re-VLOOKUP	Edit one line of SQL
Scheduled report generation	Manual repeat every time	One-click script
10GB+ dataset	Excel/Pandas OOM	Streaming, no pressure
Learning curve	Know VLOOKUP	Know standard SQL
Dependencies	Excel + multiple database clients	Just DuckDB
Reusability	Essentially zero	SQL script forever reusable

Quantified impact:

A real case — an e-commerce company needing daily cross-source operations reports:

Traditional: Data analyst spends 40 minutes daily exporting, merging, checking
DuckDB: Write SQL once, daily execution 15 seconds
Monthly time saved: 40 min × 22 workdays = 880 minutes (14.7 hours)
Cost savings: At ¥50/hour, that’s ¥735/month per person

4. How ATTACH Works Under the Hood

Understanding ATTACH’s internals helps you design and optimize cross-database queries.

4.1 ATTACH Is NOT ETL

ATTACH doesn’t copy data into DuckDB — it creates an external table reference. When you query, DuckDB pushes query predicates down to the source database, only pulling back what’s needed.

-- DuckDB executes this GROUP BY on MySQL
-- Only the aggregated results come back, not a full table scan
SELECT customer_id, COUNT(*) 
FROM mysql_db.orders 
WHERE order_date >= '2026-01-01'
GROUP BY customer_id;

DuckDB’s optimizer automatically pushes WHERE, GROUP BY, LIMIT operations down to the source database, minimizing data transfer.

4.2 Key Performance Factors

Factor	Description	Optimization
Network latency	Cross-db queries depend on network	Deploy DuckDB near your databases
Pushdown optimization	Aggregations/filters execute remotely	Use WHERE to reduce data volume
Index utilization	Source indexes still work	Index your JOIN columns
Data volume	DuckDB doesn’t cache external tables	Use CREATE TABLE AS for repeated queries

4.3 Performance Tips

-- ❌ BAD: Pull everything, filter later
SELECT * FROM mysql_db.orders; -- Could be millions of rows

-- ✅ GOOD: Push down filter + limit
SELECT * FROM mysql_db.orders 
WHERE order_date >= '2026-05-01' 
LIMIT 1000;

-- ✅ BEST: Aggregate first, then JOIN
WITH daily_stats AS (
 -- This aggregation runs on MySQL
 SELECT customer_id, DATE(order_date) AS day, SUM(amount) AS daily_total
 FROM mysql_db.orders
 WHERE order_date >= '2026-04-01'
 GROUP BY customer_id, DATE(order_date)
)
-- Then JOIN with local data
SELECT u.name, d.day, d.daily_total
FROM daily_stats d
JOIN pg_db.public.customers u ON d.customer_id = u.customer_id
ORDER BY d.daily_total DESC
LIMIT 20;

5. Advanced Use Cases

5.1 Data Migration: Cross-Database Copy

-- MySQL → DuckDB local table (one-time snapshot)
CREATE TABLE local_orders AS 
SELECT * FROM mysql_db.orders 
WHERE order_date >= '2026-01-01';

-- DuckDB → PostgreSQL (write back)
CREATE TABLE pg_db.public.report AS 
SELECT * FROM local_analytics;

5.2 Multi-Environment Comparison

-- Mount production and staging simultaneously
ATTACH 'prod_conn' AS prod (TYPE POSTGRES);
ATTACH 'staging_conn' AS staging (TYPE POSTGRES);

-- Compare data differences
SELECT 
 COALESCE(p.order_id, s.order_id) AS order_id,
 p.amount AS prod_amount,
 s.amount AS staging_amount,
 (p.amount - s.amount) AS diff
FROM prod.public.orders p
FULL OUTER JOIN staging.public.orders s 
 ON p.order_id = s.order_id
WHERE p.amount IS DISTINCT FROM s.amount;

5.3 Automated Scheduled Reports

# Cron-based daily report generation
import duckdb

con = duckdb.connect()

# ATTACH data sources
con.execute("ATTACH '...' AS mysql_db (TYPE MYSQL)")
con.execute("ATTACH '...' AS pg_db (TYPE POSTGRES)")

# Generate daily report
con.execute("""
 COPY (
 -- cross-source query...
 ) TO '/tmp/daily_report.csv' (HEADER, DELIMITER ',');
""")

# Send email via SMTP (pseudo-code)
# send_email(to='boss@company.com', attachment='/tmp/daily_report.csv')
print("✅ Daily report generated")

6. Connection Guide & Troubleshooting

6.1 MySQL Extension Setup

# Ensure MySQL client libraries are installed
apt-get install -y default-libmysqlclient-dev # Ubuntu/Debian
# or
brew install mysql-client # macOS

Then in DuckDB:

INSTALL mysql_scanner;
LOAD mysql_scanner;

6.2 Connection String Formats

Source	Connection String Example
MySQL	`host=localhost port=3306 dbname=test user=root password=secret`
PostgreSQL	`host=localhost port=5432 dbname=test user=postgres password=secret`
SQLite	`./data.db` (just the file path)

6.3 Common Pitfalls

MySQL 8.0 auth: Use mysql_native_password or update DuckDB to v1.5+ for caching_sha2_password
PostgreSQL SSL: Add sslmode=require parameter
Large table JOINs: If both sides are large, pull the smaller table into DuckDB locally first
Character encoding: Default is UTF-8; MySQL latin1 may produce garbled text

7. Monetization Strategy

This skill has significant market value because 99% of companies have data silo problems.

Target Customers

SMBs: Data scattered across multiple systems, no dedicated data team
E-commerce: Order system + CRM + finance — all independent
Retail chains: Each store + HQ + supply chain — different data sources
Traditional enterprises in transition: Legacy databases coexisting with new systems

Pricing

Service	Price	Deliverables	Timeline
One-time data integration	¥2,000-5,000 ($280-700)	Cross-source query scripts + Excel report template	1-3 days
Monthly report automation	¥500-1,500/month ($70-210)	Scheduled cross-source business reports	Monthly
Data warehouse setup	¥5,000-15,000 ($700-2,100)	Complete ETL pipeline + analytics dashboard	1-2 weeks
Data integration training	¥1,500-3,000/session ($210-420)	Teach team to use DuckDB themselves	Half day

Competitive Landscape

Solution	Price	Strength	Weakness
Traditional ETL (Kettle/DataX)	Free but needs ops	Full-featured	Complex configuration, steep learning curve
Commercial BI (Tableau/Power BI)	¥500-2,000/month	Great visualization	Expensive, weak cross-source capability
Hiring a manual analyst	¥300-500/month	No thinking required	Unreliable, churn risk
DuckDB Solution (You)	¥2,000-5,000	One-time build, permanent use	Requires basic technical client

Client Acquisition

Freelance platforms (Upwork, Fiverr): Search “data integration,” “cross-database query,” “report automation” — pitch DuckDB solutions
Industry communities: Join e-commerce, retail, or operations groups — ask “how do you generate your reports?”
This blog post: Share it as proof of expertise. Every time someone reads it, they know you can solve their data silo problem.

Sales Pitch Template

“I see your company has data spread across different systems — you probably spend hours manually merging each report. I have a solution that connects all your data sources with one SQL query. After setup, you click once and get a complete report. Integration costs ¥3,000, then ¥800/month for automated monthly reports. Interested in a free data assessment first?”

8. Conclusion

DuckDB’s ATTACH + cross-database JOIN capability is this database’s most underrated killer feature. It frees data analysts from the primitive “export → merge → VLOOKUP” workflow, compressing cross-source query time from hours to seconds.

More importantly, enterprise data silos are a universal pain point, and DuckDB provides a low-cost, easy-to-learn, immediately effective solution. Master this skill, and you can solve real-world integration problems — at a clear, billable price.

What to do today:

Run the demo script above to understand ATTACH syntax
Find the most fragmented data scenario in your company
Connect it with DuckDB and show the result to your boss
Use this case study to win external clients

Cleanup:

rm -rf day07_data/