Featured image of post Build an Automated E-commerce Sales Analytics Engine with DuckDB

Build an Automated E-commerce Sales Analytics Engine with DuckDB

Build a production-ready e-commerce sales analytics engine using DuckDB. Replace manual Excel reports with秒级 queries featuring daily trends, Pareto analysis, and user LTV estimation.

Introduction

You’ve been hired by an e-commerce client who processes 100-500MB of order data daily using Excel spreadsheets, taking 2-3 hours per week. They’re unhappy. You’re unhappy.

The solution? Build an automated sales analytics engine with DuckDB — compressing analysis time from hours to seconds, while supporting real-time BI dashboards.

You can this in multiple ways:

  • Deliver as a SaaS tool to the client
  • Package as an embeddable module for other merchants
  • Build your portfolio with a showcase project

This article expands on the DuckDB Mining Lab channel post from August 30, 2026, into a complete tutorial with all runnable code.


Step 1: Generate Sample Data

In production, you’d connect to your client’s PostgreSQL / MySQL / CSV files. For demonstration, we’ll use DuckDB’s built-in CTEs and random functions to generate a realistic e-commerce dataset — no external downloads needed.

import duckdb
import pandas as pd

# Connect in-memory (use .duckdb file in production)
con = duckdb.connect(":memory:")

# Generate simulated order data using CTEs
con.execute("""
CREATE TABLE orders AS
WITH dates AS (
 SELECT DATE '2024-01-01' + n AS order_date
 FROM unnest(generate_series(0, 364)) AS t(n)
),
categories AS (
 SELECT unnest(['Electronics', 'Clothing', 'Home & Kitchen', 'Books', 'Sports']) AS cat
),
products AS (
 SELECT 
 unnest(generate_series(1, 500)) AS product_id,
 unnest(categories) AS category,
 FLOOR(RANDOM() * 500 + 10)::DOUBLE AS price,
 FLOOR(RANDOM() * 5 + 1) AS rating
 FROM categories
 LIMIT 500
),
orders_detail AS (
 SELECT 
 p.product_id,
 p.category,
 p.price * (1 + (RANDOM() - 0.5) * 0.2) AS final_price,
 d.order_date,
 FLOOR(RANDOM() * 5 + 1)::INTEGER AS quantity,
 FLOOR(RANDOM() * 1000000) AS order_id
 FROM dates d
 CROSS JOIN products p
 WHERE RANDOM() < 0.15
)
SELECT * FROM orders_detail
""")

count = con.execute("SELECT COUNT(*) FROM orders").fetchone()[0]
print(f"📦 Orders generated: {count:,}")
con.execute("CREATE INDEX idx_date ON orders(order_date)").fetchall()

💡 Pro tip: In real projects, replace the CREATE TABLE step with read_csv_auto() or a database connection. The rest of the code stays identical.


Step 2: Core Analysis Queries (Sub-second Results)

2.1 Daily Sales Trend + Week-over-Week Comparison (Window Functions)

The most fundamental and frequently used analysis. Use DuckDB’s LAG() window function for effortless WoW calculations:

daily_sales = con.execute("""
WITH daily AS (
 SELECT 
 order_date,
 SUM(final_price * quantity) AS revenue,
 COUNT(DISTINCT order_id) AS orders,
 AVG(final_price * quantity) AS avg_order_value
 FROM orders
 GROUP BY order_date
)
SELECT 
 order_date,
 revenue,
 orders,
 ROUND(avg_order_value, 2) AS avg_order_value,
 LAG(revenue, 7) OVER (ORDER BY order_date) AS revenue_last_week,
 ROUND(
 (revenue - LAG(revenue, 7) OVER (ORDER BY order_date)) 
 / NULLIF(LAG(revenue, 7) OVER (ORDER BY order_date), 0) * 100, 2
 ) AS WoW_change_pct
FROM daily
ORDER BY order_date
""").df()

print(daily_sales.tail(7))
# Shows last 7 days of sales trend with automatic WoW comparison

Key insight: DuckDB’s window functions complete in ~100ms on million-row datasets — 10-50x faster than Pandas. This is because DuckDB uses columnar storage and vectorized execution, while Pandas processes row-by-row.

2.2 Category Pareto Analysis (80/20 Rule)

Tell your client “which 20% of categories drive 80% of revenue” — one of the most valuable insights you can provide. Built with CTEs and window functions:

category_pareto = con.execute("""
WITH cat_rev AS (
 SELECT 
 category,
 SUM(final_price * quantity) AS total_revenue,
 ROUND(
 SUM(final_price * quantity) * 100.0 
 / SUM(SUM(final_price * quantity)) OVER (), 2
 ) AS revenue_pct
 FROM orders
 GROUP BY category
),
ranked AS (
 SELECT *,
 ROW_NUMBER() OVER (ORDER BY total_revenue DESC) AS rn,
 SUM(total_revenue) OVER (ORDER BY total_revenue DESC 
 ROWS UNBOUNDED PRECEDING) AS cumulative_revenue
 FROM cat_rev
)
SELECT 
 category,
 total_revenue,
 revenue_pct,
 rn,
 ROUND(cumulative_revenue / SUM(total_revenue) OVER () * 100, 1) AS cumulative_pct
FROM ranked
ORDER BY total_revenue DESC
""").df()

print(category_pareto.to_string(index=False))

This query directly feeds into a “Category Strategy Report” for your client.

2.3 User Lifetime Value (LTV) Tiering

User segmentation is the core feature of paid subscription services. Use PERCENTILE_CONT to tier users into Bronze/Silver/Gold/Platinum:

ltv = con.execute("""
WITH user_metrics AS (
 SELECT 
 FLOOR(RANDOM() * 5000) + 1 AS user_id,
 COUNT(DISTINCT order_id) AS total_orders,
 SUM(final_price * quantity) AS lifetime_value,
 MIN(order_date) AS first_order_date,
 MAX(order_date) AS last_order_date,
 DATEDAY(MAX(order_date)) - DATEDAY(MIN(order_date)) AS active_days
 FROM orders
 GROUP BY user_id
 HAVING total_orders >= 1
)
SELECT 
 CASE 
 WHEN lifetime_value >= PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY lifetime_value) THEN 'Platinum'
 WHEN lifetime_value >= PERCENTILE_CONT(0.7) WITHIN GROUP (ORDER BY lifetime_value) THEN 'Gold'
 WHEN lifetime_value >= PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lifetime_value) THEN 'Silver'
 ELSE 'Bronze'
 END AS tier,
 COUNT(*) AS user_count,
 ROUND(AVG(lifetime_value), 2) AS avg_ltv,
 ROUND(SUM(lifetime_value), 2) AS total_revenue,
 ROUND(AVG(total_orders), 2) AS avg_orders
FROM user_metrics
GROUP BY 1
ORDER BY avg_ltv DESC
""").df()

print(ltv.to_string(index=False))

📌 ** point**: This LTV tiering query, with minor modifications, can be embedded into your client’s CRM system — a core feature of paid subscription services.


Step 3: Encapsulate as a Reusable Module

Wrap the queries into a Python class for one-click execution:

class DuckDBSalesEngine:
 """E-commerce Sales Analytics Engine - initialize once, reuse infinitely"""
 
 def __init__(self, data_path: str = ":memory:"):
 self.con = duckdb.connect(data_path)
 
 def load_csv(self, table_name: str, file_path: str):
 """Auto-detect schema and import CSV"""
 self.con.execute(
 f"CREATE TABLE {table_name} AS SELECT * FROM read_csv_auto('{file_path}')"
 )
 
 def daily_trend(self, days: int = 30) -> pd.DataFrame:
 return self.con.execute("""
 WITH daily AS (
 SELECT order_date,
 SUM(final_price * quantity) AS revenue,
 COUNT(DISTINCT order_id) AS orders
 FROM orders
 GROUP BY order_date
 )
 SELECT order_date, revenue, orders,
 LAG(revenue, 7) OVER (ORDER BY order_date) AS prev_week_rev,
 ROUND((revenue - LAG(revenue,7) OVER (ORDER BY order_date)) 
 / NULLIF(LAG(revenue,7) OVER (ORDER BY order_date),0) * 100, 2) AS wow_pct
 FROM daily
 WHERE order_date >= (SELECT MAX(order_date) - INTERVAL '{days}' DAY FROM daily)
 ORDER BY order_date
 """.format(days=days)).df()
 
 def category_pareto(self) -> pd.DataFrame:
 return self.con.execute("""
 WITH cat_rev AS (
 SELECT category,
 SUM(final_price * quantity) AS total_revenue
 FROM orders GROUP BY category
 ),
 ranked AS (
 SELECT *, 
 SUM(total_revenue) OVER (ORDER BY total_revenue DESC 
 ROWS UNBOUNDED PRECEDING) AS cum_rev
 FROM cat_rev
 )
 SELECT category, total_revenue, 
 ROUND(total_revenue/SUM(total_revenue) OVER()*100, 2) AS pct,
 ROUND(cum_rev/SUM(total_revenue) OVER()*100, 1) AS cum_pct
 FROM ranked ORDER BY total_revenue DESC
 """).df()
 
 def export_report(self, output_path: str = "sales_report.parquet"):
 """Export analysis results to Excel"""
 trend = self.daily_trend()
 pareto = self.category_pareto()
 with pd.ExcelWriter(output_path, engine='openpyxl') as writer:
 trend.to_excel(writer, sheet_name='Daily Trend', index=False)
 pareto.to_excel(writer, sheet_name='Category Pareto', index=False)
 print(f"✅ Report exported: {output_path}")

Usage is straightforward:

engine = DuckDBSalesEngine()
# engine.load_csv('orders', '/path/to/your/orders.csv')
trend = engine.daily_trend(30)
pareto = engine.category_pareto()
engine.export_report("weekly_report.xlsx")

Step 4: Streamlit Dashboard Integration

Make the analysis visual so clients can view results in their browser:

# streamlit_app.py
import streamlit as st
import duckdb
import pandas as pd

st.set_page_config(page_title="📊 E-commerce Analytics", layout="wide")
st.title("🦆 DuckDB E-commerce Sales Engine")

# Sidebar: Data source selection
st.sidebar.header("📁 Data Source")
data_source = st.sidebar.selectbox("Choose source", ["Sample Data", "Upload CSV"])

if data_source == "Sample Data":
 con = duckdb.connect(":memory:")
 # ... same sample data generation as above
else:
 uploaded = st.sidebar.file_uploader("Upload CSV", type=["csv"])
 if uploaded:
 df = pd.read_csv(uploaded)
 con = duckdb.connect(":memory:")
 con.register("orders", df)

# Main dashboard: Three KPI cards
col1, col2, col3 = st.columns(3)
with col1:
 st.metric("📈 Today's Revenue", "$12,450", "+8.3%")
with col2:
 st.metric("🛒 Today's Orders", "342", "+5.1%")
with col3:
 st.metric("💰 Avg Order Value", "$36.41", "-1.2%")

# Sales trend chart
st.subheader("📅 Last 30 Days Sales Trend")
trend_df = con.execute("""
 SELECT order_date, 
 SUM(final_price * quantity) AS revenue
 FROM orders
 GROUP BY order_date
 ORDER BY order_date
""").df()
st.line_chart(trend_df.set_index('order_date')['revenue'])

# Category Pareto chart
st.subheader("🏷️ Category Revenue Contribution (Pareto)")
pareto_df = con.execute("""
 SELECT category, 
 SUM(final_price * quantity) AS revenue,
 ROUND(SUM(final_price * quantity)*100.0/SUM(SUM(final_price * quantity)) OVER (), 1) AS pct
 FROM orders GROUP BY category
""").df()
pareto_df = pareto_df.sort_values('revenue', ascending=False)
pareto_df['cum_pct'] = pareto_df['pct'].cumsum()
st.bar_chart(pareto_df.set_index('category')[['revenue', 'pct']])

# Export button
if st.button("📥 Export Report"):
 with pd.ExcelWriter("sales_report.xlsx", engine='openpyxl') as writer:
 con.execute("SELECT * FROM orders").df().to_excel(writer, sheet_name='Raw Data', index=False)
 st.success("✅ Report downloaded!")

Run it:

pip install streamlit duckdb pandas openpyxl
streamlit run streamlit_app.py

DuckDB vs Traditional Approaches

DimensionExcel ManualPandasDuckDB
1M row processing❌ Laggy/Crashes✅ Works (slow)✅ Sub-second
Memory usageHigh (row-based)High (row-based)Low (columnar+vectorized)
SQL support❌ Complex VBA⚠️ Limited✅ Full SQL
Multi-source join❌ Manual merge⚠️ Requires ETL✅ Native support
Deployment complexityLowMediumLow (embedded)
CostHigh labor costFreeFree

本文信息

项目内容
DuckDB 版本v1.5.x(部分功能基于 v2.0 Preview)
最后验证2026-09-12
测试环境Linux / x86_64 / 16GB RAM
官方文档DuckDB Documentation
GitHubpengzz9527/duckdb-blog

如发现错误,欢迎通过 GitHub Issue 或邮件 [email protected] 反馈。

📺 Watch video tutorials → Olap Studio YouTube

Subscribe for more DuckDB & AI automation tutorials