Featured image of post The Complete Guide to Time Series Analysis with DuckDB: From MoM/YoY to User Retention

The Complete Guide to Time Series Analysis with DuckDB: From MoM/YoY to User Retention

Master all time series analysis tasks in DuckDB — MoM/YoY calculations, moving averages, user retention, Gap & Island problems — using window functions and time_bucket. 10x faster than Pandas.

The Complete Guide to Time Series Analysis with DuckDB: From MoM/YoY to User Retention

DuckDB Time Series Analysis Architecture

You’ve definitely encountered these requirements before:

  • “How much did sales grow month-over-month last month?”
  • “What’s the year-over-year trend for each quarter this year?”
  • “What’s the 7-day moving average?”
  • “When was each user’s most recent order?”

The traditional approach uses pandas with a bunch of groupby + shift + rolling operations — verbose code that’s easy to get wrong. With DuckDB, you can handle all time series operations in a single SQL query, and it runs faster.

1. Month-over-Month & Year-over-Year Calculations

Given a sales table sales(date, amount), let’s calculate both month-over-month (MoM) growth and year-over-year (YoY) growth.

SELECT
 date_trunc('month', sale_date) AS month,
 SUM(amount) AS monthly_revenue,
 -- MoM: compare with previous month
 LAG(SUM(amount)) OVER (ORDER BY date_trunc('month', sale_date)) AS prev_month_revenue,
 ROUND(
 100.0 * (SUM(amount) - LAG(SUM(amount)) OVER (ORDER BY date_trunc('month', sale_date)))
 / NULLIF(LAG(SUM(amount)) OVER (ORDER BY date_trunc('month', sale_date)), 0),
 2
 ) AS mom_growth_pct,
 -- YoY: compare with same month last year
 LAG(SUM(amount), 12) OVER (ORDER BY date_trunc('month', sale_date)) AS same_month_last_year,
 ROUND(
 100.0 * (SUM(amount) - LAG(SUM(amount), 12) OVER (ORDER BY date_trunc('month', sale_date)))
 / NULLIF(LAG(SUM(amount), 12) OVER (ORDER BY date_trunc('month', sale_date)), 0),
 2
 ) AS yoy_growth_pct
FROM sales
GROUP BY date_trunc('month', sale_date)
ORDER BY month;

Key points:

  • LAG(col, n) — Get the value from n rows before the current one. n=1 is last month, n=12 is same month last year
  • NULLIF(..., 0) — Prevent division by zero errors
  • date_trunc('month', ...) — Truncate dates to month boundaries for proper grouping

In pandas, achieving the same result requires multiple steps: groupby aggregation, then shift() for alignment, then manual calculation. DuckDB does it all in one query.

2. Moving Averages (MA)

Technical analysis enthusiasts know this well. With DuckDB’s window functions, you can compute MA5, MA20, and MA60 in just a few lines.

SELECT
 sale_date,
 amount AS daily_revenue,
 -- 5-day moving average
 AVG(amount) OVER (
 ORDER BY sale_date
 ROWS BETWEEN 4 PRECEDING AND CURRENT ROW
 ) AS ma5,
 -- 20-day moving average
 AVG(amount) OVER (
 ORDER BY sale_date
 ROWS BETWEEN 19 PRECEDING AND CURRENT ROW
 ) AS ma60,
 -- 60-day moving average
 AVG(amount) OVER (
 ORDER BY sale_date
 ROWS BETWEEN 59 PRECEDING AND CURRENT ROW
 ) AS ma60
FROM sales
ORDER BY sale_date;

Core syntax: ROWS BETWEEN n PRECEDING AND CURRENT ROW means “the current row plus the previous n rows” — exactly what a sliding window needs.

3. Time Bucket Aggregation (time_bucket)

DuckDB v1.0+ introduced the time_bucket function, which aggregates data at arbitrary time intervals. Perfect for real-time dashboards.

-- Aggregate every 15 minutes (ideal for high-frequency trading data)
SELECT
 time_bucket(INTERVAL '15 minutes', timestamp) AS time_slot,
 COUNT(*) AS trade_count,
 AVG(price) AS avg_price,
 MAX(price) - MIN(price) AS price_range,
 SUM(volume) AS total_volume
FROM trades
WHERE timestamp >= TIMESTAMP '2026-07-01'
GROUP BY time_bucket(INTERVAL '15 minutes', timestamp)
ORDER BY time_slot;

-- Daily aggregation
SELECT
 time_bucket(INTERVAL '1 day', sale_date) AS day,
 SUM(amount) AS daily_revenue,
 COUNT(DISTINCT customer_id) AS active_customers
FROM orders
GROUP BY day
ORDER BY day;

Advantages of time_bucket:

  • Automatically handles irregular time intervals
  • More flexible than manual date_trunc — supports any interval (15 minutes, 30 days, 1 quarter, etc.)
  • Output is standard timestamp, making subsequent JOINs straightforward

4. User Retention Analysis

This is one of the most critical metrics for internet products. Calculate Day-1, Day-7, and Day-30 retention rates with DuckDB.

WITH first_login AS (
 -- Each user's first login date
 SELECT
 user_id,
 MIN(login_date) AS first_date
 FROM user_logins
 GROUP BY user_id
),
login_after_first AS (
 -- Every login after the user's first one
 SELECT
 f.user_id,
 f.first_date,
 l.login_date,
 DATEDIFF('day', f.first_date, l.login_date) AS days_since_first
 FROM first_login f
 JOIN user_logins l ON f.user_id = l.user_id
 WHERE l.login_date > f.first_date
)
SELECT
 days_since_first AS retention_day,
 COUNT(DISTINCT user_id) AS retained_users,
 ROUND(100.0 * COUNT(DISTINCT user_id) / (
 SELECT COUNT(DISTINCT user_id) FROM first_login
 ), 2) AS retention_rate_pct
FROM login_after_first
WHERE days_since_first BETWEEN 0 AND 30
GROUP BY days_since_first
ORDER BY retention_day;

5. Gap & Island Problem — Identifying Active User Periods

Real-world scenario: your SaaS customers have active periods and dormant periods, and you want to automatically identify each customer’s “active phase.”

WITH groups AS (
 SELECT
 user_id,
 login_date,
 -- Consecutive login dates form the same group
 login_date - INTERVAL '1 day' * ROW_NUMBER() OVER (
 PARTITION BY user_id ORDER BY login_date
 ) AS grp
 FROM user_logins
)
SELECT
 user_id,
 MIN(login_date) AS period_start,
 MAX(login_date) AS period_end,
 COUNT(*) AS consecutive_days
FROM groups
GROUP BY user_id, grp
HAVING consecutive_days >= 3
ORDER BY user_id, period_start;

This pattern is called “Gap & Island,” one of the most classic problems in data analysis. The core idea: use date - row_number to mark consecutive ranges — identical values belong to the same group.

6. Complete Python Example: CSV to Report

import duckdb
import pandas as pd

con = duckdb.connect()

# Create sample data
con.execute("""
CREATE TABLE sales AS
SELECT
 DATE '2026-01-01' + (random() * 365)::INT AS sale_date,
 (random() * 1000)::DECIMAL(10,2) AS amount,
 ('Beijing','Shanghai','Guangzhou','Shenzhen','Hangzhou')[1 + (random()*4)::INT] AS city,
 ('Electronics','Clothing','Food','Home','Books')[1 + (random()*4)::INT] AS category
FROM range(100000)
""")

# Generate monthly sales report in one query
report = con.execute("""
WITH monthly AS (
 SELECT
 date_trunc('month', sale_date) AS month,
 SUM(amount) AS revenue,
 COUNT(*) AS order_count,
 AVG(amount) AS avg_order_value
 FROM sales
 GROUP BY month
)
SELECT
 month,
 revenue,
 order_count,
 avg_order_value,
 LAG(revenue) OVER w AS prev_month_revenue,
 ROUND(100.0 * (revenue - LAG(revenue) OVER w) / NULLIF(LAG(revenue) OVER w, 0), 2) AS mom_growth,
 ROUND(AVG(revenue) OVER (
 ORDER BY month ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
 ), 2) AS ma3_revenue
FROM monthly
WINDOW w AS (ORDER BY month)
ORDER BY month
""").fetchdf()

print(report.to_string(index=False))

7. Comparison with Traditional Tools

FeatureDuckDBPandasExcelClickHouse
MoM/YoYLAG window functionshift() + mergeComplex formulasSimilar window functions
Moving AverageROWS BETWEENrolling()No built-inSimilar window functions
Time Buckettime_bucket()resample()Pivot tablesNot supported
User RetentionCTE + JOINMulti-step logicComplex formulasSimilar window functions
Gap & IslandDate - RowNumComplex logicNearly impossibleSimilar window functions
1M rows speed<100ms~2sLaggy<100ms
Learning curveLow (SQL)Medium (Python)LowHigh

8. Overlooked Tip: WINDOW Clause Reuse

When you reference the same window definition multiple times in a single query, use the WINDOW clause to avoid repetition:

SELECT
 month,
 revenue,
 LAG(revenue) OVER w AS prev_month,
 AVG(revenue) OVER w AS ma3,
 SUM(revenue) OVER w AS cumulative
FROM monthly_sales
WINDOW w AS (ORDER BY month);

Write the window definition once, get cleaner queries and better performance (the optimizer reuses the computation).

本文信息

项目内容
DuckDB 版本v1.5.x(部分功能基于 v2.0 Preview)
最后验证2026-09-12
测试环境Linux / x86_64 / 16GB RAM
官方文档DuckDB Documentation
GitHubpengzz9527/duckdb-blog

如发现错误,欢迎通过 GitHub Issue 或邮件 [email protected] 反馈。

📺 Watch video tutorials → Olap Studio YouTube

Subscribe for more DuckDB & AI automation tutorials