DuckDB 高级 SQL 技巧大全:PIVOT、Macro、窗口函数、递归CTE、JSON处理
很多数据分析师用 DuckDB 一年,还只会 SELECT * FROM table。今天一口气讲 6 个进阶技巧,每个都有可直接运行的代码,建议收藏反复查看。
一、PIVOT / UNPIVOT:告别手写 CASE WHEN
季度销售数据想从"行格式"转成"列格式"?PIVOT 一行搞定,比一堆 CASE WHEN 优雅太多。
PIVOT:行转列
-- 原始数据:每行一个季度
SELECT * FROM sales;
-- product | quarter | amount
-- Apple | Q1 | 100
-- Apple | Q2 | 150
-- Banana | Q1 | 200
-- Banana | Q2 | 120
-- PIVOT:把季度变成列
SELECT * FROM sales
PIVOT(sum(amount) FOR quarter IN ("Q1", "Q2"));
-- product | Q1 | Q2
-- Apple | 100 | 150
-- Banana | 200 | 120
UNPIVOT:列转行
SELECT * FROM pivot_sales
UNPIVOT(amount FOR quarter IN ("Q1", "Q2"));
-- product | quarter | amount
-- Apple | Q1 | 100
-- Apple | Q2 | 150
-- Banana | Q1 | 200
-- Banana | Q2 | 120
实战场景:电商月度报表
老板每个月要"各品类各月份的销售对比表",用 PIVOT 三行搞定:
SELECT
category,
"January" AS jan, "February" AS feb, "March" AS mar
FROM monthly_sales
PIVOT(
SUM(revenue)
FOR month_name
IN ("January", "February", "March")
);
UNPIVOT 的反向操作也很有用——当你的数据被"宽表化"存储(比如 EAV 模型)时,UNPIVOT 可以直接恢复成标准长表结构,省去了繁琐的 UNION ALL。
二、SQL Macro:把常用逻辑封装成函数
DuckDB 的 Macro 不只是标量函数,还能复用整段表达式,逻辑集中维护,SQL 可读性飙升。
基础 Macro
-- 定义一个计算平方差的宏
CREATE MACRO square(x) AS x * x;
SELECT square(5) AS result;
-- 25
SELECT square(col_a - col_b) AS diff_square
FROM transactions;
实用 Macro:环比增长率
CREATE MACRO mom_growth(curr, prev)
AS ROUND((curr - prev) * 1.0 / prev, 4);
SELECT
month,
revenue,
mom_growth(revenue, LAG(revenue) OVER(ORDER BY month)) AS growth
FROM monthly_sales;
-- month | revenue | growth
-- January | 10000 | NULL
-- February| 12000 | 0.2
-- March | 11000 | -0.0833
参数化 Macro:带默认值
CREATE MACRO round_to(x, digits DEFAULT 2)
AS ROUND(x, digits);
SELECT round_to(3.14159); -- 3.14
SELECT round_to(3.14159, 3); -- 3.142
SELECT round_to(100.12345, 1); -- 100.1
Macro 的核心价值:改一处,全库生效。当你定义了一个"活跃用户"的判断逻辑,以后所有 SQL 直接调用 is_active(user),不用每次重写条件。
三、LAG / FIRST_VALUE:窗口函数实战
窗口函数是 SQL 进阶的分水岭,掌握后 80% 的数据分析需求都能解决。
LAG:取上一条记录
SELECT name, subject, score,
LAG(score) OVER(PARTITION BY name ORDER BY score) AS prev_score
FROM scores;
-- name | subject | score | prev_score
-- Bob | Science | 88 | NULL
-- Bob | Math | 92 | 88
-- Alice | Math | 85 | NULL
-- Alice | Science | 90 | 85
FIRST_VALUE / LAST_VALUE
找每个用户的最高分和最低分:
SELECT DISTINCT name,
FIRST_VALUE(score) OVER(
PARTITION BY name ORDER BY score DESC
) AS best_score,
LAST_VALUE(score) OVER(
PARTITION BY name ORDER BY score ASC
ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
) AS worst_score
FROM scores;
进阶:累计占比
WITH ranked AS (
SELECT
product_name,
revenue,
SUM(revenue) OVER(ORDER BY revenue DESC
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative,
SUM(revenue) OVER() AS total
FROM products
)
SELECT
product_name,
revenue,
ROUND(cumulative * 100.0 / total, 2) AS cumulative_pct
FROM ranked
ORDER BY revenue DESC;
-- 这就是著名的帕累托分析(二八法则)
四、CTE 链式查询:复杂逻辑拆成步骤
多层嵌套的子查询读起来像天书?用 CTE 拆成步骤,每一步独立验证。
基本 CTE 链
WITH base AS (
-- 第一步:筛选有效订单
SELECT id, amount, customer_id
FROM orders
WHERE status = 'completed'
),
ranked AS (
-- 第二步:计算客户排名
SELECT
customer_id,
SUM(amount) AS total_spend,
ROW_NUMBER() OVER(ORDER BY SUM(amount) DESC) AS rn
FROM base
GROUP BY customer_id
),
top_customers AS (
-- 第三步:取 Top 10
SELECT * FROM ranked WHERE rn <= 10
)
SELECT * FROM top_customers;
CTE + 递归:处理组织架构
WITH RECURSIVE org_tree AS (
-- 锚点:最高层管理者
SELECT id, name, manager_id, 1 AS level
FROM employees
WHERE manager_id IS NULL
UNION ALL
-- 递归:下级员工
SELECT e.id, e.name, e.manager_id, t.level + 1
FROM employees e
JOIN org_tree t ON e.manager_id = t.id
)
SELECT
LPAD('', (level-1)*4, ' ') || name AS org_chart,
level
FROM org_tree
ORDER BY level, name;
-- 结果示例:
-- CEO
-- Engineering
-- Alice
-- Bob
-- Marketing
-- Charlie
递归 CTE 的常见场景:
- 组织架构层级
- 商品分类树
- 社交网络好友关系链
- 最短路径计算(配合 Dijkstra 算法)
五、JSON 处理 + LIST 操作:非结构化数据的利器
DuckDB 处理 JSON 不需要先转成表,直接 SQL 操作,这是它比 PostgreSQL 在分析场景更灵活的原因之一。
JSON 提取
-- 提取 JSON 字段
SELECT json_extract_string(
'{"user": "alice", "age": 30, "tags": ["dev", "gopher"]}',
'$.user'
) AS username;
-- alice
展开 JSON 数组
SELECT * FROM json_each('{
"items": ["Apple", "Banana", "Cherry"]
}'::JSON -> 'items') AS j;
-- [{"value":"Apple"},{"value":"Banana"},{"value":"Cherry"}]
LIST 常用操作
SELECT
list_sort([3, 1, 2]) AS sorted, -- [1, 2, 3]
list_reverse([1, 2, 3]) AS reversed, -- [3, 2, 1]
list_distinct([1, 1, 2, 3]) AS uniq, -- [1, 2, 3]
list_concat([1, 2], [3, 4]) AS joined, -- [1, 2, 3, 4]
list_slice([1,2,3,4,5], 2, 4) AS sliced, -- [2, 3, 4]
list_avg([10, 20, 30, 40]) AS average, -- 25.0
list_generate(1, 10, 2) AS odds; -- [1, 3, 5, 7, 9]
实战:处理嵌套 API 响应
-- 假设有一个 API 返回的 JSON 数组
WITH api_data AS (
SELECT * FROM json_each('[
{"id": 1, "name": "Alice", "scores": [90, 85, 92]},
{"id": 2, "name": "Bob", "scores": [78, 88, 95]}
]')
)
SELECT
value::JSON ->> 'name' AS name,
list_avg((value::JSON -> 'scores')::LIST) AS avg_score
FROM api_data;
-- Alice | 89.0
-- Bob | 87.0
DuckDB 的 JSON 处理优势:
- 不需要先定义 schema
- 与 SQL 无缝集成
- 可直接读写 Parquet 中的 JSON 列
六、六大技巧对比:传统方式 vs DuckDB
| 需求 | 传统方式 | DuckDB |
|---|---|---|
| 行转列 | 手写多个 CASE WHEN | PIVOT(... FOR col IN (...)) |
| 逻辑复用 | 复制粘贴 / 存过程 | CREATE MACRO |
| 取上一条 | 自连接 | LAG() OVER(...) |
| 层级数据 | 递归存储过程 | WITH RECURSIVE |
| JSON | 必须先 parse 再 JOIN | 直接 json_extract_string |
| 数组操作 | 手写循环 | list_sort, list_concat 等 |
七、变现建议
掌握这些进阶技巧后,你可以:
- 接数据分析外包:报价从 ¥2000/单 起步,熟练后 ¥5000+/单
- 搭建数据产品:用 PIVOT + CTE 快速生成老板要的报表,1 小时完成别人 3 天的工作量
- SaaS 化:把常用 Macro 封装成服务,API 调用计费
- 教学变现:把这些技巧整理成课程,定价 ¥99-¥299
🔍 本文完整代码(含 15 个生产级查询模板)已发布在 duckdblab.org,建议收藏反复查阅。学习更多 DuckDB 实战经验 → duckdblab.org
