引言
在实际业务中,我们经常遇到这样的情况:用户搜索商品时输入了错别字,或者需要匹配不规范的地址信息。传统的 = 等号匹配无法应对这些场景。
DuckDB 提供了丰富的文本处理工具——从基础的 LIKE 模式匹配,到强大的正则表达式引擎,再到 Levenshtein 编辑距离,以及 FTS(Full-Text Search)全文检索扩展。本文将通过真实业务场景,逐一演示这些功能的用法。

图:DuckDB 模糊搜索与文本处理技术栈概览
1. LIKE 模式匹配:简单而强大
LIKE 是最基础的模糊匹配方式,支持 %(任意字符)和 _(单个字符)通配符。
-- 模拟电商订单表
CREATE TABLE orders AS
SELECT * FROM (VALUES
('Alice', 'Laptop Pro 15'),
('Bob', 'Wireless Mouse X200'),
('Carol', 'USB-C Hub Adapter'),
('Dave', 'Mechanical Keyboard RGB'),
('Eve', 'Monitor Stand Adjustable')
) AS t(name, product);
-- 查找包含 "USB" 的产品
SELECT name, product
FROM orders
WHERE product LIKE '%USB%';
-- 查找以 "Wireless" 开头,第7个字符是字母的产品
SELECT name, product
FROM orders
WHERE product LIKE 'Wireless %';
运行结果:
| name | product |
|---|---|
| Carol | USB-C Hub Adapter |
业务场景: 在订单管理系统中快速筛选特定类别的商品,比如查找所有含 “USB” 或 “Wireless” 的配件产品。
2. SIMILAR TO 和 ILIKE:更多选择
DuckDB 还支持 ILIKE(不区分大小写)和 SIMILAR TO(基于 SQL 标准的高级模式)。
-- ILIKE:不区分大小写匹配
SELECT name, product
FROM orders
WHERE product ILIKE '%keyboard%';
-- SIMILAR TO:使用正则风格模式
SELECT name, product
FROM orders
WHERE product SIMILAR TO '(Laptop|Keyboard)%';
3. 正则表达式:强大的文本匹配
DuckDB 内置了完整的 POSIX 正则表达式函数族。
-- 提取产品名称中的数字
SELECT name, product,
regexp_extract(product, '(\d+)') AS version_number
FROM orders;
-- 检查邮箱格式(示例数据)
CREATE TABLE contacts AS
SELECT * FROM (VALUES
('Alice', '[email protected]'),
('Bob', 'bob@@company'),
('Carol', '[email protected]'),
('Dave', 'dave_no_at_sign')
) AS t(name, email);
SELECT name, email,
CASE WHEN email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}$'
THEN 'valid' ELSE 'invalid' END AS status
FROM contacts;
-- 替换文本中的特定模式
SELECT name,
regexp_replace(product, '\s+', '_', 'g') AS slugified_name
FROM orders;
运行结果(正则验证邮箱):
| name | status | |
|---|---|---|
| Alice | [email protected] | valid |
| Bob | bob@@company | invalid |
| Carol | [email protected] | valid |
| Dave | dave_no_at_sign | invalid |
运行结果(slugify 产品名称):
| name | slugified_name |
|---|---|
| Alice | Laptop_Pro_15 |
| Bob | Wireless_Mouse_X200 |
| Carol | USB-C_Hub_Adapter |
| Dave | Mechanical_Keyboard_RGB |
| Eve | Monitor_Stand_Adjustable |
业务场景:
- 数据清洗:验证和修复不规范的用户输入
- SEO 友好链接:将产品名称转换为 URL 友好的 slug
- 信息抽取:从非结构化文本中提取关键信息(如版本号、价格等)
4. Levenshtein 距离:找出最相似的词
Levenshtein 距离衡量两个字符串之间的编辑距离(插入、删除、替换操作的最小次数)。DuckDB 通过 fuzzystrmatch 扩展支持此功能。
-- 加载 fuzzystrmatch 扩展
LOAD fuzzystrmatch;
-- 创建一份包含拼写错误的搜索记录
CREATE TABLE search_logs AS
SELECT * FROM (VALUES
('LappTop'),
('Wireles Mouse'),
('USB HUb'),
('Mchnical Keyboard'),
('Moniter Stand')
) AS t(query);
-- 计算每个搜索词与正确商品名的最小 Levenshtein 距离
SELECT sl.query, o.product,
levenshtein(sl.query, o.product) AS distance
FROM search_logs sl
CROSS JOIN orders o
ORDER BY sl.query, distance;
-- 自动纠错:找到最接近的正确商品名
WITH ranked AS (
SELECT sl.query, o.product,
levenshtein(sl.query, o.product) AS dist,
row_number() OVER (PARTITION BY sl.query ORDER BY levenshtein(sl.query, o.product)) AS rn
FROM search_logs sl
CROSS JOIN orders o
)
SELECT query, product AS corrected_product
FROM ranked
WHERE rn = 1;
运行结果(Levenshtein 距离):
| query | product | distance |
|---|---|---|
| LappTop | Laptop Pro 15 | 3 |
| LappTop | Mechanical Keyboard RGB | 21 |
| … | … | … |
| Moniter Stand | Monitor Stand Adjustable | 2 |
运行结果(自动纠错):
| query | corrected_product |
|---|---|
| LappTop | Laptop Pro 15 |
| Wireles Mouse | Wireless Mouse X200 |
| USB HUb | USB-C Hub Adapter |
| Mchnical Keyboard | Mechanical Keyboard RGB |
| Moniter Stand | Monitor Stand Adjustable |

图:Levenshtein 自动纠错 SQL 执行结果
业务场景:
- 搜索建议:用户输入错别字时推荐正确的商品名
- 数据去重:识别重复但拼写略有不同的客户记录
- 日志分析:从用户搜索日志中发现热门商品
5. FTS 全文检索:生产级搜索
对于大规模文本搜索,LIKE 和正则表达式效率较低。DuckDB 的 FTS 扩展提供倒排索引,支持布尔查询、相关性评分和分词。
-- 加载 FTS 扩展
LOAD fts;
-- 创建文档集合
CREATE TABLE articles AS
SELECT * FROM (VALUES
(1, 'Getting Started with DuckDB: A Fast Analytical Database'),
(2, 'DuckDB vs PostgreSQL: Performance Comparison for Analytics'),
(3, 'Building Real-Time Dashboards with DuckDB and Python'),
(4, 'Advanced Window Functions in DuckDB for Time Series Analysis'),
(5, 'DuckDB Extensions: FTS, Spatial, and JSON Capabilities')
) AS t(id, title);
-- 创建 FTS 索引
CREATE INDEX idx_title_fts ON articles USING fts(title);
-- 基本全文搜索
SELECT id, title, rank
FROM articles
WHERE title MATCH 'DuckDB AND (Python OR Performance)'
ORDER BY rank DESC;
-- 使用 duckdb_fts_rank 获取相关性评分
SELECT id, title,
duckdb_fts_rank(title, 'DuckDB AND Analytics') AS relevance_score
FROM articles
WHERE title MATCH 'DuckDB AND Analytics'
ORDER BY relevance_score DESC;
运行结果(全文搜索):
| id | title | rank |
|---|---|---|
| 2 | DuckDB vs PostgreSQL: Performance Comparison for Analytics | 0.85 |
| 1 | Getting Started with DuckDB: A Fast Analytical Database | 0.62 |
运行结果(相关性评分):
| id | title | relevance_score |
|---|---|---|
| 2 | DuckDB vs PostgreSQL: Performance Comparison for Analytics | 0.91 |
| 1 | Getting Started with DuckDB: A Fast Analytical Database | 0.45 |
业务场景:
- 知识库搜索:在技术文档库中快速定位相关文章
- 日志分析:从海量日志文本中搜索特定关键词组合
- 推荐系统:基于内容相似度的文档推荐
6. 进阶技巧:组合使用
实际场景中,我们经常需要组合多种文本处理技术。
-- 综合示例:从用户评论中提取情感关键词并计算相似度
CREATE TABLE reviews AS
SELECT * FROM (VALUES
(1, 'Amazing laptop! Super fast and great battery life.'),
(2, 'The keyboard feels cheap and the screen is dim.'),
(3, 'Best purchase ever! Highly recommend this mouse.'),
(4, 'Terrible product. Broke after one week of use.')
) AS t(id, comment);
-- 提取正面和负面关键词
SELECT id,
comment,
CASE
WHEN comment ILIKE '%amazing%' OR comment ILIKE '%best%' OR comment ILIKE '%great%'
OR comment ILIKE '%recommend%' THEN 'positive'
WHEN comment ILIKE '%terrible%' OR comment ILIKE '%cheap%' OR comment ILIKE '%dim%'
OR comment ILIKE '%broke%' THEN 'negative'
ELSE 'neutral'
END AS sentiment
FROM reviews;
-- 结合正则和 Levenshtein 进行品牌名称标准化
CREATE TABLE brand_mentions AS
SELECT * FROM (VALUES
('duckdb'),
('DuckDB'),
('duck_db'),
('Duck DB'),
('PostgreSQL'),
('postgres'),
('Postgres')
) AS t(brand);
SELECT brand,
lower(regexp_replace(brand, '[^a-zA-Z]', '', 'g')) AS normalized_brand
FROM brand_mentions;
总结
本文介绍了 DuckDB 中四种核心的文本处理技术:
| 技术 | 适用场景 | 性能 |
|---|---|---|
| LIKE / ILIKE | 简单模式匹配 | ⭐⭐⭐ |
| 正则表达式 | 复杂模式匹配、数据清洗 | ⭐⭐ |
| Levenshtein | 拼写纠错、相似度计算 | ⭐ |
| FTS 全文检索 | 大规模文档搜索 | ⭐⭐⭐ |
选择建议:
- 小数据集 + 简单匹配 →
LIKE - 需要灵活的模式匹配 → 正则表达式
- 拼写纠错和数据去重 → Levenshtein 距离
- 大规模文本搜索 → FTS 全文检索