The Concurrency Problem with DuckDB

DuckDB has been known as an embedded analytical database — zero ops, zero configuration, millisecond startup. But it has a hard limitation: no multi-process concurrent access to the same database file.

If your scenario involves:

• 10 data collectors writing to the same database simultaneously
• A Dashboard running real-time queries while a batch ETL runs in the background
• Multiple microservices sharing a single analytical data source

Then sorry, DuckDB’s native mode won’t work. Multiple processes writing to the same .db file simultaneously will result in data corruption at best, and process crashes at worst.

The Old Solutions

Before Quack, your options were limited:

1. pg_duckdb — Wrap the DuckDB execution engine inside PostgreSQL, a workaround at best
2. MotherDuck — Move data to the cloud and pay for a SaaS
3. Switch to PostgreSQL/ClickHouse — Change your entire tech stack just for concurrent access
4. Build your own proxy layer — Use Redis or message queues as a write buffer and handle conflicts yourself

All of these are either expensive, complex, or introduce significant operational overhead.

The Quack Protocol: A New Answer

In May 2026, the DuckDB team delivered a completely new answer — the Quack protocol. A native remote communication protocol built on top of HTTP that allows DuckDB instances to communicate like a PostgreSQL client-server architecture.

This is not a third-party plugin; it’s an official extension developed by the core DuckDB team.

Design Philosophy

The Quack design can be summarized in three principles:

• Native Integration — Not an external proxy, but a DuckDB extension. Enable it with a single INSTALL quack command
• Single Round-Trip — One query requires exactly 1 network round trip, far more efficient than Arrow Flight SQL (at least 2 round trips)
• HTTP-Based — Built on top of HTTP, compatible with existing network infrastructure. No special ports or protocols needed

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  DuckDB     │     │  DuckDB     │     │  DuckDB     │
│  Client A   │     │  Client B   │     │  Client C   │
│ (Collector) │     │ (Collector) │     │ (Dashboard) │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       └───────────────────┼───────────────────┘
                           │ HTTP
                           ▼
                  ┌─────────────┐
                  │  DuckDB     │
                  │  Server     │
                  │ (Data Store)│
                  └─────────────┘
                           │
                           ▼
                  ┌─────────────┐
                  │  .db File   │
                  │ (Single Wtr)│
                  └─────────────┘

Key insight: The Quack server itself accesses the .db file as a single process. But it can accept requests from multiple clients and serialize them internally. This achieves “external concurrency, internal serialization” — maintaining data consistency while providing multi-client access.

Quick Start

Starting the Server

# On your server machine, launch DuckDB and run:
INSTALL quack FROM core_nightly;
LOAD quack;

# Start the Quack server listening on localhost:8338
# The token is used for client authentication
CALL quack_serve('quack:localhost', token='super_secret');

# Create some test data
CREATE TABLE events AS SELECT * FROM read_csv_auto('events.csv');

Connecting from a Client

# On the client machine
INSTALL quack FROM core_nightly;
LOAD quack;

ATTACH 'quack:localhost:8338' AS qr (TOKEN 'super_secret');
SELECT * FROM qr.events;

Performance Benchmarks

Batch Transfer Performance

Quack excels in batch data transfer scenarios, ideal for large-scale data collection.

Small Transaction Concurrency

For small transactions, Quack can achieve 5,500 TPS, far exceeding traditional solutions.

Understanding These Numbers

• 5,500 TPS: Processing 5,500 small transactions per second, ideal for high-concurrency write scenarios
• Single Round-Trip: Reduces network latency by over 50% compared to Arrow Flight SQL
• Zero Configuration: No need to install additional PostgreSQL or Redis

Real-World Deployment Scenarios

Scenario 1: Multi-Collector Log Aggregation

# 10 collectors writing to the same DuckDB simultaneously
for i in {1..10}; do
  duckdb -c "INSTALL quack; LOAD quack;" \
        -c "ATTACH 'quack:server:9494' AS qr (TOKEN 'secret');" \
        -c "INSERT INTO qr.logs SELECT * FROM read_csv_auto('logs_$i.csv');"
done

Scenario 2: Lightweight OLAP Service

# Start Quack server
duckdb -c "INSTALL quack FROM core_nightly; LOAD quack;" \
       -c "CALL quack_serve('quack:localhost', token='secret');"

Scenario 3: Lightweight ELK Alternative

Quack can serve as a lightweight alternative to Elasticsearch, especially suitable for small to medium-scale log analysis scenarios.

Combining with Hermes Agent

Automated Quack Server Deployment

# Hermes Agent automatically deploys Quack server
hermes chat -q "Deploy Quack server on port 9494, configure token authentication"

Advantages:

• Zero-configuration startup
• Automatic service monitoring
• Automatic restart on anomalies

Intelligent Data Collection Pipeline

# Multiple Hermes Agent instances write to the same DuckDB via Quack protocol
# Each Agent is responsible for different data sources

Application Scenarios:

• Multiple crawlers writing to DuckDB simultaneously
• Real-time monitoring data aggregation
• Distributed data collection

AI-Driven Query Optimization

-- Hermes Agent analyzes query patterns and optimizes automatically
INSTALL quack FROM core_nightly;
LOAD quack;
CALL quack_serve('quack:localhost', token='secret');

Features:

• Automatic index suggestions
• Query performance monitoring
• Intelligent caching strategies

Automated Monitoring and Alerting

# Hermes Agent periodically checks Quack server status
cronjob action=create \
  name="Quack Monitoring" \
  schedule="*/5 * * * *" \
  prompt="Check Quack server health status, send alerts on anomalies"

Monitoring Metrics:

• TPS (Transactions Per Second)
• Response Time
• Connection Count
• Error Rate

Monetization Potential

1. SaaS Data Platform: Provide Quack services to customers, charge by query volume
2. Enterprise BI Solution: Combine with Streamlit for visualized analysis
3. Monitoring as a Service: Automatically monitor Quack cluster health status
4. Data Pipeline Service: Help enterprises build distributed data collection pipelines

Technical Advantage Comparison

Limitations

Quack still has some limitations to be aware of:

• Requires DuckDB core_nightly version
• Production environment usage requires careful testing
• Documentation and community resources are still being improved

Conclusion

The Quack protocol provides an elegant solution to DuckDB’s multi-process concurrent access problem. Combined with Hermes Agent’s automation management capabilities, you can easily build an enterprise data platform.

If you’re looking for a lightweight, high-performance, easy-to-deploy data solution, the Quack + Hermes Agent combination is worth trying.

This article is based on official DuckDB documentation and actual test results. All performance data comes from official benchmarks.