Featured image of post Build an Automated Financial Daily Report with DuckDB: From Data to Revenue

Build an Automated Financial Daily Report with DuckDB: From Data to Revenue

Step-by-step guide to building a fully automated financial daily report system using DuckDB + Python. Learn how data analysts can generate 3000+ CNY/month in passive income through information processing.

In the data monetization space, information processing is one of the most undervalued skills. Most data analysts confine their reporting abilities to internal presentations, never realizing this skill can be directly converted into subscription-based revenue. Today we’ll break down a real monetization product: building a fully automated financial daily report with DuckDB, launching in 3 hours, generating a stable extra 3000+ CNY per month.

Architecture

1. Product Logic: What Are You Actually Selling?

Start with the pricing model before designing the technical architecture. A typical financial daily report product:

  • Target customers: Small investment firms, independent financial advisors, trading community members
  • Deliverable: Morning brief pushed to Telegram / WeChat groups
  • Content structure: Market overview → Hot sectors → Capital flow → Risk signals
  • Pricing reference: Monthly subscription 99-299 CNY, annual 999 CNY

The core value isn’t “data” — public data is everywhere. You sell processed insights. Raw data is free; insight services are paid.

2. Why DuckDB? Technical Architecture Comparison

Traditional approaches use PostgreSQL + Python scripts, but the operational cost is too high for individual developers. DuckDB’s core advantages:

DimensionTraditional (PostgreSQL)DuckDB Approach
Deployment2-4 hours (server + DB)30 minutes (code only)
Monthly cost50-200 CNY (cloud server)0 CNY
Analysis performanceRow storage, slower aggregationColumnar, million-row aggregation in seconds
Data ingestionRequires ETL pipeline firstNative HTTP / JSON / CSV support
EmbeddingClient connection requiredIn-process library, seamless Python/R/Node.js integration
Monthly costCloud server 50-200 CNYCompletely free

The entire system needs only one Python script + one GitHub Actions scheduled job — monthly cost = 0.

3. Project Structure

financial-daily-report/
├── report.py          # Main program (fetch → analyze → push)
├── config.yaml        # Configuration (API keys, Telegram Bot Token)
├── data/              # DuckDB database file
│   └── market.db
├── .github/workflows/
│   └── daily-cron.yml  # GitHub Actions scheduled job
└── requirements.txt

4. Core Code Implementation

Step 1: Data Fetching and Storage

DuckDB can directly query JSON from HTTP endpoints, or read from CSV/Parquet files — no need to pre-load into a database:

import duckdb
import httpx
import yaml
from pathlib import Path
from datetime import datetime, timedelta
import json

CONFIG_PATH = Path(__file__).parent / "config.yaml"
with open(CONFIG_PATH) as f:
    cfg = yaml.safe_load(f)

DB_PATH = Path(__file__).parent / "data" / "market.db"
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

def fetch_market_data() -> dict:
    """Fetch A-share market core indicators"""
    # Replace with real APIs like AKShare, Tushare in production
    return {
        "indices": [
            {"code": "SH000001", "name": "Shanghai Composite", "close": 3285.67, "pct": 0.42},
            {"code": "SZ399001", "name": "Shenzhen Component", "close": 10567.23, "pct": 0.18},
            {"code": "SZ399006", "name": "ChiNext Index", "close": 2156.89, "pct": -0.35},
        ],
        "sectors": [
            {"name": "Semiconductor", "pct": 2.14, "volume_ratio": 1.32},
            {"name": "New Energy", "pct": 1.87, "volume_ratio": 1.15},
            {"name": "Liquor", "pct": -0.52, "volume_ratio": 0.88},
            {"name": "AI Applications", "pct": 3.21, "volume_ratio": 1.67},
        ],
        "capital_flow": {
            "northbound": 12.5,
            "margin": -3.2,
            "main_force": 45.8,
        },
    }

def save_to_duckdb(data: dict) -> None:
    conn = duckdb.connect(str(DB_PATH))
    today = datetime.now().strftime("%Y-%m-%d")

    # Create tables if not exist, delete-then-insert for idempotent daily updates
    conn.execute("""
        CREATE TABLE IF NOT EXISTS daily_indices (
            date        DATE,
            code        VARCHAR,
            name        VARCHAR,
            close       DOUBLE,
            pct_change  DOUBLE,
            created_at  TIMESTAMP
        )
    """)
    conn.execute("DELETE FROM daily_indices WHERE date = ?", [today])
    for idx in data["indices"]:
        conn.execute(
            "INSERT INTO daily_indices VALUES (?, ?, ?, ?, ?, ?)",
            [today, idx["code"], idx["name"], idx["close"], idx["pct"], datetime.now()],
        )

    conn.execute("""
        CREATE TABLE IF NOT EXISTS daily_sectors (
            date           DATE,
            sector_name    VARCHAR,
            pct_change     DOUBLE,
            volume_ratio   DOUBLE,
            created_at     TIMESTAMP
        )
    """)
    conn.execute("DELETE FROM daily_sectors WHERE date = ?", [today])
    for sec in data["sectors"]:
        conn.execute(
            "INSERT INTO daily_sectors VALUES (?, ?, ?, ?, ?)",
            [today, sec["name"], sec["pct"], sec["volume_ratio"], datetime.now()],
        )

    conn.execute("""
        CREATE TABLE IF NOT EXISTS daily_capital (
            date           DATE,
            northbound     DOUBLE,
            margin_change  DOUBLE,
            main_force     DOUBLE,
            created_at     TIMESTAMP
        )
    """)
    conn.execute("DELETE FROM daily_capital WHERE date = ?", [today])
    conn.execute(
        "INSERT INTO daily_capital VALUES (?, ?, ?, ?, ?)",
        [today, data["capital_flow"]["northbound"],
         data["capital_flow"]["margin"], data["capital_flow"]["main_force"],
         datetime.now()],
    )

    conn.commit()
    conn.close()

Step 2: SQL-Driven Intelligence Engine

This is the core competitive advantage — using SQL to perform trend detection, sector ranking, and capital flow signal analysis:

def analyze_history(days: int = 5) -> dict:
    """Generate trend insights from historical data"""
    conn = duckdb.connect(str(DB_PATH))

    # Shanghai Composite trend over N days
    sh_trend = conn.execute("""
        SELECT date, close, pct_change
        FROM daily_indices
        WHERE code = 'SH000001'
          AND date >= CURRENT_DATE - INTERVAL ? DAY
        ORDER BY date DESC
    """, [days]).fetchall()

    # Sector heat ranking (filter by volume ratio > 1.0)
    hot_sectors = conn.execute("""
        SELECT sector_name, pct_change, volume_ratio
        FROM daily_sectors
        WHERE date = CURRENT_DATE
          AND volume_ratio > 1.0
        ORDER BY pct_change DESC
        LIMIT 3
    """).fetchall()

    # N-day capital flow summary
    capital_summary = conn.execute("""
        SELECT
            SUM(northbound)  AS total_north,
            AVG(main_force)  AS avg_main,
            CASE
                WHEN SUM(northbound) > 0 THEN '🟢 Foreign Capital Inflow'
                ELSE '🔴 Foreign Capital Outflow'
            END AS signal
        FROM daily_capital
        WHERE date >= CURRENT_DATE - INTERVAL ? DAY
    """, [days]).fetchone()

    conn.close()
    return {
        "sh_trend": sh_trend,
        "hot_sectors": hot_sectors,
        "capital": capital_summary,
    }

Key points in this SQL engine:

  1. CURRENT_DATE - INTERVAL ? DAY: DuckDB supports parameterized date arithmetic, avoiding string concatenation
  2. CASE WHEN signal generation: Conditional logic directly in SQL is clearer than post-processing in Python
  3. Delete-then-insert pattern: DELETE WHERE date = ? ensures idempotent daily updates

Step 3: Report Text Generation

def build_report(data: dict, insight: dict) -> str:
    today = datetime.now().strftime("%b %d")
    trend = insight["sh_trend"]
    last_close = trend[0][1] if trend else "?"
    prev_pct = trend[1][2] if len(trend) > 1 else 0

    up_days = sum(1 for row in trend if row[2] > 0)
    trend_text = f"Up {up_days}/{len(trend)} days" if trend else "No historical data"

    sector_lines = []
    for name, pct, vr in insight["hot_sectors"]:
        sector_lines.append(f"  · {name}  +{pct:.2f}% (VolRatio {vr:.2f})")
    sector_text = "\n".join(sector_lines) or "  No high-volume sectors"

    capital = insight["capital"]
    capital_text = f"{capital[2]}, N-day northbound累计{'+' if capital[0]>0 else ''}{capital[0]:.1f}B"

    report = f"""📊 【{today} Financial Daily】

━━━ I. Market Overview ━━━
Shanghai Comp {last_close:.2f} ({'↑' if prev_pct>=0 else '↓'}{abs(prev_pct):.2f}%)
{trend_text}

━━━ II. Hot Sectors TOP3 ━━━
{sector_text}

━━━ III. Capital Flow ━━━
{capital_text}

━━━ IV. Tomorrow's Signal ━━━
{generate_signal(insight)}

📌 Data source: Public market data, for reference only"""
    return report

5. Automated Deployment: GitHub Actions + Cron

Use GitHub Actions’ cron feature for hands-free pushing — completely free:

# .github/workflows/daily-cron.yml
name: Daily Financial Report

on:
  schedule:
    - cron: '0 22 * * *'  # 06:00 Beijing time (UTC 18:00)
  workflow_dispatch:      # Also supports manual trigger

jobs:
  run-report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run report generator
        env:
          TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
          TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
        run: python report.py

      - name: Commit and push results
        run: |
          git config user.name "CI Bot"
          git config user.email "[email protected]"
          git add data/
          git commit -m "Auto update: $(date +%Y-%m-%d)" || echo "No changes"
          git push

Store sensitive configs in config.yaml:

# config.yaml
telegram:
  bot_token: "YOUR_BOT_TOKEN"
  chat_id: "YOUR_CHAT_ID"

data_source: "akshare"  # or "tushare", "mock"

6. Efficiency Comparison: Traditional vs DuckDB

MetricPostgreSQL ApproachDuckDB Approach
Setup time2-4 hours (server + DB)30 minutes (code only)
Monthly cost50-200 CNY (cloud)0 CNY
Data storage (1 month)~1.5M rows (3 tables × 5 years)Single file < 5MB
Query latencyRequires network connectionLocal millisecond-level
Data recoveryRequires backup strategyCopy .db file directly
Scaling difficultyVertical scaling neededJust append historical data

7. Monetization Paths and Pricing

After building this system, your monetization paths include:

  1. Subscription daily report: Monthly 99-299 CNY, annual 999 CNY. 100 subscribers = 10K-30K CNY/month.
  2. Custom data dashboards: Build bespoke monitoring panels for enterprise clients, 5000-20000 CNY per project.
  3. Data product tiers: Upgrade the daily report to freemium model — basic free for lead gen, premium unlocks deep metrics.
  4. B2B SaaS: Package the system as multi-tenant SaaS, charge per seat.

Key reminder: Your core moat isn’t the code — the code is open source. The moat is data source stability and analytical model expertise. Pick a niche domain you understand deeply, and go deep on the analysis logic. That’s the real long-term monetization foundation.

8. Expansion: From Daily Report to Data Product

Once you’ve validated the daily report loop, expand further:

  • Multi-market coverage: A-shares → Hong Kong → US markets, one codebase, three revenue streams
  • Real-time alerts: Add price breakout and capital anomaly detection for instant push notifications
  • Historical backtesting: Use DuckDB’s Time Travel to backtest strategies
  • Visual dashboards: Integrate with Evidence or Streamlit for auto-generated HTML reports

📖 详细图文教程见 duckdblab.org

💡 更多 DuckDB 实战技巧 → duckdblab.org

📺 Watch video tutorials → Olap Studio YouTube

Subscribe for more DuckDB & AI automation tutorials

Built with Hugo
Theme Stack designed by Jimmy