<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>DuckDB 2.0 on OLAP Studio — Independent DuckDB Technical Blog</title><link>https://duckdblab.org/en/tags/duckdb-2.0/</link><description>Recent content in DuckDB 2.0 on OLAP Studio — Independent DuckDB Technical Blog</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><lastBuildDate>Sun, 22 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://duckdblab.org/en/tags/duckdb-2.0/index.xml" rel="self" type="application/rss+xml"/><item><title>773 Rules, One Export: Auditing a Production Planning System Offline with DuckDB 2.0</title><link>https://duckdblab.org/en/post/auditing-planning-rules-with-duckdb/</link><pubDate>Sun, 22 Nov 2026 00:00:00 +0000</pubDate><guid>https://duckdblab.org/en/post/auditing-planning-rules-with-duckdb/</guid><description>&lt;img src="https://duckdblab.org/images/posts/auditing-planning-rules-with-duckdb/cover.png" alt="Featured image of post 773 Rules, One Export: Auditing a Production Planning System Offline with DuckDB 2.0" /&gt;&lt;h1 id="773-rules-one-export-auditing-a-production-planning-system-offline-with-duckdb-20"&gt;773 Rules, One Export: Auditing a Production Planning System Offline with DuckDB 2.0
&lt;/h1&gt;
 &lt;blockquote&gt;
 &lt;p&gt;Environment: DuckDB 2.0.0 as the audit engine, running on a laptop. The subject is a production Oracle Hyperion Planning application (11.1.2.4) captured as a single LCM export on 2026-11-20 — no connection to the live system, no data values, configuration only. Member and rule names are anonymized; every count below is the real count from that export, and each one is traceable to the query that produced it. Where a number comes from a method rather than a measurement, the text says so.&lt;/p&gt;

 &lt;/blockquote&gt;
&lt;h2 id="the-question-the-system-itself-cannot-answer"&gt;The question the system itself cannot answer
&lt;/h2&gt;&lt;p&gt;Halfway through a category restructure — the kind where a legacy product tree is retired and rebuilt under new parents — the project hit a question that Hyperion Planning has no screen for:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;If we delete this retired tree, what breaks?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The retired tree held 71 members. The application held 773 calculation rules, 191 forms, and 44 rulesets. Inside the product, impact analysis is one rule at a time: open a script, read it, search the next. Nobody could say, with a straight face, how many of the 773 rules still referenced the 71 members slated for deletion — and a wrong answer in either direction is expensive. Miss a live reference and a production allocation run fails in the middle of the budget cycle. Over-report, and the team burns days re-pointing rules that were never broken.&lt;/p&gt;
&lt;p&gt;So the application was exported through LCM (Lifecycle Management), the export was parsed into relational tables, and the question was answered with SQL. This article is how — and where DuckDB 2.0, specifically, carried weight that a generic &amp;ldquo;load it into a database&amp;rdquo; story would not have.&lt;/p&gt;
&lt;h2 id="step-1--parse-the-export-structurally-or-dont-parse-it-at-all"&gt;Step 1 — Parse the export structurally, or don&amp;rsquo;t parse it at all
&lt;/h2&gt;&lt;p&gt;An LCM export is a folder tree of XML: one file per form, per rule, per dimension. The tempting shortcut is regular expressions over the raw text. We measured what that shortcut costs, on the simplest inventory in the whole audit — which rules are attached to which forms:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Regex over the raw XML: &lt;strong&gt;171 attachments&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Structural XML parse (walk the document tree, read the attachment elements): &lt;strong&gt;187 attachments&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Sixteen attachments — real rules, wired to real forms — are invisible to the regex because of how the XML nests and repeats elements. Every downstream conclusion inherits the parser&amp;rsquo;s error rate, so the pipeline parses structurally (Python&amp;rsquo;s ElementTree) and loads the result into DuckDB as plain tables:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Table&lt;/th&gt;
 &lt;th&gt;Rows&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;rules&lt;/td&gt;
 &lt;td&gt;773&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;forms&lt;/td&gt;
 &lt;td&gt;191&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;members&lt;/td&gt;
 &lt;td&gt;9,600&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;form → rule attachments&lt;/td&gt;
 &lt;td&gt;187&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;rulesets&lt;/td&gt;
 &lt;td&gt;44&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;ruleset steps&lt;/td&gt;
 &lt;td&gt;574&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Two design choices matter here. First, the members table keeps the raw payload of every member alongside the flattened columns — dimension exports are heterogeneous, and the attribute you need is always the one you didn&amp;rsquo;t flatten. Second, the audit database is written in DuckDB&amp;rsquo;s &lt;strong&gt;v1.0 storage format&lt;/strong&gt;, deliberately: 2.0&amp;rsquo;s native files cannot be opened by pre-2.0 clients, and the review team&amp;rsquo;s tooling is not on 2.0 yet. In enterprise work, the newest format is a liability until everyone can read it; the compatible format is the deliverable.&lt;/p&gt;
&lt;h2 id="step-2--the-dead-reference-scan-and-the-trap-inside-it"&gt;Step 2 — The dead-reference scan, and the trap inside it
&lt;/h2&gt;&lt;p&gt;Rule scripts reference members by name, inside FIX blocks and formulas. The scan extracts every member token from every script — &lt;strong&gt;after stripping comments&lt;/strong&gt;, which is its own lesson: Essbase block comments do not nest, the first &lt;code&gt;*/&lt;/code&gt; closes the comment, and a stripper that assumes nesting invents references that exist only inside commented-out code. (Our first-pass tooling made exactly that mistake on a neighbouring check; the corrected scan works on comment-stripped live code only.)&lt;/p&gt;
&lt;p&gt;Join the extracted tokens against the retired tree and the headline appears:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;11 of the 71&lt;/strong&gt; retired members are still referenced&lt;/li&gt;
&lt;li&gt;by &lt;strong&gt;184 rules and 3 forms&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Then the trap. One of those 11 members is a &lt;em&gt;shared member&lt;/em&gt;: it sits inside the retired subtree, and also under a live parent elsewhere in the dimension. A naive subtree query condemns it — and every rule that references it through its live parent. The correct test is reachability: walk the member graph from every live root, and a member counts as dead only if it is in the retired subtree &lt;strong&gt;and&lt;/strong&gt; unreachable from any live root. That is a recursive traversal over the parent/child table — a few lines of recursive CTE, and the difference between a remediation list the business can trust and one that would have re-pointed live rules for no reason. With the shared member correctly excluded, the deliverable was a per-rule remediation list: 216 rule×member rows, each waiting on a human adjudication, because &lt;em&gt;where&lt;/em&gt; a reference should be re-pointed is a business decision, not a query result.&lt;/p&gt;
&lt;p&gt;The same join pattern, run in the other directions, produced the rest of the audit&amp;rsquo;s findings:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;521 of 773 rules are orphans&lt;/strong&gt; — attached to no form, stepped into no ruleset. They cannot execute through any normal path. Classifying one business-critical subset (the allocation chain) by hand put its 165 rules at 33 current / 120 legacy / 12 test: the orphan population is mostly archaeology, but it has to be proven rule by rule, not assumed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two snapshots, differenced as tables.&lt;/strong&gt; A second export taken two days later loads into the same schema, and the change report is a set of joins: forms +7 / −1, rules 0 added / 0 removed / &lt;strong&gt;8 changed&lt;/strong&gt;. During a restructure, &amp;ldquo;what did the last import actually change?&amp;rdquo; stops being a rumour and becomes a query.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="where-20-specifically-earns-its-place"&gt;Where 2.0 specifically earns its place
&lt;/h2&gt;&lt;p&gt;Loading XML-derived tables into SQL is not new, and most of the queries above would run on DuckDB 1.x. Three things about this audit are genuinely 2.0-shaped:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The audit loop was agent-driven.&lt;/strong&gt; The scan was not one query; it was hundreds of probes — a hypothesis, a count, a drill-down, a correction — run conversationally against the database by an AI agent driving the CLI. DuckDB 2.0&amp;rsquo;s &lt;strong&gt;agent mode&lt;/strong&gt; exists for exactly this: compact output instead of padded boxes, errors as JSON, runaway queries cut off early, estimated cost reported before a long query starts. The DuckDB team measured a 59% reduction in output volume an agent has to read on TPC-H; in an audit, that difference is what keeps a 700-rule investigation tractable instead of drowning the loop in its own result sets.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;VARIANT is the right home for what parsing throws away.&lt;/strong&gt; The members table&amp;rsquo;s raw-payload column is currently JSON text — retained, but expensive to interrogate. In 2.0, VARIANT is a first-class type whose common structure is &lt;em&gt;shredded&lt;/em&gt; into columnar storage, so drill-downs against raw attributes (&amp;ldquo;show every member whose original export carried this property&amp;rdquo;) run as column scans instead of text searches. For configuration data — where every object type carries a different attribute set — that is the difference between keeping the raw evidence and quietly dropping it at parse time.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Triggers turn the snapshot diff into a maintained table.&lt;/strong&gt; Today, the two-export comparison is a query re-run per snapshot. With 2.0&amp;rsquo;s &lt;code&gt;CREATE TRIGGER&lt;/code&gt;, loading a new snapshot can maintain the change tables incrementally — inserted, removed, and changed objects recorded as the load happens, in the same transaction shape you would use for an audit log. A restructure that runs for months generates a dozen exports; the diff stops being an event and becomes a ledger.&lt;/p&gt;
&lt;h2 id="where-it-stops"&gt;Where it stops
&lt;/h2&gt;&lt;p&gt;An offline audit has hard edges, and this one hit most of them:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;An export is a photograph, not the system.&lt;/strong&gt; We caught one case where the live outline already contained a member the export did not — added live after the export was taken. Every &amp;ldquo;this member does not exist&amp;rdquo; conclusion carries that caveat; the audit&amp;rsquo;s dead-reference findings were stated as &amp;ldquo;absent from the export &lt;em&gt;and&lt;/em&gt; unreachable in it,&amp;rdquo; and one finding was only settled by a live check.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It reads configuration, not data.&lt;/strong&gt; The audit proves what &lt;em&gt;can&lt;/em&gt; execute and what &lt;em&gt;would&lt;/em&gt; break. Whether the numbers a rule produces are right is a different verification — the one we do with export reconciliation, comparing full data extracts until the diff is zero.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No write-back, ever.&lt;/strong&gt; The deliverables are lists and evidence. Re-pointing a reference, retiring a rule, deleting a tree — those happen in the system, by a person, after adjudication.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Counts are only as honest as the parser.&lt;/strong&gt; The 171-vs-187 gap is the standing warning: every figure in this article descends from a structural parse that was itself verified two ways before any finding was drawn from it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="who-this-is-for"&gt;Who this is for
&lt;/h2&gt;&lt;p&gt;Anyone who owns a large Planning application through a restructure, a migration, or an audit: the questions that matter — &lt;em&gt;what breaks if this goes, what changed since last week, which of these 773 rules can even run&lt;/em&gt; — are set questions over the whole configuration, and the product&amp;rsquo;s own UI answers them one object at a time. Export the configuration, parse it structurally, put it in a real query engine, and the answers become queries with counts attached. The engine being DuckDB 2.0 matters less for the joins than for the working style around them: an agent that can drive the loop, raw evidence kept in a type built for it, and change that maintains itself as snapshots land. That is what &amp;ldquo;enterprise application&amp;rdquo; looks like for an embedded database — not a dashboard, but an audit you can defend line by line.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further reading
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/essbase-duckdb-offline-analytics/" &gt;Offline Essbase analytics with DuckDB&lt;/a&gt; — the same estate, worked from the data side&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/172-million-rows-on-a-laptop/" &gt;172 Million Rows on a Laptop&lt;/a&gt; — browsing and reconciling the full data export&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/duckdb-zero-diff-reconciliation/" &gt;Zero-Diff Reconciliation&lt;/a&gt; — proving a copy of enterprise data is faithful&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/duckdb-2-0-deep-dive/" &gt;DuckDB 2.0 Deep Dive&lt;/a&gt; — the release, feature by feature&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>