<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Data Validation on OLAP Studio — Independent DuckDB Technical Blog</title><link>https://duckdblab.org/en/tags/data-validation/</link><description>Recent content in Data Validation on OLAP Studio — Independent DuckDB Technical Blog</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><lastBuildDate>Sat, 21 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://duckdblab.org/en/tags/data-validation/index.xml" rel="self" type="application/rss+xml"/><item><title>Zero-Diff Reconciliation: Proving Your DuckDB Copy of Enterprise Data Is Actually Correct</title><link>https://duckdblab.org/en/post/duckdb-zero-diff-reconciliation/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><guid>https://duckdblab.org/en/post/duckdb-zero-diff-reconciliation/</guid><description>&lt;img src="https://duckdblab.org/images/posts/duckdb-zero-diff-reconciliation/cover.png" alt="Featured image of post Zero-Diff Reconciliation: Proving Your DuckDB Copy of Enterprise Data Is Actually Correct" /&gt;&lt;p&gt;&lt;img alt="Zero-Diff Reconciliation: the five-step field method for proving a DuckDB replica matches its source, with all check cells at 0.00" loading="lazy" sizes="(max-width: 767px) calc(100vw - 30px), (max-width: 1023px) 700px, (max-width: 1279px) 950px, 1232px" src="https://duckdblab.org/images/posts/duckdb-zero-diff-reconciliation/cover.png"&gt;&lt;/p&gt;

 &lt;blockquote&gt;
 &lt;p&gt;Environment: DuckDB v2.0.0 (Python). Every timing and every aggregate below is an unedited measured output from a two-part POC: a synthetic Essbase-shaped dataset (Sample Basic outline, 17,068 level-0 rows) and a scaled variant (5,281,142 rows, 317 MB export). The data is synthetic with independently generated ground truth; the pipeline, the semantics, and the timings are real. The application side of this POC is documented in our &lt;a class="link" href="https://duckdblab.org/en/post/essbase-duckdb-offline-analytics/" &gt;Essbase x DuckDB pipeline article&lt;/a&gt;. This piece is the method, generalized so it applies to any system of record you pull offline.&lt;/p&gt;

 &lt;/blockquote&gt;
&lt;h2 id="the-question-nobody-asks"&gt;The question nobody asks
&lt;/h2&gt;&lt;p&gt;When a team copies enterprise data into DuckDB for offline analysis — a cube export, an ERP extract, a warehouse pull — the reflex is to benchmark: ingest time, Parquet size, query latency. And those numbers matter. But speed is not the risk.&lt;/p&gt;
&lt;p&gt;The risk is a &lt;strong&gt;silent misalignment&lt;/strong&gt;: one hierarchy edge dropped in the export, a shared member that double-counts into a total, a scenario filter that quietly excludes half the data. And because the copy answers faster than the original system ever did, nobody notices. Wrong answers just arrive on time.&lt;/p&gt;
&lt;p&gt;The zero-diff method exists to answer the only question that actually matters:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Can you prove the replica matches the system of record, cell by cell, with a difference of exactly zero — and when it is not zero, can you locate the difference to the precise members and scenario that caused it?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;A difference of zero is &lt;em&gt;provably&lt;/em&gt; zero. A real difference is &lt;em&gt;locatable&lt;/em&gt;. Everything else is a guess.&lt;/p&gt;
&lt;h2 id="the-five-principles"&gt;The five principles
&lt;/h2&gt;&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Export at the atomic grain.&lt;/strong&gt; Verify with the finest stored data — level-0 / leaf members, full member intersections. An aggregated pull cannot be drilled down; when an aggregate is off, you cannot tell why.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hierarchy is data.&lt;/strong&gt; In a cube, rollups follow the outline. In SQL, the outline becomes tables. Every rollup is a join plus a GROUP BY, and semantics follow the data instead of a black box.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Independent ground truth.&lt;/strong&gt; The check has to come from a different code path than the replica — the source system&amp;rsquo;s own reports, or a ground truth generated independently from the raw export. A replica grading itself proves nothing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The zero-diff bar.&lt;/strong&gt; Every check cell must tie out to exactly zero. Not &amp;ldquo;within rounding.&amp;rdquo; A difference of 0.004 is a difference, and it gets localized like any other.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Separate semantics from throughput.&lt;/strong&gt; Prove correctness on a small sample first, then prove scale. They are different failure classes and need different tests.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="step-1--atomic-export-no-import-program"&gt;Step 1 — Atomic export, no import program
&lt;/h2&gt;&lt;p&gt;The source side writes its finest stored grain to a flat file: in Essbase, a &lt;code&gt;DATAEXPORT&lt;/code&gt; calc script (or MaxL) emits one full member intersection plus its value per line. The DuckDB side has no import program at all — &lt;a class="link" href="https://duckdb.org/docs/stable/data/csv/overview" target="_blank" rel="noopener"
 &gt;&lt;code&gt;read_csv&lt;/code&gt;&lt;/a&gt; turns the text into a table, with quote character and delimiter as parameters:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-sql" data-lang="sql"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;TABLE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;read_csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;basic_level0.txt&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39; &amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;#34;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;VARCHAR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Measures&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;VARCHAR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Product&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;VARCHAR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Market&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;VARCHAR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Scenario&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;VARCHAR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Value&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;DOUBLE&amp;#39;&lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;-- 17,068 rows, 0.04 s measured
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That is the entire ingest. Atomic grain, one file, zero tooling.&lt;/p&gt;
&lt;h2 id="step-2--hierarchy-as-data"&gt;Step 2 — Hierarchy as data
&lt;/h2&gt;&lt;p&gt;A fact table alone cannot reproduce cube semantics, because rollups follow the outline hierarchy. The move is to make the hierarchy into data — two small tables:&lt;/p&gt;
&lt;pre tabindex="0"&gt;&lt;code&gt;product_hier(member, parent, rollup) -- the `rollup` column is the subtle one
market_hier(member, parent)
&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;rollup&lt;/code&gt; column exists because of &lt;strong&gt;alternate rollups&lt;/strong&gt;: in Sample Basic, products 100-20, 200-20 and 300-30 belong to their primary categories &lt;em&gt;and&lt;/em&gt; roll up once more under &amp;ldquo;Diet&amp;rdquo;. A member simply appears twice in the hierarchy table, and any rollup becomes a join plus a GROUP BY:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-sql" data-lang="sql"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;JOIN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;product_hier&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;ON&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;member&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;rollup&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;primary&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Measures&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Sales&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Scenario&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Actual&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;GROUP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;BY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;BY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;One rule carries the whole design: &lt;strong&gt;never let an alternate rollup leak into the primary chain&lt;/strong&gt;, or totals get double-counted — the same discipline as keeping stored and shared members straight inside the cube.&lt;/p&gt;
&lt;h2 id="step-3--ground-truth-and-the-zero-diff-bar"&gt;Step 3 — Ground truth and the zero-diff bar
&lt;/h2&gt;&lt;p&gt;Now the check itself. Every check cell is an aggregate the source system can also produce, computed on two independent paths and diffed:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Check cell&lt;/th&gt;
 &lt;th&gt;DuckDB replica&lt;/th&gt;
 &lt;th&gt;Difference vs. ground truth&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;Category 100 (primary rollup)&lt;/td&gt;
 &lt;td&gt;1,745,240.44&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Category 200&lt;/td&gt;
 &lt;td&gt;2,558,852.13&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Category 300&lt;/td&gt;
 &lt;td&gt;1,883,168.76&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Category 400&lt;/td&gt;
 &lt;td&gt;1,869,248.40&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Overall Sales total&lt;/td&gt;
 &lt;td&gt;8,056,509.73&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Diet (alternate rollup)&lt;/td&gt;
 &lt;td&gt;1,782,909.84&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Profit (formula member, recomputed in SQL)&lt;/td&gt;
 &lt;td&gt;2,152,890.95&lt;/td&gt;
 &lt;td&gt;0.00&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Seven cells, two of them in the specialty territory (an alternate rollup and a formula member), all at exactly zero. That step is the credibility foundation of the whole route: once the semantics tie out, everything downstream is trustworthy.&lt;/p&gt;
&lt;p&gt;When a cell does not tie, the method has a standing query: drill from the failed aggregate to the level-0 intersections that compose it.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-sql" data-lang="sql"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;-- failed cell: California / Actual / Sales
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;Month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Measures&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Scenario&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Market&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;California&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Scenario&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Actual&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Measures&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Sales&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The drill-down always terminates at concrete members. A difference that localizes is a difference you can fix; one that cannot is an export problem, and that is also known — which is the point.&lt;/p&gt;
&lt;h2 id="step-4--the-failure-modes-and-their-locators"&gt;Step 4 — The failure modes, and their locators
&lt;/h2&gt;&lt;p&gt;From the POC, the failure classes that actually occur, in order of how often they bite:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Failure mode&lt;/th&gt;
 &lt;th&gt;Symptom&lt;/th&gt;
 &lt;th&gt;Locator&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;Shared/alternate member leaks into the primary chain&lt;/td&gt;
 &lt;td&gt;A total is double-counted, clean multiple&lt;/td&gt;
 &lt;td&gt;List members appearing in both rollup chains: &lt;code&gt;SELECT member, count(DISTINCT rollup) FROM product_hier GROUP BY 1 HAVING count(*) &amp;gt; 1&lt;/code&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Formula member absent from the export&lt;/td&gt;
 &lt;td&gt;A check cell is simply missing, not off&lt;/td&gt;
 &lt;td&gt;Profit is not stored at level 0; recompute it in SQL (&lt;code&gt;sum(Value * CASE Measures WHEN 'Sales' THEN 1 ELSE -1 END)&lt;/code&gt;). Every formula account you rely on needs a maintained SQL definition&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Scenario omitted from the export&lt;/td&gt;
 &lt;td&gt;Rollups tie out &lt;em&gt;confidently and completely&lt;/em&gt; wrong&lt;/td&gt;
 &lt;td&gt;&lt;code&gt;SELECT DISTINCT Scenario&lt;/code&gt; against the expected set — one missing scenario shifts every total&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Decimal/format drift&lt;/td&gt;
 &lt;td&gt;Small, systematic differences on many cells&lt;/td&gt;
 &lt;td&gt;Max absolute diff per cell; sub-unit differences that are uniform point at formatting, not logic&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Note the second row: formula members mark the semantic boundary of the route. However many formula accounts the source has, that many definitions must be maintained on the SQL side. With a handful, a few CASE expressions; with hundreds, a definition layer — and that layer, not the pipeline, is the real productization cost.&lt;/p&gt;
&lt;h2 id="step-5--then-scale-separately"&gt;Step 5 — Then scale, separately
&lt;/h2&gt;&lt;p&gt;Correctness and throughput are tested on different data. The scaled POC adds an Entity dimension with 100 organizations, 200 SKUs, keeps markets at region level: 5,281,142 level-0 rows, a 317 MB export text file. Measured on the same machine:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Step&lt;/th&gt;
 &lt;th&gt;Result&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;Ingest export text into DuckDB&lt;/td&gt;
 &lt;td&gt;5.77 s&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Convert to &lt;a class="link" href="https://duckdb.org/docs/stable/data/parquet" target="_blank" rel="noopener"
 &gt;Parquet&lt;/a&gt; (ZSTD)&lt;/td&gt;
 &lt;td&gt;2.33 s, 317 MB to 21 MB (14.0x)&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Full rollup (total Sales)&lt;/td&gt;
 &lt;td&gt;0.01 s&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Top-10 organizations by Sales&lt;/td&gt;
 &lt;td&gt;0.01 s&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Actual vs. Budget variance, all products&lt;/td&gt;
 &lt;td&gt;0.02 s&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Variance analysis is the most common job in this kind of offline work — in the source it is a dedicated report; here it is one aggregation with &lt;code&gt;FILTER&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-sql" data-lang="sql"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Product&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FILTER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Scenario&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Actual&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FILTER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Scenario&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Budget&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;var&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fact_scaled&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Measures&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Sales&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;GROUP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;BY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;BY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;var&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;LIMIT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;-- largest variance: SKU-0057, Actual 1,553,495.11 below Budget
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The scale pass runs the &lt;em&gt;same&lt;/em&gt; zero-diff check family on the bigger export. Throughput proves nothing about correctness, and correctness proves nothing about throughput — that separation is why the method has two distinct passes.&lt;/p&gt;
&lt;h2 id="the-one-page-checklist"&gt;The one-page checklist
&lt;/h2&gt;&lt;ol&gt;
&lt;li&gt;Export at the atomic grain (level 0 / leaf, full member intersections).&lt;/li&gt;
&lt;li&gt;Model the hierarchy as data: &lt;code&gt;(member, parent, rollup)&lt;/code&gt; — with &lt;code&gt;rollup&lt;/code&gt; distinguishing alternate chains.&lt;/li&gt;
&lt;li&gt;Generate ground truth on an independent path (source reports, or generated alongside the synthetic data).&lt;/li&gt;
&lt;li&gt;Build the check-cell family: every top-level rollup, every alternate rollup, every formula member you use, and the grand total.&lt;/li&gt;
&lt;li&gt;Enforce the zero-diff bar; any non-zero cell gets the drill-down query and a recorded locator.&lt;/li&gt;
&lt;li&gt;Only then: the scale pass — same checks on a large export, plus ingest / Parquet / query timings.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="when-not-to-use-the-method"&gt;When not to use the method
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Data with live security semantics.&lt;/strong&gt; Permissions and meta-read filters do not travel with an export file; the offline copy needs its own masking and access control before it leaves the person who did the export.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Thousands of formula accounts.&lt;/strong&gt; The SQL definition layer becomes the maintenance product; the pipeline is only the start of it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Write-back expectations.&lt;/strong&gt; This method proves a read-only copy is true. Entry and official calculation stay in the system of record; the replica is for looking, testing, and diffing — not for writing.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion
&lt;/h2&gt;&lt;p&gt;Zero-diff reconciliation is not a test you run once; it is the operating model for any offline copy of authoritative data. Export the atomic grain, make the hierarchy into data, grade against an independent path, hold the zero bar, localize every miss, and keep semantics and scale on separate test benches. Do that and the copy earns trust the way the source already has: by tying out, cell by cell, to exactly zero.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further Reading
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/essbase-duckdb-offline-analytics/" &gt;Essbase x DuckDB: offline analytics for a classic MOLAP cube&lt;/a&gt; — the application pipeline this method generalizes from&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdblab.org/en/post/duckdb-fts-memory-recall/" &gt;DuckDB FTS + BM25 memory recall, hands-on&lt;/a&gt; — the same &amp;ldquo;measure, don&amp;rsquo;t assume&amp;rdquo; discipline applied to a different problem&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://duckdb.org/docs/stable/data/csv/overview" target="_blank" rel="noopener"
 &gt;&lt;code&gt;read_csv&lt;/code&gt;&lt;/a&gt; and &lt;a class="link" href="https://duckdb.org/docs/stable/data/parquet" target="_blank" rel="noopener"
 &gt;Parquet&lt;/a&gt; in the DuckDB documentation&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>