Engineers automating a disaster-recovery islanding exercise stored thirty-second service metrics as Parquet in S3-compatible storage, aggregated them with DuckDB, and let an AI agent classify failover outcomes. The prototype produced structured reports without a shared lakehouse or catalog, yet the query engine did not disappear. It became an embedded component inside the agent workflow. The lesson matters because teams now face a choice between lightweight agent pipelines and managed data platforms.
How the DR analysis pipeline worked
During the exercise one region was intentionally isolated, with traffic expected to fail over to surviving regions. For every service the team had to verify whether traffic shifted, whether success rate and latency held, and how the outcome should be classified. Raw metrics were exported, transformed into self-describing Parquet assets with manifests, and kept in versioned directories per exercise. The agent then selected inputs, generated constrained DuckDB SQL or Python, and orchestrated execution to answer the three operational questions.
The architecture used three logical layers: data preparation, deterministic computation, and agent-driven interpretation, with reproducibility as a cross-cutting requirement. DuckDB operated on both sides of storage, reading raw records during ingestion, sorting and compressing them, and generating validation statistics for the manifest. During analysis it filtered time windows, joined service metadata, and computed compact fact packets with traffic volumes, timestamps, percentage changes, and SLO status. Reusable build logic was generated once and reused, while inputs, facts, configuration, and execution metadata were retained for replay.
Control came from a deterministic validation layer placed between generation and execution. The agent produced only read-only queries over predefined tables and time ranges, which were checked against an allowlist of datasets and operations. Approved queries ran in a sandboxed analysis runtime, and application code normalized rows into a fact packet rather than letting the model estimate values. The authors stress that DuckDB was not presented as universally better than Spark, Trino, or a managed lakehouse. The workload simply did not yet require a shared distributed engine.
What this means for data platform decisions
For companies adopting AI analytics, the pattern lowers the entry cost of post-incident and exercise analysis. A small team can keep Parquet files in object storage, reuse embedded aggregation logic, and ask an agent to produce explanations without operating a catalog or cluster. A large organization running repeated exercises across many services can standardize manifests, validation, and fact packets once, then scale interpretation. In both cases the visible interface changes from dashboards and hand-written queries to a request answered with evidence.
The constraint is that judgment remains trustworthy only when facts underneath it are deterministic. A QPS increase may be too small to absorb lost traffic, lower success rates may include expected business rejections, and a latency spike may be harmless if steady state returns quickly. The agent resolved such conflicts by comparing transition and steady-state windows, service tier, dependency saturation, and backlog behavior. This news does not mean SQL, tables, or governance are obsolete, and buyers should ask what is allowlisted, sandboxed, versioned, and independently verifiable before production use.
The marker to watch is the authors’ determinism split in wider adoption: exact computation stays in engines and code, contextual reasoning moves to models. If more teams publish agent pipelines with embedded engines, manifests, and four decision signals for upgrading to a managed lakehouse, the separation will harden into practice. If reproducibility and access controls lag, enterprises will keep centralized platforms as the default for regulated analysis.
