The rule engine doesn't compare "your dbt config" against "your dbt metrics" — it compares any pipeline's config against its own run history and proposes a specific, reasoned diff. That pattern isn't dbt-specific. Here's where it already stands per framework, and what Spark and Flink support would actually take.
Where each framework actually stands
Three different levels of "done," stated plainly so you don't have to guess which is which.
Reads your actual target/run_results.json — no synthetic data. Two rules today:
thread count vs. run duration, and models/tests per thread.
The only framework here backed by a real artifact reader, which is why it's the one this site leads with.
Actually has the most rule coverage — 5 of the 7 rules in the engine: executor memory (both directions), shuffle partitions, and cluster instance count (both directions).
What's missing isn't the logic, it's the input: metrics are still hand-typed sample data. No Spark REST API or event-log reader exists yet.
Nothing built. No rules, no parser, no fixtures. Proposed here because its tuning surface overlaps heavily with Spark's — see below — not because it's easy.
If you run Flink and this sounds useful, that's exactly the signal worth telling me about before any of it gets written.
Why Flink specifically, not just "more integrations"
Flink and Spark are both JVM cluster-compute engines with well-known, metric-driven tuning knobs — which is exactly the shape this rule engine already knows how to work with.
The honest version of "why Flink": checkpoint interval and state-backend sizing, parallelism vs. backpressure, and watermark/lateness tuning are Flink's equivalents of Spark's executor memory and shuffle partitions — the same "config drifted from what the workload needs" problem, wearing different config keys. That's a reason to believe the pattern transfers, not a claim that it already has.
What it would actually take
Neither of these is a rules problem. Both are a data-access problem, same as Spark's gap today.
A live reader against the Spark REST API / History Server (higher friction — needs a
driver on port 4040 or a reachable History Server most solo/small setups don't expose),
or an uploaded event log — self-serve like dbt's run_results.json, narrower
in what it can see, but no live cluster required.
Flink's own REST API (/jobs/:id/checkpoints, task manager metrics) or its
metrics reporter output (Prometheus/StatsD-style). Structurally the same problem as
Spark's — a live endpoint vs. an uploaded snapshot — just not scoped or built yet.
A different idea entirely, held to the same standard
Not a framework extension like Spark or Flink — a different axis entirely. Worth writing down honestly, including why it isn't started.
Why this one gets extra caution, not just the usual "not built yet": a wrong latency suggestion degrades gracefully - you notice, you revert. A missed security issue is silent until it's exploited, and a "security checked" label risks making someone *less* careful, not more, precisely because they trust the label. Part of this (patterns like the two above) is genuinely cheap to build - same shape as the existing rules, no new integration needed. That's a reason it's technically feasible, not a reason it's ready: the honest version would ship as narrow, named checks with an explicit, visible list of what each one does and doesn't cover - never a general "security checked" claim - and only once there's a real reason to believe it's worth the risk of getting wrong.
Nothing on this page is built. It exists to show the shape of where this could go — and to ask, before any of it gets written, whether it's actually worth building for you.