Spark & dbt · human-in-the-loop by design

Stop tuning data pipeline configs at 2am.

opti-pipe watches your pipeline runs, compares them against your configs, and tells you exactly what's wasting money or risking a failure — with a diff you can read and a number you can trust. Nothing ships without your click.

See it on live sample data →

Not ready to try it today? Get notified when this reaches production.

7Tuning rules
2Frameworks: Spark + dbt
0Auto-applied changes
99Tests backing the engine
The problem

Manual tuning is a tax every data team pays

Data engineers waste hours guessing at executor memory, batch sizes, and cluster size — and get it wrong in both directions.

Runaway cloud costs

Over-provisioned clusters and executor memory sit idle, burning budget on capacity nobody is using — and nobody notices until the bill arrives.

Failures & latency spikes

Under-provisioned memory and unoptimized parallelism cause OOM kills and queue delay — usually discovered in production, at the worst time.

What it watches

Four vectors, pulled from your actual runs

opti-pipe reads config files alongside execution telemetry — no need to eyeball dashboards across four different tools.

Latency

Execution duration

Stage/job completion time and queue delay, tracked per run to catch bottlenecks before they compound.

Throughput

Records / sec

Processing rate derived per run, so scaling issues show up before a pipeline falls behind its SLA.

Resource utilization

CPU, heap, GC pauses

Executor heap usage, allocated vs. actual CPU, and GC pause time — the signals that separate "wasteful" from "about to OOM."

Cloud cost

Cost per run

Instance type, cluster size, and schedule combine into a monthly cost estimate — and a savings number for every fix.

How it works

$ opti-pipe check spark-daily-etl-orders # pipeline already ran on schedule - nothing new to trigger here # comparing last 5 runs against the live config... rule 01 fired: spark.executor.memory 16g -> 13g $15.09/mo savings, +0.4% latency # diff, rationale, and impact are all above - nothing has changed yet $ _ waiting for your approval

I don't trust an agent to touch a production Spark config unattended, so it doesn't. You approve every change in the dashboard below, or nothing happens.

opti-pipe dashboard: pipeline list with health badges and estimated cost/savings
Pipeline health at a glance — cost, savings, and active recommendations per pipeline.
opti-pipe recommendation detail view: current vs suggested config diff with rationale
Every recommendation shows the diff, the rationale, and the estimated impact — before you click Approve. The +$34.06/mo and +$45.41/mo above are real, computed live from the bundled sample pipeline — rerun the same file yourself and you'll get the same numbers. Fixing a real OOM risk isn't free, and it says so upfront instead of hiding it.

# seven deterministic rules, no model in between

$ grep -A1 "^def _rule" analyzer.py def _rule_overprovisioned_memory(...) ## 01 cost low heap, no OOM history -> shrink spark.executor.memory def _rule_oom_or_gc_pressure(...) ## 02 reliability OOM errors or high GC pause -> grow executor memory before it fails again def _rule_shuffle_partitions(...) ## 03 latency too many tiny tasks, or too few slow ones -> resize spark.sql.shuffle.partitions def _rule_overprovisioned_cluster(...) ## 04 cost low CPU, no queue delay -> shrink executor instance count def _rule_queue_delay(...) ## 05 latency jobs waiting on capacity -> grow executor instance count def _rule_dbt_threads(...) ## 06 latency low threads, slow runs -> raise DAG parallelism def _rule_dbt_tasks_per_thread(...) ## 07 latency too many models/tests per thread, even if not slow yet -> raise threads early

Rule 02 is the one I actually needed. Some Spark job would randomly OOM, I'd lose an evening staring at heap and GC numbers trying to work out what changed, and eventually I got tired of doing it by hand and wrote this instead. If it's useful to you too, good — if not, tell me why, that's worth more to me than a compliment.
— I'm the only person who works on this

No auto-apply. Some optimization tools have an "autopilot" mode that applies changes to your cluster after every run. I didn't build one — there's no switch to turn it on. GET /recommendations recomputes live from the current config and metrics on every call, so there's no hidden state that could drift from what you last approved.

Optional AI layer

A second opinion, on demand

The seven rules above stay free, deterministic, and fully unit-tested — that doesn't change. On top of them, an optional "Ask AI" button sends your metrics and the rules' own output to a real Claude model for a plain-English critique: the same natural-language second-opinion idea larger platforms charge enterprise pricing for, here as one button with your own API key.

Summary

Plain-English health check

A few sentences on this pipeline's overall health, written from its actual run metrics.

Critique

Checks the rules' own work

Does the deterministic engine's recommendation actually look right given the numbers? The model is explicitly asked to disagree if it should.

Observations

What seven thresholds miss

Trends across runs, outliers, correlations between metrics — patterns no fixed threshold rule is built to catch.

This part costs real money per click, so I made it optional instead of automatic. Each click is a billed Claude API call, using your own Anthropic API key (a claude.ai subscription doesn't cover API access). Nothing else on this page needs one, and everything else works the same whether or not it's configured.

Under the hood

Small, inspectable, no infra to stand up

One FastAPI process. No Docker, no message queue, no external service to configure before you can see it work.

FastAPI Python 3 pytest — 99 tests Vanilla JS dashboard Spark configs (YAML) dbt configs (YAML) No Docker required
What this doesn't do yet

The honest gaps, not just the honest scope

Saying "no auto-apply" is the easy part to be honest about. Here's what's actually still missing, so you can check it against your own pipeline before you rely on this for anything.

Spark has no real integration

dbt does — it reads an actual run_results.json from your own dbt run. Spark metrics are still hand-typed sample data; there's no Spark REST API or event-log reader yet. What that would take, and why Flink's next →

Seven rules, not everything that breaks a pipeline

Memory sizing, shuffle partitions, instance count, dbt thread count (both by duration and by model-count-per-thread). Nothing here catches data skew, partition pruning, file sizing, or retry storms — that needs more rules, not a different architecture, and I haven't written them yet.

Nothing you upload is stored anywhere permanent

A pipeline you add and any run data you upload live in memory only — no database, nothing written to disk — and disappear the moment the server restarts. Click "Ask AI" and that pipeline's metrics go to Anthropic's API for that one request, nowhere else.

One shared demo, not per-company isolation

Everyone using the same link sees the same in-memory state right now. Fine for one person trying it out; not built yet for multiple teams using it at once.

See what it finds in your pipelines.

Explore the live sample data, or add your own dbt pipeline and see real recommendations.

Open the dashboard →