opti-pipe — Data Pipeline Optimizer

The problem

Manual tuning is a tax every data team pays

Data engineers waste hours guessing at executor memory, batch sizes, and cluster size — and get it wrong in both directions.

Runaway cloud costs

Over-provisioned clusters and executor memory sit idle, burning budget on capacity nobody is using — and nobody notices until the bill arrives.

Failures & latency spikes

Under-provisioned memory and unoptimized parallelism cause OOM kills and queue delay — usually discovered in production, at the worst time.

What it watches

Four vectors, pulled from your actual runs

opti-pipe reads config files alongside execution telemetry — no need to eyeball dashboards across four different tools.

Latency

Execution duration

Stage/job completion time and queue delay, tracked per run to catch bottlenecks before they compound.

Throughput

Records / sec

Processing rate derived per run, so scaling issues show up before a pipeline falls behind its SLA.

Resource utilization

CPU, heap, GC pauses

Executor heap usage, allocated vs. actual CPU, and GC pause time — the signals that separate "wasteful" from "about to OOM."

Cloud cost

Cost per run

Instance type, cluster size, and schedule combine into a monthly cost estimate — and a savings number for every fix.

How it works

$ opti-pipe check spark-daily-etl-orders # pipeline already ran on schedule - nothing new to trigger here # comparing last 5 runs against the live config... rule 01 fired: spark.executor.memory 16g -> 13g $15.09/mo savings, +0.4% latency # diff, rationale, and impact are all above - nothing has changed yet $ _ waiting for your approval

I don't trust an agent to touch a production Spark config unattended, so it doesn't. You approve every change in the dashboard below, or nothing happens.

opti-pipe dashboard: pipeline list with health badges and estimated cost/savings

Pipeline health at a glance — cost, savings, and active recommendations per pipeline.

opti-pipe recommendation detail view: current vs suggested config diff with rationale

Every recommendation shows the diff, the rationale, and the estimated impact — before you click Approve. The +$34.06/mo and +$45.41/mo above are real, computed live from the bundled sample pipeline — rerun the same file yourself and you'll get the same numbers. Fixing a real OOM risk isn't free, and it says so upfront instead of hiding it.

On the backlog

Gated behind an actual ask, not forgotten

A live Spark reader (REST API / History Server), Flink-specific tuning rules (the integration itself just shipped, ahead of rules that actually use its checkpoint/backpressure numbers), and a way to define your own rules are all logged and waiting. None of them get built ahead of demand — that's exactly how the Spark event-log reader itself went from idea to shipped: gated until enough people asked for it by name.

Need something that's not on that list? A rule, an integration, anything — tell me what and why, and leave an email so it's a real request, not a bot. That's the actual signal that decides what gets built next, not a guess on my end.

# eleven deterministic rules, no model in between

$ grep -A1 "^def _rule" analyzer.py # eleven of them. what they check and the exact thresholds stay in the # code, not on this page — run it on the sample data and judge it by # what it actually recommends.

The one I actually needed handles OOM/GC pressure. Some Spark job would randomly OOM, I'd lose an evening staring at heap and GC numbers trying to work out what changed, and eventually I got tired of doing it by hand and wrote this instead. If it's useful to you too, good — if not, tell me why, that's worth more to me than a compliment.
— I'm the only person who works on this

No auto-apply. Some optimization tools have an "autopilot" mode that applies changes to your cluster after every run. I didn't build one — there's no switch to turn it on. GET /recommendations recomputes live from the current config and metrics on every call, so there's no hidden state that could drift from what you last approved.

Optional AI layer

A second opinion, on demand

The eleven rules above stay free, deterministic, and fully unit-tested — that doesn't change. On top of them, an optional "Ask AI" button sends your metrics and the rules' own output to a real Claude model for a plain-English critique: the same natural-language second-opinion idea larger platforms charge enterprise pricing for, here as one button with your own API key.

Summary

Plain-English health check

A few sentences on this pipeline's overall health, written from its actual run metrics.

Critique

Checks the rules' own work

Does the deterministic engine's recommendation actually look right given the numbers? The model is explicitly asked to disagree if it should.

Observations

What eleven thresholds miss

Trends across runs, outliers, correlations between metrics — patterns no fixed threshold rule is built to catch.

This part costs real money per click, so I made it optional instead of automatic. Each click is a billed Claude API call, using your own Anthropic API key (a claude.ai subscription doesn't cover API access). Nothing else on this page needs one, and everything else works the same whether or not it's configured.

Under the hood

Small, inspectable, no infra to stand up

One FastAPI process. No Docker, no message queue, no external service to configure before you can see it work.

FastAPI Python 3 pytest — 200+ tests Vanilla JS dashboard Spark configs (YAML) dbt configs (YAML) No Docker required

🔒 Your uploaded data is anonymized by design. Only numeric metrics — timing, row counts, heap/memory percentages — are ever extracted from your file. Never your SQL, model names, DataFrame code, or file paths. And it's private to your browser only: no account, no other visitor can ever see it.

See what it finds in your pipelines.

Explore the live sample data, or add your own Spark, dbt, or Flink pipeline and see real recommendations.

Open the dashboard →