Spark Optimization & Capacity Planning

Paste a PySpark job or a Spark SQL query to find what makes it slow, size a cluster for your SLA, and forecast when storage runs out.

Your code

Nodes Cores per node Memory per node (GB) Input data (GB) Job type SLA (minutes)

Runtime is a rough throughput model. Calibrate it with timings from your own jobs.

Forecast (months) Retention (days) Compression ratio Replication Capacity (TB) Warn at (% full)

Uses 18 months of sample daily ingestion. The tool fits linear and exponential growth and keeps whichever predicts held-out data better.