Pack Build Optimization

29% faster builds  ·  84% smaller cache  ·  fleet-wide

neeto-deploy  ·  May 2026

Press → to advance  ·  F for fullscreen  ·  ESC for overview

TL;DR

Measured on neeto-planner-web-staging (representative Rails monorepo). Source: ClickHouse logs.app_logs + ECR manifest inspection.

The Problem

1,854 pack builds in 7 days hit the EKS arm-builds nodes:

EXPORT phase was the dominant cost — averaging ~370 s per build with no obvious cause from the surface.

Pack Build — 5 lifecycle phases

For every layer: tar(dir) → gzip → sha256 → upload-or-reuse. Layer size drives the wall-clock cost.

Where time was actually going

Per-phase duration from the before baseline (deployment 7fdc4f7c):

EXPORT alone = 56% of total build time. So we needed to know what was happening inside it.

Bundle-install creates two layers per build

Critical detail: launch-gems is created by copying build-gems, then running bundle install --without development:test --clean true to strip the dev/test gems out.

How the cache flows across builds

Quick win #1 — lifecycle cache duplicate-layer bug

Problem. When a developer deploys an app without changing any dependencies, the build pipeline should reuse the cached gem and asset layers from the previous build — not re-upload them.

But every build — even ones where the only change was app code — was pushing all 9 cache layers to ECR again.

Inspecting the actual :cache image revealed why: its manifest listed 18 layer references, but only 9 unique data blobs existed — every blob was being uploaded twice.

Next slide: the root cause + the one-line fix →

Quick win #1 — cause + fix

Cause. A missing return in neeto-deploy-lifecycle/phase/cache.go. When a layer's SHA matched the previous build's (reuse path), ReuseLayer() ran correctly — but execution then fell through and also ran AddLayerFile(), re-uploading the same blob:

if layer.Digest == previousSHA {
    if err = cache.VerifyLayer(previousSHA); err == nil {
        if err = cache.ReuseLayer(previousSHA); err != nil { /* handle */ }
        // ← MISSING `return` here. Falls through.
    }
}
return layer.Digest, cache.AddLayerFile(layer.TarPath, layer.Digest)
//                   ↑ called even when ReuseLayer already succeeded

Fix. Add return layer.Digest, nil after the successful ReuseLayer call. One line of code.

PR (merged): neeto-deploy-lifecycle#2 · Released as lifecycle:0.2

Quick win #2 — --previous-image launch reuse

Problem. Every build's EXPORT phase tars + gzips + uploads each layer of the app image to ECR. Some of those layers — launcher, config, process-types — come from the CNB lifecycle itself and only change when we bump the lifecycle version (months apart).

But pack had no way to look at the previous build's app image. So it treated every layer as new, re-uploading bytes that were byte-for-byte identical to last week's build. Every build's logs were spammed with Adding layer 'buildpacksio/lifecycle:launcher' when they could have been Reusing layer ….

Fix. In neeto-deploy-slug-compiler-web/.docker/pack-build/build.sh: look up :latest via aws ecr describe-images, then pass it to pack build as --previous-image:

if aws ecr describe-images --repository-name "$APP_IMAGE_REPOSITORY" \
     --image-ids imageTag=latest --region us-east-1 > /dev/null 2>&1; then
  previous_image_args=('--previous-image' "$LATEST_IMAGE_TAG")
fi
pack build "$APP_IMAGE_TAG" "${previous_image_args[@]}" --tag "$LATEST_IMAGE_TAG" …

Effect. The exporter now compares each new layer's SHA against the previous image's manifest. Same SHA → reference the existing blob by digest, skip the upload entirely. Logs flip from Adding layer … to Reusing layer ….

Investigation — the smoking gun

Pulled the :cache ECR image, aggregated uncompressed bytes by category:

Root cause

Upstream Paketo bundle-install in build.go — asymmetric configs:

// BUILD layer install
installProcess.Execute(..., map[string]string{
    "path":  layer.Path,
    "clean": "true",
    // ← NO "without". Installs every group.
})

// LAUNCH layer install
installProcess.Execute(..., map[string]string{
    "path":    layer.Path,
    "without": "development:test",  // ← hardcoded, launch only
    "clean":   "true",
})

First build (no cache) → build-gems gets all gems. Subsequent builds "Reuse cached layer" without re-installing → dev/test gems live in the cache forever.

Launch image was fine. But the cache layer still had to be tarred + gzipped + hashed every export — ~85 s of wasted work per build.

And the bloat reached production app images

Even though launch install runs bundle install --without development:test --clean true:

1. Launch-gems starts as a copy of build-gems (which has dev/test).
2. Pack's --previous-image optimization tells the exporter: "if this layer's SHA matches the previous build's, reference that blob — don't re-upload".
3. When Gemfile.lock doesn't change, the buildpack logs "Reusing cached layer …/launch-gems" → exporter re-uses the same SHA from the previous build's manifest.
4. Result: the polluted launch-gems blob created on the very first build is referenced by every subsequent app image. The 2.5+ GB launch-gems persists in production indefinitely.

The fix — bundle-install:0.9.0

Added BP_BUNDLE_WITHOUT env var (+ RAILS_ENV/RACK_ENV-derived default) honored by both layer installs:

// environment.go
switch railsEnv {
case "production", "staging": return "development:test"
case "development":           return "production:test:staging:heroku"
case "test":                  return "production:development:staging:heroku"
default:                      return ""           // legacy
}

// build.go — same logic now applied to BOTH layers
if environment.BundleWithout != "" {
    buildConfig["without"] = environment.BundleWithout
}

Defaults preserve back-compat. Apps with RAILS_ENV=production (i.e., every neeto product) automatically get the savings.

PR (merged): neeto-deploy-paketo-bundle-install-buildpack#12 · Tracking issue: neeto-deploy-web#7146 · 11 new tests, 0 regressions

Result #1 — cache image size

Layer 5 (build-gems) dropped from 2,652 MB → 125 MB. All other layers unchanged.

Result #2 — per-phase timing

Restoring dropped because the cache image is 84% smaller → less to pull + untar. Export dropped because there's less data to tar+gzip+hash.

Result #3 — fleet-wide impact

Top 5 apps per env, builds before vs after 2026-05-11 14:00 UTC:

Result #4 — production image sizes

3 production apps audited (had bloated 2.5+ GB launch-gems from dev/test gems):

After deleting :latest + :cache and forcing one cold rebuild: combined size 8,315 MB → 2,541 MB (−5.8 GB / −69%).

Result #5 — fleet-wide image savings

Across the apps that have rebuilt since the fleet-wide tag clear, 17 of 72 apps got measurably smaller (the other 55 simply haven't redeployed yet). Top 12 by absolute MB saved:

Why smaller images matter operationally

Image size isn't just a storage line item. Every byte gets pulled by the kubelet onto every node, on every cold start.

Compounding effect on traffic spikes: when an app gets a sudden burst, the autoscaler responds faster because new pods come ready in seconds instead of half a minute. Especially impactful for the 3 worst-offender apps (cal-web, record-web, form-web) which each had ~21 s of unnecessary pull latency on every pod startup.

What we actually shipped

Tracking issue: neeto-deploy-web#7146

What we evaluated and skipped (honestly)

The bundle-install fix already extracted the easy multi-minute wins. Every remaining option had diminishing returns relative to its implementation cost. Knowing when to stop is part of the work.

Artifacts shipped to ECR

348674388966.dkr.ecr.us-east-1.amazonaws.com/
├── neeto-deploy/paketo/lifecycle:0.2          (cache duplicate-layer fix)
│   ├── 0.2-amd64    sha256:8d6cd7…
│   └── 0.2-arm64    sha256:d79c82…
│
├── neeto-deploy/paketo/lifecycle:0.3          (SBOM placeholder files)
│   └── multi-arch   sha256:17637a…
│
├── neeto-deploy/paketo/buildpack/bundle-install:0.9.0    ← FLAGSHIP
│   ├── 0.9.0-amd64
│   ├── 0.9.0-arm64
│   └── multi-arch   sha256:736355…
│
└── neeto-deploy/paketo/buildpack/ruby:0.47.21
    └── multi-arch   sha256:fffc12…   (composite — pulls bundle-install:0.9.0)

Bottom line

Thanks 🙌

Questions?

Detailed report: pack-build-optimization-results.md
Deck source: github.com/vishal24367/pack-build-optimization-deck
Gist: gist.github.com/vishal24367/e06ad…