NeetoCI Test Optimization

~3× faster runs  ·  677 hrs/month freed  ·  fleet-wide

neeto-ci-web  ·  epic #3799  ·  May 2026

Press → to advance  ·  F for fullscreen  ·  ESC for overview

TL;DR

Cal-web: full pre/post measurement on add-minitest-distributed (4,024 tests). 8-app measurement: avg of block-pipeline runs (test_pods > 0) from production ci_jobs since 2026-05-19 15:30; baseline = 7–13 May p50.

The Problem

12,068 default.yml CI runs in 30 days across 100 products; the largest suites dominate:

A successful neeto-cal-web run takes ~20 min today. That's ~580 hours of CI/month on one product alone — and every dev waits on it.

Old architecture — one pod, serial commands

Every command runs in a single Kubernetes pod. No fan-out, no parallelism, no shared cache between blocks. EBS root volume is the only filesystem.

Where time was actually going

Per-step p50 · p95 across 18 pre-optimization neeto-cal-web default.yml runs (7–13 May 2026):

rails test ≈ 65% of in-command time (p50). rake setup, cache restore and postgres startup are the next-largest — and every one of them repeats on each parallel pod.

The architecture rewrite — blocks pipeline (epic #3799)

A CI job is now a DAG of blocks. Each block has named jobs; each job is its own K8s job; pods within a job can fan out.

Block 1 (Setup & Checks) runs 2 K8s jobs in parallel. Once it succeeds, Block 2 and Block 3 fan out together (both depend only on Block 1). Block 3's Run tests job is itself 4 K8s pods coordinated via a shared Redis queue.

New YAML format

global_job_config:                # runs on EVERY pod, before everything
  env_vars: [{ name: TZ, value: UTC }]
  prologue:
    commands: [neetoci-version ruby 4.0.1, checkout, cache restore, bundle install]
blocks:
  - name: Setup & Checks
    dependencies: []                # entry block
    task:
      jobs:                         # each entry = its own K8s job
        - { name: Install + ESLint,  commands: [...] }
        - { name: Auditors + Linters, commands: [...] }
  - name: Tests
    dependencies: [Setup & Checks]  # fan-out from Block 1
    task:
      prologue:                     # block-level prologue, runs once per pod
        commands: [neetoci-service start postgres 18, ...]
      jobs:
        - name: Run tests
          commands: [bundle exec rails test]
          parallelism: 4
          shared_redis: true        # minitest-distributed coordinator
      epilogue:
        always: { commands: [bundle exec rake simplecov_coverage:publish] }

Fully backward compatible. The parser falls back to legacy behavior when task:, dependencies:, or global_job_config: are absent — old flat commands: configs still work unchanged.

Quick win #1 — pgvector apt-installed at runtime, on every pod

Problem. Every CI pod that started Postgres ran apt-get update && apt-get install postgresql-N-pgvector against the public pgdg repo. Two index updates + a 12 MB package install, ~6–10 s per pod, multiplied by every test pod in the run.

# docker-ci/utils/neetoci-service (pre-fix)
sudo podman exec postgres bash -c \
  "apt-get update -qq && apt-get install -y -qq postgresql-${pg_major}-pgvector"

Fix. Rebuilt all postgres:* images in the internal registry with pgvector baked in (issue #3880). Deleted the runtime apt-install line from neetoci-service.

Quick win #2 — Ruby/Node/Postgres/Redis installed at runtime

Problem. Every pod paid neetoci-version ruby 4.0.1 → tarball download from the in-cluster binaries-cache service, unpack into ~/.rbenv/versions/4.0.1. Same for Node via nvm. Postgres + Redis pulled as podman images on first use. ~30–60 s of overhead per fresh pod.

Fix. New declarative docker-ci/dependencies file (#3879 → PR #3907) and Dockerfile bake steps that pre-install everything into the CI image:

RUBY_VERSIONS=(4.0.1)                     # pre-installed under ~/.rbenv
NODE_VERSIONS=(22.13)                     # pre-installed under ~/.nvm, default alias
APT_POSTGRES_VERSIONS=(18 18.3)           # pgdg apt + pgvector
REDIS_VERSIONS=(7.0.5)                    # compiled from source at /opt/redis/7.0.5/

Quick win #3 — EBS cache-restore was the wildcard

Problem. Same commit, same image, same node family (r8g) — two consecutive runs of cache restore unpacking the same 1.4 GB of node_modules + vendor/bundle + .nvm:

Download barely moved — variance was entirely in the unpack, i.e. writing 1.4 GB into the pod filesystem. Cause: r8g nodes have an EBS-only root. Every pod's scratch I/O — cache unpack, the Postgres data dir, db:schema:load, log files — lands on one network-attached gp3 volume shared by every pod on the node. Run enough pods at once and its IOPS saturate.

Fix. Moved the CI Karpenter NodePool to the r8gd family — same Graviton4 silicon, but with a physically-attached local NVMe SSD. Setting EC2NodeClass.spec.instanceStorePolicy: RAID0 tells Karpenter to RAID the instance-store NVMe and repoint kubelet + containerd ephemeral storage onto it, so pod scratch I/O hits local NVMe instead of contending for shared EBS.

Quick win #4 — cache restore was slow, not just noisy

Problem. Even on a fast node, cache restore took ~69 s. The Go cache binary — NeetoCI's fork of SemaphoreCI's cache-cli — restored every key (nvm, gems, yarn-cache, node_modules) one after another. And yarn.lock mapped to a redundant ~/.cache/yarn archive — 1.4 GB, ~30 s — that bought nothing: node_modules is already cached, so yarn install is instant on a hit.

Fix. Rebuilt the binary from toolbox PR #2 — shipped as static arm64/amd64 binaries in PR #3892:

• Parallel restore — goroutines + sync.WaitGroup; independent keys download concurrently → ~70 s → ~30 s (bound by the slowest key, gems).
• Dropped the redundant yarn-cache — yarn.lock no longer caches ~/.cache/yarn → −30 s/job and −1.4 GB of S3 per run.
• Parallel store — same goroutine pattern around compress + upload; faster cache-miss runs.
• S3 downloader — concurrency 5 → 10, part size 5 → 10 MiB.

Quick win #5 — parallelism: 4 but one pod ran the entire suite

Problem. Setting parallelism: 4 spawned 4 test pods, but with no work distributor each one re-ran the full suite. A 4,024-test run on the buggy build:

Fix. Added the minitest-distributed gem (loaded conditionally via MINITEST_COORDINATOR env var) and a new shared_redis: true per-job flag. NeetoCI provisions a per-job Redis; pods enqueue/work-steal tests until the queue drains.

Investigation — the smoking guns in the logs

Three findings from instrumenting the pod logs (JSON-event stream into the UI accordion):

1. cache restore variance: 26 s ↔ 68 s on identical inputs (slide 10). Concurrent pods on the same node fighting for EBS IOPS.
2. parallelism: 4 → only 1 pod ran tests: minitest had no distributor; the first pod that reached rails test finished the whole run before others joined (slide 12).
3. runtime apt-get for pgvector: the same apt-get update + install postgresql-N-pgvector ran in every postgres start, every pod, every run. ~10 s of pure waste (slide 8).
4. epilogue logs missing from UI: bundle exec rake simplecov_coverage:publish ran but never appeared in the pipeline view — the post-deployment script bypassed the JSON-event logger.

Root cause — five stacked issues

1. Flat YAML → flat execution. One pod ran every command serially. No way to fan out, no way to run two things at once.
2. No parallel-test distributor. Even when N pods were spawned, each had its own copy of the suite — no work-stealing.
3. EBS root volume. Per-pod scratch I/O (cache unpack, db init, log files) all went to a contended network-attached disk.
4. Runtime-installed dependencies. Every pod paid Ruby/Node/Postgres/Redis install on cold start — they should have been in the image.
5. Runtime-installed pgvector. The pgvector apt-install ran inside the postgres container on every pod that started a database.

Each one was a multi-second tax; combined they were the difference between a 7-minute and 20-minute run.

The fix — five-part rollout

Tracking issue: neeto-ci-web#3799 · 14 sub-issues, 13 PRs merged

Result #1 — per-step timing (neeto-cal-web)

Before = p50 · p95 of 18 production neeto-cal-web default.yml runs, 7–13 May 2026 (per-command durations parsed from job logs). After = median of 3+ runs on add-minitest-distributed, commit dd261621.

Result #2 — measured wall-clock, 8 apps (1st day post-merge)

Baseline = success p50 from production ci_jobs, 7–13 May 2026. Post-merge = avg of block-pipeline runs only (test_pods > 0), 19 May 15:30 onward. n = 2–6/app, still early.

Result #3 — measured fleet savings (8 apps live, 7 projected)

Top 15 projects by default.yml avg run-time. ✓ = block pipeline live; avg from production logs since merge.

✓ measured: avg of block-pipeline runs (test_pods > 0) since 2026-05-19 15:30, n = 2–6/app, baseline = success p50 of 7–13 May 2026. * = projected, suite-size model: large (≥14 min) −45%, medium (8–14 min) −35%, small (<8 min) −15%.

Result #4 — variance collapse

Two consistency improvements that don't show up in averages but matter every day:

Predictable CI is more valuable than fast CI. The new pipeline is both.

Result #5 — cluster compute (honest framing)

Parallelism reshapes wall-time but doesn't reduce total CPU-minutes much by itself:

The real compute savings come from the bake — ~3 min of setup × every pod removed:

• ~150–200 node-hours/month saved across the fleet
• ~$100–200 / month in raw EC2 (r8gd.2xlarge on-demand baseline)
• No autoscale spikes from EBS-bound cache-restore stalls

The headline win is wall-clock, not cost. But the cluster also stops thrashing — that has its own quiet value.

Why faster CI matters operationally

CI wait-time isn't a storage line item. Every minute is a developer either waiting on a green build, or context-switching away and losing flow.

Compounding effect on team rhythm: when CI is fast and predictable, smaller PRs become viable. Smaller PRs land faster, are easier to review, and break less. The 20 min → 7 min cycle is what makes the whole loop tighter.

What we actually shipped

Tracking issue: neeto-ci-web#3799

What we evaluated and skipped (honestly)

Going broad first kept the design honest. Knowing when to stop is part of the work.

Artifacts shipped

ECR (728988564940.dkr.ecr.us-east-1.amazonaws.com)
└── neeto-ci-deployment-image:v62
    ├── Ruby     4.0.1      pre-installed via rbenv
    ├── Node     22.13.1    pre-installed via nvm, default alias
    ├── Postgres 18.4 + pgvector 0.8.2  (apt cluster, fresh per pod)
    ├── Redis    7.0.5      compiled from source at /opt/redis/7.0.5/
    └── docker-ci/dependencies (declarative, easy to extend)

Internal registry (10.100.0.20:5000)
└── postgres:{13,14,15,15.1,18,18.3}     (rebuilt with pgvector baked)

Cluster (EKS, neeto-ci)
├── EC2NodeClass arm64
│   ├── instanceStorePolicy: RAID0       (NVMe → kubelet + containerd)
│   └── blockDeviceMappings              200Gi gp3, 8000 IOPS, 500 MB/s
└── NodePool default
    └── instance-family: r8gd            (Graviton4 + local NVMe SSD)

Code
└── PR #3910 → main                       Epic merged (14 commits, 65 files)

Bottom line

Thanks 🙌

Questions?

Detailed report: ci-test-optimization-results.md
Deck source: github.com/vishal24367/ci-test-optimization-deck
Gist: gist.github.com/vishal24367/674d77e…
Epic: neeto-ci-web #3799