CI/CD for a 36-Person Team: What Actually Worked (and What Didn't)

Rolling out CI/CD to a team of 36 developers working across 8 concurrent projects — including our flagship microservices HRM platform — is not a weekend task, and it's not primarily a tooling problem either. This is the honest account of what we tried, what failed outright, and what we ultimately kept, including the attempt that made things measurably worse before they got better.

What We Started With

Manual deployments via SSH, run by whichever engineer happened to be free and comfortable enough with the production server to do it. A shared staging environment that was perpetually in some broken, half-updated state, because three different projects were pushing to it independently with no coordination. And a quiet but real pattern of developers avoiding pushes late in the week, because a Thursday-afternoon deploy that broke something meant a bad Friday for whoever was on call. If any of this sounds familiar, you're not alone — it's close to the default state for a team that's grown faster than its deployment process has.

Attempt 1: Full Automation from Day One (Failed)

Our first instinct was to fix everything at once: introduce full deployment automation immediately, across every microservice in the HRM platform simultaneously. In hindsight, this was solving an adoption problem with a technology solution, and it failed for a predictable reason — the cognitive overhead of learning a new workflow, on top of an already-complex microservices setup with payroll, attendance, and onboarding services all interacting, was too much to absorb in one step.

Two weeks in, people were quietly bypassing the pipeline — going back to manual SSH deploys "just this once" because the new process was slower to trust than the old habit it was replacing. That's the tell that an automation rollout has outpaced the team's actual comfort with it.

We didn't push through this by adding more documentation or mandating compliance. We stopped, acknowledged the rollout as it stood wasn't working, and restarted with a much narrower first step.

What Actually Worked: Incremental Adoption

We broke the rollout into three phases over eight weeks, each one deliberately narrow enough that it could be adopted without requiring developers to trust the entire pipeline at once.

Phase 1 — Automated Testing Only

GitHub Actions runs unit tests and linting on every pull request. Critically, this phase changed nothing about how deployments happened — no automation touched production or even staging yet. The only thing that changed was that every PR got fast, automatic feedback on whether it broke anything, before a human reviewer even looked at it.

This built trust in the pipeline itself, in isolation from any anxiety about deployment risk. By the time Phase 2 introduced actual deploy automation, "the pipeline" was already a known, trusted quantity to the team — not a new unknown arriving at the same time as a new deployment process.

Phase 2 — Automated Staging Deploys

Merges to the develop branch now auto-deploy to staging via Docker Compose on a single VM. We deliberately kept this simple — a single VM, not a full orchestration platform — because the goal of this phase was reliability and debuggability, not sophistication. If a staging deploy failed, any engineer on the team could look at the Docker Compose logs and understand what happened, without needing deep familiarity with a more complex deployment platform.

# .github/workflows/staging-deploy.yml (simplified)
name: Deploy to Staging
on:
  push:
    branches: [develop]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy via SSH
        run: |
          ssh deploy@staging-vm "cd /app &&             git pull &&             docker compose up -d --build"

Phase 3 — Production Deploy with a Manual Gate

Production deployment is automated end-to-end, but requires a one-click manual approval step inside GitHub Actions before it runs. This was a deliberate choice, not a temporary training-wheels measure we intended to remove later: it gives the team confidence that a human consciously chose to ship to production at this moment, without removing any of the actual automation benefit — the deploy itself is still fully scripted and repeatable, just gated behind a deliberate go/no-go decision.

The Numbers

Metric	Before	After
Deployment time	~45 minutes	~8 minutes
Post-deploy incidents	Baseline	-60%
Developer confidence (survey)	Low	Measurably higher

Why the Order of Phases Mattered

It would have been faster, on paper, to combine Phases 1 and 2, or to skip straight to full automation with a manual gate. We tried something close to that combination in Attempt 1, and it failed. The value of the strict ordering wasn't really about the technical dependency between the phases — Phase 2 doesn't strictly require Phase 1 to function. It was about giving the team's trust in the pipeline time to catch up with the pipeline's actual capability, one small, verifiable step at a time.

Best Practices

1. Separate "does this pipeline work" from "do I trust this pipeline with production" — they're different questions, and conflating them is what made Attempt 1 fail.
2. Start with the phase that gives developers immediate, low-risk value — fast PR feedback is close to zero-risk and builds goodwill fast.
3. Keep early deployment automation debuggable by a generalist, not just a DevOps specialist — a single-VM Docker Compose setup that anyone can read beats a sophisticated platform nobody on the team fully understands yet.
4. Keep a manual gate on production deploys as a permanent feature, not a temporary safety net to be removed once "trust is established" — the gate and the automation aren't in tension with each other.

Common Mistakes

• Rolling out full automation across an entire complex system at once, rather than in stages the team can actually absorb.
• Treating CI/CD adoption as purely a tooling rollout, when the real blocker is often trust and habit, not technical capability.
• Removing manual approval gates too early in the name of "full automation," before the team has had enough cycles to build confidence in the pipeline's reliability.
• Measuring success purely by deployment speed, while ignoring developer sentiment — a fast pipeline nobody trusts still gets bypassed.

Security Considerations

Automating deployment access meant centralizing credentials that were previously scattered across individual developers' SSH configurations. We rotated all deploy keys as part of the rollout, scoped the CI service account's permissions to only what the deploy step actually needs, and made sure the production approval gate is tied to a named individual's action, not a shared or generic account — so every production deploy has a clear, auditable "who approved this."

FAQ

Q: Why not automate production deploys fully, with no manual gate, once the team trusted the pipeline?
We considered it, but the team preferred keeping the gate permanently — it costs almost nothing in speed (a single click) and provides a clear point of human accountability for what goes to production and when, which matters more on a payroll-adjacent HRM platform than the marginal seconds saved.

Q: How did you handle the 8 different concurrent projects during this rollout — did they all move through the phases together?
No — each project moved through the three phases on its own timeline, based on its own team's readiness. The HRM platform, being the most complex, moved more cautiously than smaller projects, which is exactly the kind of flexibility a rigid "big bang" rollout wouldn't have allowed.

Handling Rollbacks Once Automation Was in Place

Automated deployment naturally raises the next question: what happens when an automated deploy makes things worse, not better? Before this rollout, a rollback meant someone manually SSHing back to a known-good state, which was slow and error-prone under pressure — exactly the wrong conditions for a careful manual process. Once Phase 2 was in place, rollback became a matter of redeploying the previous known-good commit through the same automated pipeline used for forward deploys, which meant a rollback took roughly the same eight minutes as a normal deploy, rather than the anxious, ad hoc process it used to be.

This turned out to reduce risk aversion around deploying in the first place. When a bad deploy is expensive and slow to undo, teams naturally become cautious about deploying at all, which pushes toward larger, riskier batches of changes shipped less frequently. Once rollback was fast and routine, the team's actual deploy frequency increased, and — somewhat counterintuitively — post-deploy incidents kept falling, because smaller, more frequent deploys are individually lower-risk than the large, infrequent batches they replaced.

Extending the Approach to the Other Seven Projects

The HRM platform was the proving ground for this three-phase approach, given it was both the highest-stakes and the most architecturally complex of the eight concurrent projects. Once the phased rollout demonstrably worked there, extending the same three-phase structure to the other seven projects was considerably faster — each team could see a working reference implementation rather than being asked to trust an unproven process, which meaningfully shortened the adoption curve for every subsequent project.

A Rough Timeline Across Projects

Project	Time to Phase 3 adoption
HRM platform (first)	8 weeks
Subsequent projects (average)	2-3 weeks

Conclusion

CI/CD adoption across a team this size is a people problem as much as a tooling problem, especially across a microservices estate as varied as HRM's payroll, attendance, and onboarding services. The technical pieces — GitHub Actions, Docker Compose, a manual approval gate — are all fairly standard. What made the rollout actually succeed on the second attempt was starting with the part that gave developers immediate, low-risk value, and only expanding scope once that trust was genuinely established.