Deploying Code at WonderProxy
What We Did
In the old days, WonderProxy deployed websites with Git. The deploy process was… simple:
- Clone the Git repository directly into its home on the production server
git pullto deploy changes
The system was easy to implement and easy to use. Rollbacks were a git checkout away. It even had the super handy bonus feature of letting admins modify live code without losing the original. Heck, if you liked the modifications, you could even commit them right on prod!
Things got a little hairy when we started using CSS preprocessors. And an automated testing suite. And a generated styleguide. And a couple of code linters.
At first, all the additions weren't a problem. CSS preprocessor? Check the built CSS into the repo. Test suite? Just run it locally first. Generated styleguide? Same treatment as the CSS. Linters? Same as the test suite.
What Stopped Working
As our unofficial pre-deploy checklist grew, so did the potential for human error. Sometimes you just have to make a line of text red, right now. It looks great on prod, so you commit your change. You don't notice that the CSS preprocessor has silently redacted your red text the next time somebody adds a class, and it's not until a couple months later that somebody asks, Hey, wasn't this supposed to be red?
Even with some occasional lumps, our deploy worked for us until the team started growing. Deploying with git pull makes a few assumptions:
- Everyone who needs to deploy has write permissions to the deploy location.
- Everyone who needs to deploy has permission to restart the relevant services.
- Anyone who wants a deploy can SSH to the deploy server (i.e. is sitting in front of a computer with the right set of SSH keys).
When those assumptions stopped being true all the time, we started shopping for an automated deployment system.
What We Needed
We wanted to automate our pre-deploy checklist: we were tired of the overhead, and team members who didn't interact with the codebase often were tired of never knowing for sure exactly what to do for a deploy. Sometimes the project readme was current, and sometimes it was emphatically not.
We wanted anybody on the team to be able to deploy anything to anywhere, at any time, whether they were in front of an SSH-capable computer or not.
We wanted to be able to revert a deploy quickly. When something goes wrong on prod, nobody wants to futz around with npm or Sass; you want to get back to the previous state right now.
We wanted to keep as much of the process as possible on our own infrastructure. We've got a solid network of Puppet-controlled servers, and we'd rather rely on that than a third party.
Finally, we wanted our master branch (our 'ready to deploy' branch) to automatically deploy its new commits to a staging environment. We wanted that process to include our test suite and linters, and we wanted to be notified if anything failed. (That requirement sounds a lot like continuous delivery to me, but since we don't automatically integrate feature branches with master, I'm not sure it counts.)
What We Found
The automated deploy/continuous integration/continuous deployment market is saturated with options, both self-hosted and service-based. As we hunted around, we uncovered a couple more requirements:
- We didn't want to spend a fortune. Many popular services are free for OSS projects, but charge a premium for private projects.
- We didn't want to maintain a self-hosted solution. The goal was to reduce overhead, not shift it around.
We eventually settled on Deploybot. It's met a lot of our requirements, which is good. We're also shopping around for a new solution, which suggests that we're not entirely happy with how things are working.
In the next post, we'll talk about how we're using Deploybot, what's working for us, and what isn't. We'll also have some thoughts on where we're going to go from here.