more continuous integration
So, in a previous post, Continuous Integration I gave a quick overview of our continuous integration setup, and mentioned that I’d like to explore allowing jenkins to push our deploys for us. That work is now done. The basic workflow is now:
- Developer develops new feature
- Developer runs unit tests locally
- Developer submits patch for code review via Phabricator
- Code is approved by another person on the team
- Developer lands (pushes to master) changes
- Jenkins initiates a build, and if all tests pass, promotes the build automatically to the integration server.
- Developer, after integration testing, may promote an integration build to staging
- Developer, after staging testing, may promote a staging build to production
The two promotion steps are handled with a single click (yay one click deploy), that’s one more point for us on the Joel Test. And can be executed by any developer on the team. Our old deploy process looked something like this:
- ssh to staging server
- try to remember where the staging code lives.. was it under /var, or /sources, or /opt
- check apache vhost file to see where staging code actually lives
- cd to /var/www
- git pull (in one folder)
- svn up (in two other folders
- su to root after realizing that svn had issues checking out code because the last person who did this was logged in as root
- fix permission problems (or if in a hurry, just svn up as root)
- check website to see if everything is working
- ssh to first production server
- try to remember where the prod code lives
- find it…
- git pull in one folder
- svn up in two other folders
- same song and dance as on staging after realizing… someone did this as root last time
- chmod to fix perms on 2 directories that need to be writeable by the webserver
- test code in production
- goto 10 but for second production server.
Just to make that perfectly clear, we went from over 30 discreet, error prone steps to get the code pushed out onto staging and both production servers to ONE SINGLE CLICK (well, one for staging and one for prod, so TWO SINGLE CLICKS if you want to be pedantic).
In the ideal case, a full deploy from integration to staging and then production would take us 2 hours. On average, it was more like 4 hours. Now, we click a button, wait 5 minutes and then test the code in the new environment. No more ssh'ing all over the place, no more annoyance when you find out someone did the deploy as root before you screwing up all the permissions, no more trying to remember where the hell the code lives on this particular server. SIMPLE, one click deploy. Oh thank the gods.
Jenkins has saved my team a TON of time, this can’t be emphasized enough. If your deploy process looks anything like ours used to, please do yourselves a favor and automate that as soon as possible. You will save money, developer time, and most relevant in my case, minimize stress for your dev team. We used to log 4 hours of work for a single deploy per week, and now it doesn’t even warrant a line item, developer time spent on the actual deploy process is less than 5 minutes, that’s 3 hours and 55 minutes you just freed to spend testing/drinking beer/playing video games/etc.
While my overall experience with Jenkins has been positive, there are a few issues that would be great to see improved. It’s possible that I just haven’t figure out how to do these things yet, and I’m still playing around with the system so if you know how to do these things, drop me a line:
- Limit the number of builds Jenkins will keep on the staging/prodcution servers (I can do this in the build directory, Jenkins will clean up after itself, but I haven’t figured out how to do that on servers that Jenkins pushes code to)
- Document use cases. The most time consuming part for me was trying to figure out what plugins I needed for doing all the various pieces of my deploy.
- Plugin stability. There have been a few instances of me updating Jenkins + Plugins only to realize that everything is broken. I would hope that plugin developers were using Jenkins to continuously integrate their code to minimize regressions!
That’s it. Aside from the fact that I have an illogical dislike of Java (probably related to that one CS class I had in Java as an undergrad) I find the tool to be powerful, intuitive, and configurable.
If you’re not doing some kind of automated testing+deploy, you probably should be.