Continuous Delivery at OLX (part I)

A year ago we started an ambitious project at OLX, achieve Continuous Delivery, you may ask why? It's a fair question, and there's a good answer too, because we needed a change and like most changes this was triggered by pain. This is a very common story among IT companies, every Release is painful, is a whole bunch of features and fixes that most Operations guys don't know about, and it's infrequent, and is manual and error prone. Even though  we were releasing once every 2 weeks which is a lot for many companies, for us it was clearly not enough, we had several issues, rollbacks and even blackouts every time.
So basically we started meeting once every week to discuss how would Continuous Delivery would look like at OLX, how to build a Pipeline, WHAT the HELL was a Pipeline!? Anyway, many of us had the chance to read the famous book written by Jez Humble and were excited about the possibilities. But we were light years away from what was described there. I mean, most teams didn't even got to Continuous integration, our architecture didn't help either since it was basically a gigantic monolithic PHP code that we would checkout from SVN and rsync completely into our web servers in about 10 minutes or so.
Another issue was that the few dev teams that were Continuously integrating had different Jenkins installation, each with different set of plugins and tools. Even myself, as a Release Manager, had my own Jenkins, which I found very useful for some automatic deployment tasks that I had already started implementing. So the first challenge was to decide what to do with our Jenkins; If we would integrate the whole thing in one Jenkins it would be a mess, too many hands on the same plate, many different technologies (from different teams) and different roles as well (compiling, testing, deploying), who would be the owner of such a tool? It was clear to us that it wasn't a great solution, I mean for smaller companies it would be more than fine, but it doesn't scale.
So we decided to keep our Jenkins separated and started researching how to integrate them, sadly there's no plugin out there for these (none that we could fine anyway) and the master/slave Jenkins schema doesn't fit our problem because the Slave is just that, a plain simple slave which mirrors its master's jobs.
So we used the Bash console on Jenkins job to curl another Jenkins and trigger the remote job, it turned out to be pretty easy, and thanks to Jenkins API the caller job would loop until the callee finished, and then we could even retrieve the result, whether it failed or succeeded.

Our first experiments were made thanks to Mobile Team collaboration, this a key part: find a team willing to experiment, shake things up a little bit. So they would build the code on each commit with their Jenkins, and then call the Release Management (RM) Jenkins to get their code deployed on a QA environment, and once this was successful they could run a battery of acceptance test against the environment. Easy, right? well... it took us some time to get there.
We will continue talking about CD Project at OLX on following posts, there is a lot more in this story, including the open sourcing of our deployment tool Octopush!