Continuous Delivery at OLX (part II)

Moving on with Continuous Delivery Posts at OLX, another big challenge we had is how to deal with application properties/configuration files. At the beginning everything was made manually, each time a developer wanted to change some application config file he would make the requirement in a JIRA Ticket, specifying the environment, the file and the diff. As you can imagine this brought a lot of problems, from human typing errors to lack of visibility and in-traceability of changes. So we desperately needed a version control over these files (they were treated differently from code) and an automatic way of making the changes without compromising the stability of our environments. We wanted devs to make the changes themselves but also keep Release Management and Operation guys in the loop, giving the heads up or not whenever the could foresee a dangerous change. Another issue was security, we had in the same config file credentials with passwords and all next to feature flags or 3rd party service URL's.
So first we started splitting these files (security and the rest), credentials was not something we wanted devs or any unauthorized person to see.
Secondly we adopted Github (though this was embedded in a much larger decision since we also decided to port all of our SVN tracking system to Git, but we'll talk further about this in another post), so we created a config repository for each code repository we had. We placed the config files there (using the original path) but using branches to specify the environment.
Third we gave only read permission to devs on these repos, so they can have proper visibility on the configuration and the changes made thru time on each environment. But we didn't want them to write directly to them, we decided to use Githib's Pull Request for this so that they have to propose the change to us, we review it and approve it (or not). The approval is the commit it self, so this triggers a commit hook on a Jenkins's job that deploys the configuration automatically (only in QA environments). Github is great for this, they make Pull Request real easy, for devs they need only to edit the files online and since they don't have write permissions, Github creates a fork and offers you to create the Pull Request to the original repo automatically.

All this process had been working great for us for some time, but we still needed to deal with credential files, they weren't being tracked and some loses on these files could have been catastrophic if we hadn't have any backups (but we all know backups are not enough when you need traceability). So we needed to track them, but give only visibility to some authorized IT resources, we though of having yet another repo with more restricted permissions for credentials, next to code and config, but we would have to create yet another deployment process for these. Until we found git-crypt, this great tool let us save in the same repo, encrypted and non-encrypted files, you only need to install it and have the proper keys to see the files, otherwise you only see rubbish, even Github shows it as a binary file. So this way we saved our selves from another set of repos and custom process for credentials, we do the same as we did for normal config files, but we installed git-crypt and keys where we checkout the repo and voilà!
All these changes weren't made without some kind of resistance, many devs were used to the just typing some new configuration on a ticket, or ask what value is set in "foo.bar" on Prod (which clearly doesn't scale when you have more than 100 devs an thousands of settings), but we gave them a much powerful tool, and in time everybody got used to it and learned to love it.