When Everything Falls Apart: Stories of Version Control System Scaling
At Mozilla we have a great dependency on version control systems and software. All contributors format patches against it, all of our build automation infrastructure pulls from it, all online collaboration links against it. Over the life of various Mozilla projects, we've shifted version control systems many times, which led us to being large users of a multitude of systems: Git, Mercurial, CVS, SVN, BZR, and even RCS.
Over the years we've also developed bizarre sets of hooks and various externally-hosted barnacles that promise to add greater insight into incoming code, along with notifications and beautiful graphs to keep all the developers up-to-date with the status of the project and to keep the PHBs (pointy-haired bosses) happy with the progress of overall development.
In keeping up with this massive demand of hundreds of thousands of checkouts/clones per day, we've had to do heavy capacity planning in terms of hardware (the easy part). The hard part has been trying to scale the backend version control server software, which has led us into some interesting problems that haven't been discussed or shared at length of in detail in a public setting.
In particular this talk will be about about experiences and trials that we've faced keeping reliabile, quick response times, and general uptime of the Git and Mercurial version control systems.
Join me as we explore when, how, and how often these systems break, how to properly monitor and diagnose them, critical caching failures, backend-switching shenanigans, grumpy upstream correspondence, NFS atomicity being sketchy, and lots of throwing hands into the air and ragequitting.
Ben has been a community systems administrator at the OSU Open Source Lab, where he helped and administered dozens of high-profile open source projects. He also has ample experience participating in the OSU Linux User’s Group as their official Safety Officer, and as part of the Open Source Education Lab which aims to integrate Open Source software and ideologies into the university curriculum. Ben also played a role in the inception and coordination of the local Beaver Bar Camp.
After graduating with a degree in Computer Science and Business Administration, he joined Mozilla as a systems administrator. Mozilla gave him the opportunity to diversify his skill-set to operate on the scale of hundreds of millions of users.
In his spare time Ben is an avid embedded computer enthusiast and has spent countless nights toiling over tiny computers while cursing at them to work correctly. He enjoys the intersection of lifestyle and computers constructing things like homemade electric bicycles, restoring vintage hardware, electronics repurposing, wearable computers, and novel home automation.