One additional rule we follow is to not deploy to production on Fridays except for emergency deployments. This is because inevitably things go wrong during production deployments and we do not want our engineers to spend Fridays or weekends trying to debug a production deployment issue. We also make use of dark deployments and feature flags to further ease the production release process.
To make everyone in the team aware of how often we are breaking the "No Friday deployment" rule due to emergency deployments, we put up a small poster which tracks the number of weeks it has been since we did a Friday deployment. This is visible throughout our engineering area. Obviously the higher this number, the longer we have been able to adhere to this rule. Any Friday deployment to production will bring this number back to 0, which is what seems to have happened below:
 |
| Weeks Since Friday Deployment Counter |
This number was initially being updated manually on sticky notes. But since we use Bamboo as a continuous integration tool, we have access to its REST API to query the dates of production deployments. With this API and a little bit of code on a Raspberry Pi, there was no reason not to automate it and display the number on a digital counter.
Implementation
First off, the display we use is an 8x8 LED matrix, available here with a display driver. It’s big and bright enough to be visible within our engineering room and can be daisy chained to other matrices. Also, there is a convenient python module available to drive the display. After a bit of soldering and hooking it up to Raspberry Pi's GPIO ports, here's what the initial setup looked like: |
| Raspberry Pi with MAX7219 Dot Matrix Module |
And here's a snippet of python code to query Bamboo's REST API to get the weeks since a production deployment was done on a Friday:
BAMBOO_URL = "https://<bamboo host>/rest/api/latest" DASHBOARD_URL = BAMBOO_URL + "/deploy/dashboard" RESULTS_URL = BAMBOO_URL + "/deploy/environment/{}/results" def get_weeks_since_friday_deploy(user, passwd): weeks_since_friday_deploy = sys.maxint r = requests.get(DASHBOARD_URL, auth=(user, passwd)) r.raise_for_status() prod_env_ids = {} # first get all production environments' ids which will # be used to get deployments for project in r.json(): name = project['deploymentProject']['name'] envs = project['environmentStatuses'] for env in envs: if env['environment']['name'] == 'Production': if 'deploymentResult' not in env: continue else: prod_env_ids[name] = env['environment']['id'] for key, value in prod_env_ids.iteritems(): # no need to continue if it's already 0 if weeks_since_friday_deploy == 0: break r = requests.get(RESULTS_URL.format(value), auth=(user, passwd)) r.raise_for_status() deployments = r.json()['results'] for dep in deployments: start = datetime.datetime.fromtimestamp(dep['startedDate'] / 1e3) end = datetime.datetime.fromtimestamp(dep['finishedDate'] / 1e3) if start.isoweekday() == 5 or end.isoweekday() == 5: weeks = (datetime.datetime.now() - start).days / 7 if weeks < weeks_since_friday_deploy: weeks_since_friday_deploy = weeks weeks = (datetime.datetime.now() - end).days / 7 if weeks < weeks_since_friday_deploy: weeks_since_friday_deploy = weeks # it is possible to get a negative number in an edge case, # if the script runs very close to the time of deployment if weeks_since_friday_deploy < 0: weeks_since_friday_deploy = 0 break # once it's 0, no need to continue further if weeks_since_friday_deploy == 0: break return weeks_since_friday_deploy
Using the code above and the python library to drive the display (https://github.com/rm-hull/max7219, it is easy to display the number on the matrix and update it automatically. It is setup to run as a cron job every 5 minutes on Fridays.
---
Next step: We haven’t reached there yet but if the number goes above 9, another matrix can be daisy chained together to form a cascaded display.
Update: We did reach that point and I was able to daisy chain 2 counters.
