Prout: is your pull request out?

How do you know your pull request is on production?

Roberto Tyley

Published on Tuesday, 3 February 2015

Cloud computing   Computing   Open source   Software  

Seen on PROD (merged by @rtyley 7 minutes and 4 seconds ago). Please check your changes!
Seen on PROD (merged by @rtyley 7 minutes and 4 seconds ago). Please check your changes! Illustration: Roberto Tyley

Guardian Membership has been continuously deploying since we started the project back in May 2014. Prout arose to help with a problematic area of this wonderful setup:

Anything can break. If you’re building a site, the site can break. If you’re building a site with continuous deployment, the site can break, but the continuous deployment pipeline can break too. If either of these things have any value, then you care when they break.

A fireball erupts across Interstate 77 from a pipeline explosion. The gas company failed to conduct inspections or tests that might have revealed weaknesses in the massive pipeline.
A fireball erupts across Interstate 77 from a pipeline explosion. The gas company failed to conduct inspections or tests that might have revealed weaknesses in the massive pipeline. Photograph: West Virginia State Police/AP

One cause of site breakage is shiny new features, which is why it’s good discipline for a developer to check their changes in production as soon as they’re deployed. The continuous deployment pipeline itself can fail for so many reasons (the CI server can die, the deployment tool (RiffRaff, for us) can forget to poll for new builds, the AWS auto-scaling group can end up in a bad state requiring manual intervention, the CDN can spontaneously decide to serve only stale content… and so on) - you can monitor lots of different things, but the unknown, the unexpected, can fall between the cracks.

We wanted a single check that would span the whole pipeline - from merged pull request to running site - and, as a side-benefit, be agnostic about everything inbetween, because there’s no reason why your pipeline should be anything like our pipeline (CI, deployment tools, hosting providers - those differ even for teams within the Guardian).

Prout spans the continuous delivery pipeline from merged pull request to production deployment. Your army of mad zombie robotic squirrel scientists are not it’s concern, you might want to keep an eye on them.
Prout spans the continuous delivery pipeline from merged pull request to production deployment. Your army of mad zombie robotic squirrel scientists are not it’s concern, you might want to keep an eye on them. Illustration: Roberto Tyley

Prout only requires that you use GitHub pull requests, and that you expose the Git commit id of your build on your deployed site - a URL that Prout can hit. Prout only cares about the commit id, and definitely doesn’t care about build numbers. So for instance, if you look at the source of https://membership.theguardian.com/, you’ll see something like this near the bottom of the page:

Whenever you merge a pull request, Prout will hit your site and read that commit id - which tells it unambiguously what version of your software is being served to your users as of that moment. It also clones your GitHub repo, and compares your merged pull request with the history of the commit that’s currently on the site. From that, it can tell whether the site has your pull request or not.

Establishing what's been deployed by looking at Git history
Establishing what’s been deployed by looking at Git history. Prout walks the Git repository’s commit graph to see which pull requests are live. Live pull requests are the ones in the history of the build-commit that’s reported by the live production site. Illustration: Roberto Tyley

As soon as Prout sees that the site is running a version that includes your pull request, it’ll update the pull request with a message like this:

Seen on PROD (merged by @rtyley 7 minutes and 9 seconds ago). Please check your changes!
A pull request makes it successfully on to production Illustration: Roberto Tyley

That “merged 7 minutes and 9 seconds ago” is kind of interesting - the total transit time required by your pipeline to get a code change in front of users. You’d obviously like to keep it as low as possible, and know if it’s getting too long. If your pull request doesn’t show up promptly, then something’s going wrong - and Prout will make a point of letting you know:

Overdue on PROD (merged by @rtyley 15 minutes and 2 seconds ago). What's gone wrong?
A merged pull request has not yet made it it to production... time to investigate Illustration: Roberto Tyley

You can also get a great overview of what’s been deployed and what’s about to be deployed, just by looking at the labels in the GitHub closed pull requests view:

Merged pull requests, labelled according to deployment status
Merged pull requests, labelled according to deployment status Illustration: Roberto Tyley

How do you configure Prout?

Place a .prout.json file in any folder in your repository you want to get monitored:

{
  "checkpoints": {
    "PROD": {
      "url": "https://membership.theguardian.com/",
      "overdue": "15M"
    }
  }
}

Whenever changes from a pull request touch files anywhere within that folder, the Prout configuration will fire, and Prout will update the pull request accordingly. More details are in the project README.

Slack integration

Not everyone at the Guardian with a stake in the software we write (for instance, the in-house users of Ophan) has a GitHub account, but it’s still good for them to know about the features and fixes we release - so Prout supports hooking into Slack, to let us communicate when those features are ready to try out.

The way we let Prout know the Slack webhook URL is a hack. Slack webhooks are ‘private’ because they contain a unique code that allows you to post any message you want to that room. For a public repo like guardian/membership-frontend, how would you configure the Slack webhook for Prout without publicising the URL? The answer is a little abuse of an admin-only feature of GitHub repos - repository webhooks. You can store URLs in there, and only admin-enabled users can see them. Store a Slack webhook URL in your repository’s webhooks, and Prout will pick it up, posting a Slack message every time a pull request is successfully deployed.

Run your own Prout!

If this stuff sounds good to you, and you want to run your own instance of Prout against your own projects, you’ll want to create a dedicated GitHub account and access token that can comment and set labels on those repositories. You can quickly deploy your own instance of Prout with Heroku (that’s not essential, just convenient. You may well want to run Prout on your own infrastructure):

Deploy

Prout, like gu:who, uses GitHub as the user interface, and supplies minimal UI itself. Try it out and give us your feedback!

Continue reading

Five months in software development at the Guardian - what we’ve learned How we reimagined search for the Guardian app in one day