Octopus: Parallel deployments, API's and OctoPack!

Paul Stovell

Paul Stovell

— 3 min read
This is an old post and doesn't necessarily reflect my current thinking on a topic, and some links or images may not work. The text is preserved here for posterity.

Happy new year!

For the last month or so I've been working on a new release of Octopus, which is finally out! This release includes a few big features as well as a lot of small bug fixes and nice additions. You can read more about what changed in the release notes, but there are a few important changes that I want to call in this post.

Docs, docs, docs

I've spent a lot of time on documentation this release. Instead of Tender, Octopus is now using Confluence for product documentation. The new documentation includes lots of useful pages, including:

Package creation

NuGet packages used for Octopus don't follow all of the standard NuGet conventions, which has always made them somewhat tricky to create. While speaking with Maarten Balliauw about his upcoming book on NuGet, we bounced some ideas around, and came up with a solution. OctoPack is a tool that makes creating Octopus-flavored NuGet packages from your desktop or build server easy. Give it a try!

Edit: the source for OctoPack is available on GitHub.

API support

My goal this release was to make it possible for other tools to automate Octopus - the primary scenario being:

A CI build just finished; deploy the latest packages to my Test environment!

To that end, most actions that can be done in the Octopus UI can also now be done through a RESTful HTTP API, which Octopus itself uses - Octopus now follows an API-first design.

This week I'll publish a command line tool for triggering deployments, and a class library for building your own tools, to GitHub. Although Octopus itself is still a commercial product, I'm trying to find ways to make many of its "parts" open source, and this is one of them.

Edit: the tool is now published!

Deployment workflow changes

Build servers tend to be pretty sequential:

  1. Get the code
  2. Compile the code
  3. Test the code
  4. Publish artefacts from the code

Release management tools, on the other hand, have a lot of room for parallelism. Let's take this example:

Bob has an environment with five machines - one SQL box and 4 web boxes. The project he is deploying contains two packages - a database package, which has to go first on one machine, followed by an ASP.NET package for his website.

Octopus has to work through these high-level steps:

  1. Download the packages from his NuGet server
  2. Securely upload the packages to the applicable machines
  3. Extract and configure the DB package
  4. Extract and configure the Web package

Those steps are also sequential, and for a good reason: package uploading often takes a long time. It wouldn't be good to upload the DB package, extract it, only to find that the web servers are offline and the package couldn't be uploaded to them. Since this is the slowest part of deployment, we do it all at the start (this idea was thanks to DamianM).

However, within the step there is certainly room for parallelism. We don't need to wait for the upload to machine A to finish before we upload to machine B. We don't have to wait for the web package install on B to finish before we start installing on C.

In addition, there's also no reason why Jill has to wait for Bob to finish deploying project A to Test before she deploys project B to Production.

So from now on, most tasks in Octopus will run in parallel where possible.

One of the trickiest things about this was actually coming up with a way to capture and display the logs correctly. You can see an example of what the result in this gist. As the deployment runs, a hierarchy of logs appear, all updating at the same time - it's quite cool to watch!

Retrospective

Getting this release out took a longer than I planned, which is down to a mix of features taking longer than expected, infrastructure issues, as well as various holidays. In between this release, I migrated everything to Amazon, followed by migrating back to LeaseWeb. In future I'm going to try a more disciplined branch-oriented model so that I can make a new release every couple of weeks, and work towards Octopus 1.0.

If you'd like to know what I'm working on next, visit the Octopus Trello board. I'm also hanging out on Jabbr a lot lately, and there is an Octopus room if you have any questions.

Happy deployments!

Read more