A picture of me

Welcome, my name is Paul Stovell. I live in Brisbane and work on Octopus Deploy, an automated deployment tool.

Prior to founding Octopus Deploy, I worked for an investment bank in London building WPF applications, and before that I worked for Readify, an Australian .NET consulting firm. I also worked on a number of open source projects and was an active user group presenter. I was a Microsoft MVP for WPF from 2006 to 2013.

Aaron Stanard published a blog post about The Next Decade of .NET Open Source. It's a good summary of recent conversations and there's a lot to agree with in the post. In particular, I agree strongly with this point:

If Autofac is genuinely the better tool for your business’ case than ASP.NET Core’s built-in DI, the right behavior is to support Autofac, not lobby Microsoft to incorporate those features directly into their first party DI.

This point resonated a lot with me. I met a number of people over the years who've said something along the lines of "I really like Octopus Deploy! When Microsoft acquired In Release it wasn't very good. We told Microsoft to make it more like Octopus Deploy!".

Ummm, thanks... I guess? 😪 More on that below.

One part I think the post didn't touch on is that Microsoft projects don't exactly compete on a level playing field with community OSS projects. There's a simple thing Microsoft could do to change this, and it doesn't involve money. But, they won't.

In fact, I think the issue with .NET OSS is fundamentally down to Microsofts organization structure. While Octopus isn't OSS (though much of it is, and we'll invite customers to out GitHub repository), I think our experience is similar to that of many OSS projects that were displaced by a Microsoft product.

How Microsoft could solve it

There's a saying in business that if you want to displace a competitor, you need to build a product that's at least 10x better. It's not enough to be "just as good". Customers will say "why should I use you, we've beeen successful with ". You need a really good reason to overcome that.

However, in the .NET ecosystem, if you're Microsoft, that's not generally true. If Microsoft wants to make a document database, a messaging framework, a unit test framework or a deployment automation tool, it only needs to be 1/10th as good before the conversation immediately becomes "why should we use you over the Microsoft thing?". Microsoft become the default option, even if they're the last to the game.

What's the solution to this?

If you look at the announcements for Microsoft products that compete with their own ecosystem, one thing you'll very rarely see is any acknowledgement of the OSS projects they displace. Any Microsoft voices who'd previously promoted the OSS project will often go quiet after the Microsoft product is released.

If Microsoft decide to release a product (open source or not) in a space where there are existing partners/community projects, I think they should:

  1. Acknowledge the existing alternatives, and make it clear they've got a different use case in mind (this is, usually, the reason)
  2. Link to those alternatives, and highlight situations where those alternatives are better
  3. For all the effort they put into promoting their own product, put at least a small amount of effort into promoting the alternatives

Why it won't happen - my experience at Octopus Deploy

At Octopus Deploy, while we still integrate very well with Azure DevOps, we formally terminated our partnership with Microsoft in 2016 over issues that are similar.

Prior to 2016, Octopus Deploy was the only popular option for .NET developers to automate their application deployments, and we'd single-handedly helped thousands of dev teams to go beyond "right click, publish". In fact, at the time, Octopus Deploy was responsible for a large % of the largest Azure deployments that were happening.

Microsoft had acquired In Release previously, then turned it into Release Management, but were still charging a lot for it. In 2016 they made a bunch of changes:

  • They rebuilt it and turned it into Azure Pipelines, and made it closer to Octopus functionality wise
  • They bundled it into Visual Studio with the "right click, set up CI/CD pipeline" option (Octopus wasn't able to get into this)
  • They changed the pricing model enough for most teams to assume it's free/included in their subscription
  • They began promoting it very heavily

We suddenly found ourselves competing with a product from Microsoft that looked similar, that was being given away (perception, at least), that was integrated with VS, and that was being pushed in every Azure keynote. Overnight it became the default. We were exhibiting at Build 2016 at the time much of this was announced, and I remember people coming to our booth asking "so why should we use you over the Microsoft thing?". The "Microsoft thing" was announced only 5 minutes prior!

There were two conversations I had at Build 2016 that I think shed some light and informed my opinion here.

The first was with a product manager who worked on pipelines/release management. We talked about some integration ideas, but it didn't go anywhere. During the conversation we talked about some ways Octopus and Azure Pipelines differ - and it was clear this guy knew Octopus incredibly well. He described how channels in Octopus - an advanced feature that I'm not sure I even understand sometimes - relate to something in Azure pipelines. To know a feature like that so well, he's done his homework. I'd be incredibly surprised if people working on other products at Microsoft hadn't done their homework on the OSS alternatives.

The second was with someone who was senior within the overall TFS/Azure DevOps group that pipelines belongs to. He made it clear that TFS had its own P&L, and that people's career pathways would be tied to adoption of the products they work on, and so of course people wanted to see them succeed. If Octopus got in the way, even if that might overall be bad for Microsoft, it wasn't their problem.

So if Microsoft release a product that competes with an existing open source community project, I think there's two things you can assume:

  1. It's unlikely they didn't do their research or weren't aware of the alternative.
  2. If it's large in scale, they convinced their bosses to let them work on the project, and there's an expectation that it will gain adoption - either because it's directly tied to their bonuses/performance review, or just political capital. So there's very little reason for them to promote the alternative.

Which brings me back to Aaron's post: it's not going to work.

.NET OSS maintainers will struggle to build professionally supported OSS projects because Microsoft will immediately become the default (in the conversation, at least) in any market they enter, sapping any motivation for contributors to continue. Microsoft could balance that by promoting the alternatives, but that's very unlikely, because it's usually not in the interests of the teams involved.

There are exceptions to this like anything, but I think you see the long arc playing out. We've all seen the comic of Microsoft's org chart with the guns of each team facing each other. What's not obvious is that the .NET open source community exists within that structure too.

As I said, in 2016 we terminated the official "partnership" we had with Microsoft at the time. While a few Microsoft teams are still very friendly to us (and use Octopus), it was wierd. Azure feature teams loved us, and we built integrations with services like Service Fabric to help them with adoption. But overall it was clear that partnering with Microsoft was a waste of time, and a distraction.

(I'm informed that Octopus is still responsible for a huge % of Azure deployments, though nobody at Microsoft will tell me anymore. We've focussed on our customers and building a product they'll love. Explaining "why Octopus over Azure release pipelines" still grates, but I understand why, and we're getting better at it.)

My old friend Mitch Denny (it's been a while!) wrote about the Tyranny of NuGet.

In addition to these two popular uses, NuGet is also used by the new PowerShell Package Management tools as one possible source/provider of packages, once again for installation of software. And finally, NuGet is also exploited by deployment management tools, one of the most popular being Octopus Deploy as a way of shipping files from the package source to an agent responsible for remotely installing software. ... Personally I think that using NuGet for software installation as opposed to dependency management is an abuse of the technology.

I wrote something about this in 2013 in Wanted: a universal application packaging format for .NET. Octopus currently uses NuGet heavily, because there aren't much better choices.

I actually agree quite a bit with Mitch, even the "abusing" part. What I think Mitch missed in the discussion is that there are two parts to NuGet:

  1. The NuGet package file format
  2. The NuGet repository protocol

It's actually the repository protocol that compels Octopus to use NuGet; not the package format. In an ideal deployment automation scenario:

  1. We'd build our binaries once
  2. We'd package and stamp them with metadata (a version number at least) so we know what's in the package
  3. We'd store them in a place where we can refer back to them at any time

Keep in mind that in deployment automation, there are often more than one "living" package at a time. 1.3 is in staging and needs to go to production. 1.4 has been tested and is ready for staging tomorrow. We don't want to rebuild from source code, and we can't just have a "latest"; we need to track versions.

With MSI's, of course, you can do #1 and #2. But how do you do #3? We could chuck MSI's on a file share, but there's just no ecosystem of tooling support around that, no standardised way to sort and query.

More importantly, how does a deployment automation tool make it easy for various build tools to contribute packages to #3? How can we query #3 to work out what versions are available?

The NuGet repository protocol defines all of that, so by using NuGet packages, packages for Octopus can be stored in a ton of places:

  • Octopus itself (built-in NuGet repository)
  • TeamCity (which speaks NuGet)
  • VS Online (which (finally!) speaks NuGet)
  • Proprietary repository managers like Artifactory, Nexus, ProGet
  • MyGet
  • Self-hosed NuGet repositories

If Octopus was based on MSI's, how would we ever integrate with any of those tools?

I'm happy to admit that we're abusing NuGet. But I'd phrase it as: NuGet would be perfectly suited to what we are doing with it, but the team are pretty focussed on just being a library packaging tools and have tightly coupled NuGet to that problem domain instead of allowing it to be general purpose, which it would otherwise be perfectly capable of becoming. While NuGet is OSS and pretty open, the community using it outside of these intended purposes can't really change that direction.

That said, we're going to open Octopus to simpler packaging formats - plain old .zips and .tar.gz's with a naming convention - because NuGet just isn't suited to the non-.NET universe. But we're still left without a common repository protocol format, so this will only work with the built-in Octopus package repository.

But this is still a very compelling idea to anyone serious about a continuous delivery pipeline:

Artifact repository: it is a key resource which stores the binaries, reports, and metadata for each of your release candidates.

And until anyone can point to an alternative repository protocol that's supported by a whole ecosystem of tools our customers are using, NuGet remains a pragmatic choice for this. Abuse or not :)

I open sourced a thing! ServiceBouncer on GitHub

As a developer building Windows Services, the workflow of constantly stopping and starting services can be a bit annoying, especially if your solution consists of multiple services. ServiceBouncer is a simple tool that helps to streamline this workflow.

ServiceBouncer main window

At the start of the year I wrote about how 2014 would be my year to focus on growth:

Most of 2013 was spent on building Octopus 2.0, and I think the product that we have is awesome. My goal for 2014 is to focus on growing the business, by spending 60-80% of my time on growth-oriented tasks, rather than writing code.

Octopus grew at an average of about 40% each quarter this year, which is a huge amount. We went from a two person company to seven, and we went from working at libraries to having our own office. We took an investment from Red Gate, but our growth started before then and was all funded by revenue. And on the personal front, I got married and had a baby this year, so it was busy in many ways!

My goal was to learn a lot about marketing and specifically advertising, assuming that these would be the engines of growth. I assumed it would mean sinking thousands into AdWords and banner ads, running A/B tests, optimizing, sponsorships and other "traditional" tech marketing (the kinds of tasks you'd hire a "marketing person" to do). I spent a lot of time on those activities, and we had experts to help on them too.

It turned out that none of those things really work. No one searches for deployment automation tools, it's still not an established market. Adopting something like Octopus is a journey that involves changing your practices, your processes, and changing minds on your team. No matter how many banner ads you see, they aren't going to convince you to embark on such a journey.

When I look over what worked this year, it's all the things we did before. We blog. And not in a sleazy, content marketing, click-bait way. We blog about what we're working on, and lessons we've learned along the way. We blog plans for the future, feature concepts, and problems we're having. We're active on Twitter. When suggestions come to our UserVoice site, we actually complete them (even if it takes a while).

And fundamentally, we have a pretty good product, which solves a problem important enough that people want to talk about it. At nearly every major conference this year, someone who doesn't work for us has presented on Octopus Deploy. Every week there are new blog posts. That's all down to the product and how we engage and communicate. All of those traditional tech marketing activities took away time that I could have spent making the product even better.

All in all, it has been a great year for Octopus. And while I may have wasted too much time on advertising and other activities that turned out to be a distraction, I'm glad I gave it a shot. The best part about this job is I'm learning and growing in so many new ways every day.

Providing good customer support is the most important part of running a software business. When dealing with a support queue, it's very tempting to look at the list of tickets, and to try and work your way through them by responding to as many as you can, as quickly as you can. I had an experience today that reminded me about the importance of actually solving problems vs. simply responding to tickets.

Last week, I wanted to add a new AD domain to a Windows Azure subscription. I entered the domain, tried to verify it, and got told the domain was already in use. I couldn't figure out why, so I paid $30 for a support contract so that I'd be able to contact Microsoft support.

I lodged the ticket, and a few hours later got a reply. It turned out that I had an Office 365 account which must have had the domain already linked, so I'd need to sign in and delete it from there. As far as I knew I had closed the account and was no longer paying for it, but apparently it was still open. So I spent 10 minutes trying to remember my credentials for the service, and then eventually had to reset the password.

Upon getting into the account, I tried to delete the domain. It pretended to be deleting for 15 minutes, then eventually failed. I tried again and got the same error, so I then waited until the next day and tried again. When I got the same error, I emailed my support person again, and was told it's probably because I have users using that domain.

The steps to rectify? I need to create a new user account, then delete the other users, then I might be able to delete the domain. It sounds like a lot of work, so I'll need to wait until I can set aside the time to do it.

Now, in their defence, the support person at Microsoft has been very diligent and has done a good job - they have phoned me me every day to see where I'm up to on the ticket. They send polite emails and explain the steps properly.

The fact that it's taken too long to resolve is my fault. I'm busy, and this isn't a priority issue. I just want to add a domain, and figured it would take 10 seconds to do. Between the emails and trying to delete things and trying again and searching, it's probably taken a couple of hours (not to mention the $30). And when I see the support person's # on my phone, I screen the call, because it's usually not a good time - I just haven't had time to set aside half an hour to create the accounts and delete the other accounts and do whatever else they said I'd need to do.

We provide technical support at Octopus Deploy, and it's common for some issues to take multiple emails to solve. Being on the receiving end this time has helped me to realise a few things:

  1. If you can, just solve it for them. If the support person had just logged into my accounts and deleted everything for me (perhaps with confirmation first), it would have taken 2 minutes of my time to reply to that email and the ticket would be closed. We sell shrinkwrap software, so that's hard; but this is SaaS, so it should be easy.
  2. Try to think ahead and pre-empt what could go wrong when giving a solution. I got told to delete the domains, so I tried. Only when it failed did they point out that user accounts could be using the domain. That's probably a common issue; if they'd suggested that in the first email, it would save a roundtrip.
  3. Measure response times, not issue open times. I'm getting calls and emails each day from this (very diligent) support person. I'm sure they genuinely want to help, but I suspect the follow ups are more due to the fact that this ticket is "open" and they're reminded each day to follow up. Respond to the ticket quickly, but don't follow up every day to try and get it closed - give the customer time to work through it. Chances are, this issue isn't a priority.
  4. Don't change communication mediums. I contacted support via an online ticket system. Getting a phone call was unexpected and annoying. If I phone you, sure, call me back. But if I email you, just email me back. The fact that I'm not screaming down a phone suggests I probably want to solve this using an asynchronous communication mechanism.

Observation #1 is probably the most interesting. Between the time spent on emails and phone calls just from their side, I'm sure it would have saved Microsoft a lot of money if they could simply have deleted it for me.

Our team at Octopus has grown a lot this year, and we're going to be even bigger by the end of the year. Right now there are 7 of us (5 full-time). Next week, Damian will be starting, and in addition we're also actively hiring for four more roles. Talk about busy!

When it was just me working on Octopus, I worked from home. When Nick joined, we started to meet occasionally at libraries and coffee places to work together, while also working from home most days. When we grew to 5 full time, coordinating which days to meet up was hard. So as a team, we settled on a plan of meeting up and spending the day working together every Wednesday, which creates something consistent and reliable.

But with 5+ people, working from libraries and coffee shops just doesn't work. So a few months ago we started looking for office space, and after a long search we found something we liked. We made an offer, and after what seemed like forever, our purchase finally went unconditional:

Our office

540C Queen Street, Brisbane

It's 121 square metres, so I'm hoping to get a couple of years out of it before we outgrow it.

It sounds funny to say, but despite having now committed to a physical office space, I still can't quite predict how the space is going to be used.

I still believe that people should be able to work from home if/when they want. After all, unless there are small children in the house, there's usually no place more comfortable and quieter than your own home. And if there are small children in the house, I'd encourage you to work from home anyway, because they only stay little for so long. Take time to enjoy it while you can.

What I suspect will happen is that the office will become a "resource". We'll keep meeting up as a team every Wednesday to get that face-to-face collaboration time. If two people are working on a feature, perhaps it will be easier for them to use the office than to try and find space at a coffee shop.

While I always dreamed of having the Fog Creek private office layout, that doesn't seem to be the kind of layout that would work for us. If you want a quiet space to work, just stay home and save the commute. If you're in the office, you are there to collaborate, so we need a layout that facilitates that.

The action plan at the moment is:

  1. Get the keys
  2. Do some minor work to make it usable - there's a very noisy pipe that needs fixing
  3. Add desks, chairs and monitors
  4. Try it for a month, then decide what to do next

To make pairing easier, we've gone with some large, straight desks rather than corner desks. For chairs, of course it had to be Aerons. And I have half a dozen of these 27" Dell monitors in my garage waiting until the day we can move in (everyone gets a $4,400 laptop the day they start, so we don't need to worry about anything else).

Once we've used the space for a bit, then we'll take some time to make changes. There's a wall that I suspect we'll tear down to make for a much wider open space, and then we'll add a smaller meeting room or two (for phone calls) along the side. Then I want to paint the walls with whiteboard paint to make for a lot of collaborative space.

Once we're in I'll post some pictures. Until then, I need ideas! How would you design your ideal office space? What are some cool features we could add?

Special thanks to my wife (and our CFO) Sonia, who did so much work to find the space and organise the purchase. You rock!

In Octopus, we use RavenDB as our database for almost everything. Over time, I'm noticing that we really keep two distinct sets of data in Raven, and I'm beginning to reconsider whether having one store for both of them is the best solution.

The two sets of data I'm referring to are:

  • Definitions. Environments, machines, projects, deployment steps, variables, and so on. They define what will happen when you deploy something. There usually aren't more than a couple of thousand of any of them. As a user, you spend time crafting and modifying these.
  • Records. These collections include releases, deployments, audit events, and so on. Basically, they are a record of the actions you've done using the definitions above. These collections can balloon out to many thousands, perhaps millions of documents. As a user, these are created from simple clicks or even automatically via API's/integration.

When diving in deeper, there are some other notable characteristics:

  • Simple vs. complex: The first set tend to be very complicated documents, with a lot of nesting, dictionary properties, and so on - perfect for a document database. The second set tend to be small and flat, with just a handful of fields
  • History: when you change variables or deployment steps, we take snapshots of them, so that old releases can still use the old values while new releases get the new values. This means that we keep multiple versions of the same document in Raven.
  • Retention: retention policies only really apply to the second set of collections.

The first set of documents lend themselves well to a document database, but there are so few of them that I'm not sure we do need a database behind them. The second set of documents would probably work just as well (if not better) in a relational database.

Our documents in Raven

Right now, Raven makes a good store because it is both a document database (great for our complex definition documents) and is also able to handle our record data. But it's starting to strain. A customer with 43,000 documents sent us a backup, and it took 20+ minutes to rebuild all the indexes before it was usable. Audit events work nice until we have hundreds and thousands of them; ideally we'd keep them all and not apply a retention policy, but then we have to pay to index them all the time.

Perhaps, just maybe, a hybrid storage solution might make more sense?

Tonight I spent some time prototyping a Git-based "document" store. From a C# API point of view, the goal is for it to feel just as friendly as a document database:

using (var session = store.OpenSession()) 
   session.Store(new Project("My project") { ... });
   session.Store(new Project("Another project") { ... });
   session.Commit("Added some projects");

Under the hood, it uses JSON.NET to serialize objects to JSON, and persists them on disk, followed by committing them to a local Git repository (libgit2sharp is amazing, by the way). What's exciting is that this would give our users a whole lot of benefits that come for free when using Git: viewing history, diffs, branching, merging, pushing to TFS or GitHub, and so on.

The missing feature is obviously indexing. Since these tend to be small data sets, however, I think we can probably get away with just putting everything in memory and filtering with LINQ when we need to. We could easily load a few thousand machines into a list, and keep them there until they are invalidated.

While this would work for the first set of documents I talked about, it wouldn't work for the second set. For that, I suspect we would switch to a relational database that can be embedded (probably SQLite). These records have few columns, are quite flat, and would only need to be indexed on one or two columns.

TeamCity appears to use a similar hybrid storage model, with project definitions and build configurations stored as XML files on disk, but records of individual builds stored in a database. As for Octopus, I don't know if or when we'll make the move, but it feels like a hybrid storage model that embraces the differences between the two sets of data is only going to become more compelling in the future.

In the mean time, if anyone is interested in collaborating on a Git-backed persistance solution, let me know :-)

We use RavenDB in Octopus, and one of the features Raven promotes is the idea of Safe by Default, which actually caused us a big, unsafe problem.

Safe by default

The feature in question is "Unbounded result set protection". In short, it looks like this. Say you have 10,000,000 documents of type Foo, and you do this:


The most you'll ever get back is 128 records (embedded mode), or 1024 records for an externally hosted server.

In fact, even if you do this:


In embedded mode you'll still just get 128 records.

This "feature" has hit us plenty of times in Octopus in the past - we didn't expect people to have a lot of projects, so when someone added their 129th project, you can guess what the bug report was. But overall, not too bad, and a good bug to have if you charge for support (sadly, we don't :-)).

I always considered it an annoyance, an opinionated design that goes a little too far, until today.

In Octopus we have a built-in NuGet server, and users can push NuGet packages to it. To decide whether to delete a package, we need to work out which packages are used by a given release - if the package isn't used, it is safe for deletion.

Our query was:

 var releasedVersions = await session.Query<ReleasedPackageVersionsIndex.Result, ReleasedPackageVersionsIndex>()

Today I logged in to our demo server to find the packages had all been deleted; you can spot the bug.

We've been working with Raven for many years now and even still, bugs like this creep in - it's just too easy to call ToList() without considering what might happen. Perhaps it's our own fault, or perhaps this 'safe by default' opinion goes just a little too far. In this case, it would be safer to run out of memory or crash than to delete files.

We use Tender for most of the Octopus Deploy support. I really like Tender because:

  • It allows for public (default) and private threads - we get very few duplicates because most people search for existing threads first
  • It allows customers to report and reply to issues via email

However, the admin UI for Tender, at least from my point of view, leaves a lot to be desired. Most of our customers are overseas, so new topics are usually created overnight. Each morning, we have between 15 and 20 discussions to reply to on Tender. The default Tender admin experience requires a lot of navigation to open each thread, scroll to the bottom and reply to it.

So, I built TenderSmash:


TenderSmash is an alternative UI on top of Tender, that uses the Tender API's to list pending discussions and reply to them. I did a few things to try and optimize the experience:

  • It loads all the threads and comments up front - so you can just read, reply, read, reply, without waiting between threads.
  • It shows comments in reverse order - most of the time you only need to read the most recent comment before replying
  • It uses some quick response templates to generally reduce the amount of typing required

You can play with TenderSmash here:


TenderSmash is open source - you can find the code on GitHub.

There are many dimensions to a business:

  • Product
  • Support
  • Marketing
  • Sales
  • Operations
  • Partners
  • Vision

At any given time, the business can be growing (or shrinking) along one or many of these dimensions. Early on, I thought that launching a business was all about the product. But now, I'm starting to see that the product is just one part of the business, and that no single one of these dimensions is more important than any other.

The first lesson I've learned is that whenever one of these dimensions is growing faster than the others, it can lead to pain:

  • When you've spent 6 months working on the product, without equally investing in finding and talking to customers, it will be painful
  • When you quickly gain customers but don't have a big enough team to give them great support, it can be painful
  • When you have a beautiful roadmap and vision of the future, and a backlog of features customers really want, but not enough developers to build them, it can be painful
  • When you spend all your time on marketing and promotion, but haven’t invested in a product that can meet the expectations, it's going to be painful

To avoid pain, all of these dimensions need to grow in unison. When they aren't, it looks like this:

Dimensions of the business. Numbers are arbitrary, to give you a sense.

Right now, our pain point is support. We want to give everyone a great experience with the product and to be as helpful and responsive as possible if people encounter issues. But we're growing, and so is the support load, and getting through it can easily occupy most of the day. It's not a problem that we can't overcome, but it is an example of where we haven't grown uniformly and are therefore feeling pain.

Tomorrow, I'm interviewing candidates for a support role to try to add some balance. And that's the second lesson: as I get better at this, I hope I'll be able to anticipate when one of these dimensions is going to start to lag, and to be more prepared. Ideally, I would have hired someone to help with support 3 months ago, so that by now they’d be an expert.