A picture of me

Welcome, my name is Paul Stovell. I live in Brisbane and work on Octopus Deploy, an automated deployment tool for .NET applications.

Prior to founding Octopus Deploy, I worked for an investment bank in London building WPF applications, and before that I worked for Readify, an Australian .NET consulting firm. I also worked on a number of open source projects and was an active user group presenter. I was a Microsoft MVP for WPF from 2006 to 2013.

My old friend Mitch Denny (it's been a while!) wrote about the Tyranny of NuGet.

In addition to these two popular uses, NuGet is also used by the new PowerShell Package Management tools as one possible source/provider of packages, once again for installation of software. And finally, NuGet is also exploited by deployment management tools, one of the most popular being Octopus Deploy as a way of shipping files from the package source to an agent responsible for remotely installing software. ... Personally I think that using NuGet for software installation as opposed to dependency management is an abuse of the technology.

I wrote something about this in 2013 in Wanted: a universal application packaging format for .NET. Octopus currently uses NuGet heavily, because there aren't much better choices.

I actually agree quite a bit with Mitch, even the "abusing" part. What I think Mitch missed in the discussion is that there are two parts to NuGet:

  1. The NuGet package file format
  2. The NuGet repository protocol

It's actually the repository protocol that compels Octopus to use NuGet; not the package format. In an ideal deployment automation scenario:

  1. We'd build our binaries once
  2. We'd package and stamp them with metadata (a version number at least) so we know what's in the package
  3. We'd store them in a place where we can refer back to them at any time

Keep in mind that in deployment automation, there are often more than one "living" package at a time. 1.3 is in staging and needs to go to production. 1.4 has been tested and is ready for staging tomorrow. We don't want to rebuild from source code, and we can't just have a "latest"; we need to track versions.

With MSI's, of course, you can do #1 and #2. But how do you do #3? We could chuck MSI's on a file share, but there's just no ecosystem of tooling support around that, no standardised way to sort and query.

More importantly, how does a deployment automation tool make it easy for various build tools to contribute packages to #3? How can we query #3 to work out what versions are available?

The NuGet repository protocol defines all of that, so by using NuGet packages, packages for Octopus can be stored in a ton of places:

  • Octopus itself (built-in NuGet repository)
  • TeamCity (which speaks NuGet)
  • VS Online (which (finally!) speaks NuGet)
  • Proprietary repository managers like Artifactory, Nexus, ProGet
  • MyGet
  • Self-hosed NuGet repositories

If Octopus was based on MSI's, how would we ever integrate with any of those tools?

I'm happy to admit that we're abusing NuGet. But I'd phrase it as: NuGet would be perfectly suited to what we are doing with it, but the team are pretty focussed on just being a library packaging tools and have tightly coupled NuGet to that problem domain instead of allowing it to be general purpose, which it would otherwise be perfectly capable of becoming. While NuGet is OSS and pretty open, the community using it outside of these intended purposes can't really change that direction.

That said, we're going to open Octopus to simpler packaging formats - plain old .zips and .tar.gz's with a naming convention - because NuGet just isn't suited to the non-.NET universe. But we're still left without a common repository protocol format, so this will only work with the built-in Octopus package repository.

But this is still a very compelling idea to anyone serious about a continuous delivery pipeline:

Artifact repository: it is a key resource which stores the binaries, reports, and metadata for each of your release candidates.

And until anyone can point to an alternative repository protocol that's supported by a whole ecosystem of tools our customers are using, NuGet remains a pragmatic choice for this. Abuse or not :)

I open sourced a thing! ServiceBouncer on GitHub

As a developer building Windows Services, the workflow of constantly stopping and starting services can be a bit annoying, especially if your solution consists of multiple services. ServiceBouncer is a simple tool that helps to streamline this workflow.

ServiceBouncer main window

At the start of the year I wrote about how 2014 would be my year to focus on growth:

Most of 2013 was spent on building Octopus 2.0, and I think the product that we have is awesome. My goal for 2014 is to focus on growing the business, by spending 60-80% of my time on growth-oriented tasks, rather than writing code.

Octopus grew at an average of about 40% each quarter this year, which is a huge amount. We went from a two person company to seven, and we went from working at libraries to having our own office. We took an investment from Red Gate, but our growth started before then and was all funded by revenue. And on the personal front, I got married and had a baby this year, so it was busy in many ways!

My goal was to learn a lot about marketing and specifically advertising, assuming that these would be the engines of growth. I assumed it would mean sinking thousands into AdWords and banner ads, running A/B tests, optimizing, sponsorships and other "traditional" tech marketing (the kinds of tasks you'd hire a "marketing person" to do). I spent a lot of time on those activities, and we had experts to help on them too.

It turned out that none of those things really work. No one searches for deployment automation tools, it's still not an established market. Adopting something like Octopus is a journey that involves changing your practices, your processes, and changing minds on your team. No matter how many banner ads you see, they aren't going to convince you to embark on such a journey.

When I look over what worked this year, it's all the things we did before. We blog. And not in a sleazy, content marketing, click-bait way. We blog about what we're working on, and lessons we've learned along the way. We blog plans for the future, feature concepts, and problems we're having. We're active on Twitter. When suggestions come to our UserVoice site, we actually complete them (even if it takes a while).

And fundamentally, we have a pretty good product, which solves a problem important enough that people want to talk about it. At nearly every major conference this year, someone who doesn't work for us has presented on Octopus Deploy. Every week there are new blog posts. That's all down to the product and how we engage and communicate. All of those traditional tech marketing activities took away time that I could have spent making the product even better.

All in all, it has been a great year for Octopus. And while I may have wasted too much time on advertising and other activities that turned out to be a distraction, I'm glad I gave it a shot. The best part about this job is I'm learning and growing in so many new ways every day.

Providing good customer support is the most important part of running a software business. When dealing with a support queue, it's very tempting to look at the list of tickets, and to try and work your way through them by responding to as many as you can, as quickly as you can. I had an experience today that reminded me about the importance of actually solving problems vs. simply responding to tickets.

Last week, I wanted to add a new AD domain to a Windows Azure subscription. I entered the domain, tried to verify it, and got told the domain was already in use. I couldn't figure out why, so I paid $30 for a support contract so that I'd be able to contact Microsoft support.

I lodged the ticket, and a few hours later got a reply. It turned out that I had an Office 365 account which must have had the domain already linked, so I'd need to sign in and delete it from there. As far as I knew I had closed the account and was no longer paying for it, but apparently it was still open. So I spent 10 minutes trying to remember my credentials for the service, and then eventually had to reset the password.

Upon getting into the account, I tried to delete the domain. It pretended to be deleting for 15 minutes, then eventually failed. I tried again and got the same error, so I then waited until the next day and tried again. When I got the same error, I emailed my support person again, and was told it's probably because I have users using that domain.

The steps to rectify? I need to create a new user account, then delete the other users, then I might be able to delete the domain. It sounds like a lot of work, so I'll need to wait until I can set aside the time to do it.

Now, in their defence, the support person at Microsoft has been very diligent and has done a good job - they have phoned me me every day to see where I'm up to on the ticket. They send polite emails and explain the steps properly.

The fact that it's taken too long to resolve is my fault. I'm busy, and this isn't a priority issue. I just want to add a domain, and figured it would take 10 seconds to do. Between the emails and trying to delete things and trying again and searching, it's probably taken a couple of hours (not to mention the $30). And when I see the support person's # on my phone, I screen the call, because it's usually not a good time - I just haven't had time to set aside half an hour to create the accounts and delete the other accounts and do whatever else they said I'd need to do.

We provide technical support at Octopus Deploy, and it's common for some issues to take multiple emails to solve. Being on the receiving end this time has helped me to realise a few things:

  1. If you can, just solve it for them. If the support person had just logged into my accounts and deleted everything for me (perhaps with confirmation first), it would have taken 2 minutes of my time to reply to that email and the ticket would be closed. We sell shrinkwrap software, so that's hard; but this is SaaS, so it should be easy.
  2. Try to think ahead and pre-empt what could go wrong when giving a solution. I got told to delete the domains, so I tried. Only when it failed did they point out that user accounts could be using the domain. That's probably a common issue; if they'd suggested that in the first email, it would save a roundtrip.
  3. Measure response times, not issue open times. I'm getting calls and emails each day from this (very diligent) support person. I'm sure they genuinely want to help, but I suspect the follow ups are more due to the fact that this ticket is "open" and they're reminded each day to follow up. Respond to the ticket quickly, but don't follow up every day to try and get it closed - give the customer time to work through it. Chances are, this issue isn't a priority.
  4. Don't change communication mediums. I contacted support via an online ticket system. Getting a phone call was unexpected and annoying. If I phone you, sure, call me back. But if I email you, just email me back. The fact that I'm not screaming down a phone suggests I probably want to solve this using an asynchronous communication mechanism.

Observation #1 is probably the most interesting. Between the time spent on emails and phone calls just from their side, I'm sure it would have saved Microsoft a lot of money if they could simply have deleted it for me.

Our team at Octopus has grown a lot this year, and we're going to be even bigger by the end of the year. Right now there are 7 of us (5 full-time). Next week, Damian will be starting, and in addition we're also actively hiring for four more roles. Talk about busy!

When it was just me working on Octopus, I worked from home. When Nick joined, we started to meet occasionally at libraries and coffee places to work together, while also working from home most days. When we grew to 5 full time, coordinating which days to meet up was hard. So as a team, we settled on a plan of meeting up and spending the day working together every Wednesday, which creates something consistent and reliable.

But with 5+ people, working from libraries and coffee shops just doesn't work. So a few months ago we started looking for office space, and after a long search we found something we liked. We made an offer, and after what seemed like forever, our purchase finally went unconditional:

Our office

540C Queen Street, Brisbane

It's 121 square metres, so I'm hoping to get a couple of years out of it before we outgrow it.

It sounds funny to say, but despite having now committed to a physical office space, I still can't quite predict how the space is going to be used.

I still believe that people should be able to work from home if/when they want. After all, unless there are small children in the house, there's usually no place more comfortable and quieter than your own home. And if there are small children in the house, I'd encourage you to work from home anyway, because they only stay little for so long. Take time to enjoy it while you can.

What I suspect will happen is that the office will become a "resource". We'll keep meeting up as a team every Wednesday to get that face-to-face collaboration time. If two people are working on a feature, perhaps it will be easier for them to use the office than to try and find space at a coffee shop.

While I always dreamed of having the Fog Creek private office layout, that doesn't seem to be the kind of layout that would work for us. If you want a quiet space to work, just stay home and save the commute. If you're in the office, you are there to collaborate, so we need a layout that facilitates that.

The action plan at the moment is:

  1. Get the keys
  2. Do some minor work to make it usable - there's a very noisy pipe that needs fixing
  3. Add desks, chairs and monitors
  4. Try it for a month, then decide what to do next

To make pairing easier, we've gone with some large, straight desks rather than corner desks. For chairs, of course it had to be Aerons. And I have half a dozen of these 27" Dell monitors in my garage waiting until the day we can move in (everyone gets a $4,400 laptop the day they start, so we don't need to worry about anything else).

Once we've used the space for a bit, then we'll take some time to make changes. There's a wall that I suspect we'll tear down to make for a much wider open space, and then we'll add a smaller meeting room or two (for phone calls) along the side. Then I want to paint the walls with whiteboard paint to make for a lot of collaborative space.

Once we're in I'll post some pictures. Until then, I need ideas! How would you design your ideal office space? What are some cool features we could add?

Special thanks to my wife (and our CFO) Sonia, who did so much work to find the space and organise the purchase. You rock!

In Octopus, we use RavenDB as our database for almost everything. Over time, I'm noticing that we really keep two distinct sets of data in Raven, and I'm beginning to reconsider whether having one store for both of them is the best solution.

The two sets of data I'm referring to are:

  • Definitions. Environments, machines, projects, deployment steps, variables, and so on. They define what will happen when you deploy something. There usually aren't more than a couple of thousand of any of them. As a user, you spend time crafting and modifying these.
  • Records. These collections include releases, deployments, audit events, and so on. Basically, they are a record of the actions you've done using the definitions above. These collections can balloon out to many thousands, perhaps millions of documents. As a user, these are created from simple clicks or even automatically via API's/integration.

When diving in deeper, there are some other notable characteristics:

  • Simple vs. complex: The first set tend to be very complicated documents, with a lot of nesting, dictionary properties, and so on - perfect for a document database. The second set tend to be small and flat, with just a handful of fields
  • History: when you change variables or deployment steps, we take snapshots of them, so that old releases can still use the old values while new releases get the new values. This means that we keep multiple versions of the same document in Raven.
  • Retention: retention policies only really apply to the second set of collections.

The first set of documents lend themselves well to a document database, but there are so few of them that I'm not sure we do need a database behind them. The second set of documents would probably work just as well (if not better) in a relational database.

Our documents in Raven

Right now, Raven makes a good store because it is both a document database (great for our complex definition documents) and is also able to handle our record data. But it's starting to strain. A customer with 43,000 documents sent us a backup, and it took 20+ minutes to rebuild all the indexes before it was usable. Audit events work nice until we have hundreds and thousands of them; ideally we'd keep them all and not apply a retention policy, but then we have to pay to index them all the time.

Perhaps, just maybe, a hybrid storage solution might make more sense?

Tonight I spent some time prototyping a Git-based "document" store. From a C# API point of view, the goal is for it to feel just as friendly as a document database:

using (var session = store.OpenSession()) 
   session.Store(new Project("My project") { ... });
   session.Store(new Project("Another project") { ... });
   session.Commit("Added some projects");

Under the hood, it uses JSON.NET to serialize objects to JSON, and persists them on disk, followed by committing them to a local Git repository (libgit2sharp is amazing, by the way). What's exciting is that this would give our users a whole lot of benefits that come for free when using Git: viewing history, diffs, branching, merging, pushing to TFS or GitHub, and so on.

The missing feature is obviously indexing. Since these tend to be small data sets, however, I think we can probably get away with just putting everything in memory and filtering with LINQ when we need to. We could easily load a few thousand machines into a list, and keep them there until they are invalidated.

While this would work for the first set of documents I talked about, it wouldn't work for the second set. For that, I suspect we would switch to a relational database that can be embedded (probably SQLite). These records have few columns, are quite flat, and would only need to be indexed on one or two columns.

TeamCity appears to use a similar hybrid storage model, with project definitions and build configurations stored as XML files on disk, but records of individual builds stored in a database. As for Octopus, I don't know if or when we'll make the move, but it feels like a hybrid storage model that embraces the differences between the two sets of data is only going to become more compelling in the future.

In the mean time, if anyone is interested in collaborating on a Git-backed persistance solution, let me know :-)

We use RavenDB in Octopus, and one of the features Raven promotes is the idea of Safe by Default, which actually caused us a big, unsafe problem.

Safe by default

The feature in question is "Unbounded result set protection". In short, it looks like this. Say you have 10,000,000 documents of type Foo, and you do this:


The most you'll ever get back is 128 records (embedded mode), or 1024 records for an externally hosted server.

In fact, even if you do this:


In embedded mode you'll still just get 128 records.

This "feature" has hit us plenty of times in Octopus in the past - we didn't expect people to have a lot of projects, so when someone added their 129th project, you can guess what the bug report was. But overall, not too bad, and a good bug to have if you charge for support (sadly, we don't :-)).

I always considered it an annoyance, an opinionated design that goes a little too far, until today.

In Octopus we have a built-in NuGet server, and users can push NuGet packages to it. To decide whether to delete a package, we need to work out which packages are used by a given release - if the package isn't used, it is safe for deletion.

Our query was:

 var releasedVersions = await session.Query<ReleasedPackageVersionsIndex.Result, ReleasedPackageVersionsIndex>()

Today I logged in to our demo server to find the packages had all been deleted; you can spot the bug.

We've been working with Raven for many years now and even still, bugs like this creep in - it's just too easy to call ToList() without considering what might happen. Perhaps it's our own fault, or perhaps this 'safe by default' opinion goes just a little too far. In this case, it would be safer to run out of memory or crash than to delete files.

We use Tender for most of the Octopus Deploy support. I really like Tender because:

  • It allows for public (default) and private threads - we get very few duplicates because most people search for existing threads first
  • It allows customers to report and reply to issues via email

However, the admin UI for Tender, at least from my point of view, leaves a lot to be desired. Most of our customers are overseas, so new topics are usually created overnight. Each morning, we have between 15 and 20 discussions to reply to on Tender. The default Tender admin experience requires a lot of navigation to open each thread, scroll to the bottom and reply to it.

So, I built TenderSmash:


TenderSmash is an alternative UI on top of Tender, that uses the Tender API's to list pending discussions and reply to them. I did a few things to try and optimize the experience:

  • It loads all the threads and comments up front - so you can just read, reply, read, reply, without waiting between threads.
  • It shows comments in reverse order - most of the time you only need to read the most recent comment before replying
  • It uses some quick response templates to generally reduce the amount of typing required

You can play with TenderSmash here:


TenderSmash is open source - you can find the code on GitHub.

There are many dimensions to a business:

  • Product
  • Support
  • Marketing
  • Sales
  • Operations
  • Partners
  • Vision

At any given time, the business can be growing (or shrinking) along one or many of these dimensions. Early on, I thought that launching a business was all about the product. But now, I'm starting to see that the product is just one part of the business, and that no single one of these dimensions is more important than any other.

The first lesson I've learned is that whenever one of these dimensions is growing faster than the others, it can lead to pain:

  • When you've spent 6 months working on the product, without equally investing in finding and talking to customers, it will be painful
  • When you quickly gain customers but don't have a big enough team to give them great support, it can be painful
  • When you have a beautiful roadmap and vision of the future, and a backlog of features customers really want, but not enough developers to build them, it can be painful
  • When you spend all your time on marketing and promotion, but haven’t invested in a product that can meet the expectations, it's going to be painful

To avoid pain, all of these dimensions need to grow in unison. When they aren't, it looks like this:

Dimensions of the business. Numbers are arbitrary, to give you a sense.

Right now, our pain point is support. We want to give everyone a great experience with the product and to be as helpful and responsive as possible if people encounter issues. But we're growing, and so is the support load, and getting through it can easily occupy most of the day. It's not a problem that we can't overcome, but it is an example of where we haven't grown uniformly and are therefore feeling pain.

Tomorrow, I'm interviewing candidates for a support role to try to add some balance. And that's the second lesson: as I get better at this, I hope I'll be able to anticipate when one of these dimensions is going to start to lag, and to be more prepared. Ideally, I would have hired someone to help with support 3 months ago, so that by now they’d be an expert.

StartupAUS published a report highlighting some of the issues that startup founders in Australia face. It's a good read, covering the problems but also proposing a number of solutions. You can view the report as a PDF, or read a summary from Delimiter:

The report highlights a number of key issues facing Australia, including the fact that Australia’s startup sector is maturing at a slower rate than many other nations; that high-growth tech companies could contribute 4 percent of the nation’s GDP by 2033, compared to just 0.2 percent today (adding 540,000 jobs to the economy); and that in 2013, Australia invested just $4.5 per capita in Venture Capital for start-ups, compared to $120 in Israel, $85 in the US, $20 in South Korea and $15 in the UK.

One thing that surprised me from the report is just how small the startup community here is. After examining how badly we're performing, the report makes a number of recommendations about what should be done. Examples are the creation of an entrepreneur visa, changing the tax treatment of employee share options, better tax treatment for angel investors, and enabling crowd funding.

I don't think what we're doing at Octopus Deploy really counts as a "startup" in the sense described in the report, and perhaps that's why I found the report interesting, but also pretty irrelevant to us. Like most of the discussion about startups these days, it feels heavily geared towards helping would-be founders find VC funding. It's all about chasing funding.

While Australia might be a bad place to start a "startup", it's not too bad of a place to start a bootstrapped business like ours. There really is only one downside: the timezone makes it hard to offer great support to 95% of customers at times that suit them, so we have to wake up early and go to sleep late sometimes. The tyranny of distance naturally makes it difficult to connect with and influence trends in other areas of the planet.

I do have one idea though: our banks could do a lot more to provide solutions to help Australian companies process payments from overseas customers. Ideas like that are pretty irrelevant to the startup scene though, since almost by definition collecting revenue isn't something a startup will ever worry about :-)