Lessons learnt building a shrinkwrap .NET server product

When I'm working on a client project, especially one that's been under development for a while, I often find myself wondering what would I do differently given the chance to start over. Octopus, as my own product, is no exception. They say hindsight is 20/20, so it's often interesting to think about how to apply some of the lessons learnt.

Things that didn't go as expected

The IIS/ASP.NET/WCF/SQL stack which Octopus relies on is great for building Enterprise applications. I have plenty of experience with that stack, which is mostly why I went with it in designing Octopus.

I learnt that an enterprise stack isn't necessarily great for building ISV shrinkwrap products like Octopus. The following decisions have been beneficial, but have also come with downsides:

1. Depending on IIS
   • Getting the configuration right is very tough - people need to make sure they enable static content to be served, ASP.NET 4.0 is registered and the extension isn't blocked
   • Permissions: by default IIS AppPools run under the context of a machine-specific AppPool user. If using a remote SQL database, users get all kinds of login problems.
   • Since IIS websites get recycled, I can't use it to run any long-running or scheduled tasks. So I had to create an additional Windows Service, which is just an extra thing to maintain. Sharing configuration between the two is also difficult.
2. Using SQL server
   • A bundled database like RavenDB or SqlLite would probably have been a better choice from a deployment point of view, and a document database would probably suit my application model easier. Also, not having to rely on remote Windows authentication would probably save some of the IIS issues above.
3. Using WCF and certificates
   • The Windows Certificate Store caused so many permission-related issues that I eventually gave up, and instead base64 encoded certificates and stored them in the registry
4. Hosting PowerShell
   • Writing a custom PowerShell host that works is easy. Writing one that really, really works, is very hard
   • Not to mention x86/x64 issues and the availability of modules/snapins on each (IIS7/7.5 is painful for this)
5. NuGet
   • Overall this has been beneficial, though it had teething issues (this caused many problems)
   • CI solutions, especially TFS, are still not very good at bundling an application into a NuGet package
   • Some of the normal NuGet conventions don't make sense in the context of Octopus, but that's hard to educate people on

Lesson 1: If using IIS, have a really sophisticated installer

On the .NET platform, if you want a Windows Service that has a web UI, you have three choices:

1. Use IIS:
   • Automated configuration is messy
   • Requires users to install it with all the right options selected
   • Can't do long-running tasks reliably; you'll need a Windows Service too, and find a way to share configuration
   • Permissions are going to be a problem
2. Use HttpListener (aka HTTP.SYS)
   • Also hard to configure due to permissions
   • Miss out on a lot of IIS goodies like SSL configuration
   • ASP.NET makes a lot of assumptions that you have to cater for if hosting it in your own process
3. Use raw sockets (bypass HTTP.SYS)
   • You'll never be able to get port 80 because HTTP.SYS has probably taken it anyway
   • Same issues as above

IIS seems like the only reasonable solution. But my experiences supporting a product that depends on IIS have taught me that it can be darn hard to rely on.

If I'm going to stick with IIS, I'll need a much smarter installer. My installer needs to be able to:

• Verify that IIS is installed
• Verify that the right options are installed
• Verify that ASP.NET 4.0 is installed, registered and enabled
• Create the site and AppPool
• Configure the security credentials of the AppPool to run under an account that has permissions to contact whatever SQL database the user selects
• Verify that said user has permissions to run as an AppPool in IIS

On the one hand, it seems a shame to put so much effort into creating a clever installer when I could be focussed on the product itself. On the other hand, the solution of simply having pages and pages of documentation and a FAQ seems like a cop-out, and prevents people from easily getting started with the product.

Lesson 2: Don't use SQL Server

A lot of my IIS issues would actually probably be resolved, strangely, by not using SQL. By switching from SQL Server to an embeddable, local database, means I don't have to deal with Windows Auth issues when using a remote SQL server. That means no need to run under a custom user account most of the time, which simplifies things a lot. That's something I'm probably going to start looking at seriously over the next few weeks.

Lesson 3: Have a good test environment

I can't begin to count how many issues have been the result of differences on x86/x64, Server 2003/2008/2008R2/2008R2SP1 issues. I always knew I'd eventually need a test rig with lots of OS versions, but I hoped most things would "just work" and that could wait until later. It turns out that building a server product that runs on more than one version of Windows Server is hard work!

Conclusion

The great thing about all of this is that Octopus is bearing the pain so that others don't have to. Octopus makes it darn easy to take the ASP.NET project you built on your internal CI server, securely upload it to a locked-down server deep in a colocation facility, extract it and execute your PowerShell scripts to configure it. And that makes me happy!