Pigeon

I'm still experimenting with building games, and one of my projects is a little client/server game. Rather than using WCF and dealing with the leaky abstractions, I decided to write something small and custom.

Pigeon is an alternative to WCF designed for high throughput.

On my local machine, WCF NetTcpBinding maxes out at about 10,000 messages/second, while Pigeon achieves 40-50,000 messages/second.

Messages

Messages are encoded using Google Protocol Buffers. You just have to decorate your C# classes with the following attributes:

[Message(57)]
public class CreateCustomer
{
    [MessagePart(1)] public string FirstName { get; set; }
    [MessagePart(2)] public string LastName { get; set; }
    [MessagePart(448)] public int Age { get; set; }
}

You don't have to use the same class library/DLL on the client and server. Instead, the number 57 in the Message attribute above is used to identify the type. So long as the client and server have a type with the number 57, and attributes numbered 1, 2 and 448, even if the classes have different names, it will just work.

Client example

First you configure the client - here I'm connecting to my loopback IP address on TCP port 90001.

var builder = new MessageClientBuilder("127.0.0.1", 90001);
builder.KnownTypes.Add(typeof(CreateCustomer));

var client = builder.Build();

We need the KnownTypes.Add call to make the deserializer aware of the CreateCustomer class, so that if it is told to deserialize 57, it knows which class to create.

After we create the client, we can listen for messages from the server:

client.MessageReceieved += MessageReceieved;
client.Start();

...

private void MessageReceieved(object sender, object message) 
{
    Console.WriteLine("Got message: " + message);
}

The call to client.Start creates a new background thread, which sits in a loop raising the MessageReceieved event each time a message is read from the TCP socket. Note that this means your MessageReceived handler will be called from a background thread.

Finally, the client can send messages to the server:

client.Send(new CreateCustomer { FirstName = "Paul" });

This will queue the message for sending by another background thread, leaving your application code to continue running uninterrupted.

Server example

Writing a server is a little more complicated, since you need to track which clients are connected, and send messages to specific clients.

The server is configured in a similar way to the client - it needs a TCP port number and known types:

var builder = new MessageServerBuilder(90001);
builder.KnownTypes.Add(typeof(CreateCustomer));

var server = builder.Build();
server.MessageReceived += MessageReceived;
server.Start();

...

private void MessageReceived(object sender, MessageReceivedEventArgs e)
{
    var createCustomer = e.Message as CreateCustomer;
    if (createCustomer != null) 
    {
        // Create the customer
        var newCustomerId = SaveNewCustomerToDatabase(createCustomer.FirstName);

        // Reply back to the client (e.Sender), informing them that the customer 
        // was created
        server.Send(new CreateSuccess(newCustomerId), e.Sender);
    }
}

The server can also broadcast a message to all clients:

server.Broadcast(new HappyNewYear());

FAQ

How many threads are used?

A simple client application would use four threads:

  1. The main application thread
  2. The send thread, which sends outbound messages to the server
  3. The receive thread, which queues receieved messages from the server for dispatch
  4. The dispatch thread, which raises the MessageReceived event

A simple server application would also use five threads:

  1. The main application thread
  2. The listen thread, which accepts incoming socket requests
  3. The send thread, which sends outbound messages to any client
  4. The receieve thread, which queues receieved messages from the client for dispatch
  5. The dispatch thread, which raises the MessageReceieved event

Note that each of these threads sleep when there is no work to do

Will I run out of memory?

If your application is producing messages faster than they can be written to the sockets, or if you are receiving messages from the socket faster than your MessageReceieved event handler can handle them, messages will be discarded. The memory usage should hit a limit, since there will never be more than a fixed number of messages on the queue at once.

To illustrate, imagine an MMORPG. As the characters walk around the online world, they continually call client.Send(new Moved(currentPosition)) messages to the server. Chances are, if the server is struggling to cope with the number of messages, you'd be happy to discard the Moved message that was sent 20 seconds ago in favour of processing the Moved message that was sent 1 second ago.

A picture of me

Welcome, my name is Paul Stovell. I live in Brisbane and work on Octopus Deploy, an automated deployment tool for .NET applications.

Prior to founding Octopus Deploy, I worked for an investment bank in London building WPF applications, and before that I worked for Readify, an Australian .NET consulting firm. I also worked on a number of open source projects and was an active user group presenter. I was a Microsoft MVP for WPF from 2006 to 2013.

04 Jan 2011

Small point - for a "Google Protocol Buffers" link, this would be a better url; if you are using the specific protobuf-net implementation then fine (but the two aren't synonymous).

If it is using protobuf-net, I'm interested in how your hooking that under the covers - is that using the new v2 code to avoid the protobuf-net attributes? Or is it TypeBuilder work? (the v2 code should be of great interest if you are using TypeBuilder currently). Or is it just a fork using additional attributes?

04 Jan 2011

Also, you might want to see the the SerializeWithLengthPrefix / DeserializeWithLengthPrefix methods which might make life simpler, and there is a Serializer.NonGeneric API which might help (not sure, but it looks like you've maybe hacked that manually into Serializer?)

04 Jan 2011

Sounds good. I actually got excited for a moment, until I noticed you said the code is not on github. Oh well...

POINDRON Fred
POINDRON Fred
05 Jan 2011

Good work ! I currently use WCF in order to get informations from a distant WCF Server. In this case, I call a function like int Foo(string key).

As far as I understand pigeon, it seems that only "fire and forget" type messages can be sent; with no way to get response value (the int value for example)

Could you please tell me if there is a way to use pigeon for such requirements ?

Thanks,

Fred

06 Jan 2011

Fred - Pigeon is async in nature and you cannot expect one message to immediately answer another message. That said, message correlation should solve the problem. If you add a correlation id to your outgoing messages and made sure that the server also returns the id, the client will be able to match a former request with the incoming response.

Krzysztof - really, does it matter that much having the code at github? As long as the code is available somewhere I'm more than happy!

Enough rant.. Paul: Pigeon looks like an easy-to-get-going package for doing fast messaging. Thanks for sharing! It reminds me of another product I looked at years ago, ZeroZ. More features yes, but also harder to get going with.

06 Jan 2011

...and yes, why not let the dispatcher, which already knows about the message types, also know which handler to call for each message?

var builder = new MessageServerBuilder(90001);
builder.KnownTypes
    .Add<CreateCustomer>(createCustomer => 
         { 
             //handle CreateCustomer messages
         });

In theory, we could have an even faster message loop since the message handler doesn't have to check for each known type.

...I'll go and make it myself whether you implement it or not ;)

Adam Peled
Adam Peled
19 Jan 2011

A few questions: 1. Is the client WPF or Silverlight? 2. If it is Silverlight - should it run on trusted mode and installed (rather than on the browser) to be able to work via sockets? 3. If WPF - why not make the client also a server and work in a distributed pub/sub manner? so that it is more like server to server communication, which can handle fall-backs, faults, and problems in a better way than a TCP connection that might drop.