In my previous post, I introduced the messaging topic. Now its time to talk about what I found on the message framework universe. To get a overview about past and upcoming topics, please have look here: Content++.
There were several message frameworks I was come across and played around. The first was of course WCF comes as part of .Net and C#. This is a very convenient way to handle all the message stuff. It uses XML, JSON or binary format and is easy to handle, which makes it a good starting point for learning and prototyping. You can change the protocol after you has written your application without or minor modifications which is nice.
In a trading platform you end up with masses of data and objects to be serialized and de- serialized. Think about all market data streams and order states needs to be processed. So its worth to have a closer look over different serializing libraries and their differences. Please keep in mind this is just a raw over view and there is no claim to completeness. If you feel I has overseen a tool which is worth to be mentioned, let me know and I would add it.
- Data Contract Serializer
- .Net Binary Serializer
Data Contract Serializer comes on board with .Net and Serializes and deserializes an instance of a type into an XML stream or document using a supplied data contract. This allows you to crossing platforms with you server / client cause XML is a standard format. Using it is straight forward by adding the [DataContract] attribute to the class and the [DataMember] attribute to all fields you want to keep in.
.Net Binary Serializer is another option you would choose if you don’t go cross platform. It converts the object into a binary format which is more efficient about the payload and mostly has better performance. If you want to dig deeper into .Net serialization, I recommend to start with this read.
ProtoBuf-Net is a custom binary serializer solution which ports the Google Protocol Buffer to .Net C#. Many performance benchmarks showing much better performance and smaller payload than most XML, JSON and .Net binary serializers. If you want to go cross platform, you could use the native google version. In case you are using Visual Studio, there are nuget packages for both, ProtoBuf-Net and Google ProtoBuf, you can add to you project. The handling is quite easy and mostly without surprises.
NetSerializer is a lightweight C# serializer wich performs very good. The downside is that not all types are supported yet but the most common you would use are inside. This would be you choice if you want to stay in .Net and searching for the last performance points.
SBE, I never played with it by my self. But if we talk about serialization and trading platforms, I need to mention this project. This is the most cutting edge you would get, no matter if you look at open source or commercial solutions. The guys from Real Logic are members of the FIX High Performance Working Group, were industry standards are defined. This is the most lean and performance oriented solution you would find. Together with their Aeron messaging library it would shift the bottlenecks to your code at least. Its open source and available under Apache 2 License.
If you start to look for messaging libraries and frameworks you see no end. There are so many of them outside covering different aspects of the message problem. It was very time consuming trying out some of them, but the most difficult part was to understand the basics first. I was reading and trying different pattern from simple request <=>response to more advanced poller and actor models – and there was no light at the end of this tunnel. This topic is sooo complex and exciting. After a while I decided to reduce my effort in this and forced me to stay focused on what I really need to start.
Here is small list of messaging libs I came across. If you feel I has overseen a tool which is worth to be mentioned, let me know and I would add it!
MSMQ comes with all Windows systems by default. It provides reliable messaging and has good performance too. Fortunately there is a very good blog post explains well how it works: What Is MSMQ? There is nothing to add from my side.
ZeroMQ is a lightweight message library written in C, with outstanding performance. You can build quite simple communication pattern or very sophisticated high performance networks. By default it comes brokerless, that means app A can talk to app B directly without something in the middle. This enables very direct and fast communication. But you can build a broker into you network with ease too. Anyway, if you want to learn more about how messaging works in general read the ZeroMQ guide -> its a reference!
NetMQ, a C# port of ZeroMQ with an quite active and engaged community. Highly recommended if you go with .Net. If you want, it comes as a Nuget package or you include the source code project into your app. You would get outstanding performance with the end to end managed code experience. Currently what I would use.
RabbitMQ, if you wat to go for a brokered solution, give it a try. Great performance with a nice API. I had a very good experience when I was playing around with it.
ActiveMQ could be your choice if you are with Java or Scala. I have not much experience with it but from what I saw, its worth a look!
Redis is not a data base only. You can use it as message framework too. I never tried this, but in theory is sound really promising. You could easily build you messaging around the Redis database without need using additional tools.
Aeron is a high performance data transfer library. It exists just a couple of month but makes awesome progress. If you are a financial institution with many clients means traffic or you want to go for HFT, this would be you choice. The absolutely cutting edge, the guys from Real Logic care about balancing each Bit… There is a C++ and Java version available. Personally, for my project its not my first choice – think about a 400kW engine in a car with very very small tires;-)
Edit 23.06.2016: There is a .Net version of Aeron out there. Thanks a lot and all best wishes to the creators! Very nice and well structured GitHub repo with many code examples. I will play around with it for sure…
akka is a really interesting piece of technology. It provides an high level abstraction using the actor model. That means, it solves the problem which comes with concurrent and asynchron communication. In short, it helps you to establish reliable parallel and asynchron communication in and between your applications. This is a very interesting concept and I hope later will be time to come into this a little deeper! Akka comes from the Scala inventors and runs in JVM, but there is a .Net port available too.
OK, that’s for now. If you have some comments don’t hesitate to contact me. To get a preview about upcoming topics, please have a look here: Content++.