RabbitMQ HA testing with HAProxy
Lots of news and talk lately about RabbitMQ being scooped up by SpringSource (a division of VMWare itself a division of EMC) but there's still work to be done! RabbitMQ has fantastic performance and scalability characteristics, but one thing it doesn't do out of the box (yet) is easy HA. As can be seen by the many posts to the mailing list, it's also a common problem. There are some guides in the works and community projects like Beetle but I have no doubt that SpringSource investing time and money in RabbitMQ will bring some basic HA to the table.
In the meantime, we have a few options. One of the easiest is to simply setup RabbitMQ in active-passive configuration behind a TCP load balancer. Producers publish through the load balancer and consumers consume from all of the individual nodes. The big issue here is that there is no library support for this setup (at least for Java clients), but a little bit of hacking on the standard Java client library and you can come up with a client abstraction that nicely fails over for you.
Since we use big, expensive F5 BigIPs at work, we as lowly, untrusted developers have limited access to setup and configure them. In comes HAProxy. I setup two VMs, rabbit01 and rabbit02 and dropped HAProxy on rabbit01 to simulate the real deployment of producers being behind the load balancer. Here's the configuration I used.
global log 127.0.0.1 alert log 127.0.0.1 alert debug
defaults log global mode http option dontlognull option redispatch retries 3 contimeout 5000 clitimeout 50000 srvtimeout 50000
listen rabbitmq 192.168.56.11:5000 mode tcp stats enable balance roundrobin option forwardfor option tcpka server rabbit01 192.168.56.11:5672 check inter 5000 downinter 500 server rabbit02 192.168.56.12:5672 check inter 5000 backup
From here, it's pretty easy to test the failover. Fire up some of the test producers and consumers from the Java client library binaries and test away, using rabbitmqctl stop_app and start_app on the RabbitMQ nodes to simulate them coming up and down.
More posts to follow on an HA Java client library (i.e. as soon as I finish writing it)!
April 16th, 2010 - 19:28
Regarding HA Java client libraries, check out RabbitMQ’s experimental Java message patterns library at http://hg.rabbitmq.com/rabbitmq-java-messagepatterns/
There is a .net version of this library too.
The library provides for automatic reconnect and retransmission. Currently it only deals with one particular message pattern – point-to-point – but it should be possible to extend this to other use cases.
April 16th, 2010 - 19:57
Hey Matthias, I saw someone (Ben I think?) post about that library on the mailing list too. I’m just working on a client now that will be based on the point-to-point version in there.
April 17th, 2010 - 06:41
If you want to test against the F5 Big IPs functionally as a developer check out https://www.f5.com/trial/ It is a virtual edition of Big IP LTM.
April 17th, 2010 - 10:41
Thanks Sanjay, that could be very useful for several things I’m working on! I’ll give it a shot too.