Something Special Making it EVEN better.

RabbitMQ cluster auto-configuration debugging

So being relatively new to RabbitMQ and Erlang, I tried for the first time to auto-configure a cluster of brokers, as opposed to explicitly adding them to a cluster as I have been doing up until now. The premise is simple: 1) make sure all brokers have the same Erlang cookie so they can talk to each other, 2) create a file with a list of nodes in the cluster. The documentation is pretty clear too, "create a configuration file containing the line":

[rabbit@rabbit1, rabbit@rabbit2].

 
Makes sense, I can do that. Here's my configuration file to setup brokers on two Ubuntu VM's:

[rabbit@devins-ubuntu-vm01, rabbit@devins-ubuntu-vm02].

 
Okay, fire up RabbitMQ and everything's gonna work just fine!

=INFO REPORT==== 21-Mar-2010::18:38:40 ===
    application: rabbit
    exited: {bad_return,
                {{rabbit,start,[normal,[]]},
                 {'EXIT',
                     {{case_clause,
                          {error,
                              {cannot_read_cluster_nodes_config,
                                  "/etc/rabbitmq/rabbitmq_cluster.config",
                                  {1,erl_parse,"bad term"}}}},
                      [{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
                       {rabbit,run_boot_step,1},
                       {rabbit,'-start/2-lc$^0/1-0-',1},
                       {rabbit,'-start/2-lc$^0/1-0-',1},
                       {rabbit,start,2},
                       {application_master,start_it_old,4}]}}}}
    type: temporary

As a non-Erlangy person, that's scary. What the RabbitMQ documentation fails to tell you is that the configuration file is actually in Erlang syntax and not some configuration syntax. So what Erlang is seeing in my configuration file is actually something like

devins - ubuntu - vm01

 
I don't know how your math is but that don't compute in Erlang or otherwise. Some poking around the intertubes and it's easy to find out that to quote a string in Erlang you just use the single quote '. This worked just fine though:

['rabbit@devins-ubuntu-vm01', 'rabbit@devins-ubuntu-vm02'].

Next problem...

Although the new configuration worked out, I got a new error, but just on the vm01 node (vm02 started just fine):

=INFO REPORT==== 21-Mar-2010::19:41:19 ===
    application: rabbit
    exited: {bad_return,
                {{rabbit,start,[normal,[]]},
                 {'EXIT',
                     {{case_clause,
                          {error,
                              {unable_to_join_cluster,
                                  ['rabbit@devins-ubuntu-vm01',
                                   'rabbit@devins-ubuntu-vm02'],
                                  {merge_schema_failed,
                                      "Bad cookie in table definition rabbit_user_permission: 'rabbit@devins-ubuntu-vm01' = {cstruct,rabbit_user_permission,set,[],['rabbit@devins-ubuntu-vm01'],[],0,read_write,[],[],false,user_permission,[user_vhost,permission],[],[],{{1269,115383,61181},'rabbit@devins-ubuntu-vm01'},{{2,0},[]}}, 'rabbit@devins-ubuntu-vm02' = {cstruct,rabbit_user_permission,set,[],['rabbit@devins-ubuntu-vm02'],[],0,read_write,[],[],false,user_permission,[user_vhost,permission],[],[],{{1269,195916,246680},'rabbit@devins-ubuntu-vm02'},{{2,0},[]}}\n"}}}},
                      [{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
                       {rabbit,run_boot_step,1},
                       {rabbit,'-start/2-lc$^0/1-0-',1},
                       {rabbit,'-start/2-lc$^0/1-0-',1},
                       {rabbit,start,2},
                       {application_master,start_it_old,4}]}}}}
    type: temporary

 
Boooooo! Thankfully this one was easy though. Find the Mnesia DB directory and kill it all! There's probably a more graceful way, but this gets the job done.

devins@devins-ubuntu-vm01:~$ rm -rf /var/lib/rabbitmq/mnesia

 
Ha! Next restart, and the Rabbit is finally happy.

Debugging

What actually helped me a lot was learning about Erlang's networking utilities and using the Erlang command-line interpreter. Erlang and OTP have pretty first class support for networking and p2p in general, so you can do cool things like one line commands to ping other running Erlang instances on other machines. On each machine, you can enter the Erlang interpreter, set your remote username and a shared cookie:

devins@devins-ubuntu-vm01:~$ erl -sname testuser -setcookie SOMECOOKIE
Erlang R13B01 (erts-5.7.2) [source] [rq:1] [async-threads:0] [kernel-poll:false]

Eshell V5.7.2 (abort with ^G) (testuser@devins-ubuntu-vm01)1>

 
Now on another console, you should be able to ask Erlang if it's up and accepting connections:

devins@devins-ubuntu-vm01:~$ epmd -names
epmd: up and running on port 4369 with data:
name testuser at port 58549

 
Nice, now back in the Erlang interpreter, ping the other machine and hope for a pong back:

(testuser@devins-ubuntu-vm01)1> net_adm:ping('testuser@devins-ubuntu-vm01').
pong
(testuser@devins-ubuntu-vm01)2> net_adm:ping('testuser@devins-ubuntu-vm02').
pong
(testuser@devins-ubuntu-vm02)3> net_adm:ping(testuser@blargh).
pang

 
Notice the last ping attempt. Hard to make it out, but it actually failed with a pang message (pAng, not pOng). Subtle, but important. Also notice that I don't need to quote the string if there are no funky characters in there. Subtle again.

So, long story short, quote funky strings in your RabbitMQ cluster configuration file and you'll be a happy camper!