Consider a three node cluster with nodes 1,2 and 3.

I then issue the command.

store node(id=4, event node=1);
subscribe set(id=1,provider=2,receiver=4);

Slonik needs to wait for the store node to propogate to node 2 before 
doing the subscribe set.   The problem is that node 2 does not yet have 
an sl_node entry for node 4.

Slonik can't tell the difference between this case and

store node(id=4,event node=1);
#wait
store node(id=5,event node=4);
drop node(id=4,event node=2);
subscribe set(id=1,provider=2,receiver=4);

If slonik does not see an sl_node entry for a node 4 at node 2 how does 
it know if this is because the store node hasn't yet propogated or if 
the drop node has already blown away the data.

Is this a problem for multi-node failover as well?

Ways to avoid this problem that I can think of are:

1) Don't allow node id numbers to be re-used.  I think many users will 
come at us with pitchforks if we do this route since many places/scripts 
assume they have a node=1 and node=2 and sometimes drop then recreate a 
node with the same id so they don't break there scripts.

2) We create a node uuid that is some node identifier that a) does not 
ever appear in any slonik commands.  We use this value as the node id in 
sl_confirm and sl_event.   When we submit events the user provided node 
id get's mapped to the proper uuid.   Will this work or solve the 
problem?  I'm not sure yet.

3) We could have slonik check sl_node on all other nodes to see if the 
node is active anywhere else.  If it is we wait, if at some point the 
node is deleted from sl_node on all nodes then slonik can assume that it 
is a drop node.  I don't like this because it would involve slonik 
constantly polling all nodes in a cluster (during a wait) if it can't 
find an sl_node entry on the node it is waiting on but if it only did 
this when a node entry was missing maybe it isn't so bad.

Are other issues going to force us to implement approach 2 for 2.1 (ie 
something in multi-node failover).

Also 3 is pretty incompatible with the slon side failover but 2 is a 
chunk of work. (actually if make the existing no_id columns in sl_node, 
sl_event,sl_confirm and all the stored procedures refer to a uuid and 
then make the slonik commands map the id=blah to the new column we add 
to sl_node then this might not be so bad but a bit confusing in that a 
node id in a slonik command refers to a different column than sl_node.no_id

Thoughts?