On 12-10-16 12:56 PM, Joe Conway wrote:
> On 10/16/2012 05:50 AM, Steve Singer wrote:
>
> No, as I understand it from
>    http://slony.info/documentation/slonyupgrade.html
> we would need to:
>
>    1) Stop the slon processes on all nodes. (e.g. - old
>       version of slon)
>    2) Install the new version of slon software on all
>       nodes.
>    3) Execute a slonik script containing the command
>       update functions (id = [whatever]); for each node
>       in the cluster.
>
> We are trying to avoid #1, and in any case cannot easily do #2 (no
> upgrade in place).

Yes that is the only way you can upgrade.

>
> At the moment we are testing with clusters that are all running 2.1.0.
> It is in this configuration where failover is failing.
>
> We *attempted* to run a mixed 2.1.0/2.1.2 cluster so that we could
> failover to the new version, but slon refused to start up in a mixed
> cluster.
>
> We could possibly test a cluster with all 2.1.2, which might be
> instructive, especially if it turns out that the problem we are running
> into is solved in 2.1.2. However we would still have the challenge of
> getting from existing 2.1.0 clusters to 2.1.2 clusters without excessive
> downtime.
>

Yes this will be a challenge. The upgrade from 2.1.0 to 2.1.x is also a 
lot easier than say the upgrade from 2.1.x to 2.2.x.  I don't have a 
good solution for you if you can't shutdown the slons and do an inplace 
UPDATE FUNCTIONS.  This shouldn't necessarily require you to shutdown 
your application, but everyones application environment is different.

>>> Is bug 260 issue #2 deterministic or a race condition? Our current
>>> process works 9 out of 10 times...
>>
>> My recollection was that #260 usually tended to happen, but there are a
>> lot of other rare race conditions I had occasionally hit which lead to
>> the failover changes in 2.2
>>
>> Does your sl_listen table have any cycles in it, ie
>> a-->b
>> b--->a
>> (or even cycles through a third node)
>
> I assume you mean provider->receiver? If so, tons of cycles:
> A->C
> C->A
>

I mean origin, provider, receiver cycles like
1, 2,3
1, 3,2

2,2,3
3,3,2

is normal and isn't a cycle becaues they deal with the source for 
different event origins.

>
> I am not seeing any events in the slony tables now except SYNC events --
> does that mean slon has cleaned out the ones from yesterday when I ran
> into this?
>

There is a race condition that I occasionally have seen in testing where 
the nodes that you are 'simulating' the FAILOVER of generates say
SYNC 1,1234 but when slonik runs this SYNC isn't YET visible on any of 
the other nodes.  It is possible that after slonik picks the most ahead 
node and generates the 'fake'
FAILOVER 1,1234  on another node (say node 2). the real 1,1234 might 
show up in sl_event on a third node.

The nodes will think the failover event has been processed by other 
nodes where they didn't process the FAILOVER event but instead processed 
a SYNC event with the same number.

My best advice to mitigate this is to shutdown the slon on the node you 
are simulating the failover of and then wait a bit to make sure any 'in 
flight' nodes get processed.  I tried to address this in a better in 2.2





>
> Node D (slave2) has processed the failover and shows node C (new master)
> as the set origin. It also seems to have correct/expected rows in the
> other tables (based on comparison with a run that was successful).
>
> Node B (slave1) shows node A (original master) as the set origin.
> However sl_subscribe is correct (provider is C, B and D as the
> receivers, no extra rows), sl_path looks correct, sl_node looks correct.
>
> Node C (new master) shows node A (original master) as the set origin.
> sl_subscribe has two correct rows (provider is C, B and D as the
> receivers) and one extra row (provider B, subscriber C, active false).
> sl_path looks correct, sl_node looks correct.
>
> Node A (orig master) shows node A (original master) as the set origin.
> sl_subscribe has three incorrect rows (provider is A, B and D as the
> receivers; and provider B, subscriber C, active true). The sl_path table
> has "Event Pending" in the path rows for B->C and D->C.
>

What about the ACCEPT_SET ? I suspect it hasn't been confirmed/processed 
on all nodes.



> Joe
>
>