On 09/28/2012 07:15 AM, Steve Singer wrote:
> On 12-09-27 05:26 PM, Jan Wieck wrote:
>> On 9/27/2012 3:58 PM, Brian Fehrle wrote:
>>> Follow up:
>>>
>>> I executed this on the master:
>>> mydatabase=# select * from _slony.sl_event where ev_origin not in
>>> (select no_id from _slony.sl_node);
>>>    ev_origin |  ev_seqno  | ev_timestamp          |    
>>> ev_snapshot     |
>>> ev_type | ev_data1 | ev_data2 | ev_data3 | ev_data4 | ev_data5 |
>>> ev_data6 | ev_data7 | ev_data8
>>> -----------+------------+-------------------------------+--------------------+---------+----------+----------+----------+----------+----------+----------+----------+---------- 
>>>
>>>            3 | 5000290161 | 2012-09-27 09:48:03.749424-04 |
>>> 40580084:40580084: | SYNC    |          |          | |
>>> |          |          |          |
>>> (1 row)
>>>
>>> There is a row in sl_event that shouldn't be there, because it's
>>> referencing a node that nolonger exists. I need to add this node 
>>> back to
>>> replication, but I don't want to run into the same issue as before. I
>>> ran a cleanupEvent('10 minute') and it did nothing (even did it with 0
>>> minutes).
>>>
>>> Will this row eventually go away? will it cause issue if we attempt to
>>> add a new node to replication with node = 3? How can I safely clean 
>>> this up?
>>
>> Hmmm,
>>
>> this actually looks like a more severe race condition or even a bug.
>>
>
> The slonik_drop_node function has changed a bit in 2.2(master).  I 
> apparently added the following comment to it while doing some of the 
> FAILOVER work.  It is possible that it addresses this situation as well?
>
>
> * find the last event (including SYNC events)
> * from the node being dropped that is visible on
> * any of the remaining nodes.
> * we must wait for ALL remaining nodes to get caught up.
> *
> * we can't ignore SYNC events even though the dropped
> * node is not an origin it might have been an old
> * origin before a FAILOVER. Some behind node still
> * might need to get caught up from its provider.
> */
> ev_id = get_last_escaped_event_id((SlonikStmt *) stmt,
> stmt->no_id_list[no_id_idx],stmt->no_id_list);
>
> Also, Brian,
>
> Did your slonik script (for the DROP NODE command) include admin 
> conninfo lines to all your nodes?
>
Yes, I include all nodes that are in replication, including the one to 
be dropped.

- Brian F
>
>
>> The thing is that processing the DROP NODE and replicating the SYNC are
>> different worker threads, since the events originate on different nodes.
>> Cleaning out the sl_event is part of dropNode_int(). But the
>> remoteWorker for 3 may just have inserted that SYNC concurrently and
>> therefore it was left behind.
>>
>> My guess is that the right solution to this is to clean out everything
>> again when a STORE NODE comes along. We had been thinking of making the
>> node ID non-reusable to prevent this sort of race conditions.
>>
>>
>> Jan
>>
>