On 12-09-27 05:26 PM, Jan Wieck wrote:
> On 9/27/2012 3:58 PM, Brian Fehrle wrote:
>> Follow up:
>>
>> I executed this on the master:
>> mydatabase=# select * from _slony.sl_event where ev_origin not in
>> (select no_id from _slony.sl_node);
>>    ev_origin |  ev_seqno  | ev_timestamp          |    ev_snapshot     |
>> ev_type | ev_data1 | ev_data2 | ev_data3 | ev_data4 | ev_data5 |
>> ev_data6 | ev_data7 | ev_data8
>> -----------+------------+-------------------------------+--------------------+---------+----------+----------+----------+----------+----------+----------+----------+----------
>>            3 | 5000290161 | 2012-09-27 09:48:03.749424-04 |
>> 40580084:40580084: | SYNC    |          |          | |
>> |          |          |          |
>> (1 row)
>>
>> There is a row in sl_event that shouldn't be there, because it's
>> referencing a node that nolonger exists. I need to add this node back to
>> replication, but I don't want to run into the same issue as before. I
>> ran a cleanupEvent('10 minute') and it did nothing (even did it with 0
>> minutes).
>>
>> Will this row eventually go away? will it cause issue if we attempt to
>> add a new node to replication with node = 3? How can I safely clean this up?
>
> Hmmm,
>
> this actually looks like a more severe race condition or even a bug.
>

The slonik_drop_node function has changed a bit in 2.2(master).  I 
apparently added the following comment to it while doing some of the 
FAILOVER work.  It is possible that it addresses this situation as well?


* find the last event (including SYNC events)
* from the node being dropped that is visible on
* any of the remaining nodes.
* we must wait for ALL remaining nodes to get caught up.
*
* we can't ignore SYNC events even though the dropped
* node is not an origin it might have been an old
* origin before a FAILOVER. Some behind node still
* might need to get caught up from its provider.
*/
ev_id = get_last_escaped_event_id((SlonikStmt *) stmt,
stmt->no_id_list[no_id_idx],stmt->no_id_list);

Also, Brian,

Did your slonik script (for the DROP NODE command) include admin 
conninfo lines to all your nodes?



> The thing is that processing the DROP NODE and replicating the SYNC are
> different worker threads, since the events originate on different nodes.
> Cleaning out the sl_event is part of dropNode_int(). But the
> remoteWorker for 3 may just have inserted that SYNC concurrently and
> therefore it was left behind.
>
> My guess is that the right solution to this is to clean out everything
> again when a STORE NODE comes along. We had been thinking of making the
> node ID non-reusable to prevent this sort of race conditions.
>
>
> Jan
>