Steve, I've also noticed something even more bizarre.

Fortunately, I have log_shipping set up on a third subscriber, and grepping
through the files, I find the following:

[root at asp-dbr1 log_staging]# for i in `ls`; do grep -Hn cron_lock $i; done
slony1_log_3_00000000000011179582.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');
slony1_log_3_00000000000011180903.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');
slony1_log_3_00000000000011181730.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');
slony1_log_3_00000000000011182005.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');
slony1_log_3_00000000000011182466.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');
slony1_log_3_00000000000011183270.sql:13:insert into "public"."cron_lock"
("id","lock_until_time") values ('anonymous_marking','2011-05-11
11:40:02.456091');

[root at asp-dbr1 log_staging]# diff slony1_log_3_00000000000011179582.sql
slony1_log_3_00000000000011180903.sql
7c7
< select "_sac_uk".archiveTracking_offline('11179582', '2011-05-11
11:33:39.97637');
---
> select "_sac_uk".archiveTracking_offline('11180903', '2011-05-11
11:46:11.544073');

It seems that something has gone horribly wrong with replication such that
it's beginning to even replay events.

Does this look like an issue you may already be aware of?
--Richard



On Wed, May 11, 2011 at 2:43 PM, Richard Yen <dba at richyen.com> wrote:

>
>
> On Wed, May 11, 2011 at 2:26 PM, Steve Singer <ssinger at ca.afilias.info>wrote:
>
>> On 11-05-11 04:29 PM, Richard Yen wrote:
>>
>>> Thanks Steve.  I've put the dumps for the master and my two slaves at
>>> http://richyen.com/slony/
>>>
>>
>> As Jan suspected there isn't really anything still left in the logs about
>> events from before the failed insert.
>>
>> When should the row have been deleted from cron_lock? immediately before
>> or sometime before?
>>
> The cron runs every hour, so there should have been a DELETE  at 10:40AM,
> just milliseconds before the INSERT.
>
>
>> Did anything else happen around this time? (server restarts, moving
>> masters etc?)
>>
> No, we didn't do anything during this time.  The previous action I
> performed on this cluster was at 10:04AM, which was to INSERT a row into
> another table in the database.  The sync_interval I have is 500ms, so I'm
> fairly confident that this would have been processed long before the
> offending INSERT to cron_lock arrived at 10:40AM.  The oldest entry in
> sl_even right now is '2011-05-11 10:40:02.354653'
>
> If we can't get any more clues, I'll go ahead an process the DELETE.
>  Thanks again for your help in trying to get to the bottom of this.
> --Richard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.slony.info/pipermail/slony1-general/attachments/20110511/65628e24/attachment.htm