I think I found the cause of the problem.  The errors started with:

2005-07-19 15:17:23 PDT ERROR  remoteWorkerThread_1: "start
transaction; set enable_seqscan = off; set enable_indexscan
= on; " PGRES_FATAL_ERROR server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
2005-07-19 15:17:23 PDT ERROR  remoteWorkerThread_1: "close LOG; "
PGRES_FATAL_ERROR
2005-07-19 15:17:23 PDT ERROR  remoteWorkerThread_1: "rollback
transaction; set enable_seqscan = default; set enable_indexscan =
default; " PGRES_FATAL_ERROR
2005-07-19 15:17:23 PDT DEBUG2 remoteHelperThread_1_1: 134433.825
seconds until close cursor
2005-07-19 15:17:23 PDT ERROR  remoteWorkerThread_1: helper 1 finished
with error
2005-07-19 15:17:23 PDT ERROR  remoteWorkerThread_1: SYNC aborted

I am guessing that the backend or the connection died.  The SYNC's
kept failing with the message below.  It is bad that slon did not
detect that the connection had failed and did not abort.  This was
only fixed by restarting slon.  What is weird is there was nothing in
the postgres server logs about this.

 - Ian

On 7/20/05, Ian Burrell <ianburrell at gmail.com> wrote:
> We are getting these errors every time a SYNC tries to happen:
> 
> 2005-07-20 12:00:42 PDT ERROR  remoteWorkerThread_1: "start
> transaction; set enable_seqscan = off; set enable_indexscan
> = on; " PGRES_FATAL_ERROR
> 2005-07-20 12:00:42 PDT ERROR  remoteWorkerThread_1: "close LOG; "
> PGRES_FATAL_ERROR
> 2005-07-20 12:00:42 PDT ERROR  remoteWorkerThread_1: "rollback
> transaction; set enable_seqscan = default; set enable_indexscan =
> default; " PGRES_FATAL_ERROR
> 2005-07-20 12:00:42 PDT DEBUG2 remoteHelperThread_1_1: 209032.740
> seconds until close cursor
> 2005-07-20 12:00:42 PDT ERROR  remoteWorkerThread_1: helper 1 finished
> with error
> 2005-07-20 12:00:42 PDT ERROR  remoteWorkerThread_1: SYNC aborted
> 
> Unfortunately, it is not printing out the error from PostgreSQL which
> makes it hard to figure out why it is failing.   Also, I added line
> breaks after all the PGRES_FATAL_ERROR because they were all wrapped
> together.
> 
> Does anyone have an idea what is calling the failure?
> 
>  - Ian
>