Thanks for your reply Christopher, however moments before I found the 
problem.

This is a result of the bug # 55 that was fixed in version 1.2.21. 
Here's where I found it. http://bugs.slony.info/bugzilla/show_bug.cgi?id=55

I did the same as the author stated, I restarted my daemons with setting 
the debug level to 0 and after about 2 minutes replication started 
working again.

So all is good. Thanks again. I'll be pushing the box owners to upgrade 
to 1.2.21 even more now.

- Brian F

Christopher Browne wrote:
> Brian Fehrle <brianf at consistentstate.com> writes:
>   
>> Anyone know what could have caused this? The only think I know of that was going on was adding additional replication sets to the slony cluster at the time. I am thinking it may be more of a linux thing with too many open files rather than postgres/slony, as I haven't found anything related to slony/postgres when googling around for these errors.
>>
>> Thanks in advance for any feedback,
>>     
>
> You might check the output of "ulimit -a" on your system, as that should
> list limits.
>
> cbbrowne at cbbrowne [12:23:35] [~/architecture] [master *]
> -> % ulimit -a
> -t: cpu time (seconds)         unlimited
> -f: file size (blocks)         unlimited
> -d: data seg size (kbytes)     unlimited
> -s: stack size (kbytes)        8192
> -c: core file size (blocks)    0
> -m: resident set size (kbytes) unlimited
> -u: processes                  unlimited
> -n: file descriptors           1024
> -l: locked-in-memory size (kb) 64
> -v: address space (kb)         unlimited
> -x: file locks                 unlimited
> -i: pending signals            16382
> -q: bytes in POSIX msg queues  819200
> -e: max nice                   0
> -r: max rt priority            0
>
> On my system, the number of file descriptors is limited to 1024, by
> default.  (These parms are commonly overridable, with some
> OS-specificity.)
>
> 1024 isn't exactly a small number of files, so it's a bit of a surprise
> to get hit by this.  If you're running database backends and slons under
> a single postgres user, it's conceivable that the competition might be
> heading towards the limit, though 1024 is a pretty large limit.
>
> - I haven't had call to search for file descriptor usage, so you'll need
>   to research how to do that ;-).  Presumably it can be monitored.
>
> - Perhaps it could help to set up a "slony" Linux user, and run slons in
>   that context; that gives an extra 1024 descriptors.
>
> - If there's a process that's chewing up file descriptors and not
>   returning that, it would be hugely useful to identify that.  Perhaps
>   there's a bug in something.  It's conceivable for it to be the slons,
>   but other causes can't be ruled out.
>