Andrew Sullivan <ajs at crankycanuck.ca> writes:

> On Tue, May 30, 2006 at 03:52:14PM -0700, elein wrote:
>> > 
>> > 4) Stampeding Slony. If for any reason Slony is unable to complete the
>> > replication work available within the timeframe allotted (5 minutes I
>> > think), Slony will abandon the first connection and establish a new one
>> > to retry.
>> 
>> Exactly what "replication work" do you mean?  One table? All tables
>> being copied?  In my situation I have 6500*5 + 100 tables to copy.
>> No way is that going to be completed in 5 minutes no matter that
>> the tables are small.  (And no I did not design the schema :)
>
> That isn't the initial COPY, for sure, because none of ours will
> finish the COPY in 5 mins, either.  I suspect Rod means that there
> is a time limit on how long a snapshot-application process runs.  It
> can't be 5 mins, though, or we'd run into this after the initial
> COPY.  Chris, can you shed more light on this one?

The query of Mark's that breaks actually speaks to *an* issue fairly
eloquently.

In the initial SYNC, after the big sets of COPY commands finish, it
has to exclude all actionseq values that are from after the time (on
the master) of that SYNC.

Thus, if you have done 500,000 updates, on the master, there have been
500,000 actionseq values generated in the sl_log_* tables, and 490,000
of them have to be excluded.

This turns into a query consisting of...

 [stuff] followed by
   ... and log_actionseq not in (list of 490K values)

Assuming 6 digit values, that will add to...
   (* (+ 6 1) ;; Add in a comma
      490000)
which means that the log_actionseq component of the query will be
3.43MB in size.

Back when [if memory serves] Hannu Krosing pointed out the issue, I
ran a test which demonstrated the problem.  It did so by generating a
635K query that busted the query parser on my desktop box.

The nearness of this to 640K led to some joking about how what Bill
Gates meant was that "No one should ever need a query larger than
640K."

The patch I mentioned yesterday, which amounts to "run length
encoding," addresses this; the values are generated fairly much
sequentially, which means that if there are a lot of them, they'll
"compress" nicely to near nothingness.

>> Are you saying slony won't handle databases of >100GB?  Or tables? 
>> If the database is larger than that, exactly what patches should be
>> added for exactly what result?  For the most part I am doing
>> production work and never apply patches not in the main release for
>> obvious reasons.  If there are crucial ones, though, I need more
>> details.
>
> Rod sent some observations about group size (and a patch) to the
> list oh, about a month ago?

We put that into 1.1.5, are allowing relatively large group sizes, and
are, in 1.2, accelerating grouping growth so that you don't get stuck
plodding along with tiny group sizes for very long, but I think that's
separate from Rod's "thundering herd" problem.
-- 
output = reverse("ofni.sailifa.ac" "@" "enworbbc")
<http://dba2.int.libertyrms.com/>
Christopher Browne
(416) 673-4124 (land)