Philip Warner <pjw at rhyme.com.au> writes:
> Following recent problems with buffer sizes (again), I would be interested to know if:
>
>  (a) the slony developers think my latest suggestion (setting bounds on buffer space used when reading logs) is a good idea, 
>  (b) if you plan to implement this (or something similar), or 
>  (c) if you would be interested in patches that implement this (or something similar).
>
> Or if you think my case if just too unusual to bother with...

It seems a good idea, at least in principle.

Hey, if we can change the behaviour of slon so that it consumes Very
Little Memory, or, rather, to prevent it from Behaving Badly, that
hardly seems a bad thing.

I haven't looked deeply, but the only reason I can think of for there
to need to be a lot of memory occupied at any given time is if the
groupings of 10 queries together (e.g. - when sl_log_1 entries are
being applied, they are done in sets of 10) involves some records that
are Very Large.

Typical here would be with some application like RT, where one of the
tables, "attachements", can contain arbitrarily large records.

It would seem perfectly reasonable to short circuit this:

--> If size of current data being processed > 10MB then don't
    bother moving on from record 7 to record 8; push the 
    data out NOW and clear the data structure...

The only downside that I see is the cost of searching through the
strings to see how big they are.

I'm not certain that this is The Issue, though.  If what's consuming
memory is doing so from inside libpq, then "setting bounds" may be
rather more troublesome...

Actually, I just thought of the other place where there's an issue,
and it is indeed thornier.

That's with the "FETCH 100 FROM LOG" queries.  That's going to draw in
100 rows from sl_log_1 each time, and if some are Pathological Big
Records, _there_ lies the problem.

There isn't a "FETCH 100 FROM LOG STOP AFTER 50MB" command :-(.

I suppose what is needful is to have a way to control both the 10 and
100 figures.

Both the 10 and 100 figures are #defined in <src/slon.h>, thus:

#ifdef	SLON_CHECK_CMDTUPLES
#define SLON_COMMANDS_PER_LINE		1
#define SLON_DATA_FETCH_SIZE		100
#define SLON_WORKLINES_PER_HELPER	(SLON_DATA_FETCH_SIZE * 4)
#else
#define SLON_COMMANDS_PER_LINE		10
#define SLON_DATA_FETCH_SIZE		10
#define SLON_WORKLINES_PER_HELPER	(SLON_DATA_FETCH_SIZE * 50)
#endif

This is NOT easily changeable at runtime, but if you're running into
these problems, then I'd suggest decreasing these values in slon.h.

Ideally, it would be nice for slon to figure out suitable values by
itself.  How to do that when a bad choice would lead to slon running
out of memory and falling over seems, erm, troublesome :-(.

A way to cope with that would be if we stuffed the values into a table
on the node that the slon is servicing, and had a policy of always
starting out by taking the old values from the node and scaling them
back a bit.  If all goes well, the slon gets a bit more aggressive; if
it crashes, that naturally leads to stepping back.

Definitely worthy of more thought before implementation...
-- 
output = ("cbbrowne" "@" "ca.afilias.info")
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 673-4124 (land)