On Aug 26, 2004, at 3:08 PM, Jan Wieck wrote:

>> Well, the database that is being replicated requires less than 30 GB 
>> of disk, so is it just the overhead of a replicated, unvacuumed 
>> database that would cause it to inflate to more than double its 
>> original size during the initial copy?
>
> Er ... the copy should not require more than the vacuumed original. I 
> don't understand this. Maybe some WAL misconfiguration? Slony does the 
> data load a little unfortunate. The DB has to build all the indexes 
> during the load, not via bulk index create after. But that can't 
> account for such a bloat.

I was pretty sure WAL was configured similarly for both boxes, but I'll 
guarantee this when our new configuration is in place.

So is the fact that it was all one giant transaction the reason that 
there appeared to be no data despite that fact that the data directory 
was filling up? I.e., the transaction aborted before anything was 
committed because the disk filled up prematurely?

>> With a database of this size, is there any ability to know beforehand 
>> how much space will actually be needed for the initial data copy? 
>> Would it be more advisable to break it up into several smaller 
>> replicated sets?

 From a strategic point of view with a huge data installation, should 
Slony be able to handle that given the resources, or does common sense 
dictate a more incremental approach? I.e., create a number of smaller, 
logical sets, perform the initial copy, vacuum, and move on.

Our data model is account-based, and I was thinking of creating a set 
per account based on some modification to my meta-slonik script rather 
than attempting another monolithic replication.

But if it's useful for improving Slony, and the intent is to support 
replication of any amount of data, I'd like to contribute through 
experimentation.

>> Just a note: This is a 7.4.x installation.
>
> Which you should have on 7.4.5 as soon as you can take advantage of 
> Slony's switchover.

Well, we're already going to be on 7.4.5 before we get Slony into 
production. We've got a maintenance window opportunity, and we've still 
got some other issues involving primary keys/unique constraints on a 
big portion of the database before we can prepare for production Slony.

Anyway, I'll provide more details (as necessary) as we get into further 
testing.

-tfo