Hannu Krosing wrote:

>On T, 2005-08-09 at 14:35 -0400, Christopher Browne wrote:
>  
>
>>Hannu Krosing wrote:
>>    
>>
>
>  
>
>>>Any good advice for moving forward or am I stuck with this one ?
>>> 
>>>
>>>      
>>>
>>Oh, yeah, that was on my "to do" list, and got lost :-(
>>
>>I ran into the same problem when too much time passed between (I think)
>>submitting SUBSCRIBE SET and then starting the slon for the subscriber.
>>
>>The big part of the query is the "and log_actionseq not in (... ENORMOUS
>>LIST ...)" part, right?
>>    
>>
>
>I guess so.
>
>  
>
>>Time to look into finding a way to compress that...  (What I'm thinking
>>of doing is to have that clause combine adjacent values together...)
>>    
>>
>
>Either this, or just create a temp table and then 'COPY FROM STDIN +
>create index' and use join instead of IN () if there are more members in
>list than some predefined number.
>
>I attached python pseudocode for doing the conversions from NOT IN to
>AND-list of != and NOT BETWEEN
>When I tried the attached code on my failing query and it converted
>alist of 160035 ints to just one BETWEEN. 
>
>  
>
I'm not sure a temp table will work, based on where the actionseq values
come from :-(.

Alas, the problem is that elegant Pythonic answers aren't much good for
helping with code in C, as C hasn't got the string parsing capabilities,
and allocating memory dynamically requires a fair bit of work "by hand."

I've got a FSM whiteboarded that will efficiently parse the list of
integers (when you can use an FSM, your problem has "blazingly fast"
written all over it!); I'll have to do a bit of thinking about what to
do about processing those integers :-).

I now know the FSM part is easy; I'll have to sleep on the rest...

>>I think you're a bit out of luck on this particular subscription.
>>    
>>
>
>Fortunately doing what was suggested in the error message (HINT:
>Increase the configuration parameter "max_expr_depth".) helped me out
>this time. i upped it to 50k from default 10k. 
>
>But I think cant't raise this indefinitely :(expr_d)
>  
>
No, indeed.

When I noticed the issue, I had a subscription fail because of a query
604K in size.  The similarity to "No one would ever need more than 640K"
seemed amusing :-).

>  
>
>>What you might try when you restart it is to, during the subscription
>>period, bump up the amount of time per SYNC (e.g. - increase -s and -t
>>intervals) so that there aren't so many outstanding SYNCs when the
>>subscription gets going.
>>    
>>
>
>what does "log_actionseq not in (..." exactly check for ?
>  
>
It's excluding entries that aren't in the present SYNCs being processed.

I wouldn't be surprised if you'd get better result by either:

a) Forcing the slon to do one sync at a time ---> -g 1
b) Forcing the slon to do a whole lot of syncs at a time --> -g 1000

(It's not obvious which would improve things...)