Darcy, Chris, Jan,

> FWIW, I'm looking at their system now and am seeing persistent lock
> contention on pg_listener:
>
>  relation | database  | transaction |  pid  |     mode      | granted
> ----------+-----------+-------------+-------+---------------+---------
>     16414 | 318624738 |             | 18883 | ExclusiveLock | f
>     16414 | 318624738 |             | 19028 | ExclusiveLock | f
>     16414 | 318624738 |             | 19205 | ExclusiveLock | t
>
>
> This lock contention seems constant.       Why would pg_listener be having
> blocking exclusive locks?

To follow this up: the above lock contention is preventing vacuum from working 
on pg_listener:

INFO:  vacuuming "pg_catalog.pg_listener"
INFO:  "pg_listener": found 0 removable, 500018 nonremovable row versions in 
6175 pages
DETAIL:  500006 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages are entirely empty.
CPU 0.01s/0.12u sec elapsed 0.14 sec.
INFO:  analyzing "pg_catalog.pg_listener"
INFO:  "pg_listener": 6175 pages, 12 rows sampled, 12 estimated total rows
VACUUM

This *severe* table attenuation is the cause of pg_listener queries taking > 1 
second and starting to block.    How do we clear it up without bringing the 
system down?

--Josh

-- 
__Aglio Database Solutions_______________
Josh Berkus		       Consultant
josh at agliodbs.com	 www.agliodbs.com
Ph: 415-752-2500	Fax: 415-752-2387
2166 Hayes Suite 200	San Francisco, CA