Bug #175 - implement a monitoring thread, which stores information
about the activities of the various slon threads in the new table
sl_components

Squashed commit of the following:

commit da552dd21c87301d7d6d6762a3894c841816d48d
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Mar 18 15:01:34 2011 -0400

    Close DB connection here, pointed out by Steven

commit 0003b57ee0fb6ba855c2aa9bcfbc174671996087
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Mar 18 12:20:37 2011 -0400

    Add docs about log entries that can occur relating to monitoring thread

commit d9b69c024d34a9b41a87876270d84d1c5b2b32a6
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Mar 18 12:13:17 2011 -0400

    Add log warning if the stack gets pretty big.

commit 24da0a137120f4032858230836b39ed14da755fb
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Mar 18 12:05:55 2011 -0400

    Memory leak observed by Steve

commit d51049b54f014ee4f00f974fc9629a7d64ca9727
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Mar 18 11:45:03 2011 -0400

    upgrade schema needs to create sl_components if it is missing

commit cd3519bba5a03ce2f577747324f4a9ebc697d317
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Mar 17 13:41:38 2011 -0400

    Add sl_components to the set of tables we should consider vacuuming

commit d3f66cccb54168ca8d6a80261337a66228720cd8
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Wed Mar 16 16:40:35 2011 -0400

    Add note about interpreting sl_components for monitoring thread

commit a61f4642ed29bc727d2361a4000500b81573ea11
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Wed Mar 16 16:40:08 2011 -0400

    When exiting the monitoring thread, shut off data collection.

    And if the monitoring thread is shut off, don't collect any data in the queue.

commit d1a5be153e51cd10ac12a3dcb9cbb63856a18306
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Wed Mar 16 16:31:47 2011 -0400

    Revising failures so that things that aren't honestly fatal do not
    restart the slon.

    Some error conditions now lead to the monitoring thread terminating without running.

    Actually, there's an "eek, there's a problem" here; if the monitoring
    thread isn't running, then nothing is there to empty entries from the queue.

    So if we're closing it off, then we should turn off entry collection.
    Turn the lights off before going home...

commit d00574d2d1793a372467bc103a7772030fd939ab
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Mar 15 13:05:25 2011 -0400

    Various changes resulting from Steve Singer's review:

    1.  INITIAL_STACK_SIZE from #define to static int

    2.  Get rid of #defines of begin/commit, in favor of slon_dstrings

    3.  Add an initial query, at monitor thread start time, which
        purges old data out of sl_components.

        We removed the other way of cleaning it out, but surely still
        need a way.

    4.  Some memory allocation fixes

    5.  stack_dump() now locks the stack_lock mutex, entry_dump() comments
        warn that users of it need to take that lock.

    6.  Annotate parms of monitor_state() with "const" as needful

commit 03f52daf1fb7ae2d819ecd0c411342a594f6884d
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Mar 15 12:32:57 2011 -0400

    Remove comment per suggestion in review

commit 0543c96506eb55012c4366f78009f886b9464d59
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Mar 15 12:32:29 2011 -0400

    Add conf option, monitor_threads, which controls whether or not the monitoring thread is run

commit 7a3c5f54dcf72bcf67a49f6ed5b80ed673fc0ed5
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Mar 14 18:31:59 2011 -0400

    Remove deletion query on sl_components, per recommendation from Steve in the patch review

commit 72ed0337641eefda095cfcb706562d71d9c03413
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Mar 14 16:44:02 2011 -0400

    Revisions to docs per Steve's comments

commit c6082bc77f460e985c7a2891c45269c9648347c9
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Mar 14 16:43:22 2011 -0400

    Remove default - if we're using slon timestamps, we shouldn't try to default to the DB server's timestamp

commit e6b947c39272030208341e9fd240f7760be89410
Merge: a2e7b12 6572ed1
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 25 18:32:29 2011 -0500

    Merge branch 'master' of git://git.postgresql.org/git/slony1-engine into bug175

commit a2e7b12a90209d03ae2064d505db7909fab29b88
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 25 13:07:48 2011 -0500

    Add check on sl_components to cluster state test scripts

commit a546892938506294ee8baae24c4a1bfafc33fb46
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 25 12:43:10 2011 -0500

    Add more documentation regarding sl_components interpretation, including more on need for NTP synchronization

commit 54eaf2dd5bbdd3f18966c2d7d494cb8ddcea6dc2
Merge: f0ff740 b750db3
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 25 11:28:57 2011 -0500

    Merge branch 'master' of git://git.postgresql.org/git/slony1-engine into bug175

commit f0ff740171f3794e2113545756e44d2e8a4554cc
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 24 17:50:13 2011 -0500

    Add emacs directory locals

commit f2a4f355e08365c6b6707ca6d6e2f756637c4c63
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 24 14:42:54 2011 -0500

    Run pgindent on monitoring code

commit 21b78047c79c45d04416c8fe4a08cdb92fff79a6
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 24 12:11:47 2011 -0500

    Set initial stack size to 6.  This suffices for simple clusters, as typical contents look like:

    slonyregress1 at localhost->  select * from _slony_regress1.sl_components  order by co_actor;
           co_actor       | co_pid | co_node | co_connection_pid |   co_activity    |      co_starttime      | co_event | co_eventtype
    ----------------------+--------+---------+-------------------+------------------+------------------------+----------+--------------
     local_cleanup        |  24586 |       0 |             24907 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
     local_listen         |  24896 |       1 |             24900 | thread main loop | 2011-02-24 17:03:03+00 |          | n/a
     local_monitor        |  24586 |       0 |             24909 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
     local_sync           |  24517 |       0 |             24906 | thread main loop | 2011-02-24 17:03:05+00 |          | n/a
     remote listener      |  24586 |       2 |             24910 | thread main loop | 2011-02-24 17:03:03+00 |          | n/a
     remoteWorkerThread_2 |  24586 |       2 |             24908 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
    (6 rows)

commit a77fc6cbf80b1bacdad168a25cd51c31ef27daf2
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 24 12:04:14 2011 -0500

    Fixed memory allocation problems with monitoring thread stack

commit 1e5f6127d1a27ee1d61f36ee7ebad472fe979211
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 22 16:29:47 2011 -0500

    Remove pid parm from monitor_state() -  collect that consistently inside the function

commit 36e578301546679a8d8975525a1a84b6736012fe
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 22 16:28:15 2011 -0500

    Improving comments on sl_components

commit bac60dce1063678de66e239f2bdf6765fcdb0a99
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 22 16:27:53 2011 -0500

    Add documentation on connections, fixing some SGML tagging problems

commit 1c7b1d456ff2518e77d164ba5b6a2358bc734e84
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 18:04:10 2011 -0500

    Add section on monitoring that describes the new sl_components

commit cdbcaacb1b1c5732a739e51d0b3f0a58c5bc86c2
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 15:53:59 2011 -0500

    More activity monitoring in remote_worker threads

commit 410bff78bc8da1c0cb6b222965ac50ea9341c1ed
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 15:46:07 2011 -0500

    Remove spurious debugging code

commit 55ae2a5c266f1bf5ed5dbc0b9f5576aca0bfc096
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 15:45:13 2011 -0500

    Add in monitor_state() calls inside operation loops to indicate when these threads are doing work

commit 65d34b6de01df6117e37aadd8abe341b4469a37d
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 15:44:11 2011 -0500

    Fixing up memory allocation errors

commit ad52f7e961e72962c10c2b1ce3099003f81fa305
Merge: 76e3502 04c1986
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 18 12:32:34 2011 -0500

    Merge branch 'master' of git://git.postgresql.org/git/slony1-engine into bug175

commit 76e350232d6422b029d204344e5f8e1b7d32eea1
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 17 15:34:35 2011 -0500

    Redo monitor thread using a stack, with the optimization that if the latest entry is for the same component, we overwrite, rather than recording an extra element.

commit f14af6c3bb9cd8dcb1603e47e2744b239d06885a
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 17 15:33:35 2011 -0500

    actor is now PK for sl_components, not the previous composite

commit c642ba97a2e24db170b0927a873f497d9035d372
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 17 15:32:38 2011 -0500

    primary key for components is now on actor...
    Also, don't bother recording an update to sl_component that is newer than the present entry

commit 87089c80d660f6188eab5097c5413a4b7cf925ff
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 17 15:30:39 2011 -0500

    Shift to using malloc'ed stack array means we don't need this queue struct

commit 55746482ed77e5833e31a2d5730f8e1fae6ae51c
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Wed Feb 16 15:17:04 2011 -0500

    Allocate space for NUL for strings

commit b7253ae109b0075b66237f082af9cb5217e9c435
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 16:48:14 2011 -0500

    Fixing casts on monitoring code

commit 6b4a98137f951e31d438b816b9124c6b8e18f420
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 16:47:41 2011 -0500

    appendquery function should return void.  Change types of PIDs to pid_t

commit 07dcc554baa1a9af2db8664daebe523f59464964
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 15:58:49 2011 -0500

    Cleanup of function prototypes

commit 434716fc74a911dbeceba1521cfe5623dbec7825
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 15:52:31 2011 -0500

    Fixes resulting from SPLINT analysis

commit 37dad19295aa5606592c5372a6b5bcbb106e9fe9
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 15:51:40 2011 -0500

    set default for connpid to ensure there's guaranteed always a value

commit c9667dd5906dd269f4b7c5995350d6e8643eb227
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 13:23:49 2011 -0500

    Change some functions returning INT to return VOID to reduce warnings & such

commit b58a72e7272d1ca96792141482058a048aa3afa7
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 12:33:36 2011 -0500

    do local malloc() for copies of monitoring labels

commit f92f86ff32a2c39fd5e6282961d5aa999cad728f
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 15 12:32:53 2011 -0500

    Pass NULL rather than empty string to monitor_state()

commit 54ab45829a6242716de30f9af177513244fdbf5c
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:44:17 2011 -0500

    Numerous revisions to monitoring

commit 4c64fbe276bf579ec3949e963de8dcfb837d1fda
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:40:01 2011 -0500

    Capture connection PID for local_listen thread

commit 7136e148d027891d19bd696c1c598f57744ef783
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:39:37 2011 -0500

    Make connection PID mandatory

commit 5b174f9d33aa2a21163fd49369c06e249e4617b8
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:28:25 2011 -0500

    Capture connection PID for monitoring

commit 1b00ff5daa5a363531f2f507fb1e369f7d33ddc6
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:07:19 2011 -0500

    rename query strings, refactor

commit e2f8d344136b3531bd5fe504733b565c6e9ce5f2
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Feb 14 12:04:46 2011 -0500

    Instrument the local listen thread with monitor_state()

commit 6b45354d0cd1b0ff13c1017186b0950c69e982e0
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 11 18:18:59 2011 -0500

    Lots more debugging done

commit b94243a198af131b2dc98bc02d5410fab528ea38
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 11 18:18:36 2011 -0500

    Change default timing for monitoring thread to dump results every 500ms, add docs for this option

commit ed670f0e6b5f06aa80d1fe8e407765cc67948b78
Merge: 17bdfdf 692fa16
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Feb 11 15:49:11 2011 -0500

    Merge branch 'master' of git://git.postgresql.org/git/slony1-engine into bug175

commit 17bdfdf63295138d5439384f48c6ad80c72af03c
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 10 17:57:12 2011 -0500

    Lots of changes to monitor thread.  It doesn't quite work, but it's now getting close

commit b8101ab3b85aec1f25f287b059bc583c9c7739ce
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 10 17:44:17 2011 -0500

    Change to use UPDATE so we'll frequently get to do HOT updates

commit a32838fde66ef9481681f80ac4ea82103fd43914
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 10 17:36:19 2011 -0500

    Query listed co_event too many times

commit 7cb0500a59265ca585fce82437069535a44a40e5
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Feb 10 17:35:53 2011 -0500

    Activity isn't mandatory - Listeners aren't normally busy doing things

commit b81d6f811ae109481ae6e6a055cae8a1f8afdbbe
Merge: 7239af7 f7ceb47
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Feb 8 15:36:17 2011 -0500

    Merge branch 'master' into bug175

commit 7239af790b0e103ceefee82a3cb0a7f0803b372e
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Mon Dec 20 18:11:52 2010 -0500

    Debugging monitoring code - next steps...

commit 3d1380144a3b1dc320b37154547f76494235385e
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:55:56 2010 -0500

    Revise monitoring API to reduce parms needed, and start instrumenting remote worker main loop

commit 16f722db1e6bf2be2b055a82f66999d8e5babd35
Merge: e08b3cf 379e4d6
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:43:27 2010 -0500

    Merge branch 'master' into bug175

    Conflicts:
    	src/backend/slony1_funcs.sql

commit e08b3cf11c0d36eec936071df493ab6a93859046
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:39:55 2010 -0500

    Loose implementation of API for queueing updates to states of slony components, and draining the queue to record the queries in DB

commit 0eaac89aec7f86b601d8e1f9711123b784f62008
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:38:52 2010 -0500

    Add API for recording monitoring of component states

commit 99cbd9cc2ab8ac07da0b5cf2f822b1cd69d47246
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:38:11 2010 -0500

    Add monitor_interval parameter to control how often the monitoring queue gets flushed

commit 8c2997b2a6ec5ffeef40a575570ab1574c9ee7d1
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:37:39 2010 -0500

    Reformatting & such of monitoring table

commit 5a8cc861c59d591ebbf6de00b54a265e3cdac05b
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Fri Dec 17 17:36:55 2010 -0500

    Function to record component state for monitoring

commit 125ccb17e9e80db37d3f78e2ea62b56395d68ab7
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Dec 16 16:17:47 2010 -0500

    Upon startup of connections, capture the PID into SlonConn structure

commit 31e3547ad4d128f4f1c38165673fb4d3319c7cce
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Thu Dec 16 16:17:18 2010 -0500

    Put proper comments on sl_components

commit aeed5716509029ce84a5b6fc19bc5e76c00d3d4f
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Dec 14 13:16:07 2010 -0500

    Add preliminary code for monitoring thread

commit b61746b0fab22c8c67df3318dc42813407912287
Author: Christopher Browne <cbbrowne at ca.afilias.info>
Date:   Tue Dec 14 13:15:37 2010 -0500

    Add a preliminary schema for monitoring table

Branch
------
master

Details
-------
http://git.postgresql.org/gitweb?p=slony1-engine.git;a=commitdiff;h=59fa974c06a1cae9f355809ea81eaa6d443f0a16

Modified Files
--------------
RELEASE                           |    2 +
doc/adminguide/loganalysis.sgml   |   28 ++-
doc/adminguide/monitoring.sgml    |  110 +++++++++
doc/adminguide/prerequisites.sgml |   20 ++
doc/adminguide/slonconf.sgml      |   25 ++
doc/adminguide/slony.sgml         |    3 +-
share/slon.conf-sample            |    3 +
src/backend/slony1_base.sql       |   28 +++
src/backend/slony1_funcs.sql      |   44 ++++
src/slon/Makefile                 |    2 +
src/slon/cleanup_thread.c         |    4 +
src/slon/confoptions.c            |   26 ++-
src/slon/dbutils.c                |   21 ++-
src/slon/local_listen.c           |    6 +
src/slon/monitor_thread.c         |  455 +++++++++++++++++++++++++++++++++++++
src/slon/remote_listen.c          |    5 +
src/slon/remote_worker.c          |    7 +-
src/slon/slon.c                   |   17 ++
src/slon/slon.h                   |   37 +++-
src/slon/sync_thread.c            |    3 +
tools/test_slony_state-dbi.pl     |   27 +++
tools/test_slony_state.pl         |   26 ++
22 files changed, 881 insertions(+), 18 deletions(-)