On Sun, 1 Jan 2006, Brian Hirt wrote:

> Hello:
>
> As were getting closer to moving to 8.1 and UTF8 i decided to do some
> testing.  I'm running slony 1.1.5-rc2.  The provider database is
> encoded in LATIN1/postgresql 8.0.  The slave is UTF8/postgresql
> 8.1.   Out of the box, slony fails to transfer the data.  When I
> first start slon, the initial copy fails with this error:
>
> postmaster log:
> ERROR:  invalid UTF-8 byte sequence detected near byte 0xae
> CONTEXT:  COPY developer_converted_to_entity, line 71, column
> developer_name: "Spatializer? "
>
> slony log:
> 2006-01-01 13:51:25 MST ERROR  remoteWorkerThread_1: copy from stdin
> on local node - PGRES_FATAL_ERROR ERROR:  invalid UTF-8 byte sequence
> detected near byte 0xae
>
> What's happening is that the data is coming from the provider in
> LATIN1 and that raw data is given to the subscriber which is
> expecting UTF8 -- the LATIN1 binary data is obviously not UTF8 and
> the subscriber node query fails during conversion.
>
> I've come up with a simple solution that is working well, but
> probably not desired for typical users.
>
> What I've done is change slon_connectdb() in dbutils.c to always set
> the client encoding to UTF8.  This pushes any encoding issues to the

This does not work if the master database is SQL_ASCII (I don't think it
works if it is MULE_INTERNAL either). The reason is, all character
sequences in this encoding are valid but not all sequences can be
translated to utf8. It is entirely possible that people are using text
fields as byte arrays in SQL_ASCII databases. See my email:

http://gborg.postgresql.org/pipermail/slony1-general/2005-December/003430.html

If we set the client encoding to utf8 on such databases, we will corrupt
user data.

Gavin