9.3 to 9.5 upgrade problems

Hi all,

I have a master (web1) and two slaves (web2, webserv), one slave is quite far from the master, the db is 112 Gig, so
pg_basebackupis my last resort. 

I followed the page here:
https://www.postgresql.org/docs/9.5/static/pgupgrade.html

including the rsync stuff.  I practiced it _twice_, once in PG 9.5 beta, and again a week ago, on two VM's I created
locally. Both practice sessions worked perfect. 

I just ran it on the live databases.  The master seems ok, its running PG 9.5 now, I can login to it, and no errors in
thelog. 

Neither slave works.  After I'd gotten done with the pgupgrade steps, both slaves gave me this error:

FATAL:  database system identifier differs between the primary and standby

Sure enough pg_controldata show'd their database system id different (all three web1, web2, webserv were different.  no
matchesat all), so I'm assuming the rsync didnt rsync right, or I missed a step and ran it to early, or something ...
I'mnot quite sure. 

I needed to get the live website back up and running again, so I let the master go, ran analyze, and when it was
finished,used the steps here to try and resync: 

https://wiki.postgresql.org/wiki/Binary_Replication_Tutorial

on Master:
select pg_start_backup('clone',true);
rsync -av --exclude pg_xlog --exclude postgresql.conf /pub/pg95/* web2:/pub/pg95/
select pg_stop_backup();
rsync -av /pub/pg95/pg_xlog web2:/pub/pg95/


That ran pretty quick, and pg_controldata shows matching numbers, but when I start the slave I get:

,,2016-07-03 06:06:57.173 CDT,: LOG:  entering standby mode
,,2016-07-03 06:06:57.205 CDT,: LOG:  redo starts at 369/D6002228
,,2016-07-03 06:06:57.984 CDT,: LOG:  consistent recovery state reached at 369/DCC5DB90
,,2016-07-03 06:06:57.984 CDT,: LOG:  database system is ready to accept read only connections
,,2016-07-03 06:06:57.984 CDT,: LOG:  invalid record length at 369/DD038ED0
,,2016-07-03 06:06:58.344 CDT,: LOG:  started streaming WAL from primary at 369/DD000000 on timeline 1
web,[unknown],2016-07-03 06:07:11.176 CDT,[local]: FATAL:  role "andy" does not exist

I can login as myself on the master, but not on the slave.  when I "psql -U postgres" on the slave I get:

psql: FATAL:  cache lookup failed for database 16401

This is only on web2, its close to web1, so I'm hoping I can get it fixed and then rsync it quickly to the far away
slave.

I'm at a loss here, any hints or suggestions would be appreciated.

Thanks,

-Andy