Обсуждение: 7.2 is slow?

Поиск
Список
Период
Сортировка

7.2 is slow?

От
Tatsuo Ishii
Дата:
With the freshly retrieved current source, now PostgreSQL is running
fine on an AIX 5L box. Thanks Tom.

BTW, I have done some benchmarking using pgbench on this machine and
found that 7.2 is almost two times slower than 7.1. The hardware is a
4way machine. Since I thought that 7.2 improves the performance for
SMP machines, I'm now wondering why 7.2 is so slow.

postgresql.conf paramters changed from default values are:

max_connections = 1024
wal_sync_method = fdatasync
shared_buffers = 4096
deadlock_timeout = 1000000

configure option is: --enable-multibyte=EUC_JP

Of cousre, these setting are identical for both 7.1 and 7.2.

See attached graph...

Re: 7.2 is slow?

От
Bruce Momjian
Дата:
> With the freshly retrieved current source, now PostgreSQL is running
> fine on an AIX 5L box. Thanks Tom.
> 
> BTW, I have done some benchmarking using pgbench on this machine and
> found that 7.2 is almost two times slower than 7.1. The hardware is a
> 4way machine. Since I thought that 7.2 improves the performance for ^^^^
> SMP machines, I'm now wondering why 7.2 is so slow.

Ewe.  I will remind people that this multi-cpu setup is exactly the type
of machine we wanted to speed up with the new light-weight locking code
that reduced spinlock looping.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Tatsuo Ishii wrote:
> 
> With the freshly retrieved current source, now PostgreSQL is running
> fine on an AIX 5L box. Thanks Tom.
> 
> BTW, I have done some benchmarking using pgbench on this machine and
> found that 7.2 is almost two times slower than 7.1.

Is this an AIX specific problem or do all/all SMP/all 4way computers 
have it ?

Is this a bug that needs to be addresse before release of final ?

Or would we just prominently warn people that the new release is 2x 
slower and advise upgrading only if they have powerful enough computers 
(system load < 0.5 during normal operation)?

------------------------
Hannu


Re: 7.2 is slow?

От
Tatsuo Ishii
Дата:
> > BTW, I have done some benchmarking using pgbench on this machine and
> > found that 7.2 is almost two times slower than 7.1.
> 
> Is this an AIX specific problem or do all/all SMP/all 4way computers 
> have it ?

Not sure. As far as I can tell, nobody except me has tested 7.2 on big
boxes.

> Is this a bug that needs to be addresse before release of final ?

I hope this would be solved before final. At least I would like to
know what's going on.

Anyway, I will do some testings on a smaller machine (that is my
laptop) to see if I see the same performance degration on it.
--
Tatsuo Ishii


Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Tatsuo Ishii wrote:
> 
> > > BTW, I have done some benchmarking using pgbench on this machine and
> > > found that 7.2 is almost two times slower than 7.1.
> >
> > Is this an AIX specific problem or do all/all SMP/all 4way computers
> > have it ?
> 
> Not sure. As far as I can tell, nobody except me has tested 7.2 on big
> boxes.
> 
> > Is this a bug that needs to be addresse before release of final ?
> 
> I hope this would be solved before final. At least I would like to
> know what's going on.
> 
> Anyway, I will do some testings on a smaller machine (that is my
> laptop) to see if I see the same performance degration on it.

How did you test ?

I could do the same test on Dual Pentium III / 800 w/1024 MB
with IBM 45 G/7200 IDE disk.

So we could compare different platforms as well :)

-------------
Hannu


Re: 7.2 is slow?

От
Mathijs Brands
Дата:
On Mon, Dec 17, 2001 at 12:43:05PM +0200, Hannu Krosing allegedly wrote:
> Tatsuo Ishii wrote:
> > 
> > > > BTW, I have done some benchmarking using pgbench on this machine and
> > > > found that 7.2 is almost two times slower than 7.1.
> > >
> > > Is this an AIX specific problem or do all/all SMP/all 4way computers
> > > have it ?
> > 
> > Not sure. As far as I can tell, nobody except me has tested 7.2 on big
> > boxes.
> > 
> > > Is this a bug that needs to be addresse before release of final ?
> > 
> > I hope this would be solved before final. At least I would like to
> > know what's going on.
> > 
> > Anyway, I will do some testings on a smaller machine (that is my
> > laptop) to see if I see the same performance degration on it.
> 
> How did you test ?
> 
> I could do the same test on Dual Pentium III / 800 w/1024 MB
> with IBM 45 G/7200 IDE disk.
> 
> So we could compare different platforms as well :)

I could do some testing on a Sun 450 / 4x400 MHz / 4 GB, if that's helpful.

Cheers,

Mathijs


Re: 7.2 is slow?

От
bpalmer
Дата:
> > > > Is this an AIX specific problem or do all/all SMP/all 4way computers
> > > > have it ?

I'll have 4 way and 8 way xeon boxes tues evening that I can test this
against (though I won't get to test till wed unless I don't sleep)

- Brandon

----------------------------------------------------------------------------c: 646-456-5455
              h: 201-798-4983b. palmer,  bpalmer@crimelabs.net           pgp:crimelabs.net/bpalmer.pgp5
 



Re: 7.2 is slow?

От
Tatsuo Ishii
Дата:
> > How did you test ?
> > 
> > I could do the same test on Dual Pentium III / 800 w/1024 MB
> > with IBM 45 G/7200 IDE disk.
> > 
> > So we could compare different platforms as well :)
> 
> I could do some testing on a Sun 450 / 4x400 MHz / 4 GB, if that's helpful.
> 
> Cheers,
> 
> Mathijs
> 

> I'll have 4 way and 8 way xeon boxes tues evening that I can test this
> against (though I won't get to test till wed unless I don't sleep)
>
> - Brandon

Thanks to everyone. Here are the methods I used for testings including
generating graphs (actually very simple).

(1) Tweak postgresql.conf to allow large concurrent users. I tested up   to 1024 on AIX, but for the comparison I think
testingup to 128   users is enough. Here are example settings:
 
   max_connections = 128   shared_buffers = 4096   deadlock_timeout = 100000
   You might want to tweak wal_sync_method to get the best   performance. However this should not affect the comparison
between  7.1 and 7.2.
 

(2) Run:
   sh bench.sh
   It will invoke pgbench for various concurrent users. So you need   to install pgbench beforehand (it's in
contrib/pgbench.Just type   make install there to install pgbench).
 
   This will take while.

(3) (2) will generate a file named "bench.data". The file have rows       where the first column is the number of
concurrentusers and       second one is the tps. Rename it to bench-7.2.data.
 

(4) Do (1) and (2) for PostgreSQL 7.1 and rename bench.data to   bench-7.1.data.

(5) Run plot.sh to see the result graph. Note that plot.sh requires   gnuplot.
---
Tatsuo Ishii

Re: 7.2 is slow?

От
Hannu Krosing
Дата:
It seems that on dual PIII we are indeed faster than 7.1.3 for 
small number of clients but slower for large number (~ 40)

My initial results on dual PIII/800 are as follows
                                     7.1.3      7.2b4  7.2b4-FULL
==================================================================
./pgbench -i -p 5433 
./pgbench -p 5433 -c  1 -t 100       240/251    217/223    177/181
./pgbench -p 5433 -c  5 -t 100        93/ 94    211/217    207/212
./pgbench -p 5433 -c 10 -t 100        57/ 58    145/148    160/163
------------------------------------------------------------------
./pgbench -i -s 10 -p 5433 
./pgbench -p 5433 -c   1 -t 100      171/177    162/166    169/173
./pgbench -p 5433 -c   5 -t 100      140/143    191/196    202/207
./pgbench -p 5433 -c  10 -t 100      132/135    165/168    159/163
./pgbench -p 5433 -c  25 -t 100       65/ 66     60/ 60     75/ 76
./pgbench -p 5433 -c  50 -t 100       60/ 61     43/ 43     55/ 59
./pgbench -p 5433 -c 100 -t 100       48/ 48     23/ 23     34/ 34
------------------------------------------------------------------

One of thereasons seems to be that vacuum has chaged

after oding 

psql -p 5433 -c 'vacuum full'

the result of

./pgbench -p 5433 -c 100 -t 100

was 34/34 - still ~25% slower than 7.1.3 but much better 
than with non-full vacuum (which I guess is used by pgbench

The third column 7.2b4-FULL is done by running  "psql -p 5433 -c 'vacuum full'"
between each pgbench run - now the lines cross somwhere 
between 25 and 50 concurrent users

One of the reasons pg is slower on last limes of my test is that 
postgres is slower when vacuum is not done often enough - 
on fresh db
"./pgbench -p 5433 -c 100 -t 10"  gives   67/75 as result
indicating that one reason is just our non-overwriting storage manager.


I also tried to outsmart pg by running the new vacuum 
concurrently, but was disappointed.

vacuuming in 'normal' psql gave me 20/20 tps and running 
with nice psql gave 21/21 tps
running ./pgbench -p 5433 -c 100 -t 100 as first benchmark gave the 
same result as running it after vacuum full


-----------------------------------------------------------------------
PS. I hope to get single-processor results from the same computer in 
about 6 hours as well (after my co-worker arrives home and can reboot 
his computer to single-user)

Inxc - after you have rebooted to single-processor mode, pleas start 
the postgres daemon by

su - hannu
cd db/7.2b4/
bin/pg_ctl -D data -l logfile

and ther run above pgbench commands from 
cd /home/hannu/src/postgresql-7.1.3/contrib/pgbench/
-----------------------------------------------------------------------


Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Tatsuo Ishii wrote:
> 
> 
> Thanks to everyone. Here are the methods I used for testings including
> generating graphs (actually very simple).
> 
> (1) Tweak postgresql.conf to allow large concurrent users. I tested up
>     to 1024 on AIX, but for the comparison I think testing up to 128
>     users is enough. Here are example settings:
> 
>     max_connections = 128
>     shared_buffers = 4096
>     deadlock_timeout = 100000
> 
>     You might want to tweak wal_sync_method to get the best
>     performance. However this should not affect the comparison between
>     7.1 and 7.2.
>
> (2) Run:
> 
>     sh bench.sh

I have no more time today, but I'll redo the tests with your script
tomorrow
(after I have found where to stick database name and port :)

----------------
Hannu


Re: 7.2 is slow?

От
Tom Lane
Дата:
Hannu Krosing <hannu@tm.ee> writes:
> ./pgbench -i -s 10 -p 5433 
> ./pgbench -p 5433 -c   1 -t 100      171/177    162/166    169/173
> ./pgbench -p 5433 -c   5 -t 100      140/143    191/196    202/207
> ./pgbench -p 5433 -c  10 -t 100      132/135    165/168    159/163
> ./pgbench -p 5433 -c  25 -t 100       65/ 66     60/ 60     75/ 76
> ./pgbench -p 5433 -c  50 -t 100       60/ 61     43/ 43     55/ 59
> ./pgbench -p 5433 -c 100 -t 100       48/ 48     23/ 23     34/ 34

You realize, of course, that when the number of clients exceeds the
scale factor you're not really measuring anything except update
contention on the "branch" rows?  Every transaction tries to update
the balance for its branch, so if you have more clients than branches
then there will be lots of transactions blocked waiting for someone
else to commit.  With a 10:1 ratio, there will be several transactions
blocked waiting for *each* active transaction; and when that guy
commits, all the others will waken simultaneously and contend for the
chance to update the branch row.  One will win, the others will go
back to sleep, having done nothing except wasting CPU time.  Thus a
severe falloff in measured TPS is inevitable when -c >> -s.  I don't
think this scenario has all that much to do with real-world loads,
however.

I think you are right that the difference between 7.1 and 7.2 may have
more to do with the change in VACUUM strategy than anything else.  Could
you retry the test after changing all the "vacuum" commands in pgbench.c
to "vacuum full"?
        regards, tom lane


Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Tom Lane wrote:
> 
> Hannu Krosing <hannu@tm.ee> writes:
> > ./pgbench -i -s 10 -p 5433
> > ./pgbench -p 5433 -c   1 -t 100      171/177    162/166    169/173
> > ./pgbench -p 5433 -c   5 -t 100      140/143    191/196    202/207
> > ./pgbench -p 5433 -c  10 -t 100      132/135    165/168    159/163
> > ./pgbench -p 5433 -c  25 -t 100       65/ 66     60/ 60     75/ 76
> > ./pgbench -p 5433 -c  50 -t 100       60/ 61     43/ 43     55/ 59
> > ./pgbench -p 5433 -c 100 -t 100       48/ 48     23/ 23     34/ 34
> 
> You realize, of course, that when the number of clients exceeds the
> scale factor you're not really measuring anything except update
> contention on the "branch" rows? 

Oops! I thought that the deciding table would be tellers and this -s 10 
would be ok for up to 100 users

I will retry this with Tatsuos using -s 128(if it still fits on disk 
- taking about 160MB/1Mtuple needs 1.6GB for test with -s 100 and 
I currently have only 1.3G free)

I re-run some of them with -s 50  (on 7.2b4)

each one after running "psql -p 5433 -c 'vacuum full;checkpoint;'"
                                     tps 
./pgbench -p 5433 -i -s 50
./pgbench -p 5433 -c   1 -t 1000     93/ 93
./pgbench -p 5433 -c   3 -t  333    106/107
./pgbench -p 5433 -c   5 -t  200    106/107
./pgbench -p 5433 -c   8 -t  125    112/113
./pgbench -p 5433 -c  10 -t  100     94/ 95
./pgbench -p 5433 -c  25 -t   40     98/ 91
./pgbench -p 5433 -c  50 -t   20     70/ 74

> Every transaction tries to update
> the balance for its branch, so if you have more clients than branches
> then there will be lots of transactions blocked waiting for someone
> else to commit.  With a 10:1 ratio, there will be several transactions
> blocked waiting for *each* active transaction; and when that guy
> commits, all the others will waken simultaneously and contend for the
> chance to update the branch row.  One will win, the others will go
> back to sleep, having done nothing except wasting CPU time.  Thus a
> severe falloff in measured TPS is inevitable when -c >> -s.  I don't
> think this scenario has all that much to do with real-world loads,
> however.

It probably models a real-world ill-tuned database :)

And it seems that we fall off more rapidly on 7.2 than we did on 7.1 , 
even so much so that we will be slower in the end.

> I think you are right that the difference between 7.1 and 7.2 may have
> more to do with the change in VACUUM strategy than anything else.  Could
> you retry the test after changing all the "vacuum" commands in pgbench.c
> to "vacuum full"?

The third column should be the equivalent of doing so (I did run 
'vacuum full' between each pgbench and AFACT pgbencg runs vacuun only 
before each run)

--------------
Hannu


Re: 7.2 is slow?

От
"Zeugswetter Andreas SB SD"
Дата:
> I think you are right that the difference between 7.1 and 7.2 may have
> more to do with the change in VACUUM strategy than anything else.  Could
> you retry the test after changing all the "vacuum" commands in pgbench.c
> to "vacuum full"?

Might there also be a difference in chosen query plans ?
Wasn't 7.1 more willing to choose an index over seq scan,
even though the scan would be faster in the single user case ?
Or was that change after 7.0 ?

The seq scan would be slower that the index in the case of 
many concurrent accesses.

Andreas


Re: 7.2 is slow?

От
Jussi Mikkola
Дата:
I haven't tested with the new 7.2 betas, but here are some results from <br />7.1. <p>We have a developement computer,
IBMx series 250, with 4 processors <br />(PIII Xeon 750Mhz), 1 Gb memory and  2SCSI disks (u160). <p>The software is
writingnew rows to a table, and after this it reads the <br />id from that row. There are currently about 50
connectionsdoing the <br />same thing. <p>When I run this test with the Redhat 7.1, with SMP kernel, I noticed, that
the<br />processors are more than 90% idle. Disks utilisation is not the <br />bottleneck either, since there is very
lowdisk usage. Some data is <br />written to disks every 4-5 seconds. Fsync is turned of. In transactions, <br />this
meansabout 200 inserted rows per second. The software that is used <br />to give the feed, is capable of several
thousandrows per second. <p>Okey, so I tried this also with the same computer, but using the not SMP <br />supported
kernel.So only with one processor. The result was about 600 <br />rows per second. The configuration file was
unchanged.Now, the <br />processor is about 100% utilized. <p>I didn't find any parameters that should help in this,
butif you have a <br />version of 7.2 that you would like to get information about, let me <br />know, so I'll test.
<p>Jussi<p>Zeugswetter Andreas SB SD wrote: <blockquote type="CITE">> I think you are right that the difference
between7.1 and 7.2 may have <br />> more to do with the change in VACUUM strategy than anything else.  Could <br
/>>you retry the test after changing all the "vacuum" commands in pgbench.c <br />> to "vacuum full"? <p>Might
therealso be a difference in chosen query plans ? <br />Wasn't 7.1 more willing to choose an index over seq scan, <br
/>eventhough the scan would be faster in the single user case ? <br />Or was that change after 7.0 ? <p>The seq scan
wouldbe slower that the index in the case of <br />many concurrent accesses. <p>Andreas
<p>---------------------------(endof broadcast)--------------------------- <br />TIP 4: Don't 'kill -9' the
postmaster</blockquote><pre>-- 
Jussi Mikkola                    Project Manager
Bonware Oy                       gsm +358 40 830 7561
Tekniikantie 12                  tel +358 9 2517 5570
02150 Espoo                      fax +358 9 2517 5571 
Finland                          www.bonware.com</pre>  

Re: 7.2 is slow?

От
Bruce Momjian
Дата:
> I haven't tested with the new 7.2 betas, but here are some results from
> 7.1.
> 
> We have a developement computer, IBM x series 250, with 4 processors
> (PIII Xeon 750Mhz), 1 Gb memory and  2SCSI disks (u160).
> 
> The software is writing new rows to a table, and after this it reads the
> id from that row. There are currently about 50 connections doing the
> same thing.
> 
> When I run this test with the Redhat 7.1, with SMP kernel, I noticed, that
> the
> processors are more than 90% idle. Disks utilisation is not the
> bottleneck either, since there is very low disk usage. Some data is
> written to disks every 4-5 seconds. Fsync is turned of. In transactions,
> this means about 200 inserted rows per second. The software that is used
> to give the feed, is capable of several thousand rows per second.
> 
> Okey, so I tried this also with the same computer, but using the not SMP
> supported kernel. So only with one processor. The result was about 600
> rows per second. The configuration file was unchanged. Now, the
> processor is about 100% utilized.
> 
> I didn't find any parameters that should help in this, but if you have a
> version of 7.2 that you would like to get information about, let me
> know, so I'll test.

Yes!  This sleeping case is the problem we expected to see on SMP
machines in >= 7.1 because of lock contention and a select() that can't
sleep for less than 1/100 second.  Please try the current 7.2 snapshot
and let us know what performance you get.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: 7.2 is slow?

От
Jussi Mikkola
Дата:
Hi !

Yes, now I've tested with 7.2b4. The result is about the  same as with 7.1.

About 200 messages with four processors and about 600 messages with one
processor.

Jussi

>
>
> Yes!  This sleeping case is the problem we expected to see on SMP
> machines in >= 7.1 because of lock contention and a select() that can't
> sleep for less than 1/100 second.  Please try the current 7.2 snapshot
> and let us know what performance you get.
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

--
Jussi Mikkola                    Project Manager
Bonware Oy                       gsm +358 40 830 7561
Tekniikantie 12                  tel +358 9 2517 5570
02150 Espoo                      fax +358 9 2517 5571
Finland                          www.bonware.com





Re: 7.2 is slow?

От
Tom Lane
Дата:
"Zeugswetter Andreas SB SD" <ZeugswetterA@spardat.at> writes:
> Might there also be a difference in chosen query plans ?

If so, it'd affect the results across-the-board, but AFAICT
Tatsuo is seeing comparable results for small numbers of clients.
        regards, tom lane


Re: 7.2 is slow?

От
Ashley Cambrell
Дата:
Hi All,

I have experienced similar problems after moving my main server from UP 
(PII 300 box) to SMP (PII400 box).  I remeber someone said look at the 
iostat figures for the different runs, but I haven't had time to check 
it out.

Out of curosity, what does iostat say when run in SMP vs UP?

Ashley Cambrell

Jussi Mikkola wrote:

>Hi !
>
>Yes, now I've tested with 7.2b4. The result is about the  same as with 7.1.
>
>About 200 messages with four processors and about 600 messages with one
>processor.
>
>Jussi
>
>>
>><snip>
>>




Re: 7.2 is slow?

От
Luis Amigo
Дата:
We're getting similar problem.
We're currently working on TPC-H benchmarking using postgresql 7.2b3.
From one up to 8 paralell conexions (we've got 8 MIPS processors), uptime
increases from 1 to 8,
but increasing above 8 makes performance drop rapidly to uptimes even lower
than 2 for 22 conexions.
As we've could trail, when we have more processes than processors, we're
getting an increasing number of collisions.
when collision happens both processes get idle for a while, then collision may
happen again and so...
Regards
Luis Amigo
Universidad de Cantabria




Re: 7.2 is slow?

От
Tom Lane
Дата:
Jussi Mikkola <jussi.mikkola@bonware.com> writes:
> Yes, now I've tested with 7.2b4. The result is about the  same as with 7.1.

> About 200 messages with four processors and about 600 messages with one
> processor.

That's annoying.  The LWLock changes were intended to solve the
inefficiency with multiple CPUs, but it seems like we still have a
problem somewhere.

Could you recompile the backend with profiling enabled and try to get
a profile from your test case?  To build a profilable backend, it's
sufficient to do
cd .../src/backendgmake cleangmake PROFILE=-pg allgmake install-bin

(assuming you are using gcc).  Then restart the postmaster, and you
should notice "gmon.out" files being dropped into the various database
subdirectories anytime a backend exits.  Next run your test case,
and as soon as it finishes copy the gmon.out file to a safe place.
(You'll only be able to get the profile from the last process to exit,
so try to make sure that this is representative.  Might be worth
repeating the test a few times to make sure that the results don't
vary a whole lot.)  Finally, do
gprof .../bin/postgres gmon.out >resultfile

to produce a legible result.

Oh, one more thing: on Linuxen you are likely to find that all the
reported routine runtimes are zero, rendering the results useless.
Apply the attached patch (for 7.2beta) to fix this.
        regards, tom lane

*** src/backend/postmaster/postmaster.c.orig    Wed Dec 12 14:52:03 2001
--- src/backend/postmaster/postmaster.c    Mon Dec 17 19:38:29 2001
***************
*** 1823,1828 ****
--- 1823,1829 ---- {     Backend    *bn;                /* for backend cleanup */     pid_t        pid;
+     struct itimerval svitimer;      /*      * Compute the cancel key that will be assigned to this backend. The
***************
*** 1858,1869 ****
--- 1859,1874 ----     beos_before_backend_startup(); #endif 
+     getitimer(ITIMER_PROF, &svitimer);
+      pid = fork();      if (pid == 0)                /* child */     {         int            status; 
+         setitimer(ITIMER_PROF, &svitimer, NULL);
+          free(bn); #ifdef __BEOS__         /* Specific beos backend startup actions */


Re: 7.2 is slow?

От
Tom Lane
Дата:
Luis Amigo <lamigo@atc.unican.es> writes:
> We're getting similar problem.
> We're currently working on TPC-H benchmarking using postgresql 7.2b3.
> From one up to 8 paralell conexions (we've got 8 MIPS processors),

MIPS?  Which spinlock implementation is getting used?  (Look in
src/include/storage/s_lock.h and src/backend/storage/lmgr/s_lock.c)

If you're falling back to the default SysV-semaphore based spinlock
implementation, I wouldn't be surprised to see a performance problem...
        regards, tom lane


Re: 7.2 is slow? [compile problem]

От
Ashley Cambrell
Дата:
I haven't actually tried to compile 7.2 from the CVS, but there seems to 
be a problem?  [maybe on my side]

make[3]: Leaving directory 
`/home/ash/ash-server/Work/build/pgsql/src/backend/utils'
gcc -O2 -Wall -Wmissing-prototypes -Wmissing-declarations  
-Wl,-rpath,/tmp//lib -export-dynamic access/SUBSYS.o bootstrap/SUBSYS.o 
catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o 
lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o 
optimizer/SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o 
rewrite/SUBSYS.o storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -lz 
-lcrypt -lresolv -lnsl -ldl -lm -lreadline  -o postgres
nodes/SUBSYS.o: In function `pprint':
nodes/SUBSYS.o(.text+0xdc95): undefined reference to `MIN'
nodes/SUBSYS.o(.text+0xdcfd): undefined reference to `MIN'
collect2: ld returned 1 exit status
make[2]: *** [postgres] Error 1
make[2]: Leaving directory 
`/home/ash/ash-server/Work/build/pgsql/src/backend'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/ash/ash-server/Work/build/pgsql/src'
make: *** [all] Error 2

In ./src/backend/nodes/print.c:

/* outdent */
if (indentLev > 0)
{   indentLev--;   indentDist = MIN(indentLev * INDENTSTOP, MAXINDENT);
}


If I add
#ifndef MIN
#define MIN(a,b) (((a)<(b)) ? (a) : (b))
#endif
to print.c it compiles fine.

Ashley Cambrell


<snip>

>
>That's annoying.  The LWLock changes were intended to solve the
>inefficiency with multiple CPUs, but it seems like we still have a
>problem somewhere.
>
>Could you recompile the backend with profiling enabled and try to get
>a profile from your test case?  To build a profilable backend, it's
>sufficient to do
>
>    cd .../src/backend
>    gmake clean
>    gmake PROFILE=-pg all
>    gmake install-bin
>
>(assuming you are using gcc).  Then restart the postmaster, and you
>should notice "gmon.out" files being dropped into the various database
>subdirectories anytime a backend exits.  Next run your test case,
>and as soon as it finishes copy the gmon.out file to a safe place.
>(You'll only be able to get the profile from the last process to exit,
>so try to make sure that this is representative.  Might be worth
>repeating the test a few times to make sure that the results don't
>vary a whole lot.)  Finally, do
>
>    gprof .../bin/postgres gmon.out >resultfile
>
>to produce a legible result.
>
>Oh, one more thing: on Linuxen you are likely to find that all the
>reported routine runtimes are zero, rendering the results useless.
>Apply the attached patch (for 7.2beta) to fix this.
>
>            regards, tom lane
>
>*** src/backend/postmaster/postmaster.c.orig    Wed Dec 12 14:52:03 2001
>--- src/backend/postmaster/postmaster.c    Mon Dec 17 19:38:29 2001
>***************
>*** 1823,1828 ****
>--- 1823,1829 ----
>  {
>      Backend    *bn;                /* for backend cleanup */
>      pid_t        pid;
>+     struct itimerval svitimer;
>  
>      /*
>       * Compute the cancel key that will be assigned to this backend. The
>***************
>*** 1858,1869 ****
>--- 1859,1874 ----
>      beos_before_backend_startup();
>  #endif
>  
>+     getitimer(ITIMER_PROF, &svitimer);
>+ 
>      pid = fork();
>  
>      if (pid == 0)                /* child */
>      {
>          int            status;
>  
>+         setitimer(ITIMER_PROF, &svitimer, NULL);
>+ 
>          free(bn);
>  #ifdef __BEOS__
>          /* Specific beos backend startup actions */
>
>---------------------------(end of broadcast)---------------------------
>TIP 5: Have you checked our extensive FAQ?
>
>http://www.postgresql.org/users-lounge/docs/faq.html
>





Re: 7.2 is slow? [compile problem]

От
"Christopher Kings-Lynne"
Дата:
I just got the same problem on latest CVS on freebsd/i386

gmake[4]: Entering directory `/home/chriskl/pgsql/src/backend/utils/time'
gcc -pipe -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -I../../../.
./src/include   -c -o tqual.o tqual.c
/usr/libexec/elf/ld -r -o SUBSYS.o tqual.o
gmake[4]: Leaving directory `/home/chriskl/pgsql/src/backend/utils/time'
/usr/libexec/elf/ld -r -o SUBSYS.o fmgrtab.o adt/SUBSYS.o cache/SUBSYS.o
error/SUBSYS.o fmgr/SUBSYS.o hash/SUBSYS.o init/SUBSYS.o misc/SUBS
YS.o mmgr/SUBSYS.o sort/SUBSYS.o time/SUBSYS.o
gmake[3]: Leaving directory `/home/chriskl/pgsql/src/backend/utils'
gcc -pipe -O2 -Wall -Wmissing-prototypes -Wmissing-declarations  -R/home/chr
iskl/local/lib -export-dynamic access/SUBSYS.o bootstrap/SUBSYS
.o catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o
lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o optimizer/
SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o rewrite/SUBSYS.o
storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -lz -lcrypt -lcomp
at -lm -lutil -lreadline  -o postgres
nodes/SUBSYS.o: In function `pprint':
nodes/SUBSYS.o(.text+0xda71): undefined reference to `MIN'
nodes/SUBSYS.o(.text+0xdade): undefined reference to `MIN'
gmake[2]: *** [postgres] Error 1
gmake[2]: Leaving directory `/home/chriskl/pgsql/src/backend'
gmake[1]: *** [all] Error 2
gmake[1]: Leaving directory `/home/chriskl/pgsql/src'
gmake: *** [all] Error 2

Chris

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Ashley Cambrell
> Sent: Thursday, 20 December 2001 8:51 AM
> To: Tom Lane
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] 7.2 is slow? [compile problem]
>
>
> I haven't actually tried to compile 7.2 from the CVS, but there seems to
> be a problem?  [maybe on my side]
>
> make[3]: Leaving directory
> `/home/ash/ash-server/Work/build/pgsql/src/backend/utils'
> gcc -O2 -Wall -Wmissing-prototypes -Wmissing-declarations
> -Wl,-rpath,/tmp//lib -export-dynamic access/SUBSYS.o bootstrap/SUBSYS.o
> catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o
> lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o
> optimizer/SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o
> rewrite/SUBSYS.o storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -lz
> -lcrypt -lresolv -lnsl -ldl -lm -lreadline  -o postgres
> nodes/SUBSYS.o: In function `pprint':
> nodes/SUBSYS.o(.text+0xdc95): undefined reference to `MIN'
> nodes/SUBSYS.o(.text+0xdcfd): undefined reference to `MIN'
> collect2: ld returned 1 exit status
> make[2]: *** [postgres] Error 1
> make[2]: Leaving directory
> `/home/ash/ash-server/Work/build/pgsql/src/backend'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/ash/ash-server/Work/build/pgsql/src'
> make: *** [all] Error 2
>
> In ./src/backend/nodes/print.c:
>
> /* outdent */
> if (indentLev > 0)
> {
>     indentLev--;
>     indentDist = MIN(indentLev * INDENTSTOP, MAXINDENT);
> }
>
>
> If I add
> #ifndef MIN
> #define MIN(a,b) (((a)<(b)) ? (a) : (b))
> #endif
> to print.c it compiles fine.
>
> Ashley Cambrell
>
>
> <snip>
>
> >
> >That's annoying.  The LWLock changes were intended to solve the
> >inefficiency with multiple CPUs, but it seems like we still have a
> >problem somewhere.
> >
> >Could you recompile the backend with profiling enabled and try to get
> >a profile from your test case?  To build a profilable backend, it's
> >sufficient to do
> >
> >    cd .../src/backend
> >    gmake clean
> >    gmake PROFILE=-pg all
> >    gmake install-bin
> >
> >(assuming you are using gcc).  Then restart the postmaster, and you
> >should notice "gmon.out" files being dropped into the various database
> >subdirectories anytime a backend exits.  Next run your test case,
> >and as soon as it finishes copy the gmon.out file to a safe place.
> >(You'll only be able to get the profile from the last process to exit,
> >so try to make sure that this is representative.  Might be worth
> >repeating the test a few times to make sure that the results don't
> >vary a whole lot.)  Finally, do
> >
> >    gprof .../bin/postgres gmon.out >resultfile
> >
> >to produce a legible result.
> >
> >Oh, one more thing: on Linuxen you are likely to find that all the
> >reported routine runtimes are zero, rendering the results useless.
> >Apply the attached patch (for 7.2beta) to fix this.
> >
> >            regards, tom lane
> >
> >*** src/backend/postmaster/postmaster.c.orig    Wed Dec 12 14:52:03 2001
> >--- src/backend/postmaster/postmaster.c    Mon Dec 17 19:38:29 2001
> >***************
> >*** 1823,1828 ****
> >--- 1823,1829 ----
> >  {
> >      Backend    *bn;                /* for backend cleanup */
> >      pid_t        pid;
> >+     struct itimerval svitimer;
> >
> >      /*
> >       * Compute the cancel key that will be assigned to this backend. The
> >***************
> >*** 1858,1869 ****
> >--- 1859,1874 ----
> >      beos_before_backend_startup();
> >  #endif
> >
> >+     getitimer(ITIMER_PROF, &svitimer);
> >+
> >      pid = fork();
> >
> >      if (pid == 0)                /* child */
> >      {
> >          int            status;
> >
> >+         setitimer(ITIMER_PROF, &svitimer, NULL);
> >+
> >          free(bn);
> >  #ifdef __BEOS__
> >          /* Specific beos backend startup actions */
> >
> >---------------------------(end of broadcast)---------------------------
> >TIP 5: Have you checked our extensive FAQ?
> >
> >http://www.postgresql.org/users-lounge/docs/faq.html
> >
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>



Re: 7.2 is slow? [compile problem]

От
Bruce Momjian
Дата:
OK, I just committed a fix.  MIN() was used in the pretty node print
patch;  should have been Min().

---------------------------------------------------------------------------

> I just got the same problem on latest CVS on freebsd/i386
> 
> gmake[4]: Entering directory `/home/chriskl/pgsql/src/backend/utils/time'
> gcc -pipe -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -I../../../.
> ./src/include   -c -o tqual.o tqual.c
> /usr/libexec/elf/ld -r -o SUBSYS.o tqual.o
> gmake[4]: Leaving directory `/home/chriskl/pgsql/src/backend/utils/time'
> /usr/libexec/elf/ld -r -o SUBSYS.o fmgrtab.o adt/SUBSYS.o cache/SUBSYS.o
> error/SUBSYS.o fmgr/SUBSYS.o hash/SUBSYS.o init/SUBSYS.o misc/SUBS
> YS.o mmgr/SUBSYS.o sort/SUBSYS.o time/SUBSYS.o
> gmake[3]: Leaving directory `/home/chriskl/pgsql/src/backend/utils'
> gcc -pipe -O2 -Wall -Wmissing-prototypes -Wmissing-declarations  -R/home/chr
> iskl/local/lib -export-dynamic access/SUBSYS.o bootstrap/SUBSYS
> .o catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o
> lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o optimizer/
> SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o rewrite/SUBSYS.o
> storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -lz -lcrypt -lcomp
> at -lm -lutil -lreadline  -o postgres
> nodes/SUBSYS.o: In function `pprint':
> nodes/SUBSYS.o(.text+0xda71): undefined reference to `MIN'
> nodes/SUBSYS.o(.text+0xdade): undefined reference to `MIN'
> gmake[2]: *** [postgres] Error 1
> gmake[2]: Leaving directory `/home/chriskl/pgsql/src/backend'
> gmake[1]: *** [all] Error 2
> gmake[1]: Leaving directory `/home/chriskl/pgsql/src'
> gmake: *** [all] Error 2
> 
> Chris
> 
> > -----Original Message-----
> > From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Ashley Cambrell
> > Sent: Thursday, 20 December 2001 8:51 AM
> > To: Tom Lane
> > Cc: pgsql-hackers@postgresql.org
> > Subject: Re: [HACKERS] 7.2 is slow? [compile problem]
> >
> >
> > I haven't actually tried to compile 7.2 from the CVS, but there seems to
> > be a problem?  [maybe on my side]
> >
> > make[3]: Leaving directory
> > `/home/ash/ash-server/Work/build/pgsql/src/backend/utils'
> > gcc -O2 -Wall -Wmissing-prototypes -Wmissing-declarations
> > -Wl,-rpath,/tmp//lib -export-dynamic access/SUBSYS.o bootstrap/SUBSYS.o
> > catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o
> > lib/SUBSYS.o libpq/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o
> > optimizer/SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o
> > rewrite/SUBSYS.o storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o -lz
> > -lcrypt -lresolv -lnsl -ldl -lm -lreadline  -o postgres
> > nodes/SUBSYS.o: In function `pprint':
> > nodes/SUBSYS.o(.text+0xdc95): undefined reference to `MIN'
> > nodes/SUBSYS.o(.text+0xdcfd): undefined reference to `MIN'
> > collect2: ld returned 1 exit status
> > make[2]: *** [postgres] Error 1
> > make[2]: Leaving directory
> > `/home/ash/ash-server/Work/build/pgsql/src/backend'
> > make[1]: *** [all] Error 2
> > make[1]: Leaving directory `/home/ash/ash-server/Work/build/pgsql/src'
> > make: *** [all] Error 2
> >
> > In ./src/backend/nodes/print.c:
> >
> > /* outdent */
> > if (indentLev > 0)
> > {
> >     indentLev--;
> >     indentDist = MIN(indentLev * INDENTSTOP, MAXINDENT);
> > }
> >
> >
> > If I add
> > #ifndef MIN
> > #define MIN(a,b) (((a)<(b)) ? (a) : (b))
> > #endif
> > to print.c it compiles fine.
> >
> > Ashley Cambrell
> >
> >
> > <snip>
> >
> > >
> > >That's annoying.  The LWLock changes were intended to solve the
> > >inefficiency with multiple CPUs, but it seems like we still have a
> > >problem somewhere.
> > >
> > >Could you recompile the backend with profiling enabled and try to get
> > >a profile from your test case?  To build a profilable backend, it's
> > >sufficient to do
> > >
> > >    cd .../src/backend
> > >    gmake clean
> > >    gmake PROFILE=-pg all
> > >    gmake install-bin
> > >
> > >(assuming you are using gcc).  Then restart the postmaster, and you
> > >should notice "gmon.out" files being dropped into the various database
> > >subdirectories anytime a backend exits.  Next run your test case,
> > >and as soon as it finishes copy the gmon.out file to a safe place.
> > >(You'll only be able to get the profile from the last process to exit,
> > >so try to make sure that this is representative.  Might be worth
> > >repeating the test a few times to make sure that the results don't
> > >vary a whole lot.)  Finally, do
> > >
> > >    gprof .../bin/postgres gmon.out >resultfile
> > >
> > >to produce a legible result.
> > >
> > >Oh, one more thing: on Linuxen you are likely to find that all the
> > >reported routine runtimes are zero, rendering the results useless.
> > >Apply the attached patch (for 7.2beta) to fix this.
> > >
> > >            regards, tom lane
> > >
> > >*** src/backend/postmaster/postmaster.c.orig    Wed Dec 12 14:52:03 2001
> > >--- src/backend/postmaster/postmaster.c    Mon Dec 17 19:38:29 2001
> > >***************
> > >*** 1823,1828 ****
> > >--- 1823,1829 ----
> > >  {
> > >      Backend    *bn;                /* for backend cleanup */
> > >      pid_t        pid;
> > >+     struct itimerval svitimer;
> > >
> > >      /*
> > >       * Compute the cancel key that will be assigned to this backend. The
> > >***************
> > >*** 1858,1869 ****
> > >--- 1859,1874 ----
> > >      beos_before_backend_startup();
> > >  #endif
> > >
> > >+     getitimer(ITIMER_PROF, &svitimer);
> > >+
> > >      pid = fork();
> > >
> > >      if (pid == 0)                /* child */
> > >      {
> > >          int            status;
> > >
> > >+         setitimer(ITIMER_PROF, &svitimer, NULL);
> > >+
> > >          free(bn);
> > >  #ifdef __BEOS__
> > >          /* Specific beos backend startup actions */
> > >
> > >---------------------------(end of broadcast)---------------------------
> > >TIP 5: Have you checked our extensive FAQ?
> > >
> > >http://www.postgresql.org/users-lounge/docs/faq.html
> > >
> >
> >
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
> >
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org
> 

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: 7.2 is slow? [compile problem]

От
Tom Lane
Дата:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> OK, I just committed a fix.  MIN() was used in the pretty node print
> patch;  should have been Min().

Mea maxima (or MINima?) culpa.  Thanks ...
        regards, tom lane


Re: 7.2 is slow?

От
Tom Lane
Дата:
Jussi Mikkola <jussi.mikkola@bonware.com> writes:
> Well, here's the profile, but yes, almost all the times are zero.

It looks to me like this profile only covers the postmaster, not a
backend.  You want to use gmon.out from down inside the database's
subdirectory ($PGDATA/base/something/gmon.out).

> Both without the patch, and with it. Did I miss something? (Yes, I did
> make and install afterwards ;-)

[ scratches head ]  Dunno.  You did restart the postmaster after
installing the new executable, right?
        regards, tom lane


Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Tom Lane wrote:
> 
> Hannu Krosing <hannu@tm.ee> writes:
> > ./pgbench -i -s 10 -p 5433
> > ./pgbench -p 5433 -c   1 -t 100      171/177    162/166    169/173
> > ./pgbench -p 5433 -c   5 -t 100      140/143    191/196    202/207
> > ./pgbench -p 5433 -c  10 -t 100      132/135    165/168    159/163
> > ./pgbench -p 5433 -c  25 -t 100       65/ 66     60/ 60     75/ 76
> > ./pgbench -p 5433 -c  50 -t 100       60/ 61     43/ 43     55/ 59
> > ./pgbench -p 5433 -c 100 -t 100       48/ 48     23/ 23     34/ 34
> 
> You realize, of course, that when the number of clients exceeds the
> scale factor you're not really measuring anything except update
> contention on the "branch" rows?  Every transaction tries to update
> the balance for its branch, so if you have more clients than branches
> then there will be lots of transactions blocked waiting for someone
> else to commit.  With a 10:1 ratio, there will be several transactions
> blocked waiting for *each* active transaction; and when that guy
> commits, all the others will waken simultaneously and contend for the
> chance to update the branch row.  One will win, the others will go
> back to sleep, having done nothing except wasting CPU time.  Thus a
> severe falloff in measured TPS is inevitable when -c >> -s.  I don't
> think this scenario has all that much to do with real-world loads,
> however.

I did some benchmarking and the interesting part is that 7.2b4 is up to 
2.5X faster than 7.1.3 for _small_ scale factors and up to 25% slower 
when there is no contention (-s128, clients <= 128)

Perhaps the waiting on lock somehow organizes things to happen in some 
order that avoids some stupidity in some other locking logic ?

I run benchmark (with added vacuum full for 7.2b4) on Dual PIII 800MHz 
with 1 G of RAM and an IDE disk. The results are mean from six runs 
with two slowes removed (there was other activity going on sometimes)

they are for scale factors 1, 10 and 128 

in order to measure real performance of roughly the _same_ dataset each
test run did the same total number of transactions 512 with each client 
doing 512/nr_of_trx.

Re: 7.2 is slow?

От
Tom Lane
Дата:
Hannu Krosing <hannu@tm.ee> writes:
> in order to measure real performance of roughly the _same_ dataset each
> test run did the same total number of transactions 512 with each client 
> doing 512/nr_of_trx.

That means you're only measuring a few transactions per backend (as few
as 4, near the upper end of the scale).  I think the results may say
more about backend-startup transients than true peak throughput.
Could you try it again with a run about ten times that long?
        regards, tom lane


Re: 7.2 is slow?

От
Hannu Krosing
Дата:

Tom Lane wrote:

>Hannu Krosing <hannu@tm.ee> writes:
>
>>in order to measure real performance of roughly the _same_ dataset each
>>test run did the same total number of transactions 512 with each client 
>>doing 512/nr_of_trx.
>>
>
>That means you're only measuring a few transactions per backend (as few
>as 4, near the upper end of the scale).  I think the results may say
>more about backend-startup transients than true peak throughput.
>Could you try it again with a run about ten times that long?
>
I did run 4096trx on 7.2b4 with  -s 1, best 4-of-6
   512trx 4096trx ratio
1    180.59    90.15    2.00
2    221.52    80.92    2.74
4    203.72    75.60    2.69
8    179.54    69.29    2.59
16    156.68    63.15    2.48
32    123.48    57.73    2.14
64    89.99    54.14    1.66
128    61.84    48.97    1.26

so it seems that large number of of transactions degrades tps 
performance faster than
connection setup overhead.

the

I'll try running the whole suite again with higher number of 
transactions i  a few days

------------------
Hannu




Re: 7.2 is slow?

От
Jussi Mikkola
Дата:
Hi Tom!

Well, here's the profile, but yes, almost all the times are zero. Both without
the patch, and with it. Did I miss something? (Yes, I did make and install
afterwards ;-)

I made a new database, so as it was growing, the times were even slower than
before. (It would be nice, if I could create a large database from the
beginning.;)

Is this of any help? Do you need the same result with less cpus?

Jussi


Tom Lane wrote:

> Jussi Mikkola <jussi.mikkola@bonware.com> writes:
> > Yes, now I've tested with 7.2b4. The result is about the  same as with 7.1.
>
> > About 200 messages with four processors and about 600 messages with one
> > processor.
>
> That's annoying.  The LWLock changes were intended to solve the
> inefficiency with multiple CPUs, but it seems like we still have a
> problem somewhere.
>
> Could you recompile the backend with profiling enabled and try to get
> a profile from your test case?  To build a profilable backend, it's
> sufficient to do
>
>         cd .../src/backend
>         gmake clean
>         gmake PROFILE=-pg all
>         gmake install-bin
>
> (assuming you are using gcc).  Then restart the postmaster, and you
> should notice "gmon.out" files being dropped into the various database
> subdirectories anytime a backend exits.  Next run your test case,
> and as soon as it finishes copy the gmon.out file to a safe place.
> (You'll only be able to get the profile from the last process to exit,
> so try to make sure that this is representative.  Might be worth
> repeating the test a few times to make sure that the results don't
> vary a whole lot.)  Finally, do
>
>         gprof .../bin/postgres gmon.out >resultfile
>
> to produce a legible result.
>
> Oh, one more thing: on Linuxen you are likely to find that all the
> reported routine runtimes are zero, rendering the results useless.
> Apply the attached patch (for 7.2beta) to fix this.
>
>                         regards, tom lane
>
> *** src/backend/postmaster/postmaster.c.orig    Wed Dec 12 14:52:03 2001
> --- src/backend/postmaster/postmaster.c Mon Dec 17 19:38:29 2001
> ***************
> *** 1823,1828 ****
> --- 1823,1829 ----
>   {
>         Backend    *bn;                         /* for backend cleanup */
>         pid_t           pid;
> +       struct itimerval svitimer;
>
>         /*
>          * Compute the cancel key that will be assigned to this backend. The
> ***************
> *** 1858,1869 ****
> --- 1859,1874 ----
>         beos_before_backend_startup();
>   #endif
>
> +       getitimer(ITIMER_PROF, &svitimer);
> +
>         pid = fork();
>
>         if (pid == 0)                           /* child */
>         {
>                 int                     status;
>
> +               setitimer(ITIMER_PROF, &svitimer, NULL);
> +
>                 free(bn);
>   #ifdef __BEOS__
>                 /* Specific beos backend startup actions */

--
Jussi Mikkola                    Project Manager
Bonware Oy                       gsm +358 40 830 7561
Tekniikantie 12                  tel +358 9 2517 5570
02150 Espoo                      fax +358 9 2517 5571
Finland                          www.bonware.com


Flat profile:

Each sample counts as 0.01 seconds.
 no time accumulated

  %   cumulative   self              self     total
 time   seconds   seconds    calls  Ts/call  Ts/call  name
  0.00      0.00     0.00     2056     0.00     0.00  LWLockAssign
  0.00      0.00     0.00     2002     0.00     0.00  LWLockRelease
  0.00      0.00     0.00     2000     0.00     0.00  LWLockAcquire
  0.00      0.00     0.00     1161     0.00     0.00  PostmasterRandom
  0.00      0.00     0.00      837     0.00     0.00  isblank
  0.00      0.00     0.00      476     0.00     0.00  LockBuffer
  0.00      0.00     0.00      464     0.00     0.00  CharRemap
  0.00      0.00     0.00      319     0.00     0.00  AllocSetAlloc
  0.00      0.00     0.00      319     0.00     0.00  MemoryContextAlloc
  0.00      0.00     0.00      250     0.00     0.00  call_hash
  0.00      0.00     0.00      250     0.00     0.00  hash_search
  0.00      0.00     0.00      249     0.00     0.00  next_token
  0.00      0.00     0.00      243     0.00     0.00  AllocSetFree
  0.00      0.00     0.00      243     0.00     0.00  BasicOpenFile
  0.00      0.00     0.00      243     0.00     0.00  pfree
  0.00      0.00     0.00      239     0.00     0.00  XLogFlush
  0.00      0.00     0.00      238     0.00     0.00  AddBufferToFreelist
  0.00      0.00     0.00      238     0.00     0.00  PinBuffer
  0.00      0.00     0.00      238     0.00     0.00  RelationNodeCacheGetRelation
  0.00      0.00     0.00      238     0.00     0.00  StartBufferIO
  0.00      0.00     0.00      238     0.00     0.00  TerminateBufferIO
  0.00      0.00     0.00      238     0.00     0.00  UnpinBuffer
  0.00      0.00     0.00      238     0.00     0.00  _mdfd_blind_getseg
  0.00      0.00     0.00      238     0.00     0.00  mdblindwrt
  0.00      0.00     0.00      238     0.00     0.00  relpath
  0.00      0.00     0.00      238     0.00     0.00  smgrblindwrt
  0.00      0.00     0.00      238     0.00     0.00  tag_hash
  0.00      0.00     0.00      234     0.00     0.00  StreamClose
  0.00      0.00     0.00      233     0.00     0.00  DLAddHead
  0.00      0.00     0.00      233     0.00     0.00  DLNewElem
  0.00      0.00     0.00      232     0.00     0.00  BackendStartup
  0.00      0.00     0.00      232     0.00     0.00  ConnCreate
  0.00      0.00     0.00      232     0.00     0.00  ConnFree
  0.00      0.00     0.00      232     0.00     0.00  RandomSalt
  0.00      0.00     0.00      232     0.00     0.00  StreamConnection
  0.00      0.00     0.00      205     0.00     0.00  read_to_eol
  0.00      0.00     0.00      187     0.00     0.00  GUC_yylex
  0.00      0.00     0.00      135     0.00     0.00  pgstat_ispgstat
  0.00      0.00     0.00      134     0.00     0.00  CleanupProc
  0.00      0.00     0.00      134     0.00     0.00  DLFreeElem
  0.00      0.00     0.00      134     0.00     0.00  DLRemove
  0.00      0.00     0.00      133     0.00     0.00  pgstat_beterm
  0.00      0.00     0.00      133     0.00     0.00  pgstat_send
  0.00      0.00     0.00       59     0.00     0.00  ShmemAlloc
  0.00      0.00     0.00       44     0.00     0.00  seg_alloc
  0.00      0.00     0.00       36     0.00     0.00  my_log2
  0.00      0.00     0.00       33     0.00     0.00  InternalIpcSemaphoreCreate
  0.00      0.00     0.00       33     0.00     0.00  IpcSemaphoreCreate
  0.00      0.00     0.00       33     0.00     0.00  IpcSemaphoreUnlock
  0.00      0.00     0.00       28     0.00     0.00  DynaHashAlloc
  0.00      0.00     0.00       24     0.00     0.00  pqsignal
  0.00      0.00     0.00       19     0.00     0.00  newNode
  0.00      0.00     0.00       16     0.00     0.00  lappend
  0.00      0.00     0.00       16     0.00     0.00  lcons
  0.00      0.00     0.00       16     0.00     0.00  nconc
  0.00      0.00     0.00       14     0.00     0.00  AllocSetContextCreate
  0.00      0.00     0.00       14     0.00     0.00  AllocSetInit
  0.00      0.00     0.00       14     0.00     0.00  MemoryContextCreate
  0.00      0.00     0.00       13     0.00     0.00  MemoryContextStrdup
  0.00      0.00     0.00       12     0.00     0.00  ShmemInitStruct
  0.00      0.00     0.00       12     0.00     0.00  string_hash
  0.00      0.00     0.00        8     0.00     0.00  WriteCLOGPage
  0.00      0.00     0.00        8     0.00     0.00  hash_create
  0.00      0.00     0.00        8     0.00     0.00  hdefault
  0.00      0.00     0.00        8     0.00     0.00  init_htab
  0.00      0.00     0.00        7     0.00     0.00  AllocateFile
  0.00      0.00     0.00        6     0.00     0.00  FreeFile
  0.00      0.00     0.00        5     0.00     0.00  ShmemInitHash
  0.00      0.00     0.00        5     0.00     0.00  hash_estimate_size
  0.00      0.00     0.00        5     0.00     0.00  hash_select_dirsize
  0.00      0.00     0.00        5     0.00     0.00  on_shmem_exit
  0.00      0.00     0.00        4     0.00     0.00  SetConfigOption
  0.00      0.00     0.00        4     0.00     0.00  find_option
  0.00      0.00     0.00        4     0.00     0.00  on_proc_exit
  0.00      0.00     0.00        4     0.00     0.00  set_config_option
  0.00      0.00     0.00        3     0.00     0.00  GUC_yy_load_buffer_state
  0.00      0.00     0.00        3     0.00     0.00  MemoryContextSwitchTo
  0.00      0.00     0.00        3     0.00     0.00  NumLWLocks
  0.00      0.00     0.00        3     0.00     0.00  SSDataBase
  0.00      0.00     0.00        3     0.00     0.00  lconsi
  0.00      0.00     0.00        3     0.00     0.00  parse_int
  0.00      0.00     0.00        2     0.00     0.00  CreateLockFile
  0.00      0.00     0.00        2     0.00     0.00  FindExec
  0.00      0.00     0.00        2     0.00     0.00  GUC_yy_flex_alloc
  0.00      0.00     0.00        2     0.00     0.00  GUC_yy_flush_buffer
  0.00      0.00     0.00        2     0.00     0.00  GUC_yy_get_next_buffer
  0.00      0.00     0.00        2     0.00     0.00  GUC_yy_init_buffer
  0.00      0.00     0.00        2     0.00     0.00  IgnoreSystemIndexes
  0.00      0.00     0.00        2     0.00     0.00  LWLockConditionalAcquire
  0.00      0.00     0.00        2     0.00     0.00  LWLockShmemSize
  0.00      0.00     0.00        2     0.00     0.00  SInvalShmemSize
  0.00      0.00     0.00        2     0.00     0.00  StreamServerPort
  0.00      0.00     0.00        2     0.00     0.00  ValidateBinary
  0.00      0.00     0.00        2     0.00     0.00  ValidatePgVersion
  0.00      0.00     0.00        2     0.00     0.00  XLOGPathInit
  0.00      0.00     0.00        2     0.00     0.00  tokenize_file
  0.00      0.00     0.00        1     0.00     0.00  AbortBufferIO
  0.00      0.00     0.00        1     0.00     0.00  BaseInit
  0.00      0.00     0.00        1     0.00     0.00  BootstrapMain
  0.00      0.00     0.00        1     0.00     0.00  BufferShmemSize
  0.00      0.00     0.00        1     0.00     0.00  BufferSync
  0.00      0.00     0.00        1     0.00     0.00  CLOGPhysicalWritePage
  0.00      0.00     0.00        1     0.00     0.00  CLOGShmemInit
  0.00      0.00     0.00        1     0.00     0.00  CLOGShmemSize
  0.00      0.00     0.00        1     0.00     0.00  CheckPointCLOG
  0.00      0.00     0.00        1     0.00     0.00  ClearDateCache
  0.00      0.00     0.00        1     0.00     0.00  ClosePostmasterPorts
  0.00      0.00     0.00        1     0.00     0.00  CreateCacheMemoryContext
  0.00      0.00     0.00        1     0.00     0.00  CreateCheckPoint
  0.00      0.00     0.00        1     0.00     0.00  CreateDataDirLockFile
  0.00      0.00     0.00        1     0.00     0.00  CreateDummyCaches
  0.00      0.00     0.00        1     0.00     0.00  CreateLWLocks
  0.00      0.00     0.00        1     0.00     0.00  CreateOptsFile
  0.00      0.00     0.00        1     0.00     0.00  CreateSharedInvalidationState
  0.00      0.00     0.00        1     0.00     0.00  CreateSharedMemoryAndSemaphores
  0.00      0.00     0.00        1     0.00     0.00  CreateSocketLockFile
  0.00      0.00     0.00        1     0.00     0.00  CreateSpinlocks
  0.00      0.00     0.00        1     0.00     0.00  DLNewList
  0.00      0.00     0.00        1     0.00     0.00  DebugFileOpen
  0.00      0.00     0.00        1     0.00     0.00  DummyProcKill
  0.00      0.00     0.00        1     0.00     0.00  EnableExceptionHandling
  0.00      0.00     0.00        1     0.00     0.00  FlushBufferPool
  0.00      0.00     0.00        1     0.00     0.00  FreeSpaceShmemSize
  0.00      0.00     0.00        1     0.00     0.00  GUC_yy_create_buffer
  0.00      0.00     0.00        1     0.00     0.00  GUC_yy_get_previous_state
  0.00      0.00     0.00        1     0.00     0.00  GUC_yyrestart
  0.00      0.00     0.00        1     0.00     0.00  GUC_yywrap
  0.00      0.00     0.00        1     0.00     0.00  GetCurrentTransactionId
  0.00      0.00     0.00        1     0.00     0.00  GetRedoRecPtr
  0.00      0.00     0.00        1     0.00     0.00  GetUndoRecPtr
  0.00      0.00     0.00        1     0.00     0.00  InitBufTable
  0.00      0.00     0.00        1     0.00     0.00  InitBufferPool
  0.00      0.00     0.00        1     0.00     0.00  InitBufferPoolAccess
  0.00      0.00     0.00        1     0.00     0.00  InitCommunication
  0.00      0.00     0.00        1     0.00     0.00  InitDummyProcess
  0.00      0.00     0.00        1     0.00     0.00  InitFreeList
  0.00      0.00     0.00        1     0.00     0.00  InitFreeSpaceMap
  0.00      0.00     0.00        1     0.00     0.00  InitLocalBuffer
  0.00      0.00     0.00        1     0.00     0.00  InitLockTable
  0.00      0.00     0.00        1     0.00     0.00  InitLocks
  0.00      0.00     0.00        1     0.00     0.00  InitProcGlobal
  0.00      0.00     0.00        1     0.00     0.00  InitShmemAllocation
  0.00      0.00     0.00        1     0.00     0.00  InitShmemIndex
  0.00      0.00     0.00        1     0.00     0.00  InternalIpcMemoryCreate
  0.00      0.00     0.00        1     0.00     0.00  IpcInitKeyAssignment
  0.00      0.00     0.00        1     0.00     0.00  IpcMemoryCreate
  0.00      0.00     0.00        1     0.00     0.00  LWLockReleaseAll
  0.00      0.00     0.00        1     0.00     0.00  LockMethodInit
  0.00      0.00     0.00        1     0.00     0.00  LockMethodTableInit
  0.00      0.00     0.00        1     0.00     0.00  LockMethodTableRename
  0.00      0.00     0.00        1     0.00     0.00  LockShmemSize
  0.00      0.00     0.00        1     0.00     0.00  MemoryContextInit
  0.00      0.00     0.00        1     0.00     0.00  PMSignalInit
  0.00      0.00     0.00        1     0.00     0.00  PostmasterMain
  0.00      0.00     0.00        1     0.00     0.00  PreallocXlogFiles
  0.00      0.00     0.00        1     0.00     0.00  ProcessConfigFile
  0.00      0.00     0.00        1     0.00     0.00  ReadControlFile
  0.00      0.00     0.00        1     0.00     0.00  RecordSharedMemoryInLockFile
  0.00      0.00     0.00        1     0.00     0.00  RemovePgTempFiles
  0.00      0.00     0.00        1     0.00     0.00  ResetAllOptions
  0.00      0.00     0.00        1     0.00     0.00  ResetBufferPool
  0.00      0.00     0.00        1     0.00     0.00  ResetLocalBufferPool
  0.00      0.00     0.00        1     0.00     0.00  SHMQueueElemInit
  0.00      0.00     0.00        1     0.00     0.00  SHMQueueInit
  0.00      0.00     0.00        1     0.00     0.00  SIBufferInit
  0.00      0.00     0.00        1     0.00     0.00  ServerLoop
  0.00      0.00     0.00        1     0.00     0.00  SetDataDir
  0.00      0.00     0.00        1     0.00     0.00  SetRedoRecPtr
  0.00      0.00     0.00        1     0.00     0.00  SetThisStartUpID
  0.00      0.00     0.00        1     0.00     0.00  ShutdownBufferPoolAccess
  0.00      0.00     0.00        1     0.00     0.00  TouchSocketLockFile
  0.00      0.00     0.00        1     0.00     0.00  UnlockBuffers
  0.00      0.00     0.00        1     0.00     0.00  UpdateControlFile
  0.00      0.00     0.00        1     0.00     0.00  XLOGShmemInit
  0.00      0.00     0.00        1     0.00     0.00  XLOGShmemSize
  0.00      0.00     0.00        1     0.00     0.00  XLogFileInit
  0.00      0.00     0.00        1     0.00     0.00  XLogFileOpen
  0.00      0.00     0.00        1     0.00     0.00  XLogInsert
  0.00      0.00     0.00        1     0.00     0.00  XLogWrite
  0.00      0.00     0.00        1     0.00     0.00  ZeroProcSemaphore
  0.00      0.00     0.00        1     0.00     0.00  assign_defaultxactisolevel
  0.00      0.00     0.00        1     0.00     0.00  assign_xlog_sync_method
  0.00      0.00     0.00        1     0.00     0.00  checkDataDir
  0.00      0.00     0.00        1     0.00     0.00  crypt_getpwdfilename
  0.00      0.00     0.00        1     0.00     0.00  crypt_openpwdfile
  0.00      0.00     0.00        1     0.00     0.00  element_alloc
  0.00      0.00     0.00        1     0.00     0.00  free_name_value_list
  0.00      0.00     0.00        1     0.00     0.00  initMasks
  0.00      0.00     0.00        1     0.00     0.00  init_ps_display
  0.00      0.00     0.00        1     0.00     0.00  issue_xlog_fsync
  0.00      0.00     0.00        1     0.00     0.00  load_hba
  0.00      0.00     0.00        1     0.00     0.00  load_hba_and_ident
  0.00      0.00     0.00        1     0.00     0.00  load_ident
  0.00      0.00     0.00        1     0.00     0.00  load_password_cache
  0.00      0.00     0.00        1     0.00     0.00  mdabort
  0.00      0.00     0.00        1     0.00     0.00  mdinit
  0.00      0.00     0.00        1     0.00     0.00  mdsync
  0.00      0.00     0.00        1     0.00     0.00  on_exit_reset
  0.00      0.00     0.00        1     0.00     0.00  parse_bool
  0.00      0.00     0.00        1     0.00     0.00  pg_fdatasync
  0.00      0.00     0.00        1     0.00     0.00  pg_fsync
  0.00      0.00     0.00        1     0.00     0.00  pgstat_close_sockets
  0.00      0.00     0.00        1     0.00     0.00  pgstat_init
  0.00      0.00     0.00        1     0.00     0.00  pgstat_start
  0.00      0.00     0.00        1     0.00     0.00  pqinitmask
  0.00      0.00     0.00        1     0.00     0.00  proc_exit
  0.00      0.00     0.00        1     0.00     0.00  reset_shared
  0.00      0.00     0.00        1     0.00     0.00  save_ps_display_args
  0.00      0.00     0.00        1     0.00     0.00  set_ps_display
  0.00      0.00     0.00        1     0.00     0.00  shmem_exit
  0.00      0.00     0.00        1     0.00     0.00  smgrabort
  0.00      0.00     0.00        1     0.00     0.00  smgrinit
  0.00      0.00     0.00        1     0.00     0.00  smgrshutdown
  0.00      0.00     0.00        1     0.00     0.00  smgrsync

 %         the percentage of the total running time of the
time       program used by this function.

cumulative a running sum of the number of seconds accounted
 seconds   for by this function and those listed above it.

 self      the number of seconds accounted for by this
seconds    function alone.  This is the major sort for this
           listing.

calls      the number of times this function was invoked, if
           this function is profiled, else blank.

 self      the average number of milliseconds spent in this
ms/call    function per call, if this function is profiled,
       else blank.

 total     the average number of milliseconds spent in this
ms/call    function and its descendents per call, if this
       function is profiled, else blank.

name       the name of the function.  This is the minor sort
           for this listing. The index shows the location of
       the function in the gprof listing. If the index is
       in parenthesis it shows where it would appear in
       the gprof listing if it were to be printed.

             Call graph (explanation follows)


granularity: each sample hit covers 4 byte(s) no time propagated

index % time    self  children    called     name
                0.00    0.00       8/2056        CLOGShmemInit [102]
                0.00    0.00    2048/2056        InitBufferPool [131]
[1]      0.0    0.00    0.00    2056         LWLockAssign [1]
-----------------------------------------------
                0.00    0.00       1/2002        WriteCLOGPage [61]
                0.00    0.00       1/2002        XLogInsert [177]
                0.00    0.00       1/2002        InitBufferPool [131]
                0.00    0.00       1/2002        InitShmemIndex [142]
                0.00    0.00       1/2002        GetUndoRecPtr [129]
                0.00    0.00       1/2002        LockMethodTableInit [148]
                0.00    0.00       2/2002        CheckPointCLOG [104]
                0.00    0.00       2/2002        XLogFlush [16]
                0.00    0.00       5/2002        CreateCheckPoint [108]
                0.00    0.00      11/2002        ShmemInitStruct [59]
                0.00    0.00     238/2002        LockBuffer [6]
                0.00    0.00     238/2002        TerminateBufferIO [21]
                0.00    0.00    1500/2002        BufferSync [100]
[2]      0.0    0.00    0.00    2002         LWLockRelease [2]
-----------------------------------------------
                0.00    0.00       1/2000        CheckPointCLOG [104]
                0.00    0.00       1/2000        XLogInsert [177]
                0.00    0.00       1/2000        XLogFlush [16]
                0.00    0.00       1/2000        InitBufferPool [131]
                0.00    0.00       1/2000        GetUndoRecPtr [129]
                0.00    0.00       1/2000        LockMethodTableInit [148]
                0.00    0.00       2/2000        WriteCLOGPage [61]
                0.00    0.00       4/2000        CreateCheckPoint [108]
                0.00    0.00      12/2000        ShmemInitStruct [59]
                0.00    0.00     238/2000        LockBuffer [6]
                0.00    0.00     238/2000        StartBufferIO [20]
                0.00    0.00    1500/2000        BufferSync [100]
[3]      0.0    0.00    0.00    2000         LWLockAcquire [3]
-----------------------------------------------
                0.00    0.00       1/1161        SSDataBase [78]
                0.00    0.00     232/1161        BackendStartup [30]
                0.00    0.00     928/1161        RandomSalt [33]
[4]      0.0    0.00    0.00    1161         PostmasterRandom [4]
-----------------------------------------------
                0.00    0.00     837/837         next_token [12]
[5]      0.0    0.00    0.00     837         isblank [5]
-----------------------------------------------
                0.00    0.00     476/476         FlushBufferPool [121]
[6]      0.0    0.00    0.00     476         LockBuffer [6]
                0.00    0.00     238/2002        LWLockRelease [2]
                0.00    0.00     238/2000        LWLockAcquire [3]
-----------------------------------------------
                0.00    0.00     464/464         ConnCreate [31]
[7]      0.0    0.00    0.00     464         CharRemap [7]
-----------------------------------------------
                0.00    0.00     319/319         MemoryContextAlloc [9]
[8]      0.0    0.00    0.00     319         AllocSetAlloc [8]
-----------------------------------------------
                0.00    0.00       1/319         crypt_getpwdfilename [183]
                0.00    0.00       1/319         load_ident [192]
                0.00    0.00       1/319         CreateOptsFile [112]
                0.00    0.00       1/319         mdinit [195]
                0.00    0.00       2/319         load_hba [190]
                0.00    0.00       2/319         LockMethodTableInit [148]
                0.00    0.00      13/319         MemoryContextCreate [57]
                0.00    0.00      13/319         MemoryContextStrdup [58]
                0.00    0.00      19/319         newNode [51]
                0.00    0.00      28/319         DynaHashAlloc [49]
                0.00    0.00     238/319         relpath [24]
[9]      0.0    0.00    0.00     319         MemoryContextAlloc [9]
                0.00    0.00     319/319         AllocSetAlloc [8]
-----------------------------------------------
                0.00    0.00     250/250         hash_search [11]
[10]     0.0    0.00    0.00     250         call_hash [10]
-----------------------------------------------
                0.00    0.00       1/250         InitShmemIndex [142]
                0.00    0.00      11/250         ShmemInitStruct [59]
                0.00    0.00     238/250         RelationNodeCacheGetRelation [19]
[11]     0.0    0.00    0.00     250         hash_search [11]
                0.00    0.00     250/250         call_hash [10]
                0.00    0.00     238/238         tag_hash [26]
                0.00    0.00      12/12          string_hash [60]
                0.00    0.00       1/59          ShmemAlloc [43]
                0.00    0.00       1/1           element_alloc [185]
-----------------------------------------------
                0.00    0.00     249/249         tokenize_file [95]
[12]     0.0    0.00    0.00     249         next_token [12]
                0.00    0.00     837/837         isblank [5]
-----------------------------------------------
                0.00    0.00     243/243         pfree [15]
[13]     0.0    0.00    0.00     243         AllocSetFree [13]
-----------------------------------------------
                0.00    0.00       1/243         CLOGPhysicalWritePage [101]
                0.00    0.00       1/243         XLogFileInit [175]
                0.00    0.00       1/243         XLogFileOpen [176]
                0.00    0.00       1/243         ReadControlFile [156]
                0.00    0.00       1/243         UpdateControlFile [172]
                0.00    0.00     238/243         _mdfd_blind_getseg [4052]
[14]     0.0    0.00    0.00     243         BasicOpenFile [14]
-----------------------------------------------
                0.00    0.00       1/243         crypt_openpwdfile [184]
                0.00    0.00       1/243         load_ident [192]
                0.00    0.00       1/243         LockMethodTableInit [148]
                0.00    0.00       2/243         load_hba [190]
                0.00    0.00     238/243         _mdfd_blind_getseg [4052]
[15]     0.0    0.00    0.00     243         pfree [15]
                0.00    0.00     243/243         AllocSetFree [13]
-----------------------------------------------
                0.00    0.00       1/239         CreateCheckPoint [108]
                0.00    0.00     238/239         BufferSync [100]
[16]     0.0    0.00    0.00     239         XLogFlush [16]
                0.00    0.00       2/2002        LWLockRelease [2]
                0.00    0.00       1/2           LWLockConditionalAcquire [88]
                0.00    0.00       1/2000        LWLockAcquire [3]
                0.00    0.00       1/1           XLogWrite [178]
-----------------------------------------------
                0.00    0.00     238/238         UnpinBuffer [22]
[17]     0.0    0.00    0.00     238         AddBufferToFreelist [17]
-----------------------------------------------
                0.00    0.00     238/238         BufferSync [100]
[18]     0.0    0.00    0.00     238         PinBuffer [18]
-----------------------------------------------
                0.00    0.00     238/238         BufferSync [100]
[19]     0.0    0.00    0.00     238         RelationNodeCacheGetRelation [19]
                0.00    0.00     238/250         hash_search [11]
-----------------------------------------------
                0.00    0.00     238/238         BufferSync [100]
[20]     0.0    0.00    0.00     238         StartBufferIO [20]
                0.00    0.00     238/2000        LWLockAcquire [3]
-----------------------------------------------
                0.00    0.00     238/238         BufferSync [100]
[21]     0.0    0.00    0.00     238         TerminateBufferIO [21]
                0.00    0.00     238/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00     238/238         FlushBufferPool [121]
[22]     0.0    0.00    0.00     238         UnpinBuffer [22]
                0.00    0.00     238/238         AddBufferToFreelist [17]
-----------------------------------------------
                0.00    0.00     238/238         smgrblindwrt [25]
[23]     0.0    0.00    0.00     238         mdblindwrt [23]
                0.00    0.00     238/238         _mdfd_blind_getseg [4052]
-----------------------------------------------
                0.00    0.00     238/238         _mdfd_blind_getseg [4052]
[24]     0.0    0.00    0.00     238         relpath [24]
                0.00    0.00     238/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00     238/238         BufferSync [100]
[25]     0.0    0.00    0.00     238         smgrblindwrt [25]
                0.00    0.00     238/238         mdblindwrt [23]
-----------------------------------------------
                0.00    0.00     238/238         hash_search [11]
[26]     0.0    0.00    0.00     238         tag_hash [26]
-----------------------------------------------
                0.00    0.00       2/234         ClosePostmasterPorts [106]
                0.00    0.00     232/234         ServerLoop [165]
[27]     0.0    0.00    0.00     234         StreamClose [27]
-----------------------------------------------
                0.00    0.00       1/233         SSDataBase [78]
                0.00    0.00     232/233         BackendStartup [30]
[28]     0.0    0.00    0.00     233         DLAddHead [28]
-----------------------------------------------
                0.00    0.00       1/233         SSDataBase [78]
                0.00    0.00     232/233         BackendStartup [30]
[29]     0.0    0.00    0.00     233         DLNewElem [29]
-----------------------------------------------
                0.00    0.00     232/232         ServerLoop [165]
[30]     0.0    0.00    0.00     232         BackendStartup [30]
                0.00    0.00     232/1161        PostmasterRandom [4]
                0.00    0.00     232/233         DLNewElem [29]
                0.00    0.00     232/233         DLAddHead [28]
-----------------------------------------------
                0.00    0.00     232/232         ServerLoop [165]
[31]     0.0    0.00    0.00     232         ConnCreate [31]
                0.00    0.00     464/464         CharRemap [7]
                0.00    0.00     232/232         StreamConnection [34]
                0.00    0.00     232/232         RandomSalt [33]
-----------------------------------------------
                0.00    0.00     232/232         ServerLoop [165]
[32]     0.0    0.00    0.00     232         ConnFree [32]
-----------------------------------------------
                0.00    0.00     232/232         ConnCreate [31]
[33]     0.0    0.00    0.00     232         RandomSalt [33]
                0.00    0.00     928/1161        PostmasterRandom [4]
-----------------------------------------------
                0.00    0.00     232/232         ConnCreate [31]
[34]     0.0    0.00    0.00     232         StreamConnection [34]
-----------------------------------------------
                0.00    0.00      35/205         load_ident [192]
                0.00    0.00     170/205         load_hba [190]
[35]     0.0    0.00    0.00     205         read_to_eol [35]
-----------------------------------------------
                0.00    0.00     187/187         ProcessConfigFile [155]
[36]     0.0    0.00    0.00     187         GUC_yylex [36]
                0.00    0.00       2/2           GUC_yy_get_next_buffer [85]
                0.00    0.00       1/1           GUC_yy_create_buffer [123]
                0.00    0.00       1/3           GUC_yy_load_buffer_state [75]
                0.00    0.00       1/1           GUC_yyrestart [125]
                0.00    0.00       1/1           GUC_yywrap [126]
                0.00    0.00       1/1           GUC_yy_get_previous_state [124]
-----------------------------------------------
                0.00    0.00     135/135         reaper [3444]
[37]     0.0    0.00    0.00     135         pgstat_ispgstat [37]
-----------------------------------------------
                0.00    0.00     134/134         reaper [3444]
[38]     0.0    0.00    0.00     134         CleanupProc [38]
                0.00    0.00     134/134         DLRemove [40]
                0.00    0.00     134/134         DLFreeElem [39]
-----------------------------------------------
                0.00    0.00     134/134         CleanupProc [38]
[39]     0.0    0.00    0.00     134         DLFreeElem [39]
-----------------------------------------------
                0.00    0.00     134/134         CleanupProc [38]
[40]     0.0    0.00    0.00     134         DLRemove [40]
-----------------------------------------------
                0.00    0.00     133/133         reaper [3444]
[41]     0.0    0.00    0.00     133         pgstat_beterm [41]
                0.00    0.00     133/133         pgstat_send [42]
-----------------------------------------------
                0.00    0.00     133/133         pgstat_beterm [41]
[42]     0.0    0.00    0.00     133         pgstat_send [42]
-----------------------------------------------
                0.00    0.00       1/59          CLOGShmemInit [102]
                0.00    0.00       1/59          reset_shared [206]
                0.00    0.00       1/59          LockMethodTableInit [148]
                0.00    0.00       1/59          InitProcGlobal [140]
                0.00    0.00       1/59          hash_search [11]
                0.00    0.00       2/59          XLOGShmemInit [173]
                0.00    0.00       2/59          InitBufferPool [131]
                0.00    0.00       5/59          ShmemInitHash [67]
                0.00    0.00       7/59          CreateSharedMemoryAndSemaphores [114]
                0.00    0.00      38/59          init_htab [64]
[43]     0.0    0.00    0.00      59         ShmemAlloc [43]
-----------------------------------------------
                0.00    0.00      44/44          init_htab [64]
[44]     0.0    0.00    0.00      44         seg_alloc [44]
                0.00    0.00       6/28          DynaHashAlloc [49]
-----------------------------------------------
                0.00    0.00      10/36          hash_estimate_size [68]
                0.00    0.00      10/36          hash_select_dirsize [69]
                0.00    0.00      16/36          init_htab [64]
[45]     0.0    0.00    0.00      36         my_log2 [45]
-----------------------------------------------
                0.00    0.00      33/33          IpcSemaphoreCreate [47]
[46]     0.0    0.00    0.00      33         InternalIpcSemaphoreCreate [46]
-----------------------------------------------
                0.00    0.00      33/33          InitProcGlobal [140]
[47]     0.0    0.00    0.00      33         IpcSemaphoreCreate [47]
                0.00    0.00      33/33          InternalIpcSemaphoreCreate [46]
                0.00    0.00      33/33          IpcSemaphoreUnlock [48]
-----------------------------------------------
                0.00    0.00      33/33          IpcSemaphoreCreate [47]
[48]     0.0    0.00    0.00      33         IpcSemaphoreUnlock [48]
-----------------------------------------------
                0.00    0.00       3/28          init_htab [64]
                0.00    0.00       6/28          seg_alloc [44]
                0.00    0.00      19/28          hash_create [62]
[49]     0.0    0.00    0.00      28         DynaHashAlloc [49]
                0.00    0.00      28/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00      11/24          PostmasterMain [153]
                0.00    0.00      13/24          BootstrapMain [98]
[50]     0.0    0.00    0.00      24         pqsignal [50]
-----------------------------------------------
                0.00    0.00       3/19          lconsi [79]
                0.00    0.00      16/19          lcons [53]
[51]     0.0    0.00    0.00      19         newNode [51]
                0.00    0.00      19/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00      16/16          tokenize_file [95]
[52]     0.0    0.00    0.00      16         lappend [52]
                0.00    0.00      16/16          lcons [53]
-----------------------------------------------
                0.00    0.00      16/16          lappend [52]
[53]     0.0    0.00    0.00      16         lcons [53]
                0.00    0.00      16/19          newNode [51]
-----------------------------------------------
                0.00    0.00      16/16          tokenize_file [95]
[54]     0.0    0.00    0.00      16         nconc [54]
-----------------------------------------------
                0.00    0.00       1/14          PostmasterMain [153]
                0.00    0.00       1/14          mdinit [195]
                0.00    0.00       1/14          CreateCacheMemoryContext [107]
                0.00    0.00       2/14          MemoryContextInit [151]
                0.00    0.00       9/14          hash_create [62]
[55]     0.0    0.00    0.00      14         AllocSetContextCreate [55]
                0.00    0.00      14/14          MemoryContextCreate [57]
-----------------------------------------------
                0.00    0.00      14/14          MemoryContextCreate [57]
[56]     0.0    0.00    0.00      14         AllocSetInit [56]
-----------------------------------------------
                0.00    0.00      14/14          AllocSetContextCreate [55]
[57]     0.0    0.00    0.00      14         MemoryContextCreate [57]
                0.00    0.00      14/14          AllocSetInit [56]
                0.00    0.00      13/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00      13/13          tokenize_file [95]
[58]     0.0    0.00    0.00      13         MemoryContextStrdup [58]
                0.00    0.00      13/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00       1/12          CLOGShmemInit [102]
                0.00    0.00       1/12          LockMethodTableInit [148]
                0.00    0.00       1/12          InitProcGlobal [140]
                0.00    0.00       2/12          XLOGShmemInit [173]
                0.00    0.00       2/12          InitBufferPool [131]
                0.00    0.00       5/12          ShmemInitHash [67]
[59]     0.0    0.00    0.00      12         ShmemInitStruct [59]
                0.00    0.00      12/2000        LWLockAcquire [3]
                0.00    0.00      11/250         hash_search [11]
                0.00    0.00      11/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00      12/12          hash_search [11]
[60]     0.0    0.00    0.00      12         string_hash [60]
-----------------------------------------------
                0.00    0.00       8/8           CheckPointCLOG [104]
[61]     0.0    0.00    0.00       8         WriteCLOGPage [61]
                0.00    0.00       2/2000        LWLockAcquire [3]
                0.00    0.00       1/2002        LWLockRelease [2]
                0.00    0.00       1/1           CLOGPhysicalWritePage [101]
-----------------------------------------------
                0.00    0.00       3/8           CreateDummyCaches [110]
                0.00    0.00       5/8           ShmemInitHash [67]
[62]     0.0    0.00    0.00       8         hash_create [62]
                0.00    0.00      19/28          DynaHashAlloc [49]
                0.00    0.00       9/14          AllocSetContextCreate [55]
                0.00    0.00       8/8           init_htab [64]
-----------------------------------------------
                0.00    0.00       3/8           CreateDummyCaches [110]
                0.00    0.00       5/8           ShmemInitHash [67]
[63]     0.0    0.00    0.00       8         hdefault [63]
-----------------------------------------------
                0.00    0.00       8/8           hash_create [62]
[64]     0.0    0.00    0.00       8         init_htab [64]
                0.00    0.00      44/44          seg_alloc [44]
                0.00    0.00      38/59          ShmemAlloc [43]
                0.00    0.00      16/36          my_log2 [45]
                0.00    0.00       3/28          DynaHashAlloc [49]
-----------------------------------------------
                0.00    0.00       1/7           crypt_openpwdfile [184]
                0.00    0.00       1/7           load_hba [190]
                0.00    0.00       1/7           load_ident [192]
                0.00    0.00       1/7           checkDataDir [182]
                0.00    0.00       1/7           ProcessConfigFile [155]
                0.00    0.00       2/7           ValidatePgVersion [93]
[65]     0.0    0.00    0.00       7         AllocateFile [65]
-----------------------------------------------
                0.00    0.00       1/6           load_hba [190]
                0.00    0.00       1/6           load_ident [192]
                0.00    0.00       1/6           checkDataDir [182]
                0.00    0.00       1/6           ProcessConfigFile [155]
                0.00    0.00       2/6           ValidatePgVersion [93]
[66]     0.0    0.00    0.00       6         FreeFile [66]
-----------------------------------------------
                0.00    0.00       1/5           InitBufTable [130]
                0.00    0.00       1/5           InitFreeSpaceMap [136]
                0.00    0.00       1/5           InitShmemIndex [142]
                0.00    0.00       2/5           LockMethodTableInit [148]
[67]     0.0    0.00    0.00       5         ShmemInitHash [67]
                0.00    0.00       5/5           hash_select_dirsize [69]
                0.00    0.00       5/59          ShmemAlloc [43]
                0.00    0.00       5/12          ShmemInitStruct [59]
                0.00    0.00       5/8           hdefault [63]
                0.00    0.00       5/8           hash_create [62]
-----------------------------------------------
                0.00    0.00       1/5           FreeSpaceShmemSize [122]
                0.00    0.00       2/5           BufferShmemSize [99]
                0.00    0.00       2/5           LockShmemSize [150]
[68]     0.0    0.00    0.00       5         hash_estimate_size [68]
                0.00    0.00      10/36          my_log2 [45]
-----------------------------------------------
                0.00    0.00       5/5           ShmemInitHash [67]
[69]     0.0    0.00    0.00       5         hash_select_dirsize [69]
                0.00    0.00      10/36          my_log2 [45]
-----------------------------------------------
                0.00    0.00       1/5           InitBufferPoolAccess [132]
                0.00    0.00       1/5           InitProcGlobal [140]
                0.00    0.00       1/5           InitDummyProcess [134]
                0.00    0.00       2/5           InternalIpcMemoryCreate [143]
[70]     0.0    0.00    0.00       5         on_shmem_exit [70]
-----------------------------------------------
                0.00    0.00       1/4           BootstrapMain [98]
                0.00    0.00       3/4           PostmasterMain [153]
[71]     0.0    0.00    0.00       4         SetConfigOption [71]
                0.00    0.00       4/4           set_config_option [74]
-----------------------------------------------
                0.00    0.00       4/4           set_config_option [74]
[72]     0.0    0.00    0.00       4         find_option [72]
-----------------------------------------------
                0.00    0.00       1/4           StreamServerPort [91]
                0.00    0.00       1/4           smgrinit [211]
                0.00    0.00       2/4           CreateLockFile [81]
[73]     0.0    0.00    0.00       4         on_proc_exit [73]
-----------------------------------------------
                0.00    0.00       4/4           SetConfigOption [71]
[74]     0.0    0.00    0.00       4         set_config_option [74]
                0.00    0.00       4/4           find_option [72]
                0.00    0.00       3/3           parse_int [80]
                0.00    0.00       1/1           parse_bool [198]
-----------------------------------------------
                0.00    0.00       1/3           GUC_yylex [36]
                0.00    0.00       1/3           GUC_yy_get_next_buffer [85]
                0.00    0.00       1/3           GUC_yy_flush_buffer [84]
[75]     0.0    0.00    0.00       3         GUC_yy_load_buffer_state [75]
-----------------------------------------------
                0.00    0.00       1/3           PostmasterMain [153]
                0.00    0.00       2/3           CreateDummyCaches [110]
[76]     0.0    0.00    0.00       3         MemoryContextSwitchTo [76]
-----------------------------------------------
                0.00    0.00       1/3           CreateLWLocks [111]
                0.00    0.00       2/3           LWLockShmemSize [89]
[77]     0.0    0.00    0.00       3         NumLWLocks [77]
-----------------------------------------------
                0.00    0.00       1/3           PostmasterMain [153]
                0.00    0.00       2/3           ServerLoop [165]
[78]     0.0    0.00    0.00       3         SSDataBase [78]
                0.00    0.00       1/1           on_exit_reset [197]
                0.00    0.00       1/1           pgstat_close_sockets [201]
                0.00    0.00       1/1           set_ps_display [208]
                0.00    0.00       1/1           BootstrapMain [98]
                0.00    0.00       1/1161        PostmasterRandom [4]
                0.00    0.00       1/233         DLNewElem [29]
                0.00    0.00       1/233         DLAddHead [28]
-----------------------------------------------
                0.00    0.00       3/3           tokenize_file [95]
[79]     0.0    0.00    0.00       3         lconsi [79]
                0.00    0.00       3/19          newNode [51]
-----------------------------------------------
                0.00    0.00       3/3           set_config_option [74]
[80]     0.0    0.00    0.00       3         parse_int [80]
-----------------------------------------------
                0.00    0.00       1/2           CreateDataDirLockFile [109]
                0.00    0.00       1/2           CreateSocketLockFile [115]
[81]     0.0    0.00    0.00       2         CreateLockFile [81]
                0.00    0.00       2/4           on_proc_exit [73]
-----------------------------------------------
                0.00    0.00       1/2           PostmasterMain [153]
                0.00    0.00       1/2           CreateOptsFile [112]
[82]     0.0    0.00    0.00       2         FindExec [82]
                0.00    0.00       2/2           ValidateBinary [92]
-----------------------------------------------
                0.00    0.00       2/2           GUC_yy_create_buffer [123]
[83]     0.0    0.00    0.00       2         GUC_yy_flex_alloc [83]
-----------------------------------------------
                0.00    0.00       2/2           GUC_yy_init_buffer [86]
[84]     0.0    0.00    0.00       2         GUC_yy_flush_buffer [84]
                0.00    0.00       1/3           GUC_yy_load_buffer_state [75]
-----------------------------------------------
                0.00    0.00       2/2           GUC_yylex [36]
[85]     0.0    0.00    0.00       2         GUC_yy_get_next_buffer [85]
                0.00    0.00       1/3           GUC_yy_load_buffer_state [75]
-----------------------------------------------
                0.00    0.00       1/2           GUC_yyrestart [125]
                0.00    0.00       1/2           GUC_yy_create_buffer [123]
[86]     0.0    0.00    0.00       2         GUC_yy_init_buffer [86]
                0.00    0.00       2/2           GUC_yy_flush_buffer [84]
-----------------------------------------------
                0.00    0.00       1/2           BootstrapMain [98]
                0.00    0.00       1/2           PostmasterMain [153]
[87]     0.0    0.00    0.00       2         IgnoreSystemIndexes [87]
-----------------------------------------------
                0.00    0.00       1/2           XLogFlush [16]
                0.00    0.00       1/2           CreateCheckPoint [108]
[88]     0.0    0.00    0.00       2         LWLockConditionalAcquire [88]
-----------------------------------------------
                0.00    0.00       1/2           CreateSharedMemoryAndSemaphores [114]
                0.00    0.00       1/2           CreateLWLocks [111]
[89]     0.0    0.00    0.00       2         LWLockShmemSize [89]
                0.00    0.00       2/3           NumLWLocks [77]
-----------------------------------------------
                0.00    0.00       1/2           CreateSharedMemoryAndSemaphores [114]
                0.00    0.00       1/2           SIBufferInit [164]
[90]     0.0    0.00    0.00       2         SInvalShmemSize [90]
-----------------------------------------------
                0.00    0.00       2/2           PostmasterMain [153]
[91]     0.0    0.00    0.00       2         StreamServerPort [91]
                0.00    0.00       1/1           CreateSocketLockFile [115]
                0.00    0.00       1/4           on_proc_exit [73]
-----------------------------------------------
                0.00    0.00       2/2           FindExec [82]
[92]     0.0    0.00    0.00       2         ValidateBinary [92]
-----------------------------------------------
                0.00    0.00       1/2           BootstrapMain [98]
                0.00    0.00       1/2           checkDataDir [182]
[93]     0.0    0.00    0.00       2         ValidatePgVersion [93]
                0.00    0.00       2/7           AllocateFile [65]
                0.00    0.00       2/6           FreeFile [66]
-----------------------------------------------
                0.00    0.00       1/2           BootstrapMain [98]
                0.00    0.00       1/2           PostmasterMain [153]
[94]     0.0    0.00    0.00       2         XLOGPathInit [94]
-----------------------------------------------
                0.00    0.00       1/2           load_hba [190]
                0.00    0.00       1/2           load_ident [192]
[95]     0.0    0.00    0.00       2         tokenize_file [95]
                0.00    0.00     249/249         next_token [12]
                0.00    0.00      16/16          nconc [54]
                0.00    0.00      16/16          lappend [52]
                0.00    0.00      13/13          MemoryContextStrdup [58]
                0.00    0.00       3/3           lconsi [79]
-----------------------------------------------
                0.00    0.00       1/1           shmem_exit [209]
[96]     0.0    0.00    0.00       1         AbortBufferIO [96]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[97]     0.0    0.00    0.00       1         BaseInit [97]
                0.00    0.00       1/1           InitCommunication [133]
                0.00    0.00       1/1           DebugFileOpen [118]
                0.00    0.00       1/1           smgrinit [211]
                0.00    0.00       1/1           InitBufferPoolAccess [132]
-----------------------------------------------
                0.00    0.00       1/1           SSDataBase [78]
[98]     0.0    0.00    0.00       1         BootstrapMain [98]
                0.00    0.00      13/24          pqsignal [50]
                0.00    0.00       1/4           SetConfigOption [71]
                0.00    0.00       1/2           ValidatePgVersion [93]
                0.00    0.00       1/2           IgnoreSystemIndexes [87]
                0.00    0.00       1/2           XLOGPathInit [94]
                0.00    0.00       1/1           InitLocalBuffer [137]
                0.00    0.00       1/1           BaseInit [97]
                0.00    0.00       1/1           ZeroProcSemaphore [179]
                0.00    0.00       1/1           InitDummyProcess [134]
                0.00    0.00       1/1           CreateDummyCaches [110]
                0.00    0.00       1/1           CreateCheckPoint [108]
                0.00    0.00       1/1           SetRedoRecPtr [167]
                0.00    0.00       1/1           proc_exit [205]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[99]     0.0    0.00    0.00       1         BufferShmemSize [99]
                0.00    0.00       2/5           hash_estimate_size [68]
-----------------------------------------------
                0.00    0.00       1/1           FlushBufferPool [121]
[100]    0.0    0.00    0.00       1         BufferSync [100]
                0.00    0.00    1500/2000        LWLockAcquire [3]
                0.00    0.00    1500/2002        LWLockRelease [2]
                0.00    0.00     238/238         PinBuffer [18]
                0.00    0.00     238/238         StartBufferIO [20]
                0.00    0.00     238/238         RelationNodeCacheGetRelation [19]
                0.00    0.00     238/239         XLogFlush [16]
                0.00    0.00     238/238         smgrblindwrt [25]
                0.00    0.00     238/238         TerminateBufferIO [21]
-----------------------------------------------
                0.00    0.00       1/1           WriteCLOGPage [61]
[101]    0.0    0.00    0.00       1         CLOGPhysicalWritePage [101]
                0.00    0.00       1/243         BasicOpenFile [14]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[102]    0.0    0.00    0.00       1         CLOGShmemInit [102]
                0.00    0.00       8/2056        LWLockAssign [1]
                0.00    0.00       1/59          ShmemAlloc [43]
                0.00    0.00       1/12          ShmemInitStruct [59]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[103]    0.0    0.00    0.00       1         CLOGShmemSize [103]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[104]    0.0    0.00    0.00       1         CheckPointCLOG [104]
                0.00    0.00       8/8           WriteCLOGPage [61]
                0.00    0.00       2/2002        LWLockRelease [2]
                0.00    0.00       1/2000        LWLockAcquire [3]
-----------------------------------------------
                0.00    0.00       1/1           ResetAllOptions [159]
[105]    0.0    0.00    0.00       1         ClearDateCache [105]
-----------------------------------------------
                0.00    0.00       1/1           ServerLoop [165]
[106]    0.0    0.00    0.00       1         ClosePostmasterPorts [106]
                0.00    0.00       2/234         StreamClose [27]
-----------------------------------------------
                0.00    0.00       1/1           CreateDummyCaches [110]
[107]    0.0    0.00    0.00       1         CreateCacheMemoryContext [107]
                0.00    0.00       1/14          AllocSetContextCreate [55]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[108]    0.0    0.00    0.00       1         CreateCheckPoint [108]
                0.00    0.00       5/2002        LWLockRelease [2]
                0.00    0.00       4/2000        LWLockAcquire [3]
                0.00    0.00       1/2           LWLockConditionalAcquire [88]
                0.00    0.00       1/1           GetUndoRecPtr [129]
                0.00    0.00       1/1           smgrsync [213]
                0.00    0.00       1/1           FlushBufferPool [121]
                0.00    0.00       1/1           CheckPointCLOG [104]
                0.00    0.00       1/1           XLogInsert [177]
                0.00    0.00       1/239         XLogFlush [16]
                0.00    0.00       1/1           UpdateControlFile [172]
                0.00    0.00       1/1           PreallocXlogFiles [154]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[109]    0.0    0.00    0.00       1         CreateDataDirLockFile [109]
                0.00    0.00       1/2           CreateLockFile [81]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[110]    0.0    0.00    0.00       1         CreateDummyCaches [110]
                0.00    0.00       3/8           hdefault [63]
                0.00    0.00       3/8           hash_create [62]
                0.00    0.00       2/3           MemoryContextSwitchTo [76]
                0.00    0.00       1/1           CreateCacheMemoryContext [107]
-----------------------------------------------
                0.00    0.00       1/1           reset_shared [206]
[111]    0.0    0.00    0.00       1         CreateLWLocks [111]
                0.00    0.00       1/3           NumLWLocks [77]
                0.00    0.00       1/2           LWLockShmemSize [89]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[112]    0.0    0.00    0.00       1         CreateOptsFile [112]
                0.00    0.00       1/2           FindExec [82]
                0.00    0.00       1/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[113]    0.0    0.00    0.00       1         CreateSharedInvalidationState [113]
-----------------------------------------------
                0.00    0.00       1/1           reset_shared [206]
[114]    0.0    0.00    0.00       1         CreateSharedMemoryAndSemaphores [114]
                0.00    0.00       7/59          ShmemAlloc [43]
                0.00    0.00       1/1           BufferShmemSize [99]
                0.00    0.00       1/1           LockShmemSize [150]
                0.00    0.00       1/1           XLOGShmemSize [174]
                0.00    0.00       1/1           CLOGShmemSize [103]
                0.00    0.00       1/2           LWLockShmemSize [89]
                0.00    0.00       1/2           SInvalShmemSize [90]
                0.00    0.00       1/1           FreeSpaceShmemSize [122]
                0.00    0.00       1/1           IpcMemoryCreate [145]
                0.00    0.00       1/1           CreateSpinlocks [116]
                0.00    0.00       1/1           InitShmemAllocation [141]
                0.00    0.00       1/1           InitShmemIndex [142]
                0.00    0.00       1/1           CLOGShmemInit [102]
                0.00    0.00       1/1           InitBufferPool [131]
                0.00    0.00       1/1           InitLocks [139]
                0.00    0.00       1/1           InitLockTable [138]
                0.00    0.00       1/1           InitProcGlobal [140]
                0.00    0.00       1/1           SIBufferInit [164]
                0.00    0.00       1/1           CreateSharedInvalidationState [113]
                0.00    0.00       1/1           InitFreeSpaceMap [136]
-----------------------------------------------
                0.00    0.00       1/1           StreamServerPort [91]
[115]    0.0    0.00    0.00       1         CreateSocketLockFile [115]
                0.00    0.00       1/2           CreateLockFile [81]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[116]    0.0    0.00    0.00       1         CreateSpinlocks [116]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[117]    0.0    0.00    0.00       1         DLNewList [117]
-----------------------------------------------
                0.00    0.00       1/1           BaseInit [97]
[118]    0.0    0.00    0.00       1         DebugFileOpen [118]
-----------------------------------------------
                0.00    0.00       1/1           shmem_exit [209]
[119]    0.0    0.00    0.00       1         DummyProcKill [119]
                0.00    0.00       1/1           LWLockReleaseAll [146]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[120]    0.0    0.00    0.00       1         EnableExceptionHandling [120]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[121]    0.0    0.00    0.00       1         FlushBufferPool [121]
                0.00    0.00     476/476         LockBuffer [6]
                0.00    0.00     238/238         UnpinBuffer [22]
                0.00    0.00       1/1           BufferSync [100]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[122]    0.0    0.00    0.00       1         FreeSpaceShmemSize [122]
                0.00    0.00       1/5           hash_estimate_size [68]
-----------------------------------------------
                0.00    0.00       1/1           GUC_yylex [36]
[123]    0.0    0.00    0.00       1         GUC_yy_create_buffer [123]
                0.00    0.00       2/2           GUC_yy_flex_alloc [83]
                0.00    0.00       1/2           GUC_yy_init_buffer [86]
-----------------------------------------------
                0.00    0.00       1/1           GUC_yylex [36]
[124]    0.0    0.00    0.00       1         GUC_yy_get_previous_state [124]
-----------------------------------------------
                0.00    0.00       1/1           GUC_yylex [36]
[125]    0.0    0.00    0.00       1         GUC_yyrestart [125]
                0.00    0.00       1/2           GUC_yy_init_buffer [86]
-----------------------------------------------
                0.00    0.00       1/1           GUC_yylex [36]
[126]    0.0    0.00    0.00       1         GUC_yywrap [126]
-----------------------------------------------
                0.00    0.00       1/1           XLogInsert [177]
[127]    0.0    0.00    0.00       1         GetCurrentTransactionId [127]
-----------------------------------------------
                0.00    0.00       1/1           reaper [3444]
[128]    0.0    0.00    0.00       1         GetRedoRecPtr [128]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[129]    0.0    0.00    0.00       1         GetUndoRecPtr [129]
                0.00    0.00       1/2000        LWLockAcquire [3]
                0.00    0.00       1/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00       1/1           InitBufferPool [131]
[130]    0.0    0.00    0.00       1         InitBufTable [130]
                0.00    0.00       1/5           ShmemInitHash [67]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[131]    0.0    0.00    0.00       1         InitBufferPool [131]
                0.00    0.00    2048/2056        LWLockAssign [1]
                0.00    0.00       2/59          ShmemAlloc [43]
                0.00    0.00       2/12          ShmemInitStruct [59]
                0.00    0.00       1/2000        LWLockAcquire [3]
                0.00    0.00       1/1           InitBufTable [130]
                0.00    0.00       1/1           InitFreeList [135]
                0.00    0.00       1/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00       1/1           BaseInit [97]
[132]    0.0    0.00    0.00       1         InitBufferPoolAccess [132]
                0.00    0.00       1/5           on_shmem_exit [70]
-----------------------------------------------
                0.00    0.00       1/1           BaseInit [97]
[133]    0.0    0.00    0.00       1         InitCommunication [133]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[134]    0.0    0.00    0.00       1         InitDummyProcess [134]
                0.00    0.00       1/1           SHMQueueElemInit [162]
                0.00    0.00       1/1           SHMQueueInit [163]
                0.00    0.00       1/5           on_shmem_exit [70]
-----------------------------------------------
                0.00    0.00       1/1           InitBufferPool [131]
[135]    0.0    0.00    0.00       1         InitFreeList [135]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[136]    0.0    0.00    0.00       1         InitFreeSpaceMap [136]
                0.00    0.00       1/5           ShmemInitHash [67]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[137]    0.0    0.00    0.00       1         InitLocalBuffer [137]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[138]    0.0    0.00    0.00       1         InitLockTable [138]
                0.00    0.00       1/1           LockMethodTableInit [148]
                0.00    0.00       1/1           LockMethodTableRename [149]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[139]    0.0    0.00    0.00       1         InitLocks [139]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[140]    0.0    0.00    0.00       1         InitProcGlobal [140]
                0.00    0.00      33/33          IpcSemaphoreCreate [47]
                0.00    0.00       1/59          ShmemAlloc [43]
                0.00    0.00       1/12          ShmemInitStruct [59]
                0.00    0.00       1/5           on_shmem_exit [70]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[141]    0.0    0.00    0.00       1         InitShmemAllocation [141]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[142]    0.0    0.00    0.00       1         InitShmemIndex [142]
                0.00    0.00       1/5           ShmemInitHash [67]
                0.00    0.00       1/250         hash_search [11]
                0.00    0.00       1/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00       1/1           IpcMemoryCreate [145]
[143]    0.0    0.00    0.00       1         InternalIpcMemoryCreate [143]
                0.00    0.00       2/5           on_shmem_exit [70]
                0.00    0.00       1/1           RecordSharedMemoryInLockFile [157]
-----------------------------------------------
                0.00    0.00       1/1           reset_shared [206]
[144]    0.0    0.00    0.00       1         IpcInitKeyAssignment [144]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[145]    0.0    0.00    0.00       1         IpcMemoryCreate [145]
                0.00    0.00       1/1           InternalIpcMemoryCreate [143]
-----------------------------------------------
                0.00    0.00       1/1           DummyProcKill [119]
[146]    0.0    0.00    0.00       1         LWLockReleaseAll [146]
-----------------------------------------------
                0.00    0.00       1/1           LockMethodTableInit [148]
[147]    0.0    0.00    0.00       1         LockMethodInit [147]
-----------------------------------------------
                0.00    0.00       1/1           InitLockTable [138]
[148]    0.0    0.00    0.00       1         LockMethodTableInit [148]
                0.00    0.00       2/319         MemoryContextAlloc [9]
                0.00    0.00       2/5           ShmemInitHash [67]
                0.00    0.00       1/2000        LWLockAcquire [3]
                0.00    0.00       1/59          ShmemAlloc [43]
                0.00    0.00       1/12          ShmemInitStruct [59]
                0.00    0.00       1/1           LockMethodInit [147]
                0.00    0.00       1/2002        LWLockRelease [2]
                0.00    0.00       1/243         pfree [15]
-----------------------------------------------
                0.00    0.00       1/1           InitLockTable [138]
[149]    0.0    0.00    0.00       1         LockMethodTableRename [149]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[150]    0.0    0.00    0.00       1         LockShmemSize [150]
                0.00    0.00       2/5           hash_estimate_size [68]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[151]    0.0    0.00    0.00       1         MemoryContextInit [151]
                0.00    0.00       2/14          AllocSetContextCreate [55]
-----------------------------------------------
                0.00    0.00       1/1           reset_shared [206]
[152]    0.0    0.00    0.00       1         PMSignalInit [152]
-----------------------------------------------
                0.00    0.00       1/1           main [2822]
[153]    0.0    0.00    0.00       1         PostmasterMain [153]
                0.00    0.00      11/24          pqsignal [50]
                0.00    0.00       3/4           SetConfigOption [71]
                0.00    0.00       2/2           StreamServerPort [91]
                0.00    0.00       1/1           EnableExceptionHandling [120]
                0.00    0.00       1/1           MemoryContextInit [151]
                0.00    0.00       1/14          AllocSetContextCreate [55]
                0.00    0.00       1/3           MemoryContextSwitchTo [76]
                0.00    0.00       1/1           ResetAllOptions [159]
                0.00    0.00       1/1           checkDataDir [182]
                0.00    0.00       1/1           SetDataDir [166]
                0.00    0.00       1/1           free_name_value_list [186]
                0.00    0.00       1/1           ProcessConfigFile [155]
                0.00    0.00       1/2           IgnoreSystemIndexes [87]
                0.00    0.00       1/2           FindExec [82]
                0.00    0.00       1/1           CreateDataDirLockFile [109]
                0.00    0.00       1/1           RemovePgTempFiles [158]
                0.00    0.00       1/2           XLOGPathInit [94]
                0.00    0.00       1/1           reset_shared [206]
                0.00    0.00       1/1           DLNewList [117]
                0.00    0.00       1/1           CreateOptsFile [112]
                0.00    0.00       1/1           pqinitmask [204]
                0.00    0.00       1/1           load_ident [192]
                0.00    0.00       1/1           load_hba_and_ident [191]
                0.00    0.00       1/1           load_password_cache [193]
                0.00    0.00       1/3           SSDataBase [78]
                0.00    0.00       1/1           ServerLoop [165]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[154]    0.0    0.00    0.00       1         PreallocXlogFiles [154]
                0.00    0.00       1/1           XLogFileInit [175]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[155]    0.0    0.00    0.00       1         ProcessConfigFile [155]
                0.00    0.00     187/187         GUC_yylex [36]
                0.00    0.00       1/7           AllocateFile [65]
                0.00    0.00       1/6           FreeFile [66]
-----------------------------------------------
                0.00    0.00       1/1           XLOGShmemInit [173]
[156]    0.0    0.00    0.00       1         ReadControlFile [156]
                0.00    0.00       1/243         BasicOpenFile [14]
-----------------------------------------------
                0.00    0.00       1/1           InternalIpcMemoryCreate [143]
[157]    0.0    0.00    0.00       1         RecordSharedMemoryInLockFile [157]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[158]    0.0    0.00    0.00       1         RemovePgTempFiles [158]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[159]    0.0    0.00    0.00       1         ResetAllOptions [159]
                0.00    0.00       1/1           ClearDateCache [105]
                0.00    0.00       1/1           assign_xlog_sync_method [181]
                0.00    0.00       1/1           assign_defaultxactisolevel [180]
-----------------------------------------------
                0.00    0.00       1/1           ShutdownBufferPoolAccess [169]
[160]    0.0    0.00    0.00       1         ResetBufferPool [160]
                0.00    0.00       1/1           ResetLocalBufferPool [161]
-----------------------------------------------
                0.00    0.00       1/1           ResetBufferPool [160]
[161]    0.0    0.00    0.00       1         ResetLocalBufferPool [161]
-----------------------------------------------
                0.00    0.00       1/1           InitDummyProcess [134]
[162]    0.0    0.00    0.00       1         SHMQueueElemInit [162]
-----------------------------------------------
                0.00    0.00       1/1           InitDummyProcess [134]
[163]    0.0    0.00    0.00       1         SHMQueueInit [163]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[164]    0.0    0.00    0.00       1         SIBufferInit [164]
                0.00    0.00       1/2           SInvalShmemSize [90]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[165]    0.0    0.00    0.00       1         ServerLoop [165]
                0.00    0.00     232/232         ConnCreate [31]
                0.00    0.00     232/232         BackendStartup [30]
                0.00    0.00     232/234         StreamClose [27]
                0.00    0.00     232/232         ConnFree [32]
                0.00    0.00       2/3           SSDataBase [78]
                0.00    0.00       1/1           initMasks [187]
                0.00    0.00       1/1           init_ps_display [188]
                0.00    0.00       1/1           ClosePostmasterPorts [106]
                0.00    0.00       1/1           TouchSocketLockFile [170]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[166]    0.0    0.00    0.00       1         SetDataDir [166]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[167]    0.0    0.00    0.00       1         SetRedoRecPtr [167]
-----------------------------------------------
                0.00    0.00       1/1           reaper [3444]
[168]    0.0    0.00    0.00       1         SetThisStartUpID [168]
-----------------------------------------------
                0.00    0.00       1/1           shmem_exit [209]
[169]    0.0    0.00    0.00       1         ShutdownBufferPoolAccess [169]
                0.00    0.00       1/1           UnlockBuffers [171]
                0.00    0.00       1/1           smgrabort [210]
                0.00    0.00       1/1           ResetBufferPool [160]
-----------------------------------------------
                0.00    0.00       1/1           ServerLoop [165]
[170]    0.0    0.00    0.00       1         TouchSocketLockFile [170]
-----------------------------------------------
                0.00    0.00       1/1           ShutdownBufferPoolAccess [169]
[171]    0.0    0.00    0.00       1         UnlockBuffers [171]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[172]    0.0    0.00    0.00       1         UpdateControlFile [172]
                0.00    0.00       1/243         BasicOpenFile [14]
                0.00    0.00       1/1           pg_fsync [200]
-----------------------------------------------
                0.00    0.00       1/1           reset_shared [206]
[173]    0.0    0.00    0.00       1         XLOGShmemInit [173]
                0.00    0.00       2/59          ShmemAlloc [43]
                0.00    0.00       2/12          ShmemInitStruct [59]
                0.00    0.00       1/1           ReadControlFile [156]
-----------------------------------------------
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
[174]    0.0    0.00    0.00       1         XLOGShmemSize [174]
-----------------------------------------------
                0.00    0.00       1/1           PreallocXlogFiles [154]
[175]    0.0    0.00    0.00       1         XLogFileInit [175]
                0.00    0.00       1/243         BasicOpenFile [14]
-----------------------------------------------
                0.00    0.00       1/1           XLogWrite [178]
[176]    0.0    0.00    0.00       1         XLogFileOpen [176]
                0.00    0.00       1/243         BasicOpenFile [14]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[177]    0.0    0.00    0.00       1         XLogInsert [177]
                0.00    0.00       1/2000        LWLockAcquire [3]
                0.00    0.00       1/1           GetCurrentTransactionId [127]
                0.00    0.00       1/2002        LWLockRelease [2]
-----------------------------------------------
                0.00    0.00       1/1           XLogFlush [16]
[178]    0.0    0.00    0.00       1         XLogWrite [178]
                0.00    0.00       1/1           XLogFileOpen [176]
                0.00    0.00       1/1           issue_xlog_fsync [189]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[179]    0.0    0.00    0.00       1         ZeroProcSemaphore [179]
-----------------------------------------------
                0.00    0.00       1/1           ResetAllOptions [159]
[180]    0.0    0.00    0.00       1         assign_defaultxactisolevel [180]
-----------------------------------------------
                0.00    0.00       1/1           ResetAllOptions [159]
[181]    0.0    0.00    0.00       1         assign_xlog_sync_method [181]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[182]    0.0    0.00    0.00       1         checkDataDir [182]
                0.00    0.00       1/2           ValidatePgVersion [93]
                0.00    0.00       1/7           AllocateFile [65]
                0.00    0.00       1/6           FreeFile [66]
-----------------------------------------------
                0.00    0.00       1/1           crypt_openpwdfile [184]
[183]    0.0    0.00    0.00       1         crypt_getpwdfilename [183]
                0.00    0.00       1/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00       1/1           load_password_cache [193]
[184]    0.0    0.00    0.00       1         crypt_openpwdfile [184]
                0.00    0.00       1/1           crypt_getpwdfilename [183]
                0.00    0.00       1/7           AllocateFile [65]
                0.00    0.00       1/243         pfree [15]
-----------------------------------------------
                0.00    0.00       1/1           hash_search [11]
[185]    0.0    0.00    0.00       1         element_alloc [185]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[186]    0.0    0.00    0.00       1         free_name_value_list [186]
-----------------------------------------------
                0.00    0.00       1/1           ServerLoop [165]
[187]    0.0    0.00    0.00       1         initMasks [187]
-----------------------------------------------
                0.00    0.00       1/1           ServerLoop [165]
[188]    0.0    0.00    0.00       1         init_ps_display [188]
-----------------------------------------------
                0.00    0.00       1/1           XLogWrite [178]
[189]    0.0    0.00    0.00       1         issue_xlog_fsync [189]
                0.00    0.00       1/1           pg_fdatasync [199]
-----------------------------------------------
                0.00    0.00       1/1           load_hba_and_ident [191]
[190]    0.0    0.00    0.00       1         load_hba [190]
                0.00    0.00     170/205         read_to_eol [35]
                0.00    0.00       2/319         MemoryContextAlloc [9]
                0.00    0.00       2/243         pfree [15]
                0.00    0.00       1/7           AllocateFile [65]
                0.00    0.00       1/2           tokenize_file [95]
                0.00    0.00       1/6           FreeFile [66]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[191]    0.0    0.00    0.00       1         load_hba_and_ident [191]
                0.00    0.00       1/1           load_hba [190]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[192]    0.0    0.00    0.00       1         load_ident [192]
                0.00    0.00      35/205         read_to_eol [35]
                0.00    0.00       1/319         MemoryContextAlloc [9]
                0.00    0.00       1/7           AllocateFile [65]
                0.00    0.00       1/2           tokenize_file [95]
                0.00    0.00       1/6           FreeFile [66]
                0.00    0.00       1/243         pfree [15]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[193]    0.0    0.00    0.00       1         load_password_cache [193]
                0.00    0.00       1/1           crypt_openpwdfile [184]
-----------------------------------------------
                0.00    0.00       1/1           smgrabort [210]
[194]    0.0    0.00    0.00       1         mdabort [194]
-----------------------------------------------
                0.00    0.00       1/1           smgrinit [211]
[195]    0.0    0.00    0.00       1         mdinit [195]
                0.00    0.00       1/14          AllocSetContextCreate [55]
                0.00    0.00       1/319         MemoryContextAlloc [9]
-----------------------------------------------
                0.00    0.00       1/1           smgrsync [213]
[196]    0.0    0.00    0.00       1         mdsync [196]
-----------------------------------------------
                0.00    0.00       1/1           SSDataBase [78]
[197]    0.0    0.00    0.00       1         on_exit_reset [197]
-----------------------------------------------
                0.00    0.00       1/1           set_config_option [74]
[198]    0.0    0.00    0.00       1         parse_bool [198]
-----------------------------------------------
                0.00    0.00       1/1           issue_xlog_fsync [189]
[199]    0.0    0.00    0.00       1         pg_fdatasync [199]
-----------------------------------------------
                0.00    0.00       1/1           UpdateControlFile [172]
[200]    0.0    0.00    0.00       1         pg_fsync [200]
-----------------------------------------------
                0.00    0.00       1/1           SSDataBase [78]
[201]    0.0    0.00    0.00       1         pgstat_close_sockets [201]
-----------------------------------------------
                0.00    0.00       1/1           main [2822]
[202]    0.0    0.00    0.00       1         pgstat_init [202]
-----------------------------------------------
                0.00    0.00       1/1           main [2822]
[203]    0.0    0.00    0.00       1         pgstat_start [203]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[204]    0.0    0.00    0.00       1         pqinitmask [204]
-----------------------------------------------
                0.00    0.00       1/1           BootstrapMain [98]
[205]    0.0    0.00    0.00       1         proc_exit [205]
                0.00    0.00       1/1           shmem_exit [209]
                0.00    0.00       1/1           smgrshutdown [212]
-----------------------------------------------
                0.00    0.00       1/1           PostmasterMain [153]
[206]    0.0    0.00    0.00       1         reset_shared [206]
                0.00    0.00       1/1           IpcInitKeyAssignment [144]
                0.00    0.00       1/59          ShmemAlloc [43]
                0.00    0.00       1/1           PMSignalInit [152]
                0.00    0.00       1/1           XLOGShmemInit [173]
                0.00    0.00       1/1           CreateLWLocks [111]
                0.00    0.00       1/1           CreateSharedMemoryAndSemaphores [114]
-----------------------------------------------
                0.00    0.00       1/1           main [2822]
[207]    0.0    0.00    0.00       1         save_ps_display_args [207]
-----------------------------------------------
                0.00    0.00       1/1           SSDataBase [78]
[208]    0.0    0.00    0.00       1         set_ps_display [208]
-----------------------------------------------
                0.00    0.00       1/1           proc_exit [205]
[209]    0.0    0.00    0.00       1         shmem_exit [209]
                0.00    0.00       1/1           ShutdownBufferPoolAccess [169]
                0.00    0.00       1/1           AbortBufferIO [96]
                0.00    0.00       1/1           DummyProcKill [119]
-----------------------------------------------
                0.00    0.00       1/1           ShutdownBufferPoolAccess [169]
[210]    0.0    0.00    0.00       1         smgrabort [210]
                0.00    0.00       1/1           mdabort [194]
-----------------------------------------------
                0.00    0.00       1/1           BaseInit [97]
[211]    0.0    0.00    0.00       1         smgrinit [211]
                0.00    0.00       1/1           mdinit [195]
                0.00    0.00       1/4           on_proc_exit [73]
-----------------------------------------------
                0.00    0.00       1/1           proc_exit [205]
[212]    0.0    0.00    0.00       1         smgrshutdown [212]
-----------------------------------------------
                0.00    0.00       1/1           CreateCheckPoint [108]
[213]    0.0    0.00    0.00       1         smgrsync [213]
                0.00    0.00       1/1           mdsync [196]
-----------------------------------------------
                0.00    0.00     238/238         mdblindwrt [23]
[4052]   0.0    0.00    0.00     238         _mdfd_blind_getseg [4052]
                0.00    0.00     238/238         relpath [24]
                0.00    0.00     238/243         BasicOpenFile [14]
                0.00    0.00     238/243         pfree [15]
-----------------------------------------------

 This table describes the call tree of the program, and was sorted by
 the total amount of time spent in each function and its children.

 Each entry in this table consists of several lines.  The line with the
 index number at the left hand margin lists the current function.
 The lines above it list the functions that called this function,
 and the lines below it list the functions this one called.
 This line lists:
     index    A unique number given to each element of the table.
        Index numbers are sorted numerically.
        The index number is printed next to every function name so
        it is easier to look up where the function in the table.

     % time    This is the percentage of the `total' time that was spent
        in this function and its children.  Note that due to
        different viewpoints, functions excluded by options, etc,
        these numbers will NOT add up to 100%.

     self    This is the total amount of time spent in this function.

     children    This is the total amount of time propagated into this
        function by its children.

     called    This is the number of times the function was called.
        If the function called itself recursively, the number
        only includes non-recursive calls, and is followed by
        a `+' and the number of recursive calls.

     name    The name of the current function.  The index number is
        printed after it.  If the function is a member of a
        cycle, the cycle number is printed between the
        function's name and the index number.


 For the function's parents, the fields have the following meanings:

     self    This is the amount of time that was propagated directly
        from the function into this parent.

     children    This is the amount of time that was propagated from
        the function's children into this parent.

     called    This is the number of times this parent called the
        function `/' the total number of times the function
        was called.  Recursive calls to the function are not
        included in the number after the `/'.

     name    This is the name of the parent.  The parent's index
        number is printed after it.  If the parent is a
        member of a cycle, the cycle number is printed between
        the name and the index number.

 If the parents of the function cannot be determined, the word
 `<spontaneous>' is printed in the `name' field, and all the other
 fields are blank.

 For the function's children, the fields have the following meanings:

     self    This is the amount of time that was propagated directly
        from the child into the function.

     children    This is the amount of time that was propagated from the
        child's children to the function.

     called    This is the number of times the function called
        this child `/' the total number of times the child
        was called.  Recursive calls by the child are not
        listed in the number after the `/'.

     name    This is the name of the child.  The child's index
        number is printed after it.  If the child is a
        member of a cycle, the cycle number is printed
        between the name and the index number.

 If there are any cycles (circles) in the call graph, there is an
 entry for the cycle-as-a-whole.  This entry shows who called the
 cycle (as parents) and the members of the cycle (as children.)
 The `+' recursive calls entry shows the number of function calls that
 were internal to the cycle, and the calls entry for each member shows,
 for that member, how many times it was called from other members of
 the cycle.


Index by function name

  [96] AbortBufferIO         [141] InitShmemAllocation   [178] XLogWrite (gmon-start.c)
  [17] AddBufferToFreelist (gmon-start.c) [142] InitShmemIndex [179] ZeroProcSemaphore (gmon-start.c)
   [8] AllocSetAlloc (gmon-start.c) [143] InternalIpcMemoryCreate (gmon-start.c) [4052] _mdfd_blind_getseg
(gmon-start.c)
  [55] AllocSetContextCreate  [46] InternalIpcSemaphoreCreate (gmon-start.c) [180] assign_defaultxactisolevel
(gmon-start.c)
  [13] AllocSetFree (gmon-start.c) [144] IpcInitKeyAssignment [181] assign_xlog_sync_method
  [56] AllocSetInit (gmon-start.c) [145] IpcMemoryCreate  [10] call_hash (gmon-start.c)
  [65] AllocateFile           [47] IpcSemaphoreCreate    [182] checkDataDir (gmon-start.c)
  [30] BackendStartup (gmon-start.c) [48] IpcSemaphoreUnlock [183] crypt_getpwdfilename
  [97] BaseInit                [3] LWLockAcquire         [184] crypt_openpwdfile (gmon-start.c)
  [14] BasicOpenFile           [1] LWLockAssign          [185] element_alloc (gmon-start.c)
  [98] BootstrapMain          [88] LWLockConditionalAcquire [72] find_option (gmon-start.c)
  [99] BufferShmemSize         [2] LWLockRelease         [186] free_name_value_list (gmon-start.c)
 [100] BufferSync            [146] LWLockReleaseAll       [62] hash_create
 [101] CLOGPhysicalWritePage (gmon-start.c) [89] LWLockShmemSize [68] hash_estimate_size
 [102] CLOGShmemInit           [6] LockBuffer             [11] hash_search
 [103] CLOGShmemSize         [147] LockMethodInit (gmon-start.c) [69] hash_select_dirsize
   [7] CharRemap (gmon-start.c) [148] LockMethodTableInit [63] hdefault (gmon-start.c)
 [104] CheckPointCLOG        [149] LockMethodTableRename [187] initMasks (gmon-start.c)
  [38] CleanupProc (gmon-start.c) [150] LockShmemSize     [64] init_htab (gmon-start.c)
 [105] ClearDateCache          [9] MemoryContextAlloc    [188] init_ps_display
 [106] ClosePostmasterPorts   [57] MemoryContextCreate     [5] isblank (gmon-start.c)
  [31] ConnCreate (gmon-start.c) [151] MemoryContextInit [189] issue_xlog_fsync (gmon-start.c)
  [32] ConnFree (gmon-start.c) [58] MemoryContextStrdup   [52] lappend
 [107] CreateCacheMemoryContext [76] MemoryContextSwitchTo [53] lcons
 [108] CreateCheckPoint       [77] NumLWLocks             [79] lconsi
 [109] CreateDataDirLockFile [152] PMSignalInit          [190] load_hba (gmon-start.c)
 [110] CreateDummyCaches      [18] PinBuffer             [191] load_hba_and_ident
 [111] CreateLWLocks         [153] PostmasterMain        [192] load_ident (gmon-start.c)
  [81] CreateLockFile (gmon-start.c) [4] PostmasterRandom (gmon-start.c) [193] load_password_cache
 [112] CreateOptsFile (gmon-start.c) [154] PreallocXlogFiles (gmon-start.c) [194] mdabort
 [113] CreateSharedInvalidationState [155] ProcessConfigFile [23] mdblindwrt
 [114] CreateSharedMemoryAndSemaphores [33] RandomSalt (gmon-start.c) [195] mdinit
 [115] CreateSocketLockFile  [156] ReadControlFile (gmon-start.c) [196] mdsync
 [116] CreateSpinlocks       [157] RecordSharedMemoryInLockFile [45] my_log2
  [28] DLAddHead              [19] RelationNodeCacheGetRelation [54] nconc
  [39] DLFreeElem            [158] RemovePgTempFiles      [51] newNode
  [29] DLNewElem             [159] ResetAllOptions        [12] next_token (gmon-start.c)
 [117] DLNewList             [160] ResetBufferPool       [197] on_exit_reset
  [40] DLRemove              [161] ResetLocalBufferPool   [73] on_proc_exit
 [118] DebugFileOpen         [162] SHMQueueElemInit       [70] on_shmem_exit
 [119] DummyProcKill (gmon-start.c) [163] SHMQueueInit   [198] parse_bool (gmon-start.c)
  [49] DynaHashAlloc (gmon-start.c) [164] SIBufferInit    [80] parse_int (gmon-start.c)
 [120] EnableExceptionHandling [90] SInvalShmemSize       [15] pfree
  [82] FindExec               [78] SSDataBase (gmon-start.c) [199] pg_fdatasync
 [121] FlushBufferPool       [165] ServerLoop (gmon-start.c) [200] pg_fsync
  [66] FreeFile               [71] SetConfigOption        [41] pgstat_beterm
 [122] FreeSpaceShmemSize    [166] SetDataDir            [201] pgstat_close_sockets
 [123] GUC_yy_create_buffer  [167] SetRedoRecPtr         [202] pgstat_init
  [83] GUC_yy_flex_alloc (gmon-start.c) [168] SetThisStartUpID [37] pgstat_ispgstat
  [84] GUC_yy_flush_buffer    [43] ShmemAlloc             [42] pgstat_send (gmon-start.c)
  [85] GUC_yy_get_next_buffer (gmon-start.c) [67] ShmemInitHash [203] pgstat_start
 [124] GUC_yy_get_previous_state (gmon-start.c) [59] ShmemInitStruct [204] pqinitmask
  [86] GUC_yy_init_buffer    [169] ShutdownBufferPoolAccess (gmon-start.c) [50] pqsignal
  [75] GUC_yy_load_buffer_state [20] StartBufferIO (gmon-start.c) [205] proc_exit
  [36] GUC_yylex              [27] StreamClose            [35] read_to_eol (gmon-start.c)
 [125] GUC_yyrestart          [34] StreamConnection       [24] relpath
 [126] GUC_yywrap             [91] StreamServerPort      [206] reset_shared (gmon-start.c)
 [127] GetCurrentTransactionId [21] TerminateBufferIO (gmon-start.c) [207] save_ps_display_args
 [128] GetRedoRecPtr         [170] TouchSocketLockFile    [44] seg_alloc (gmon-start.c)
 [129] GetUndoRecPtr         [171] UnlockBuffers          [74] set_config_option
  [87] IgnoreSystemIndexes    [22] UnpinBuffer           [208] set_ps_display
 [130] InitBufTable          [172] UpdateControlFile     [209] shmem_exit
 [131] InitBufferPool         [92] ValidateBinary (gmon-start.c) [210] smgrabort
 [132] InitBufferPoolAccess   [93] ValidatePgVersion      [25] smgrblindwrt
 [133] InitCommunication (gmon-start.c) [61] WriteCLOGPage (gmon-start.c) [211] smgrinit
 [134] InitDummyProcess       [94] XLOGPathInit          [212] smgrshutdown (gmon-start.c)
 [135] InitFreeList          [173] XLOGShmemInit         [213] smgrsync
 [136] InitFreeSpaceMap      [174] XLOGShmemSize          [60] string_hash
 [137] InitLocalBuffer       [175] XLogFileInit (gmon-start.c) [26] tag_hash
 [138] InitLockTable         [176] XLogFileOpen (gmon-start.c) [95] tokenize_file (gmon-start.c)
 [139] InitLocks              [16] XLogFlush
 [140] InitProcGlobal        [177] XLogInsert

Re: 7.2 is slow?

От
Jussi Mikkola
Дата:
I haven't tested with the new 7.2 betas, but here are some results from
7.1.

We have a developement computer, IBM x series 250, with 4 processors
(PIII Xeon 750Mhz), 1 Gb memory and  2SCSI disks (u160).

The software is writing new rows to a table, and after this it reads the
id from that row. There are currently about 50 connections doing the
same thing.

When I run this test with the Redhat 7.1 SMP kernel, I noticed, that the
processors are more than 90% idle. Disks utilisation is not the
bottleneck either, since there is very low disk usage. Some data is
written to disks every 4-5 seconds. Fsync is turned of. In transactions,
this means about 200 inserted rows per second. The software that is used
to give the feed, is capable of several thousand rows per second.

Okey, so I tried this also with the same computer, but using the not SMP
supported kernel. So only with one processor. The result was about 600
rows per second. The configuration file was unchanged. Now, the
processor is about 100% utilized.

I didn't find any parameters that should help in this, but if you have a
version of 7.2 that you would like to get information about, let me
know, so I'll test.

Jussi





Re: 7.2 is slow?

От
Hannu Krosing
Дата:
Jussi Mikkola wrote:
> 
> Hi Tom!
> 
> Well, here's the profile, but yes, almost all the times are zero. Both without
> the patch, and with it. Did I miss something? (Yes, I did make and install
> afterwards ;-)
> 
> I made a new database, so as it was growing, the times were even slower than
> before. (It would be nice, if I could create a large database from the
> beginning.;)
> 
> Is this of any help? Do you need the same result with less cpus?
> 
> Jussi
> 
> Tom Lane wrote:
> 
> > Jussi Mikkola <jussi.mikkola@bonware.com> writes:
> > > Yes, now I've tested with 7.2b4. The result is about the  same as with 7.1.
> >
> > > About 200 messages with four processors and about 600 messages with one
> > > processor.
> >

Was this solved with latest LWLock patches ?

--------------
Hannu