Обсуждение: Shared memory estimation for postgres
I am a newbie to databases and Postgres and I am trying to analyze the shared memory being calculated and allocated by Postgres in the method "CreateSharedMemoryAndSemaphores" for different major versions for different postgres.conf file My idea was to create a utility in Postgres and calll out the methods like BufferShmemSize(), LockShmemSize() etc being used in the CreateSharedMemoryAndSemaphores() to find the value. Till now what I have done: I have created a new utility and am trying to link the src/backend code to it but I'm not able to get it working correctly. Is there any other interest idea / way where I use a postgres.conf file to calculate the above mentioned shared memory value ? I have found this discussion thread from the past but haven't made much sense to me in terms of where to look. Old discussion thread: http://postgresql.nabble.com/postgresql-conf-basic-analysis-tool-td1948070.html -- View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Do a web search on setting shared memory the hard way, and I think you'll see what you really need to do. -- Jay Sent from my iPad > On Nov 10, 2016, at 5:57 PM, leoaaryan <leoaaryan@gmail.com> wrote: > > I am a newbie to databases and Postgres and I am trying to analyze the shared > memory being calculated and allocated by Postgres in the method > "CreateSharedMemoryAndSemaphores" for different major versions for different > postgres.conf file > > My idea was to create a utility in Postgres and calll out the methods like > BufferShmemSize(), LockShmemSize() etc being used in the > CreateSharedMemoryAndSemaphores() to find the value. > > Till now what I have done: I have created a new utility and am trying to > link the src/backend code to it but I'm not able to get it working > correctly. > > Is there any other interest idea / way where I use a postgres.conf file to > calculate the above mentioned shared memory value ? > > I have found this discussion thread from the past but haven't made much > sense to me in terms of where to look. > Old discussion thread: > http://postgresql.nabble.com/postgresql-conf-basic-analysis-tool-td1948070.html > > > > -- > View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868.html > Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
Hi Jay, If you are talking about http://evol-monkey.blogspot.com/2013/08/setting-sharedbuffers-hard-way.html and the "pg_buffercache" extensions then yes I have gone through it. The easiest way to find the value for the shared memory computation is to change the logging level to DEBUG3 and start postgres DB engine and it will give the calculated value in the log file. I believe postgres as a DB needs to be running for any extension to run (I may be wrong here) and I dont want to start the database for this analysis. Please correct me if I'm wrong in my concepts or if I've not understood anything. -leoaaryan. -- View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929872.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
On Fri, Nov 11, 2016 at 8:40 AM, leoaaryan <leoaaryan@gmail.com> wrote: > The easiest way to find the value for the shared memory computation is to > change the logging level to DEBUG3 and start postgres DB engine and it will > give the calculated value in the log file. > > I believe postgres as a DB needs to be running for any extension to run (I > may be wrong here) and I dont want to start the database for this analysis. > > Please correct me if I'm wrong in my concepts or if I've not understood > anything. Some time ago a patch has been proposed to project to a system view the shared memory allocations currently happening on an instance: https://www.postgresql.org/message-id/20140504114417.GM12715@awork2.anarazel.de This could be plugged into its own extension rather easily. Note though that DSM segments (dynamic shared memory) do not have names associated to them which would be helpful if you'd like to track that as well. But perhaps you don't care much. -- Michael
On 11 November 2016 at 06:57, leoaaryan <leoaaryan@gmail.com> wrote: > I am a newbie to databases and Postgres and I am trying to analyze the shared > memory being calculated and allocated by Postgres in the method > "CreateSharedMemoryAndSemaphores" for different major versions for different > postgres.conf file Note that this follows on from a number of other posts, including http://stackoverflow.com/questions/39607940/is-it-possible-to-know-the-memory-being-allocated-by-the-method-createsharedmem http://stackoverflow.com/questions/40433784/is-it-possible-to-call-a-postgres-internal-method-from-a-util Please, PLEASE link to related prior discussion when you post somewhere. It's really annoying having to look it up or wasting people's time duplicating discussion that's already happened. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Hi Michael, I think the method "pg_get_shmem_allocations" mentioned in the patch will give the allocated shared memory when the postgres db server is running. I'm trying to get the same without running the server if possible. Please correct me if I have failed to understand the discussion thread contents. -- View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929886.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Hi Craig, Sorry for the multiple point of contact for the same question. I'll keep in mind to attach similar corresponding links in future if any. http://stackoverflow.com/questions/39607940/is-it-possible-to-know-the-memory-being-allocated-by-the-method-createsharedmem and http://stackoverflow.com/questions/40433784/is-it-possible-to-call-a-postgres-internal-method-from-a-util are the other two posts from my side on stackoverflow to understand more about the feasibility of the nature of task I'm trying to accomplish. -- View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929887.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
On Fri, Nov 11, 2016 at 1:26 PM, leoaaryan <leoaaryan@gmail.com> wrote: > I think the method "pg_get_shmem_allocations" mentioned in the patch will > give the allocated shared memory when the postgres db server is running. I'm > trying to get the same without running the server if possible. That's up to "read the code and create a formula based on the system parameter to calculate the amount" then. -- Michael
Hi Michael, Thanks for all the help and time. I have already developed a code where I can exactly calculate the to be allocated shared memory value based on the Postgres 9.5.4 code (i went through the code, found out the sizes and offset of all the structures used in the memory calculation process and then use the values from postgres.conf file to calculate the required value). But the problem is if there is any change in the structures or anything is newly added in the next major version, I need to look at the code again and see what changed and then modify the hardcoded values of the structure size. I'm trying to avoid that. -leoaaryan -- View this message in context: http://postgresql.nabble.com/Shared-memory-estimation-for-postgres-tp5929868p5929891.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
<p dir="ltr"><p dir="ltr">On 11 Nov. 2016 13:00, "leoaaryan" <<a href="mailto:leoaaryan@gmail.com">leoaaryan@gmail.com</a>>wrote:<br /> ><br /> > Hi Michael,<br /> ><br /> >Thanks for all the help and time. I have already developed a code where I<br /> > can exactly calculate the to beallocated shared memory value based on the<br /> > Postgres 9.5.4 code (i went through the code, found out the sizesand offset<br /> > of all the structures used in the memory calculation process and then use<br /> > the valuesfrom postgres.conf file to calculate the required value).<br /> ><br /> > But the problem is if there is anychange in the structures or anything is<br /> > newly added in the next major version, I need to look at the code againand<br /> > see what changed and then modify the hardcoded values of the structure size.<br /> > I'm trying toavoid that.<p dir="ltr">Earlier I suggested adding a command line flag to the backend that, like --version, prints thedesired info and exits.<p dir="ltr">It's still most unclear to me what the underlying problem you're trying to solve hereis. Why you want this info and why you're so keen to avoid starting a backend to find it.
On 2016-11-11 09:56:21 +0900, Michael Paquier wrote: > On Fri, Nov 11, 2016 at 8:40 AM, leoaaryan <leoaaryan@gmail.com> wrote: > > The easiest way to find the value for the shared memory computation is to > > change the logging level to DEBUG3 and start postgres DB engine and it will > > give the calculated value in the log file. > > > > I believe postgres as a DB needs to be running for any extension to run (I > > may be wrong here) and I dont want to start the database for this analysis. > > > > Please correct me if I'm wrong in my concepts or if I've not understood > > anything. > > Some time ago a patch has been proposed to project to a system view > the shared memory allocations currently happening on an instance: > https://www.postgresql.org/message-id/20140504114417.GM12715@awork2.anarazel.de > This could be plugged into its own extension rather easily. Note > though that DSM segments (dynamic shared memory) do not have names > associated to them which would be helpful if you'd like to track that > as well. But perhaps you don't care much. FWIW, there's a number of purposes where it'd be more useful to have the total amount of shared memory output without starting postgres. E.g. to compute the amount of hugepages to configure as root... I wondered if we could have a readonly guc that's output by something like -C total_shared_memory or such (which doesn't start a server...). Andres