Обсуждение: Per backend relation statistics tracking
Hi hackers, We currently have some stats in pg_stat_all_tables[|indexes] but we lack visibility into which specific backends are generating this activity. PFA, a patch series to add $SUBJECT. It currently displays a few stats through a new pg_stat_backend view (same as the one already introduced in [1]). This view displays one row per server process, showing statistics related to the current activity of that process. It currently provides something like: pid | seq_scan | seq_tup_read | idx_tup_fetch | idx_scan | idx_tup_read | vacuum_count | last_vacuum | analyze_count | last_analyze | stats_reset ---------+----------+--------------+---------------+----------+--------------+--------------+-------------------------------+---------------+--------------+------------- 3583274 | 0 | 0 | 20 | 11 | 20 | 0 | | 0 | | 3583278 | 3 | 420 | 77 | 60 | 77 | 1 | 2025-08-12 05:47:37.546794+00| 0 | | 3583371 | 4 | 1 | 11444 | 11428 | 13167 | 0 | | 0 | | 3583372 | 4 | 1 | 11404 | 11388 | 13110 | 0 | | 0 | | 3583373 | 4 | 1 | 11412 | 11396 | 13116 | 0 | | 0 | | 3583272 | 0 | 0 | 0 | 0 | 0 | 0 | | 0 | | 3583270 | 0 | 0 | 0 | 0 | 0 | 0 | | 0 | | (7 rows) Let's see some use case examples: - Debug a specific increase in IO reads: When pg_stat_get_backend_io() shows an increase in reads/read_bytes for backends, this new view allows deeper investigation to determine if those reads are caused by table sequential scans and how many rows are involved. - One could check if some backends are triggering manual vacuums or analyzes and check the last time they did so. - Verify load distribution among sessions: Check if database activities are evenly distributed across backends (in the pgbench example above, backends 3583371, 3583372, 3583373 show similar activity patterns). Uneven distribution could indicate outdated application versions on some hosts or suboptimal connection pool configurations. - Set up monitoring alerts for backends showing unusual database access patterns. Also thanks to the pid, we can join pg_stat_activity and then, for example: - If backends from one host show significantly more sequential scans than others, this could indicate an outdated application version on that host running inefficient queries. - One could categorize applications based on workload patterns and track how different applications use the engine. Patch series structure: 0001 - Adds a new PgStat_BackendRelPending struct to store those pending statistics and then relies on the existing per backend statistics machinery that has been added in 9aea73fc61d. It also adds a new counter (heap_scan) to record the number of sequential scans initiated on tables. 0002 - Adds the pg_stat_backend view This view displays one row per server process, showing statistics related to the current activity of that process. It currently displays the pid, the number of sequential scans initiated on a table and the time at which these statistics were last reset. It's built on top of a new function (pg_stat_get_backend_statistics()). The idea is the same as pg_stat_activity and pg_stat_get_activity(). From 0003 to 0010 - Adds one counter at a time. These are purely mechanical changes split for easier review (they don't need to be committed separately). A few remarks: - What I don't like that much is that due to the fact that we want to avoid exposing PendingBackendStats to the outside world (see the related comment in pgstat_backend.c): then those counters need extra functions calls. Those functions are very simple and could just be inlined as macros (avoiding the extra functions calls). Maybe we could re-consider hiding PendingBackendStats to the outside world? - We could add more stats, for example the n_tup_* ones. I did add those in this patch series, as I want to get your feedback first about the whole idea. - With those new counters in place, PgStat_BackendPending grows from 2880 to 2952 (72) bytes and PgStat_Backend from 2920 to 2992 (72) bytes. I think that the memory increase is relatively small. [1]: https://www.postgresql.org/message-id/aJhOfZ2c7XdHXOAU%40ip-10-97-1-34.eu-west-3.compute.internal Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Вложения
- v1-0001-Adding-per-backend-relation-statistics-tracking.patch
- v1-0002-Adding-the-pg_stat_backend-view.patch
- v1-0003-Adding-seq_tup_read-to-pg_stat_backend.patch
- v1-0004-Adding-idx_tup_fetch-to-pg_stat_backend.patch
- v1-0005-Adding-idx_scan-to-pg_stat_backend.patch
- v1-0006-Adding-idx_tup_read-to-pg_stat_backend.patch
- v1-0007-Adding-vacuum_count-to-pg_stat_backend.patch
- v1-0008-Adding-last_vacuum-to-pg_stat_backend.patch
- v1-0009-Adding-analyze_count-to-pg_stat_backend.patch
- v1-0010-Adding-last_analyze-to-pg_stat_backend.patch
Thanks for the patches. I have not gone through them in detail yet, but +1 on adding backend activity stats. This provides another level of drill down to spot anomalous sessions or different patterns across applications. I also think we will want more than just relation stats. For example, columns from pg_statio already look useful on a per-backend aggregate level. Beyond that, I can imagine future additions like number of transactions, subtransactions, I/O stats, conflicts, etc. All of these seem like valuable per-backend aggregates. That is why I think we should be careful about naming. pg_stat_backend feels very generic, but right now it only shows relation stats. Maybe we call it pg_stat_backend_tables to start? Then if we later add I/O, we could have pg_stat_backend_io, or for conflicts, pg_stat_backend_conflicts, etc. That way we keep things more flexible, instead of trying to fit everything into one view. It also helps us avoid having to rename views in the future. What do you think? -- Sami Imseih Amazon Web Services (AWS)
On Mon, Aug 25, 2025 at 05:51:38PM -0500, Sami Imseih wrote: > I have not gone through them in detail yet, but +1 on adding backend activity > stats. This provides another level of drill down to spot anomalous sessions or > different patterns across applications. I also think we will want more than > just relation stats. For example, columns from pg_statio already look useful on > a per-backend aggregate level. Beyond that, I can imagine future additions like > number of transactions, subtransactions, I/O stats, conflicts, etc. All of these > seem like valuable per-backend aggregates. FWIW, I am not excited by this proposal. WAL and pg_stat_io are interesting for monitoring purposes because they can be used to check the balance of the activity across all the backends. The current pg_statio_* relations are interesting because they provide data to the fields for each relation. Adding these fields to the backend level stats spread based on the backend PID without the knowledge of the relation they're related with makes it much less interesting IMO, because we lose a lot of granularity value that we have with the pg_statio_* relations, at the cost of more bloat, particularly if these numbers are distributed across many relations. Even if I see this data, I am not sure how I would use it in correlation with the existing pg_statio_* to tune something, the existing views being sufficient to tune relation-level parameters, no? I have equally some doubts about the value of the vacuum and analyze count, including the time of their latest runs. Manual vacuums are something that some do because autovacuum is not able to keep up, still the encouraged patterns involve running these in periods of lower-activity. How would knowing about the whole number of vacuums and/or analyze be useful if these are run with cron jobs, which I suspect involve connections that live only for the duration of one single job? Transactions and subtransactions may be interesting to consider. Perhaps mainly useful to evaluate the balance activity, say with a connection pooler with some specific configuration or when checking transaction activity across multiple databases by joining with pg_stat_activity? -- Michael
Вложения
> Adding these fields to the backend level stats spread based on the > backend PID without the knowledge of the relation they're related with > makes it much less interesting IMO, because we lose a lot of > granularity value that we have with the pg_statio_* relations, at the > cost of more bloat, particularly if these numbers are distributed > across many relations. I think the flip side of the argument is that the current per-table metrics don't tell us which of our backends ( by user, application_name ) are contributing to a specific type of activity. Would it be interesting to try to answer a question such as "Does application_name = AppA perform more sequential scans than AppB?" or "does UserA perform more index scans than UserB?" Currently, we don't have an easy way to figure out such information, without some sort of a DML trigger. -- Sami
Hi, On 2025-08-12 07:48:10 +0000, Bertrand Drouvot wrote: > From 9e2f8cb9a87f1d9be91f2f39ef25fbb254944968 Mon Sep 17 00:00:00 2001 > From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> > Date: Mon, 4 Aug 2025 08:14:02 +0000 > Subject: [PATCH v1 01/10] Adding per backend relation statistics tracking > > This commit introduces per backend relation stats tracking and adds a > new PgStat_BackendRelPending struct to store the pending statistics. To begin with, > this commit adds a new counter (heap_scan) to record the number of sequential > scans initiated on tables. > > This commit relies on the existing per backend statistics machinery that has been > added in 9aea73fc61d. > --- > src/backend/access/heap/heapam.c | 3 ++ > src/backend/utils/activity/pgstat_backend.c | 59 +++++++++++++++++++++ > src/include/pgstat.h | 14 +++++ > src/include/utils/pgstat_internal.h | 3 +- > src/tools/pgindent/typedefs.list | 1 + > 5 files changed, 79 insertions(+), 1 deletion(-) > 73.9% src/backend/utils/activity/ > 7.4% src/include/utils/ > 15.4% src/include/ > 3.2% src/ > > diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c > index 0dcd6ee817e..d9d6fb6c6ea 100644 > --- a/src/backend/access/heap/heapam.c > +++ b/src/backend/access/heap/heapam.c > @@ -467,7 +467,10 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) > * and for sample scans we update stats for tuple fetches). > */ > if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN) > + { > pgstat_count_heap_scan(scan->rs_base.rs_rd); > + pgstat_count_backend_rel_heap_scan(); > + } > } > I don't like that this basically doubles the overhead of keeping stats by tracking everythign twice. The proper solution is to do that not in the hot path (i.e. in scans), but when summarizing stats to be flushed to the shared stats. FWIW, I think this was done wrongly for the per-backend IO stats too. I've seen the increased overhead in profiles - and IO related counters aren't incremented remotely as often as the scan related counters are. Greetings, Andres Freund
Hi, On Tue, Aug 26, 2025 at 08:12:45AM +0900, Michael Paquier wrote: > On Mon, Aug 25, 2025 at 05:51:38PM -0500, Sami Imseih wrote: > > I have not gone through them in detail yet, but +1 on adding backend activity > > stats. This provides another level of drill down to spot anomalous sessions or > > different patterns across applications. I also think we will want more than > > just relation stats. For example, columns from pg_statio already look useful on > > a per-backend aggregate level. Beyond that, I can imagine future additions like > > number of transactions, subtransactions, I/O stats, conflicts, etc. All of these > > seem like valuable per-backend aggregates. > > Even if I see this data, I am not sure how I > would use it in correlation with the existing pg_statio_* to tune > something, the existing views being sufficient to tune relation-level > parameters, no? Right, but the "opposite" is also true, how would you: - Debug a specific increase in IO reads: When pg_stat_get_backend_io() shows an increase in reads/read_bytes for backends, this new view allows deeper investigation to determine if those reads are caused by table sequential scans and how many rows are involved. - Verify load distribution among sessions: Check if database activities are evenly distributed across backends. Uneven distribution could indicate outdated application versions on some hosts or suboptimal connection pool configurations. - Set up monitoring alerts for backends showing unusual database access patterns. - If backends from one host show significantly more sequential scans than others, this could indicate an outdated application version on that host running inefficient queries. - One could categorize applications based on workload patterns and track how different applications use the engine. With "only" the relation's stats at your disposal? The relations are the "destinations" of the activity while the backends are the "source". I think it's good to track both. > I have equally some doubts about the value of the vacuum and analyze > count, including the time of their latest runs. Manual vacuums are > something that some do because autovacuum is not able to keep up, > still the encouraged patterns involve running these in periods of > lower-activity. How would knowing about the whole number of vacuums > and/or analyze be useful if these are run with cron jobs, which I > suspect involve connections that live only for the duration of one > single job? This one was more to track "unwanted" manual vacuums. The timestamp could help to know which relation is involved (with a where clause on the timestamp). > Transactions and subtransactions may be interesting to consider. > Perhaps mainly useful to evaluate the balance activity, say with a > connection pooler with some specific configuration or when checking > transaction activity across multiple databases by joining with > pg_stat_activity? Yes exactly. And also to get XID (virtual transactions excluded) consumption rate per backend like in [1]. In [1], the number of XIDs generated per backend has been added. [1]: https://www.postgresql.org/message-id/aJhOfZ2c7XdHXOAU%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Mon, Aug 25, 2025 at 07:22:43PM -0500, Sami Imseih wrote: > > Adding these fields to the backend level stats spread based on the > > backend PID without the knowledge of the relation they're related with > > makes it much less interesting IMO, because we lose a lot of > > granularity value that we have with the pg_statio_* relations, at the > > cost of more bloat, particularly if these numbers are distributed > > across many relations. > > I think the flip side of the argument is that the current per-table metrics > don't tell us which of our backends ( by user, application_name ) > are contributing to a specific type of activity. > Would it be interesting to try to answer a question such as > "Does application_name = AppA perform more sequential scans than AppB?" > or "does UserA perform more index scans than UserB?" Right. I think we can see the relations as the "destinations" of the activity and the backends as the "source" of it. Tracking both sides help to visualize the activity from different angles. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Mon, Aug 25, 2025 at 08:28:04PM -0400, Andres Freund wrote: > Hi, > > On 2025-08-12 07:48:10 +0000, Bertrand Drouvot wrote: > > From 9e2f8cb9a87f1d9be91f2f39ef25fbb254944968 Mon Sep 17 00:00:00 2001 > > From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> > > Date: Mon, 4 Aug 2025 08:14:02 +0000 > > Subject: [PATCH v1 01/10] Adding per backend relation statistics tracking > > > > This commit introduces per backend relation stats tracking and adds a > > new PgStat_BackendRelPending struct to store the pending statistics. To begin with, > > this commit adds a new counter (heap_scan) to record the number of sequential > > scans initiated on tables. > > > > This commit relies on the existing per backend statistics machinery that has been > > added in 9aea73fc61d. > > --- > > src/backend/access/heap/heapam.c | 3 ++ > > src/backend/utils/activity/pgstat_backend.c | 59 +++++++++++++++++++++ > > src/include/pgstat.h | 14 +++++ > > src/include/utils/pgstat_internal.h | 3 +- > > src/tools/pgindent/typedefs.list | 1 + > > 5 files changed, 79 insertions(+), 1 deletion(-) > > 73.9% src/backend/utils/activity/ > > 7.4% src/include/utils/ > > 15.4% src/include/ > > 3.2% src/ > > > > diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c > > index 0dcd6ee817e..d9d6fb6c6ea 100644 > > --- a/src/backend/access/heap/heapam.c > > +++ b/src/backend/access/heap/heapam.c > > @@ -467,7 +467,10 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) > > * and for sample scans we update stats for tuple fetches). > > */ > > if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN) > > + { > > pgstat_count_heap_scan(scan->rs_base.rs_rd); > > + pgstat_count_backend_rel_heap_scan(); > > + } > > } > > > > I don't like that this basically doubles the overhead of keeping stats by > tracking everythign twice. The proper solution is to do that not in the hot > path (i.e. in scans), but when summarizing stats to be flushed to the shared > stats. I do agree, like when the relations stats are flushed then we do update the database ones too. I'll use the same approach in the next revision. > FWIW, I think this was done wrongly for the per-backend IO stats too. I've > seen the increased overhead in profiles That's indeed something that could be improved for the backends IO stats too. I'll work on this for 19 (that's probably too late for 18 and not that alarming?). > and IO related counters aren't > incremented remotely as often as the scan related counters are. You mean the flush are not triggered as often? If so, yeah that's also something you've mentioned ([1]) and that I've in mind to look at. [1]: https://www.postgresql.org/message-id/erpzwxoptqhuptdrtehqydzjapvroumkhh7lc6poclbhe7jk7l%40l3yfsq5q4pw7 Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Tue, Aug 26, 2025 at 06:38:41AM +0000, Bertrand Drouvot wrote: > Hi, > > On Mon, Aug 25, 2025 at 08:28:04PM -0400, Andres Freund wrote: > > I don't like that this basically doubles the overhead of keeping stats by > > tracking everythign twice. The proper solution is to do that not in the hot > > path (i.e. in scans), but when summarizing stats to be flushed to the shared > > stats. > > I do agree, like when the relations stats are flushed then we do update the > database ones too. I'll use the same approach in the next revision. Something along the lines like in the attached (I'm just providing 0001 here, will do the others if we agree on the proposal)? Remark: We could avoid the new branch in pgstat_relation_flush_cb() if we split tables and indexes stats, but I don't think that should be a blocker for per backend relations stats. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Вложения
Hi, On Mon, Aug 25, 2025 at 05:51:38PM -0500, Sami Imseih wrote: > Thanks for the patches. > > I have not gone through them in detail yet, but +1 on adding backend activity > stats. Thanks for sharing your thoughts. > This provides another level of drill down to spot anomalous sessions or > different patterns across applications. I also think we will want more than > just relation stats. For example, columns from pg_statio already look useful on > a per-backend aggregate level. Beyond that, I can imagine future additions like > number of transactions, subtransactions, I/O stats, conflicts, etc. All of these > seem like valuable per-backend aggregates. > > That is why I think we should be careful about naming. pg_stat_backend feels > very generic, but right now it only shows relation stats. Maybe we call it > pg_stat_backend_tables to start? Then if we later add I/O, we could have > pg_stat_backend_io, or for conflicts, pg_stat_backend_conflicts, etc. That way > we keep things more flexible, instead of trying to fit everything into > one view. It also helps us avoid having to rename views in the future. > > What do you think? My initial idea was to provide just a single view for "basic" counters (i.e things that do not need multiple lines per backend (like pg_stat_get_backend_io() output would need). That would mean a single view for "basic" counters and a set of functions for more "sophisticated" ones (like pg_stat_get_backend_io()) that need multiple rows per backend. Having said that, we could imagine adding pg_stat_get_backend_wal() output to pg_stat_backend too. Thoughts? -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On 2025-08-26 06:38:41 +0000, Bertrand Drouvot wrote: > > and IO related counters aren't > > incremented remotely as often as the scan related counters are. > > You mean the flush are not triggered as often? If so, yeah that's also something > you've mentioned ([1]) and that I've in mind to look at. I mean that we increment the counters less frequently. pgstat_count_heap_getnext() is called for every tuple on a page, which is obviously much more frequent than once per page like for IO. Greetings, Andres Freund
> > That is why I think we should be careful about naming. pg_stat_backend feels > > very generic, but right now it only shows relation stats. Maybe we call it > > pg_stat_backend_tables to start? Then if we later add I/O, we could have > > pg_stat_backend_io, or for conflicts, pg_stat_backend_conflicts, etc. That way > > we keep things more flexible, instead of trying to fit everything into > > one view. It also helps us avoid having to rename views in the future. > > > > What do you think? > > My initial idea was to provide just a single view for "basic" counters (i.e > things that do not need multiple lines per backend (like pg_stat_get_backend_io() > output would need). That would mean a single view for "basic" counters and > a set of functions for more "sophisticated" ones (like pg_stat_get_backend_io()) > that need multiple rows per backend. Having said that, we could imagine adding > pg_stat_get_backend_wal() output to pg_stat_backend too. > > Thoughts? The data that is multiple rows per backend is unique from that perspective. The stuff we are talking about now will be one row with data aggregated up to the PID. I worry that a single view will grow very wide, and we will have to eventually split it. So we may as well start thinking about having multiple views in advance. > Having said that, we could imagine adding > pg_stat_get_backend_wal() output to pg_stat_backend too. For this one, I think we should just overload the function pg_stat_get_backend_wal, and if a PID is not passed in, return all of them; and just create a new view called pg_stat_backend_wal that returns all the PIDs. sort of like how we have pg_stat_get_activity and pg_stat_activity, etc. Would that not be more consistent? -- Sami
Hi, On Tue, Aug 26, 2025 at 07:18:10AM -0400, Andres Freund wrote: > Hi, > > On 2025-08-26 06:38:41 +0000, Bertrand Drouvot wrote: > > > and IO related counters aren't > > > incremented remotely as often as the scan related counters are. > > > > You mean the flush are not triggered as often? If so, yeah that's also something > > you've mentioned ([1]) and that I've in mind to look at. > > I mean that we increment the counters less frequently. > pgstat_count_heap_getnext() is called for every tuple on a page, which is > obviously much more frequent than once per page like for IO. I see. So let's rely on the existing "relation" increments and make use of them when flushing the relation stats to populate the backend stats (as done in v2 shared up-thread). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Tue, Aug 26, 2025 at 04:55:09PM -0500, Sami Imseih wrote: > I worry that a single view will grow very wide, and we will have to eventually > split it. So we may as well start thinking about having multiple views > in advance. I gave it more thoughts and I now think that multiple views is better. We could start with pg_stat_backend_relations here. > > Having said that, we could imagine adding > > pg_stat_get_backend_wal() output to pg_stat_backend too. > > For this one, I think we should just overload the function > pg_stat_get_backend_wal, > and if a PID is not passed in, return all of them; and just create a > new view called > pg_stat_backend_wal that returns all the PIDs. sort of like how we have > pg_stat_get_activity and pg_stat_activity, etc. Yes, and it's also how the new view is designed in this patch. I think I'll add a new pg_stat_backend_wal view if the current proposal goes in (for consistency). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Wed, Aug 27, 2025 at 01:57:13PM +0000, Bertrand Drouvot wrote: > Hi, > > On Tue, Aug 26, 2025 at 04:55:09PM -0500, Sami Imseih wrote: > > I worry that a single view will grow very wide, and we will have to eventually > > split it. So we may as well start thinking about having multiple views > > in advance. > > I gave it more thoughts and I now think that multiple views is better. We > could start with pg_stat_backend_relations here. > PFA v3, using pg_stat_backend_relation instead of pg_stat_backend. In passing, adding PGDLLIMPORT for backend_has_relstats and PendingBackendStats that were missing in v2. As compared to v1, only 0001 and 0002 are shared. Let's discuss those first before moving foward with 0003+. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Вложения
Hi, On Tue, Aug 26, 2025 at 06:38:41AM +0000, Bertrand Drouvot wrote: > On Mon, Aug 25, 2025 at 08:28:04PM -0400, Andres Freund wrote: > > FWIW, I think this was done wrongly for the per-backend IO stats too. I've > > seen the increased overhead in profiles > > That's indeed something that could be improved for the backends IO stats too. I'll > work on this for 19 (that's probably too late for 18 and not that alarming?). The concern for the backends IO stats is addressed in [1]. [1]: https://www.postgresql.org/message-id/flat/aLWonxez0MBSqEvA%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com