Обсуждение: documentation on HOT

Поиск
Список
Период
Сортировка

documentation on HOT

От
"Jonathan S. Katz"
Дата:
Hi,

While working on the 2020-02-10 release announcement, in order to better 
describe one of the fixes, I tried to find a reference in the docs that 
described what HOT is, how it works, etc. in a user-friendly manner.

This lead me to the acronyms page[1], which lead me to a document in the 
repo[2]. The closest thing I found was a blog post that Laurenz Albe 
wrote on the topic[3].

Given the importance of HOT, it seems like this would be a good topic to 
document. I would suggest something higher-level for general users in 
the "Indexes"[4] section, and something lower-level in internals[5] 
(which could perhaps be derived from [2]).

Thoughts on this?

Thanks,

Jonathan

[1] https://www.postgresql.org/docs/current/acronyms.html
[2] 
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/access/heap/README.HOT;hb=HEAD
[3] 
https://www.cybertec-postgresql.com/en/hot-updates-in-postgresql-for-better-performance/
[4] https://www.postgresql.org/docs/current/indexes.html
[5] https://www.postgresql.org/docs/current/internals.html

Вложения

Re: documentation on HOT

От
"David G. Johnston"
Дата:
On Sun, Feb 6, 2022 at 6:08 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:

Given the importance of HOT, it seems like this would be a good topic to
document. I would suggest something higher-level for general users in
the "Indexes"[4] section, and something lower-level in internals[5]
(which could perhaps be derived from [2]).

Thoughts on this?

I'm doubting there is any disagreement that this is needed.  Three of us said as much recently [1] while discussing a user question regarding our documentation for expression indexes.


I presently haven't felt moved to fill the need myself as so I was fine with at least providing a high-level summary in the glossary which we also the general user to understand the idea without having to dive into the README.  The details for internals can continue to be handled there in the interest of at least getting something committed for the majority of users.

David J.

Re: documentation on HOT

От
"Jonathan S. Katz"
Дата:
On 2/6/22 8:56 PM, David G. Johnston wrote:
> On Sun, Feb 6, 2022 at 6:08 PM Jonathan S. Katz <jkatz@postgresql.org 
> <mailto:jkatz@postgresql.org>> wrote:
> 
> 
>     Given the importance of HOT, it seems like this would be a good
>     topic to
>     document. I would suggest something higher-level for general users in
>     the "Indexes"[4] section, and something lower-level in internals[5]
>     (which could perhaps be derived from [2]).
> 
>     Thoughts on this?
> 
> 
> I'm doubting there is any disagreement that this is needed.  Three of us 
> said as much recently [1] while discussing a user question regarding our 
> documentation for expression indexes.

I did try to search for previous discussion on this (not well enough 
apparently) and I was unable to find the thread. That is also part of 
the challenge with the term HOT.

> I presently haven't felt moved to fill the need myself as so I was fine 
> with at least providing a high-level summary in the glossary which we 
> also the general user to understand the idea without having to dive into 
> the README.

I don't agree that such an explanation belongs in the glossary. That 
feels like it would be too brief, and I think that does a disservice to 
the topic and our users.

I agree with Bruce's point that we should have a new section (or 
subsection). As I mentioned in my previous post, given HOT involves 
indexing, I would suggest putting it there.

I think that something that follows the general outline of Laurenz's 
post would satisfy the user requirements. It explains at a high level 
what HOT is, it's advantages, and how it works.

>  The details for internals can continue to be handled there 
> in the interest of at least getting something committed for the majority 
> of users.

I do think there should be some reference of it to the docs. Even if we 
have a page in the "Internals" section that says "For more information 
on how HOT works, please see <link>".

That said, is there any reasoning why the HOT README (or something 
similar to it) is not in the "Internals" section of the documentation, 
similar to other indexing topics?

Thanks,

Jonathan

Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Sun, Feb  6, 2022 at 09:29:56PM -0500, Jonathan Katz wrote:
> I agree with Bruce's point that we should have a new section (or
> subsection). As I mentioned in my previous post, given HOT involves
> indexing, I would suggest putting it there.
> 
> I think that something that follows the general outline of Laurenz's post
> would satisfy the user requirements. It explains at a high level what HOT
> is, it's advantages, and how it works.

Attached is a patch that adds a new HOT section to the storage chapter.
You can see the results here:

    https://momjian.us/tmp/pgsql/storage-hot.html

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Thu, Jul 21, 2022 at 10:02:18PM -0400, Bruce Momjian wrote:
> On Sun, Feb  6, 2022 at 09:29:56PM -0500, Jonathan Katz wrote:
> > I agree with Bruce's point that we should have a new section (or
> > subsection). As I mentioned in my previous post, given HOT involves
> > indexing, I would suggest putting it there.
> > 
> > I think that something that follows the general outline of Laurenz's post
> > would satisfy the user requirements. It explains at a high level what HOT
> > is, it's advantages, and how it works.
> 
> Attached is a patch that adds a new HOT section to the storage chapter.

I wasn't happy with the last paragraph so I added some more details. 
URL contents updated too:

    https://momjian.us/tmp/pgsql/storage-hot.html

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
"Jonathan S. Katz"
Дата:
On 7/22/22 8:51 AM, Bruce Momjian wrote:
> On Thu, Jul 21, 2022 at 10:02:18PM -0400, Bruce Momjian wrote:
>> On Sun, Feb  6, 2022 at 09:29:56PM -0500, Jonathan Katz wrote:
>>> I agree with Bruce's point that we should have a new section (or
>>> subsection). As I mentioned in my previous post, given HOT involves
>>> indexing, I would suggest putting it there.
>>>
>>> I think that something that follows the general outline of Laurenz's post
>>> would satisfy the user requirements. It explains at a high level what HOT
>>> is, it's advantages, and how it works.
>>
>> Attached is a patch that adds a new HOT section to the storage chapter.
> 
> I wasn't happy with the last paragraph so I added some more details.
> URL contents updated too:
> 
>     https://momjian.us/tmp/pgsql/storage-hot.html

Thanks! This is great. Probably the most concise and clear explanation I 
have seen for HOT, which is exactly what we need for the docs :)

A few suggestions:

s/Fortunately, there is/To help reduce overhead, PostgreSQL has/

s/In summary, h/H/

 > You can increase the change of HOT updates by using non-default table 
fillfactor settings.

I think we should expand on this and explain how adjusting "fillfactor" 
will affect this. I think that may change the final sentence too.

Thanks,

Jonathan

Вложения

Re: documentation on HOT

От
"David G. Johnston"
Дата:
On Fri, Jul 22, 2022 at 8:09 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
On 7/22/22 8:51 AM, Bruce Momjian wrote:
> On Thu, Jul 21, 2022 at 10:02:18PM -0400, Bruce Momjian wrote:
>> On Sun, Feb  6, 2022 at 09:29:56PM -0500, Jonathan Katz wrote:
>>> I agree with Bruce's point that we should have a new section (or
>>> subsection). As I mentioned in my previous post, given HOT involves
>>> indexing, I would suggest putting it there.
>>>
>>> I think that something that follows the general outline of Laurenz's post
>>> would satisfy the user requirements. It explains at a high level what HOT
>>> is, it's advantages, and how it works.
>>
>> Attached is a patch that adds a new HOT section to the storage chapter.
>
> I wasn't happy with the last paragraph so I added some more details.
> URL contents updated too:
>
>       https://momjian.us/tmp/pgsql/storage-hot.html

Thanks! This is great. Probably the most concise and clear explanation I
have seen for HOT, which is exactly what we need for the docs :)

A few suggestions:

s/Fortunately, there is/To help reduce overhead, PostgreSQL has/

s/In summary, h/H/

 > You can increase the change of HOT updates by using non-default table
fillfactor settings.

I think we should expand on this and explain how adjusting "fillfactor"
will affect this. I think that may change the final sentence too.


I think we need to expose the information regarding columns used in predicates here.

"(Here, "indexed column" means any column referenced
at all in an index definition, including for example columns that are
tested in a partial-index predicate but are not stored in the index.)"

I get it is an implementation detail but explaining the name seems like a good thing to do as well:

"Without HOT, every version of a row in an update chain has its own index
entries, even if all indexed columns are the same.  With HOT, a new tuple
placed on the same page and with all indexed columns the same as its
parent row version does not get new index entries.  This means there is
only one index entry for the entire update chain on the heap page.
An index-entry-less tuple is marked with the HEAP_ONLY_TUPLE flag."

Where the last sentence becomes: "Those index-entry-less tuples (yeah, still dislike triple-hypenation...) are thus named "Heap-Only Tuples".

(I've actually incorporated this as I think it should be down below, as a lead-in to the listing of conditions for when the optimization can be used.)

Then maybe "can be removed during select" should be reworded as:

"No longer visible heap-only tuples can be removed during normal
operation, including <command>SELECT</command>s, instead of requiring
periodic vacuum operations."

The original heap entry the index points to cannot be removed. "Old versions of heap-only tuples" vs. "No longer visible heap-only tuples" is probably a style choice.  There are basically three different "versions" in context here though so avoiding "old versions" has some appeal to me.

I'm not a fan of:

"Fortunately, there is an automatic system..."

I'd like to give credit to the fact we engineered a solution to the downsides, so change the lead-in paragraph to the conditions listing to be:

"To mitigate these downsides PostgreSQL implements an optimization whereby sometimes only the heap tuple is created, not the index entry, when performing an update.  In a case of giving things obvious and meaningful names, this is the Heap-Only Tuple (HOT) Optimization.  This optimization is possible when:"

David J.

Re: documentation on HOT

От
Peter Geoghegan
Дата:
On Fri, Jul 22, 2022 at 8:09 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> Thanks! This is great. Probably the most concise and clear explanation I
> have seen for HOT, which is exactly what we need for the docs :)

I'm delighted that this is finally happening!

> I think we should expand on this and explain how adjusting "fillfactor"
> will affect this. I think that may change the final sentence too.

Definitely -- this is by far the most important reason to tune heap
fill factor, which can make a big difference. There should be a link
from the fill factor docs to the new HOT section, at a minimum.

Another thing: would be good to point out how to monitor the number of
HOT updates vs non-HOT updates using views like pg_stat_user_tables.

One minor correction: Opportunistic pruning isn't limited to heap-only
tuples -- "HOT pruning" is actually a bit of a misnomer, that somehow
caught on in the Postgres community. Opportunistic pruning can and
will happen even with non-HOT updates -- the only difference on the
heap page level is that 4 byte LP_DEAD line pointers will accumulate
over time, which only VACUUM is able to clean up (we need LP_DEAD
items to stick around until then as tombstones, so that index scans
don't ever get confused). But pruning will still be able to free most
of the space on the heap page, in almost the same way.

Of course the indexes on the table are another matter -- each and
every index will accumulate versions when there are many non-HOT
updates, which can be a huge problem. My point is that *from the point
of view of an affected heap page*, pruning (whether opportunistic or
performed by VACUUM) can go ahead without HOT tuples/updates, and
isn't really that different in terms of the amount of space freed each
time.

Bottom-up index deletion is effective is no small part because pruning
was always pretty effective even with non-HOT updates, except within
indexes -- it pretty much compensates for that remaining problem. The
heuristics that it uses in the nbtree code work quite well because of
the natural tendency of those 4 byte LP_DEAD item pointers to
concentrate on individual heap pages. We can fit so many more 4 byte
LP_DEAD items than full sized live heap tuples that it becomes pretty
obvious (from the point of view of the nbtree code that starts out by
looking at an index leaf page) which pointed-to heap blocks are likely
to enable it to free space via index tuple deletes -- it often visits
the heap pages with the most TIDs.

Before bottom-up index deletion was added, I would notice that pgbench
variant workloads with non-HOT updates would result in a
pgbench_accounts table that is maybe 1%-2% larger after a few hours,
while the indexes on the same table would grow by 100% - 200%.
Opportunistic pruning would work pretty well for the heap, but
wouldn't do anything at all to control bloat in indexes. So
opportunistic pruning was always very effective with non-HOT updates
in one important way, but totally ineffective in another way.

There is a section about bottom-up index deletion in the docs (in the
B-Tree internals chapter) that already references HOT. It should link
to this new chapter now, I think.

-- 
Peter Geoghegan



Re: documentation on HOT

От
Bruce Momjian
Дата:
On Fri, Jul 22, 2022 at 11:09:49AM -0400, Jonathan Katz wrote:
> On 7/22/22 8:51 AM, Bruce Momjian wrote:
> > URL contents updated too:
> > 
> >     https://momjian.us/tmp/pgsql/storage-hot.html
> 
> Thanks! This is great. Probably the most concise and clear explanation I
> have seen for HOT, which is exactly what we need for the docs :)
> 
> A few suggestions:
> 
> s/Fortunately, there is/To help reduce overhead, PostgreSQL has/

So, I can't do that without making the later part of the sentence
redundant.  I adjusted it in the attached patch.
> 
> s/In summary, h/H/

Uh, I don't want to remove that since I am restating what I said above
--- do you have another approach?

> > You can increase the change of HOT updates by using non-default table
> fillfactor settings.
> 
> I think we should expand on this and explain how adjusting "fillfactor" will
> affect this. I think that may change the final sentence too.

Okay, done in the attached patch.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Fri, Jul 22, 2022 at 09:25:43AM -0700, David G. Johnston wrote:
> On Fri, Jul 22, 2022 at 8:09 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> I think we need to expose the information regarding columns used in predicates
> here.
> 
> "(Here, "indexed column" means any column referenced
> at all in an index definition, including for example columns that are
> tested in a partial-index predicate but are not stored in the index.)"

Okay, I clarified this in the attached patch.

> I get it is an implementation detail but explaining the name seems like a good
> thing to do as well:
> 
> "Without HOT, every version of a row in an update chain has its own index
> entries, even if all indexed columns are the same.  With HOT, a new tuple
> placed on the same page and with all indexed columns the same as its
> parent row version does not get new index entries.  This means there is
> only one index entry for the entire update chain on the heap page.
> An index-entry-less tuple is marked with the HEAP_ONLY_TUPLE flag."

I don't see how the chain is useful for people trying to understand how
to benefit from this feature.

> Where the last sentence becomes: "Those index-entry-less tuples (yeah, still
> dislike triple-hypenation...) are thus named "Heap-Only Tuples".
> 
> (I've actually incorporated this as I think it should be down below, as a
> lead-in to the listing of conditions for when the optimization can be used.)
> 
> Then maybe "can be removed during select" should be reworded as:
> 
> "No longer visible heap-only tuples can be removed during normal
> operation, including <command>SELECT</command>s, instead of requiring
> periodic vacuum operations."

I added a no-longer-visible qualifier to the patch.

> The original heap entry the index points to cannot be removed. "Old versions of
> heap-only tuples" vs. "No longer visible heap-only tuples" is probably a style
> choice.  There are basically three different "versions" in context here though
> so avoiding "old versions" has some appeal to me.
> 
> I'm not a fan of:
> 
> "Fortunately, there is an automatic system..."
> 
> I'd like to give credit to the fact we engineered a solution to the downsides,
> so change the lead-in paragraph to the conditions listing to be:

Yeah, good point.  We didn't stumble upon this feature.  I have adjusted
that wording.

> "To mitigate these downsides PostgreSQL implements an optimization whereby
> sometimes only the heap tuple is created, not the index entry, when performing
> an update.  In a case of giving things obvious and meaningful names, this is
> the Heap-Only Tuple (HOT) Optimization.  This optimization is possible when:"

Sorry, I don't like the above since it isn't precise and the "In a case
of giving things obvious and meaningful names" seems odd.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Fri, Jul 22, 2022 at 10:05:04AM -0700, Peter Geoghegan wrote:
> On Fri, Jul 22, 2022 at 8:09 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> > Thanks! This is great. Probably the most concise and clear explanation I
> > have seen for HOT, which is exactly what we need for the docs :)
> 
> I'm delighted that this is finally happening!
> 
> > I think we should expand on this and explain how adjusting "fillfactor"
> > will affect this. I think that may change the final sentence too.
> 
> Definitely -- this is by far the most important reason to tune heap
> fill factor, which can make a big difference. There should be a link
> from the fill factor docs to the new HOT section, at a minimum.

Done in the attached patch, and updated URL:

    https://momjian.us/tmp/pgsql/storage-hot.html

> Another thing: would be good to point out how to monitor the number of
> HOT updates vs non-HOT updates using views like pg_stat_user_tables.

I added a sentence about that, good idea.

> One minor correction: Opportunistic pruning isn't limited to heap-only
> tuples -- "HOT pruning" is actually a bit of a misnomer, that somehow
> caught on in the Postgres community. Opportunistic pruning can and
> will happen even with non-HOT updates -- the only difference on the
> heap page level is that 4 byte LP_DEAD line pointers will accumulate
> over time, which only VACUUM is able to clean up (we need LP_DEAD
> items to stick around until then as tombstones, so that index scans
> don't ever get confused). But pruning will still be able to free most
> of the space on the heap page, in almost the same way.

I see your point. I changed the benefits from pruning to no new page
item identifiers.

> There is a section about bottom-up index deletion in the docs (in the
> B-Tree internals chapter) that already references HOT. It should link
> to this new chapter now, I think.

Already done in the attached patch, I think.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Fri, Jul 22, 2022 at 01:07:41PM -0400, Bruce Momjian wrote:
> > > You can increase the change of HOT updates by using non-default table
> > fillfactor settings.
> > 
> > I think we should expand on this and explain how adjusting "fillfactor" will
> > affect this. I think that may change the final sentence too.
> 
> Okay, done in the attached patch.

I have improved the wording of the last paragraph in this patch.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Peter Geoghegan
Дата:
On Fri, Jul 22, 2022 at 2:11 PM Bruce Momjian <bruce@momjian.us> wrote:
> I have improved the wording of the last paragraph in this patch.

I think that it would be worth prominently explaining where heap-only
tuples get their name from: it comes from the fact there are (by
definition) no entries for a heap-only tuple in any index, ever.
Indexes are nevertheless capable of locating heap-only tuples during
index scans, by dealing with a little additional indirection: they
must traverse groups of related tuple versions, all for the same
logical row that was HOT updated one or more times -- this group of
related tuples is called a HOT chain.

This seems like a useful thing to emphasize because it places the
emphasis on what *doesn't* happen. Mostly what doesn't happen in
indexes.

New item identifiers actually *are* needed for heap-only tuples
(perhaps we could get away with it, but we don't). However, that
doesn't really matter too much in practice. Heap-only tuples can still
have their line pointers set to LP_UNUSED directly during pruning,
without having to be set to LP_DEAD for a time first (a situation
which VACUUM alone can correct by setting the LP_DEAD items to
LP_UNUSED during its second heap pass).

So heap-only tuples "skip the step" where they have to become LP_DEAD
stubs/tombstones. Which is possible precisely because indexes don't
need to be considered (they're "heap-only").

I agree that pruning should be discussed here, though -- I wouldn't go
as far as treating pruning as 100% unrelated to HOT. Perhaps something
along the lines of this works:

"It is possible for opportunistic pruning to completely remove all
bloat caused by HOT updates (bloat from HOT chains), without leaving
any residual garbage that only VACUUM is capable of cleaning up.
Pruning a page affected by non-HOT updates or deletes is somewhat less
effective, though, because small tombstone items (dead item
identifiers) must remain until such time as VACUUM can verify that no
remaining index tuples reference the items."

Again, the emphasis is on what *doesn't* have to happen because
indexes aren't making life hard for us. From the point of view of
indexes, ignorance is bliss. The really nice important point about
pruning and HOT is that it becomes possible (with care from the DBA
and application) to practically eliminate the role of VACUUM. We may
not even require a little help from VACUUM, under ideal conditions.

-- 
Peter Geoghegan



Re: documentation on HOT

От
Bruce Momjian
Дата:
On Fri, Jul 22, 2022 at 03:33:20PM -0700, Peter Geoghegan wrote:
> On Fri, Jul 22, 2022 at 2:11 PM Bruce Momjian <bruce@momjian.us> wrote:
> > I have improved the wording of the last paragraph in this patch.
> 
> I think that it would be worth prominently explaining where heap-only
> tuples get their name from: it comes from the fact there are (by
> definition) no entries for a heap-only tuple in any index, ever.
> Indexes are nevertheless capable of locating heap-only tuples during
> index scans, by dealing with a little additional indirection: they
> must traverse groups of related tuple versions, all for the same
> logical row that was HOT updated one or more times -- this group of
> related tuples is called a HOT chain.
> 
> This seems like a useful thing to emphasize because it places the
> emphasis on what *doesn't* happen. Mostly what doesn't happen in
> indexes.
> 
> New item identifiers actually *are* needed for heap-only tuples
> (perhaps we could get away with it, but we don't). However, that
> doesn't really matter too much in practice. Heap-only tuples can still
> have their line pointers set to LP_UNUSED directly during pruning,
> without having to be set to LP_DEAD for a time first (a situation
> which VACUUM alone can correct by setting the LP_DEAD items to
> LP_UNUSED during its second heap pass).
> 
> So heap-only tuples "skip the step" where they have to become LP_DEAD
> stubs/tombstones. Which is possible precisely because indexes don't
> need to be considered (they're "heap-only").

Good points.  I have updated the attached patch and URL to mention that
HOT rows are _completely_ removed, and why that is possible, and I
clarified the page item identifier mention.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Peter Geoghegan
Дата:
On Sat, Jul 23, 2022 at 8:51 AM Bruce Momjian <bruce@momjian.us> wrote:
> Good points.  I have updated the attached patch and URL to mention that
> HOT rows are _completely_ removed, and why that is possible, and I
> clarified the page item identifier mention.

I think that this version looks very good, but I do have some minor notes:

* You wrote "Specifically, updates cause additional rows to be added to tables."

Perhaps this could be rephrased: "Specifically, updates add new
physical tuples to tables to represent each new version."

I think that the term "row" should only refer to the simple/abstract
idea of a row from a table, while the term tuple should be preferred
when referring to a physical embodiment of a row, like one version of
a row. Perhaps it's worth following that convention across the board
here (not just in this sentence that I have highlighted).

* You wrote "This can also require new index entries for each updated
row, and removal of old versions of rows can be expensive"

I believe that the operative word in this sentence (which appears in
the first paragraph) is "can". I think that it would be good to go
just a bit further with that. Maybe add another sentence immediately
afterwards that conveys "and now we're going to discuss when and how
new versions from updates can sometimes avoid the need for a new round
of index entries".

* You wrote "New index entries are not needed to represent updated rows"

It seems to me that this undersells the key benefit. You could perhaps
add another sentence. Something like: "This avoids the immediate cost
of adding new successor versions to each and every index, and avoids
the cost of removing the obsolete versions from each and every index
later on."

* You refer to opportunistic pruning as something that happens "during
normal operation", but that doesn't seem to get the idea of
"opportunistic" across.

It seems like it would be worth writing a sentence or two more on
this, just to get that aspect across. Opportunistic cleanup occurs
when a query happens to notice that a heap page that it had to read as
part of query processing needed to be cleaned up in passing. We do it
there and then because it happens to be relatively cheap and
convenient to do it that way. That sort of thing.

Overall, I think that this is suitable to commit, and I don't want to
make too much of a fuss. It's great that we're doing this.

Thanks
-- 
Peter Geoghegan



Re: documentation on HOT

От
"David G. Johnston"
Дата:
On Sat, Jul 23, 2022 at 11:34 AM Peter Geoghegan <pg@bowt.ie> wrote:
On Sat, Jul 23, 2022 at 8:51 AM Bruce Momjian <bruce@momjian.us> wrote:
> Good points.  I have updated the attached patch and URL to mention that
> HOT rows are _completely_ removed, and why that is possible, and I
> clarified the page item identifier mention.

I think that this version looks very good, but I do have some minor notes:

* You wrote "Specifically, updates cause additional rows to be added to tables."

Perhaps this could be rephrased: "Specifically, updates add new
physical tuples to tables to represent each new version."

I think that the term "row" should only refer to the simple/abstract
idea of a row from a table, while the term tuple should be preferred
when referring to a physical embodiment of a row, like one version of
a row. Perhaps it's worth following that convention across the board
here (not just in this sentence that I have highlighted).

I concur, suggesting the following:

"Specifically, updates result in multiple rows versions (tuples) existing on the table."

"There is sufficient free space on the page containing the old tuple for the updated tuple."

"Old tuples can be completely removed..."


Overall, I think that this is suitable to commit, and I don't want to
make too much of a fuss. It's great that we're doing this.


Agreed.  The other suggestion listed are not clear-cut winners in my mind.

The following, though, seems to just come out of nowhere.  It would be better setup as a "(See <link> for why this is possible.)" instead of dropping "page item identifiers" on the reader.

+     This removal is possible because indexes
+     do not reference their <link linkend="storage-page-layout">page
+     item identifiers</link>.

As a related thought, this has done a great job of being usable for a DBA operating at a high-level of system knowledge and interaction.  I don't think burying it in storage.sgml is desirable,  Maybe "Performance Tips" under "Avoid Unnecessary Indexes" (yes, a bit of a stretch, but nothing else seems to fit better, except maybe in concurrency control since we are discussing overcoming the limitation of our concurrency control choice.

Summary paragraph:
"can only happen if" => "can only be created if"

David J.

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Sat, Jul 23, 2022 at 11:33:40AM -0700, Peter Geoghegan wrote:
> On Sat, Jul 23, 2022 at 8:51 AM Bruce Momjian <bruce@momjian.us> wrote:
> > Good points.  I have updated the attached patch and URL to mention that
> > HOT rows are _completely_ removed, and why that is possible, and I
> > clarified the page item identifier mention.
> 
> I think that this version looks very good, but I do have some minor notes:
> 
> * You wrote "Specifically, updates cause additional rows to be added to tables."
> 
> Perhaps this could be rephrased: "Specifically, updates add new
> physical tuples to tables to represent each new version."

Uh, that seems more confusing than what I have.  I also considered
"tuples", but if you are saying "old version of a row", you are taking
about an old version of a logical row, not an old version of a physical
tuple, really.

> I think that the term "row" should only refer to the simple/abstract
> idea of a row from a table, while the term tuple should be preferred
> when referring to a physical embodiment of a row, like one version of
> a row. Perhaps it's worth following that convention across the board
> here (not just in this sentence that I have highlighted).

Yes, if we were talking about tuples unrelated to the versions of the
rows they represent, then yes, it would make sense.

> * You wrote "This can also require new index entries for each updated
> row, and removal of old versions of rows can be expensive"
> 
> I believe that the operative word in this sentence (which appears in
> the first paragraph) is "can". I think that it would be good to go
> just a bit further with that. Maybe add another sentence immediately
> afterwards that conveys "and now we're going to discuss when and how
> new versions from updates can sometimes avoid the need for a new round
> of index entries".

I ended up adding index cleanup further up in the text --- please see my
patch in the next email I send in this thread.
> 
> * You wrote "New index entries are not needed to represent updated rows"
> 
> It seems to me that this undersells the key benefit. You could perhaps
> add another sentence. Something like: "This avoids the immediate cost
> of adding new successor versions to each and every index, and avoids
> the cost of removing the obsolete versions from each and every index
> later on."

Same.

> 
> * You refer to opportunistic pruning as something that happens "during
> normal operation", but that doesn't seem to get the idea of
> "opportunistic" across.
> 
> It seems like it would be worth writing a sentence or two more on
> this, just to get that aspect across. Opportunistic cleanup occurs
> when a query happens to notice that a heap page that it had to read as
> part of query processing needed to be cleaned up in passing. We do it
> there and then because it happens to be relatively cheap and
> convenient to do it that way. That sort of thing.

Yes, we need that, but not in this section --- I would like to it
though.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson




Re: documentation on HOT

От
Bruce Momjian
Дата:
On Sat, Jul 23, 2022 at 12:12:38PM -0700, David G. Johnston wrote:
> I concur, suggesting the following:
> 
> "Specifically, updates result in multiple rows versions (tuples) existing on
> the table."

See my reply to Peter G. about tuples.

> "There is sufficient free space on the page containing the old tuple for the
> updated tuple."

Same.

> "Old tuples can be completely removed..."

It is really old row versions, not old tuples since tuples, in my mind,
don't have a chain of versions.  There are cannot-be-viewed-by-anyone
tuples, but that is not the point here.

>     Overall, I think that this is suitable to commit, and I don't want to
>     make too much of a fuss. It's great that we're doing this.
> 
> 
> 
> Agreed.  The other suggestion listed are not clear-cut winners in my mind.
> 
> The following, though, seems to just come out of nowhere.  It would be better
> setup as a "(See <link> for why this is possible.)" instead of dropping "page
> item identifiers" on the reader.
> 
> +     This removal is possible because indexes
> +     do not reference their <link linkend="storage-page-layout">page
> +     item identifiers</link>.

I added parentheses around that.
> 
> As a related thought, this has done a great job of being usable for a DBA
> operating at a high-level of system knowledge and interaction.  I don't think
> burying it in storage.sgml is desirable,  Maybe "Performance Tips" under "Avoid
> Unnecessary Indexes" (yes, a bit of a stretch, but nothing else seems to fit
> better, except maybe in concurrency control since we are discussing overcoming
> the limitation of our concurrency control choice.

Uh, not sure.  Anyone else have an opinion?

> Summary paragraph:
> "can only happen if" => "can only be created if"

Yes, good point.

Updated patch attached.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения

Re: documentation on HOT

От
Bruce Momjian
Дата:
On Tue, Jul 26, 2022 at 07:29:10PM -0400, Bruce Momjian wrote:
> > Summary paragraph:
> > "can only happen if" => "can only be created if"
> 
> Yes, good point.
> 
> Updated patch attached.

I applied this patch back to PG 11.  (PG 10 had too many conflicts.)  I
also added a mention about how indexes can prevent HOT to the index
introduction section;  applied patch of that addition attached.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Indecision is a decision.  Inaction is an action.  Mark Batterson


Вложения