Re: documentation structure

On Thu, Mar 21, 2024 at 12:43 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> which is a bit odd: why are the two WAL chapters in the middle of the
> chapters 62 and 63 talking about AMs?  Maybe put 66 right after 63
> instead.    Also, is it really better to have 62/63 first and 66
> later?  It sounds to me like 66 is more user-oriented and the other two
> are developer-oriented, so I'm inclined to suggest putting them the
> other way around, but I'm not really sure about this.  (Also, starting
> chapter 66 straight with 66.1 BTree without any intro text looks a bit
> odd; maybe one short introductory paragraph is sufficient?)

I had similar thoughts. I think that we should consider some changes
to the chapter ordering, but I didn't want to try to change too many
things all at once, since somebody only has to hate one thing about
the patch to sink the whole thing.

But since you brought it up, what I've been thinking about is that the
whole division into parts might need to be rethought a bit. I feel
like "VII. Internals" is a mix of about four different kinds of
content. First, the biggest portion of it is information about
developing certain kinds of C extensions -- all the "Writing a
Whatever" chapters, the "Whatever Access Method Interface Definition"
chapters, "Generic WAL Records", "Custom WAL Resource Managers", and
all the index-related chapters. Second, we've got some information
that I think is mainly of interest to people developing PostgreSQL
itself, namely, "PostgreSQL Coding Conventions", "Native Language
Support", and "System Catalog Declarations and Initial Contents". You
*might* care about these if you're developing extensions, or even if
you're not a developer at all, but then again you might not. Third,
we've got some reference material, namely "System Catalogs", "System
Views", and perhaps "Frontend/Backend Protocol". I distinguish these
from the previous two categories because I think you could care about
this stuff as a random user, or a developer of products that
interoperate with PostgreSQL but don't link with it or share any
common code. Finally, there's just a bunch of random bits and bobs
that we've decided to document here for one reason or another, either
because somebody else did a bunch of the work, like "Overview of
PostgreSQL Internals", or because some developer did something and
someone said "hey, that should be documented!", like "Backup Manifest
Format."

So my first thought is to pull out the stuff that's mainly for
PostgreSQL core developers and move it to an appendix. I propose we
create an appendix called "Developer Guide" and that it absorb the
existing appendix I, "The Source Code Repository", possibly all or
part of appendix J, "Documentation", and the chapters from "VII.
Internals" that are mostly of developer interest. I think that
possibly some of what's in "J. Documentation" should actually be moved
into the "Installation" chapter where we talk about building the
source code, because it doesn't make much sense to document the build
tool chain in one part of the documentation and the documentation
toolchain someplace else entirely, but "J.6. Style Guide" is developer
information, not build instructions.

My second thought is that the stuff from "VII. Internals" that I
categorized as reference material should move into section "VI.
Reference". I think we should also consider moving appendix F,
"Additional Supplied Modules and Extensions," and appendix G,
"Additional Supplied Programs" to the reference section. However,
prior to doing that, I think that appendix G needs some cleanup or
possibly we should just find a way to remove it outright. We're
shipping an appendix G with two major subsections, one of which is
completely empty and has been since v14, and the other of which
contains only two things. I think we should just remove the empty
sub-section entirely. I'm not sure what to do about the only with only
2 things in it (vacuumlo and oid2name). Would it be a bad idea to just
merge those bits into the client applications reference section?

My third thought is about what to do with the material in "VII.
Internals" that is about developing specific kind of extensions, like,
say, "Writing a Foreign Data Wrapper." If you look at "V. Server
Programming", you see that we actually have some very similar sections
there, like chapter 47, "Background Worker Processes" and chapter 50,
"Archive Modules". I think it's not very clear in the current
structure where topics that are relevant for extension developers
should go under "Server Programming" or under "Internals", and it
looks to me like different people have just done different things and
it's all a bit haphazard. One idea is to decide that the correct
answer is "Server Programming" and move all of the internals chapters
that fall into this category over to there. I don't think this is the
right answer, because that section also contains information about a
bunch of stuff that's strictly SQL-level, like rules and triggers. So
what I think we should do is create either [A] a new top-level part,
just before or just after what's currently called "VI. Reference" or
[B] a new appendix or [C] a new "Reference" section, that is
specifically for documentation of server APIs intended for extension
use. And then all the chapters under "V. Server Programming" or "VII.
Internals" that are documenting APIs would get moved there.

If we adopted all of the patches that I proposed in my previous email
and all of the suggestions that I just dropped in the preceding wall
of text, then the internals section would be left with only these
chapters:

- Overview of PostgreSQL Internals
- Genetic Query Optimizer
- Database Physical Storage
- Transaction Processing
- How the Planner Uses Statistics
- Backup Manifest Format

A lot of those chapters are pretty dated and maybe not that useful in
2024, but this email is already long enough and full of
sufficiently-aggressive proposals that I'm not inclined to opine too
much further on what we might want to do if and when we've done
everything I just proposed. For now, suffice it to say that I think we
might choose to either rewrite and expand some of these to make them
more useful, or demote some of them to some less prominent place in
the documentation, or just delete some of them entirely; but we can
figure that out if and when we get there.

> I was looking at the PL chapters earlier today too, wondering whether
> this would be valuable; but I worry that there are too many
> sub-sub-sections there, so it could end up being a bit messy.  I didn't
> look at the resulting output though.

That thought occurred to me as well. I certainly think that if we
perform the sort of aggressive purging of the top-level index for
which I'm advocating, there are going to be some people who are grumpy
that the stuff they're trying to find isn't where it used to be, or
who legitimately had trouble finding the content that they want. It
seems to me that if you're looking for the documentation on one of the
individual procedural languages and you don't see it, you'll try
clicking on "Procedural Languages" and then you'll find that it's now
under there. Now, what's maybe a bit unfortunate is that chapter
indexes only show two levels of section headings, and the PL/pgsql
chapter in particular has a lot of <sect2> items. If those get demoted
to <sect3> as I am proposing, they won't show up the chapter index any
more. I do think there's a possibility that this could be a problem
for someone.

On the other hand, the table of contents for
https://www.postgresql.org/docs/devel/plpgsql.html is so long right
now that it doesn't fit on the page, so maybe losing a level of
subsections won't be so bad. Alternately, maybe we could revise the
structure of the section a bit to ameliorate the problem. It seems to
me that most of the first-level section headers are actually pretty
clear about what you're likely to find underneath. If you're looking
for WHILE and you see a section called "Control Structures", it seems
like you're chances of guessing that WHILE will be underneath that
section are pretty good. The major exception that I see is 45.2,
"Basic Statements," which isn't very clear about what might be covered
there. But what if we split that apart into separate sections called
"Assignment", "Executing SQL", and "Doing Nothing at All"? And maybe
we'd even pull "Returning" out of "Control Structures" as well. I
think that would be clear enough for people to find what they need
without the extra level of headers.

(For the sake of completeness, let me note that PL/python and PL/perl
have a few <sect2> headings as well, but I don't think it would create
a problem for users if all of those got changed to <sect3>. PL/Tcl has
no <sect2> headings.)

I'm not sure if this kind of rearrangement is actually necessary or
not; but my point here is that if we think that people will have
problems or we find out that they actually did have problems, we can
look at doing this kind of stuff to compensate. What I don't think we
should do is decide that the only workable solution is to keep having
so many separate chapters at the top level. We're way, way beyond the
point where you can easily find anything on that page, and trying to
emphasize everything just ends up emphasizing nothing. We need to push
in a direction where every chapter and every appendix is expected to
have a large amount of content under it, so that the top-level index
becomes a way of finding the kind of content you want (SQL reference
pages, extension APIs, built-in SQL-callable functions, whatever) and
then you use that page to find the specific content that you want
within that category.

--
Robert Haas
EDB: http://www.enterprisedb.com