Обсуждение: New 'pg' consolidated metacommand patch

Hackers,

Attached is a patch for a `pg' command that consolidates various PostgreSQL functionality into a single command, along
thelines of how `git' commands are run from a single 'git' executable.  In other words, 

  `pg upgrade`   # instead of `pg_upgrade`
  `pg resetwal`   # instead of `pg_resetwal`

This has been discussed before on the -hackers list, but I don't recall seeing a patch.  I'm submitting this patch
mostlyas a way of moving the conversation along, fully expecting the community to want some (or all) of what I wrote to
bechanged. 

I'd also appreciate +1 and -1 votes on the overall idea, in case this entire feature, regardless of implementation, is
simplysomething the community does not want. 

Once again, this is mostly intended as a starting point for discussion.


The patch moves some commands from BINDIR to LIBEXECDIR where `pg' expects to find them.  For commands named pg_foo,
theexecutable is still named pg_foo and the sources are still located in src/bin/pg_foo/, but the command can now be
runas `pg foo`, `pg foo --version`, `pg foo FOO SPECIFIC ARGS`, etc. 

The command pgbench (no underscore) maps to 'pg bench'.

Commands without a "pg" prefix stay the same, so "createdb" => "pg createdb", etc.

The 'psql' and 'postgres'  executables (and the 'postmaster' link) have been left in BINDIR, as has 'ecpg'.  The 'pg'
executablehas been added to BINDIR. 

All other executables have been moved to LIBEXECDIR where they retain their old names and can still be run directly
fromthe command line.  If we committed this patch for v14, I think it makes sense that packagers could put the
LIBEXECDIRin the PATH so that 3rd-party scripts which call pg_ctl, initdb, etc. continue to work.  For that reason, I
didnot change the names of the executables, merely their location.  During conversations with Robert off-list, we
discussedrenaming the executables to things like 'pg-ctl' (hyphen rather than underscore), mostly because that's the
moremodern way of doing it and follows what 'git' does.  To avoid breaking scripts that execute these commands by the
oldname, this patch doesn't go that far.  It also leaves the usage() functions alone such that when they report their
ownprogname in the usage text, they do so under the old name.  This would need to change at some point, but I'm unclear
onwhether that would be for v14 or if it would be delayed. 

The binaries 'createuser' and 'dropuser' might be better named 'createrole' and 'droprole'.  I don't currently have
aliasesin this patch, but it might make sense to allow 'pg createrole' as a synonym for 'pg createuser' and 'pg
droprole'as a synonym for 'pg dropuser'.  I have not pursued that yet, largely because as soon as you go that route, it
startsmaking sense to have things like 'pg drop user', 'pg cluster db' and so forth, with the extra spaces.  How far
wouldpeople want me to go in this direction? 

Prior to this patch, postgres binaries that need to execute other postgres binaries determine the BINDIR using
find_my_execand trimming off their own executable name.  They can then assume the other binary is in that same
directory. After this patch, binaries need to find the common prefix ROOTDIR = commonprefix(BINDIR,LIBEXECDIR) and then
assumethe other binary is either in ROOTDIR/binsuffix or ROOTDIR/libexecsuffix.  This may cause problems on windows if
BINDIRand LIBEXECDIR are configured on different drives, as there won't be a common prefix of C:\my\pg\bin and
D:\my\pg\libexec. I'm hoping somebody with more Windows savvy expresses an opinion about how to handle this. 

The handling of the old libexec directory in pg_upgrade is not as robust as it could be.  I'll look to improve that for
asubsequent version of the patch, assuming the overall idea of the patch seems acceptable. 

I've updated some of the doc/sgml/* files, but don't want to spend too much time changing documentation until we have
someconsensus that the patch is moving in the right direction. 



—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

> On May 26, 2020, at 4:59 PM, David G. Johnston <david.g.johnston@gmail.com> wrote:
>
> On Tue, May 26, 2020 at 4:19 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> I'd also appreciate +1 and -1 votes on the overall idea, in case this entire feature, regardless of implementation,
issimply something the community does not want. 
>
> -1, at least as part of core.  My question would be how much of this is would be needed if someone were to create an
externalproject that installed a "pg" command on top of an existing PostgreSQL installation.  Or put differently, how
manyof the changes to the existing binaries are required versus nice-to-have? 

If the only goal of something like this were to have a frontend that could execute the various postgres binaries, then
I'dsay no changes to those binaries would be needed, and the frontend would not be worth very much.  The value in
havingthe frontend is that it makes it less difficult to introduce new commands to the postgres suite of commands, as
youdon't need to worry about whether another executable by the same name might happen to already exist somewhere.  Even
introducinga command named "pg" has already gotten such a response on this thread.  By having the commands installed in
postgres'slibexec rather than bin, you can put whatever commands you want in libexec without worrying about conflicts.
Thatstill leaves open the question of whether existing commands get moved into libexec, and if so, if they keep the
samename.  An external project for this would be worthless in this regard, as the community wouldn't get any benefit
whendebating the merits of introducing a new command vs. the potential for conflicts. 

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

> On May 27, 2020, at 1:13 AM, Dave Page <dpage@pgadmin.org> wrote:
>
> Hi
>
> On Wed, May 27, 2020 at 12:19 AM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>
> I think it makes sense that packagers could put the LIBEXECDIR in the PATH so that 3rd-party scripts which call
pg_ctl,initdb, etc. continue to work.   
>
> Having packages that futz with the PATH is generally a bad idea, especially those that support side-by-side
installationsof different versions. None of ours (EDBs) will be doing so. 

I probably phrased that badly.  The operative word in that sentence was "could".  If we rename the binaries, people can
stillmake links to them from the old name, but if we don't rename them, then either links or PATH changes *could* be
used. I'm not trying to recommend any particular approach.  Mentioning "packagers" probably wasn't helpful, as "people"
worksjust as well in that sentence. 

There is also the option of not moving the binaries at all, and only putting new commands into libexec, while
grandfatheringexisting ones in bin. 

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

> On May 27, 2020, at 1:50 AM, Magnus Hagander <magnus@hagander.net> wrote:
>
> On Wed, May 27, 2020 at 1:19 AM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> Hackers,
>
> Attached is a patch for a `pg' command that consolidates various PostgreSQL functionality into a single command,
alongthe lines of how `git' commands are run from a single 'git' executable.  In other words, 
>
>   `pg upgrade`   # instead of `pg_upgrade`
>   `pg resetwal`   # instead of `pg_resetwal`
>
> This has been discussed before on the -hackers list, but I don't recall seeing a patch.  I'm submitting this patch
mostlyas a way of moving the conversation along, fully expecting the community to want some (or all) of what I wrote to
bechanged. 
>
> As mentioned at least once before, the "pg" name is already taken in posix. Granted it has been removed now, but it
wasremoved from posix in 2018, which I think is nowhere near soon enough to "steal. See for example
https://en.wikipedia.org/wiki/Pg_(Unix)

Care to recommend a different name?

> All other executables have been moved to LIBEXECDIR where they retain their old names and can still be run directly
fromthe command line.  If we committed this patch for v14, I think it makes sense that packagers could put the
LIBEXECDIRin the PATH so that 3rd-party scripts which call pg_ctl, initdb, etc. continue to work.  
>
> I would definitely not expect a packager to change the PATH, as also mentioned by others. More likely options would
beto symlink the binaries into the actual  bindir, or just set both those directories to the same one (in the path) for
anumber of releases as a transition. 

There is nothing in the patch that expects packagers to muck with the PATH.  The idea, badly phrased, was that by
keepingthe names of the executables and only changing locations, people would have more options for how to deal with
thechange. 

> But you should definitely poll the packagers separately to make sure something is done that works well for them --
especiallywhen it comes to integrating with for example the debian/ubuntu wrapper system that already supports multiple
parallelinstalls. And mind that they don't typically follow hackers actively (I think), so it would be worthwhile to
bringtheir attention specifically to the thread. In many ways I'd find them more important to get input from than most
"otherhackers" :) 

Yeah, good advice.  Since I've already floated this on -hackers, I might wait a few days for comment, then if it looks
encouraging,ask on other lists. 

> For that reason, I did not change the names of the executables, merely their location.  During conversations with
Robertoff-list, we discussed renaming the executables to things like 'pg-ctl' (hyphen rather than underscore), mostly
becausethat's the more modern way of doing it and follows what 'git' does.  To avoid breaking scripts that execute
thesecommands by the old name, this patch doesn't go that far.  It also leaves the usage() functions alone such that
whenthey report their own progname in the usage text, they do so under the old name.  This would need to change at some
point,but I'm unclear on whether that would be for v14 or if it would be delayed. 
>
> Ugh, yeah, please don't do that. Renaming them just to make it "look more modern" helps nobody, really. Especially if
thesuggestion is people should be using the shared-launcher binary anyway.  
>
> usage() seems more reasonable to change as part of a patch like this.
>
>
> The binaries 'createuser' and 'dropuser' might be better named 'createrole' and 'droprole'.  I don't currently have
aliasesin this patch, but it might make sense to allow 'pg createrole' as a synonym for 'pg createuser' and 'pg
droprole'as a synonym for 'pg dropuser'.  I have not pursued that yet, largely because as soon as you go that route, it
startsmaking sense to have things like 'pg drop user', 'pg cluster db' and so forth, with the extra spaces.  How far
wouldpeople want me to go in this direction? 
>
> I'd say a createrole would make sense, but certainly not a "create role". You'd end up with unlimited number of
commands.But in either of them, I'd say keep aliases completely out of it for a first iteration. 

Ok.

> Prior to this patch, postgres binaries that need to execute other postgres binaries determine the BINDIR using
find_my_execand trimming off their own executable name.  They can then assume the other binary is in that same
directory. After this patch, binaries need to find the common prefix ROOTDIR = commonprefix(BINDIR,LIBEXECDIR) and then
assumethe other binary is either in ROOTDIR/binsuffix or ROOTDIR/libexecsuffix.  This may cause problems on windows if
BINDIRand LIBEXECDIR are configured on different drives, as there won't be a common prefix of C:\my\pg\bin and
D:\my\pg\libexec. I'm hoping somebody with more Windows savvy expresses an opinion about how to handle this. 
>
> Maybe the "pg" binary could just pass down it's own location as a parameter to the binary it calls, thereby making
surethat binary has direct access to both?  

Perhaps.  Thus far, I've avoided making the binaries dependent on being called from 'pg'.  Having them depend on a
parameterthat 'pg' passes would be a big change. 

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

## Magnus Hagander (magnus@hagander.net):

> Ugh, yeah, please don't do that. Renaming them just to make it "look more
> modern" helps nobody, really. Especially if the suggestion is people should
> be using the shared-launcher binary anyway.

Quick, let's invent a fancy name like "microcommand" for doing this
like we're used to; then we tell people that's the new "modern" (anybody
cares to write a Medium article for that? (why Medium? it's neither
Rare nor Well Done)). What might make sense for (some) version control
systems and is tempting in languages which have forgotten howto shared
library might not be the best architecture for everything. What has
become of the collection of small dedicated tools?

Regards,
Christoph

-- 
Spare Space

On Wed, May 27, 2020 at 4:51 AM Magnus Hagander <magnus@hagander.net> wrote:
> As mentioned at least once before, the "pg" name is already taken in posix. Granted it has been removed now, but it
wasremoved from posix in 2018, which I think is nowhere near soon enough to "steal. See for example
https://en.wikipedia.org/wiki/Pg_(Unix)

The previous discussion of this general topic starts at
http://postgr.es/m/CA+TgmoZQmDY7nLrQ96nLm-wrnmNPY90qdMvZ6LtJO941GwgLMg@mail.gmail.com
and the discussion of this particular issue starts at
https://www.postgresql.org/message-id/15135.1586703479%40sss.pgh.pa.us

I think I agree with what Andres said on that thread: rather than
waiting a long time to see what happens, we should grab the name
before somebody else does. As also discussed on that thread, perhaps
we should have the official name of the binary be 'pgsql' with 'pg' as
a symlink that some packagers might choose to omit.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, May 27, 2020 at 4:51 AM Magnus Hagander <magnus@hagander.net> wrote:
>> For that reason, I did not change the names of the executables, merely their location.  During conversations with
Robertoff-list, we discussed renaming the executables to things like 'pg-ctl' (hyphen rather than underscore), mostly
becausethat's the more modern way of doing it and follows what 'git' does.  To avoid breaking scripts that execute
thesecommands by the old name, this patch doesn't go that far.  It also leaves the usage() functions alone such that
whenthey report their own progname in the usage text, they do so under the old name.  This would need to change at some
point,but I'm unclear on whether that would be for v14 or if it would be delayed. 
>
> Ugh, yeah, please don't do that. Renaming them just to make it "look more modern" helps nobody, really. Especially if
thesuggestion is people should be using the shared-launcher binary anyway. 

The way things like 'git' work is that 'git thunk' just looks in a
designated directory for an executable called git-thunk, and invokes
it if it's found. If you want to invent your own git subcommand, you
can. I guess 'git help' wouldn't know to list it, but you can still
get the metacommand to execute it. That only works if you use a
standard naming, though. If the meta-executable has to hard-code the
names of all the individual executables that it calls, then you can't
really make that work.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2020-05-27 01:19, Mark Dilger wrote:
> Attached is a patch for a `pg' command that consolidates various PostgreSQL functionality into a single command,
alongthe lines of how `git' commands are run from a single 'git' executable.  In other words,
 
> 
>    `pg upgrade`   # instead of `pg_upgrade`
>    `pg resetwal`   # instead of `pg_resetwal`
> 
> This has been discussed before on the -hackers list, but I don't recall seeing a patch.  I'm submitting this patch
mostlyas a way of moving the conversation along, fully expecting the community to want some (or all) of what I wrote to
bechanged.
 
> 
> I'd also appreciate +1 and -1 votes on the overall idea, in case this entire feature, regardless of implementation,
issimply something the community does not want.
 

I'm not excited about this.

First, consider that git has over 170 subcommands.  PostgreSQL currently 
has 36, and we're probably not going to add dozens more any time soon. 
So the issue is not of the same scope.  It also seems to me that the way 
git is organized this becomes a self-perpetuating system:  They are 
adding subcommands all the time without much care where you might in 
other situations think harder about combining them and keeping the 
surface area smaller.  For example, we wouldn't really need separate 
commands clusterdb, reindexdb, vacuumdb if we had better support in psql 
for "run this command in each database [in parallel]".

git (and svn etc. before it) also has a much more consistent operating 
model that is sensible to reflect in the command structure.  They all 
more or less operate on a git repository, in apparently 170 different 
ways.  The 36 PostgreSQL commands don't all work in the same way.  Now 
if someone were to propose a way to combine server tools, perhaps like 
pginstancetool {init|controldata|resetwal|checksum}, and perhaps also in 
a way that actually saves code duplication and inconsistency, that would 
be something to consider.  Or maybe a client-side tool that does 
pgclienttool {create user|drop user|create database|...} -- but that 
pretty much already exists by the name of psql.  But just renaming 
everything that's shipped with PostgreSQL to one common bucket without 
regard to how it actually works and what role it plays would be 
unnecessarily confusing.

Also consider some practical concerns with the command structure you 
describe: Tab completion of commands wouldn't work anymore, unless you 
supply custom tab completion setups.  The direct association between a 
command and its man page would be broken.  Shell scripting becomes more 
challenging:  Instead of writing common things like "if which 
pg_waldump; then" you'd need some custom code, to be determined.  These 
are all solvable, but just a sum of slight annoyances, for no real benefit.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

> On May 27, 2020, at 2:42 PM, Christopher Browne <cbbrowne@gmail.com> wrote:
>
> There is merit to having a new, harmonious set of "pg commands."  But it's eminently easy to get into trouble (and
getpeople mad) by changing things that have been working fine for many years.  Half the battle (against the "getting
peoplemad" part) is making sure that it's clear that people were listened to.  Listening to the community is one of the
importantthings to do :-). 

I totally agree.

There are options for keeping the existing tools and not modifying them.  If the "pg" command (or "pgsql" command, if
weuse that naming) knows, for example, how to execute pg_ctl, that's no harm to people who prefer to just run pg_ctl
directly. It only becomes a problem when this patch, or one like it, decides that "pg_ctl" needs to work differently,
havea different set of command line options, etc.  The only thing I changed about pg_ctl and friends in the v1 patch is
thatthey moved from BINDIR to LIBEXECDIR, and internally they were updated to be able to still work despite the move.
Thatchange was partly designed to spark conversation.  If people prefer they get moved back into BINDIR, fine by me.
Ifpeople instead prefer that the patch include links between the old BINDIR location and the new LIBEXECDIR location,
that'salso fine by me.  The "pg" command doesn't really care either.  I'm intentionally not calling the shots here.
I'masking the community members, many of whom expressed an interest in something along the lines of this patch.  I'm
happyto do the grunt work of the patch to meet the community needs. 

Dave Page expressed an interest upthread in standardizing the interfaces of the various commands.  He didn't say this,
butI assume he is thinking about things like -d meaning --debug in initdb but meaning --dbname=CONNSTR in
pg_basebackup. We could break backwards compatibility by changing one or both of those commands to interpret those
optionsin some new standardized way.  Or, we could preserve backwards compatibililty by having "pg" take --dbname and
--debugoptions and pass them to the subcommand according to the grandfathered rules of the subcommand.  I tend towards
preservingcompability, but maybe somebody on this list wants to argue for the other side?  For new commands introduced
afterthis patch gets committed (assuming it does), options could be passed from "pg" through to the subcommand
unmolested. That supports Robert's idea that people could install new subcommands from contrib modules without the "pg"
commandneeding to know anything about them.  This, too, is still open to conversation and debate. 

I'd like to hear from more community members on this.  I'm listening.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2020-05-27 23:42, Christopher Browne wrote:
> d) systemd is a Controversial System; the folk that seem particularly 
> irate about it seem to be "Old Bearded Sysadmins" that hate the idea of 
> redoing their understandings of how Unix systems initialize. Personally, 
> my feelings are ambivalent; I'm using it where I find some use, and have 
> not been displeased with my results.  And since modern systems now have 
> USB and network devices added and dropped on a whim, there's a critical 
> need for something newer with more dynamic responses than old SysV 
> Init.  But I certainly "get" that some aren't so happy with it, and I'm 
> not thrilled at the ongoing scope creep that never seems to end.

It is worth noting that systemd did not go for a one-binary-for-all 
approach.  It has different binaries for different parts of the 
functionality.  systemctl for controlling services, journalctl for 
controlling the journal, etc.  Just as a data point to show that there 
is no single "new" way to do things.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, May 27, 2020 at 4:00 PM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> First, consider that git has over 170 subcommands.  PostgreSQL currently
> has 36, and we're probably not going to add dozens more any time soon.
> So the issue is not of the same scope.  It also seems to me that the way
> git is organized this becomes a self-perpetuating system:  They are
> adding subcommands all the time without much care where you might in
> other situations think harder about combining them and keeping the
> surface area smaller.  For example, we wouldn't really need separate
> commands clusterdb, reindexdb, vacuumdb if we had better support in psql
> for "run this command in each database [in parallel]".

I see your point here, but I think it's clear that some people are
already concerned about the proliferation of binaries on namespace
grounds. For example, it was proposed that pg_verifybackup should be
part of pg_basebackup, and then later and by someone else that it
should be part of pg_checksums. It's quite different from either of
those things and I'm pretty confident it shouldn't be merged with
either one, but there was certainly pressure in that direction. So
apparently for some people 36 is already too many. It's not clear to
me why it should be a problem to have a lot of commands, as long as
they all start with "pg_", but if it is then we should do something
about it rather than waiting until we have more of them.

> git (and svn etc. before it) also has a much more consistent operating
> model that is sensible to reflect in the command structure.  They all
> more or less operate on a git repository, in apparently 170 different
> ways.  The 36 PostgreSQL commands don't all work in the same way.  Now
> if someone were to propose a way to combine server tools, perhaps like
> pginstancetool {init|controldata|resetwal|checksum}, and perhaps also in
> a way that actually saves code duplication and inconsistency, that would
> be something to consider.  Or maybe a client-side tool that does
> pgclienttool {create user|drop user|create database|...} -- but that
> pretty much already exists by the name of psql.  But just renaming
> everything that's shipped with PostgreSQL to one common bucket without
> regard to how it actually works and what role it plays would be
> unnecessarily confusing.

This doesn't strike me as a very practical proposal because
"pginstancetool checksums" is really stinking long compared to
"pg_checksums", where as "pg checksums" is no different, or one
keystroke better if you assume that the underscore requires pressing
shift.

> Also consider some practical concerns with the command structure you
> describe: Tab completion of commands wouldn't work anymore, unless you
> supply custom tab completion setups.  The direct association between a
> command and its man page would be broken.  Shell scripting becomes more
> challenging:  Instead of writing common things like "if which
> pg_waldump; then" you'd need some custom code, to be determined.  These
> are all solvable, but just a sum of slight annoyances, for no real benefit.

There are some potential benefits, I think, such as:

1. It seems to be the emerging standard for command line interfaces.
There's not only the 'git' example but also things like 'aws', which
is perhaps more similar to the case proposed here in that there are a
bunch of subcommands that do quite different sorts of things. I think
a lot of developers are now quite familiar with the idea of a main
command with a bunch of subcommands, and they expect to be able to
type 'git help' or 'aws help' or 'pg help' to get a list of commands,
and then 'pg help createdb' for help with that. If you don't know what
pg commands exist today, how do you discover them? You're right that
not everyone is going this way but it seems to be pretty common
(kubectl, yum, brew, npm, heroku, ...).

2. It lowers the barrier to adding more commands. For example, as
Chris Browne says, we could have a 'pg completion' command to emit
completion information for various shells. True, that would be more
necessary with this proposal. But I bet there's stuff that could be
done even today -- I think most modern shells have pretty powerful
completion facilities. Someone could propose it, but what are the
chances that a patch adding a pg_completion binary would be accepted?
I feel like the argument that this is too narrow to justify the
namespace pollution is almost inevitable.

3. It might help us achieve some better consistency between commands.
Right now we have a lot of warts in the way things are named, like
pgbench vs. pg_ctl vs. createdb, and also pg_basebackup vs.
pg_test_fsync (why not pg_base_backup or pg_testfsync?). Standardizing
on something like this would probably help us be more consistent
there. Over time, we might be able to also clean other things up.
Maybe we could get all of our client-side utilities to share a common
config file, and have a 'pg config' utility to configure it. Maybe we
could have common options that are shared by all commands as 'git'
does. These things aren't impossible without unifying the interface,
but unifying the interface does help to make it clearer why the other
things should also be unified.

Now maybe it's just not worth it. I'm pretty sure that if we made this
change I would spend some time cursing this because my fingers would
type commands that don't work any more, and that could be annoying,
and I suspect a lot of other people might feel similarly. I don't
think this is something we HAVE to do, but I am a little worried that
we're otherwise locked into a system that isn't particularly scalable
to more commands and doesn't really have any sort of unifying design
either.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company