Обсуждение: [HACKERS] removing tsearch2

On Thu, Feb 9, 2017 at 7:37 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-02-09 19:19:21 -0500, Stephen Frost wrote:
>> * Robert Haas (robertmhaas@gmail.com) wrote:
>> > On Thu, Feb 9, 2017 at 4:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> > > Also, our experience with contrib/tsearch2 suggests that the extension
>> > > shouldn't be part of contrib, because we have zero track record of getting
>> > > rid of stuff in contrib, no matter how dead it is.
>> >
>> > Let's nuke tsearch2 to remove this adverse precedent, and then add the
>> > new thing.
>> >
>> > Anybody who still wants tsearch2 can go get it from an old version, or
>> > somebody can maintain a fork on github.
>>
>> Works for me.
>
> +1

OK, that's three votes in favor of removing tsearch2 (from core,
anyone who wants it can maintain a copy elsewhere).  Starting a new
thread to make sure we collect all the relevant votes, but I really,
really think it's past time for this to go away.  The last actual
change to tsearch2 which wasn't part of a wider cleanup was
3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
7 years since there's been any real work done on this -- and the
release where we brought tsearch into core is now EOL, plus three more
releases besides.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Feb 10, 2017 at 8:28 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Feb 9, 2017 at 7:37 PM, Andres Freund <andres@anarazel.de> wrote:
>> On 2017-02-09 19:19:21 -0500, Stephen Frost wrote:
>>> * Robert Haas (robertmhaas@gmail.com) wrote:
>>> > On Thu, Feb 9, 2017 at 4:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> > > Also, our experience with contrib/tsearch2 suggests that the extension
>>> > > shouldn't be part of contrib, because we have zero track record of getting
>>> > > rid of stuff in contrib, no matter how dead it is.
>>> >
>>> > Let's nuke tsearch2 to remove this adverse precedent, and then add the
>>> > new thing.
>>> >
>>> > Anybody who still wants tsearch2 can go get it from an old version, or
>>> > somebody can maintain a fork on github.
>>>
>>> Works for me.
>>
>> +1
>
> OK, that's three votes in favor of removing tsearch2 (from core,
> anyone who wants it can maintain a copy elsewhere).  Starting a new
> thread to make sure we collect all the relevant votes, but I really,
> really think it's past time for this to go away.  The last actual
> change to tsearch2 which wasn't part of a wider cleanup was
> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
> 7 years since there's been any real work done on this -- and the
> release where we brought tsearch into core is now EOL, plus three more
> releases besides.

+1.
-- 
Michael

> On 10 Feb 2017, at 12:28, Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Feb 9, 2017 at 7:37 PM, Andres Freund <andres@anarazel.de> wrote:
>> On 2017-02-09 19:19:21 -0500, Stephen Frost wrote:
>>> * Robert Haas (robertmhaas@gmail.com) wrote:
>>>> On Thu, Feb 9, 2017 at 4:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>>> Also, our experience with contrib/tsearch2 suggests that the extension
>>>>> shouldn't be part of contrib, because we have zero track record of getting
>>>>> rid of stuff in contrib, no matter how dead it is.
>>>>
>>>> Let's nuke tsearch2 to remove this adverse precedent, and then add the
>>>> new thing.
>>>>
>>>> Anybody who still wants tsearch2 can go get it from an old version, or
>>>> somebody can maintain a fork on github.
>>>
>>> Works for me.
>>
>> +1
>
> OK, that's three votes in favor of removing tsearch2 (from core,
> anyone who wants it can maintain a copy elsewhere).  Starting a new
> thread to make sure we collect all the relevant votes, but I really,
> really think it's past time for this to go away.  The last actual
> change to tsearch2 which wasn't part of a wider cleanup was
> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
> 7 years since there's been any real work done on this -- and the
> release where we brought tsearch into core is now EOL, plus three more
> releases besides.

FWIW, +1

cheers ./daniel

Robert Haas <robertmhaas@gmail.com> writes:
> OK, that's three votes in favor of removing tsearch2 (from core,
> anyone who wants it can maintain a copy elsewhere).  Starting a new
> thread to make sure we collect all the relevant votes, but I really,
> really think it's past time for this to go away.  The last actual
> change to tsearch2 which wasn't part of a wider cleanup was
> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
> 7 years since there's been any real work done on this -- and the
> release where we brought tsearch into core is now EOL, plus three more
> releases besides.

I'm abstaining on the actual vote, but I just wanted to note that there's
at least two layers of stub compatibility functions in core that are
there only to support tsearch2, cf commits
eb43e851d6b3fa0c4516efcfdf29a183f7717a43
6595dd04d136d5c97ae05fc580572c8f00042143
... and there might be more that I missed in quickly scanning the
commit log.  We should probably investigate whether those can be
gotten rid of without damaging the maintainability of the external
module.  I *think* these were only needed as short-term hacks but
don't want to say for sure because ENOCAFFEINE.
        regards, tom lane

On 2/10/17 6:28 AM, Robert Haas wrote:
> On Thu, Feb 9, 2017 at 7:37 PM, Andres Freund <andres@anarazel.de> wrote:
>> On 2017-02-09 19:19:21 -0500, Stephen Frost wrote:
>>> * Robert Haas (robertmhaas@gmail.com) wrote:
>>>> On Thu, Feb 9, 2017 at 4:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>>> Also, our experience with contrib/tsearch2 suggests that the extension
>>>>> shouldn't be part of contrib, because we have zero track record of getting
>>>>> rid of stuff in contrib, no matter how dead it is.
>>>>
>>>> Let's nuke tsearch2 to remove this adverse precedent, and then add the
>>>> new thing.
>>>>
>>>> Anybody who still wants tsearch2 can go get it from an old version, or
>>>> somebody can maintain a fork on github.
>>>
>>> Works for me.
>>
>> +1
> 
> OK, that's three votes in favor of removing tsearch2 (from core,
> anyone who wants it can maintain a copy elsewhere).  Starting a new
> thread to make sure we collect all the relevant votes, but I really,
> really think it's past time for this to go away.  The last actual
> change to tsearch2 which wasn't part of a wider cleanup was
> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
> 7 years since there's been any real work done on this -- and the
> release where we brought tsearch into core is now EOL, plus three more
> releases besides.

+1

-- 
-David
david@pgmasters.net

On Fri, Feb 10, 2017 at 9:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> OK, that's three votes in favor of removing tsearch2 (from core,
>> anyone who wants it can maintain a copy elsewhere).  Starting a new
>> thread to make sure we collect all the relevant votes, but I really,
>> really think it's past time for this to go away.  The last actual
>> change to tsearch2 which wasn't part of a wider cleanup was
>> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
>> 7 years since there's been any real work done on this -- and the
>> release where we brought tsearch into core is now EOL, plus three more
>> releases besides.
>
> I'm abstaining on the actual vote, but I just wanted to note that there's
> at least two layers of stub compatibility functions in core that are
> there only to support tsearch2, cf commits
> eb43e851d6b3fa0c4516efcfdf29a183f7717a43
> 6595dd04d136d5c97ae05fc580572c8f00042143
> ... and there might be more that I missed in quickly scanning the
> commit log.  We should probably investigate whether those can be
> gotten rid of without damaging the maintainability of the external
> module.  I *think* these were only needed as short-term hacks but
> don't want to say for sure because ENOCAFFEINE.

That's not a bad idea, but I think it's an independent issue.  If the
hacks are still needed for an external module, we shouldn't go out of
our way to remove them even if we nuke tsearch2 (but we don't need to
maintain them going forward unless we get a complaint).  If they hacks
aren't still needed, they could be removed whether or not we keep
tsearch2 in contrib.  Unless I'm confused?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:
> That's not a bad idea, but I think it's an independent issue.  If the
> hacks are still needed for an external module, we shouldn't go out of
> our way to remove them even if we nuke tsearch2 (but we don't need to
> maintain them going forward unless we get a complaint).  If they hacks
> aren't still needed, they could be removed whether or not we keep
> tsearch2 in contrib.  Unless I'm confused?

Technically, it's probably independent of whether we keep tsearch2 in
contrib.  I think (but haven't researched it) that it's more a matter
of whether we care to still support direct upgrades from pre-release-N
versions of tsearch2, for some N.  Politically though, I think it'll
be easier to make an aggressive decision in that regard if we are
tossing tsearch2 out of contrib.
        regards, tom lane

On 02/10/2017 06:41 AM, David Steele wrote:
> On 2/10/17 6:28 AM, Robert Haas wrote:
>> On Thu, Feb 9, 2017 at 7:37 PM, Andres Freund <andres@anarazel.de> wrote:
>>> On 2017-02-09 19:19:21 -0500, Stephen Frost wrote:
>>>> * Robert Haas (robertmhaas@gmail.com) wrote:
>>>>> On Thu, Feb 9, 2017 at 4:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>>>> Also, our experience with contrib/tsearch2 suggests that the extension
>>>>>> shouldn't be part of contrib, because we have zero track record of getting
>>>>>> rid of stuff in contrib, no matter how dead it is.
>>>>>
>>>>> Let's nuke tsearch2 to remove this adverse precedent, and then add the
>>>>> new thing.
>>>>>
>>>>> Anybody who still wants tsearch2 can go get it from an old version, or
>>>>> somebody can maintain a fork on github.
>>>>
>>>> Works for me.
>>>
>>> +1
>>
>> OK, that's three votes in favor of removing tsearch2 (from core,
>> anyone who wants it can maintain a copy elsewhere).  Starting a new
>> thread to make sure we collect all the relevant votes, but I really,
>> really think it's past time for this to go away.  The last actual
>> change to tsearch2 which wasn't part of a wider cleanup was
>> 3ca7eddbb7c4803729d385a0c9535d8a972ee03f in January 2009, so it's been
>> 7 years since there's been any real work done on this -- and the
>> release where we brought tsearch into core is now EOL, plus three more
>> releases besides.
> 
> +1
> 

+1

-- 
Josh Berkus
Containers & Databases Oh My!

On Fri, Feb 10, 2017 at 3:28 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> Works for me.
>>
>> +1
>
> OK, that's three votes in favor of removing tsearch2 (from core,
> anyone who wants it can maintain a copy elsewhere).

+1.

I'd also be in favor of either removing contrib/isn, or changing it so
that the ISBN country code prefix enforcement went away. That would
actually not imply and real loss of functionality from a practical
perspective, since you can still enforce that the check digit is
correct without any of that. I think that the existing design of some
parts of contrib/isn is just horrible.

-- 
Peter Geoghegan

On Fri, Feb 10, 2017 at 1:18 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> I'd also be in favor of either removing contrib/isn, or changing it so
> that the ISBN country code prefix enforcement went away. That would
> actually not imply and real loss of functionality from a practical
> perspective, since you can still enforce that the check digit is
> correct without any of that. I think that the existing design of some
> parts of contrib/isn is just horrible.

There's no question that contrib/isn is not well-designed, but whether
to remove it is a question for a different thread than this one.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 02/10/2017 10:18 AM, Peter Geoghegan wrote:
> On Fri, Feb 10, 2017 at 3:28 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>>> Works for me.
>>>
>>> +1
>>
>> OK, that's three votes in favor of removing tsearch2 (from core,
>> anyone who wants it can maintain a copy elsewhere).
> 
> +1.
> 
> I'd also be in favor of either removing contrib/isn, or changing it so
> that the ISBN country code prefix enforcement went away. That would
> actually not imply and real loss of functionality from a practical
> perspective, since you can still enforce that the check digit is
> correct without any of that. I think that the existing design of some
> parts of contrib/isn is just horrible.

+1 to quick-fix it, -1 to just delete it.

There's a bunch of these things in /contrib which really ought to be
PGXN extensions (also CUBE, earthdistance, etc.).  However, one of the
steps in that would be getting the mainstream platforms to package them
so that users have a reasonable upgrade path, so I would not propose
doing it for 10.

-- 
Josh Berkus
Containers & Databases Oh My!

On Fri, Feb 10, 2017 at 12:32 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> That's not a bad idea, but I think it's an independent issue.  If the
>> hacks are still needed for an external module, we shouldn't go out of
>> our way to remove them even if we nuke tsearch2 (but we don't need to
>> maintain them going forward unless we get a complaint).  If they hacks
>> aren't still needed, they could be removed whether or not we keep
>> tsearch2 in contrib.  Unless I'm confused?
>
> Technically, it's probably independent of whether we keep tsearch2 in
> contrib.  I think (but haven't researched it) that it's more a matter
> of whether we care to still support direct upgrades from pre-release-N
> versions of tsearch2, for some N.  Politically though, I think it'll
> be easier to make an aggressive decision in that regard if we are
> tossing tsearch2 out of contrib.

I agree that it's OK to desupport direct upgrades from ancient
releases to the very latest code at some point in time, and if
somebody thinks that's important work to reduce future maintenance or
just to tidy up, I'm not going to oppose it strenuously.  But I
wouldn't be in any rush either.

Anyway, it seems like the consensus here is unanimous.  Unless there
are a LARGE number of contrary votes in the meanwhile, I'll go make
this happen sometime next week.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 02/10/2017 01:27 PM, Josh Berkus wrote:
> On 02/10/2017 10:18 AM, Peter Geoghegan wrote:
>> On Fri, Feb 10, 2017 at 3:28 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>>>> Works for me.
>>>> +1
>>> OK, that's three votes in favor of removing tsearch2 (from core,
>>> anyone who wants it can maintain a copy elsewhere).
>> +1.
>>
>> I'd also be in favor of either removing contrib/isn, or changing it so
>> that the ISBN country code prefix enforcement went away. That would
>> actually not imply and real loss of functionality from a practical
>> perspective, since you can still enforce that the check digit is
>> correct without any of that. I think that the existing design of some
>> parts of contrib/isn is just horrible.
> +1 to quick-fix it, -1 to just delete it.
>
> There's a bunch of these things in /contrib which really ought to be
> PGXN extensions (also CUBE, earthdistance, etc.).  However, one of the
> steps in that would be getting the mainstream platforms to package them
> so that users have a reasonable upgrade path, so I would not propose
> doing it for 10.
>



Part of the reason for keeping a number of extensions is that it helps
test our extension infrastructure. Also they server as good pieces of
example code. So I don't want to get rid of them all, or even any of
them that have any degree of significant use. I think these days
tsearch2 is very largely redundant, so that means there's a good reason
not to keep it. But that's not true of cube, isn etc.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2/10/17 2:24 PM, Andrew Dunstan wrote:
>> There's a bunch of these things in /contrib which really ought to be
>> PGXN extensions (also CUBE, earthdistance, etc.).  However, one of the
>> steps in that would be getting the mainstream platforms to package them
>> so that users have a reasonable upgrade path, so I would not propose
>> doing it for 10.
>
> Part of the reason for keeping a number of extensions is that it helps
> test our extension infrastructure. Also they server as good pieces of
> example code. So I don't want to get rid of them all, or even any of
> them that have any degree of significant use. I think these days
> tsearch2 is very largely redundant, so that means there's a good reason
> not to keep it. But that's not true of cube, isn etc.

That's based on an assumption that PGXN shouldn't be treated as part of 
the community effort, which I think is a mistake. Having a robust, 
community run extension/package/module framework has proven to be 
extremely valuable for other programming environments, and IMHO we 
should be striving to improve in that area.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

On Fri, Feb 10, 2017 at 1:41 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Anyway, it seems like the consensus here is unanimous.  Unless there
> are a LARGE number of contrary votes in the meanwhile, I'll go make
> this happen sometime next week.

Done.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2/13/17 2:37 AM, Magnus Hagander wrote:
>     That's based on an assumption that PGXN shouldn't be treated as part
>     of the community effort, which I think is a mistake. Having a
>     robust, community run extension/package/module framework has proven
>     to be extremely valuable for other programming environments, and
>     IMHO we should be striving to improve in that area.
>
>
> Until pgxn has a way of helping users on for example Windows (or other
> platforms where they don't have a pgxs system and a compiler around),
> it's always going to be a "second class citizen".

I view that as more of a failing of pgxs than pgxn. Granted, the most 
common (only?) pgxn client right now is written in python, but it's 
certainly possible to run that on windows with some effort (BigSQL does 
it), and I'm fairly certain it's not that hard to package a python 
script as a windows .exe.

> It's certainly part of the community efforts in many ways, but it's a
> significant loss of usability compared to things that are included. And
> from the perspective of the testing the infrastructure, you'd loose a
> lot of platform coverage (unless you can find a way to integrate pgxn
> building with the buildfarm).

Right; I think we need at least some amount of pgxn buildfarm coverage. 
There probably also needs to be a way to officially bless certain 
distributions. Unless there's a pretty significant need for an official 
extension to be in contrib, it should go into PGXN.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

On Tue, Feb 14, 2017 at 8:37 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
> Right; I think we need at least some amount of pgxn buildfarm coverage.
> There probably also needs to be a way to officially bless certain
> distributions. Unless there's a pretty significant need for an official
> extension to be in contrib, it should go into PGXN.

I'm not sure a need-based test is going to be entirely the right
thing.  The advantage of having something in contrib is that the core
developers will maintain it for you.  When things change from release
to release, everything in contrib necessarily gets updated; things on
PGXN or elsewhere only get updated if their owners update them.  There
are several good things about that.  First, it means that people can
rely on the stuff in contrib mostly sticking around for future
releases, except occasionally when we decide to drop a module.
Second, it gives maintainers of external modules examples of what they
need to do to update their code when we change the core APIs.  Third,
it's probably actually more efficient for one person to go sweep
through a large number of modules and fix them all at once than if
those things were all on PGXN with separate owners.  Now, you can
overdo that: I don't want to be responsible for every module on PGXN
or anything close to it.  But on balance I think there's a good case
for shipping a substantial set of modules in contrib.

I think, in general, that we've done a good job increasing the quality
of contrib over time.  Pretty much all of the new stuff we've added is
fairly solid, and we cleaned things up significantly by moving test
code to src/test/modules.  But we've still got some older modules in
contrib whose quality and general usefulness is pretty questionable
IMV: earthdistance, intagg, intarray, isn, and of course the
eternally-deprecated but never-actually-removed xml2.  I'm not in a
hurry to expend a lot of energy removing any of that stuff, and I
think we might be reaching our quota of backward-incompatible changes
for this release, but it's questionable in my mind whether having
those things there works out to a net plus.  There's a lot of good
stuff in contrib, though, and I don't think we should be averse to
adding more in the future.  It shouldn't be just that any random
contrib module somebody writes can get dropped into the core
distribution, but I think we ought to be open to reasonable proposals
to add things there when it makes sense.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Feb 14, 2017, at 5:37 AM, Jim Nasby <Jim.Nasby@BlueTreble.com> wrote:

>> Until pgxn has a way of helping users on for example Windows (or other
>> platforms where they don't have a pgxs system and a compiler around),
>> it's always going to be a "second class citizen".
>
> I view that as more of a failing of pgxs than pgxn. Granted, the most common (only?) pgxn client right now is written
inpython, but it's certainly possible to run that on windows with some effort (BigSQL does it), and I'm fairly certain
it'snot that hard to package a python script as a windows .exe. 

Yeah, that’s outside of PGXN’s mandate. It doesn’t do any installing at all, just distribution (release, search,
download).Even the Python client just looks to see what build support is in a distribution it downloads to decide how
tobuild it (make, configure, etc.), IIRC. 

David

On Feb 14, 2017, at 9:37 AM, Magnus Hagander <magnus@hagander.net> wrote:

> It's a failing in one of the two at least. It either needs to be easier to build the things on windows, or pgxn would
needto learn to do binary distributions.  

PGXN makes no effort to support installation on any platform at all. Happy to work with anyone who wants to add binary
distribution,but supporting multiple platforms might be a PITA. Maybe there’d be a way to integrate with the RPM and
.deband Windows repos (is there something like that for Windows?). 

> Even if we get the building easier on windows, it'll likely remain a second class citizen (though better than today's
thirdclass), given the amount of windows machines that actually have a compiler on them for start. Pgxs in Windows
wouldbe a big improvement, but it won't solve the problem.  

Yep.

David

First, just to clarify: my reasons for proposing "core adoption" of PGXN 
are not technical in nature. My desire is to have an extension/add-on 
system that's officially endorsed and embraced by the official 
community, similar to CPAN, pypy, npm, etc. There's no technical reason 
we need PGXN to be a first class citizen, but IMHO making it a first 
class citizen would greatly enhance the usefulness of Postgres and 
strengthen & expand the community. That community expansion should 
eventually result in an increase in new contributors to the database 
code itself.

On 2/14/17 8:24 AM, Robert Haas wrote:
> On Tue, Feb 14, 2017 at 8:37 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
>> Right; I think we need at least some amount of pgxn buildfarm coverage.
>> There probably also needs to be a way to officially bless certain
>> distributions. Unless there's a pretty significant need for an official
>> extension to be in contrib, it should go into PGXN.
>
> I'm not sure a need-based test is going to be entirely the right
> thing.  The advantage of having something in contrib is that the core
> developers will maintain it for you.

With fairly minimal effort, that could be true of something in another 
repository though.

> When things change from release
> to release, everything in contrib necessarily gets updated; things on
> PGXN or elsewhere only get updated if their owners update them.  There

Right, and the intricate tie between contrib and rest of the database is 
why I'm not proposing ditching contrib completely. There's clearly some 
cases when that close tie makes the contrib code significantly simpler. 
(Though, it'd certainly be great if we found a way to reduce the 
multi-version support overhead for all extension creators!)

> are several good things about that.  First, it means that people can
> rely on the stuff in contrib mostly sticking around for future
> releases, except occasionally when we decide to drop a module.
> Second, it gives maintainers of external modules examples of what they
> need to do to update their code when we change the core APIs.  Third,
> it's probably actually more efficient for one person to go sweep
> through a large number of modules and fix them all at once than if
> those things were all on PGXN with separate owners.  Now, you can
> overdo that: I don't want to be responsible for every module on PGXN
> or anything close to it.  But on balance I think there's a good case
> for shipping a substantial set of modules in contrib.

I agree with your points, but AFAIK there's no reason that needs to 
happen in contrib. There could certainly be a dedicated git.PG.org repo 
for official extensions. AFAICT that would meet all your points 
(understanding that code that really needed to be tied to specific 
versions could remain in contrib).

Another option would be to start with just publishing most/all of what's 
in contrib on PGXN. Most OS packaging solutions for contrib seem rather 
clunky/arbitrary to me, so having pgxnclient as an install option would 
probably be welcome to some users. This would mean an additional step in 
the release process, but AFAIK that could be hidden behind a single make 
target (whoever's doing the release would also need access to the 
relevant account on pgxn.org).

... points about current contrib modules that I agree with...

> There's a lot of good
> stuff in contrib, though, and I don't think we should be averse to
> adding more in the future.  It shouldn't be just that any random
> contrib module somebody writes can get dropped into the core
> distribution, but I think we ought to be open to reasonable proposals
> to add things there when it makes sense.

Right now contrib is serving two completely separate purposes:

1) location for code that (for technical reasons) should be tied to 
specific PG versions
2) indication of "official endorsement" of a module by the community

My view is that we've turned #2 into a nail simply because of the hammer 
we built for #1 (certainly understandable since contrib way pre-dates 
extensions, let alone PGXN).

The community has discussions (about once a year) about what should or 
shouldn't stay in contrib, and a lot of the decision making process ends 
up driven by #2. If we had another official distribution channel for 
extensions, we could completely separate #1 and #2: contrib is strictly 
for official community extensions that are greatly simplified by being 
in the main repo; all other official extensions live at <insert git URL 
here> / <list of URLs on PGXN.org>. With some effort (hopefully from 
newly attracted extension authors), #1 could probable be eliminated as 
well, to the benefit of other external projects.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> First, just to clarify: my reasons for proposing "core adoption" of PGXN 
> are not technical in nature.

What do you think "core adoption" means?  Surely not that anything
associated with PGXN would be in the core distro.

> Right now contrib is serving two completely separate purposes:

> 1) location for code that (for technical reasons) should be tied to 
> specific PG versions
> 2) indication of "official endorsement" of a module by the community

This argument ignores what I think is the real technical reason for
keeping contrib, which is to have a set of close-at-hand test cases
for extension and hook mechanisms.  Certainly, not every one of the
existing contrib modules is especially useful for that purpose, but
quite a few of them are.
        regards, tom lane

On 2/14/17 2:05 PM, Tom Lane wrote:
> Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
>> First, just to clarify: my reasons for proposing "core adoption" of PGXN
>> are not technical in nature.
>
> What do you think "core adoption" means?  Surely not that anything
> associated with PGXN would be in the core distro.

No, certainly not. If anything, PGXN being a first class citizen would 
allow for potentially removing code from core, since there would then be 
a first-class way for users to add it.

>> Right now contrib is serving two completely separate purposes:
>
>> 1) location for code that (for technical reasons) should be tied to
>> specific PG versions
>> 2) indication of "official endorsement" of a module by the community
>
> This argument ignores what I think is the real technical reason for
> keeping contrib, which is to have a set of close-at-hand test cases
> for extension and hook mechanisms.  Certainly, not every one of the
> existing contrib modules is especially useful for that purpose, but
> quite a few of them are.

I was under the impression that a lot of that had moved to test, but 
yes, that's a consideration. That said, if local caching was added to 
pgxnclient I don't think it'd require much change to just pull those 
cases from PGXN instead of the core repo. Alternatively, they could be 
pulled from a git repo that's houses the source code for the official 
PGXN modules (or what PGXN calls a distribution).

Those kind of changes would actually help any extension author that 
wants to depend on another extension (namely, automatically pulling 
dependent extensions from PGXN). I have make targets that currently 
accomplish this. They'd be nicer with some additions to both extensions 
and PGXN, but IMHO they're workable right now.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

On 02/14/2017 12:05 PM, Tom Lane wrote:
> Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
>> First, just to clarify: my reasons for proposing "core adoption" of PGXN 
>> are not technical in nature.
> 
> What do you think "core adoption" means?  Surely not that anything
> associated with PGXN would be in the core distro.

One part of this would need to be having a designated committee of the
Postgres community pick a set of "blessed" extensions for packagers to
package.  Right now, contrib serves that purpose (badly).  One of the
reasons we haven't dealt with the extension distribution problem is that
nobody wanted to take on the issue of picking a list of blessed extensions.

> 
>> Right now contrib is serving two completely separate purposes:
> 
>> 1) location for code that (for technical reasons) should be tied to 
>> specific PG versions
>> 2) indication of "official endorsement" of a module by the community
> 
> This argument ignores what I think is the real technical reason for
> keeping contrib, which is to have a set of close-at-hand test cases
> for extension and hook mechanisms.  Certainly, not every one of the
> existing contrib modules is especially useful for that purpose, but
> quite a few of them are.

Yes.  But there's a bunch that aren't, and those are the ones which we
previously discussed, the ones with indifferent maintenance like ISN and
Intarray.

You have to admit that it seems really strange in the eyes of a new user
that ISN is packaged with PostgreSQL, whereas better-written and more
popular extensions (like plv8, pg_partman or pgq) are not.


-- 
Josh Berkus
Containers & Databases Oh My!

On 2/14/17 2:19 PM, Josh Berkus wrote:
> One part of this would need to be having a designated committee of the
> Postgres community pick a set of "blessed" extensions for packagers to
> package.  Right now, contrib serves that purpose (badly).  One of the
> reasons we haven't dealt with the extension distribution problem is that
> nobody wanted to take on the issue of picking a list of blessed extensions.

That was my idea behind "all other official extensions live at <insert 
git URL here> / <list of URLs on PGXN.org>".

Adding some kind of reputation system to PGXN would probably be even 
more useful, but I certainly don't think that's mandatory for having 
officially blessed "core extensions".
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

On 2017-02-14 12:19:56 -0800, Josh Berkus wrote:
> On 02/14/2017 12:05 PM, Tom Lane wrote:
> > Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> >> First, just to clarify: my reasons for proposing "core adoption" of PGXN 
> >> are not technical in nature.
> > 
> > What do you think "core adoption" means?  Surely not that anything
> > associated with PGXN would be in the core distro.
> 
> One part of this would need to be having a designated committee of the
> Postgres community pick a set of "blessed" extensions for packagers to
> package.  Right now, contrib serves that purpose (badly).  One of the
> reasons we haven't dealt with the extension distribution problem is that
> nobody wanted to take on the issue of picking a list of blessed extensions.

I don't see the trust problem being solved by them being blessed unless
they're part of the regularly scheduled postgres back-branch
releases. Which essentially requires them to be in core, or increase the
release maintenance/management cost further.

We sure could have levels between "random github repo" and "in core
extension", but I don't see "bless external extensions" supplanting all
contrib stuff.  There's a few extension in contrib/ where that level
would make sense, and surely more outside, but I think moving all of
contrib to something externally managed would be horrible idea.


> You have to admit that it seems really strange in the eyes of a new user
> that ISN is packaged with PostgreSQL, whereas better-written and more
> popular extensions (like plv8, pg_partman or pgq) are not.

These actually seem like easy cases. plv8 has a massive external
dependency, pg_partman is a) relatively new, b) there's in-core
development of extensions, and pgq isn't exactly trivial, partially
written in python, ...

On Feb 14, 2017, at 9:37 AM, Magnus Hagander <magnus@hagander.net> wrote:

> It's a failing in one of the two at least. It either needs to be easier to build the things on windows, or pgxn would need to learn to do binary distributions.

PGXN makes no effort to support installation on any platform at all. Happy to work with anyone who wants to add binary distribution, but supporting multiple platforms might be a PITA. Maybe there’d be a way to integrate with the RPM and .deb and Windows repos (is there something like that for Windows?).

On Feb 17, 2017, at 12:54 AM, Magnus Hagander <magnus@hagander.net> wrote:

> If we could somehow integrate PGXN with both the RPM build process, the DEB build process and a Windows build process
(whetherdriven by PGXN or just "fed enough data" by PGXN is a different question), I think that would go a long way
towardsthe goal. 

My thought was that someone could rsync the PGXN repo every hour or something and build any new modules there. That’s
howthe search site is built: Every five minutes, it rsyncs rsync://master.pgxn.org/pgxn, parses the output to see new
releases,and updates the index. 

> Also being able to use this somehow to drive continuous builds and tests (kind of like a buildfarm-lite for a subset
ofplatforms) would be useful for reaching a point where extensions outside of core can come at least close to what we
deliverin core. 

Personally I just use Travis and Coveralls on GitHub for that sort of thing.

Best,

David

On 2/14/17 15:19, Josh Berkus wrote:
> You have to admit that it seems really strange in the eyes of a new user
> that ISN is packaged with PostgreSQL, whereas better-written and more
> popular extensions (like plv8, pg_partman or pgq) are not.

I don't know.  Seems pretty standard coming from a Python or Ruby
ecosystem, say.  ;-)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2/14/17 2:39 PM, Andres Freund wrote:
>> One part of this would need to be having a designated committee of the
>> Postgres community pick a set of "blessed" extensions for packagers to
>> package.  Right now, contrib serves that purpose (badly).  One of the
>> reasons we haven't dealt with the extension distribution problem is that
>> nobody wanted to take on the issue of picking a list of blessed extensions.>
> I don't see the trust problem being solved by them being blessed unless
> they're part of the regularly scheduled postgres back-branch
> releases. Which essentially requires them to be in core, or increase the
> release maintenance/management cost further.

I don't see why we'd have to touch blessed PGXN extensions any time 
there's a back-branch release. We don't do that with what's in core 
today. If we did want to do that (say, so that the extension version 
always matched the PG version), it wouldn't be hard to automate.

Obviously if there's a bug in an extension we'd want to do something, 
but tying that to the release would be completely optional.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

On Fri, Feb 17, 2017 at 09:54:37AM +0100, Magnus Hagander wrote:
> If we could somehow integrate PGXN with both the RPM build process, the DEB
> build process and a Windows build process (whether driven by PGXN or just "fed
> enough data" by PGXN is a different question), I think that would go a long way
> towards the goal. Also being able to use this somehow to drive continuous
> builds and tests (kind of like a buildfarm-lite for a subset of platforms)
> would be useful for reaching a point where extensions outside of core can come
> at least close to what we deliver in core.

Just stating the obvious, but one of the reasons CPAN works so well is
that most of the modules are written in Perl and hence don't need
per-platform compilation.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Feb 27, 2017, at 12:04 PM, Bruce Momjian <bruce@momjian.us> wrote:

> Just stating the obvious, but one of the reasons CPAN works so well is
> that most of the modules are written in Perl and hence don't need
> per-platform compilation.

There are a *lot* of C-baded modules on CPAN; and my guess is that, more often than not, Perl modules depend on other
C-basedmodules. 

I daresay a lot of PostgreSQL extensions can be written in pure SQL or PL/pgSQL.

Best,

David

On Mon, Feb 27, 2017 at 01:00:20PM -0800, David E. Wheeler wrote:
> On Feb 27, 2017, at 12:04 PM, Bruce Momjian <bruce@momjian.us> wrote:
> 
> > Just stating the obvious, but one of the reasons CPAN works so well is
> > that most of the modules are written in Perl and hence don't need
> > per-platform compilation.
> 
> There are a *lot* of C-baded modules on CPAN; and my guess is that, more often than not, Perl modules depend on other
C-basedmodules.
 
> 
> I daresay a lot of PostgreSQL extensions can be written in pure SQL or PL/pgSQL.

Oh, does CPAN distribute compiled modules or requires users to compile
them.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Feb 27, 2017, at 1:53 PM, Bruce Momjian <bruce@momjian.us> wrote:

> Oh, does CPAN distribute compiled modules or requires users to compile
> them.

Like PGXN, it formally does not care, but its implementation expects source code distributions what will be built and
installedby users. Note that the vast majority of those modules, -- even pure Perl modules -- are built with make. 

So users typically get their Perl modules in one of these ways:

1. As binaries from their distribution’s package manager. These tend to be updated manually by volunteers and not
integratedinto CPAN, though there are solutions such as [rpmcpan](https://github.com/iovation/rpmcpan) and
[PPM](http://www.activestate.com/activeperl/ppm-perl-modules)which do regular distro package builds. 

2. As source code from CPAN, from which they are compiled (when necessary), built, and installed by the user or a build
systemsuch as [Homebrew](https://brew.sh). 

Best,

David