Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 21:38, Andres Freund wrote:
> On 2017-09-25 21:31:11 +0300, Alvaro Hernandez wrote:
>>>>> - Distribution and testing are non-trivial: many OS/archs combinations.
>>>>>
>>> Yes, it is. Why would we want to increase that burden to this community?
>>
>>      That's a different story, and one I cannot argue against. If easying
>> postgresql integration with other tools is not something of a priority or
>> something the core group cannot add to all the stuff on their shoulders, all
>> my due respect. PostgreSQL users will do without, someway, somehow. But IMHO
>> this should be a really high priority, and saying that this would turn
>> PostgreSQL into an Oracle code base is going too far ;)
> Well, we can certainly use more help doing maintenance-y stuff, which
> will in turn allow to get more stuff into core in a good state medium+
> term... ;)
>
> Less jokingly: I'm doubtful that the "not a priority" take is really
> fair - there's a lot of priorities, and they compete for scant
> resources. Which means people have to argue convincingly if they want to
> add to that burden - just actually asking the question whether it's a
> good use of resources doesn't mean it's not. Just that it's not yet
> clear.
>
>
    OK, let me try to do that. I believe data integration is a 
priority. World is no longer an isolated database where many apps talk 
to it. Nowdays heterogeneous architectures are almost the norm. CDC is 
often the best solution for many of the data integration tasks like data 
warehousing, data consolidation, stream processing, etc. From this 
perspective, it would be key to have a general tool or good starting 
point for CDC or even higher level functionality tools. Think of 
PostgreSQL's Golden Gate.
    Now, developing any software like this faces two significant 
challenges, which turn into deterrents for developing such software:

- If you want to develop your own output plugin, then your market is 
reduced as you have to exclude all the managed solutions (until, and 
only if, you would convince them to include your plugin... highly 
unlikely, very difficult). And probably another % of enterprise 
environments which will hesitate to link your own plugin to the running 
production database. Last but not least, you need to compile and test 
(and testing this is far from easy) on multiple OSes/architectures.

- If you stick to in-core plugins, then you need to support at least 
three different output formats if you want to support 9.4+: 
test_decoding (and pray it works!), pgoutput, and the "new" in-core 
plugin that was proposed at the beginning of this thread, if that would 
see the light.

    Both are strong enough arguments to make building this kind 
software far less interesting. Actually, I don't know of any software 
like this, and this may already be a consequence of what I'm saying. And 
I truly believe PostgreSQL should offer this, as part of its ecosystem 
(not necessarily in core).
    On the other hand, in-core may help encourage building this 
solutions. If there would be an in-core, uniform, flexible, output 
plugin, and ideally backported to 9.4 (I know, I know....), included in 
very PostgreSQL... it would really open the doors to many integration 
applications. Output plugins are great for your own use, for controlled 
environments, for specific set of users. But only in-core plugins help 
for general-purpose solutions.

    Álvaro

-- 

Alvaro Hernandez


-----------
OnGres



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers