Re: The plan for FDW-based sharding

Hi,

On 03/01/2016 08:02 PM, Bruce Momjian wrote:
> On Tue, Mar  1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote:
>> Note that I am not saying that other discussed approaches are any
>> better, I am saying that we should know approximately what we
>> actually want and not just beat FDWs with a hammer and hope sharding
>> will eventually emerge and call that the plan.
>
> I will say it again --- FDWs are the only sharding method I can think
> of that has a chance of being accepted into Postgres core.

I don't quite see why that would be the case. Firstly, it assumes that 
FDW-based approach is going to work, but given the lack of prototype or 
even a technical analysis discussing the missing pieces, that's very 
difficult to judge.

I find it a bit annoying that there are objections from people who 
implemented (or attempted to implement) sharding on PostgreSQL, yet no 
reasonable analysis of their arguments and how the FDW approach will 
address them. My my understanding is they deem FDWs a bad foundation for 
sharding because it was designed for a different purpose, but the 
abstractions are a bad fit for sharding (which assumes isolated nodes, 
certain form of execution etc.)

> It is a plan, and if it fails, it fails. If is succeeds, that's
> good. What more do you want me to say? I know of no other way to
> answer the questions you asked above.

Well, wouldn't it be great if we could do the decision based on some 
facts and not mere belief that it'll help. That's exactly what Petr is 
talking about - the fear that we'll spend a few years working on 
sharding based on FDWs, only to find out that it does not work too well. 
That'd be a pretty bad outcome, wouldn't it?

My other worry is that we'll eventually mess the FDW infrastructure, 
making it harder to use for the original purpose. Granted, most of the 
improvements proposed so far look sane and useful for FDWs in general, 
but sooner or later that ceases to be the case - there sill be changes 
needed merely for the sharding. Those will be tough decisions.

While I disagree with Simon on various things, I absolutely understand 
why he was asking about a prototype, and some sort of analysis of what 
usecases we expect to support initially/later/never, and what pieces are 
missing to get the sharding working. IIRC at the FOSDEM Dev Meeting 
you've claimed you're essentially working on a prototype - once we have 
the missing FDW pieces, we'll know if it works. I disagree that - it's 
not a prototype if it takes several years to find the outcome.

Also, in another branch of this thread you've said this (I don't want to 
sprinkle the thread with responses, so I'll just respond here):

> In a way, I don't see any need for an FDW sharding prototype
> because, as I said, we already know XC/XL work, so copying what they
> do doesn't help. What we need to know is if we can get near the XC/XL
>  benchmarks with an acceptable addition of code, which is what I
> thought I already said. Perhaps this can be done with FDWs, or some
> other approach I have not heard of yet.

I don't quite understand the reasoning presented here. The XC/XL are not 
based on FDWs at all, therefore the need for prototype of the FDW-based 
sharding is entirely independent to the fact that these solutions seem 
to work quite well.

regards

--
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services