On 11 March 2016 at 16:09, Bruce Momjian <bruce@momjian.us> wrote:
Ideally everything would have a well-defined plan, it is sometimes hard to do.
BDR helped for logical decoding etc - having something concrete really helped shape and guide each part of it as it was (or is/will be, in some cases) migrated from BDR to core.
That said, it was necessary because for many of the things it needs there weren't really good, isolated improvements to make with obvious utility for other projects. Sure, commit timestamps are handy, replication origins will be handy, etc. They can be used by other projects and will be. Some are already. But unlike the FDW enhancements they're not things that will be used simply by being present without even requiring any special user action, so they had an understandably higher barrier to cross for acceptance.
Once you get to the point where you're not making FDW improvements that help a broad set of users and start doing things that'll really only aid some hypothetical sharding system that also requires other infrastructure changes, hooks, etc ... that's when I think it's going to be proof-of-concept prototype time.
Similar to our approach on parallelism (which is also super-important and doesn't many active developers), sometimes you just need to create infrastructure and see how well it solves problems.
Yep. Again, like BDR and logical decoding. We've had quite a lot of surprises as we find unexpected corner cases and challenges over time. Andres's original work on logical decoding went through a number of significant revisions as more was learned about the problem to solve. Sometimes you can only do that by actually building it. Logical decoding as it stands in core is only partway through that evolution as it is - I think we now have a good understanding of why logical decoding of prepared xacts, streaming of in-progress xacts etc will be needed down the track, but it would've been hard to come up with that at the start when we didn't have experience using what we've already got.
The weird thing is that if you do implement an ill-defined feature, there really isn't much huge positive feedback --- people just use the feature, and the complaints stop.
... eventually.
Sometimes the bug reports start. Occasionally you get a "thanks, this looks interesting/handy". But usually just bug reports or complaints that whatever you built isn't good enough to meet some random person's particular use case. Ah well.