Re: Does larger i/o size make sense?

<div dir="ltr"><div class="gmail_extra"><br /><div class="gmail_quote">On Thu, Aug 22, 2013 at 8:53 PM, Kohei KaiGai
<spandir="ltr"><<a href="mailto:kaigai@kaigai.gr.jp" target="_blank">kaigai@kaigai.gr.jp</a>></span> wrote:<br
/><blockquoteclass="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":652"
style="overflow:hidden">An idea that I'd like to investigate is, PostgreSQL allocates a set of<br /> continuous buffers
tofit larger i/o size when block is referenced due to<br /> sequential scan, then invokes consolidated i/o request on
thebuffer.<br /> It probably make sense if we can expect upcoming block references<br /> shall be on the neighbor
blocks;that is typical sequential read workload.</div></blockquote></div><br /></div><div class="gmail_extra">I think
itmakes more sense to use scatter gather i/o or async i/o to read to regular sized buffers scattered around memory than
torestrict the buffers to needing to be contiguous.<br /><br /></div><div class="gmail_extra">As others said, Postgres
dependson the OS buffer cache to do readahead. The scenario where the above becomes interesting is if it's paired with
amove to directio or other ways of skipping the buffer cache. Double caching is a huge waste and leads to lots of
inefficiencies.<br /><br /></div><div class="gmail_extra">The blocking issue there is that Postgres doesn't understand
muchabout the underlying hardware storage. If there were APIs to find out more about it from the kernel -- how much
furtherbefore the end of the raid chunk, how much parallelism it has, how congested the i/o channel is, etc -- then
Postgresmight be on par with the kernel and able to eliminate the double buffering inefficiency and might even be able
todo better if it understands its own workload better.<br /><br /></div><div class="gmail_extra">If Postgres did that
thenit would be necessary to be able to initiate i/o on multiple buffers in parallel. That can be done using scatter
gatheri/o such as readv() and writev() but that would mean blocking on reading blocks that might not be needed until
thefuture. Or it could be done using libaio to initiate i/o and return control as soon as the needed data is available
whileother i/o is still pending.<br /><br /></div><div class="gmail_extra"><br /></div><div class="gmail_extra">-- <br
/>greg<br/></div></div>