Re: Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

> -----Original Message-----
> From: David Rowley [mailto:david.rowley@2ndquadrant.com]
> Sent: Wednesday, August 19, 2015 9:00 AM
> To: Kevin Grittner
> Cc: Kaigai Kouhei(海外 浩平); pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with
> crazy number of rows
> 
> On 19 August 2015 at 08:54, Kevin Grittner <kgrittn@ymail.com> wrote:
> 
> 
>     Kouhei Kaigai <kaigai@ak.jp.nec.com> wrote:
> 
>     >         long        lbuckets;
> 
>     >         lbuckets = 1 << my_log2(hash_table_bytes / bucket_size);
> 
>     >     Assert(nbuckets > 0);
> 
>     > In my case, the hash_table_bytes was 101017630802, and bucket_size was
> 48.
>     > So, my_log2(hash_table_bytes / bucket_size) = 31, then lbuckets will
> have
>     > negative number because both "1" and my_log2() is int32.
>     > So, Min(lbuckets, max_pointers) chooses 0x80000000, then it was set
> on
>     > the nbuckets and triggers the Assert().
> 
>     > Attached patch fixes the problem.
> 
>     So you changed the literal of 1 to 1U, but doesn't that just double
>     the threshold for failure?  Wouldn't 1L (to match the definition of
>     lbuckets) be better?
> 
> 
> 
> 
> I agree, but I can only imagine this is happening because the maximum setting
> of work_mem has been modified with the code you're running.
> 
> hash_tables_bytes is set based on work_mem
> 
> hash_table_bytes = work_mem * 1024L;
> 
> The size of your hash table is 101017630802 bytes, which is:
> 
> david=# select pg_size_pretty(101017630802);
> 
>  pg_size_pretty
> ----------------
>  94 GB
> (1 row)
> 
> david=# set work_mem = '94GB';
> ERROR:  98566144 is outside the valid range for parameter "work_mem" (64 ..
> 2097151)
>
Hmm. Why I could set work_mem = '96GB' without error.

It was described in the postgresql.conf.
 postgres=# SHOW work_mem;  work_mem ----------  96GB (1 row)

> So I think the only way the following could cause an error, is if bucket_size
> was 1, which it can't be.
> 
> lbuckets = 1 << my_log2(hash_table_bytes / bucket_size);
> 
> 
> I think one day soon we'll need to allow larger work_mem sizes, but I think there's
> lots more to do than this change.
>
I oversight this limitation, but why I can bypass GUC limitation check?

--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>