Re: On-disk bitmap index patch

Tom Lane wrote:
>
> 
> I'm surprised no one caught me making this bogus computation.  I
> realized this morning it's wrong: if there are 10000 distinct values
> then on the average the 1-bits would be about 10000 bits apart, not 100.

Right - I didn't think 10000 was *that* bad, but was too sleepy to try 
working it out :-).

> 
>
> I don't believe the 100x numbers that have been
> bandied around in this discussion, but 10x is plenty enough to be
> interesting.
> 

Yep - I have not managed to get 100x in any of my tests. However, I do 
see some about half that for the TPCH scale 10 dataset:

tpch=# \i relsizes.sql            (BTREE)        relname         | relpages
------------------------+---------- customer               |    41019 customer_c_custkey     |     3288
customer_c_mktsegment |     5779 customer_c_nationkey   |     3288 lineitem               |  1535724
lineitem_l_linenumber |   131347 lineitem_l_orderkey    |   131347 orders                 |   307567 orders_o_custkey
   |    32847 orders_o_orderpriority |    65876 orders_o_orderstatus   |    41131
 


tpch=# \i relsizes.sql            (MAINLY BITMAP)        relname         | relpages
------------------------+---------- customer               |    41019 customer_c_custkey     |     3288
customer_c_mktsegment |      157 customer_c_nationkey   |      336 lineitem               |  1535724
lineitem_l_linenumber |     7571 lineitem_l_orderkey    |   131347 orders                 |   307567 orders_o_custkey
   |    32847 orders_o_orderpriority |     1427 orders_o_orderstatus   |      717
 

The orders_o_orderpriority and orders_o_orderstatus bitmap indexes are 
46 and 57 times smaller than their btree counterparts (hmm...might we 
see even better compression for larger scale factors?).

An obvious deduction is that the TPCH dataset is much more amenable to 
run compression than my synthetic Zipfian data was. The interesting 
question is how well "real" datasets are run compressable, I suspect 
"better than my Zipfian data" is a safe assumption!

Cheers

Mark