Re: Making a tree with "millions and millions" of dynamic nodes

Christian Fowler <google@NOSPAM.gravesweeper.com> wrote in message news:<6b-dnQJnDI6YvlCiXTWc-g@speakeasy.net>...
> I have a VERY LARGE pile of geographic data that I am importing into a database (db of choice is postgres,
> though may hop to oracle if necessary). The data is strictly hierarchical - each node has one, and only one
> parent. The depth should not exceed 6 or 7 levels. The initial import will have about 6 million leaves, and
> 3 million branches.
> For selection, it is crucial for me to get:
>
> 1. path generation speed
> 2. immediate sibling speed
> 3. children count speed
>

I don't know how high your requirement for having an SQL interface is.
At my company we wrote a real-time database for the local number
portability, with over 40M entries, with support for variable length
phone numbers ( up to 10 digits ). The customer wanted the longest
phone number match possible. example there could 1, 12, 123 in the
database and the number 1234 would match to 123.

We did it with C++ and a memory mapped file for persistence.
We had to use a 64bit cpu and 12G of RAM.

we can do over 100K insert/search/delete's per second.
a relatively flat or equal performance for each operation ( depends on
number length ).

If real-time performance is that important, then you might have to
customize.