Re: Large databases, performance

Shridhar,

It's one hell of a DB you're building. I'm sure I'm not the only one interested
so to satisfy those of us who are nosey: can you say what the application is?

I'm sure we'll all understand if it's not possible for you mention such
information.


--
Nigel J. Andrews


On Thu, 3 Oct 2002, Shridhar Daithankar wrote:

> Hi,
>
> Today we concluded test for database performance. Attached are results and the
> schema, for those who have missed earlier discussion on this.
>
> We have (almost) decided that we will partition the data across machines. The
> theme is, after every some short interval a burst of data will be entered in
> new table in database, indexed and vacuume. The table(s) will be inherited so
> that query on base table will fetch results from all the children. The
> application has to consolidate all the data per node basis. If the database is
> not postgresql, app. has to consolidate data across partitions as well.
>
> Now we need to investigate whether selecting on base table to include children
> would use indexes created on children table.
>
> It's estimated that when entire data is gathered, total number of children
> tables would be around 1K-1.1K across all machines.
>
> This is in point of average rate of data insertion i.e. 5K records/sec and
> total data size, estimated to be 9 billion rows max i.e. estimated database
> size is 900GB. Obviously it's impossible to keep insertion rate on an indexed
> table high as data grows. So partitioning/inheritance looks better approach.
>
> Postgresql is not the final winner as yet. Mysql is in close range. I will keep
> you guys posted about the result.
>
> Let me know about any comments..
>
> Bye
>  Shridhar