Re: Loading 500m json files to database

On 3/24/20 11:53 AM, Kevin Brannen wrote:
> From: Rob Sargent <robjsargent@gmail.com>
>
>> Sorry if I missed it, but have we seen the size range of these json files?
> Not that I've seen, but that's an implementation detail for whoever is doing the work. As someone else pointed out,
pickthe value as needed, whether that's 10, 100, 1000, or whatever. But issuing 1000 lines of "\copy file" sort of
commandsat a time isn't a big deal by itself. OTOH, you have a good point that 1000 could be too much work for the
serverto handle, especially if the "-1" flag is also used. As always: test, test, test... 😊
 
>

My fear is this:  the community helps OP load 500M "records" in a 
reasonable timeframe then OP's organization complains postgres is slow 
once they start using it... because the transition from file system to 
rdbms was ill conceived.  Are the json files large documents or 
arbitrary content or are they small data structures of recognizable 
types. And so on