Обсуждение: Regarding Copy in postgresql

Поиск
Список
Период
Сортировка

Regarding Copy in postgresql

От
sreeaurovindh viswanathan
Дата:
I have a text file which has about 140 million records spaced by tab.I
am trying it to read and put it into database.I have used Copy to do
it.For the delimiter i tried with \t but it did not work.Hence i
copied the tab space and pasted it in copy command and then it
executed.Does the copy command read the file sequentially.Atleast from
the ordering that i got it doesn't seem so.. If that is the case how
is the orderering of read..


Thanks
Sree aurovindh V

Re: Regarding Copy in postgresql

От
Daniel Staal
Дата:
--As of March 22, 2012 9:56:37 AM +0530, sreeaurovindh viswanathan is
alleged to have said:

> I have a text file which has about 140 million records spaced by tab.I
> am trying it to read and put it into database.I have used Copy to do
> it.For the delimiter i tried with \t but it did not work.Hence i
> copied the tab space and pasted it in copy command and then it
> executed.Does the copy command read the file sequentially.Atleast from
> the ordering that i got it doesn't seem so.. If that is the case how
> is the orderering of read..

--As for the rest, it is mine.

The copy itself should read the file sequentially, I think.  (I can't think
of a single good reason not to do so from a programming standpoint.)
However a 'select' to pull the data back out of the database is inherently
unordered; it's order will have no relation (except by chance) to the order
the data was put into the database.

If you need to know if all of the data in the text file were loaded, it
would be best to either count the records inserted, or select the last
record and see if it was loaded.  If you need a more predictable way to
find which was the last record read, you will either need to use the count
(again), or to find some ordering in the text file that you can use in an
'order by' clause.  (If you are are on a Unix-like platform, it may be
worth feeding your text file through `sort` just for this purpose.)

Daniel T. Staal

---------------------------------------------------------------
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

Re: Regarding Copy in postgresql

От
sreeaurovindh viswanathan
Дата:
Dear Daniel Staal,

Thanks for your update.Now i could understand why it became out of
order.I used copy command to copy the data from text file into
database.These rows did not have a primary key attached to it.Hence i
made an autogenerate column and dumped the data to it.Here I used the
select statement and this is why the out of order copying of rows took
place.

Thanks for your insight.

Regards
Sree aurovindh V

On Thu, Mar 22, 2012 at 10:50 AM, Daniel Staal <DStaal@usa.net> wrote
> --As of March 22, 2012 9:56:37 AM +0530, sreeaurovindh viswanathan is
> alleged to have said:
>
>> I have a text file which has about 140 million records spaced by tab.I
>> am trying it to read and put it into database.I have used Copy to do
>> it.For the delimiter i tried with \t but it did not work.Hence i
>> copied the tab space and pasted it in copy command and then it
>> executed.Does the copy command read the file sequentially.Atleast from
>> the ordering that i got it doesn't seem so.. If that is the case how
>> is the orderering of read..
>
>
> --As for the rest, it is mine.
>
> The copy itself should read the file sequentially, I think.  (I can't think
> of a single good reason not to do so from a programming standpoint.) However
> a 'select' to pull the data back out of the database is inherently
> unordered; it's order will have no relation (except by chance) to the order
> the data was put into the database.
>
> If you need to know if all of the data in the text file were loaded, it
> would be best to either count the records inserted, or select the last
> record and see if it was loaded.  If you need a more predictable way to find
> which was the last record read, you will either need to use the count
> (again), or to find some ordering in the text file that you can use in an
> 'order by' clause.  (If you are are on a Unix-like platform, it may be worth
> feeding your text file through `sort` just for this purpose.)
>
> Daniel T. Staal
>
> ---------------------------------------------------------------
> This email copyright the author.  Unless otherwise noted, you
> are expressly allowed to retransmit, quote, or otherwise use
> the contents for non-commercial purposes.  This copyright will
> expire 5 years after the author's death, or in 30 years,
> whichever is longer, unless such a period is in excess of
> local copyright law.
> ---------------------------------------------------------------
>
> --
> Sent via pgsql-novice mailing list (pgsql-novice@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-novice