Re: moving data from windows to linux

Alan Hodgson wrote:
> On Wednesday 25 April 2007 08:36, Francois Deliege <fdeliege@cs.aau.dk>
> wrote:
>
>> I am moving from 8.2.4 under Windows to 8.2.4 under FreeBSD 6.2.
>> The databases are all encoded in UTF8 on the windows server.
>> The threads I have seen in the archives are all about version or
>> database encoding issues.
I think I identified the "moving from Windows to FreeBSD" problem.  In
short:
FreeBSD 6.2 cannot handle files larger than 1.7GB on NTFS partition mounted.

--> Would be nice to enable splits to be handled by pg_dump /
pg_restore.  :-)

Here are some pieces of evidence:

Looking at the data that causes the problem in the music-data dump file:
dimension1# awk '/16862\t8[7-9]/' /mnt/windows/music-data
16862   87      0.99999997673498697     2.7562404582937301e-010
5.3289215505938802e-011 1.63906667677656e-014   3.6428502118254301e-013
1.3056815099222999e-009       9.8685777446191209e-016
1.2298273123735399e-017 1.2144251138520999e-011 4.8726130130392397e-014
1.9123218467089399e-012 4.0612663109915703e-012
2.1516222137270299e-010       4.4099191584683199e-012
2.1392297825760101e-008
16862   88      0.99983678662586195     5.0406110505424003e-005
1.5563266660749601e-006 1.6705707371203099e-007 1.63349359798036e-011
3.48994254

Not a lot of data for 16862 88 compared to 16862 87 !!!  That's what was
causing the problem...
dimension1# sed '/16862\t88/,//d' < /mnt/windows/music-data >
/usr/home/fdeliege/musicdata-ok
dimension1# ls -l musicdata-ok
-rw-r--r--  1 root  fdeliege  1836729357 Apr 26 13:57 musicdata-ok

The size of the file is 1.7GB
A short python script to visualize the data mounted on the windows
partition from my FreeBSD box...
dimension1# python pythonscript.py /mnt/windows/music-data 1836729340 1000
e-011   3.48994254

The same running script running from windows gives me what I expected, a
lot of numbers.  :-)

Now, I am splitting the 150 GB dump file into 1G chunks... time for a beer.

Cheers,

Francois