Re: import function's progress report

Hi

On Wed, Dec 31, 2008 at 7:14 AM, Quan Zongliang <quanzongliang@gmail.com> wrote:
> Hi, all
>
> This is a progress report about import function.
> A screenshot with dialog test mode (/t) attached.
> Only the file page's code is finished.
>
> 1. Target page:
> Now it is blank.
> Reserved for standalone version to select target database and table.

OK. Note that the different pages should be on a wxNotebook - don't
use buttons to control the UI (we've - OK, I've - made that mistake
before).

> 2. Source pages: file / Source / ODBC
> Import from csv format file / other pgsql database / ODBC(for MS-Windows)
> First, csv file will be supported. Remainders only in plan.

The different sources should be on a single 'Source' tab. At the top
of that tab, use a radio button set to allow the user to select the
source type, and have that swap in/out framed sets of controls
underneath. For a very basic example of this, look at the Report form
which changes the stylesheet options based on whether you select HTML
or XML output.

> 3. Columns page:
> Set data format ( date & number), space trim and column's order
> I plan to support express, user can input a pgsql express to treat data.

Hmm, pgsql expressions could be slow as they can only be evaluated in
insert statements, which is really not good for such a tool. I would
suggest:

- Consider embedding Python (which is on the TODO anyway) and allowing
transforms to be scripted on the client.
- Convert the import data to a COPY compatible format so it can be fed
directly to the server without using INSERT which is *much* slower.

> 4. Options/Status/Others page:
> Only some ideas:
>  Skip empty rows
>  Import rows match regex
>  Don't import rows match regex
>  Commit per (n) rows.
>  Some log options
>  Test insert with (n) rows before insert to DB
>  Generate insert statement to a file.
>  Stop when (n) errors
> and so on.

Sounds good :-)

> Limited encoding can be Support, see attached file "encodinglist.txt".
> ASCII, Japanese EUCJP/Shift_JIS and Chinese GB2312/Big5 had been tested.

Cool :-)


--
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com