Обсуждение: call for creating test suite

Hello,

  is here someone who will create testsuite for psqlodbc driver?
I have this idea:

- it contains executable, and dir with text files
- each text file describe one test
- text file contains specification:
   - test name
   - what is the right result (failure with specified sqlstate could
       be the right result)
   - SQLXXX odbc commands with parameters which will be executed
     in same order they are written to text file
- executable
   - goes throught test dir execute each test and copare expected
     result (readed from text file) againist real result for each
     test; it takes in mind if the failure is on the last SQLXXX
   - it stops on first error which is raised during execution of text
     file
   - write if each test is ok or not ok to stdout

Comments and volunteers are welcome.
This could improve bug hunting and checking if there is no known
bug in new release or development snapshot.

Please help us,

Luf

Ludek Finstrle <luf@pzkagis.cz> writes:

>   is here someone who will create testsuite for psqlodbc driver?

Isn't it possible to re-use/build upon some existing ODBC test suite?
Since ODBC is a standard interface, it would look like a waste to
duplicate such an effort across different ODBC drivers.

At worst a postgreSQL test-suite should hold ONLY postgreSQL-specific
tests, setup scripts, patches, etc., but no "standard" tests IMHO.

We (continuent) surely could invest some time developing such a suite
provided we can easily run it against our software.

There already is something that looks a lot like an ODBC test suite in
the 'tests' directory of unixODBC. Anyone had a look at it/heard about
it?

> >   is here someone who will create testsuite for psqlodbc driver?
>
> Isn't it possible to re-use/build upon some existing ODBC test suite?
> Since ODBC is a standard interface, it would look like a waste to
> duplicate such an effort across different ODBC drivers.

It's possible. I only specify what I want from it. I don't know other
odbc test suites.
If it fits my requirements it'll be simple to reproduce error from
log output of psqlodbc in the future. I think we process log output
through grep, awk and we get the input for test suite.

I'm open to discussion about the test suite fetures and architecture.
I post it here in this case too.

> We (continuent) surely could invest some time developing such a suite
> provided we can easily run it against our software.

Maybe I understand this bad. Are you volunteer?

> There already is something that looks a lot like an ODBC test suite in
> the 'tests' directory of unixODBC. Anyone had a look at it/heard about
> it?

No, I don't know it. Could you explain it to us? Everyone who feel that
we may use another ODBC test suite is welcome with his opinion and
his knowledge.

If someone would write it afresh the test suite could be written whatever
language you want. I think about PERL as it could be the fastest and
easiest way for me. But anyone could feel free to use what he want (after
discussion here).
I don't need the test suite know the constants. I get the values
so it could only know integral numbers, real numbers (maybe not
I don't know whole ODBC spec), string and maybe some other basic
types.

Regards,

Luf

Ludek Finstrle <luf@pzkagis.cz> writes:

>> We (continuent) surely could invest some time developing such a suite
>> provided we can easily run it against our software.
>
> Maybe I understand this bad. Are you volunteer?

Yes, we will contribute to any test suite that fits our needs enough
for us to use it regularly (I guess that's how things usually
go!). One of our requirements is that it should NOT be too strongly
tied to a specific DBMS (postgreSQL or other).


>> There already is something that looks a lot like an ODBC test suite in
>> the 'tests' directory of unixODBC. Anyone had a look at it/heard about
>> it?
>
> No, I don't know it. Could you explain it to us? Everyone who feel that
> we may use another ODBC test suite is welcome with his opinion and
> his knowledge.

I haven't look at it yet, only planned to. Unless it's terribly
written/not flexible, it should be faster to start from this than to
reinvent the wheel.


> If someone would write it afresh the test suite could be written whatever
> language you want. I think about PERL as it could be the fastest and
> easiest way for me.

You gonna like it: I heard that unixodbc/tests use perl intensively.


> I don't need the test suite know the constants. I get the values
> so it could only know integral numbers, real numbers (maybe not
> I don't know whole ODBC spec), string and maybe some other basic
> types.

Which constants are you talking about?


Cheers,

Marc.

On Friday 13 January 2006 14:50, Ludek Finstrle wrote:
> If someone would write it afresh the test suite could be written whatever
> language you want. I think about PERL as it could be the fastest and
> easiest way for me. But anyone could feel free to use what he want (after
> discussion here).

A year or so ago I wrote a proglet (in C) to do some basic ODBC retrieval...
it was used as part of a test framework to do some simple queries and check
that the expected data was being returned from the database.

I guess this might be useful for PostgreSQL ODBC testing.

I'll hunt out my code over the weekend and see if its still usable.

Greetings from (frozen) Luxembourg.

--
David Somers
SIP: FWD dial **44808449868206; voipuser.org dial 08449868206
PGP Key = 7E613D4E
Fingerprint = 53A0 D84B 7F90 F227 2EAB  4FD7 6278 E2A8 7E61 3D4E

> Yes, we will contribute to any test suite that fits our needs enough
> for us to use it regularly (I guess that's how things usually
> go!). One of our requirements is that it should NOT be too strongly
> tied to a specific DBMS (postgreSQL or other).

Well, nice to hear it. Thank you. I don't need tied test suite to DBMS.
Only text files (or what we discuss here) have to be in some kind
specific. The next question is the license of the test suite.

> >> There already is something that looks a lot like an ODBC test suite in
> >> the 'tests' directory of unixODBC. Anyone had a look at it/heard about
> >> it?
> >
> > No, I don't know it. Could you explain it to us? Everyone who feel that
>
> I haven't look at it yet, only planned to. Unless it's terribly
> written/not flexible, it should be faster to start from this than to
> reinvent the wheel.

It sounds well.

> > If someone would write it afresh the test suite could be written whatever
> > language you want. I think about PERL as it could be the fastest and
> > easiest way for me.
>
> You gonna like it: I heard that unixodbc/tests use perl intensively.

It was only idea that string parsing and hashes in PERL are easy to
write.

> > I don't need the test suite know the constants. I get the values
> > so it could only know integral numbers, real numbers (maybe not
> > I don't know whole ODBC spec), string and maybe some other basic
> > types.
>
> Which constants are you talking about?

I mean that there is no entry in text files like this:
SQLEndTran(SQL_HANDLE_DBC,<handle>,SQL_COMMIT)
it would be
SQLEndTran(2,<handle>,0)

SQL_HANDLE_DBC is ODBC constant with its value = 2.

Egh I forgot that we have to get new handle and replace the old handle
with new one. Or similar support od ODBC handles.
I mean the test suite have to can get results and output params from
SQLXXX calls which need it (e.g. SQLAllocHandle).

So next request for test suite is support in text files something like:
SQLAllocHandle (1,0,@) (-,-,<handle>)
You don't have to follow this syntax exactly. It only mean that SQLXXX
calls could have to parameter sets. There are input parameters in first ()
and there are are output parameters in second (). @ means only output param,
- means only input param. We have to remember to handle the ODBC handle
specially. There can be more specialities. It need go throught ODBC spec
and define it fully.

I only specify what I know. The inspiration from unixODBC test suite
could be very good.

Regards,

Lud

Am 12 Jan 2006 um 15:59 hat Ludek Finstrle geschrieben:

> Hello,
>
>   is here someone who will create testsuite for psqlodbc driver?

Hi,

as for my problems with updateable cursors and driver versions, I
have extracted some code from my ODBC wrapper classes to be
a simple C test application with the same 'workflow' as of my GUI.

It is based on Dave's google posting and I could clearlier see if my
wrapper implementation is correct:

http://groups.google.com/group/pgsql.interfaces.odbc/browse_frm/thread/df155225d9
d4a291/e9903c90492969b2#e9903c90492969b2

Here you could grab the code:

http://cvs.sourceforge.net/viewcvs.py/lbdmf/CPP/RegressionTests/Database/Plain-
ODBC-Test/test.cpp?rev=1.5&view=markup

It works on Windows/Linux/Mac OS X.

It has no very magic code to be copyrighted. So I think it should be GPL if Dave
agree.

Regards, Lothar

--
Lothar Behrens    |    Rapid Prototyping ...
Rosmarinstr 3        |
40235 Düsseldorf      |    www.lollisoft.de

Fri, Jan 13, 2006 at 04:33:38PM +0100, Marc Herbert wrote:
> Ludek Finstrle <luf@pzkagis.cz> writes:
> >> We (continuent) surely could invest some time developing such a suite
> >> provided we can easily run it against our software.
> >
> > Maybe I understand this bad. Are you volunteer?
>
> Yes, we will contribute to any test suite that fits our needs enough
> for us to use it regularly (I guess that's how things usually
> go!). One of our requirements is that it should NOT be too strongly
> tied to a specific DBMS (postgreSQL or other).

Is there some progress? Or do you have some time plan? Or ...?

Regards,

Luf

> - it contains executable, and dir with text files
> - each text file describe one test

Sounds like a language to be defined first. If I would create a test
suite, first I would
find the smallest peaces of code to be implemented in a native language
like C or C++.

These peaces reflect typical or untypical usecases of the SQLXXX API.

As I have some experience using plugins, I would suggest to implement
these test
peaces as plugins to be loaded by the name of the test.

To have more flexibility, I would define a language to get test data
for it, but delegate
the handling of the test to the plugin.

Regards,

Lothar

Thu, Feb 02, 2006 at 03:26:26AM -0800, lothar.behrens@lollisoft.de napsal(a):
> > - it contains executable, and dir with text files
> > - each text file describe one test
>
> Sounds like a language to be defined first. If I would create a test
> suite, first I would
> find the smallest peaces of code to be implemented in a native language
> like C or C++.

I don't think so. We need to describe what we want. Then we could choose
the best suitable language for it. Maybe it is C or C++ or Perl or
whatever suite the developer.

> These peaces reflect typical or untypical usecases of the SQLXXX API.
>
> As I have some experience using plugins, I would suggest to implement
> these test
> peaces as plugins to be loaded by the name of the test.

It isn't easy to add new test (plugin). When we could describe each test
with text file (which could be automatically generated from mylog output)
we could easy add the new test to testsuite. So each new release has
no old bug. When we'll collect enough tests we have rock solid ODBC driver.

> To have more flexibility, I would define a language to get test data
> for it, but delegate
> the handling of the test to the plugin.

BTW we get the most flexibility by using ODBC API calls with parameters
obtained from mylog output. We cover all the situations this way.

I'm not sure now if I undestand you well. What do you mean with
"language to get test data"?

Regards,

Luf

P.S. I'm sorry if I misunderstand you.

If you talk about a description in a text file, how do you interpret
that description in
an application to do that test ?

In my understanding, each description has a fixed format - eg. a
language. The simplest
one I could mention here may be ini:

[Test XYZ]
Param1 = Value1
Param2 = Value2
ParamN = ValueN

You may be able to generate these description data by scanning mylog
data. But in
my mind, the result is like a language a specific parser could
understand.

Regardles, how you would create the input files - detecting a login
scenario in mylog,
you would create a login test file reflecting that scenario.

When did you know, a new scenario has begun in mylog ?

I think, reverse engeneering the mylog to a description or script in a
language is very
difficult to handle and maintain.

I don't know the format of mylog, but I know how difficult it is to
scan log files - even only
to pick out some aspects only.

Who understands these mylog files ?

If I think about a plugin, I think about extensibility of the test
suite - regardless of the
language of implementation.

And more, if you want to compare a test between different databases,
who maintains
the generation of new tests from foreign logfile formats ?

What I mean, is, a programmer who uses ODBC API, may use typical
scenarious.
Like login, select, update, start/stop transactions. All these have a
typical call sequence
of ODBC API functions.

That is, what I mean with code peaces.

If you use these peaces to test pgSQL against, you can find bugs in the
implementation
of new stuff or unknown bugs that appear while you fixing other bugs.

The ODBC API dictates typical usecases. I think, these should be a
reference for tests.

New features in the ODBC driver should end up in a test for that
feature, thus extention.
I mean, if the infrastructure is available, it would be easy to extend.
Also it would be
easy to contribute new ones that don't break tested ones.

I hope to be constructive :-)

Regards,

Lothar

Thu, Feb 02, 2006 at 04:46:42AM -0800, lothar.behrens@lollisoft.de wrote:
> If you talk about a description in a text file, how do you interpret
> that description in an application to do that test ?
>
> In my understanding, each description has a fixed format - eg. a
> language.

What application do you mean? I hope you mean testsuite.

There is ODBC API specification. So we know all SQLXXX call and
it's parameters (input, output, sepcific stmt, maybe some more?).
mylog output isn't ready right now. But it may contains something
like:
SQLXXX(param1,param2,...)
It isn't problematic to parse it with e.g. perl and it is more
readable than ini (it has less limitations).

> You may be able to generate these description data by scanning mylog
> data. But in my mind, the result is like a language a specific parser
> could understand.

Ok.

> Regardles, how you would create the input files - detecting a login
> scenario in mylog, you would create a login test file reflecting that
> scenario.
>
> When did you know, a new scenario has begun in mylog ?

It isn't problematic. BTW there is no problem to replay whole run
with mylog. And the scenario begin at the first line of mylog ;-)
We need to create minimalistic reproduceable application now when
we want hunt the bug. It is the same way with every testsuites.

> I think, reverse engeneering the mylog to a description or script in a
> language is very difficult to handle and maintain.

Why? I see no problem. It gives me more power. I can creata test
from whatever programming language I want. It just easy enable mylog
output and create test text file from it (with some free available
script). Do I miss something important?

I think it's more problematic to rewrite the problem into DLL or
another prog. language with somem restrictions.

> I don't know the format of mylog, but I know how difficult it is to
> scan log files - even only
> to pick out some aspects only.

There is no problem. Unix tools like grep, or using perl is very very
simple. When I know what I need to find.

> Who understands these mylog files ?

I hope I understand these files :-) But I want change the mylog
output to generate better logging when it goes into SQLXXX and
when it returns something from SQLXXX.
BTW mylog output isn't so unreadable ;-)

> If I think about a plugin, I think about extensibility of the test
> suite - regardless of the language of implementation.

I think ODBC specification is fixed. I see no way to extend it.

> And more, if you want to compare a test between different databases,
> who maintains the generation of new tests from foreign logfile formats ?

Good point. I'm unable to answer this. But I see one big problem
with this. I see noone who do this for us :-(
BTW there is maybe some chance to get it with unixODBC or iODBC or
from MS ODBC manager tracing?

Still it is easier for normal user to generate mylog output and
send it to us then writing special application to detect the difference.
The test suite with text files from mylog output could be very
good for replying bugs without having reproduceable application.

> What I mean, is, a programmer who uses ODBC API, may use typical
> scenarious.

"May" - this is the real problem. Each implementation use different
access to ODBC driver. I see differences between VB, C++, BC++, Delphi,
...

> If you use these peaces to test pgSQL against, you can find bugs in the
> implementation of new stuff or unknown bugs that appear while you
> fixing other bugs.

I see no difference here againist text files. I can record all
peaces into text files ;-)

> New features in the ODBC driver should end up in a test for that
> feature, thus extention.

Yes I agree. No problem with text files. No difference.

> I hope to be constructive :-)

I hope so

Luf

"lothar.behrens@lollisoft.de" <lothar.behrens@lollisoft.de> writes:

> I think, reverse engeneering the mylog to a description or script in a
> language is very
> difficult to handle and maintain.
>
> I don't know the format of mylog, but I know how difficult it is to
> scan log files - even only
> to pick out some aspects only.

I don't think either it will be easy to base and manage a test suite
on log outputs.

I agree with Lothar on basically everything else he said in the
previous message (except I would replace "peace" with "piece" for
better clarity :-)

IMHO, a test suite should be made of snippets of real code, and be as
close as possible to real ODBC use, for two main reasons:

- introducing distance between the tests and the real applications may
introduce bugs or slightly different behaviours, and you end up
debugging the tests instead of the actual code.

- The more the extra dependencies besides ODBC (on external parsers,
languages, tools), the more the effort to run the suite, the less
people running it/using it/contributing to it. For instance, as Lothar
put it, by parsing postgreSQL files you prevent, say mysql (and
continuent FWIW) ODBC developers to contribute. And you create a
dependency on the format of logs, which are meant to be human-readable
and not stable.


That does imply that an ODBC test suite should be mostly written in
C/C++. At worst some Perl or alike could be used to parse & format the
results of the runs but ideally some C/C++ testing framework would
provide that. We have used the CppUnit framework to do that in Carob
and are mostly satisfied about it. Main drawbacks are the limited
documentation (compared to the numerous possibilities and flexibility)
and the verbosity of the code of tests. You can browse carob testing
code here: <https://forge.continuent.org/scm/?group_id=10>, in
directory "carob/test/"

Similar alternatives to CppUnit do exist.


Moreover, I think we definitely want to give a try to the unixodbc
test suite before doing anything. If not to opt for it and contribute
to it, at least to learn from it.


Cheers,

Marc.

A piece :-) of code may be better than a piece of database vendor
related log data.

But creating a different output from mylog into a more general list of
SQLXXX function
calls would be a good starting point to create reusable code pieces -
in any language.

Cascaded code translators then would generate test snippets for an
existing test
framework.

This way, we would get many test snippets very fast. They must be
approved for
insertion into the test suite, but I think not more work than an
approval procedure.

The idea to generate test descriptions from logfiles in general is
good. But the whole
system shouldn't related to pgSQLODBC only - in my opinion :-)

If you agree, then it should be designed more modular.

Such as input filters to translate vendor log data to a more general
APIXXX call log data.
Then we would not depend on the input. And ending with different output
translators.

I agree with Marc, to first have a look at the unixODBC test suite.

Ehhh, here is a (good - in my opinion) parsing generator tool:
http://www.devincook.com/goldparser/

Lothar