Обсуждение: Add LINE: hint when schemaname.typename is a non-existent schema

The attached patch adds a LINE: ... hint when schemaname.typename
results in a schema which does not exist. I came across this when a
missing comma in a SELECT list resulted in an error without a location
in a query a few thousand lines long.

Before:

(postgres@[local]:5432 14:41:25) [postgres]> select test.id 'all' as
example from test;
ERROR:  3F000: schema "test" does not exist
LOCATION:  get_namespace_oid, namespace.c:2826

After:

(postgres@[local]:5433 14:42:32) [postgres]> select test.id 'all' as
example from test;
ERROR:  3F000: schema "test" does not exist
LINE 1: select test.id 'all' as example from test;
               ^
LOCATION:  LookupTypeName, parse_type.c:171

-Ryan Kelly

On Mon, Feb 2, 2015 at 2:44 PM, Ryan Kelly <rpkelly22@gmail.com> wrote:
> The attached patch adds a LINE: ... hint when schemaname.typename
> results in a schema which does not exist. I came across this when a
> missing comma in a SELECT list resulted in an error without a location
> in a query a few thousand lines long.
>
> Before:
>
> (postgres@[local]:5432 14:41:25) [postgres]> select test.id 'all' as
> example from test;
> ERROR:  3F000: schema "test" does not exist
> LOCATION:  get_namespace_oid, namespace.c:2826
>
> After:
>
> (postgres@[local]:5433 14:42:32) [postgres]> select test.id 'all' as
> example from test;
> ERROR:  3F000: schema "test" does not exist
> LINE 1: select test.id 'all' as example from test;
>                ^
> LOCATION:  LookupTypeName, parse_type.c:171
>
> -Ryan Kelly

Please add your patch to
https://commitfest.postgresql.org/action/commitfest_view/open so do we
don't forget about it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Ryan Kelly <rpkelly22@gmail.com> writes:
> The attached patch adds a LINE: ... hint when schemaname.typename
> results in a schema which does not exist. I came across this when a
> missing comma in a SELECT list resulted in an error without a location
> in a query a few thousand lines long.

I think this is a good problem to attack, but the proposed patch seems
a bit narrow and brute-force.  Surely this isn't the only place where
we're missing an errposition pointer on a "no such schema" error?

It's probably not reasonable to add a pstate argument to 
LookupExplicitNamespace itself, given that namespace.c isn't part
of the parser.  But you could imagine adding an interface function
in the parser that calls LookupExplicitNamespace and then throws a
position-aware error on failure, and then making sure that all schema
lookups in the parser go through that.
        regards, tom lane

On Tue, Feb 3, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Ryan Kelly <rpkelly22@gmail.com> writes:
>> The attached patch adds a LINE: ... hint when schemaname.typename
>> results in a schema which does not exist. I came across this when a
>> missing comma in a SELECT list resulted in an error without a location
>> in a query a few thousand lines long.
>
> I think this is a good problem to attack, but the proposed patch seems
> a bit narrow and brute-force.  Surely this isn't the only place where
> we're missing an errposition pointer on a "no such schema" error?

I managed to find four code paths that failed to report an error position if
the schema did not exist:
- Schema-qualified types (e.g. select schemaname.typename 'foo')
- Schema-qualified operators (e.g. select 1 operator(schemaname.+) 1)
- Schema-qualified table names in CREATE TABLE (e.g. create table
schemaname.tablename (colname integer))
- Schema-qualified function calls (e.g. select schemaname.funcname())

> It's probably not reasonable to add a pstate argument to
> LookupExplicitNamespace itself, given that namespace.c isn't part
> of the parser.  But you could imagine adding an interface function
> in the parser that calls LookupExplicitNamespace and then throws a
> position-aware error on failure, and then making sure that all schema
> lookups in the parser go through that.

While searching for other instances of this problem, I found that using
a schema that does not exist in conjunction with a collation reported the
position. That code uses setup_parser_errposition_callback and the
documentation for that function seems to indicate that the usage of it in my
newly attached patch are consistent. I do not know, however, if this is the
cleanest approach or if I should actually create a function like
validate_explicit_namespace to handle this (and I'm also not sure where
such a function should live, if I do create it).

-Ryan Kelly

Ryan Kelly <rpkelly22@gmail.com> writes:
> On Tue, Feb 3, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It's probably not reasonable to add a pstate argument to
>> LookupExplicitNamespace itself, given that namespace.c isn't part
>> of the parser.  But you could imagine adding an interface function
>> in the parser that calls LookupExplicitNamespace and then throws a
>> position-aware error on failure, and then making sure that all schema
>> lookups in the parser go through that.

> While searching for other instances of this problem, I found that using
> a schema that does not exist in conjunction with a collation reported the
> position. That code uses setup_parser_errposition_callback and the
> documentation for that function seems to indicate that the usage of it in my
> newly attached patch are consistent. I do not know, however, if this is the
> cleanest approach or if I should actually create a function like
> validate_explicit_namespace to handle this (and I'm also not sure where
> such a function should live, if I do create it).

Yeah, setup_parser_errposition_callback would work too.  I'm not sure
offhand which I like better.  One thing to keep in mind is that the
callback approach results in adding an error cursor position to *any*
error thrown while the callback is active, not only "schema not found".
There are pluses and minuses to that.  I've seen error cursors attached
to very bizarre internal problems that (more or less by chance) showed up
while the parser was looking up a table name, but weren't really connected
to the table name at all.  OTOH, most of the time you'd just as soon not
be too picky about what conditions you provide a cursor for.

The main place I'd question what you did is the callback placement around
make_oper_cache_key --- that might work, but it seems very bizarre, and
perhaps more than usually prone to the "cursor given for unrelated
problem" issue.  Perhaps better to push it down inside that function
so it surrounds just the namespace lookup call.

Also the diffs in parse_utilcmd.c are very confusing and seem to change
more code than is necessary --- why did that happen?
        regards, tom lane

The following review has been posted through the commitfest application:
make installcheck-world:  not tested
Implements feature:       tested, failed
Spec compliant:           not tested
Documentation:            not tested

Tom suggested few changes already which I too think author needs to address.
So marking it "Waiting on Author".

However, I see following, example does not work well.

postgres=# create or replace function
f1(a abc.test.id%type) returns int as
$$ select 1; $$
language sql;
ERROR:  schema "abc" does not exist

Is that expected?
I guess we need it at all places in parse_*.c where we will look for namespace.
Please fix.


Also, like Tom's suggestion on make_oper_cache_key, can we push down this
inside func_get_detail() as well, just to limit it for namespace lookup?

However, patch is not getting applied cleanly on latest sources. Need rebase.

> On Tom comments on parse_utilcmd.c:
I guess the block is moved after the pstate and CreateStmtContext are setup
properly.  I guess, we can move just after pstate setup, so that it will
result into minimal changes?

Can we have small test-case? Or will it be too much for this feature?

The new status of this patch is: Waiting on Author

In the interest of moving forward, I have updated this patch because
Ryan has been inactive for over a month now.

Tom Lane wrote:

> Yeah, setup_parser_errposition_callback would work too.  I'm not sure
> offhand which I like better.  One thing to keep in mind is that the
> callback approach results in adding an error cursor position to *any*
> error thrown while the callback is active, not only "schema not found".
> There are pluses and minuses to that.  I've seen error cursors attached
> to very bizarre internal problems that (more or less by chance) showed up
> while the parser was looking up a table name, but weren't really connected
> to the table name at all.  OTOH, most of the time you'd just as soon not
> be too picky about what conditions you provide a cursor for.

I think we can live with cursor positions in some weird corner cases.
If we later find out that we don't like it for some reason, we can
reduce the scope that this applies to.

> The main place I'd question what you did is the callback placement around
> make_oper_cache_key --- that might work, but it seems very bizarre, and
> perhaps more than usually prone to the "cursor given for unrelated
> problem" issue.  Perhaps better to push it down inside that function
> so it surrounds just the namespace lookup call.

Agreed; the attached patch does it that way.  (I notice that we have the
pstate as first arg in many places; I put at the end for
make_oper_cache_key, together with location.  Is there some convention
to have it as first arg?)

> Also the diffs in parse_utilcmd.c are very confusing and seem to change
> more code than is necessary --- why did that happen?

The reason appears to be that Ryan wanted to have the pstate set, but
that was only created after looking other things up, so he moved a
largish block down; this was pretty bogus AFAICT.  The attached patch
fixes this by first creating the pstate, then doing the namespace
lookup, then doing the rest of the setup.

It's a bit disappointing to see so little changes in regression expected
output ...

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Agreed; the attached patch does it that way.  (I notice that we have the
> pstate as first arg in many places; I put at the end for
> make_oper_cache_key, together with location.  Is there some convention
> to have it as first arg?)

Yes, parser-related functions always have pstate as first arg if they
need it at all.  Please follow that convention.
        regards, tom lane

Álvaro,

I think, there are few open questions here and thus marking it back to "Waiting on Author".

Please have your views on the review comments already posted.
Also make changes as Tom suggested about placing pstate at the beginning.

I am more concerned about this:

1.
postgres=# create or replace function
f1(a abc.test.id%type) returns int as
$$ select 1; $$
language sql;
ERROR:  schema "abc" does not exist

Is that expected?

2.
Also what about pushing setup_parser_errposition_callback() inside func_get_detail() as well, just to limit it for
namespacelookup?
 

Thanks

The new status of this patch is: Waiting on Author

Jeevan Chalke <jeevan.chalke@gmail.com> writes:
> I am more concerned about this:

> 1.
> postgres=# create or replace function
> f1(a abc.test.id%type) returns int as
> $$ select 1; $$
> language sql;
> ERROR:  schema "abc" does not exist

> Is that expected?

Yes, or at least, if it's not what we want it's not this patch's fault.
That behavior is pre-existing.

> Also what about pushing setup_parser_errposition_callback() inside func_get_detail() as well, just to limit it for
namespacelookup?
 

Adding a pstate parameter to func_get_detail would be rather invasive;
not sure it's worth it.
        regards, tom lane

Jeevan Chalke wrote:
> Álvaro,
> 
> I think, there are few open questions here and thus marking it back to "Waiting on Author".
> 
> Please have your views on the review comments already posted.

For some reason I missed your previous email.

> Also make changes as Tom suggested about placing pstate at the beginning.

Sure.

> 1.
> postgres=# create or replace function
> f1(a abc.test.id%type) returns int as
> $$ select 1; $$
> language sql;
> ERROR:  schema "abc" does not exist
> 
> Is that expected?

Type resolution for function arguments happens at execution time,
in interpret_function_parameter_list() as called by CreateFunction().
We don't have a pstate at that point which is why location is not
reported.  Fixing that seems like a whole new project.

> 2.
> Also what about pushing setup_parser_errposition_callback() inside func_get_detail() as well, just to limit it for
namespacelookup?
 

Hm, that sounds more reasonable.  Actually it means we have to drill
all the way down to FuncnameGetCandidates.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Jeevan,

Thanks for the review.

Jeevan Chalke wrote:

> I think, there are few open questions here and thus marking it back to "Waiting on Author".
> 
> Please have your views on the review comments already posted.
> Also make changes as Tom suggested about placing pstate at the beginning.

Pushed with that one change.

> postgres=# create or replace function
> f1(a abc.test.id%type) returns int as
> $$ select 1; $$
> language sql;
> ERROR:  schema "abc" does not exist
> 
> Is that expected?

We can leave this for a future patch.

> 2.
> Also what about pushing setup_parser_errposition_callback() inside func_get_detail() as well, just to limit it for
namespacelookup?
 

Seemed messy enough that I left this as-is; and as Tom said elsewhere, I
think it's okay to have cursor position in other errors too.  At the
very least, the user will know for certain what's the function being
processed that caused whatever failure.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services