Re: invalidly encoded strings

Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>   
>> Tom Lane wrote:
>>     
>>> In the short run it might be best to do it in scan.l after all.  
>>>       
>
>   
>> I have not come up with a way of doing that and handling the bytea case. 
>>     
>
> AFAICS we have no realistic choice other than to reject \0 in SQL
> literals; to do otherwise requires API changes throughout that stack of
> modules.  And once you admit \0 is no good, it's not clear that
> \somethingelse is any better for bytea-using clients.  Moreover, given
> that we are moving away from backslash escapes as fast as we can sanely
> travel, expending large amounts of energy to make them work better
> doesn't seem like a good use of development manpower.
>
>   
>   

Perhaps we're talking at cross purposes.

The problem with doing encoding validation in scan.l is that it lacks 
context. Null bytes are only the tip of the bytea iceberg, since any 
arbitrary sequence of bytes can be valid for a bytea. So we can only do 
validation of encoding on a literal when we know it isn't destined for a 
bytea. That's what I haven't come up with a way of doing in the scanner 
(and as you noted upthread it's getting pretty darn late in the cycle 
for us to be looking for ways to do things).

I still don't see why it's OK for us to do validation from the foo_recv 
functions but not the corresponding foo_in functions. At least in the 
short term that would provide us with fairly complete protection against 
accepting invalidly encoded data into the database, once we fix up 
chr(), without having to mess with the scanner, parser, COPY code etc. 
We could still get corruption from UDFs and UDTs - it's hard to see how 
we can avoid that danger. Yes, we would need to make sure that any 
additions to the type system acted properly, and yes we should fix up 
any validation inefficiencies (like possibly inlining calls in the UTF8 
validation code). Personally, I don't see those as killer objections.

cheers

andrew