Обсуждение: Regarding useObjects

Dear All

I see support for useObjects was dropped some time in 2020, with this commit : 529e5dc3a8f071ad4946daf3bde1422c21a99524 .

Desperate mode question : was it replaced by smth else ? Any workaround ?

Serious mode question : Since eventually we'll have to bite the bullet and do an upgrade, how would you propose dealing with 1000s lines of java code using arrays of primitives ?

thank you

> I see support for useObjects was dropped some time in 2020, with this commit :
529e5dc3a8f071ad4946daf3bde1422c21a99524.
 

Could you provide a test case that demonstrates the failure?
It is hard to understand what you mean by "useObjects was dropped"

Vladimir

I see support for useObjects was dropped some time in 2020, with this commit : 529e5dc3a8f071ad4946daf3bde1422c21a99524 .

Could you provide a test case that demonstrates the failure?
It is hard to understand what you mean by "useObjects was dropped"

There used to be a variable inside : src/main/java/org/postgresql/jdbc/PgArray.java which was like :

private final boolean useObjects;
this.useObjects = true;

so there was the option to change this to false and recompile. This variable was removed between REL42.2.0 and REL42.3.0 . So not quite support, sorry for my wording.

Vladimir

-- 
Achilleas Mantzios IT DEV - HEAD IT DEPT Dynacom Tankers Mgmt

> so there was the option to change this to false and recompile

We try doing our best to keep backward compatibility both
compilation-wise and behaviour-wise.
However, we treat "private fields and methods" as private, and we do
not expect people would depend on them.
If you have to modify sources or depend on private APIs I would
suggest raising an issue or PR so it could be incorporated
into the driver itself.

Would you please provide a test case of what you are doing with PgArray?
What is exactly the sequence of JDBC calls, the actual and the
expected outcomes in your case?

The mentioned "useObjects" has been removed in
https://github.com/pgjdbc/pgjdbc/pull/1194 (Refactor decoding arrays)

Vladimir

Good Day Vladimir

On 9/27/23 09:03, Vladimir Sitnikov wrote:
>> so there was the option to change this to false and recompile
> We try doing our best to keep backward compatibility both
> compilation-wise and behaviour-wise.
> However, we treat "private fields and methods" as private, and we do
> not expect people would depend on them.
> If you have to modify sources or depend on private APIs I would
> suggest raising an issue or PR so it could be incorporated
> into the driver itself.
>
> Would you please provide a test case of what you are doing with PgArray?
> What is exactly the sequence of JDBC calls, the actual and the
> expected outcomes in your case?

E.g. for the case of floats, we have to change all:

java.sql.Array arr = rs.getArray(1);

Object objarr = arr.getArray();

float flarr[] = (float[]) objarr;

to

java.sql.Array arr = rs.getArray(1);

Object objarr = arr.getArray();

Float flarr[] = (Float[]) objarr;

>
> The mentioned "useObjects" has been removed in
> https://github.com/pgjdbc/pgjdbc/pull/1194 (Refactor decoding arrays)
>
> Vladimir

On Wed, 27 Sept 2023 at 05:00, Achilleas Mantzios - cloud <a.mantzios@cloud.gatewaynet.com> wrote:

Good Day Vladimir

On 9/27/23 09:03, Vladimir Sitnikov wrote:
>> so there was the option to change this to false and recompile
> We try doing our best to keep backward compatibility both
> compilation-wise and behaviour-wise.
> However, we treat "private fields and methods" as private, and we do
> not expect people would depend on them.
> If you have to modify sources or depend on private APIs I would
> suggest raising an issue or PR so it could be incorporated
> into the driver itself.
>
> Would you please provide a test case of what you are doing with PgArray?
> What is exactly the sequence of JDBC calls, the actual and the
> expected outcomes in your case?

E.g. for the case of floats, we have to change all:

java.sql.Array arr = rs.getArray(1);

Object objarr = arr.getArray();

float flarr[] = (float[]) objarr;

to

java.sql.Array arr = rs.getArray(1);

Object objarr = arr.getArray();

Float flarr[] = (Float[]) objarr;

So the problem with using float instead of Float is that it is impossible to have a null float and arrays can have nulls.

Thanks Dave, we wrote much of the code before 2008, and we had to implement our own version of nulls in arrays of primitives. So this code was stuck everywhere. Plus we got like 120+ remote sites (slaves) ( which run ... even older code) so we must fully support them as well. Its only about 200 source files in the central system's repo, so we will have to do it sooner or later.

The hard part would be migrating from pgsql 10 to 16.

Dave 

>
> The mentioned "useObjects" has been removed in
> https://github.com/pgjdbc/pgjdbc/pull/1194 (Refactor decoding arrays)
>
> Vladimir

> So the problem with using float instead of Float is that it is impossible to have a null float and arrays can have nulls.

There's a feature request for retrieving primitive arrays: https://github.com/pgjdbc/pgjdbc/issues/2939

back in pre-postgresql-42* days we used to use -888 and -888.888 to represent int NULL and float NULL respectively, quite ...primitive but it worked. We should retain this old mapping , but once all systems are up to date we will start storing and reading NULLs in arrays as NULLs.

Vladimir

On Wed, 27 Sept 2023 at 06:07, Achilleas Mantzios - cloud <a.mantzios@cloud.gatewaynet.com> wrote:

On 9/27/23 12:55, Vladimir Sitnikov wrote:

> So the problem with using float instead of Float is that it is impossible to have a null float and arrays can have nulls.

There's a feature request for retrieving primitive arrays: https://github.com/pgjdbc/pgjdbc/issues/2939

back in pre-postgresql-42* days we used to use -888 and -888.888 to represent int NULL and float NULL respectively, quite ...primitive but it worked. We should retain this old mapping , but once all systems are up to date we will start storing and reading NULLs in arrays as NULLs.

Using that mapping is not something that we would entertain. 

If you are going to actually store NULL in the array, how would that work with primitives ?

We support our central system (master) plus 120 slave systems communicating via satellites (running very old versions of everything , linux, postgersql , java, jdbc, etc) using custom replication code (a hack of DBmirror) that we wrote back in 2003 or so. Data come back and forth, so we have to support all those 120 archaic slaves while we upgrade and after we upgrade the central system.

When everything is up to date, (which will be some years from now), then we will start actually storing NULLs inside arrays instead of -888 and -888.888 . BUT even then, we are not willing to update the actual old data (-888) to the new NULL version, because that would trigger a massive replication traffic from our central system to the slaves. Also updating all past / historic values would cause bloating, huge autovacuum activity, among other things on those remote slaves that run unmanned (in simple english : I am the only DBA for all 1 + 120 postgresql instances) . So we should support historic data as they are, until something bigger happens (like a vast update in our topology/logic/etc).

Dave

Vladimir

On Wed, 27 Sept 2023 at 06:28, Achilleas Mantzios - cloud <a.mantzios@cloud.gatewaynet.com> wrote:

On 9/27/23 13:17, Dave Cramer wrote:

On Wed, 27 Sept 2023 at 06:07, Achilleas Mantzios - cloud <a.mantzios@cloud.gatewaynet.com> wrote:

On 9/27/23 12:55, Vladimir Sitnikov wrote:

> So the problem with using float instead of Float is that it is impossible to have a null float and arrays can have nulls.

There's a feature request for retrieving primitive arrays: https://github.com/pgjdbc/pgjdbc/issues/2939

back in pre-postgresql-42* days we used to use -888 and -888.888 to represent int NULL and float NULL respectively, quite ...primitive but it worked. We should retain this old mapping , but once all systems are up to date we will start storing and reading NULLs in arrays as NULLs.

Using that mapping is not something that we would entertain. 

If you are going to actually store NULL in the array, how would that work with primitives ?

We support our central system (master) plus 120 slave systems communicating via satellites (running very old versions of everything , linux, postgersql , java, jdbc, etc) using custom replication code (a hack of DBmirror) that we wrote back in 2003 or so. Data come back and forth, so we have to support all those 120 archaic slaves while we upgrade and after we upgrade the central system.

When everything is up to date, (which will be some years from now), then we will start actually storing NULLs inside arrays instead of -888 and -888.888 . BUT even then, we are not willing to update the actual old data (-888) to the new NULL version, because that would trigger a massive replication traffic from our central system to the slaves. Also updating all past / historic values would cause bloating, huge autovacuum activity, among other things on those remote slaves that run unmanned (in simple english : I am the only DBA for all 1 + 120 postgresql instances) . So we should support historic data as they are, until something bigger happens (like a vast update in our topology/logic/etc).

So it sounds like this would work for you if someone implemented primitive arrays. 

Yes or we could just upgrade JDBC driver in the central system, and touch those ~ 200 source files to just work (not throw exception), but still retaining the -888 representation. 

But an implementation of primitive arrays for ints,longs,floats,doubles would be better at this stage.

Dave