Обсуждение: Re: A Windows x64 port of PostgreSQL
On Wed, Jul 2, 2008 at 9:09 AM, Bernd Helmle <mailings@oopsware.de> wrote: > --On Mittwoch, Juli 02, 2008 07:39:29 -0400 Ken Camann <kjcamann@gmail.com> > wrote: > >> I assume it would only really matter if you did this to >> a pointer, and perhaps that is happening somewhere (but I doubt it >> since postgres runs fine on 64-bit POSIX OSes). These would be easy >> to fix, but very annoying given the number of them. > > At least, every Datum depends on something like that, see > src/include/postgres.h: > > sizeof(Datum) == sizeof(long) >= sizeof(void *) >= 4 Oh I see. Between this and looking again at the warning list, I see that it will probably take a lot more work than I thought. There are about 450 occurrences of the assumption that sizeof(size_t) == sizeof(int). Many of them come from string functions, but many are in important files like the tuplestore.c, bufpage.c, etc. I assume these these would need to be checked very carefully, and probably by someone who understands the internals better than I do. I'd really like to help out with this, but I'm not sure I can work on a patch even if I change these things for myself. Fixing this code would touch a lot of important internals in postgres (albeit in a small way), so my patch would probably not be accepted. I assume this kind of thing has to be done by someone a little closer to the project because it would modify so many things. On the other hand, if I just do this for myself and don't share it then I lose x64 support again when 8.4 comes out...and I really want the on-disk bitmap indices and better partitioning (if the latter ends up coming). So I'm not really sure what to do. I've never contributed to anything before, and I've actually never even used CVS. I will probably start making changes for myself, but before I do I'd like to know if I should bother doing it in such a way that might be useful to the community. Thanks, Ken
On Wed, Jul 2, 2008 at 8:32 PM, Ken Camann <kjcamann@gmail.com> wrote: > I'd really like to help out with this, but I'm not sure I can work on > a patch even if I change these things for myself. Fixing this code > would touch a lot of important internals in postgres (albeit in a > small way), so my patch would probably not be accepted. I assume this > kind of thing has to be done by someone a little closer to the project > because it would modify so many things. We don't work like that - anyone can have a patch accepted for any part of the code, no matter how much previous experience they have. One of the most complex patches we've ever dealt with (HOT) was largely written by a new contributor. > So I'm not really sure what to do. I've never contributed to anything > before, and I've actually never even used CVS. I will probably start > making changes for myself, but before I do I'd like to know if I > should bother doing it in such a way that might be useful to the > community. You might prefer to learn git rather than CVS - it will probably make life easier to keep your work in sync with CVS head through the use of the git mirror. Contributing is simple though. First, read the FAQ (http://wiki.postgresql.org/wiki/Developer_FAQ). Then, tell us you're working on the project, and start coding. Try to break the work down into logical sections to aid reviewing and discussion. Post WIP patches for people to look over, and don't be afraid to ask if you're not sure about something. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
"Ken Camann" <kjcamann@gmail.com> writes:
> Oh I see. Between this and looking again at the warning list, I see
> that it will probably take a lot more work than I thought. There are
> about 450 occurrences of the assumption that sizeof(size_t) ==
> sizeof(int).
[ blink... ] There are *zero* occurrences of the assumption that
sizeof(size_t) == sizeof(int), unless maybe in some of that grotty
#ifdef WIN32 code. Postgres has run on 64-bit platforms for many
years now.
regards, tom lane
On Wed, Jul 2, 2008 at 8:43 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > "Ken Camann" <kjcamann@gmail.com> writes: >> Oh I see. Between this and looking again at the warning list, I see >> that it will probably take a lot more work than I thought. There are >> about 450 occurrences of the assumption that sizeof(size_t) == >> sizeof(int). > > [ blink... ] There are *zero* occurrences of the assumption that > sizeof(size_t) == sizeof(int), unless maybe in some of that grotty > #ifdef WIN32 code. Postgres has run on 64-bit platforms for many > years now. Hi Tom. I knew about the previous 64 bit platform support, which is why I was so surprised to see the problem. Unless I am missing an important #define that somehow makes this stuff go away (but I don't think so, given how much of it there is) it does happen to be in there. If I haven't done anything wrong, I would assume no one noticed because those architectures define sizeof(long) to be >= sizeof(size_t). Well actually, let me be as strict as possible because I don't know the latest C standards very well (I am a C++ programmer). Am I correct that the standard says that sizeof(size_t) must be sizeof(void*), and that no compiler has ever said otherwise? I think so, given what size_t is supposed to mean. So I tend use sizeof(void*) and sizeof(size_t) interchangeably. Sorry for the confusion if that is less clear. According to postgres.h (not conditionally defined by anything) states that all the code assumes: sizeof(Datum) == sizeof(long) >= sizeof(void *) >= 4 where the first equation is reflexively true because Datum is a long typedef. EMT64/AMD64 is new compared to the older architectures, I would guess the older ones predate the time when it became a somewhat de facto standard to leave "long int" at 4 bytes, and make "long long" the new 64-bit type. In fact this definition is so common that it will soon be the de jour C++ standard definition. I assume ISO C still will not fix byte lengths to the declarators since they've fought it for so long. In any case, if sizeof(long) = 4 this fails to be true. This is more interesting still (in c.h) /** Size* Size of any memory resident object, as returned by sizeof.*/ typedef size_t Size; /** Index* Index into any memory resident array.** Note:* Indices are non negative.*/ typedef unsigned int Index; /** Offset* Offset into any memory resident array.** Note:* This differs from an Index in that an Index isalways* non negative, whereas Offset may be negative.*/ typedef signed int Offset; There seems to be an interesting mix of size_t, long, and int in use for memory. No one has noticed possibly because the shared buffers per single user have never been bigger than 2GB for anyone. Postgres documentation recommends "big" numbers like 20 or 30 MB, and the default is much smaller. In order to have had problems with this, you'd probably need all the following to happen at once: 1.) a huge enterprise (with lots of money to buy memory but using postgres and not Oracle) doing data warehousing on enormous tables 2.) on a platform where sizeof(int) = sizeof(long) = 4 but sizeof(void*) = 8 3.) a DBA who wanted the shared buffers > 2 GB 4.) An operating system supporting > 2GB of memory 5.) An operating system willing to allocate continuous blocks > 2 GB 6.) An cstdlib implementation of malloc willing to allocate continuous blocks > 2 GB 7.) Exactly the right query to make it explode. I think it happens to work out that not all of those have happened simultaneously yet. Anyway, there are a lot of other sizeof(int) == sizeof(size_t) assumptions in totally unimportant places, here's one in bootstrap.c "int len; len = strlen(str); //possible loss of data" That kind is very common. -Ken
"Ken Camann" <kjcamann@gmail.com> writes:
> Well actually, let me be as strict as possible because I don't know
> the latest C standards very well (I am a C++ programmer). Am I
> correct that the standard says that sizeof(size_t) must be
> sizeof(void*), and that no compiler has ever said otherwise?
I'm not sure that the spec actually requires that in so many words,
but it's hard to imagine a sane implementation where it isn't true.
(Now, I've worked on some less-than-sane hardware, but that stuff
all died the real death in the eighties or so. Flat memory models
have been the only kind anyone would tolerate for a long time now.)
> EMT64/AMD64 is new compared to the older architectures, I
> would guess the older ones predate the time when it became a somewhat
> de facto standard to leave "long int" at 4 bytes, and make "long long"
> the new 64-bit type.
Apparently your definition of "de facto standard" is "any idiotic
decision Microsoft cares to make". AFAIK there is *no* system other
than WIN64 where long is narrower than size_t; and I rather doubt that
there ever will be any. "long long" is generally understood to mean
a type that the hardware supports, but not very efficiently (ie, it's
double the native word width) --- and one would hope that pointers
are not in that category. On a 64-bit machine LL really ought to
denote 128-bit arithmetic ... I wonder what curious syntax Microsoft
will invent when they realize their compilers ought to support that?
Anyway, back to the immediate problem. What would probably make sense
to try as a first step is something like
#ifndef WIN64
typedef unsigned long Datum; /* XXX sizeof(long) >= sizeof(void *) */
#else
typedef unsigned long long Datum; /* Microsoft's out in left field */
#endif
and see how many warnings that eliminates ...
regards, tom lane
On Thu, Jul 3, 2008 at 12:39 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > "Ken Camann" <kjcamann@gmail.com> writes: >> EMT64/AMD64 is new compared to the older architectures, I >> would guess the older ones predate the time when it became a somewhat >> de facto standard to leave "long int" at 4 bytes, and make "long long" >> the new 64-bit type. > > Apparently your definition of "de facto standard" is "any idiotic > decision Microsoft cares to make". AFAIK there is *no* system other > than WIN64 where long is narrower than size_t; and I rather doubt that > there ever will be any. "long long" is generally understood to mean > a type that the hardware supports, but not very efficiently (ie, it's > double the native word width) --- and one would hope that pointers > are not in that category. On a 64-bit machine LL really ought to > denote 128-bit arithmetic ... I wonder what curious syntax Microsoft > will invent when they realize their compilers ought to support that? Actually, it isn't my definition. I haven't really worked with enough compilers to make a claim like that, I got that impression (and the phrase "de facto") from the proceedings of the C++0x standards committee where they finalized long long as being required to be 8 bytes. I think it ultimately does come from Microsoft/Intel, because they did follow the old width convention in the 16 bit days, that is sizeof(int) was 2 (word width) and sizeof(long) was 4. When 32-bit arrived (much too late, at Microsoft) most x86 compilers that had formerly used the segmented memory model made int 4 bytes like people felt "it was supposed to be" but left long at 4 the way it was so as not to bloat all the variables to double words on such a register-poor architecture as x86. I actually think of Borland Turbo C++ and Intel more than Microsoft when I think of this decision. For that reason, I would have thought you would see the same thing on all x86 systems...but now I realize that's stupid. Once you leave Windows its a gcc world, so it would be the way it always should have been, on every POSIX system. Even then though, if I were to use Linux on x64, wouldn't sizeof(int) be 4 and not 8? I bring this up for more reasons than spamming the list with trivia, it leads to a very important question: > Anyway, back to the immediate problem. What would probably make sense > to try as a first step is something like > > #ifndef WIN64 > typedef unsigned long Datum; /* XXX sizeof(long) >= sizeof(void *) */ > #else > typedef unsigned long long Datum; /* Microsoft's out in left field */ > #endif > > and see how many warnings that eliminates ... It's a question about style. If Microsoft Visual C really is the only one like this, then I guess there is no harm in #ifdef _WIN64 instead of #ifdef (some other name that captures the peculiarity of what is happening but isn't MSFT dependent). win32.h (not written by me) already defines SIZEOF_SIZE_T and SIZEOF_LONG_INT (or something like that)...It might be a better idea to use those two. But the thing is, this isn't the only issue. There is the fact that "int" appears in c.h for memory offsets and not long. As long as I might have to change a whole lot of this stuff to make exceptions for windows, I was wondering what the community thinks of the way this is all currently handled and what advice they have might have for me when changing stuff. There seems to be two problems that affect 64-bit POSIX systems too: 1. An object in memory can have size "Size" (= size_t). So its big (maybe 8 bytes). 2. An index into the buffer containing that object has index "Index" (= int) So its smaller (maybe 4 bytes). Now you can't index your big object, unless sizeof(size_t) = sizeof(int). But sizeof(size_t) must be at least 8 bytes on just about any 64-bit system. And sizeof(int) is still 4 most of the time, right? But I am thinking that if I knew more about postgres, maybe I would realize something important, like the way that it works ensures that Offset is never greater than a certain amount, because (for example, I have no idea if this is true or not) maybe its only used to point inside a disk block or something. Who would know this? -Ken
A bit long - the summary is that "intptr_t" should probably be used, assuming I understand the problem this thread is talking about: Ken Camann wrote: > 1. An object in memory can have size "Size" (= size_t). So its big > (maybe 8 bytes). > 2. An index into the buffer containing that object has index "Index" > (= int) So its smaller (maybe 4 bytes). Now you can't index your big > object, unless sizeof(size_t) = sizeof(int). But sizeof(size_t) must > be at least 8 bytes on just about any 64-bit system. And sizeof(int) > is still 4 most of the time, right I believe one of the mistakes here is an assumption that "int" is always the correct type to use for an index. This is not correct. "int" will be a type that is probably the most efficient word size for the target machine, and since "int" is usually ~32 bits these days, it will have a range that is sufficient for most common operations, therefore, it is commonly used. But, the C and C++ specifications do not define that an index into an array is of type "int". Rather, they defined E1[E2] as *((E1) + (E2)), and then the + operator is defined such that if one operand E1 is a pointer and operand E2 is an integer type, the result will be a pointer to the E2th element of E1 with the same pointer type as E1. "integer type" is not "int". It is any integer type. If the useful range of the array is 256 values, a "char" is acceptable for use as a "char" is an integer type. The optimizer might promote the "char" to a 32-bit or 64-bit machine register before calculating the result of the addition, but this is irrelevant to the definition of the C language. I think one could successfully argue that ptrdiff_t is the correct value to use for an array index that might use a range larger than "int" on a machine where sizeof(int) < sizeof(void*). ptrdiff_t represents the difference between two pointers. If P and Q are void* and I is ptrdiff_t, and Q - P = I, then &P[I] = Q. Though, I think it might be easier to use size_t. If I is of type size_t, and P = malloc(I), then P[0] ... P[I-1] are guaranteed to be addressable using a size_t. There is also the usable range, even on a machine with sizeof(size_t) of 64 bits. I don't think any existing machine can actually address 64-bits worth of continuous memory. 48-bits perhaps. Technically, sizeof(size_t) does not need to be sizeof(void*), and in fact, the C standard has this to say: "The types used for size_t and ptrdiff_t should not have an integer conversion rank greater than that of signed long int unless the implementation supports objects large enough to make this necessary." It doesn't define sizeof(size_t) in terms of sizeof(void*). The C standard defines long int as: "Their implementation-defined values shall be equal or greater in magnitude (absolute value) to those shown, with the same sign. ... — minimum value for an object of type long int LONG_MIN -2147483647 // −(2**31 − 1) — maximum value for an object of type long int LONG_MAX +2147483647 // 2**31 − 1" Based upon this definition, it appears that Windows 64 is compatible with the standard. That GCC took a different route that is also compatible with the standard is inconvenient, but a reality that should be dealt with. More comments from the C standard on this issue: "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type." The "portable" answer to this problem, is supposed to be intptr_t: "7.18.1.4 Integer types capable of holding object pointers The following type designates a signed integer type with the property that any valid pointer to void can be converted to this type, then converted back to pointer to void, and the result will compare equal to the original pointer: intptr_t The following type designates an unsigned integer type with the property that any valid pointer to void can be converted to this type, then converted back to pointer to void, and the result will compare equal to the original pointer: uintptr_t These types are optional." If Windows 64 has this type (not sure - I don't use Windows 64), then I believe intptr_t is the portable way to solve this problem. Note, though, that intptr_t does not guarantee that it can hold every integer value. For example, on a 32-bit platform, sizeof(intptr_t) might be 32 bits, and sizeof(long long) might be 64 bits. There is also this portable type: " 7.18.1.5 Greatest-width integer types The following type designates a signed integer type capable of representing any value of any signed integer type: intmax_t The following type designates an unsigned integer type capable of representing any value of any unsigned integer type: uintmax_t These types are required." I think this means that if PostgreSQL were to be designed to support all ISO C compliant platforms, PostgreSQL would have to use a union of intptr_t and intmax_t. Or, PostgreSQL will choose to not support some platforms. Windows 64 seems as if it may continue to be as popular as Windows 32, and should probably be supported. Cheers, mark -- Mark Mielke <mark@mielke.cc>
Am Donnerstag, 3. Juli 2008 schrieb Ken Camann: > > Anyway, back to the immediate problem. What would probably make sense > > to try as a first step is something like > > > > #ifndef WIN64 > > typedef unsigned long Datum; /* XXX sizeof(long) >= sizeof(void *) */ > > #else > > typedef unsigned long long Datum; /* Microsoft's out in left field > > */ #endif > > > > and see how many warnings that eliminates ... > > It's a question about style. If Microsoft Visual C really is the only > one like this, then I guess there is no harm in #ifdef _WIN64 instead > of #ifdef (some other name that captures the peculiarity of what is > happening but isn't MSFT dependent). win32.h (not written by me) > already defines SIZEOF_SIZE_T and SIZEOF_LONG_INT (or something like > that)...It might be a better idea to use those two. Style is something that we can worry about later, once we know how the code is supposed to behave. Coding Datum to be >= sizeof(* void) _and_ >= sizeof(long) and whatever else isn't that hard to do stylishly later on. > But the thing is, this isn't the only issue. There is the fact that > "int" appears in c.h for memory offsets and not long. As long as I > might have to change a whole lot of this stuff to make exceptions for > windows, I was wondering what the community thinks of the way this is > all currently handled and what advice they have might have for me when > changing stuff. Present us the actual problems as you discover them, and we will find a solution. Right now we are just guessing. > There seems to be two problems that affect 64-bit POSIX systems too: Well, 64-bit POSIX works just fine, so unless you can present an actual failure in practice, I suggest you do not worry about this.
Peter Eisentraut <peter_e@gmx.net> writes:
> Present us the actual problems as you discover them, and we will find a
> solution. Right now we are just guessing.
>> There seems to be two problems that affect 64-bit POSIX systems too:
> Well, 64-bit POSIX works just fine, so unless you can present an actual
> failure in practice, I suggest you do not worry about this.
I think the main thing Ken is missing is that there are large swaths of
the system that don't deal in objects larger than 1Gb, and thus do not
have any need of 64-bit sizes. In the places where it actually matters,
we use long or size_t.
To get a working WIN64 port it'd be necessary to go around and replace
long with size_t/ssize_t in the places where it matters --- but there
are not 450 of them, I don't think. And I'd advise not touching the
places that use int; that will just bloat the patch and make it harder
to review, without actually buying any functionality.
regards, tom lane
Tom Lane wrote: > To get a working WIN64 port it'd be necessary to go around and replace > long with size_t/ssize_t in the places where it matters --- but there > are not 450 of them, I don't think. And I'd advise not touching the > places that use int; that will just bloat the patch and make it harder > to review, without actually buying any functionality Plus - changing them all to 64-bit integers even for cases that will not ever require > 32-bit integers, is likely to be slower in all cases except for cases those that can be optimized to use only registers. I would use "int" by choice for any size that will never extend beyond 1 Gb as it is likely to perform the best. Cheers, mark -- Mark Mielke <mark@mielke.cc>
On Thu, Jul 03, 2008 at 01:45:06AM -0400, Ken Camann wrote: > When 32-bit arrived (much too late, at Microsoft) most x86 compilers > that had formerly used the segmented memory model made int 4 bytes > like people felt "it was supposed to be" but left long at 4 the way it > was so as not to bloat all the variables to double words on such a > register-poor architecture as x86. The usual way to talk about these things is that unix systems went for LP64 and Microsoft apparently went for LLP64. This link http://www.unix.org/version2/whatsnew/lp64_wp.html talks about it. I don't think anyone is going to change their position on this now. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
kjcamann@gmail.com ("Ken Camann") writes:
> On Thu, Jul 3, 2008 at 12:39 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> "Ken Camann" <kjcamann@gmail.com> writes:
>>> EMT64/AMD64 is new compared to the older architectures, I
>>> would guess the older ones predate the time when it became a somewhat
>>> de facto standard to leave "long int" at 4 bytes, and make "long long"
>>> the new 64-bit type.
>>
>> Apparently your definition of "de facto standard" is "any idiotic
>> decision Microsoft cares to make". AFAIK there is *no* system other
>> than WIN64 where long is narrower than size_t; and I rather doubt that
>> there ever will be any. "long long" is generally understood to mean
>> a type that the hardware supports, but not very efficiently (ie, it's
>> double the native word width) --- and one would hope that pointers
>> are not in that category. On a 64-bit machine LL really ought to
>> denote 128-bit arithmetic ... I wonder what curious syntax Microsoft
>> will invent when they realize their compilers ought to support that?
>
> Actually, it isn't my definition. I haven't really worked with enough
> compilers to make a claim like that, I got that impression (and the
> phrase "de facto") from the proceedings of the C++0x standards
> committee where they finalized long long as being required to be 8
> bytes. I think it ultimately does come from Microsoft/Intel, because
> they did follow the old width convention in the 16 bit days, that is
> sizeof(int) was 2 (word width) and sizeof(long) was 4.
AFAIK, we oughtn't care what C++ standards say, because PostgreSQL is
implemented in C, and therefore needs to follow what the *C* standards
say.
> When 32-bit arrived (much too late, at Microsoft) most x86 compilers
> that had formerly used the segmented memory model made int 4 bytes
> like people felt "it was supposed to be" but left long at 4 the way it
> was so as not to bloat all the variables to double words on such a
> register-poor architecture as x86. I actually think of Borland Turbo
> C++ and Intel more than Microsoft when I think of this decision. For
> that reason, I would have thought you would see the same thing on all
> x86 systems...but now I realize that's stupid. Once you leave Windows
> its a gcc world, so it would be the way it always should have been, on
> every POSIX system. Even then though, if I were to use Linux on x64,
> wouldn't sizeof(int) be 4 and not 8?
#include <stdio.h>
int main (int argc, char **argv) { int i; long j; long long k; printf ("size of int: %d\n", sizeof(i)); printf
("sizeof long: %d\n", sizeof(j)); printf ("size of long long: %d\n", sizeof(k));
}
Output on 32 bit Linux:
size of int: 4
size of long: 4
size of long long: 8
Output on 64 bit Linux:
size of int: 4
size of long: 8
size of long long: 8
--
output = ("cbbrowne" "@" "linuxfinances.info")
http://linuxfinances.info/info/
"The real romance is out ahead and yet to come. The computer
revolution hasn't started yet. Don't be misled by the enormous flow of
money into bad defacto standards for unsophisticated buyers using poor
adaptations of incomplete ideas." -- Alan Kay
chris wrote: >> C++0x standards >> committee where they finalized long long as being required to be 8 > > AFAIK, we oughtn't care what C++ standards say, because PostgreSQL is > implemented in C, and therefore needs to follow what the *C* standards > say. I agree the C++ standards should matter one bit to postgresql, but AFAIK C99 also says "long long" is at least 64 bits too -- but if we're talking C99, we'd be better off using whichever of int64_t or int_least64_t or int_fast64_t we really meant anyway. Since we don't I assume we're trying to be compatible with pre-c99 C too which AFAICT means you can't assume much about "long long" either. Pre-C99 you can't really count on much. I've spent time where "int" was 20 bits; and on another platform where int was 32 bits and long 40 bits.