Обсуждение: Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.

Hi,

Please keep the list in copy, especially if that's about Windows specific as
I'm definitely not very knowledgeable about it.

On Fri, Apr 01, 2022 at 09:18:03AM +0000, Wilm Hoyer wrote:
> 
> If you don't wanna go the manifest way, maybe the RtlGetVersion function is the one you need:
> https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-rtlgetversion?redirectedfrom=MSDN

Thanks for the info!  I tried to use the function but trying to include either
wdm.h or Ntddk.h errors out.  Unfortunately I don't know how to look for a file
in Windows so I don't even know if those files are present.

I searched a bit and apparently some people are using this function directly
opening some dll, which seems wrong.

> Another Idea on windows machines would be to use the commandline to execute
> ver in a separate Process and store the result in a file.

That also seems hackish, I don't think that we want to rely on something like
that.

> >> While winver.exe on the same vm says windows 11, version 21H2, build 22000.493.
> 
> > So, what GetVersionEx returns is actually "it depends", and this is documented:
> 
> >> With the release of Windows 8.1, the behavior of the GetVersionEx API 
> >> has changed in the value it will return for the operating system 
> >> version. The value returned by the GetVersionEx function now depends 
> >> on how the application is manifested.
> >>
> >> Applications not manifested for Windows 8.1 or Windows 10 will return 
> >> the Windows 8 OS version value (6.2). Once an application is 
> >> manifested for a given operating system version, GetVersionEx will 
> >> always return the version that the application is manifested for in 
> >> future releases. To manifest your applications for Windows 8.1 or 
> >> Windows 10, refer to Targeting your application for Windows.
> 
> The documentation is a bit unclear - with the correct functions you should get the:
> Minimum( actualOS-Version, Maximum(Manifested OS Versions))
> The Idea behind, as I understand it, is to better support virtualization and
> backward compatibility - you manifest only Windows 8.1 -> than you always get
> a System that behaves like Windows 8.1 in every aspect.  (Every Aspect not
> true in some corner cases due to security patches)

Well, it clearly does *NOT* behave as a Windows 8.1, even if for some reason
large pages relies on security patches.

Their API is entirely useless, so I'm still on the opinion that we should
unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
day.

On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> I searched a bit and apparently some people are using this function directly
> opening some dll, which seems wrong.

I was wondering about this whole business, and the manifest approach
is a *horrible* design for an API where the goal is to know if your
run-time environment is greater than a given threshold.

>> Another Idea on windows machines would be to use the commandline to execute
>> ver in a separate Process and store the result in a file.
>
> That also seems hackish, I don't think that we want to rely on something like
> that.

Hmm.  That depends on the dependency set, I guess.  We do that on
Linux at some extent to for large pages in sysv_shmem.c.  Perhaps this
could work for Win10 if this avoids the extra loopholes with the
manifests.

> Their API is entirely useless,

This I agree.

> so I'm still on the opinion that we should
> unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
> day.

Are we sure that this is not going to cause failures in environments
where the flag is not supported?
--
Michael

On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
>> I searched a bit and apparently some people are using this function
>> directly opening some dll, which seems wrong.

> I was wondering about this whole business, and the manifest approach is a *horrible* design for an API where the goal
isto know if your run-time environment is greater than a given threshold. 

Agreed for the use case at hand, where you want to use one core API Function or another depending on the OS Version.

One Blog from Microsoft, I remember, told that one reason for the change were the increase of false installation error
messages"Install Error - Your system does not meet the minimum supported operating system and service pack level." 
where the software in question was written for Windows XP and the user tried to install it on, say, Windows 8.
That is just a Developer-Pilot error, where the Developers forgot to anticipate future OS Versions and instead of
checkingfor Version at least, where checking for Version equality of all at design time known Windows Version. 
Since you can develop only against OS APIs known at design time, and Microsoft claims to be pretty good at maintaining
backwardcompatible facades for their APIs, there is some reason in that decision. 
(To only see the Versions and APIs you told the OS with the manifest, you knew about at compile time).
The core Problem at hand is, that ms broke the promise of backward compatibility, since the function in question is
workingdifferently, depending on windows version, although with the above reasoning we should get the exact same
behavioron windows 10 as on windows 8.1 (as PostgreSql, per default, only claims to know about Windows 8.1 features). 

That said, I can understand the design decision. Personally, I still don't like it a bit, since developers should be
allowedto make some stupid mistakes. 

>>> Another Idea on windows machines would be to use the commandline to
>>> execute ver in a separate Process and store the result in a file.
>>
>> That also seems hackish, I don't think that we want to rely on
>> something like that.

>Hmm.  That depends on the dependency set, I guess.  We do that on Linux at some extent to for large pages in
sysv_shmem.c. Perhaps this could work for Win10 if this avoids the extra loopholes with the >manifests. 

I used the following hack to get the "real" Major and Minor Version of Windows - it's in C# (.Net) and needs to be
adjusted(you can compile as x64 and use a long-long as return value ) to return the Service Number too and translated
itinto C. 
I share it anyways, as it might help - please be kind, as it really is a little hack.

Situation:
Main Application can't or is not willing to add a manifest file into its resources.

Solution:
Start a small executable (which has a manifest file compiled into its resources), let it read the OS Version and code
theVersion into its return Code. 

CInt32 is basically an integer redefinition, where one can access the lower and higher Int16 separately.

The Main Programm eventually calls this (locate the executable, adjust the Process startup to be minimal, call the
executableas separate process and interpret the return value as Version): 
private static Version ReadModernOsVersionInternal()
{
    String codeBase = Assembly.GetExecutingAssembly().CodeBase;
    Uri uri = new Uri(codeBase);

    String localPath = uri.LocalPath;
    String pathDirectory = Path.GetDirectoryName(localPath);

    if (pathDirectory != null)
    {
        String fullCombinePath = Path.Combine(pathDirectory, "Cf.Utilities.ReadModernOSVersion");

        ProcessStartInfo processInfo = new ProcessStartInfo
        {
            FileName = fullCombinePath,
            CreateNoWindow = true,
            UseShellExecute = false
        };

        Process process = new Process
        {
            StartInfo = processInfo
        };

        process.Start();

        if (process.WaitForExit(TimeSpan.FromSeconds(1).Milliseconds))
        {
            CInt32 versionInteger = process.ExitCode;
            return new Version(versionInteger.HighValue, versionInteger.LowValue);
        }
    }

    return new Version();
}


The small Version Check executable:

static Int32 Main(String[] args)
{
    return OsVersionErmittler.ErmittleOsVersion();
}

and

static class OsVersionErmittler
{
    /// <summary>
    /// Ermittelt die OsVersion und übergibt diese als High und LowWord.
    /// </summary>
    /// <returns></returns>
    public static CInt32 ErmittleOsVersion()
    {
        OperatingSystem version = Environment.OSVersion;
        if (version.Platform == PlatformID.Win32NT && version.Version >= new Version(6, 3))
        {
            String versionString = version.VersionString;
            return new CInt32((Int16) version.Version.Major, (Int16) version.Version.Minor);
        }
        return 0;
    }
}

The shortened manifest of the small executable:
<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
    <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
    <application>
      <!-- Eine Liste der Windows-Versionen, unter denen diese Anwendung getestet
           und für die sie entwickelt wurde. Wenn Sie die Auskommentierung der entsprechenden Elemente aufheben,
           wird von Windows automatisch die kompatibelste Umgebung ausgewählt. -->

      <!-- Windows Vista -->
      <!--<supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}" />-->

      <!-- Windows 7 -->
      <!--<supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}" />-->

      <!-- Windows 8 -->
      <!--<supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}" />-->

      <!-- Windows 8.1 -->
      <supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}" />

      <!-- Windows 10 -->
      <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}" />

    </application>
  </compatibility>

 </assembly>


I hope I'm not intrusive, otherwise, feel free to ignore this mail,
Wilm.

On Wed, Apr 27, 2022 at 05:13:12PM +0900, Michael Paquier wrote:
> On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> > so I'm still on the opinion that we should
> > unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
> > day.
> 
> Are we sure that this is not going to cause failures in environments
> where the flag is not supported?

I don't know for sure as I have no way to test, but it would be very lame for
an OS to provide a #define explicitly intended for one use case if that use
case can't handle that flag yet.

On Wed, Apr 27, 2022 at 03:04:23PM +0000, Wilm Hoyer wrote:
>
> I used the following hack to get the "real" Major and Minor Version of
> Windows - it's in C# (.Net) and needs to be adjusted (you can compile as x64
> and use a long-long as return value ) to return the Service Number too and
> translated it into C.
> I share it anyways, as it might help - please be kind, as it really is a
> little hack.
>
> Situation:
> Main Application can't or is not willing to add a manifest file into its
> resources.
>
> Solution:
> Start a small executable (which has a manifest file compiled into its
> resources), let it read the OS Version and code the Version into its return
> Code.

Thanks for sharing.

Having to compile another tool just for that seems like a very high price to
pay, especially since we don't have any C# code in the tree.  I'm not even sure
that compiling this wouldn't need additional requirements and/or if it would
work on our oldest supported Windows versions.

On Wed, Apr 27, 2022 at 03:04:23PM +0000, Wilm Hoyer wrote:
>>
>> I used the following hack to get the "real" Major and Minor Version of
>> Windows - it's in C# (.Net) and needs to be adjusted (you can compile
>> as x64 and use a long-long as return value ) to return the Service
>> Number too and translated it into C.
>> I share it anyways, as it might help - please be kind, as it really is
>> a little hack.
>>
>> Situation:
>> Main Application can't or is not willing to add a manifest file into
>> its resources.
>>
>> Solution:
>> Start a small executable (which has a manifest file compiled into its
>> resources), let it read the OS Version and code the Version into its
>> return Code.

> Thanks for sharing.

You are welcome.

> Having to compile another tool just for that seems like a very high price to pay, especially since we don't have any
C#code in the tree.  I'm not even sure that compiling this wouldn't need additional  requirements and/or if it would
workon our oldest supported Windows versions. 

With "translate it into C" I meant "tread it as pseudo code, for a solution in plain C" (e.g. substitude
Environment.OSVersionwith IsWindowsVersionOrGreater or GetVersion ) 

On Wed, Apr 27, 2022 at 05:13:12PM +0900, Michael Paquier wrote:
> On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> > so I'm still on the opinion that we should unconditionally use the
> > FILE_MAP_LARGE_PAGES flag if it's defined and call it a day.
>
> Are we sure that this is not going to cause failures in environments
> where the flag is not supported?

I'm not that familiar with the Microsoft OS or C (that's why I haven't migrated the c# code to C in the first place) to
havea clear answer to that question. 

If there is any risk and you want to avoid it, I can share a search result when I faced the same issue for our
application.I declined this solution in favor of the previously shared one. 
It's from NUnit (and needs migration to C as well - but since it just involves the Registry this should be pretty
forward).
Just in case the Framework is not known: NUnit is the most popular .Net port of the Unit Testing Framework JUnit. There
exitsa C port too (CUnit) Maybe in there you can find an OS Version check too. 

// Copyright (c) Charlie Poole, Rob Prouse and Contributors. MIT License - see LICENSE.txt
[...]
namespace NUnit.Framework.Internal
{
    [SecuritySafeCritical]
    public class OSPlatform
    {
[...]
        /// <summary>
        /// Gets the actual OS Version, not the incorrect value that might be
        /// returned for Win 8.1 and Win 10
        /// </summary>
        /// <remarks>
        /// If an application is not manifested as Windows 8.1 or Windows 10,
        /// the version returned from Environment.OSVersion will not be 6.3 and 10.0
        /// respectively, but will be 6.2 and 6.3. The correct value can be found in
        /// the registry.
        /// </remarks>
        /// <param name="version">The original version</param>
        /// <returns>The correct OS version</returns>
        private static Version GetWindows81PlusVersion(Version version)
        {
            try
            {
                using (var key = Registry.LocalMachine.OpenSubKey(@"SOFTWARE\Microsoft\Windows NT\CurrentVersion"))
                {
                    if (key != null)
                    {
                        var buildStr = key.GetValue("CurrentBuildNumber") as string;
                        int.TryParse(buildStr, out var build);

                        // These two keys are in Windows 10 only and are DWORDS
                        var major = key.GetValue("CurrentMajorVersionNumber") as int?;
                        var minor = key.GetValue("CurrentMinorVersionNumber") as int?;
                        if (major.HasValue && minor.HasValue)
                        {
                            return new Version(major.Value, minor.Value, build);
                        }

                        // If we get here, we are not Windows 10, so we are Windows 8
                        // or 8.1. 8.1 might report itself as 6.2, but will have 6.3
                        // in the registry. We can't do this earlier because for backwards
                        // compatibility, Windows 10 also has 6.3 for this key.
                        var currentVersion = key.GetValue("CurrentVersion") as string;
                        if(currentVersion == "6.3")
                        {
                            return new Version(6, 3, build);
                        }
                    }
                }
            }
            catch (Exception)
            {
            }
            return version;
        }
[...]
   }
}

Finally, my reasoning to use the executable solution in favor of the NUnit one:
I found no guarantee from Microsoft regarding the keys and values in the registry - hence a change with an update or in
anewer Windows is not likely, but still possible. That's no problem for a heavily used and supported framework like
NUnit- they are likely to adopt within days of a new Windows release. I on the other hand wanted a solution with small
tono support. That's why I decided to implement a solution that's as in line as possible with the official Microsoft
advicefor targeting newer OS Versions. 

Best regards
Wilm.

On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> Their API is entirely useless, so I'm still on the opinion that we should
> unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
> day.

Now that the minimal runtime version is Windows 10 in v16~ thanks to
495ed0e, we could be much more aggressive and do the attached, which
is roughly what Thomas has proposed upthread at the exception of
assuming that FILE_MAP_LARGE_PAGES always exists, because updates are
forced by MS in this environment.  We could make it conditional, of
course, with an extra #ifdef painting.
--
Michael