[Fwd: Maximum filesize of one file]

> I have a 40 million record database (about 40GB and growing) and I
> notice that some of my multicolumn indices are beginning to approach
> 2GB on disk.
>
> Will there be any problems spanning the 32 bit limit here (a la tables
> in version 6.4)?

Please refer to the attached messages from linux-raid@vger.rutgers.edu.

CNReturn-Path: <owner-linux-raid-outgoing@vger.rutgers.edu>
Received: from nic.funet.fi (nic.funet.fi [128.214.248.6])
    by mail.sinyih.com.tw (8.9.1/8.9.1) with ESMTP id RAA19559
    for <cn@mail.sinyih.com.tw>; Sun, 14 Mar 1999 17:57:56 +0800
Received: from vger.rutgers.edu ([128.6.190.2]:49714 "EHLO vger.rutgers.edu" ident: "NO-IDENT-SERVICE[2]") by
nic.funet.fiwith ESMTP id <10060-29820>; Sun, 14 Mar 1999 03:56:04 +0200 
Received: by vger.rutgers.edu via listexpand id <160871-212>; Sat, 13 Mar 1999 20:56:57 -0500
Received: by vger.rutgers.edu id <160865-215>; Sat, 13 Mar 1999 20:56:53 -0500
Received: from pop.uniserve.com ([204.244.156.3]:3119 "HELO pop.uniserve.com" ident: "NO-IDENT-SERVICE[2]") by
vger.rutgers.eduwith SMTP id <160864-215>; Sat, 13 Mar 1999 20:56:06 -0500 
Received: from shell.uniserve.ca [204.244.186.218]
    by pop.uniserve.com with smtp (Exim 1.82 #4)
    id 10M066-0001xz-00; Sat, 13 Mar 1999 17:53:46 -0800
Date:     Sat, 13 Mar 1999 17:53:43 -0800 (PST)
From: Tom <tom@uniserve.com>
X-Sender: tom@shell.uniserve.ca
To: Matti Aarnio <matti.aarnio@sonera.fi>
cc: Dietmar Stein <dstein2203@t-online.de>, linux-raid@vger.rutgers.edu
Subject: Re: Maximum filesize of one file
In-Reply-To: <19990310234257Z92257-891+152@mea.tmt.tele.fi>
Message-ID: <Pine.BSF.4.02A.9903131749270.24376-100000@shell.uniserve.ca>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: owner-linux-raid@vger.rutgers.edu
Precedence: bulk
X-Loop: majordomo@vger.rutgers.edu
X-Orcpt: rfc822;linux-raid-outgoing
X-Mozilla-Status2: 00000000

On Thu, 11 Mar 1999, Matti Aarnio wrote:

> dstein2203@t-online.de (Dietmar Stein) asked:
> > Hi guys,
> >
> > not really the topic - I know - but I read that the maximum size of one
> > file is currently 2GB on linux like on other operating systems (e.g
> > HP-UX supports filesizes above 2GB but HP does not take support for it).
> >
> > Is there a way to increase the size for testing? I was looking for a
> > parameter in the kernel like ulimit but didn't find until now. Can
> > someone tell me how to change/increase?
>
>     With Linux 2.0 kernel series at 32-bit systems (e.g. i386)
>     the answer is: 2G is absolute limit, period.
>
>     The longer answer contains the size limit explanations
>     which are available at 64-bit systems (for 2.0 just Alpha
>     machines).  See files at  ftp://mea.tmt.tele.fi/linux/LFS/
>
>     At the above mentioned place there is also patch for the kernel
>     to support sizes over 2G at 32-bit platforms, however they
>     are not fully ready to be used quite yet -- some issues are
>     still open regarding glibc 2.1 support syncing.

  I don't know about this.  I believe the primary issue is support for >
2GB files is a design limiation in ext2fs.  I don't think it has anything
to do with the architecture.  Supposedly, ext3fs will fix this.

  For example, FreeBSD has support > 2GB because it uses UFS, and UFS
supports > 2GB files.  I understand that UFS is available for Linux too,
and when you use it, you get > 2GB files too.  I also understand that
other non-ext2fs filesystems for Linux > 2GB files too.


Tom


Return-Path: <owner-linux-raid-outgoing@vger.rutgers.edu>
Received: from nic.funet.fi (nic.funet.fi [128.214.248.6])
    by mail.sinyih.com.tw (8.9.1/8.9.1) with ESMTP id SAA19574
    for <cn@mail.sinyih.com.tw>; Sun, 14 Mar 1999 18:36:17 +0800
Received: from vger.rutgers.edu ([128.6.190.2]:18512 "EHLO vger.rutgers.edu" ident: "NO-IDENT-SERVICE[2]") by
nic.funet.fiwith ESMTP id <10167-6944>; Sun, 14 Mar 1999 04:33:47 +0200 
Received: by vger.rutgers.edu via listexpand id <160367-215>; Sat, 13 Mar 1999 21:34:55 -0500
Received: by vger.rutgers.edu id <157678-212>; Sat, 13 Mar 1999 21:34:48 -0500
Received: from mea.tmt.tele.fi ([194.252.70.162]:1124 "EHLO mea.tmt.tele.fi" ident: "IDENT-NONSENSE") by
vger.rutgers.eduwith ESMTP id <160249-212>; Sat, 13 Mar 1999 21:34:30 -0500 
Received: by mea.tmt.tele.fi id <92257-406>; Sun, 14 Mar 1999 04:30:58 +0200
Subject: Re: Maximum filesize of one file
In-Reply-To: <Pine.BSF.4.02A.9903131749270.24376-100000@shell.uniserve.ca> from Tom at "Mar 13, 99 05:53:43 pm"
To: tom@uniserve.com (Tom)
Date:     Sun, 14 Mar 1999 04:30:58 +0200 (EET)
Cc: matti.aarnio@sonera.fi, dstein2203@t-online.de,
        linux-raid@vger.rutgers.edu
From: Matti Aarnio <matti.aarnio@sonera.fi>
Phone: +358-20402082 (office, with redirection to cellular)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Message-Id: <19990314023104Z92257-406+2@mea.tmt.tele.fi>
Sender: owner-linux-raid@vger.rutgers.edu
Precedence: bulk
X-Loop: majordomo@vger.rutgers.edu
X-Orcpt: rfc822;linux-raid-outgoing
X-Mozilla-Status2: 00000000

Tom <tom@uniserve.com> pronouncedth:
> On Thu, 11 Mar 1999, Matti Aarnio wrote:
> >     With Linux 2.0 kernel series at 32-bit systems (e.g. i386)
> >     the answer is: 2G is absolute limit, period.
> >
> >     The longer answer contains the size limit explanations
> >     which are available at 64-bit systems (for 2.0 just Alpha
> >     machines).  See files at  ftp://mea.tmt.tele.fi/linux/LFS/
> >
> >     At the above mentioned place there is also patch for the kernel
> >     to support sizes over 2G at 32-bit platforms, however they
> >     are not fully ready to be used quite yet -- some issues are
> >     still open regarding glibc 2.1 support syncing.
>
>   I don't know about this.  I believe the primary issue is support for >
> 2GB files is a design limiation in ext2fs.  I don't think it has anything
> to do with the architecture.  Supposedly, ext3fs will fix this.

    I am happily using ext2 filesystem at my Alpha, and do use
    filesizes exceeding 2G quite regularly, thank you.  I have
    done so since 1.3 series kernels well before 2.* came out.
    (Not earlier simply because I didn't have an Alpha before that..)

>   For example, FreeBSD has support > 2GB because it uses UFS, and UFS
> supports > 2GB files.  I understand that UFS is available for Linux too,
> and when you use it, you get > 2GB files too.  I also understand that
> other non-ext2fs filesystems for Linux > 2GB files too.

    No you don't.  The VFS layer limitations prevent that at 32-bit
    machines (without my patches, that is.)

    In fact because the UFS and EXT2 share same principal block
    addressing scheme at the device, they both have SAME maximum
    file size limits with given filesystem block size.  (As do all
    SysV filesystems, Minix FS, etc. all which use the triply-in-
    directed block addressing scheme coming to us from the original
    AT&T creation so long ago...)

> Tom

/Matti Aarnio <matti.aarnio@sonera.fi>