Обсуждение: [NOVICE] - SAN/NAS/DAS - Need advises
Hi all,
I am working on upgrading my hardware and wondering how Postgres could work with SAN, NAS and DAS .
Can someone advise me or share experiences ?
Regards,
Fel
I am working on upgrading my hardware and wondering how Postgres could work with SAN, NAS and DAS .
Can someone advise me or share experiences ?
Regards,
Fel
On Tue, 7 Sep 2010 15:03:44 +0200, fel <fellsin@hotmail.com> wrote: > Hi all, > > I am working on upgrading my hardware and wondering how Postgres could > work with SAN, NAS and DAS . > Can someone advise me or share experiences ? Unless you want to spend *A LOT* of money, DAS is the way to go. You can get quite a bit of the same functionality without the financial overhead from the use of a volume manager + DAS. JD > > Regards, > Fel -- PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
While I agree with JD, we ended up using a fiber solution through a fiber switch with multi-path drivers (IBM DS4300). It did end up costing a few thousand dollars with all of the drives, but the performance made it worth it.
The big thing you want to remember to consider with any storage option is the overall I/O of your storage. A single 5400RPM drive with all of your pgdata on it (as well as the logs, let's say) is going to have serious performance implications that you're not going to have with a RAID10 array of 16 drives, for example.
Personally, having been in IT for _quite_ a few years, I'm still very leery of using network-based storage on database servers specifically. I know other people do it quite successfully out there in the world, but I personally neither want nor need my storage communication going over Ethernet (or such).
The big thing you want to remember to consider with any storage option is the overall I/O of your storage. A single 5400RPM drive with all of your pgdata on it (as well as the logs, let's say) is going to have serious performance implications that you're not going to have with a RAID10 array of 16 drives, for example.
Personally, having been in IT for _quite_ a few years, I'm still very leery of using network-based storage on database servers specifically. I know other people do it quite successfully out there in the world, but I personally neither want nor need my storage communication going over Ethernet (or such).
On Tue, 7 Sep 2010 15:03:44 +0200, fel <fellsin@hotmail.com> wrote:
> Hi all,
>
> I am working on upgrading my hardware and wondering how Postgres could
> work with SAN, NAS and DAS .
> Can someone advise me or share experiences ?
Unless you want to spend *A LOT* of money, DAS is the way to go. You can
get quite a bit of the same functionality without the financial overhead
from the use of a volume manager + DAS.
JD
>
> Regards,
> Fel
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
On Tue, Sep 7, 2010 at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote: > On Tue, 7 Sep 2010 15:03:44 +0200, fel <fellsin@hotmail.com> wrote: >> Hi all, >> >> I am working on upgrading my hardware and wondering how Postgres could >> work with SAN, NAS and DAS . >> Can someone advise me or share experiences ? > > Unless you want to spend *A LOT* of money, DAS is the way to go. You can > get quite a bit of the same functionality without the financial overhead > from the use of a volume manager + DAS. With the right supplier, you can plug in literally 100 hard drives to a regular server with DAS and for a fraction of the cost of a SAN. -- To understand recursion, one must first understand recursion.
On 2010-09-07 20:42, Scott Marlowe wrote: > With the right supplier, you can plug in literally 100 hard drives to > a regular server with DAS and for a fraction of the cost of a SAN. > Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives with a HP MDS600 with 15K SAS drives. The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The fibre channel version is about 20% more expensive pr TB. So of course it is a "fraction of the cost of a SAN", but it is a fairly small one. -- Jesper
On 9/7/10 12:06 PM, Jesper Krogh wrote: > On 2010-09-07 20:42, Scott Marlowe wrote: >> With the right supplier, you can plug in literally 100 hard drives to >> a regular server with DAS and for a fraction of the cost of a SAN. > Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives > with a HP MDS600 with 15K SAS drives. > > The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The > fibre channel version is about 20% more expensive pr TB. > > So of course it is a "fraction of the cost of a SAN", but it is a > fairly small one. Are you really comparing equal systems? "8gbit Fibre Channel" means a single Fibre Channel shared by 42 disks, whereas "3GBitDAS SAS" means 42 3gbit channels running in parallel. It seems like you'd really need some realistic benchmarks thatemulate your actual server load before you'd know how these two systems compare. Craig
On Tue, Sep 7, 2010 at 1:36 PM, Craig James <craig_james@emolecules.com> wrote: > On 9/7/10 12:06 PM, Jesper Krogh wrote: >> >> On 2010-09-07 20:42, Scott Marlowe wrote: >>> >>> With the right supplier, you can plug in literally 100 hard drives to >>> a regular server with DAS and for a fraction of the cost of a SAN. >> >> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS >> drives >> with a HP MDS600 with 15K SAS drives. >> >> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The >> fibre channel version is about 20% more expensive pr TB. >> >> So of course it is a "fraction of the cost of a SAN", but it is a >> fairly small one. > > Are you really comparing equal systems? "8gbit Fibre Channel" means a > single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42 > 3gbit channels running in parallel. It seems like you'd really need some > realistic benchmarks that emulate your actual server load before you'd know > how these two systems compare. Well, not usually. Most SAS DAS systems use a single multi-lane cable that gives you 4x3GB channels, etc. However, unless you're doing little than sequentially scanned reports of a large size being read, the difference between 8gb and 3gb is not going to matter. There are lots of very hard working transactional databases that are lucky to see more than 20 or 40 megabytes a second getting trasnferred spread out over 30 or 40 drives. What really matters here is if the 8gb SAN is as fast as or faster than the DAS setup. For most people measuring the speed of the interface is a lot like the famous Tom Lane quote about benchmarking jet fighters versus airliners by measuring the amount of runway they need. If you can get 10k tps on the SAN and 10k tps on the DAS So to the OP, what are hoping to get from the SAN that you won't get from the DAS? Also, how reliable are these two in comparison to each other is kinda of important. Speed of the interface isn't a real big deal for a database server Size of the battery backed cache in each one is And how each survives the power plug pull test. If your SAN salesman balks at a power on test you don't have to run it, you'll know. -- To understand recursion, one must first understand recursion.
Hi everybody, I have been reading this thread and I got the idea that SANs to avoid, but would somebody please give a bit of Comparison/perspective on NAS? Regards, Tena Sakai tsakai@gallo.ucsf.edu On 9/7/10 12:36 PM, "Craig James" <craig_james@emolecules.com> wrote: > On 9/7/10 12:06 PM, Jesper Krogh wrote: >> On 2010-09-07 20:42, Scott Marlowe wrote: >>> With the right supplier, you can plug in literally 100 hard drives to >>> a regular server with DAS and for a fraction of the cost of a SAN. >> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives >> with a HP MDS600 with 15K SAS drives. >> >> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The >> fibre channel version is about 20% more expensive pr TB. >> >> So of course it is a "fraction of the cost of a SAN", but it is a >> fairly small one. > > Are you really comparing equal systems? "8gbit Fibre Channel" means a single > Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42 3gbit > channels running in parallel. It seems like you'd really need some realistic > benchmarks that emulate your actual server load before you'd know how these > two systems compare. > > Craig >
On 2010-09-07 22:47, Scott Marlowe wrote: > Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS >>> drives >>> with a HP MDS600 with 15K SAS drives. >>> >>> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The >>> fibre channel version is about 20% more expensive pr TB. >>> >>> So of course it is a "fraction of the cost of a SAN", but it is a >>> fairly small one. >>> >> Are you really comparing equal systems? "8gbit Fibre Channel" means a >> single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42 >> 3gbit channels running in parallel. It seems like you'd really need some >> realistic benchmarks that emulate your actual server load before you'd know >> how these two systems compare. >> > Well, not usually. Most SAS DAS systems use a single multi-lane cable > that gives you 4x3GB channels, etc. > > However, unless you're doing little than sequentially scanned reports > of a large size being read, the difference between 8gb and 3gb is not > going to matter. There are lots of very hard working transactional > databases that are lucky to see more than 20 or 40 megabytes a second > getting trasnferred spread out over 30 or 40 drives. > > What really matters here is if the 8gb SAN is as fast as or faster > than the DAS setup. For most people measuring the speed of the > interface is a lot like the famous Tom Lane quote about benchmarking > jet fighters versus airliners by measuring the amount of runway they > need. > If you can get 10k tps on the SAN and 10k tps on the DAS > So to the OP, what are hoping to get from the SAN that you won't get > from the DAS? Also, how reliable are these two in comparison to each > other is kinda of important. Speed of the interface isn't a real big > deal for a database server Size of the battery backed cache in each > one is And how each survives the power plug pull test. If your SAN > salesman balks at a power on test you don't have to run it, you'll > know. > All wise words, that I can acknowledge with "hands on" experience. I was basically only reacting to the "you can do the choice based on cost alone.. DAS is soo-much-cheaper". In the comparison, I get an equal amount of disks with same characteristica, in the same raid-configuration. 1-2GB Battery backed cache on each. So on paper, I think the systems are directly comparable. In the real world it is more about "feelings" since I never get to benchmark both of them. Jesper -- Jesper
* Scott Marlowe <scott.marlowe@gmail.com> wrote: > What really matters here is if the 8gb SAN is as fast as or faster > than the DAS setup. For most people measuring the speed of the > interface is a lot like the famous Tom Lane quote about benchmarking > jet fighters versus airliners by measuring the amount of runway they > need. If you really want to know what's better, you'll have to test the candidates with real loads: record your real runtime load w/ blktrace and replay them on the candidates. Nominal performance parameters as insufficient as a dd or iobench test. For example, take an IBM XIV or DS8k w/ thin provisioning: simply writing zeros (or just the same pattern to all blocks) will give you almost the raw bus speed (maybe adding a little bit overhead inside the storage system), since spindles will idle. Caches (bein RAM or SSD) can make the storage look way faster than it really is (in rare cases, bad cache decisions could make also make it slower). Synthetic tests don't allow you to predict how spindle seeks, bus jams, etc will behave in real production. It *heavily* depends on your workload which storage type (and concrete product) will perform in production, nominal parameters are too vague for enterprise usage. (no, you can't even tell that DAS is faster than SAN). You need to test your workload and then decide which product to use. cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ----------------------------------------------------------------------
* Tena Sakai <tsakai@gallo.ucsf.edu> wrote: Hi, > I have been reading this thread and I got the idea that > SANs to avoid, but would somebody please give a bit of > Comparison/perspective on NAS? same as for SAN: measure it on your real workload and compare different products w/ similar stability and accessibility properties. cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ----------------------------------------------------------------------
Not sure if this will make it as I'm not subscribed. If it does, please CC me on any replies. > Are you really comparing equal systems? "8gbit Fibre Channel" means a > single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" > means 42 > > 3gbit channels running in parallel. Saw this on osdir while Googling and I thought I'd try to quash misinformation. The above statement is factually incorrect on many levels. Let me explain: The current, 2010, Nexsan SASBeast carries 42x15k SAS drives and has _FOUR_ 8G FC ports, 2 per controller, for a total of 8.5GB/s bidirectional aggregate raw FC bandwidth, not the single port 2.125GB/s inferred by the OP. 42 dual ported 3Gb/s SAS drives have a bidirectional aggregate maximum raw bandwidth of 25.2 GB/s, though current SAS drives only hit about 150MB/s while streaming large blocks, or half the SAS wire rate. The ratio of raw max disk to raw max SAN bandwidth with the Nexsan SASBeast is 2.96:1. By the same token, if I read the MDS600 PDF correctly, with a dual domain setup using dual ported SAS drives, a total of eight SFF8088 connectors on 4 I/o modules provide 32 SAS2 links for a maximum of 19.2GB/s bidirectional aggregate raw bandwidth. 70 dual ported 3Gb/s SAS drives have a bidirectional aggregate maximum raw bandwidth of 42GB/s. The ratio of raw max disk to raw max SAS2 wire bandwidth to the DAS host is 2.19:1. Note that both designs are using SAS Expanders to collate multiple drives into a single upstream SAS channel, in the case of Nexsan, this is to the SAN controllers, and in the case of HP it is to the host, or possibly an SAS switch. Noting that actual drive performance is limited to about 150MB/s, the 42 drives of the Nexsan can spin 6.3GB/s aggregate throughput, which is well below the 8.5GB/s FC bandwidth of the SASBeast. The 70 drives of the MDS600 can spin 10.5GB/s aggregate throughput, which is well below the max host bandwidth of 19.2GB/s. In both cases, total performance is limited by the RAID controllers, not the chassis backplane wiring or drive bandwidth. In the case of the SASBeast the relatively low RAID controller performance of this inexpensive FC SAN array limits maximum sustained throughput to approximately 1.2GB/s, depending on the RAID level used and the number of spindles per array, even in the presence of 4GB of cache (2GB per controller). Sustained 1.2GB/s is more than plenty of disk bandwidth for many, probably most, production applications. The typical configuration of the MDS600 is to connect all 70 drives through 8 SAS2 channels to an SAS2 switch, with a P700m RAID card with 256 or 512MB cache in each server attached to the MDS600. The downside to this JBOD SAS setup is that storage assignment to each server is at the drive level. So, if you have 70 disks and 7 servers, and you want an equal amount of storage and performance per server, the maximum array you can create per server uses only 10 disks. If you want to do RAID6 with a spare for each server, you've lost 3 disks per server to redundancy, or 21 your seventy disks--almost 1/3rd of your disk capacity is lost to redundancy. Also, your stripe size is limited to 9 spindles (remember, 1 spare per array), so each server will get a maximum of 7 spindles of post RAID6 parity stripe performance with a relatively low end RAID controller, for somewhere around 300-400MB/s. Now, with the SASBeast, as it's fiber channel, you can connect a huge number of hosts. The maximum number of LUNs you can export is 254, so you could potentially assign LUNs to 254 servers on your SAN. If you want maximum performance and redundancy, you simply assign 2 disks as spares, and dump the other 40 into a RAID6 array. You can now carve up this 40 drive array into say, 38 LUNs, assigning each to a different server. The LUNs are logical, and each is spread over the entire 40 drive array. Each server will get all 38 spindles worth of stripe performance because the RAID6 stripe is over 40 disks, not 9 as in the MDS600 case. There is a reason even inexpensive FC SAN arrays cost more than JBOD storage systems. Even though they may have less theoretical bandwidth, in deployment, they have equal or better performance for many workloads, and far far greater flexibility. With an SASBeast (or any Nexsan array) I can reassign, say, a 3TB LUN from a downed production database server with a dead motherboard to standby server with a few mouse clicks, power on the spare server, and have it boot directly from the SAN LUN as the previous server did. My production db server is back up in under 2 minutes--from the time I'm aware of the dead server that is. Try doing that with a DAS JBOD MDS600. :) Well, ok, if you're running a VMWare ESX cluster with multiple hosts connected to the MDS600 you could do it just as fast. Same is true for the SASBeast, but in that case, I could have 254 ESX hosts in my cluster. :) -- Stan