Обсуждение: [NOVICE] - SAN/NAS/DAS - Need advises

On Tue, 7 Sep 2010 15:03:44 +0200, fel <fellsin@hotmail.com> wrote:
> Hi all,
>
> I am working on upgrading my hardware and wondering how Postgres could
> work with SAN, NAS and DAS .
> Can someone advise me or share experiences ?

Unless you want to spend *A LOT* of money, DAS is the way to go. You can
get quite a bit of the same functionality without the financial overhead
from the use of a volume manager + DAS.

JD

>
> Regards,
> Fel

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997

On Tue, Sep 7, 2010 at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote:
> On Tue, 7 Sep 2010 15:03:44 +0200, fel <fellsin@hotmail.com> wrote:
>> Hi all,
>>
>> I am working on upgrading my hardware and wondering how Postgres could
>> work with SAN, NAS and DAS .
>> Can someone advise me or share experiences ?
>
> Unless you want to spend *A LOT* of money, DAS is the way to go. You can
> get quite a bit of the same functionality without the financial overhead
> from the use of a volume manager + DAS.

With the right supplier, you can plug in literally 100 hard drives to
a regular server with DAS and for a fraction of the cost of a SAN.


--
To understand recursion, one must first understand recursion.

On 2010-09-07 20:42, Scott Marlowe wrote:
> With the right supplier, you can plug in literally 100 hard drives to
> a regular server with DAS and for a fraction of the cost of a SAN.
>
Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives
with a HP MDS600 with 15K SAS drives.

The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The
fibre channel version is about 20% more expensive pr TB.

So of course it is a "fraction of the cost of a SAN", but it is a
fairly small one.

--
Jesper

On 9/7/10 12:06 PM, Jesper Krogh wrote:
> On 2010-09-07 20:42, Scott Marlowe wrote:
>> With the right supplier, you can plug in literally 100 hard drives to
>> a regular server with DAS and for a fraction of the cost of a SAN.
> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives
> with a HP MDS600 with 15K SAS drives.
>
> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The
> fibre channel version is about 20% more expensive pr TB.
>
> So of course it is a "fraction of the cost of a SAN", but it is a
> fairly small one.

Are you really comparing equal systems?  "8gbit Fibre Channel" means a single Fibre Channel shared by 42 disks, whereas
"3GBitDAS SAS" means 42 3gbit channels running in parallel.  It seems like you'd really need some realistic benchmarks
thatemulate your actual server load before you'd know how these two systems compare. 

Craig

On Tue, Sep 7, 2010 at 1:36 PM, Craig James <craig_james@emolecules.com> wrote:
> On 9/7/10 12:06 PM, Jesper Krogh wrote:
>>
>> On 2010-09-07 20:42, Scott Marlowe wrote:
>>>
>>> With the right supplier, you can plug in literally 100 hard drives to
>>> a regular server with DAS and for a fraction of the cost of a SAN.
>>
>> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS
>> drives
>> with a HP MDS600 with 15K SAS drives.
>>
>> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The
>> fibre channel version is about 20% more expensive pr TB.
>>
>> So of course it is a "fraction of the cost of a SAN", but it is a
>> fairly small one.
>
> Are you really comparing equal systems?  "8gbit Fibre Channel" means a
> single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42
> 3gbit channels running in parallel.  It seems like you'd really need some
> realistic benchmarks that emulate your actual server load before you'd know
> how these two systems compare.

Well, not usually.  Most SAS DAS systems use a single multi-lane cable
that gives you 4x3GB channels, etc.

However, unless you're doing little than sequentially scanned reports
of a large size being read, the difference between 8gb and 3gb is not
going to matter.    There are lots of very hard working transactional
databases that are lucky to see more than 20 or 40 megabytes a second
getting trasnferred spread out over 30 or 40 drives.

What really matters here is if the 8gb SAN is as fast as or faster
than the DAS setup.  For most people measuring the speed of the
interface is a lot like the famous Tom Lane quote about benchmarking
jet fighters versus airliners by measuring the amount of runway they
need.

If you can get 10k tps on the SAN and 10k tps on the DAS

So to the OP, what are hoping to get from the SAN that you won't get
from the DAS?  Also, how reliable are these two in comparison to each
other is kinda of important.  Speed of the interface isn't a real big
deal for a database server  Size of the battery backed cache in each
one is  And how each survives the power plug pull test.  If your SAN
salesman balks at a power on test you don't have to run it, you'll
know.

--
To understand recursion, one must first understand recursion.

Hi everybody,

I have been reading this thread and I got the idea that
SANs to avoid, but would somebody please give a bit of
Comparison/perspective on NAS?

Regards,

Tena Sakai
tsakai@gallo.ucsf.edu


On 9/7/10 12:36 PM, "Craig James" <craig_james@emolecules.com> wrote:

> On 9/7/10 12:06 PM, Jesper Krogh wrote:
>> On 2010-09-07 20:42, Scott Marlowe wrote:
>>> With the right supplier, you can plug in literally 100 hard drives to
>>> a regular server with DAS and for a fraction of the cost of a SAN.
>> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS drives
>> with a HP MDS600 with 15K SAS drives.
>>
>> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The
>> fibre channel version is about 20% more expensive pr TB.
>>
>> So of course it is a "fraction of the cost of a SAN", but it is a
>> fairly small one.
>
> Are you really comparing equal systems?  "8gbit Fibre Channel" means a single
> Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42 3gbit
> channels running in parallel.  It seems like you'd really need some realistic
> benchmarks that emulate your actual server load before you'd know how these
> two systems compare.
>
> Craig
>

On 2010-09-07 22:47, Scott Marlowe wrote:
> Ok, recently I have compared prices a NexSan SASBeast with 42 15K SAS
>>> drives
>>> with a HP MDS600 with 15K SAS drives.
>>>
>>> The first is 8gbit Fibre Channel, the last is 3Gbit DAS SAS. The
>>> fibre channel version is about 20% more expensive pr TB.
>>>
>>> So of course it is a "fraction of the cost of a SAN", but it is a
>>> fairly small one.
>>>
>> Are you really comparing equal systems?  "8gbit Fibre Channel" means a
>> single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS" means 42
>> 3gbit channels running in parallel.  It seems like you'd really need some
>> realistic benchmarks that emulate your actual server load before you'd know
>> how these two systems compare.
>>
> Well, not usually.  Most SAS DAS systems use a single multi-lane cable
> that gives you 4x3GB channels, etc.
>
> However, unless you're doing little than sequentially scanned reports
> of a large size being read, the difference between 8gb and 3gb is not
> going to matter.    There are lots of very hard working transactional
> databases that are lucky to see more than 20 or 40 megabytes a second
> getting trasnferred spread out over 30 or 40 drives.
>
> What really matters here is if the 8gb SAN is as fast as or faster
> than the DAS setup.  For most people measuring the speed of the
> interface is a lot like the famous Tom Lane quote about benchmarking
> jet fighters versus airliners by measuring the amount of runway they
> need.
>
If you can get 10k tps on the SAN and 10k tps on the DAS
> So to the OP, what are hoping to get from the SAN that you won't get
> from the DAS?  Also, how reliable are these two in comparison to each
> other is kinda of important.  Speed of the interface isn't a real big
> deal for a database server  Size of the battery backed cache in each
> one is  And how each survives the power plug pull test.  If your SAN
> salesman balks at a power on test you don't have to run it, you'll
> know.
>
All wise words, that I can acknowledge with "hands on" experience.

I was basically only reacting to the "you can do the choice based
on cost alone.. DAS is soo-much-cheaper". In the comparison, I get
an equal amount of disks with same characteristica, in the same
raid-configuration. 1-2GB Battery backed cache on each. So on
paper, I think the systems are directly comparable. In the real
world it is more about "feelings" since I never get to benchmark both of
them.


Jesper
--
Jesper

* Scott Marlowe <scott.marlowe@gmail.com> wrote:

> What really matters here is if the 8gb SAN is as fast as or faster
> than the DAS setup.  For most people measuring the speed of the
> interface is a lot like the famous Tom Lane quote about benchmarking
> jet fighters versus airliners by measuring the amount of runway they
> need.

If you really want to know what's better, you'll have to test the
candidates with real loads: record your real runtime load w/ blktrace
and replay them on the candidates.

Nominal performance parameters as insufficient as a dd or iobench test.

For example, take an IBM XIV or DS8k w/ thin provisioning: simply
writing zeros (or just the same pattern to all blocks) will give you
almost the raw bus speed (maybe adding a little bit overhead inside
the storage system), since spindles will idle. Caches (bein RAM or
SSD) can make the storage look way faster than it really is (in rare
cases, bad cache decisions could make also make it slower).

Synthetic tests don't allow you to predict how spindle seeks, bus
jams, etc will behave in real production. It *heavily* depends on
your workload which storage type (and concrete product) will perform
in production, nominal parameters are too vague for enterprise usage.
(no, you can't even tell that DAS is faster than SAN).

You need to test your workload and then decide which product to use.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

* Tena Sakai <tsakai@gallo.ucsf.edu> wrote:

Hi,

> I have been reading this thread and I got the idea that
> SANs to avoid, but would somebody please give a bit of
> Comparison/perspective on NAS?

same as for SAN: measure it on your real workload and compare
different products w/ similar stability and accessibility properties.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Not sure if this will make it as I'm not subscribed.  If it does, please
CC me on any replies.

> Are you really comparing equal systems? "8gbit Fibre Channel" means a
> single Fibre Channel shared by 42 disks, whereas "3GBit DAS SAS"
> means 42 > > 3gbit channels running in parallel.

Saw this on osdir while Googling and I thought I'd try to quash
misinformation.  The above statement is factually incorrect on many
levels.  Let me explain:

The current, 2010, Nexsan SASBeast carries 42x15k SAS drives and has
_FOUR_ 8G FC ports, 2 per controller, for a total of 8.5GB/s
bidirectional aggregate raw FC bandwidth, not the single port 2.125GB/s
inferred by the OP.  42 dual ported 3Gb/s SAS drives have a
bidirectional aggregate maximum raw bandwidth of 25.2 GB/s, though
current SAS drives only hit about 150MB/s while streaming large blocks,
or half the SAS wire rate.  The ratio of raw max disk to raw max SAN
bandwidth with the Nexsan SASBeast is 2.96:1.

By the same token, if I read the MDS600 PDF correctly, with a dual
domain setup using dual ported SAS drives, a total of eight SFF8088
connectors on 4 I/o modules provide 32 SAS2 links for a maximum of
19.2GB/s bidirectional aggregate raw bandwidth.   70 dual ported 3Gb/s
SAS drives have a bidirectional aggregate maximum raw bandwidth of
42GB/s.  The ratio of raw max disk to raw max SAS2 wire bandwidth to the
DAS host is 2.19:1.

Note that both designs are using SAS Expanders to collate multiple
drives into a single upstream SAS channel, in the case of Nexsan, this
is to the SAN controllers, and in the case of HP it is to the host, or
possibly an SAS switch.

Noting that actual drive performance is limited to about 150MB/s, the 42
drives of the Nexsan can spin 6.3GB/s aggregate throughput, which is
well below the 8.5GB/s FC bandwidth of the SASBeast.  The 70 drives of
the MDS600 can spin 10.5GB/s aggregate throughput, which is well below
the max host bandwidth of 19.2GB/s.

In both cases, total performance is limited by the RAID controllers, not
the chassis backplane wiring or drive bandwidth.  In the case of the
SASBeast the relatively low RAID controller performance of this
inexpensive FC SAN array limits maximum sustained throughput to
approximately 1.2GB/s, depending on the RAID level used and the number
of spindles per array, even in the presence of 4GB of cache (2GB per
controller).  Sustained 1.2GB/s is more than plenty of disk bandwidth
for many, probably most, production applications.

The typical configuration of the MDS600 is to connect all 70 drives
through 8 SAS2 channels to an SAS2 switch, with a P700m RAID card with
256 or 512MB cache in each server attached to the MDS600.  The downside
to this JBOD SAS setup is that storage assignment to each server is at
the drive level.

So, if you have 70 disks and 7 servers, and you want an equal amount of
storage and performance per server, the maximum array you can create per
server uses only 10 disks.  If you want to do RAID6 with a spare for
each server, you've lost 3 disks per server to redundancy, or 21 your
seventy disks--almost 1/3rd of your disk capacity is lost to redundancy.
 Also, your stripe size is limited to 9 spindles (remember, 1 spare per
array), so each server will get a maximum of 7 spindles of post RAID6
parity stripe performance with a relatively low end RAID controller, for
somewhere around 300-400MB/s.

Now, with the SASBeast, as it's fiber channel, you can connect a huge
number of hosts.  The maximum number of LUNs you can export is 254, so
you could potentially assign LUNs to 254 servers on your SAN.  If you
want maximum performance and redundancy, you simply assign 2 disks as
spares, and dump the other 40 into a RAID6 array.  You can now carve up
this 40 drive array into say, 38 LUNs, assigning each to a different
server.  The LUNs are logical, and each is spread over the entire 40
drive array.  Each server will get all 38 spindles worth of stripe
performance because the RAID6 stripe is over 40 disks, not 9 as in the
MDS600 case.

There is a reason even inexpensive FC SAN arrays cost more than JBOD
storage systems.  Even though they may have less theoretical bandwidth,
in deployment, they have equal or better performance for many workloads,
and far far greater flexibility.  With an SASBeast (or any Nexsan array)
I can reassign, say, a 3TB LUN from a downed production database server
with a dead motherboard to standby server with a few mouse clicks, power
on the spare server, and have it boot directly from the SAN LUN as the
previous server did.  My production db server is back up in under 2
minutes--from the time I'm aware of the dead server that is.  Try doing
that with a DAS JBOD MDS600. :)  Well, ok, if you're running a VMWare
ESX cluster with multiple hosts connected to the MDS600 you could do it
just as fast.  Same is true for the SASBeast, but in that case, I could
have 254 ESX hosts in my cluster. :)

--
Stan