Raid...

[JC/DV]

OK, with Daniel going RAID and getting excellent results, I'm contemplating this in the near future. I'm rather ignorant on RAID.

My mobo will support RAID on the SATA ports, but I don't want my system on the RAID setup....

So, should I keep it internal or could I go external with something like this via 1394b? http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=2719711&CatId=207

[ppatton]

how many sata ports do you have in your computer? if you have 1 for your OS drive and multiple for other hard drives, these other ports should be able to be configured as a raid array. but if you have just 2 sata ports total, something on the line of your link will work.

paul

[JC/DV]

I have 4 sata ports. Sata0 contains my system drive. So you're saying I can configure the other 3 to run as raid and not the first one?

[ppatton]

yes, in computer management, disk management you should be able to set them up as a raid 0 array of all 3 of the remaining drives of equal size and create 1 array out of this

paul

[Billy]

Just remember that all the drives you stripe together should all be the same size, and preferably all the same model. Raid0 gives you speed but not redundancy, so don't forget to back up because losing one drive in an array will cause you to lose all your data on all your drives.

[JC/DV]

Just remember that all the drives you stripe together should all be the same size, and preferably all the same model. Raid0 gives you speed but not redundancy, so don't forget to back up because losing one drive in an array will cause you to lose all your data on all your drives.
That's what scares me about raid... loosing the data! If I were to do this, I would purchase same drives brand new and start from scratch.... also another question... say I get 1 TB of raid (2-500 GB drives) would it only be 500 GB worth of actual storage space? Like I said, I'm REALLY ignorant on RAID. (I know it can kill wasps and other insects though:grinning-smiley-021 )

[Billy]

That's what scares me about raid... loosing the data! If I were to do this, I would purchase same drives brand new and start from scratch.... also another question... say I get 1 TB of raid (2-500 GB drives) would it only be 500 GB worth of actual storage space? Like I said, I'm REALLY ignorant on RAID. (I know it can kill wasps and other insects though:grinning-smiley-021 )No, you'd have the full 1000 gigs, just half the data would be on one drive and half on the other. But when it writes a file it writes half on one and half on the other, so to retrieve the full file it has to have the data off both drives.

We've been using Raid0's for quite a while, and we've had some drives die, but we only use the Raids for video files, which in our case, usually don't remain on the drive for long anyway. We also use batch capture so if an array fails we can always recreate a project via batch capture and reloading the project file, which resides on different drives.

There's another type of Raid, Raid5, I think, that has speed and redundancy. But it takes more drives to achieve that. Another type has redundancy but not speed. Raid0 works for us the way we do things at this point in time so I haven't really kept up with the other types.

[ssvp]

Dude.. keep it very simple. Get an eSATA (True eSATA) RAID enclosure. Don't let WINXP setup your RAID as a software RAID. Go with the hardware raid on your mobo.

If your going RAID I would suggest any RAID certified drives since their MTBF's are higher and the drives internally have better shock/vibration protection.

I have several RAID1 configs, my fastest by far is my eSATA 4x150GB Western Digital 10,000RPM Raptors.. It's sweet as hell..

Of course SD/DV only needs 25Mb/s so a single SATA drive will do just fine..:)

[Billy]

Dude.. keep it very simple. Get an eSATA (True eSATA) RAID enclosure. Don't let WINXP setup your RAID as a software RAID. Go with the hardware raid on your mobo.

Of course SD/DV only needs 25Mb/s so a single SATA drive will do just fine..:)
I don't know why, but all the VT dealers and VT techno geeks say go with a software striped raid, not the hardware.

As far as DV being 25Mb/s, that's for a single stream. If you want to stack more than 1 clip on a timeline then you'll need more than that. I look for 100Mb/s when I put my Raids together. It's too easy to put together a fast Raid to allow the hard drives to be the bottleneck in getting real time performance from my systems.

[RatVega]

I have several RAIDs, the oldest has been up for over three years in its current configuration. With good hardware a RAID can be pretty reliable. For real reliability, you need to go RAID5 or better, which can get spendy.

Use real RAID software. When my last RAID started breaking up, the software I had used successfully for a year couldn't characterize the problem. I now have software that can assist in diagnosing where the problem comes from.

As far as speed goes, be aware that anything above 60% of the theoretical spec is a bonus. If you go FW 800, expect to see real-world data rates of around 480Mb/s. The same holds true with virtually all the channel speeds; what you can sustain is a good deal less thaqn what you can peak.

Our latest RAID is an external 4x500 (2TB) affair that runs on its own SATA board. It can run in excess of 200MB (megabytes) per second for reasonable periods. It is RAID 0 managed by SoftRAID 3.5.1 It ran about $1100 when we built it several months ago.

Anything less than a RAID 5 is like dirtbike riding; if you're good (and careful) you should be OK, but crashes are a reality you'll want to plan for.

[JC/DV]

Anything less than a RAID 5 is like dirtbike riding; if you're good (and careful) you should be OK, but crashes are a reality you'll want to plan for.
Thanks for all the inputs.... this alone is what I'm kind of looking for. I'm not good at dirt bikes and don't like them, so I guess RAID5 is what I need to look into. I'm going to continue my researching and will keep absorbing all your guys' input...

[ppatton]

I had a server set up with raid 5 running server 2003, 4TB, that couldn't handle the data rates that I needed it to. of course it was recording 100 security cameras 24/7 as soon as I changed to raid 0 my problems ceased. until the drives started dying, then of course the data was lost. But I doubt your throughput will be anything near what I was trying to do. But the drives dying didn't start for 3 years on raid 0 setup. So rather than loose 4TB of video data I set the arrays up as smaller 1TB arrays that way I would only loose part of what is recorded.

paul

[RatVega]

I was a bit rushed the other day, so I'll add a few thoughts...

Paul has a couple of good points, specifically that RAID 5 will be slower by virtue of the overhead used to manage it, and that many RAIDs are designed for bandwidth requirements way past what a mid-tier video guy really needs.

There is RAID 5 hardware out there that can support serious channel rates, but it gets very expensive very quickly so most of us are compelled to compromise. With the advent of SATA (and SATA II & III) raw speed comes pretty cheap so a software RAID 5 may be practical. I don't have numbers on what the degradation is but if guys are getting over 400MB/sec with 8-drive RAID 0 arrays, sustaining "normal" video data rates in RAID 5 should be possible.

Drive reliability has pretty much always been the killer, but the latest drives are getting pretty good. Patrick mentioned buying RAID-qualified drives, this is a good suggestion unless they carry such a premium that they're impractical. Starting with fresh, matched, full-warranty drives is a must. Many of the "bargain" drives are offered with seriously reduced warranty and I take this to say there is a higher statistical probability failure, something that is a calculated risk in the general sense but suicidal in an array.

BTW, there is life beyond SATA, but it's pretty much substandard or obscenely expensive.

A FW800 RAID array sounds attractive but in fact will fall short early in the bandwidth arena. No matter the configuration, it will be limited to 800Mbps (theoretical) or around 500Mbps real world. That's barely enough to keep a single stream of uncompressed 720p playing...

On the other end, there are UltraSCSI arrays out there that are lightning fast and close to bullet-proof but they are priced like new cars. And no, they don't offer first-time buyer rebates... :grinning-smiley-021

[ppatton]

Drive reliability has pretty much always been the killer, but the latest drives are getting pretty good. Patrick mentioned buying RAID-qualified drives, this is a good suggestion unless they carry such a premium that they're impractical. Starting with fresh, matched, full-warranty drives is a must. Many of the "bargain" drives are offered with seriously reduced warranty and I take this to say there is a higher statistical probability failure, something that is a calculated risk in the general sense but suicidal in an array.



I'll agree 100% with this. This is a must in highend configuation. The same server I was talking about in my earlier post used high quality drives, (the first set was replaced due to a 90% failure of all drives, inside of 3 months) (all built on Monday) a bad batch. But were all replaced under warranty. The second set lasted 3+ years before they started dying. But replacements had been budgeted for at the 3 year mark (time for the warranty to run out) But by dividing up the size of the arrays all data wasn't lost, even though a lot was. But in another sese I did have additional recordings of all the lost data on other computers as well, rudundancy can also be your friend.

paul

[Mike Downey]

I don't know why, but all the VT dealers and VT techno geeks say go with a software striped raid, not the hardware.

As far as DV being 25Mb/s, that's for a single stream. If you want to stack more than 1 clip on a timeline then you'll need more than that. I look for 100Mb/s when I put my Raids together. It's too easy to put together a fast Raid to allow the hard drives to be the bottleneck in getting real time performance from my systems.


I would not do that. It takes away from system resources when you do it this way.

It is easier to change motherboards when you do it this way.


I have talked with a few builders about this and the consensus seems to be something like this.


1. Motherboard Raid

2. Internal Raid Card

3. Software Raid

4 External Raid.


In that order. :)



Mike

[Billy]

I guess it all depends. The guy who's built all my machines, other guys who build VT machines, and even NewTek themselves recommend software striping of drives.

From what little googling I've done of software vs hardware striping of drives, what I've gleaned is that software striping can, and is in a lot of cases, faster that hardware striping if you have a fast processor (which most machines nowadays qualify as). Software striping does use some CPU cycles but not enough to worry about unless you need every cycle you can get. So the info I have is that if speed is your major concern (which would be the case with Raid0), software striping is the way to go.

Another advantage is that it's a lot cheaper to go with the software stripe.

I've had software striped raids for several years now and haven't had any problems with my VT playing back several layers of video in real time, or any problems with not enough CPU cycles.

[ppatton]

I've found with hardware raid, figuring out what drive is bad can be a chore, plus have to shudown computer to view bios to find what port/drive is bad. with software I can leave computer management open and see the status of each individual drive.

paul

[RatVega]

Frankly, RAID technology has evolved right along with the rest of computer technology, so old rules of thumb get dated as well. There was once a good reason to have a screen saver (would you believe it was to save the screen from early wear-out?)

A "mother board RAID" (never seen one) would require a pretty special mother board (even more so in the past) to set up a large array.

A hardware RAID controller with dedicated hi-rel features would be nice but expensive. As Billy says, it depends a lot on the purpose of the RAID.

Software RAID has come a long ways in terms of flexibility and monitoring, and is quite affordable. Since this thread started, I've received a free update for my RAID software that expands its capabilities and will install seamlessly.

External RAID would only be inferior based on the bus architecture and the "parts count" side of statistical failure probability. SATA doesn't suffer from the bus arbitration issues associated with many earlier storage bus technologies. Truth be known, I expect that the vast majority of older RAIDs were external simply because of the drive count required for a large array.

If money were no object we'd all have advanced hardware controllers and huge arrays, but for most of us money is an object. For my money, I'll build my own software-controlled external SATA RAID and buy a new Honda with the difference in price... :)

[Mike Downey]

I guess it all depends. The guy who's built all my machines, other guys who build VT machines, and even NewTek themselves recommend software striping of drives.

From what little googling I've done of software vs hardware striping of drives, what I've gleaned is that software striping can, and is in a lot of cases, faster that hardware striping if you have a fast processor (which most machines nowadays qualify as). Software striping does use some CPU cycles but not enough to worry about unless you need every cycle you can get. So the info I have is that if speed is your major concern (which would be the case with Raid0), software striping is the way to go.

Another advantage is that it's a lot cheaper to go with the software stripe.

I've had software striped raids for several years now and haven't had any problems with my VT playing back several layers of video in real time, or any problems with not enough CPU cycles.


I would love to test this because it wouldn't take much to show me I am wrong! :)


I had Software Raids on occasion for years and they do work well. I am not sure that the speed difference is enough to really see but it would be fun to find out :)

Another thing I needed to qualify is that the externals that I was talking about was USB and Firewire... not SATA. :)


Mike

[Mike Downey]

If money were no object we'd all have advanced hardware controllers and huge arrays, but for most of us money is an object. For my money, I'll build my own software-controlled external SATA RAID and buy a new Honda with the difference in price... :)


I am with ya on this one! :)


Mike

[Billy]

I would love to test this because it wouldn't take much to show me I am wrong! :)
I don't have any personal comparisons. All the machines I bought came with the video drives software striped, and I just carried on the tradition. However, casual googling says software is a little faster than hardware with fast processors.

I had Software Raids on occasion for years and they do work well. I am not sure that the speed difference is enough to really see but it would be fun to find out :)All the more reason not to spend the extra bucks for the hardware. :)

Another thing I needed to qualify is that the externals that I was talking about was USB and Firewire... not SATA. :)I didn't think you could make an external drive(s) dynamic. :confused-smiley-013

[RatVega]

I had Software Raids on occasion for years and they do work well. I am not sure that the speed difference is enough to really see but it would be fun to find out :)

Another thing I needed to qualify is that the externals that I was talking about was USB and Firewire... not SATA. :)


Mike
For all the technology and hype, RAIDs boil down to just a couple of concepts and a couple of implementation styles.

Striping (RAID 0) allows the controller (whether hardware or software) to read/write to more than one drive at the same time, which increases the speed until you run out of channel bandwidth.

Mirroring (RAID 1) literally makes two copies of everything then checks to make sure both are the same. This is an "instant back-up" but uses more space and requires more work so it's slower but far more reliable.

Parity (RAID 5) can be added to increase the data reliability in either case but requires more space and overhead work so fast, secure RAIDS of the past were frequently built using a hardware controller to do all the extra work.

The "classic" fast, reliable RAID was a hardware RAID 50: a striped, parity arrangement with its own processor to parcel out the data and compute the parity checksums and figure out the errors and rebuild the data on the fly when an error occured. This was (and still is) expensive but pretty darned reliable. What's changed is that we now have a huge increase in processing power with the newer processors, and the software involved keeps getting better.

Why SATA excels (and I've been told is isn't "the best", just "the best affordable") is that the architecture puts one fast drive on each channel so an eight channel controller can do it's thing without any concern for channel arbitration or saturation. I think the basic SATA chip allows for two drives and understands the constructs of a RAID, so it's prepared to play the game. And of course, the SATA drive interface is capable of dumping data at a faster rate (and getting faster) than most previous (affordable) drives.

This is just a rough sketch, there are a lot of other options/considerations, but you get the idea... :)

[ssvp]

Why SATA excels (and I've been told is isn't "the best", just "the best affordable") is that the architecture puts one fast drive on each channel so an eight channel controller can do it's thing without any concern for channel arbitration or saturation. I think the basic SATA chip allows for two drives and understands the constructs of a RAID, so it's prepared to play the game. And of course, the SATA drive interface is capable of dumping data at a faster rate (and getting faster) than most previous (affordable) drives.

This is just a rough sketch, there are a lot of other options/considerations, but you get the idea... :)

Yeah they are affordable.. I just built out a Hitachi SAN that has 36TB's it's all SATA drives.. Hitachi is more and more going SATA for SAN's but sticking to SCSI for NAS..

[RatVega]

I have an acquaintance who is in the heavy commercial end of mass storage and is the source of the comment. He was talking about long-term reliability in both data and hardware. The arguement starts to sound like one for buying Mercedes as fleet cars: with proper maintenance, the second 250,000 miles cost you less... This kind of thinking makes sense for the guys at Yahoo but is pretty much lost on us little guys.

Thinking back, I can't remember ever having a SCSI drive fail but I've lost several ATA drives.