path of a disk IO

[linux2006]

Hi all,

I know this is a pretty big question but just trying to learn as much as I can about this topic.

I wanted to understand how IO travels in a SAN. Lets say for example, I have a linux host connected to a SAN.

If I copy a 2MB file, can someone walk me through what happens. I realize a LOT happens but looking for any info you can provide (which will probably lead to me having more questions to look up).

I'm looking for the path from host, to cache, to HBA card, over the fiber channel, to the SAN HBA, disk cache, and onto disk. (and if I missed anything).

I'm not looking for SAN vendor specifics, just more of a general flow.

For example, is that 2MB file broken up into 4K IO's (524,288 of them??) and are they all sent seperately over the fiber channel...etc..etc..

Any help or pointers you could provide would be greatly appreciated.

[Víctor Cespón]

Yes, that's pretty much how it goes. Fiber channel is the basic protocol, but it transports SCSI frames, so the operating system talks to the disk through SCSI commads, like a local disk.

The fiber channel protocol transports payloads of up to 2112 bytes at 1, 2, 4 or 8 Gb/s. To maintain the correct timing, every 8 bits are encoded into 10 bits, so that there's no more than 5 0's or 1's in a row.

This encoding is done by the server HBA and then reversed by the storage controller. Once the data is in the storage controller it stores it on a battery-backed RAM and sends the ACK back to the server. The storage controller accumulates writes and sorts them to minimize the amount of movements on the disk heads.

For reads, the storage controller tries to maintain as much as possible on the cache, when a hosts requests something it's sent from the cache and not from the disks.

Also, to have redundancy and increase the throughput, usually there are two controllers and several paths to the storage array. The OS multipath driver sends the I/Os over all paths and if one fails, moves the traffic to the others.

Some basic pages:
http://en.wikipedia.org/wiki/Fibre_Channel
http://www.recoverdata.com/fc_tutorial.htm
http://en.wikipedia.org/wiki/8B/10B_encoding