SAN - The (not so) Little Network That Could...

• A SAN is a creature combining 2 hot buzzwords (Storage, Networks) using a pointless word (Area).
• A SAN is a way to acknowledge that "it is no wonder the IT managers go nuts - the disk space keeps filling!"
• In short - a SAN puts the hosts in one place, the disks in another place, and lets everyone fight over a network.
• Linux is quite in infant with regards to its SAN support - here is one area where its older brothers and sisters fare better...
• ...But with recent kernels, this is changing.

What's In A SAN?

• A SAN is a network, much like the Internet is a network.
• Except that SANS were meant for LANs
• And SANs allow hosts to talk to.... disks (or more likely, disk arrays, RAIDs, JBODs (Just a Bunch Of Disks) and similar beasts.
• Mind you - a RAID can be a small thing costing 10K$, or a large beast costing several 100K$.
• Most SANs employ the Fiber-Channel protocol, over fiber-optic links (1Gbps, 2Gbps, lately also 4Gbps)...
• ...thought someone from IBM will tell you that iSCSI is the wave of the future...
• ...and the real weirdos will say that Infiniband is king!

Why Use SANs?

• In the past - each server had its own internal disks - hard to replace them or add more.
• Disks were moved outside (up to 7 disks per SCSI controller) - requires down-time for adding disks, many small disks for many servers cost a lot.
• Disks were moved to NAS servers (that expose CIFS or NFS shares) - not enough control by the applications (e.g. what if i want raw device access?, how does it scale to many servers and lots of TB of data? also, Ethernet performance is limiting)
• Disks were moved to RAIDs that expose block-devices directly, across a fiber-channel network.

A Little About SCSI

• SCSI defines a BUS-based protocol - you connect several devices to a BUS, give a unique ID to each device, and allow them to communicate.
• On the server (host) you install an HBA (Host-BUS Adapter) to communicate with the SCSI BUS.
• A SCSI address is composed of 4 numbers:
  1. Host number (the number of the SCSI HBA on the host)
  2. Port (or channel) - in case an HBA has more then one SCSI port
  3. Target ID (identifies a storage controller)
  4. LUN (Logical Unit Number) - identifies a single SCSI disk/tape/whatever in a given target.

The SCSI Architecture

• In The SCSI Architecture, there are initiators and targets.
• An initiator is like a client.
• A target is like a server.
• Initiators send SCSI commands to targets, and receive responses.

SCSI Commands

• SCSI commands pass data in one direction only. If you want to send some data and then receive some data, you need to use two separate SCSI commands.
• Some common SCSI commands: READ, WRITE, Send Diagnostics, Receive Diagnostics, Inquiry, Test Unit Ready.
• Each command returns a status code. A special code named 'mode sense' means there is an extra 'sense data' to be received, which contains an explanation for the cause of the problem.
• In order to allow a pipeline of commands, it is possible to attach a tag to a command. The tag is a single byte, so at most 255 commands may be pending.

An Example - The SCSI Inquiry Command

• The SCSI Inquiry Command allows us to get information about the device.
• When we send the SCSI command, we need to state which 'page' we want to access.
• All SCSI targets must respond to inquiries to page 0x80. often they have a vendor-specific response to page 0x83 too.
• An example, using the sg_inq command from the sg3-utils package:

  [choo@simey ~]# sg_inq /dev/sg0
  standard INQUIRY:
    PQual=0  Device_type=5  RMB=1  version=0x02  [SCSI-2]
    [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
    SCCS=0  ACC=0  TGPS=0  3PC=0  Protect=0  BQue=0
    EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
    [RelAdr=0]  WBus16=0  Sync=1  Linked=0  [TranDis=0]  CmdQue=0
      length=44 (0x2c)   Peripheral device type: cd/dvd
   Vendor identification: YAMAHA  
   Product identification: CRW-F1E         
   Product revision level: 1.0c
  

A Little About Fiber-Channel

• Fiber-Channel (FC) is like Ethernet
• Thought built with multi-pathing in mind (see below)
• and runs SCSI on top of it, for good measure (and backward compatibility with existing software and operating systems).

Fiber-Channel Terminology

• WWN (World-Wide Name) - the address of a controller (HBA on a host, controller on a RAID) connected to a Fiber-Channel network.
• LUN Masking - The operation of defining which WWNs may access a given LUN on a RAID attached to a Fiber-Channel network.
• Zones - Similar to VLANs in Ethernet - allows splitting the Fiber-Channel network to separate logical networks, where traffic cannot flow between different zones. Zones may span multiple fibre-channel switches.
• Name Server - a service on a Fiber-Channel switch, that collects a list of WWNs connected to it, and allows a host to see who is connected to the Fiber-Channel network (actually - the zones that contain the client sending the query).

More Fiber-Channel Terminology

• PLOGI (Port Login) - the first operation an initiator performs in order to talk to a target, is to login into the target.
• Multi-Pathing - the ability to connect several different paths between two entities, to use for load-balancing and/or for high-availability.

About RAIDs

• A RAID (Redundant Array of Inexpensive Disks) is a very expensive (10K$ for cheapo things and up to 500K$ and more for large high-end) container for disks.
• RAIDs support redundancy - i.e. the same data is stored on more then one physical disk, so a single disk failure does not stop I/O.
• RAIDs normally have multiple controllers, for high-availability and for load-balancing. A giant RAID might have 20 such controllers...
• RAIDs allow defining virtual disks, each exposed as a separate LUN, and support LUN masking.
• Finally, mid-range and high-end RAIDs support applications such as taking snapshots of LUNs, remote mirroring, etc.

SAN Virtualization

• As SANs grow, it is hard to manage them - knowing which host uses which LUNs. It is also hard to perform backups in today's environment, where you can't afford down-time for backup purposes (and you can't backup an active file-system).
• RAIDs were the first to virtualize things - virtual disks, snapshots, mirroring...
• Reliance on the applications in the high-end RAIDs result a vendor lock-in - which mean high prices.
• For starters, LVM (Logical Volume Management) in the hosts allow performing snapshots in the hosts rather then in the RAID - but then you cannot access these snapshots via another host.
• There is growing tendency to move all this virtualization to the network level - where you allow sharing snapshots and virtual volumes between hosts, have centralized management, and can mix RAIDs from different vendors (and thus reduce RAID prices).

Linux, SCSI and SANs

• Linux supported SCSI since very long ago - drivers for various SCSI controllers were in the kernel for years.
• Linux supports Fiber-Channel HBAs from various vendors.
• There are many non-SCSI devices that use the SCSI protocol on top of another protocol (e.g. USB) - Linux supports many of those too.

The Linux SCSI Sub-System

The Linux SCSI sub-system is split into 3 layers:

• SCSI low-layer - contains the HBA drivers for the different vendors.
• SCSI mid-layer - a single driver that allows HBA drivers and upper-layer drivers to find each other, and supports operations common to all SCSI devices.
• SCSI upper-layer - contains devices for different device types:
  • sd - a block driver for SCSI disks.
  • st - a character driver for SCSI tapes.
  • sr - a block driver for SCSI CDs/DVDs.
  • sg - a generic character driver - used to bypass the above drivers, or to talk to other types of devices, such as scanners.

SCSI Drivers Loading Order

• The mid-layer driver must be loaded before the others.
• When an HBA driver is loaded, it causes the mid-layer to perform a SCSI scan, in order to find any LUNs accessible to it, and attach upper-layer devices to them.
• This means that the order of HBA drivers loading is crucial to device naming: if we have HBA 1 that sees disk A, and HBA 2 that sees disk B, loading the the driver for HBA 1 first, will mean that disk A will be seen via /dev/sda, while disk B will be seen via /dev/sdb.
• In a SAN this is even more problematic - the order of discovery of different RAIDs will change the target ID (and device file attaching) for them. Anyone said "data corruption"?

Looking At SCSI "things" - /proc/scsi

• The /proc file-system contains a lot of info about SCSI devices.
• /proc/scsi/scsi contains a summary of the SCSI devices seen by the OS. A typical entry:

  Host: scsi0 Channel: 00 Id: 00 Lun: 00
    Vendor: YAMAHA   Model: CRW-F1E          Rev: 1.0c
    Type:   CD-ROM                           ANSI SCSI revision: 02
  

Looking At SCSI "things" - /proc/sg/

• The /proc/sg/ directory contains data seen by the SCSI generic driver 'sg'.
• The /proc/scsi/sg/devices contains a concise summary in a table. the column headers are found in /proc/scsi/sg/device_hdr:

  [choo@simey ~]# cat /proc/scsi/sg/device_hdr /proc/scsi/sg/devices 
  host    chan    id      lun     type    opens   qdepth  busy    online
  0       0       0       0       5       0       5       0       1
  

• If 'online' is 0 - the device was marked as offline due to some errors, and any SCSI commands sent to it are failed by the Linux kernel.
• 'busy' shows the number of pending commands. An idle device with a 'busy' field of more then 0 implies stuck SCSI commands.
• 'qdepth' is the number of commands that may be pending between the HBA driver and the target. NOTE: there is still another queue of commands that were not yet handled by the HBA, which can be much larger.

Looking At SCSI "things" - /sys/class/scsi/

• Kernel 2.6 added the sys file-system - which is an ordered way of exposing the entire peripherals hierarchy seen by the OS.
• /sys/class/scsi_device/ contains info about all SCSI devices:

  / # ls /sys/class/scsi_device/
  6:0:0:0/  6:0:0:5/  6:0:1:0/  6:0:1:5/  7:0:0:0/  7:0:0:5/  7:0:1:0/  7:0:1:5/
  6:0:0:1/  6:0:0:6/  6:0:1:1/  6:0:1:6/  7:0:0:1/  7:0:0:6/  7:0:1:1/  7:0:1:6/
  6:0:0:2/  6:0:0:7/  6:0:1:2/  6:0:1:7/  7:0:0:2/  7:0:0:7/  7:0:1:2/  7:0:1:7/
  6:0:0:3/  6:0:0:8/  6:0:1:3/  6:0:1:8/  7:0:0:3/  7:0:0:8/  7:0:1:3/  7:0:1:8/
  6:0:0:4/  6:0:0:9/  6:0:1:4/  6:0:1:9/  7:0:0:4/  7:0:0:9/  7:0:1:4/  7:0:1:9/
  

• Each directory contains a link named 'device' to a directory with data about one LUN.
• Some files in this directory:
  • vendor - the Vendor of the device.
  • model - the model of the device.
  • type - the type of the device (0 - disk).
• Note: the /sys file-system contains a large cross-reference, so each device can be found in several different paths - explore it at will...

Linux HBA Drivers

• First, we have the old SCSI HBA drivers for pure SCSI Controllers, such as aic79xx.o for various Adaptec HBAs, advansys.o for ConnectCom Solutions, Inc drivers, etc.
• The we have drivers for SCSI-over-something - SCSI over ide (ide-scsi.o), SCSI over USB (usb-storage.o for USB storage devices).
• Fiber-Channel HBAs - In the last 2-3 years, the 2 major vendors of Fiber-Channel HBAs - QLogic and Emulex - open-sourced their drivers, and they are now a part of the standard kernel (qlaXXXX.o for QLogic, lpfcXXXX.o for Emulex)...
• ...althought you still often need to take the latest drivers from the vendor's site, to overcome various problems.

Linux And SCSI Hotplug

• Until recently, Linux's SCSI disk layer discovered new devices when the HBA driver was loaded into memory.
• For normal usage that means only during system boot.
• In a SAN Environment, disks are added without shutting servers down. Thus, we want to discover new SCSI disks without a reboot.
• In kernel 2.6.12, the SCSI disks layer was given the ability to discover new disks in a plug&play manner.

Linux Hotplug And Fiber-Channel

• The Fiber-Channel protocol allows an HBA to be notified when a new LUN is seen on the SAN.
• The Fiber-Channel HBA reports LUN events to the SCSI mid-layer as soon as it sees them - hence, plug&play.
• It also notifies the SCSI mid-layer when LUNs are deleted.
• On drawback - if a RAID gets disconnected from the Fiber-Channel network, even for a few seconds, a similar event propagates and the device is immediately removed from the system.
• As a result - such events will cause I/O errors on the device.

Two Words About LVM

• Logical Volume Manager (LVM) is a tool to decouple file-systems (and raw devices) from the physical disks.
• with LVM, we create a volume group (vg), from one or more disks.
• On top of a vg, we can create logical volumes (lv).
• File-systems get created on top of the logical volumes.
• To add new capacity, we simply add a new device to the vg, and then either create new logical volumes, or expand existing ones.
• Since the volumes are now not tied to physical devices, we can add features such as snapshots, mirroring, etc.

Two Words About LVM And SANs

• LVM also solves the problem of changing device discovery order in a SAN - each device that belongs to a vg has a signature written onto it. Thus, when the system boots, it discovers the vg-s using these signatures, no matter the order of booting the devices.
• The names of the device files associated with the logical volumes remain fixed, making it easy to shuffle around RAIDs in the SAN.

References

1. IBMs SAN redbook http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg245470.html?Open
2. The Linux SCSI generic driver - http://sg.torque.net/sg/
3. LVM HowTo - http://www.tldp.org/HOWTO/LVM-HOWTO/index.html
4. Linux Hotplugging - http://linux-hotplug.sourceforge.net/

Originally written by guy keren