Sunfire Storage Array Configuration

Overview

The Sunfire 68K which comprises titan and myosin is connected to a Sun enterprise StorEdge T3+ storage array cabinet via fibre channel HBAs. This cabinet consists of 4 enclosures containing two arrays per enclosure connected in a partner configuration, where enclosure arrays are joined in a partner group. This configuration provides dynamic multi-pathing access to the disk arrays for high-availability and redudancy. Each enclosure array can hold 9 scsi disks in various RAID configurations for a maximum of two volumes per array. Two of the enclosures are connected to titan, whereas the other two enclosures are connected to myosin - providing a total of four disk arrays per partition. Each enclosure is accessible via IP from either myosin or titan, which have virtual interfaces configured on the 192.168.50.X network.

For the array volumes which are configured and online (the T3+ software shows this as mounted with the proc list command), the volume is visible within the operating system as a disk device. Use the solaris format command to enumerate the devices that are made available from the array. Because the operating system can access the array volumes via multiple paths, each volume will be listed twice in the output of format.

The logical disks made available to the OS by the disk arrays are not used directly by titan and myosin but are instead, managed by Veritas Volume Manager. Veritas enables creation of logical volumes with a high degree of configurability, where logical disks can be moved, mirrored, partitioned, reserved for backup, or striped across disk groups and volumes to meet application-specific demands. Our current configuration is relatively straightforward, where array volumes map to logical disks which are in turn members of disk groups and volumes. These higher-level logical volumes are made available to the operating system through the Veritas software to which the specific OS mount points are mapped.

Array Configuration

Enclosure	IP	Volume	Listed Capacity	RAID	Data Disks	Standby	Status	Comments
myosin-t31	192.168.50.70	v0	512.3 GB	5	u1d1-8	u1d9	Online
		v1-raid0	146.3 GB	0	u2d1-2	none	Online
		v1-raid5	365.9 GB	5	u2d3-8	u2d9	Online
myosin-t32	192.168.50.71	v0	512.3 GB	5	u1d1-8	u1d9	Online
		v1-raid0	146.3 GB	0	u2d1-2	none	Online
		v1-raid5	365.9 GB	5	u2d3-8	u2d9	Online
titan-t31	192.168.50.72	v0	477.192 GB	5	u1d1-8	u1d9	Online
		v1	477.192	5	u2d1-8	u2d9	Online
titan-t32	192.168.50.73	v0	477.192	5	u1d1-8	u1d9	Online
		v1	477.192	5	u2d1-8	u2d9	Offline	Defunct array which previously contained Mascot

Volume Configuration

myosin

• Three volumes: backup, home, and scratch. Home and backup are configured on the array as RAID 5. Scratch is RAID 1..
  • backup - from myosin-t31 v0, myosin-t32 v0 - mounted as /backup
  • scratch - from myosin-t31 v1_raid0, myosin-t31 v1_raid0 - mounted as /scratch
  • home - from myson-t31, v1_raid5, myosin-t32 v1_raid5 - mounted as as /home

• Generated using the Veritas vxprint command:

myosin# vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm rootdisk     c0t0d0s2     -        35363560 -        -        -       -
dm rootmir      c1t0d0s2     -        35358848 -        -        -       -

v  rootvol      root         ENABLED  24577792 -        ACTIVE   -       -
pl rootvol-01   rootvol      ENABLED  24577792 -        ACTIVE   -       -
sd rootdisk-B0  rootvol-01   ENABLED  1        0        -        -       Block0
sd rootdisk-02  rootvol-01   ENABLED  24577791 1        -        -       -
pl rootvol-02   rootvol      ENABLED  24577792 -        ACTIVE   -       -
sd rootmir-01   rootvol-02   ENABLED  24577792 0        -        -       -

v  swapvol      swap         ENABLED  8194168  -        ACTIVE   -       -
pl swapvol-01   swapvol      ENABLED  8194168  -        ACTIVE   -       -
sd rootdisk-01  swapvol-01   ENABLED  8194168  0        -        -       -
pl swapvol-02   swapvol      ENABLED  8194168  -        ACTIVE   -       -
sd rootmir-02   swapvol-02   ENABLED  8194168  0        -        -       -

Disk group: backupdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg backupdg     backupdg     -        -        -        -        -       -

dm backupd01    c4t3d0s2     -        1000624128 -      -        -       -
dm backupd02    c7t2d0s2     -        1000624128 -      -        -       -

v  backup       fsgen        ENABLED  2001248256 -      ACTIVE   -       -
pl backup-01    backup       ENABLED  2001248256 -      ACTIVE   -       -
sd backupd01-01 backup-01    ENABLED  1000624128 0      -        -       -
sd backupd02-01 backup-01    ENABLED  1000624128 0      -        -       -

Disk group: homedg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg homedg       homedg       -        -        -        -        -       -

dm home01       c4t3d2s2     -        714731520 -       -        -       -
dm home02       c7t2d2s2     -        714731520 -       -        -       -

v  home         fsgen        ENABLED  1429463040 -      ACTIVE   -       -
pl home-01      home         ENABLED  1429463040 -      ACTIVE   -       -
sd home01-01    home-01      ENABLED  714731520 0       -        -       -
sd home02-01    home-01      ENABLED  714731520 0       -        -       -

Disk group: scratchdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg scratchdg    scratchdg    -        -        -        -        -       -

dm scratch01    c4t3d1s2     -        285892608 -       -        -       -
dm scratch02    c7t2d1s2     -        285892608 -       -        -       -

v  scratch      fsgen        ENABLED  571785216 -       ACTIVE   -       -
pl scratch-01   scratch      ENABLED  571785216 -       ACTIVE   -       -
sd scratch01-01 scratch-01   ENABLED  285892608 0       -        -       -
sd scratch02-01 scratch-01   ENABLED  285892608 0       -        -       -

Management

StorEdgeT3+ Arrays

Access to the arrays is only available from the vasopressin HP9000. From the shell command line, simply telnet to the designated IP address for the particular enclosure identified above. Please contact Rob Vogelbacher for login information.

Once in the array console, array volumes can be enumerated and configured with the vol command.

• List current volumes and status:
  • vol list
• Delete a volume:
  • vol unmount volume-name
  • vol remove volume-name
• Add an array:
  • vol add volume-name data undn-n raid n standby und9

Status of the array can be monitored with the sys stat command.

For more information, please read the Sun StorEdge T3 and T3+ Array Administrator Guide

Veritas Volume Management

Most Veritas volume management functions can be performed with the various commands starting with 'vx' that are located in /usr/sbin/

For a menu-based volume configuration management, try the vxdiskadm command.

The Veritas Volume Manager Administrator's Guide is an indispensible resource for understanding the intricacies and capabilities of this software. The web is also a good resource for troubleshooting, as Veritas is the most commonly used storage array/disk management software.

Troubleshooting Tips

Some useful tips/commands for troubleshooting the storage arrays:

• Veritas commands
  • /usr/sbin/vxdiskadm - menu based utility for importing/exporting disk groups, onlining and offlining devices, and verifying configuration
  • /usr/sbin/vxmend - can be used to clean plexes after bringing back defunct disks/devices online
  • /usr/sbin/vxprint - list configuration information
  • /usr/sbin/vxvol - used to stop and start volumes

RJV - 9.24.06

• Steps taken to bring back home and backup partitions online
  • export diskgroup backupdg and homedg using vxdiskadm, option 9
  • import disgroups using vxdiskadm, option 8
  • clean plexes using vxmend:

vxmend -o force fix clean backup-01
vxmend  -f force fix clean home-01
vxmend off backup-01
vxmend off home-01
vxmend on backup-01
vxmend on home-01

• • start volumes with vxvol start backup, vxvol start home
  • fsck devices with fsck -y /dev/vx/dsk/backupdg/backup, fsck -y /dev/vx/dsk/homedg/home
  • mount volumes - mount /backup, mount /home

ToDos

• Document Titan setup
• Update firmware on arrays
• Setup remote logging of enclosures
• SNMP traps?

-- RobVogelbacher - 28 Nov 2005