tobert.org

The random and rare writings of Al Tobey.

Saturday, January 31, 2009

Why OpenSolaris is growing on me ...

I crashed my OpenSolaris 2008.11 and it was awesome. The crash was entirely my fault and I knew it was likely, but I can't resist playing with shiny new toys. I'm testing ZFS on iSCSI LUN's and was taking packet traces of iscsitgt and COMSTAR for the developers of LIO Target v3.0 so they can get all the bits right for automatic MPxIO setup when using a Solaris initiator. When I installed COMSTAR on OpenSolaris 2008.11, I noticed that one of the things included in the tarball I found is an updated iSCSI initiator package with some interesting infrastructural changes (iscsidm). I decided to give it a whirl on my test cluster with 10 2.5TB LUN's connected to a hopped-up Dell r905 running OS2008.11.
atobey@opensolaris01:~# wget http://www.opensolaris.org/os/project/iser/SUNWiscsitr-bins-121108.i386.tar.gz
atobey@opensolaris01:~# tar -xzvf SUNWiscsitr-bins-121108.i386.tar.gz
atobey@opensolaris01:~# pfexec su -
# Hmm, good chance this is gonna end badly ...
root@opensolaris01:~# zfs snapshot rpool/ROOT/opensolaris-1@2009-01-31-01:53:47
# Ok, now that I have a backout plan, continue ...
root@opensolaris01:~# pkgadd -d . SUNWiscsidmr
root@opensolaris01:~# pkgadd -d . SUNWiscsidmu
root@opensolaris01:~# pkgadd -d . SUNWiscsir
root@opensolaris01:~# pkgadd -d . SUNWiscsiu
root@opensolaris01:~# rm -f /etc/iscsi/* # only known way to truly clean up all iSCSI configuration
root@opensolaris01:~# reboot
# get some coffee, reconnect ...
atobey@opensolaris01:~# pfexec su -
root@opensolaris01:~# iscsiadm modify discovery --static disable
root@opensolaris01:~# iscsiadm modify initiator-node -N iqn.2008-12.org.tobert.opensolaris01
root@opensolaris01:~# iscsiadm modify initiator-node --node-alias opensolaris01
root@opensolaris01:~# iscsiadm modify initiator-node --configured-sessions 2 # required for MPxIO
root@opensolaris01:~# iscsiadm modify initiator-node --authentication chap
root@opensolaris01:~# iscsiadm modify initiator-node --CHAP-name torgiscsi
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box01:lio0,192.168.1.11:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box01:lio0,192.168.2.11:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box02:lio0,192.168.1.12:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box02:lio0,192.168.2.12:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box03:lio0,192.168.1.13:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box03:lio0,192.168.2.13:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box04:lio0,192.168.1.14:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box04:lio0,192.168.2.14:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box05:lio0,192.168.1.15:3260
root@opensolaris01:~# iscsiadm add static-config iqn.2009-01.org.tobert.box05:lio0,192.168.2.15:3260
root@opensolaris01:~# iscsiadm modify initiator-node --CHAP-secret
# enter password
root@opensolaris01:~# iscsiadm modify discovery --static enable
root@opensolaris01:~# zpool import -o cachefile=/etc/zfs/tank.cache tank
root@opensolaris01:~# zpool list
NAME        SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank       26.3T   800G  25.5T     2%  ONLINE  -
rpool        68G  20.8G  47.2G    30%  ONLINE  -
root@opensolaris01:~# zpool status tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          c0t234d0  ONLINE       0     0     0
          c0t235d0  ONLINE       0     0     0
          c0t230d0  ONLINE       0     0     0
          c0t231d0  ONLINE       0     0     0
          c0t226d0  ONLINE       0     0     0
          c0t227d0  ONLINE       0     0     0
          c0t222d0  ONLINE       0     0     0
          c0t223d0  ONLINE       0     0     0
          c0t216d0  ONLINE       0     0     0
          c0t217d0  ONLINE       0     0     0

errors: No known data errors
The devices are just regular iSCSI devices without MPxIO since I didn't configure it for this test. I fired off an instance of ffsb set to run for a day over 800GB of data with a read-heavy profile. This test in various forms has run perfectly for a couple months now. The initiator crashed the kernel sometime during the night. This is what I saw on my serial console:
apnainci[ccp[uc6]p/ut2h]re/atdh=rfefafdfff0=fff07b9f2ff113c80: abs8s3ertion 15fa4i6l0e:d B:A D TR0, fAiPl:e :t y.p.e/=.e./com6
dd
Lr=0f fffofcfc0u0r7rbe9d2 3ian mo80 gduleenun "uix:naisxs" fdauiel +t7oe  a( )N
 L fpfofifff0n07bter9 2d3ad0e riedfme:riedmn_ce
.tyanscki_nagb ofritl_eo nsey+s1tde7m s(.).
ffffff007b923b20 idm:idm_task_abort+9a ()
ffffff007b923b70 idm:idm_update_state+1fb ()
ffffff007b923ba0 idm:idm_state_s8_cleanup+85 ()
ffffff007b923be0 idm:idm_conn_event_handler+144 ()
ffffff007b923c60 genunix:taskq_thread+193 ()
ffffff007b923c70 unix:thread_start+8 ()

dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
 done
dump aborted: please record the above information!
rebooting...
Clearly the idm module still needs some polishing, and I'm sure it'll get it. My story so far is not here to disparage the idm or COMSTAR work; it's here as a backdrop to why I dig OpenSolaris more and more the longer I use it. Backing out patches always sucks, on nearly every OS, and with nearly every package manager. With ZFS, life sucks a lot less when you find yourself backed into a corner:
root@opensolaris01:~# zfs list -t all
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
rpool                                         52.8G  14.1G    75K  /rpool
rpool/ROOT                                    4.08G  14.1G    18K  legacy
rpool/ROOT/opensolaris                         106M  14.1G  2.31G  -
rpool/ROOT/opensolaris-1                      3.90G  14.1G  3.66G  /
rpool/ROOT/opensolaris-1@install              71.0M      -  2.21G  -
rpool/ROOT/opensolaris-1@2009-01-13-23:06:36  40.0M      -  2.26G  -
rpool/ROOT/opensolaris-1@2009-01-31-01:53:47  78.4M      -  3.66G  -
rpool/dump                                    16.0G  14.1G  16.0G  -
rpool/export                                   761M  14.1G    19K  /export
rpool/export/home                              761M  14.1G    32K  /export/home
rpool/export/home/atobey                       761M  14.1G   761M  /export/home/atobey
rpool/swap                                    32.0G  46.1G    16K  -
root@opensolaris01:~# zfs clone rpool/ROOT/opensolaris-1@2009-01-31-01:53:47 rpool/ROOT/opensolaris-2
root@opensolaris01:~# zfs list -t all
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
rpool                                         52.8G  14.1G    75K  /rpool
rpool/ROOT                                    4.08G  14.1G    18K  legacy
rpool/ROOT/opensolaris                         106M  14.1G  2.31G  -
rpool/ROOT/opensolaris-1                      3.90G  14.1G  3.66G  /
rpool/ROOT/opensolaris-1@install              71.0M      -  2.21G  -
rpool/ROOT/opensolaris-1@2009-01-13-23:06:36  40.0M      -  2.26G  -
rpool/ROOT/opensolaris-1@2009-01-31-01:53:47  78.4M      -  3.66G  -
rpool/ROOT/opensolaris-2                      74.8M  14.1G  3.66G  -
rpool/dump                                    16.0G  14.1G  16.0G  -
rpool/export                                   761M  14.1G    19K  /export
rpool/export/home                              761M  14.1G    32K  /export/home
rpool/export/home/atobey                       761M  14.1G   761M  /export/home/atobey
rpool/swap                                    32.0G  46.1G    16K  -
root@opensolaris01:~# zfs set mountpoint=/ rpool/ROOT/opensolaris-2
root@opensolaris01:~# zfs set canmount=noauto rpool/ROOT/opensolaris-2
# just because they're there on the other clones ...
root@opensolaris01:~# zfs set org.opensolaris.libbe:policy=static rpool/ROOT/opensolaris-2
# ran uuidgen on a linux box
root@opensolaris01:~# zfs set org.opensolaris.libbe:uuid=35f4532d-e452-453b-86a9-1de8e1b296f9
root@opensolaris01:~# sed -i 's/opensolaris-1/opensolaris-2/g' /rpool/boot/grub/menu.lst
root@opensolaris01:~# reboot
ZFS is cool. Maybe someday when btrfs grows up, Linux can do this for its root filesystems too.

Notes:
  • The shell stuff in this post are reenactments of a sort because I didn't think to blog about this until I was all done.

  • pkg image-update will also set up snapshots/clones for easy backout, which is why rpool/ROOT/opensolaris-1 existed.

  • Network connection was 2 ixgbe's to 5 Linux boxes with 2 e1000e's each

  • I used sed on the GRUB config to save space. Don't try it at home unless you really understand what it's doing.

  • Hooray for /usr/gnu and having it at the front of $PATH

  • pfexec su - for root access isn't really best practice, but it sure saves a lot of typing.
Posted by Albert P. Tobey at 11:11 AM 2 comments
Labels: , , , , , , , ,

Wednesday, January 14, 2009

Installing Solaris Cluster Express 12/08 on OpenSolaris 2008.11

I googled and checked around the usual places to see if Sun Cluster Express is supported on OpenSolaris 2008.11 and didn't find any useful information. Since SXCE 105 is not an option for me. (Its iSCSI initiator is buggy and the iSCSI boot infestation is a whole different rant.) I'm sticking with 2008.11.

To get started, I downloaded SCX 12/08 for SXCE, which comes as a zip file full of SysV packages. I didn't try too hard to find tomcat (SUNWtcatu) packages for OpenSolaris and just pulled them from the SXCE ISO. I don't think the version of SXCE matters much, but it's probably best to use 101a. I used 105. All of my systems are x86.

You can't just grab SUNWtactu, so I manually figured out the dependencies. Here they are:
SUNWctpls
SUNWdtcor
SUNWj5dev
SUNWj5rt
SUNWmfrun
SUNWtcatr
SUNWtcatu


I had the SXCE 105 ISO on a linux box so I loopback mounted it there and rsynced the packages. If you're on Solaris, you'll want lofi.

Then I installed each package one at a time so I could watch for errors. I got a couple warnings from SUNWj5rt but they were harmless.
pkgadd -d . SUNWdtcor
pkgadd -d . SUNWctpls
pkgadd -d . SUNWmfrun
pkgadd -d . SUNWj5rt
pkgadd -d . SUNWj5dev
pkgadd -d . SUNWtcatu
pkgadd -d . SUNWtcatr


Once this is done the installer runs just fine.
   Enter 1 to view installation summary and Enter 2 to view installation logs
  [1] {"!" exits}
Installation Summary Report
Install Summary
Solaris Cluster Express Suite : Installed
Java DB : Installed, Configure After Install
Solaris(TM) Cluster Express 12/08 : Installed, Configure After Install
Solaris(TM) Cluster Express Agents 12/08 : Installed, Configure After Install
Solaris(TM) Cluster Express Geographic Edition 12/08 : Installed, Configure
After Install
Quorum Server : Installed, Configure After Install
High Availability Session Store 4.4 : Installed
All Shared Components : Installed, Configure After Install
Configuration Data
The configuration log is saved in :  /var/sadm/install/logs/JavaES_Install_log.
1791792999


Hope this helps somebody.

http://www.opensolaris.org/os/
http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/SCXdocs/SCX/
Posted by Albert P. Tobey at 6:43 PM 1 comments
Labels: , , , ,
Subscribe to: Posts (Atom)