Tall tales for System Admins

A healthy dose of mdb

2008-10-16T09:20:00.000-06:00

Since the past several months, we have been encountering a baffling problem with Solaris 10 systems hanging on their way up after a reboot (or simply a boot) in the multi-user mode (run level 3). Since we do our system patching at regular intervals (and since solaris runs so well that we don't need to reboot our servers otherwise), we didn't notice this until we came upon our latest patch cycle last month.

We noticed that our systems were not booting up the normal way. The only way to boot was to "trick" the system into booting into single-user mode and then exiting out to the multi-user run-level.

This was a problem happening only on systems running Solaris 10, with Veritas Volume Manager 5.0 MP1. So naturally we leaned towards tackling this first as a possible VxVM problem (introduced by some patch during the patch cycle). And the patch rev we had decided to apply had the Veritas Storage Foundation 5.0 MP1 RP4 in it.

Again, to reiterate the problem description --

Description:

Systems running Solaris 10 and Veritas Volume Manager tend to hang when booted with a normal init 6 (ie while booting into Run Level 3).

Workaround:

Boot the system in run level S (single user mode) and then exit out of it to boot into multi-user mode.

Details:

Opened a case with Veritas and sent them VRTSexplorer outputs and copies of our messages file from problem host nodeA.

After several iterations of generating explorers and veritas configuration information, Veritas still didn't have anything substantial they could pin this issue on. They asked me to generate a crash dump of the hanging system (usually should be able to coredump a running host by breaking it and running "sync" from the OBP).

After repeated attempts at generating the core, I was unable to do so. It seems like the system hangs before the dump device is initialized/configured by the OS.

Using the work around, I enabled the solaris deadman timer (which incidentally we should have on all our servers). This involves setting the following line in the /etc/system file --

* set snooping = 1

What the deadman timer does is sents a high-priority hardware interrupt to each CPU (or Core or strand depending on the platform) and updates a counter upon successful response by the CPU. In case of a system being hung (due to hardware issues especially), this count might not increase with an increase in clock tick (interrupts aresent to CPUs every tick). When this counter is not incremented, the kernel panics and kills itself.

This didn't work because we didn't encounter that kind of a problem (but is a good idea to have enabled nonetheless).

Another alternative for panicking a system is as follows

Boot the server in kernel debug mode (in SPARC systems it is done by running boot -k from the OBP). This loads the kmdb module into the kernel as it boots up. Breaking the system while in kernel debug mode will not drop it into the OBP but instead launch an mdb interface.

One can generate a coredump by running $ < i="" tried="" that="" but="" no="">

[17]> $ <>
panic[cpu17]/thread=2a101a3fca0: BAD TRAP: type=9 rp=2a101a3f760 addr=0 mmu_fsr=0

sched: trap type = 0x9
pid=0, pc=0x0, sp=0x2a101a3f001, tstate=0x1606, context=0x0
g1-g7: 1011be4, 1, 1870000, 20040, 1860c00, 0, 2a101a3fca0

000002a101a3f480 unix:die+9c (9, 2a101a3f760, 0, 0, 2a101a3f540, 1901ec0)
%l0-3: 0000000000000000 0000000000000009 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000001097000
000002a101a3f560 unix:trap+6cc (2a101a3f760, 10000, 0, 0, 30003dac000, 2a101a3fca0)
%l0-3: 0000000000000000 0000000001859480 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000000010200
000002a101a3f6b0 unix:ktl0+64 (0, 20040, 1860c00, 1, 1260ce4, 1901ec0)
%l0-3: 0000030003dac000 0000000000000088 0000000000001606 0000000001020354
%l4-7: 0000000000000000 00000000018f1800 000000000000000c 000002a101a3f760
000002a101a3f800 unix:debug_enter+108 (0, a, a, 1826400, 0, 1870000)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30
000002a101a3f8d0 unix:abort_seq_softintr+94 (1826400, 18f1800, 30003dac000, 2a101a3fd78, 1, 189f800)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30

syncing file systems... done
skipping system dump - no dump device configured
rebooting...

SC Alert: Host System has Reset

SC Alert: Failed to send email alert for recent event.

SC Alert: Indicator SYS/ACT is now STANDBY BLINK

So back to the kernel debug mode again. This time I decided to try and investigate the kernel (you can do this against a coredump as well).

[17]> ::msgbuf
MESSAGE
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 supports power management.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
DMA restricted to lower 4GB due to errata
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 Firmware version v1.9.0.0 (IR)
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: IOC Operational.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: Initiator WWNs: 0x5080020000262858-0x508002000026285b
PCI-device: scsi@0, mpt0
mpt0 is /pci@780/pci@0/pci@9/scsi@0
sd1 at mpt0: target 0 lun 0
sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@0,0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0 (sd1) online
root on /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a fstype ufs
px1 at root: 0x7c0 0x0
px1 is /pci@7c0
PCI Express-device: pci@0, pxb_plx5
pxb_plx5 is /pci@7c0/pci@0
PCI-device: pci@1, pxb_plx6
pxb_plx6 is /pci@7c0/pci@0/pci@1
PCI-device: pci@0, px_pci0
px_pci0 is /pci@7c0/pci@0/pci@1/pci@0
PCI-device: ide@8, uata0
uata0 is /pci@7c0/pci@0/pci@1/pci@0/ide@8
WARNING: px1: spurious interrupt from ino 0x4
uata-0#0

sd2 at uata0: target 0 lun 0
sd2 is /pci@7c0/pci@0/pci@1/pci@0/ide@8/sd@0,0
sd3 at mpt0: target 1 lun 0
sd3 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0 (sd3) online
sd4 at mpt0: target 2 lun 0
sd4 is /pci@780/pci@0/pci@9/scsi@0/sd@2,0
/pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd4) online
sd5 at mpt0: target 3 lun 0
sd5 is /pci@780/pci@0/pci@9/scsi@0/sd@3,0
/pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd5) online
iscsi0 at root
iscsi0 is /iscsi
root remounted on /pseudo/vxio@0:0 fstype ufs
virtual-device: cnex0
cnex0 is /virtual-devices@100/channel-devices@200
PCI-device: isa@2, ebus0
ebus0 is /pci@7c0/pci@0/pci@1/pci@0/isa@2
pseudo-device: dld0
dld0 is /pseudo/dld@0
PCI-device: usb@5, ohci0
ohci0 is /pci@7c0/pci@0/pci@1/pci@0/usb@5
PCI-device: usb@6, ohci1
ohci1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6
cpu0: UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
cpu1: UltraSPARC-T1 (cpuid 1 clock 1000 MHz)
cpu1 initialization complete - online
cpu2: UltraSPARC-T1 (cpuid 2 clock 1000 MHz)
cpu2 initialization complete - online
cpu3: UltraSPARC-T1 (cpuid 3 clock 1000 MHz)
cpu3 initialization complete - online
cpu4: UltraSPARC-T1 (cpuid 4 clock 1000 MHz)
cpu4 initialization complete - online
cpu5: UltraSPARC-T1 (cpuid 5 clock 1000 MHz)
cpu5 initialization complete - online
cpu6: UltraSPARC-T1 (cpuid 6 clock 1000 MHz)
cpu6 initialization complete - online
cpu7: UltraSPARC-T1 (cpuid 7 clock 1000 MHz)
cpu7 initialization complete - online
cpu8: UltraSPARC-T1 (cpuid 8 clock 1000 MHz)
cpu8 initialization complete - online
cpu9: UltraSPARC-T1 (cpuid 9 clock 1000 MHz)
cpu9 initialization complete - online
cpu10: UltraSPARC-T1 (cpuid 10 clock 1000 MHz)
cpu10 initialization complete - online
cpu11: UltraSPARC-T1 (cpuid 11 clock 1000 MHz)
cpu11 initialization complete - online
cpu12: UltraSPARC-T1 (cpuid 12 clock 1000 MHz)
cpu12 initialization complete - online
cpu13: UltraSPARC-T1 (cpuid 13 clock 1000 MHz)
cpu13 initialization complete - online
cpu14: UltraSPARC-T1 (cpuid 14 clock 1000 MHz)
cpu14 initialization complete - online
cpu15: UltraSPARC-T1 (cpuid 15 clock 1000 MHz)
cpu15 initialization complete - online
cpu16: UltraSPARC-T1 (cpuid 16 clock 1000 MHz)
cpu16 initialization complete - online
cpu17: UltraSPARC-T1 (cpuid 17 clock 1000 MHz)
cpu17 initialization complete - online
cpu18: UltraSPARC-T1 (cpuid 18 clock 1000 MHz)
cpu18 initialization complete - online
cpu19: UltraSPARC-T1 (cpuid 19 clock 1000 MHz)
cpu19 initialization complete - online
cpu20: UltraSPARC-T1 (cpuid 20 clock 1000 MHz)
cpu20 initialization complete - online
cpu21: UltraSPARC-T1 (cpuid 21 clock 1000 MHz)
cpu21 initialization complete - online
cpu22: UltraSPARC-T1 (cpuid 22 clock 1000 MHz)
cpu22 initialization complete - online
cpu23: UltraSPARC-T1 (cpuid 23 clock 1000 MHz)
cpu23 initialization complete - online
USB 1.10 device (usb3eb,3301) operating at full speed (USB 1.x) on USB 1.10 root
hub: hub@1, hubd1 at bus address 2
hubd1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1
/pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1 (hubd1) online
PCI-device: pci@0,2, px_pci1
px_pci1 is /pci@7c0/pci@0/pci@1/pci@0,2
PCI-device: pci@1, pxb_plx1
pxb_plx1 is /pci@780/pci@0/pci@1
PCI-device: pci@2, pxb_plx2
pxb_plx2 is /pci@780/pci@0/pci@2
PCI-device: pci@8, pxb_plx3
pxb_plx3 is /pci@780/pci@0/pci@8
PCI-device: pci@2, pxb_plx7
pxb_plx7 is /pci@7c0/pci@0/pci@2
PCI-device: pci@8, pxb_plx8
pxb_plx8 is /pci@7c0/pci@0/pci@8
PCI-device: pci@9, pxb_plx9
pxb_plx9 is /pci@7c0/pci@0/pci@9
NOTICE: e1000g0 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[0] : Adapter copper link is down.
NOTICE: e1000g1 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[1] : Adapter copper link is down.
NOTICE: e1000g3 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[3] : Adapter copper link is down.
NOTICE: pciex8086,105e - e1000g[0] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: pciex8086,105e - e1000g[1] : Adapter 1000Mbps full duplex copper link is
up.
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
NOTICE: pciex8086,105e - e1000g[3] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-3-1700 dmpnode 300/0x0 has migrated from enclosure FAKE_E
NCLR_SNO to enclosure DISKS

NOTICE: e1000g2 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11

Look at the process table...

[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f30028 ssmagent.bin
R 9 1 9 9 0 0x42000000 000006001292f070 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 0000060017e43080 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e42468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017dc8018 vxconfigd

Pick a suspect thread and dig deeper --

[17]> 0000060017e43080::findstack -v
kmdb: thread 60017e43080 isn't in memory
[17]> 0000060017e43080::walk thread
30003eba480
[17]> 0000060017e43080::walk thread| ::findstack -v
stack pointer for thread 30003eba480: 2a101514ff1
[ 000002a101514ff1 cv_wait_sig_swap_core+0x130() ]
000002a1015150a1 waitid+0x484(0, 60017e42468, 0, 60017e430e8, 0, 1)
000002a101515171 waitsys32+0x10(0, 38, ffbffa80, 83, 39590, 3a57c)
000002a1015152e1 syscall_trap32+0xcc(0, 38, ffbffa80, 83, 39590, 3a57c)
[17]>

[17]> 0000060017e42468::walk thread| ::findstack -v
stack pointer for thread 30003fae200: 2a101dbeff1
[ 000002a101dbeff1 cv_wait_sig_swap_core+0x130() ]
000002a101dbf0a1 waitid+0x484(0, 0, 0, 60017e424d0, 7, 1)
000002a101dbf171 waitsys32+0x10(7, 0, ffbffaa8, 3, 2909e8, 0)
000002a101dbf2e1 syscall_trap32+0xcc(7, 0, ffbffaa8, 3, 2909e8, 0)

[17]> 0000060017dc8018::walk thread | ::findstack -v
stack pointer for thread 30003eb0420: 2a10072ec21
[ 000002a10072ec21 cv_wait+0x38() ]
000002a10072ecd1 delay+0x90(1, 1870000, df39, df3a, 1, 18f0818)
000002a10072ed81 vxio`volkio_flush_cached_io+0xdc(7006ea30, 30003eb0420,
7006789c, 7006e9e8, 0, 0)
000002a10072ee31 vxio`vol_commit_flush_cached_io+0x60(60011503240, 70046d28, 0
, 0, 0, 0)
000002a10072eee1 vxio`vol_ktrans_commit+0x6fc(7006b2f0, 1, 7006789c, 7006d92c
, 60011503240, 0)
000002a10072ef91 vxio`volsioctl_real+0x4c8(70034000, 564f4c86, 0, 100003,
60011403768, 2a10072fadc)
000002a10072f0e1 fop_ioctl+0x20(60012a7f900, 564f4c86, 0, 100003, 60011403768
, 12f1770)
000002a10072f191 ioctl+0x184(1, 600115e8a50, 0, 564f4c86, 0, 564f4c86)
000002a10072f2e1 syscall_trap32+0xcc(1, 564f4c86, 0, 564f4c86, 0, 820770)
[17]>

Found something in the last one, but no immediate red flags. But then I saw this (ssmagent running). SSM Agent is a SNMP agent that is used to monitor our systems and report back to Netcool, our Fault Management server.

[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f5e028 ssmagent.bin
R 43 1 42 42 0 0x42020000 0000060017db7078 dhcpagent
R 9 1 9 9 0 0x42000000 0000060012922458 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 000006001172cc20 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e10468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017db4018 vxconfigd
[17]> 0000060017f5e028::walk thread| ::findstack -v
stack pointer for thread 30003fb55a0: 2a101e02961
[ 000002a101e02961 sema_p+0x130() ]
000002a101e02a11 biowait+0x6c(60017dda100, 0, 1870000, 30003dbe000, 2000,
60017dda100)
000002a101e02ac1 ufs`ufs_getpage_miss+0x2ec(60019a0f300, 40000, 4de,
600129cea20, fdba0000, 2a101e03760)
000002a101e02bc1 ufs`ufs_getpage+0x694(300014b7e00, 40000, 1, 0, 1, 3)
000002a101e02d21 fop_getpage+0x44(60019a0f300, 600114659c0, 60011403978, 3,
fdba0000, 3)
000002a101e02df1 segvn_fault+0xb04(8000, 600129cea20, 3, 2000, 40000, 0)
000002a101e02fc1 as_fault+0x4c8(600129cea20, 600129d9200, fdba0000,
60011736320, 189eb00, 0)
000002a101e030d1 pagefault+0x68(fdba14a8, 0, 3, 0, 60017f5e028, 600117362a8)
000002a101e03191 trap+0xd50(2a101e03b90, 10000, 0, 3, fdba14a8, 0)
000002a101e032e1 utl0+0x4c(ff3f40fc, ff3f5a70, 1, 0, ff3f4910, 821)
stack pointer for thread 30003f08820: 2a102005091
[ 000002a102005091 cv_wait_sig_swap_core+0x130() ]
000002a102005141 lwp_park+0x130(0, 1, 30003f089c6, 30003f08820, 0, 100000)
000002a102005231 syslwp_park+0x54(0, 0, 0, 0, ff092010, 1)
000002a1020052e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092010, 1)
stack pointer for thread 30003fb4560: 2a101d9f091
[ 000002a101d9f091 cv_wait_sig_swap_core+0x130() ]
000002a101d9f141 lwp_park+0x130(0, 1, 30003fb4706, 30003fb4560, 0, 100000)
000002a101d9f231 syslwp_park+0x54(0, 0, 0, 0, ff092020, 1)
000002a101d9f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092020, 1)
stack pointer for thread 30003ef77c0: 2a1008e9091
[ 000002a1008e9091 cv_wait_sig_swap_core+0x130() ]
000002a1008e9141 lwp_park+0x130(0, 1, 30003ef7966, 30003ef77c0, 0, 100000)
000002a1008e9231 syslwp_park+0x54(0, 0, 0, 0, ff092030, 1)
000002a1008e92e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092030, 1)
stack pointer for thread 30003fd35c0: 2a101db6f91
[ 000002a101db6f91 cv_timedwait_sig+0x16c() ]
000002a101db7041 cv_waituntil_sig+0x8c(60017dc8592, 60017dc8558, 2a101db7ad0,
2, 18f0800, 2)
000002a101db7111 poll_common+0x4e8(60012996580, 60017f5e028, 2a101db7ad0, 0,
fe57bcd0, 2)
000002a101db7201 pollsys+0xf8(fe57bcd0, 1, fe57bd70, 0, 2a101db7ad0, 0)
000002a101db72e1 syscall_trap32+0xcc(fe57bcd0, 1, fe57bd70, 0, fe57bd70, 0)
stack pointer for thread 30003efa460: 2a101df3091
[ 000002a101df3091 cv_wait_sig_swap_core+0x130() ]
000002a101df3141 lwp_park+0x130(0, 1, 30003efa606, 30003efa460, 0, 100000)
000002a101df3231 syslwp_park+0x54(0, 0, 0, 0, ff092050, 1)
000002a101df32e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092050, 1)
stack pointer for thread 30003fe55e0: 2a10089efc1
[ 000002a10089efc1 cv_timedwait_sig+0x16c() ]
000002a10089f071 cv_waituntil_sig+0x8c(30003fe5786, 30003fe5788, 2a10089fa10,
2, 18f0800, 2)
000002a10089f141 lwp_park+0x130(fdffbd50, 0, 30003fe5786, 30003fe55e0, 0,
100000)
000002a10089f231 syslwp_park+0x54(0, fdffbd50, 0, 0, ff092060, 1)
000002a10089f2e1 syscall_trap32+0xcc(0, fdffbd50, 0, 0, ff092060, 1)
stack pointer for thread 30003ff4a20: 2a10083f091
[ 000002a10083f091 cv_wait_sig_swap_core+0x130() ]
000002a10083f141 lwp_park+0x130(0, 0, 30003ff4bc6, 30003ff4a20, 0, 100000)
000002a10083f231 syslwp_park+0x54(0, 0, 0, 0, ff092070, 1)
000002a10083f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092070, 1)
[17]>

It wasn't vxconfigd, but was ssmagent (a Netcool/Micromuse/IBM SNMP monitoring agent) instead that was sitting in a blocked i/o wait state (biowait) and effectively preventing VxVM from starting up the rootdg and mounting the encapsulated volumes.
An svcadm disable ssmagent from the running OS, fixed the problem and system now boots just fine.

Oracle 10G R2 and ASM in a Solaris Non-Global Zone

2008-08-22T11:16:00.000-06:00

It's been a while since I've made any posts. I am looking to migrate the site shortly (host it privately). But here's one I thought I'd share with my thousands of anonymous readers.

Really folks...if you like what you read here...post a few lines. If you hate it...post that too!

Global Zone configuration

Add the oracle user and dba group

vistatst-01:$() # grep dba /etc/group
dba::600:
vistatst-01:$() # grep oracle /etc/passwd
oracle:x:101:600:Oracle DBA:/u01/app/oracle:/usr/bin/ksh

Configure LUNs for Oracle ASM formatting

c4t6006016030C01C00EC2DC1FE8052DD11d0s2 is ASM disk...Let's inspect it's partition table...
       2      5    01          0 104852480 104852479
       6      0    00       2560 104849920 104852479

c4t6006016030C01C009698C4458152DD11d0s2 is ASM disk...Let's inspect it's partition table...
       2      5    01          0 104852480 104852479
       6      0    00       2560 104849920 104852479

c4t6006016030C01C0046A8487A8152DD11d0s2 is ASM disk...Let's inspect it's partition table...
       2      5    01          0 104852480 104852479
       6      0    00       2560 104849920 104852479

c4t6006016030C01C00CEF5BFAA8152DD11d0s2 is ASM disk...Let's inspect it's partition table...
       2      5    01          0 104852480 104852479
       6      0    00       2560 104849920 104852479

vistatst-01:$() #

Here we have chosen to set up slice 6 as the "disk" with cylinder 0 being excluded by manually partitioning it. The starting sector count (as you can observe is 2560 instead of sector 0).

Make oracle:dba the owner of these LUNS.

vistatst-01:$() # for i in `vxdisk list|grep invalid|awk '{print $1}>
lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw

lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw

lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw

lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw

And change the ownerships of the underlying files...

crw-r-----   1 oracle   dba      118, 426 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw
crw-r-----   1 oracle   dba      118, 418 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw
crw-r-----   1 oracle   dba      118, 402 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw
crw-r-----   1 oracle   dba      118, 434 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw

Configure Solaris Non-Global Zone (NGZ)

vistatst-01:$() # zonecfg -z vistatst-01-z01 set limitpriv=default,proc_priocntl
vistatst-01:$() # zonecfg -z vistatst-01-z01 export
create -b
set zonepath=/zonefiles/vistatst-01-z01
set autoboot=true
set limitpriv=default,proc_priocntl
set ip-type=shared
add net
set address=10.10.10.119
set physical=ce0
end
add device
set match=/dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6
end
add device
set match=/dev/dsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6
end
add device
set match=/dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s6
end
add device
set match=/dev/dsk/c4t6006016030C01C009698C4458152DD11d0s6
end
add device
set match=/dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s6
end
add device
set match=/dev/dsk/c4t6006016030C01C0046A8487A8152DD11d0s6
end
add device
set match=/dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6
end
add device
set match=/dev/dsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6
end
add dataset
set name=vinmarttstpool
end
vistatst-01:$() #
vistatst-01:$() # zoneadm -z vistatst-01-z01 reboot

Set up CSSD to start up

If CSSD doesn't start up at this point -- run the following command from $ORACLE_HOME/bin --

#./localconfig reset
vinmartst-01:$(bin) # ./localconfig reset
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Adding to inittab
Aug 14 11:58:36 vinmartst-01 root: Oracle Cluster Synchronization Service starting by user request.
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Aug 14 11:58:36 vinmartst-01 root: Cluster Ready Services completed waiting on dependencies.
CSS is active on these nodes.
        vinmartst-01
CSS is active on all nodes.
Oracle CSS service is installed and running under init(1M)

Verify that cssd is running in the process table --

vinmartst-01:$(bin) # ps -ef|grep css
  oracle  9120  8207   0 11:58:36 ?           0:00 /u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin
    root  9220  8373   0 11:59:59 console     0:00 grep css

Verify that the LUNs are visible to the NGZ --

vinmartst-01:$(.oracle) # ls -alrt /dev/rdsk
total 4
drwxr-xr-x   2 root     root         512 Aug 12 16:34 .
drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..
crw-r-----   1 oracle   dba      118, 422 Aug 14 13:46 c4t6006016030C01C009698C4458152DD11d0s6
crw-r-----   1 oracle   dba      118, 430 Aug 14 13:55 c4t6006016030C01C00EC2DC1FE8052DD11d0s6
crw-r-----   1 oracle   dba      118, 438 Aug 14 14:10 c4t6006016030C01C00CEF5BFAA8152DD11d0s6
crw-r-----   1 oracle   dba      118, 406 Aug 14 14:10 c4t6006016030C01C0046A8487A8152DD11d0s6

vinmartst-01:$(.oracle) # ls -lart /dev/dsk
total 4
brw-r-----   1 oracle   dba      118, 430 Aug 12 16:34 c4t6006016030C01C00EC2DC1FE8052DD11d0s6
brw-r-----   1 oracle   dba      118, 438 Aug 12 16:34 c4t6006016030C01C00CEF5BFAA8152DD11d0s6
brw-r-----   1 oracle   dba      118, 422 Aug 12 16:34 c4t6006016030C01C009698C4458152DD11d0s6
brw-r-----   1 oracle   dba      118, 406 Aug 12 16:34 c4t6006016030C01C0046A8487A8152DD11d0s6
drwxr-xr-x   2 root     root         512 Aug 12 16:34 .
drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..

Run Oracle Installer and do the rest of the stuff that needs to be done!

References

Dry-running Splunk -- "The IT Search Engine" - I

2008-05-23T12:45:00.001-06:00

This is a study we'd (a former colleague of mine and yours truly) done last year of a log analysis tool called Splunk
What you will read in this article are the results/excerpts of that study.

Some of the questions we asked are as follows --

Why use a log analysis tool?
What do most shops use
What does a tool such as splunk buy us (as an IT shop)
What are it's benefits and pit-falls?
What is the cost of ownership?

Why use a log-analysis tool?

The biggest reason to use such a tool would be to move from a Reactive to Proactive Systems Management paradigm

With the number of systems (about 900+ *nix servers in that shop) and the criticality (many systems cost millions of dollars in down-time) of availability of these, it is imperative to find a tool that can actually be used quickly and effortlessly to analyze valuable log information

If such a tool can look at various layers of a "delivered stack" (aka hardware, os, application, network, san, etc), it would be a gold-mine by virtue of being able to link the stack "end-to-end" and by speeding up the analysis process.

What do most shops use?

Most shops I've been in do log analysis like this --

a) Don't do any log analysis unless absolutely required. And if it is required, admins log into the individual servers and parse through the logs using vi (or using a combination of grep/awk/sed if they are script-savvy)

b) Have a centralized ssh (or god forbid! rsh) trusted admin host from where they launch a log parser script that filters specific key words and that gets emailed to a mailbox or to the individual admins' email boxes

c) have a centralized log host where they run a script akin to the one mentioned above

I've worked in shops of varying sizes -- from a ISP/Telecom giant who ran 4000+ sun servers to a 50-server tiny sweatshop. Most of the shops I've been in fall some where in between (with hosts ranging from 200 - 1000 in number). That's a lot of hosts to manage and a lot of logging that needs to be parsed.

What does a log-analysis tool buy an IT shop?

You've all probably thought about this -- a centralized, easy-to-use log analysis tool buys an IT shop valuable time!

So what does Splunk claim to do?

In there own words --

“The Splunk Server indexes IT data from ANY source. No need to configure it for specific formats, write regular expressions or change your logging output. Search mountains of data by time, keywords, type of event, source, host or relationships to other events. “

Some key features of Splunk:

Universal Indexing
Can index terabytes of data all from one place
Capable of indexing approx. 22,000 events/second at density of 150 bytes/event.

How does splunk acquire data?

Access data from any live source:

Mounted files: NFS/SMB, CIFS/AFP, NAS/SAN, FIFO,
Remote files: rsync, scp/ftp/rcp,
Network ports: UDP & TCP, syslog/syslog-ng, log4j/log4php, JMX/JMS, SNMP
Databases: SQL/ODBC
Splunk Servers: Access data locally on production hosts and forward it to another Splunk Server over SSL/TCP

The actual evaluation results will be the next article.

Servers running on Hot Air

2008-04-01T11:56:00.000-06:00

This infoworld article caught my eye and induced some much needed mirth.

http://www.infoworld.com/article/08/04/01/14FE-april-fool-hp-windy_1.html

NFS HA Service Group in VCS 5.x on Solaris 10

2008-03-07T13:17:00.004-06:00

Remember to delete the service maps for NFS using svccfg command

With Solaris 10 and VCS 5.x, nfsd HAS TO run under VCS control. In order to achieve that, the following needs ton happen (on every node that will host the NFS share) --

Disable/Delete the NFS services from SMF

# svccfg delete -f svc:/network/nfs/server:default
# svccfg delete -f svc:/network/nfs/status:default
# svccfg delete -f svc:/network/nfs/nlockmgr:default

Manually restart lockd, statd and automountd

# /usr/lib/nfs/lockd
# /usr/lib/nfs/statd
# /usr/lib/fs/autofs/automount
# /usr/lib/autofs/automountd

NOTE: In this example (see below), the NFSgrp is configured only for one node. To add another node, add the node name and number to SystemList and AutoStartList

       group NFSgrp (
               SystemList = { hostA = 0 }
               AutoStartList = { hostA }
        )

        DiskGroup nfsDG (
               Critical = 0
               DiskGroup = testdg
        )

        Volume nfsVOL (
              Critical = 0
              Volume = testnfshome
              DiskGroup = testdg
        )


        IP IPres (
                Device = bge0
               Address = "10.10.10.22"
                NetMask = "255.255.255.0"
        )
        Mount Mountres (
                MountPoint = "/nfs/testnfs"
                BlockDevice = "/dev/vx/dsk/testdg/testnfshome"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
        )
        NFS NFSres (
                Nservers = 16
        )

        NFSLock NFSLockres (
                PathName = "/nfs/testnfs"
        )
        NIC NICres (
                Device = bge0
        )
        Share Shareres (
                PathName = "/nfs/testnfs"
                Options = "-o rw -d \"test home dirs\""
        )



        // IPres requires Shareres
        IPres requires NICres
        nfsVOL requires nfsDG
        Mountres requires nfsVOL
        NFSLockres requires Mountres
        Shareres requires NFSLockres
        Shareres requires NFSres

        // resource dependency tree
        //
        // group NFSgrp
        // {
        // IP IPres
        //      {
        //      NIC NICres
        //      Share Shareres
        //          {
        //          NFSLock NFSLockres
        //              {
        //              Mount Mountres
        //                  {
        //                  Volume nfsVOL
        //                      {
        //                      DG nfsDG
        //                      }
        //                  }
        //              }
        //          NFS NFSres
        //          }
        //       }
        // }

Solaris 8 Zone -- etude/BrandZ trial run

2008-01-10T16:07:00.000-06:00

Solaris8 BrandZ pre-requisites

Requires kernel patch 127111-05 (or latest version) for sparc. Find all dependencies and fulfill then (ie patch requirements).

# ls
SUNWs8brandr  SUNWs8brandu  SUNWs8p2v
# pwd
/mypool/software/sol8p2v/s8ma-1_0-rr/Product

# pkgadd -d .

The following packages are available:
  1  SUNWs8brandr     Solaris 8 Migration Assistant: solaris8 brand support (Root)
                      (sparc) 11.10.0,REV=2007.10.08.16.51
  2  SUNWs8brandu     Solaris 8 Migration Assistant: solaris8 brand support (Usr)
                      (sparc) 11.10.0,REV=2007.10.08.16.51
  3  SUNWs8p2v        Solaris 8 p2v Tool
                      (sparc) 11.10.0,REV=2007.10.08.16.51

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

The SUNWs8brandr and SUNWs8brandu packages need to be added to the Solaris 10 Host OS (Global Zone).

Zone configuration

Then configure the Zone --

# zonecfg -z s8-zone
s8-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:s8-zone> create -t SUNWsolaris8
zonecfg:s8-zone> set zonepath=/mypool/zones/s8-zone
zonecfg:s8-zone>
zonecfg:s8-zone> set autoboot=true
zonecfg:s8-zone> add net
zonecfg:s8-zone:net> set address=192.168.99.100
zonecfg:s8-zone:net> set physical=bge1
zonecfg:s8-zone:net> end
zonecfg:s8-zone> add fs
zonecfg:s8-zone:fs> set type=zfs
zonecfg:s8-zone:fs> set dir=/mypool/vol1
zonecfg:s8-zone:fs> end
special not specified
zonecfg:s8-zone:fs> set special=share/zone/s8-zone
zonecfg:s8-zone:fs> end
zonecfg:s8-zone>

zonecfg:sol8zone> add attr
zonecfg:sol8zone:attr> set name=hostid
zonecfg:sol8zone:attr> set type=string
zonecfg:sol8zone:attr> set value=8325f14d
zonecfg:sol8zone:attr> end
zonecfg:sol8zone> verify
zonecfg:sol8zone> commit
zonecfg:sol8zone> exit
dwailsun:$() # zonecfg -z sol8zone info
zonename: sol8zone
zonepath: /mypool/zones/sol8zone
brand: solaris8
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: shared
fs:
        dir: /mypool/vol1
        special: share/zone/sol8zone
        raw not specified
        type: zfs
        options: []
net:
        address: 192.168.99.100
        physical: bge1
attr:
        name: hostid
        type: string
        value: 8325f14d
dwailsun:$() # zonecfg -z sol8zone info attr
attr:
        name: hostid
        type: string
        value: 8325f14d
dwailsun:$() #

Install the zone

dwailsun:$() # zonecfg -z sol8zone export > /var/tmp/safe/sol8zone.config
dwailsun:$(safe) # zoneadm -z s8-zone install -u -a /mypool/software/sol8p2v/solaris8-image.flar
could not verify fs /mypool/vol1: could not access zfs dataset 'share/zone/s8-zone'
zoneadm: zone s8-zone failed to verify

dwailsun:$(safe) # zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
mypool               3.75G  15.4G  39.3K  /mypool
mypool/software      3.22G  6.78G  3.22G  /mypool/software
mypool/vol1          66.6K  5.00G  34.0K  /mypool/vol1
mypool/vol1/s8-zone  32.6K  5.00G  32.6K  /mypool/vol1/s8-zone
mypool/www            544M  3.47G   544M  /mypool/www
mypool/zones         34.0K  5.00G  34.0K  /mypool/zones
dwailsun:$(safe) # zfs set mountpoint=legacy mypool/vol1/s8-zone
dwailsun:$(safe) # zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
mypool               3.75G  15.4G  39.3K  /mypool
mypool/software      3.22G  6.78G  3.22G  /mypool/software
mypool/vol1          65.3K  5.00G  32.6K  /mypool/vol1
mypool/vol1/s8-zone  32.6K  5.00G  32.6K  legacy
mypool/www            544M  3.47G   544M  /mypool/www
mypool/zones         34.0K  5.00G  34.0K  /mypool/zones
dwailsun:$(safe) # zoneadm -z s8-zone install -u -a /mypool/software/sol8p2v/solaris8-image.flar
      Log File: /var/tmp/s8-zone.install.987.log
        Source: /mypool/software/sol8p2v/solaris8-image.flar
    Installing: This may take several minutes...
Postprocessing: This may take several minutes...

        Result: Installation completed successfully.
      Log File: /mypool/zones/sol8zone/root/var/log/s8-zone.install.987.log

Solaris8 P2V

Run sol8-p2v --

dwailsun:$(safe) # /usr/lib/brand/solaris8/s8_p2v s8-zone
[Fri Dec 28 12:36:01 PST 2007]         S20_apply_patches:  Unpacking patch:  109 147-44
[Fri Dec 28 12:36:01 PST 2007]         S20_apply_patches: Installing patch:  109 147-44

Checking installed patches...
Patch 109147-44 has already been applied.
See patchadd(1M) for instructions.

Patchadd is terminating.
[Fri Dec 28 12:36:09 PST 2007]         S20_apply_patches:  Unpacking patch:  111 023-03
[Fri Dec 28 12:36:09 PST 2007]         S20_apply_patches: Installing patch:  111 023-03

Checking installed patches...
Patch 111023-03 has already been applied.
See patchadd(1M) for instructions.

Patchadd is terminating.
[Fri Dec 28 12:36:11 PST 2007]         S20_apply_patches:  Unpacking patch:  111 431-01
[Fri Dec 28 12:36:11 PST 2007]         S20_apply_patches: Installing patch:  111 431-01

Checking installed patches...
This patch is obsoleted by patch 108993-67 which has already
been applied to this system.

Patchadd is terminating.
[Fri Dec 28 12:36:13 PST 2007]         S20_apply_patches:  Unpacking patch:  112 605-04
[Fri Dec 28 12:36:13 PST 2007]         S20_apply_patches: Installing patch:  112 605-04

Checking installed patches...
This patch is obsoleted by patch 108993-67 which has already
been applied to this system.

Patchadd is terminating.
[Fri Dec 28 12:36:15 PST 2007]         S20_apply_patches:  Unpacking patch:  112 050-04
[Fri Dec 28 12:36:15 PST 2007]         S20_apply_patches: Installing patch:  112 050-04

Checking installed patches...
Patch 112050-04 has already been applied.
See patchadd(1M) for instructions.

Patchadd is terminating.
[Fri Dec 28 12:36:17 PST 2007]         S20_apply_patches:  Unpacking patch:  109 221-01
[Fri Dec 28 12:36:17 PST 2007]         S20_apply_patches: Installing patch:  109 221-01

Checking installed patches...
This patch is obsoleted by patch 109318-39 which has already
been applied to this system.

Patchadd is terminating.
dwailsun:$(safe) #

dwailsun:$(safe) # zoneadm -z s8-zone boot
dwailsun:$(safe) # zoneadm list -v
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   3 s8-zone          running    /mypool/zones/sol8zone         solaris8 shared
dwailsun:$(safe) # zlogin -C s8-zone
[Connected to zone 's8-zone' console]


You did not enter a selection.
What type of terminal are you using?
 1) ANSI Standard CRT
 2) DEC VT52
 3) DEC VT100
 4) Heathkit 19
 5) Lear Siegler ADM31
 6) PC Console
 7) Sun Command Tool
 8) Sun Workstation
 9) Televideo 910
 10) Televideo 925
 11) Wyse Model 50
 12) X Terminal Emulator (xterms)
 13) Other
Type the number of your choice and press Return: 12
Configuring network interface addresses: bge1.
RPC: Timed out

Then it goes through and does the sysidcfg bit...

System identification is completed.

rebooting system due to change(s) in /etc/default/init

Dec 28 12:41:25 rpcbind: rpcbind terminating on signal.
System identification is completed.


[NOTICE: Zone rebooting]

SunOS Release 5.8 Version Generic_Virtual 64-bit
Copyright 1983-2000 Sun Microsystems, Inc.  All rights reserved

Hostname: sol8virt
The system is coming up.  Please wait.
starting rpc services: rpcbind done.
syslog service starting.
Print services started.
Dec 28 14:41:37 sol8virt sendmail[4102]: My unqualified host name (sol8virt) unknown; sleeping for retry
The system is ready.

sol8virt console login:

# uname -a
SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70
# exit

[Connection to zone 's8-zone' pts/5 closed]
dwailsun:$(safe) # uname -a
SunOS dwailsun 5.10 Generic_127111-05 sun4u sparc SUNW,A70
dwailsun:$(safe) # zlogin s8-zone
[Connected to zone 's8-zone' pts/5]
Last login: Fri Dec 28 14:43:35 on pts/5
Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004
# uname -a
SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70
#
# cat /etc/release
                       Solaris 8 2/04 s28s_hw4wos_05a SPARC
           Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.
                            Assembled 08 January 2004
#

Think of a optimal battery of tests that can help us determine whether this virtualized solaris 8 is a viable platform for servers that cannot be migrated....

Adding packages -- pkgadd works

# uname -a
SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70
# pkginfo|grep -i smc
application SMCgcc         gcc
application SMCliconv      libiconv
application SMClintl       libintl
application SMCosh471      openssh
application SMCossl        openssl
application SMCzlib        zlib

Set up sshd after adding these packages, complete with start up scripts, sshd privsep user id in the system accounts files (passwd and shadow).

# /etc/init.d/sshd start
Could not load host key: /usr/local/etc/ssh_host_key
Could not load host key: /usr/local/etc/ssh_host_dsa_key
Disabling protocol version 1. Could not load host key
# ps -ef|grep sshd
    root  5086  4609  0 15:18:13 ?        0:00 /usr/local/sbin/sshd

Installing Oracle 8i

Setting up Oracle 8i was a breeze. Simply dumped the 2 cds of Oracle 8i 64-bit installation media onto a solaris8 zone visible fileystem and ran the runInstaller with all defaults and the demo database (scott/tiger) getting created as the end step.

Make sure to copy the media to local disk when installing inside the zone. The reason being, even though the cdrom can be exported to the local zone from the Global zone this way --

add fs
set dir=/mnt
set special=/cdrom
set type=lofs
add options ro
add options nodevices
end

We would have issues ejecting and inserting new cdroms, etc.

dwailsun:$() # ssh oracle@sol8virt
oracle@sol8virt's password:
Last login: Thu Jan  3 11:27:25 2008 from 10.119.10.4
Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004
Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004
$ ps -ef|grep ora
  oracle 22608 22152  0 11:25:48 ?        0:00 ora_reco_brandz
  oracle 22610 22152  0 11:25:48 ?        0:00 ora_snp0_brandz
  oracle 22626 22152  0 11:26:55 ?        0:00 /export/shared/oracle/OraHome1/bin/tnslsnr LISTENER -inherit
  oracle 22614 22152  0 11:25:48 ?        0:00 ora_snp2_brandz
  oracle 22687 22685  0 11:56:04 ?        0:00 /usr/local/sbin/sshd -R
  oracle 22695 22689  0 11:56:09 pts/6    0:00 grep ora
  oracle 22689 22687  0 11:56:04 pts/6    0:00 -ksh
  oracle 22604 22152  4 11:25:48 ?        1:04 ora_ckpt_brandz
  oracle 22600 22152  0 11:25:48 ?        0:00 ora_dbw0_brandz
  oracle 22598 22152  0 11:25:48 ?        0:00 ora_pmon_brandz
  oracle 22620 22152  0 11:25:48 ?        0:00 ora_d000_brandz
  oracle 22602 22152  0 11:25:48 ?        0:02 ora_lgwr_brandz
  oracle 22618 22152  0 11:25:48 ?        0:00 ora_s000_brandz
  oracle 22616 22152  0 11:25:48 ?        0:00 ora_snp3_brandz
  oracle 22612 22152  0 11:25:48 ?        0:00 ora_snp1_brandz
  oracle 22606 22152  0 11:25:48 ?        0:00 ora_smon_brandz
$

Using a Wiki for quick Documentation

2007-10-10T13:26:00.000-06:00

A few thoughts on using new(er) technology to manage/maintain and collaborate docs and projects.

In my previous gig with a major Pharmaceutical company, I had taken on the onus of documenting the environment at the onset. That way I could learn the "whole picture" while creating good reference material. But the ordinary way of writing was both hard and cumbersome.

1) Old hacks and tricks in vi-edited text files on my workstation were hard to manage and were hard to distribute publicly.
2) Environment design, etc were pain-stakingly created with visio and took hours.
3) Word is a lousy product to use for technical documentation, imho.

So decided to try out a bunch of technologies. Set up two content management engines (both FOSS -- Joomla! and Drupal) that were basically LAMP apps (Linux, Apache, MySQL, PHP) and were very powerful. Joomla! caught my eye due to the flexibility and out-of-the-box toolset it provided. Drupal was minimalistic and needed more effort to "mold". I went on and created a couple of public-consumption web sites using Joomla! (http://www.medhajournal.com and http://www.invadingthesacred.com).

However, the most powerful tool that struck a optimistic chord in mine and the rest of the teams' heart was a minimalist tool called Moinmoin (http://moinmoin.wikiwikiweb.de) that ran on a python-based engine, maintained the content without any RDBMS backing and was a breeze to setup and use.

With excellent markup tags, it has made documentation a breeze and we actually ended up creating a lot of docs using this tool and managing some multi-million dollar projects (infrastructure side) as well.

Of course there were the PM-drones who ran Project, GANTT charts, etc, and they have their own rightful place in the hierarchy of things in the PM framework. But to make a dreary task (that most admins fear) surprisingly easy, the Wiki was the perfect tool.

Thoughts on Virtualization

2007-08-22T13:55:00.000-06:00

I am a regular of the ZDNet blog by Paul Murphy and thought I'd add to his thoughts on Virtualization and all the brouhaha that's going on these days --

Virtualization? uh huh… by ZDNet's Paul Murphy -- Virtualization is popular because it was popular - and not because there's a practical reason to do it.

The most interesting thing I discovered in the process of working on a "high-visibility" project (ERP solution) is that most mgt-types don't understand what Virtualization has to offer. Someone high up (high-up enough I guess) decides that Virtualization is the answer to all evils that haunt a modern datacenter. The claims are that --

Virtualization reduces server sprawl
Virtualization reduces power and cooling footprints
It empowers the IT support organization to be agile (read build more boxes fast) and really support a dynamic business (with lots of development type activities going on)
It is a cure for many problems..blah blah

But when you look at what you're saving on the standard UNIX platforms (except Sun), the costs amount to something exorbitant. I won't name the vendor, but it charges for everything starting from it's multi-pathing software to Resource Mgt software to Virtualization, and they charge by the core.

Soon you start thinking, does this really buy me the cost savings by reducing server-sprawl?
Then the vendor will say, "Why look at this as a consolidation platform? Why don't you think about the flexibility you'll get by using this model? Moving workloads around on the fly, etc?"

The problem with that is that Workload management (called SLOs I believe) calls for very detailed and in-depth recording of metrics (what kind of loads are generated by applications, starting by categorizing by application types, etc.

So you first identify the right kinds of metrics to track. The collect the data for a reasonable period of time (say 3-4 months). Then, only after munging all that data, is it possible to say with any authority that a certain amount of resources are required for a particular workload (and build a system that can manage those resource requirements on the fly). T

his entire process might take about 1 year (from start to finish) before being a viable option (some shops I've been in are better equipped to do this kind of measurements than others -- depending on how "modern" the IT organization usually is -- does it "REALLY" employ standards such as ITIL or not, etc).

I'd say that something like Sun's container model on the Cool-threads servers would be more appropriate for all the above criteria. Consolidation, Resource management, flexibility, etc.

SRM has been free with Solaris since Solaris 9.
Solaris 10 has the virtualization pieces completely free.
The hardware is cheap(er than the competition's for sure)

Setting up Veritas Cluster server

2007-08-22T11:54:00.000-06:00

Install the VCS Packages after patching the server to appropriate/recommended Patch list.

VCS LICENSE KEY : !@$-@$%-(*&^-$%@-$%%-!

List of VCS Packages:

VRTSappqw VRTSvcs VRTSvcsqw
VRTScscm VRTSvcsag VRTSvcsw
VRTSgab VRTSvcsdc VRTSvlic
VRTSllt VRTSvcsmg VRTSweb
VRTSoraqw VRTSvcsmn VRTSperl VRTSvcsor

edit /etc/llthosts (on both servers - for a 2 node cluster)

0 hostd02
1 hostd03

edit /etc/llttab

set-node hostd03 #here the nodename will change with each host
set-cluster 54 #Set the appropriate cluster ID
link qfe1 /dev/qfe:1 - ether - - #heartbeat 1
link qfe5 /dev/qfe:5 - ether - - #heartbeat 2
link-lowpri qfe0 /dev/qfe:0 - ether - - #Low-pri heartbeat

Edit the /etc/gabtab file with

cat > /etc/gabtab <<EOGAB
gabconfig -c -n 2
EOGAB

#Here the number after the "-n" varies with the number of nodes in cluster

Edit the main.cf (/etc/VRTSvcs/conf/config) to match your reqs

##Only on the first/main server of the Cluster

##Start of main.cf##

include "types.cf"
include "OracleTypes.cf"

cluster OneBill_Prod (
UserNames = { admin = "cDRpdxPmHpzS." }
Administrators = { admin }
CounterInterval = 5
)

system hostd02 (
)

system hostd03 (
)

group network_grp (
SystemList = { hostd02 = 0, hostd03 = 1 }
PrintTree = 0
Parallel = 1
AutoStartList = { hostd02, hostd03 }
)

NIC OneBillv1_nic (
Device = qfe0
NetworkType = ether
)

Phantom OneBillv1_phantom (
)

group oracle_grp (
SystemList = { hostd02 = 0, hostd03 = 1 }
PrintTree = 0
AutoStartList = { hostd02 }
)

DiskGroup orashrdg_dg (
DiskGroup = orashrdg
)

IP OneBillv1_vip (
Device = qfe0
Address = "112.64.90.54"
NetMask = "255.255.255.0"
IfconfigTwice = 1
)

Mount au1_mnt (
MountPoint = "/au1"
BlockDevice = "/dev/vx/dsk/orashrdg/au1"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount bu1_mnt (
MountPoint = "/bu1"
BlockDevice = "/dev/vx/dsk/orashrdg/bu1"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u01_mnt (
MountPoint = "/au1"
BlockDevice = "/dev/vx/dsk/orashrdg/au1"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount bu1_mnt (
MountPoint = "/bu1"
BlockDevice = "/dev/vx/dsk/orashrdg/bu1"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u01_mnt (
MountPoint = "/u01"
BlockDevice = "/dev/vx/dsk/orashrdg/u01"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u02_mnt (
MountPoint = "/u02"
BlockDevice = "/dev/vx/dsk/orashrdg/u02"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u03_mnt (
MountPoint = "/u03"
BlockDevice = "/dev/vx/dsk/orashrdg/u03"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u04_mnt (
MountPoint = "/u04"
BlockDevice = "/dev/vx/dsk/orashrdg/u04"
BlockDevice = "/dev/vx/dsk/orashrdg/u04"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Mount u05_mnt (
MountPoint = "/u05"
BlockDevice = "/dev/vx/dsk/orashrdg/u05"
FSType = vxfs
MountOpt = rw
FsckOpt = "-y"
)

Proxy OneBillv1_proxy (
TargetResName = OneBillv1_nic
)

Volume au1_vol (
Volume = au1
DiskGroup = orashrdg
)

Volume bu1_vol (
Volume = bu1
DiskGroup = orashrdg
)

Volume u01_vol (
Volume = u01
DiskGroup = orashrdg
)

Volume u02_vol (
Volume = u02
DiskGroup = orashrdg
)

Volume u03_vol (
Volume = u03
DiskGroup = orashrdg
)

Volume u04_vol (
Volume = u04
DiskGroup = orashrdg
)

Volume u05_vol (
Volume = u05
DiskGroup = orashrdg
)

OneBillv1_vip requires OneBillv1_proxy
au1_mnt requires au1_vol
au1_mnt requires orashrdg_dg
bu1_mnt requires bu1_vol
bu1_vol requires orashrdg_dg
u01_mnt requires u01_vol
u01_vol requires orashrdg_dg
u02_mnt requires u02_vol
u02_vol requires orashrdg_dg
u03_mnt requires u03_vol
u03_vol requires orashrdg_dg
u04_mnt requires u04_vol
u04_vol requires orashrdg_dg
u05_mnt requires u05_vol
u05_vol requires orashrdg_dg

##End of main.cf##

Copy OracleTypes.cf, etc to the config directory

From /etc/VRTSvcs/conf/config run

opt/VRTSvcs/bin/hacf -verify .

###(Fix errors as you get them)

Setting up GAB and LLT

sbin/gabconfig -U
/sbin/lltconfig -U
/sbin/lltconfig -c
/sbin/gabconfig -c -n 2
/sbin/lltconfig -a list

##Make sure Filesystems (Shared Filesystems) are commented out of the /etc/vfstab file

#Make sure each node in the cluster has the host/IP information of every other in it's local hosts file#

Reboot the servers, bringing up the main server/node up first

On each node of the cluster

/sbin/vxlicinst -k <KEY>
/opt/VRTSvcs/bin/hastop -local -force
/opt/VRTSvcs/bin/hastart

Create Mount points on all nodes for Shared Filesystems

for i in au1 bu1 u01 u02 u03 u04 u05
do
if [ ! -d $i ]; then
mkdir $i
fi
done

Test failovers by bringing down resources and checking the failover

Sun Cluster Cheat Sheet -- 4

2007-07-18T14:53:00.000-06:00

Data Services in the Cluster

HAStoragePlus helps configure a local filesystem into a highly available one. It provides following capabilities:

additional filesystem checks
mounts and unmounts
enables Sun cluster to failover local file systems (to failover, local file system must reside on global dgs with affinity switchovers enabled)

Data Service Agent -- is a specially written software that allows a data service in a cluster to operate properly.

Data Service Agent (or Agent) does the following to a standard application:

stop/start application
monitor faults
validate configuration
provides a registration information file that allows Sun Cluster to store all the info about the methods.

Sun Cluster 2.x runs Fault Monitoring components on failover node, and can initiate a takeover. On Cluster 3.x software, it is not allowed. Monitor can either monitor to restart or failover on primary (active host) node.

Failover resource groups:

Logical host resource -- SUNW.Logicalhostname Data Storage Resource -- SUNW.HAStoragePlus NFS resource -- SUNW.nfs

Shutdown a resource group:

scswitch -F -g

Turn on a resourec group:

scswitch -Z -g

Switch a failover group over to another node:

scswitch -z -g  -h

Restart a resource group:

scswitch -R -h  -g

Evacuate all resources and rgs from a node:

scswitch -S -h node

Disable a res and it's fault monitor:

scswitch -n -j

Enable a resource and it's fault monitor:

scswitch -e -j

Clear the STOP_FAILED flag:

scswitch -c -j  -h  -f STOP_FAILED

How to add a diskgroup and voluem to Cluster configuration

1. Create the disk group and volume.

2. Register the local disk group with the cluster.

        root@aesnsra1:../ # scconf -a -D type=vxvm,name=patroldg2,nodelist=aesnsra2
        root@aesnsra2:../ # scswitch -z -h aesnsra2 -D patroldg2

3. Create your file system.

4. Update /etc/vfstab to change '-' boot options

example:

        /dev/vx/dsk/patroldg2/patroldg02 /dev/vx/rdsk/patroldg2/patroldg02 /patrol02 vxfs 3 no suid

5. Set up a resource group with a HAStoragePlus resource for local filesystem:

        root@aesnsra2:../ # scrgadm -a -g aescib1-hastp-rg -h aescib1
        root@aesnsra2:../ # scrgadm -a -g aescib1-hastp-rg -j sapmntdg01-rs -t SUNW.HAStoragePlus -x FilesystemMountPoints=/sapmnt

6. Bring the resource group online which will mount the specified filesystem:

        root@aesnsra2:../ # scswitch -Z -g hastp-aesnsra2-rg

7. Enable resource

        root@aesnsra2:../# scswitch -e -j osdumps-dev-rs

Optional step:

8. reboot and test.

Fault monitor operations

Disable the fault monitor for a resource:

scswitch -n -M -j

Enable the Fault monitor for a resource:

scswitch -e  -M -j

scstat -g       #shows status of all resource groups

Using scrgadm to register and configure Data service software

eg:

scrgadm -a -t SUNW.nfs

scrgadm -a -t SUNW.HAStoragePlus
scrgadm -p

Create a fail over res:

scrgadm -a -f nfs-rg -h node1,node2 \
-y Pathprefix=/global/nfs/admin

Add logical host name res to rg:

scrgadm -a -L -g nfs-rg -l clustername-nfs

Create a HAStoragePlus res:

scrgadm -a -j nfs-stor -g nfs-rg \
-t SUNW.HAStoragePlus \
-x FilesystemMountpoints=/global/nfs -x AffinityOn=True

Create SUNW.nfs resource:

scrgadm -a -j nfs-res -g nfs-rg \
-t SUNW.nfs -y Resource_dependencies=nfs-stor

Print the various resource/resource group dependencies via scrgadm:

scrgadm -pvv|grep -i depend     #And then parse this output

Enable res and res monitors, manage rg and switch rg to online state:

scswitch -Z -f nfs-rg

scstat -g

Show current RG configuration:

scrgadm -p[v[v]] [ -t resource_type_name ] [ -g resgrpname ] \
[ -j resname ]

Resizing a VxVM/VxfS vol/fs under sun cluster

# vxassist -g aesnfsp growby saptrans 5g

# scconf -c -D name=aesnfsp,sync

root@aesrva1:../ # vxprint -g aesnfsp -v saptrans
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  saptrans     fsgen        ENABLED  188743680 -       ACTIVE   -       -

root@aesrva1:../ # fsadm -F vxfs -b 188743680 /saptrans
UX:vxfs fsadm: INFO: /dev/vx/rdsk/aesnfsp/saptrans is currently 178257920 sector
s - size will be increased

# root@aesrva1:../ # scconf -c -D name=aesnfsp,sync

Command Quick Reference

scstat

scconf

scrgadm

scha_

scdidadm

Sun Terminal Concentrator (Annex NTS)

Enable setup mode by pressing TC test button until TC power indicator starts to blink rapidly, then release the button and press it briefly.

On entering the Setup mode, a "monitor:" prompt is displayed.

Set up IP address using:

monitor::addr

Setting up Load source:

monitor::seq

Specifying image:

monitor::image

Telnet into the TC IP address:

enter "cli"

Elevate to privileged acct using "su"

Run "admin" at the TC OS prompt:

get admin: subprompt:

show port=1 type mode
set port= type  mode  #Choose various options
quit (to exit the boot prompt)
boot

Sun cluster Cheat Sheet -- 3

2007-07-18T14:50:00.000-06:00

Displays existing DG resources in the Cluster

scstat -D

Registering VxVM DGs

scconf -a -D type=vxvm,name=. \
nodelist=:, \
preferenced=true,failback=enabled

nodelist should contain only nodes that are physically connected to the disks of that dg.
preferenced=true/false affects whether nodelist indiciates an order of failover preference. On a two-node cluster, this options is only meaningful if failback is enabled.
failback=disabled/enabled affects whether a preferred node "takes back" it's device group when it joins the cluster. The default value is disabled. When faileback is disabled, preferenced is set to false. If it is enabled, preferenced also must be set to true.

Moving DGs across nodes of a cluster

When VxVM dgs are registered as Sun Cluster resources, NEVER USE vxdg import/deport commands to change ownership (node-wise) of the dgs. This will cause SC to treat dg as failed resource.

Use the following command instead:

# scswitch -z -D  -h

Resyncing Device Groups

scconf -c -D name=,sync

Changing DG configuration

scconf -c -D name=,preferenced=,failback=

Maintenance mode

scswitch -m -D

NOTE: all volumes in the dg must be unopened or unmounted (not being used) in order to do that.

To come back out of maintenance mode

scswitch -z -D  -h

Repairing DID device database after replacing JBOD disks

'Make sure you know which disk to update ...'

scdidadm -l c1t1d0

returns node1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d7

scdidadm -l d7

returns node1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d7

Then use following cmds to update and verify the DID info:

scdidadm -R d7
scdidadm -l -o diskid d7

returns a large string with disk id.

Replacing a failed disk in a A5200 Array (similar concept with other FC disk arrays)

vxdisk list - get the failed disk name

vxprint -g dgname -- determine state of the volume(s) that might be affected

On the hosting node, replace the failed disk:

luxadm remove enclosure,position
luxadm insert enclosure,position

On either node of the cluster (that hosts the dg):

scdidadm -l c#t#d#
scdidadm -R d#

On the hosting node:

vxdctl enable

vxdiskadm (replace failed disk in vxvm)

vxprint -g 
vxtask list     #ensure that resyncing is completed

Remove any relocated submirrors/plexes (if hot-relocation had to move something out of the way):

vxunreloc repaired-diskname

Solaris Vol Mgr (SDS) in Sun Clustered Env

Preferred method of using Soft partitions is to use single slices to create mirrors and then create volumes (soft partitions) from that (kind of similar to VxVM public region in an initialized disk).

Shared Disksets and Local Disksets

Only disks that are physically located in the multi-ported storage will be members of shared disksets. Only disks that are in the same diskset operate as a unit; they can be used together to build mirrored volumes, and primary ownership of the diskset transfers as a while from node to node.

Boot disks are the local disksets. This is a pre-requisite in order to have shared disksets.

Replica management

Add local replicas manually.
Put local state db replicas on slice 7 of disks (as a convention) in order to maintain uniformity. Shared disksets have to have replicas on slice 7.
Spread local replicas evenly across disks and controllers.
Support for Shared disksets is provided by Pkg SUNWmdm

Modifying /kernel/drv/md.conf

nmd == max num of volumes (default 128)
md_nsets == max is 32, default 4.

Creating shared disksets and mediators

scdidadm -l c1t3d0

-- returns d17 as DID device

scdidadm -l d17
metaset -s  -a -h    # creates metaset
metaset -s  -a -m    # creates mediator
metaset -s  -s /dev/did/rdsk/d9 /dev/did/rdsk/d17
metaset # returns values
metadb -s 
medstat -s  (reports mediator status)

Remaining syntax vis-a-vis Sun Cluster is identical to that for VxVM.

IPMP and sun cluster

IPMP is cluster un-aware. To work around that, Sun Cluster uses Cluster-specific public network mgr daemon (pnmd) to integrate IPMP into the cluster.

pmnd daemon has two capabilities:

populate CCR with public network adapter status
facilitate application failover

When pnmd detects all members of a local IPMP group have failed, it consults a file called /var/cluster/run/pnm_callbacks. This file contains entries that would have been created by the activation of Log icalHostname and SharedAddress resources. It is the job of hafoip_ipmp_callback to device whether to migrate resources to another node.

scstat -i       #view IPMP configuration

Sun Cluster Cheat Sheet -- 2

2007-07-18T14:46:00.000-06:00

Sun Cluster Set up

don't mix PCI and SBus SCSI devices

Quorum Device Rules

A quorum device must be available to both nodes in a 2-node cluster
QD info is maintained globally in the CCR db
QD should contain user data
Max and optimal number of votes contributed by QDs must be N -1
- (where N == number of nodes in the cluster)
If # of QDs >= # of nodes, Cluster cannot come up easily if there are too
- many failed/errored QDs.
QDs are not required in clusters with more than 2 nodes, but recommended
- for higher cluster availability.
QDs are manually configured after Sun Cluster s/w installation is done.
QDs are configured using DID devices

Quorum Math and Consequences

A running cluster is always aware of (Math):
- --> Total possible Q votes (number of nodes + disk quorum votes) --> Total present Q votes (number of booted nodes + available QD votes) --> Total needed Q votes ( >= 50% of possible votes)
Consequences:
- --> Node that cannot find adequate Q votes will freeze, waiting for
  - other nodes to join the cluster
  --> Node that is booted in the cluster but can no longer find the
  - needed number of votes kernel panics

installmode Flag -- allows for cluster nodes to be rebooted after/during initial

installation without causing the other (active) node(s) to panic.

Cluster status

# Reporting the clsuter membership and quorum vote information

# /usr/cluster/bin/scstat -q

Verifying cluster configuration info

# scconf -p

Run scsetup to correct any configuration mistakes and/or to:

* add or remove quorum disks * add, remove, enable, disable cluster transport components * register/unregister vxVM dgs * add/remove node access from a VxVM dg * change clsuter private host names * change cluster name

Shutting down cluster on all nodes:

# scshutdown -y g 15

# scstat (verifies cluster status)

Cluster Daemons

lahirdx@aescib1:/home/../lahirdx > ps -ef|grep cluster|grep -v grep
    root     4     0  0   May 07 ?       352:39 cluster
    root   111     1  0   May 07 ?        0:00 /usr/cluster/lib/sc/qd_userd
    root   120     1  0   May 07 ?        0:00 /usr/cluster/lib/sc/failfastd
    root   123     1  0   May 07 ?        0:00 /usr/cluster/lib/sc/clexecd
    root   124   123  0   May 07 ?        0:00 /usr/cluster/lib/sc/clexecd
    root  1183     1  0   May 07 ?       46:45 /usr/cluster/lib/sc/rgmd
    root  1154     1  0   May 07 ?        0:07 /usr/cluster/lib/sc/rpc.fed
    root  1125     1  0   May 07 ?       23:49 /usr/cluster/lib/sc/sparcv9/rpc.pmfd
    root  1153     1  0   May 07 ?        0:03 /usr/cluster/lib/sc/cl_eventd
    root  1152     1  0   May 07 ?        0:04 /usr/cluster/lib/sc/cl_eventlogd
    root  1336     1  0   May 07 ?        2:17 /var/cluster/spm/bin/scguieventd -d

    root  1174     1  0   May 07 ?        0:03 /usr/cluster/bin/pnmd
    root  1330     1  0   May 07 ?        0:01 /usr/cluster/lib/sc/scdpmd
    root  1339     1  0   May 07 ?        0:00 /usr/cluster/lib/sc/cl_ccrad

FF Panic rule -- failfast will shutdown the node (panic the kernel) if specified daemon is not restarted within 30s.
cluster -- system proc created by the kernel to encap kernel threads that make up the core kernel range of operatiosn. It directly panics the kernel if it's sent a KILL signal (SIGKILL). Other signals have no effect.
clexecd -- this is used by cluster kernel threads to execure userland cmds (such as run_reserve and dofsck cmds). It is also used to run cluster cmds remotely (eg: scshutdown).A failfast driver panics the kernel if this daemon is killed and not restarted in 30s.
cl_eventd -- This daemon registers and forwards cluster events s(eg: nodes entering and leaving the cluster). With a min of SC 3.1 10/03, user apps can register themselves to receive cluster events. The daemon automatically get's respawned by rpc.pmfd if it is killed.
rgmd -- This is the resource group mgr, which manages the state of all cluster-unaware applications. A failfast driver panics the kernel if this daemon is killed by not started in 30s.
rpc.fed -- this is the "fork-and-exec" daemon - -which handles reqs from rgmd to spawn methods for specific data services. failfast will hose the box if this is killed and not restarted in 30s.
scguieventd -- this daemon processes cluster events for the SunPlex Mgr GUI, so that the display can be updated in real time. It's not automatically started if it stops. If you are having trouble with SunPlex Mgr, might have to restart the daemon or reboot the specific node.
rpc.pmfd -- This is the process monitoring facility. It is i used as a general mech to initiate restarts and failure action scripts for some cluster f/w daemons, and for most app daemons and app fault monitors. FF panic rule holds good.
pnmd -- This is the public Network mgt daemon, and manages n/w status info received from the local IPMP (in.mpathd) running on each node in the cluster. It is automatically restarted by rpc.pmfd if it dies.
scdpmd -- multi-threaded DPM daemon runs on each node. DPM daemon is started by an rc script when a node boots. It montiors the availability of logical path that is visible thru various multipath drivers (MPxIO), HDLM, Powerpath, etc. Automatically restarted by rpc.pmfd if it dies.

Validating basic cluster config

The sccheck (/usr/cluster/bin/sccheck) cmd validates the cluster configuration:
/var/cluster/sccheck is the repository where it stores the reports generated.

Disk Path Monitoring

scdpm -p all:all #prints all disk paths in the clsuter and their status
scinstall -pv #check the clsuter installation status -- package revisions, patches applied, etc
Cluster release file: /etc/cluster/release

Shutting down cluster

scshutdown -y -g 30

Booting nodes in non-cluster mode

```
        boot -x
    
```

Placing node in maintenance mode

scconf -c -q node=,maintstate

Reset the maintenance mode by rebooting the node or running

scconf -c -q reset By placing a node in a cluster in maintenance mode, we reduce the number of reqd. quorum votes and ensuring that cluster operation is not disrupted as a result thereof).
Sunplex manager is available on https::3000

VxVM Rootdg requirements for Sun Cluster

* vxio major number has to be identical on all nodes of the cluster (check for vxio entry in /etc/name_to_major)

* vxvm installed on all nodes physically connected to shared storage -- on non-storage nodes, yvxvm can be used to encapsulate and mirror the boot disk. If not using VxVM on a non-storage node, use SVM. All is required in such a case is the vxio major number be identical to all other nodes of the cluster (add an entry in /etc/name_to_major file).

* VxVM license is reqd. on all nodes not connected to a A5x00 storedge array.

* Std rootdg created on all nodes where vxVM is installed. Options to initialize rootdg on each node are:

--> Encap boot disk so it can be mirroered. Preserve all data and creating volumes inside rootdg to encap /global/.devices/node@# --> If disk has more than 5 slices on it, it cannot be encap'ed. --> Initialize other local disks into rootdg.

* Unique volume name and minor number across the nodes for the /global/.devices/node@# file system if the boot disk is encap'ed -- the /global/.devices/node@# fs must be on devices with a unique name on each node, because it's mounted on each node for the same reason. The normal Solaris OS /etc/mnttab logic redates global fs and still demands that each device have a unique major/minor number. VxVM doesn't support changing minor numbers of individual volumes. The entire disk group has to be re-minored.

Use the following command:

#vxdg  [ -g diskgroup ] [ -f ]  reminor
           [diskgroup ]  new-base-minor

From the vxdg man pages:

     reminor   Changes the base minor number for  a  disk  group,
               and  renumbers  all devices in the disk group to a
               range starting at that number.  If the device  for
               a  volume  is  open,  then  the  old device number
               remains in effect until the system is rebooted  or
               until  the disk group is deported and re-imported.
               Also, if you close an open volume, then  the  user
               can   execute  vxdg reminor  again  to  cause  the
               renumbering to take effect  without  rebooting  or
               reimporting.

               A new device number may also overlap with  a  tem-
               porary  renumbering for a volume device. This also
               requires a reboot or reimport for the  new  device
               numbering to take effect.  A temporary renumbering
               can happen in the following situations:  when  two
               volumes  (for  example,  volumes  in two different
               disk groups) share the same  permanently  assigned
               device number, in which case one of the volumes is
               renumbered temporarily to use an alternate  device
               number; or when the persistent device number for a
               volume was changed, but the active  device  number
               could  not be changed to match.  The active number
               may be left unchanged after  a  persistent  device
               number change either because the volume device was
               open, or because the new number was in use as  the
               active device number for another volume.

               vxdg fails if you try to use a  range  of  numbers
               that  is  currently  in use as a persistent (not a
               temporary) device number.  You can  force  use  of
               the  number range with use of the -f option.  With
               -f, some device renumberings may not  take  effect
               until  a  reboot or a re-import (just as with open
               volumes).  Also, if you force volumes in two  disk
               groups  to use the same device number, then one of
               the volumes is temporarily renumbered on the  next
               reboot.   Which volume device is renumbered should
               be considered random, except that  device  number-
               ings in the rootdg disk group take precedence over
               all others.
               The -f option should be used  only  when  swapping
               the  device number ranges used by two or more disk
               groups.  To swap the number ranges  for  two  disk
               groups,  you  would  use  -f  when renumbering the
               first disk group to use the range  of  the  second
               disk  group.  Renumbering the second disk group to
               the first range does not require the use of -f.

Sun Cluster does not work with Veritas DMP. DMP can be disabled before installing the software by putting in dummy symlinks, etc.
scvxinstall -- is a shell script that automates VxVM installation in a Sun Clustered env.
scvxinstall automates the following things:
- tries to disable DMP (vxdmp)
- installs correct cluster package
- automatically negotiates a vxio major number and properly edits /etc/name_to_major
- automates rootdg initialization process and encapsulates boot disk
  - --> gives different device names for the /global/.devices/node@# volumes on each side --> edites teh vfstab properly for this same volume. The problem is this particular line as DID device on it, and VxVM doesn't understand DID devices. --> installs a script to "reminor" the rootdg on the reboot --> reboots the node so that VxVM operates properly.

Sun Cluster Cheat Sheet -- 1

2007-07-18T14:13:00.000-06:00

Cluster Configuration Repository (CCR)

/etc/cluster/ccr (directory)

Important Files

/etc/cluster/ccr/infrastructure

Global Services

One node is to specific global services. All other nodes communicate with the global services (devices, filesystems) via the Cluster interconnect.

Global Naming (DID Devices)

- /dev/did/dsk and /dev/did/rdsk

DID used only for naming globally -- not access
DID device names cannot/are not used in VxVM
DID device names are used in Sun/Solaris Volume Manager

Global Devices

provide global access to devices irrespective of there physical location.
Most commonly SDS/SVM/VxVM devices are used as global devices. LVM software is unaware of the implementation of global nature on these devices.

/global/.devices/node@nodeID

nodeID is an integer representing the node in the cluster

Global Filesystems

# mount -o global, logging /dev/vx/dsk/nfsdg/vol01 /global/nfs

or edit the /etc/vfstab file to contain the following:

        /dev/vx/dsk/nfsdg/vol01    /dev/vx/rdsk/nfsdg/vol01    /global/nfs    ufs    2    yes    global,logging

Global Filesystem is also known as (aka) Cluster Filesystem (CFS) or PxFS (Proxy File system)

NOTE: Local failover filesystems (ie. directly attached to a storage device) cannot be used for scalable services -- one would have to use global filesystems for it.

Console Software

SUNWccon There are three wariants of the cluster console software:
- cconsole ( access the node consoles through the TC or other remote console access method )
- crlogin (uses rlogin as underlying transport)
- ctelnet (uses telnet as underlying transport)
  /opt/SUNWcluster/bin/ &

Cluster Control Panel

/opt/SUNWcluster/bin/ccp [ clustername ] &

All necessary info for cluster admin is stored in the following two files:

--> /etc/clusters Eg: sc-cluster sc-node1 sc-node2
--> /etc/serialports

        sc-node1 sc-tc 5002             # Connect via TCP port on TC
        sc-node2 sc-tc 5003
        sc-10knode1 sc10k-ssp 23        # connect via E10K SSP
        sc-10knode2 sc10k-ssp 23
        sc-15knode1 sf15k-mainsc 23     # Connect via 15K Main SC
        e250node1 RSCIPnode1 23         # Connect via LAN RSC on a E250
        node1 sc-tp-ws 23               # Connect via a tip launchpad
        sf1_node1 sf1_mainsc 5001       # Connect via passthru on midframe

Playing with ideas for an Enterprise File Transfer mechanism

2007-04-03T10:16:00.000-06:00

Overview

This is a post of an old high-level architecture design I'd worked on, to see how Open-source technology might fit into a Financial Organization's (my then-employer's client) Enterprise.

Long-term Goal

To design an Enterprise File Transfer mechanism based on high encryption/compression transport mechanism of Secure Shell (SSH).
Solution should be easy to use, easy to deploy and cost effective.
Solution should be scalable and robust.
Solution should integrate the two major OS platforms (UNIX and Windows) “seamlessly

Technical Solution Specs

Secure FTP solution for the Enterprise
Breakup of the requirements

Centralized “drop box” or “landing zone” type facility
Automated “feeds” type mechanism for further propagation/distribution of data/files
Should we try and integrate a transparent layer of version control that'll maintain audit trails, etc?
Easy client access (preferably web-based)
Additional client access from Windows environment (fat clients as required)
Tight integration with existing authentication mechanism(s) – Active Directory + UNIX logins
Low learning curve (meaning, simple solution)
Fine-grained access control
Ease of administration (centralized user account + privilege management)
traceable usage history (auditable transfer logs, user access logs, user activity logs)
Easily replicable process of deployment/re-deployment

Possible Solutions (commercial and open-source)

Tumbleweed-based Secure transfer mechanism (already investigated and POC'ed)
Combination of Windows SSH/SCP/SFTP servers and clients and the pre-existing OpenSSH servers (on UNIX) with Jscape's SFTP/FTPS applet based web front-end (opensource solution).
Vandyke Software's Vshell software (which gives SSH servers + client command lines for both Windows and UNIX)
SSH Tectia software suite

Other Considerations

Inter-operability of different technologies

The biggest challenge in designing a comprehensive “seamless” Secure FTP solution for any enterprise is in the inter-operability between the disparate platforms. For example, the UNIX servers and the Windows servers need to be able to seamlessly communicate with each other in order for such a solution to be viable.

Integrated authentication mechanism

The UNIX systems can be made to authenticate against an LDAP server or the Microsoft Active Directory. With the introduction of either an LDAP_PAM module for LDAP based authentication or third-party plug-ins (such as Vintella's VAS module) that will “hook up” with AD.

Keeping such an environment in mind, our solution should be designed to be able to automatically adapt to an eventual roll-out of such an authentication system.

This would, as a result, allow for easy, centralized management of user accounts and privileges.

Overview of the Open-technology solution

Web-based SFTP system

This solution is designed around existing OpenSSH server and client software currently running in the Enterprise (UNIX/UNIX-like platforms). Additional software components like Windows commercial SSH product(s) and Jscape Secure FTP applet will be required to realize this design.

Major components:

UNIX (Solaris/Linux platforms)

OpenSSH Server
Apache Web server
Jscape's Secure FTP applet
HTML page to load the SFTP applet

Windows

Commercial SSH/SFTP server
IIS web server
Jscape's Secure FTP applet
HTML page to load the SFTP applet

Application Client-side Requirements

Operating Systems (supported): Windows 98/2000/XP/ME, Linux, Solaris, Mac OS X
Browser: Internet Explorer or Netscape Navigator/Mozilla (gecko-based) browsers
Java VM: Java Plug-in 1.4.2 or higher enabled

NOTE: For Macintosh users, the MRJ (Mac Runtime for Java) does not include the necessary crypto classes required to establish a secure connection. If using MRJ, you will need to install the Sun JCE (Java Cryptography extensions) reference implementation.●

Strengths and Advantages

This solution leverages the existing SSH infrastructure in-house (OpenSSH on UNIX platform already exists in most shops or is available for free downloads) and a cost-effective OpenSource Java Applet based Web interface.
This is an extremely simple solution and with pre-determined “drop zone” servers in place at each location, using the mechanism of key-based authentication and command-line tools, administrators will be able to automate and schedule “feeds” transmissions to requested targets.

Additional Requirements

Design customizable scripting framework using Perl (and/or similar programming language) and XML that would allow for automated feeds to be implemented.

Using the Centralized SSH2-key based root access method to automate network inventory

2007-03-09T12:49:00.000-06:00

Leveraging Centralized SSH2 based trusts to monitor network interface status on solaris servers

Since SSH2 key-based trusts have been established in this landscape (at root level), the automation of a variety of tasks becomes easily achievable. The SSH2 key-based trust ensures secure and encrypted transport mechanism (that reinforces security-oriented approach to system administration). Leveraging tools such as sudo (1m) or powerbroker an additional layer of security and auditability can be added.

Using TLRC and ndd_get.sh to collect Network-related information

The following two scripts can be used to make network interface related metrics collections.

tlrc.pl (Test Login Run Command) is a perl script that reads input from a colon-separated text file (of very specific format) or from the command-line and can execute any command on the remote host(s) specified with STDOUT/STDERR logging, etc.

tlrc.pl (test login run command) --

#!/usr/bin/env perl

use Getopt::Std;
use Net::Ping;

my %Args;

getopts( 'l:i:c:o:n:adT:th', \%Args );

if ( $Args{h} ) {
&printUsage &amp;& exit 0;
}

my $hlist = $Args{i} || "/path/to/inventory.txt";
my $ssh = "/usr/bin/ssh";

my $rsh = "/usr/bin/rsh";
my $p = Net::Ping->new();
my $lid = $Args{l} || "nobody";
my $outfile = $Args{o} || "tlrc.out";
my @shlcmd = $Args{c};
my $conprot = $Args{T} or "ssh";

if ( $conprot = "rsh|remsh|rlogin" ) {
$conprot = "rsh";
}
else {
&printUsage &amp;& exit 1;
}

open( RHL, "< $hlist" ) or die "Unable to open input file $hlist: $! \n"; @rhl = ;
close(RHL);
open( WOF, "|tee $outfile" )
or die "Unable to open output file $outfile for writes: $! \n";
open( WHL, ">> hlist.tlrc" );

if ( $Args{c} ) {
die "Can't execute $Args{c} with the \"-t\" switch \n"
if ( ( $Args{t} or $Args{d} ) );
runCmd(@shlcmd);
}

if ( $Args{d} ) {
die "Can't munge dmesg and run login tests at the same time! \n"
if $Args{t};
&dmesgMunger;
}

if ( $Args{t} ) {
&loginTest;
}

sub printUsage {
print
"Usage: $0 [ -l <> ][ -i ][ -c ][ -n ]|[ -a ]|[ -t ]|[ -h ] \n";
print
"\t -l -- pass the login name you want to use for this session \n
\t -i -- pass the input file (colon-delimited) with list of hosts an
d pingability status \n
\t -c -- quoted Command you want to run remotely \n
\t -n -- comma delimited list of hosts you want to run remote c
ommand specified with \"cmdstring\" on \
\t -a -- specifies all hosts in input file to run remote command specified with
\"cmdstring\" on \
\t -T -- specifies the Connection type -- ssh or rsh \
\t -t -- Optional switch to the -c or -h switches, it will only run testing port
ion of the script \
\t -h -- print this message \n";
}

sub runCmd {

my @cmdstring = @_;

if ( $Args{a} ) {
foreach $line (@rhl) {
next if ( $line =~ m/^#/ );
next if ( $line =~ m/^$/ );
my ( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage )
= split( ':', $line );
chomp( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage );
if ( $pstate == 0 ) {
if ( $canlogin == 0 ) {
if ( $contype == 0 ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $contype == 1 ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
print "Cannot understand connection type! \n";
}
}
else {
print "Cannot log into the server! \n";
}
}
else {
print "$name is unpingable -- can't reach! \n";
}
}
}
elsif ( $Args{n} ) {
$hlist = $Args{n};
@hostlist = split( ' ', $hlist );
foreach $name (@hostlist) {
if ( $conprot = "ssh" ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $conprot = "rsh" ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
die "Unknown Option with \"-T\" switch! \n";
}
}
}
}

sub ssh_cmd {

my ( $id, $host, @cmd ) = @_;
print "$ssh $id\@$host '@cmd' \n";
@sshout = qx/$ssh $id\@$host '@cmd'/;

#or die "Can't run cmd : $! \n";
print WOF "$host \n";
print WOF "@sshout \n";
}

sub rsh_cmd {

my ( $id, $host, @cmd ) = @_;
print "$rsh -l $id $host '@cmd' \n";

@rshout = qx/$rsh -l $id $host '@cmd' /;

#or die "can't run $rsh -l $id $host '@cmd' : $! \n";

print WOF "$host \n";
print WOF "@rshout \n";
}

sub dmesgMunger {

&getToday;
&runCmd(
"cat /var/adm/messages|grep \"$today\"|egrep -v \"vas|auth\|lw8\|mail.info\|Wait
ing\|Networker savegroup\|local1|checked|wrap|Normal\"|egrep -i \"scsi|disk|err|
fatal|pers|mem|link|fcp|AFT|ASFR|PSYND|ESYND|full|vx_nospace|vxfs|vxvm\""
);
}

sub getToday {
my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst ) =
localtime(time);
chomp( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst );
$year += 1900;
$mon += 1;

my %months = (
1 => 'Jan',
2 => 'Feb',
3 => 'Mar',
4 => 'Apr',
5 => 'May',
6 => 'Jun',
7 => 'Jul',
8 => 'Aug',
9 => 'Sep',
10 => 'Oct',
11 => 'Nov',
12 => 'Dec',
);

if ( $mday < mday = " $mday" today = "$months{$mon} $mday" line ="~" pstate ="=" npstate =" $p-">ping( $name, 1 );
if ( $npstate == 0 ) {

#$p->close();
print "Running \"$ssh $lid\@$name\"...\n";
my @sshout =
system( "$ssh", "-l", "$lid", "$name", "\'exit\'" );
$exitval = $? >> 8;
chomp $exitval;
print WOF
"attempt to log into $name ended with status $exitval \n";
print WHL "$name:$domain:$ip:$pstate:$exitval\n";
}
else {
print WOF "Unable to ping $name \n";
}
}
if ( $pstate == 1 ) {
print
"$hlist says $name is inaccessible.\nBut I will try to ping $name again anyway..
.\n";
my $npstate = $p->ping( $name, 1 );
if ( $npstate == 0 ) {

#$p->close();
print "Running \"$ssh $lid\@$name\"...\n";
my @sshout =
system( "$ssh", "-l", "$lid", "$name", "\'exit;\'" );
$exitval = $? >> 8;
chomp $exitval;
print WOF
"attempt to log into $name ended with status $exitval \n";
print WHL "$name:$domain:$ip:$pstate:$exitval\n";
}
else {
print WOF "Unable to ping $name \n";
}
}
}
close(WOF);
}

inventory.txt (the input file passed to tlrc.pl) --

#HOSTNAME:DOMAIN NAME:IP:PINGABLE(1 == no; 0 == yes):Login(1 == no;0 == yes):Connection(0=ssh:1=telnet/rsh):SERIAL:HOSTID:USAGE(P|NP)

This is a colon-delimited file with fields as listed above. Not all of them are required for running the script, but can be useful in certain cases (eg: hostid, serial #).

ndd_get.sh is a Korn-shell based script that returns the NIC link-related statistics in a comma-separated output format.

#!/usr/bin/env ksh

NDD=/usr/sbin/ndd
ID=`/usr/xpg4/bin/id -u`
HOSTNAME=`/usr/bin/hostname`

printUsage() {
echo "Usage: $0 [ -a ]|[-n -i ]|[-h] \n";
}

splitter() {
interface=$1
INSTANCE=`echo $interface|awk -F\e '{print $2}'`
BASEDEV=`echo $interface|awk -F\e '{print $1}'`
ADAPTER="$BASEDEV"e
}

macipget() {
IF=$1
IFCONFIG=/usr/sbin/ifconfig
IP=`$IFCONFIG $IF|grep inet|awk '{print $2}'`
MAC=`$IFCONFIG $IF|grep ether|awk '{print $2}'`
}

nddget() {
#set -x
AD=$1
INST=$2
$NDD -set /dev/$AD instance $INST
LSTAT=`$NDD -get /dev/$AD link_status`
LSPEED=`$NDD -get /dev/$AD link_speed`
LMODE=`$NDD -get /dev/$AD link_mode`
IS_100FDX=`$NDD -get /dev/$AD adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$AD adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$AD adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$AD adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100fdx_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100hdx_cap`
LP_10FDX=`$NDD -get /dev/$AD lp_10fdx_cap`
LP_10HDX=`$NDD -get /dev/$AD lp_10hdx_cap`
LP_AUTONEG=`$NDD -get /dev/$AD lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat="down"
else
linkstat="up"
fi
if [ $LSPEED -eq 0 ]; then
linkspeed="10"
else
linkspeed="100"
fi
if [ $LMODE -eq 0 ]; then
linkmode="Half Duplex"
else
linkmode="Full Duplex"
fi
if [ $AUTONEG -eq 0 ]; then
autoneg="Off"
else
autoneg="on"
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg="Off"
else
lp_autoneg="On"
fi
IF=$AD$INST
macipget $IF
print "$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg"
}

kstatget() {
#set -x
AD=$1
INST=$2

linkspeed=`/usr/bin/kstat -p $AD|grep -i link_|\
grep "$AD:$INST"|grep link_speed|awk '{print $2}'`

is_up=`/usr/bin/kstat -p $AD|grep -i link_|\
grep "$AD:$INST"|grep link_up| awk '{print $2}'`
if [ $is_up -eq 1 ]; then
linkstat="UP"
else
linkstat="DOWN"
fi
LINK_MODE=`/usr/bin/kstat -p $AD|grep -i link_|\
grep $AD:$INST|grep link_duplex|awk '{print $2}'`
case $LINK_MODE in
2) linkmode="Full Duplex";;
1) linkmode="Half Duplex";;
*) linkmode="Unknown";;
esac

$NDD -set /dev/$AD instance $INST
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_AUTONEG=`/usr/bin/kstat -p $AD|\
grep $AD:$INST|grep lp_cap_autoneg|awk '{print $2}'`
if [ $AUTONEG -eq 0 ]; then
autoneg="Off"
else
autoneg="On"
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg="Off"
else
lp_autoneg="On"
fi
IF=$AD$INST
macipget $IF
print "$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg"

}

bgekstatget() {
#set -x
AD=$1
INST=$2

linkspeed=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_| grep link_speed|awk '{print $2}'`

is_up=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_status| awk '{print $2}'`
if [ $is_up -eq 1 ]; then
linkstat="UP"
else
linkstat="DOWN"
fi
LINK_MODE=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_duplex|awk '{print $2}'`
case $LINK_MODE in
2) linkmode="Full Duplex";;
1) linkmode="Half Duplex";;
*) linkmode="Unknown";;
esac

AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep autoneg|awk '{print $2}'`
LP_AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep lp_| grep autoneg |awk '{print $2}'`
if [ $AUTONEG -eq 0 ]; then
autoneg="Off"
else
autoneg="On"
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg="Off"
else
lp_autoneg="On"
fi

IF=$AD$INST
macipget $IF
print "$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg"

}

dmfeget() {

AD=$1
INST=$2
EADAPT=$AD$INST
#$NDD -set /dev/$EADAPT
# NOte the ndd set is not required since dmfe interfaces are directly
# set up as device files (such as /dev/dmfe0, /dev/dmfe1)

LSTAT=`$NDD -get /dev/$EADAPT link_status`
LSPEED=`$NDD -get /dev/$EADAPT link_speed`
LMODE=`$NDD -get /dev/$EADAPT link_mode`
IS_100FDX=`$NDD -get /dev/$EADAPT adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$EADAPT adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$EADAPT adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$EADAPT adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$EADAPT adv_autoneg_cap`
LP_AUTONEG=`$NDD -get /dev/$ADAPT lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat="down"
else
linkstat="up"
fi
if [ $LSPEED -eq 0 ]; then
linkspeed="10"
else
linkspeed="100"
fi
if [ $LMODE -eq 0 ]; then
linkmode="Half Duplex"
else
linkmode="Full Duplex"
fi
if [ $AUTONEG -eq 0 ]; then
autoneg="Off"
else
autoneg="on"
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg="Off"
else
lp_autoneg="On"
fi
macipget $EADAPT

print "$HOSTNAME,$EADAPT,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$l
p_autoneg"

}

getParms() {
#set -x
case $ADAPTER in
qfe) nddget $ADAPTER $INSTANCE;;
hme) nddget $ADAPTER $INSTANCE;;
eri) nddget $ADAPTER $INSTANCE;;
ce) kstatget $ADAPTER $INSTANCE;;
bge) bgekstatget $ADAPTER $INSTANCE;;
dmfe) dmfeget $ADAPTER $INSTANCE;;
*) echo "Error: Unknown adapter! \n" &&amp; exit 1;;
esac
}

nicStatAll() {
#set -x
/usr/sbin/ifconfig -a|nawk '/UP/{print $1}'|egrep -v "lo0|clprivnet"| \
awk -F: '{print $1}' |sort -nr|uniq > /tmp/iflist;
for interface in `cat /tmp/iflist`
do
if [ $interface = ":*" ]; then
next
fi
# Deprecated code -- left behind for old time's sake
#count=`echo $interface|wc -m|sed -e"s!^[ /t]!!g"`
#count1=`expr $count - 2`
#count2=`expr $count - 1`
#int=`echo $interface|cut -c 1-${count1}`
#dev=/dev/${int}
#inst=`echo $interface|cut -c ${count2}`
case $interface in
eri*) INSTANCE=`echo $interface|awk -F\i '{print $2}'`
BASEDEV=`echo $interface|awk -F\i '{print $1}'`
ADAPTER="$BASEDEV"i;;
*) splitter $interface;;
esac
getParms
done
}

if [ $ID -ne 0 ]; then
echo "ERROR: You are not root! Only root can run this script!\n";
exit 1;
fi

while getopts an:i:h arg
do
case $arg in
a) nicStatAll &&amp; exit 0;;
n) ADAPTER=${OPTARG};;
i) INSTANCE=${OPTARG};;
h) printUsage &&amp; exit 0;;
*) printUsage &&amp; exit 1;;
esac
done
shift $(($OPTIND - 1))

if [ ! -z ${ADAPTER} ]; then
if [ ! -z ${INSTANCE} ]; then
getParms
else
printUsage && exit 1
fi
else
printUsage && exit 1
fi

On the centralized management host (whose SSH2-based Key is trusted by the monitored hosts) run the following command to perform the inventory:

admin:(dev) $ sudo ./tlrc.pl -l root -a \
-c "/path/to/nddget.sh -a" \
-o ~/logs/ndd_get_today.txt

/usr/bin/ssh root@host1 '/path/to/ndd_get.sh -a'
/usr/bin/ssh root@host2 '/path/to/ndd_get.sh -a'
host1
host1,bge2,IP,MAC,UP,100,Full Duplex,On,On
host1,bge1,IP,MAC,UP,100,Full Duplex,On,On
host1,bge0,IP,MAC,UP,100,Full Duplex,On,On

host2
host2,bge2,10.228.147.62,0:3:ba:49:45:51,UP,100,Full Duplex,On,On
host2,bge1,10.228.143.62,0:3:ba:49:45:50,UP,100,Full Duplex,On,On
host2,bge0,10.228.139.62,0:3:ba:49:45:4f,UP,100,Full Duplex,On,On

/usr/bin/rsh -l root host3 '/path/to/ndd_get.sh -a'
/usr/bin/rsh -l root host4 '/path/to/ndd_get.sh -a'

host3
host3,qfe1,IP,MAC,up,100,Full Duplex,Off,Off
host3,qfe0,IP,MAC,up,100,Full Duplex,Off,Off
host3,ce0,IP,MAC,UP,1000,Full Duplex,On,On

Look at the text output created thus:

admin:(logs) $ more ndd_get_today.txt

host1
host1,bge2,IP,MAC,UP,100,Full Duplex,On,On
host1,bge1,IP,MAC,UP,100,Full Duplex,On,On
host1,bge0,IP,MAC,UP,100,Full Duplex,On,On

host2
host2,bge2,IP,MAC,UP,100,Full Duplex,On,On
host2,bge1,IP,MAC,UP,100,Full Duplex,On,On
host2,bge0,IP,MAC,UP,100,Full Duplex,On,On

Now look at the sudo log file to see if there's associated logging captured.

admin:(log) $ sudo tail sudo.log
Sep  5 16:32:36 : lahirdx : TTY=pts/27 ; PWD=/export/home/lahirdx/dev ;
    USER=root ; COMMAND=/usr/bin/ssh aesdbc1
Sep  6 09:49:31 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ;
    USER=root ; COMMAND=./tlrc.pl -a -c /export/patches/Scripts/bin/ndd_get.sh
    -a -o /export/home/lahirdx/logs/ndd_get_9606.txt
Sep  6 09:49:40 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ;
    USER=root ; COMMAND=./tlrc.pl -l root -a -c
    /export/patches/Scripts/bin/ndd_get.sh -a -o /export/home/lahirdx/logs/ndd_get_9606.txt

NOTE: Look at the full command line, who executed a particular command, when etc getting captured in the logs. Also, it is imperative to ensure that the "/path/to/ndd_get.sh" is the same on all the monitored hosts. This author recommends creating a system V package to deploy commonly used scripts and tools under /opt/tools (or similar directory structure) to ensure standardization of the environment.

More Zone Configuration stuff

2007-01-02T14:08:00.000-06:00

# more zonecfg-hints.txt
LOFS mount:

global# newfs /dev/rdsk/c1t0d0s0
global# mount /dev/dsk/c1t0d0s0 /mystuff
global# zonecfg -z my-zone
zonecfg:my-zone> add fs
zonecfg:my-zone:fs> set dir=/usr/mystuff
zonecfg:my-zone:fs> set special=/mystuff
zonecfg:my-zone:fs> set type=lofs
zonecfg:my-zone:fs> end
* Use a UFS mount:

global# newfs /dev/rdsk/c1t0d0s0
global# zonecfg -z my-zone
zonecfg:my-zone> add fs
zonecfg:my-zone:fs> set dir=/usr/mystuff
zonecfg:my-zone:fs> set special=/dev/dsk/c1t0d0s0
zonecfg:my-zone:fs> set raw=/dev/rdsk/c1t0d0s0
zonecfg:my-zone:fs> set type=ufs
zonecfg:my-zone:fs> end
* Export the device node and mount from the non-global zone:

global# zonecfg -z my-zone
zonecfg:my-zone> add device
zonecfg:my-zone:device> set match=/dev/rdsk/c1t0d0s0
zonecfg:my-zone:device> end
zonecfg:my-zone> add device
zonecfg:my-zone:device> set match=/dev/dsk/c1t0d0s0
zonecfg:my-zone:device> end
my-zone# newfs /dev/rdsk/c1t0d0s0
my-zone# mount /dev/dsk/c1t0d0s0 /usr/mystuff
* Mount the FS directly from the Global zone when the non-global zone is run
ning:

global# mount /dev/dsk/c1t0d0s0 /export/zones/zone1/root/mnt
* Using lofiadm

#

Solaris 10 Zone Configuration Advanced tricks

2007-01-02T13:56:00.000-06:00

Cannot manipulate system device files in local zones (only in global zones). But one can export special devices such as Sun Volume Manager devices, network devices, etc from global to local zones.

EG: Creating an SVM soft partition that I want to use exclusively for a local zone (z1 in this case).

dwailsun:(/) # metainit d43 -p d4 300m
d43: Soft Partition is setup

dwailsun:(/) # metastat -p
d3 1 1 c0t0d0s3
d1 1 1 c0t0d0s1
d0 1 1 c0t0d0s0
d43 -p d4 -o 1638496 -b 614400
d4 1 1 c0t0d0s4
d42 -p d4 -o 819264 -b 819200
d41 -p d4 -o 32 -b 819200

dwailsun:(/) # newfs /dev/md/rdsk/d43
newfs: construct a new file system /dev/md/rdsk/d43: (y/n)? y
Warning: 480 sector(s) in last cylinder unallocated
/dev/md/rdsk/d43: 614400 sectors in 610 cylinders of 16 tracks, 63 sectors
300.0MB in 39 cyl groups (16 c/g, 7.88MB/g, 3776 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 16224, 32416, 48608, 64800, 80992, 97184, 113376, 129568, 145760,
468576, 484768, 500960, 516128, 532320, 548512, 564704, 580896, 597088,
613280,

Now trying out various methods of provisioning this logical volume to the zone "z1".

Can we export it (from the global zone) as a filesystem?
We could -- but then we'll have to mount it up on global zone first. So let's skip this one...

dwailsun:(/) # zonecfg -z z1
zonecfg:z1> add fs
zonecfg:z1:fs>
zonecfg:z1:fs> end
dir not specified
special not specified
type not specified
zonecfg:z1:fs> cancel

Can we directly add the SVM metadevices to the local zone?
After much heartache I found it it's not possible (yet). We can however provide solaris native slices (/dev//c0t0d0s0 type devices) via the following method.

zonecfg:z1> add device
zonecfg:z1:device> set match=/dev/md/rdsk/d43
zonecfg:z1:device> end
zonecfg:z1> info
zonepath: /zones/z1
autoboot: true
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address: 192.168.1.2
physical: eri0
device
match: /dev/md/rdsk/d43
zonecfg:z1>exit

dwailsun:(/) #
dwailsun:(/) # zlogin z1
[Connected to zone 'z1' pts/4]
Last login: Tue Apr 25 16:06:22 on pts/4
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
dwailsun-z1:(/) #
dwailsun-z1:(/) # newfs /dev/md/rdsk/d43
newfs: /dev/md/rdsk/d43: No such file or directory

dwailsun-z1:(/) # exit

[Connection to zone 'z1' pts/4 closed]

zonecfg:z1> add device
zonecfg:z1:device> set match=/dev/md/dsk/d43

zth: /zones/z1
autoboot: true
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address: 192.168.1.2
physical: eri0
device
match: /dev/md/rdsk/d43
device
match: /dev/md/dsk/d43
onecfg:z1:device> zonecfg:z1> verify
zonecfg:z1> commit
zonecfg:z1> info
zonepath: /zones/z1
autoboot: true
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address: 192.168.1.2
physical: eri0
device
match: /dev/md/rdsk/d43
device
match: /dev/md/dsk/d43
zonecfg:z1> exit
dwailsun:(/) # zlogin z1
[Connected to zone 'z1' pts/4]
Last login: Tue Apr 25 16:06:45 on pts/4
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
dwailsun-z1:(/) # newfs /dev/md/dsk/d43
newfs: /dev/md/dsk/d43: No such file or directory

Obviously that doesn't work. So let's blow that away and try a different option...

zonecfg:z1> remove device match=/dev/md/rdsk/d43
zonecfg:z1> remove device match=/dev/md/dsk/d43

Let's try by adding the metadevice after newfs'ing it in the global zone but not as a device, but an UFS filesystem instead.

zonecfg:z1> add fs
zonecfg:z1:fs> set dir=/build
zonecfg:z1:fs> set special=/dev/md/dsk/d43
zonecfg:z1:fs> set raw=/dev/md/rdsk/d43
zonecfg:z1:fs> set type=ufs
zonecfg:z1:fs> end

dwailsun-z1:(/) # df -k /build
Filesystem kbytes used avail capacity Mounted on
/build 288151 13753 245583 6% /build

dwailsun-z1:(/) # cat /etc/mnttab |grep build
/build /build ufs rw,intr,largefiles,logging,xattr,onerror=panic,dev=154002b 1146064117

And even though this doesn't show up in the local zone's /etc/vfstab, the UFS filesystem is being exported (exclusively) for the local zone (z1) from the global zone (in this case).

dwailsun:(/) # zoneadm list -vc
ID NAME STATUS PATH
0 global running /
2 z2 running /zones/z2
- z1 installed /zones/z1

dwailsun:(/) # zlogin -C -e\@ z1
[Connected to zone 'z1' console]

dwailsun-z1 console login: root
Password:
Apr 26 09:28:19 dwailsun-z1 login: ROOT LOGIN /dev/console
Last login: Wed Apr 26 09:22:48 on pts/5
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
dwailsun-z1:(/) # init 6
dwailsun-z1:(/) # svc.startd: The system is coming down. Please wait.
svc.startd: 57 system services are now being stopped.
Apr 26 09:28:30 dwailsun-z1 syslogd: going down on signal 15
svc.startd: The system is down.

[NOTICE: Zone rebooting]

zonecfg:z1> add inherit-pkg-dir
Zone z1 already installed; add inherit-pkg-dir not allowed.
zonecfg:z1> export -f /var/tmp/tools/z1.cfg

dwailsun:(/) # zoneadm -z z1 uninstall
Are you sure you want to uninstall zone z1 (y/[n])? y

Configuring a zone to access the CDROM drive

zonecfg:z1> add fs
zonecfg:z1:fs> set dir=/cdrom
zonecfg:z1:fs> set special=/cdrom
zonecfg:z1:fs> set type=l
zonecfg:z1:fs> set options=[nodevices]
zonecfg:z1:fs> verify
resource specification incomplete
z1: Insufficient specification
zonecfg:z1:fs> end
zonecfg:z1> verify
zonecfg:z1> info
zonepath: /zones/z1
autoboot: true
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
fs:
dir: /cdrom
special: /cdrom
raw not specified
type: l
options: [nodevices]
net:
address: 192.168.1.2
physical: eri0
device
match: /dev/md/rdsk/d43
device
match: /dev/md/dsk/d43
zonecfg:z1>
zonecfg:z1> commit

Note: If CD-ROM access is granted to a zone, other zones should not be configured to access the same CD-ROM drive.

Kickstart and PXELinux

2006-12-22T13:50:00.000-06:00

As part of a vertical consolidation and virtualization project, I had set up a Kickstart configuration in order to boot x86 servers and install the VMWare ESX Server software. While I wrote this document specifically for ESX installations, the underlying OS was a modified Redhat 7.x distro (or something that seemed awfully similar). So here are the steps to implement and use a PXEBooting Kickstart environment.

Prerequisites

Linux server (can use a manually built ESX server for this purpose) with dhcpd (dhcp daemon), tftpd (tftp daemon) and nfs server services turned on.
Have the dump of the ESX installation media CD in a directory/filesystem large enough to hold it.

mount /dev/cdrom /mnt

mkdir –p /images/install

cp –pra /mnt/* /images/install

umount /mnt

cd /images/ install/images

mount –o loop ./bootnet.img /mnt

cd /mnt

mkdir –p /tftpboot/pxelinux.cfg

cp boot.msg initrd.img syslinux.cfg vmlinuz /tftpboot

cd /tftpboot

umount /mnt

cp syslinux.cfg pxelinux.cfg/default

Details

Having copied these files over, now make a copy of the /tftpboot/pxelinux.cfg/default to /tftpboot/pxelinux.cfg/bootnet
Edit /tftpboot/pxelinux.cfg/bootnet to make it look like this:

default ks
prompt 0
timeout 0
display boot.msg
F1 boot.msg
label linux
kernel vmlinuz
append initrd=initrd.img
label ks
kernel vmlinuz
append apic ks=nfs:10.165.214.160:/vmimages/kickstart/ks.cfg ramdisk_size=10240 initrd=initrd.img

NOTE: Make sure that there isn’t actually a line break in the line starting with append (the entries following the “append” directive need to be on the same line and “\” has been put in for ease of readability only).

Get a copy of the “syslinux” distribution (run the commands listed below):

http://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-3.07.tar.gz

Explode the tarball like this:

# cp syslinux-3.07.tar.gz /tmp
# tar zxvf /tmp/syslinux-3.07.tar.gz
# cd /tmp/syslinux-3.07
# cp syslinux.0 /tftpboot

Edit the dhcp configuration file like this:

# vi /etc/dhcpd.conf

# ddns-update-style none;
ignore client-updates;
allow bootp;
allow booting;
option ip-forwarding false;
option mask-supplier false;
get-lease-hostnames off;
use-host-decl-names off;
option routers 10.165.214.1;

subnet 10.165.214.0 netmask 255.255.255.0 {
default-lease-time 1800;
max-lease-time 3600;
option subnet-mask 255.255.255.0;
}

host hostA {
hardware ethernet 00:AA:BB:CC:DD:EE;
fixed-address 10.165.214.164;
#option option-214 "0AA5D6A4";
#next-server 10.165.214.160;
filename "/tftpboot/pxelinux.0";
}

NOTE: Choose the correct subnet (that your servers will reside in).

Make sure that the tftpd daemon is installed and edit the file /etc/xinetd.d/tftp to look like this:

service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = /tftpboot
disable = no
per_source = 11
cps = 100 2
}

It is to be noted that the default ESX server install comes with gcc (2.96) and make (GNU Make version 3.79.1). But the tftp daemon and the dhcp daemon (server-side) aren’t installed by default.
Download the latest versions of the dhcp sources (can obtain the software by following instructions at this url: http://www.isc.org/index.pl?/sw/dhcp/) and build it by following the instructions in the README file or INSTALL file of the distribution (usually it is a standard method of ./configure; make; make install)
Similarly, obtain a copy of the latest tftp source from (ftp://ftp.kernel.org/pub/software/network/tftp/). Following the build instructions will result in the installation of the tftpd binary on the system.
Start the dhcp daemon by running:

dhcpd -cf /etc/dhcpd.conf

Start the tftp daemon by sending a SIGHUP to the xinetd daemon

kill -HUP `ps -ef|grep xinetd|grep -v grep|awk '{print $2}'`

The NFS server services need to be started up after creating the appropriate NFS shares in /etc/exports. The contents of the /etc/exports file should look like this:

#
/vmimages/install *(ro,no_root_squash)
/vmimages/kickstart *(ro,no_root_squash)
/tftpboot *(ro,no_root_squash)

Start NFS services by running the following commands:

# cd /etc/rc.d/init.d
# ./portmap start
# ./nfs start
# ./nfslock start

After ensuring that the target ESX server (hardware) is plugged into the right subnet, boot the server and choose the PXE boot option. The booting from the PXE Ethernet ROM happens like this:

Server boots and the PXE ROM loads, tries to acquire an IP address from the DHCP server (bootp).
Once the IP address is obtained, the pxelinux.0 binary loads and obtains the vmlinuz kernel image via tftp.
The initial ramdisk image (initrd, also acquired via tftp) is loaded into memory (to create the root filesystem in RAM).
The kickstart configuration (ks.cfg) is acquired via NFS and used by the installer to do the “hands-free” installation.

Using the ks-config.pl script

#!/usr/bin/perl -w

# Version: 1.2 (SCCS)

use File::Copy;

use Getopt::Std;

my %Args;
my $dhcp_config = "/etc/dhcpd.conf";
my $null = "/dev/null";
my $tftpdir = "/tftpboot";
my $pxedir = "$tftpdir/pxelinux.cfg";
my $netboot_cfg = "$pxedir/netboot.1";

#**************
# Main
#**************

getopts( "m:s:i:h", \%Args );

if ($Args{h}) {
printUsage() && exit 0;
}

$mac = $Args{m} || printUsage() && die "Unable to continue without a mac address \n";

$hostname = $Args{s} || printUsage() && die "Unable to continue without a server name \n";

$ip = $Args{i} || printUsage() && die "Unable to continue without an IP address \n";

copy("$dhcp_config", "$dhcp_config.$$") or die "Unable to backup $dhcp_config: $! n";

open(DHCP, ">> $dhcp_config") or die "Unable to write to $dhcp_config : $! \n";

print DHCP "\n";
print DHCP "host $hostname\{ \n";
print DHCP "hardware ethernet $mac; \n";
print DHCP "fixed-address $ip; \n";
print DHCP "filename \"$tftpdir/pxelinux.0\"; \n";
print DHCP "\} \n";
close(DHCP);

print "Killing dhcp daemon...\n";
system("/usr/bin/pkill dhcpd");
print "Starting dhcp daemon....\n";
$start_cmd = "/usr/sbin/dhcpd -cf $dhcp_config";
system("$start_cmd");

sub printUsage {
print "Correct Usage: $0 -m \"\" -s -i \"\" | -h \n";
}

Using the script (modify to your heart’s content) , one could set up the dhcpd entries (knowing well that our friendly vi editor isn’t necessarily the most user-friendly (especially to the UNIX un-initiates..)) rather quickly/easily.

Here’s how:

# chmod +x ks-config.pl
# ./ks-config.pl –m “11:22:33:44:55:66” –s buildserver01 –i “10.165.214.55”

Would set up entries in dhcpd.conf like this:

[root@ksserver bin]# ./ks-config.pl -m "11:22:33:44:55:66" -s buildserver01 > -i "10.165.214.55"
Killing dhcp daemon...
Starting dhcp daemon....
Internet Systems Consortium DHCP Server V3.0.1
Copyright 2004 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on LPF/eth0/00:11:0a:53:e8:98/10.165.214.0/24
Sending on LPF/eth0/00:11:0a:53:e8:98/10.165.214.0/24
Sending on Socket/fallback/fallback-net

Take a look at the /etc/dhcpd.conf file later:

[root@ksserver bin]# tail /etc/dhcpd.conf
hardware ethernet 00:11:AA:BB:CC:DD;
fixed-address 10.165.214.174;
filename "/tftpboot/pxelinux.0";
}

host buildserver01{
hardware ethernet 11:22:33:44:55:66;
fixed-address 10.165.214.55;
filename "/tftpboot/pxelinux.0";
}

Perl Praises and Beefs

2006-12-19T14:53:00.000-06:00

An insightful member of the slashdot community had made this observation here (http://slashdot.org/comments.pl?sid=189778&cid=15623704). Thought I'd spin it off and blog it.

Praise

The power of perl is irrefutable -- it helps slap together quick and clean solutions to irritating admin problems. The flip-side of being a perl jockey I guess is that one tends to try and create a solution to many a problem that already has a solution - because searching CPAN can be a pain at times.
Use of the more flexible features of the languages (such as Hashes, hash of hashes etc) data/number munging and organization becomes more manageable.
Using Perl's almost endless modules, a lot of relatively complicated tasks can be simplified.
Annoyance factor of numerous tasks (especially Administrative and reporting) can be reduced drastically with the help of Perl.

Beefs

The beef I guess is that unlike Python or Perl's other competitors, Perl modules don't come tightly integrated with the core distro. Agreed that Perl probably has a lot more modules than any of those other languages do, but a larger than ordinary de facto distribution (why not include important modules like Digest::MD5, Crypt modules, SSH modules etc?) would be desirable (especially in those situations where you don't have access to the internet directly from within corporate networks and can't install the modules with the "perl -MCPAN -e shell" option) . There might be those Perl veterans who would say -- "build your own distro with your custom modules already packaged" -- and while that might be a very smart thing to do, many a time (when one keeps moving from one environment to another -- some call it job hopping, it helps to be able to download one single perl distro package or rpm or the source+compile and have basic administrative scripts work -- especially those that rely on centralized automation (ssh-based trusts, copies across the network, etc).
Also, perl's syntax can be terse and difficult for noobies to understand (or even older perl-hands for that matter -- when someone has written code without appropriate comments, etc).
Tinkering with Python recently, I found it's simplicity refreshing and it's syntax easier to comprehend (especially when compared with Perl's (imho) complicated "scoping" requirements, etc).
Sometimes (and I guess it depends on the person writing the code) Perl tends to over-complicate things that can be easily handled via Shell scripts.

That much said (several months back), I'd have to say (after having written a pretty big (by my standards) application in Python, I like certain aspects of Python a lot (comprehensibility of code lists at the top) but I still found myself wanting to revert back to Perl at the slightest opportunity. I was irritated with the way you have to declare the variable type in Python (eg: can't do a floating point operation on two numbers unless you qualify the variable as type float whereas in Perl it is automatically detected) -- now I know there are those who will say "Perl does it wrong" and while that might be true, it is also true that a pressed-for-time Systems Admin could do with worrying about other things in the larger scheme of things...

I guess I must've have internalized the terse syntactical idiosyncrasies of Perl somewhere along the way and thinking in "Python" seemed a wee bit alien.

What I did like about Python is the ease with which one could develop GUI interfaces .
I actually had to write a reporting tool on the windows platform -- and using the win32com.client module, coding was a piece of cake. I also played with the Tkinter module on UNIX and coding with it was a breeze as well.

Using Perl range operators within Shell scripts

2006-12-19T14:28:00.000-06:00

While this might be totally redundant, I couldn't for the love of God remember how range operators were used within the various shells (and specifically the Korn shell). Tried various tricks but I guess I must've gotten "perl-ized" somewhere along the way. So here's my workaround for quick shell scripts (usually fired off from the command line on the fly) that require range operators (say you want to iterate through the numbers 1 - 100 and do some activity with each value.

for i in `perl -e 'foreach $i (1..100) { print "$i \n"; }'`
do
echo "I is $i";
done

Here we'll invoke a perl one-liner to print out the values of $i in the range 1 to 100. We read that and do something with it within a "ksh for loop". It's easier than having to type in:

for i in 1 2 3 4 5 .... 100
do
echo "I is $i"
done

Sidenote: Hopefully I'll get to update this blog more often that I usually do -- if there's no one watching, at least it's something for me to refer to (as a cheatsheet on the fly).

Some Ksh/Bash hackeries

2006-08-18T09:06:00.000-06:00

${parameter%word}
Remove smallest suffix pattern. The WORD is expanded to produce
a pattern. It then expands to the value of PARAMETER, with the
smallest portion of the suffix matched by the pattern deleted.

x=file.c
echo ${x%.c}.o
-->file.o

${parameter%%word}

Remove largest suffix pattern. The WORD is expanded to produce
a pattern. It then expands to the value of PARAMETER, with the
largest portion of the suffix matched by the pattern deleted.

x=posix/src/std
echo ${x%%/*}
-->posix

${parameter#word}
Remove smallest prefix pattern. The WORD is expanded to produce
a pattern. It then expands to the value of PARAMETER, with the
smallest portion of the prefix matched by the pattern deleted.

x=$HOME/src/cmd
echo ${x#$HOME}
-->/src/cmd

${parameter##word}
Remove largest prefix pattern. The WORD is expanded to produce
a pattern. It then expands to the value of PARAMETER, with the
largest portion of the prefix matched by the pattern deleted.

x=/one/two/three
echo ${x##*/}
-->three

Given
a=/a/b/c/d
b=b.xxx

csh bash result
--- ---- ------
$a:h ${a%/*} /a/b/c
$a:t ${a##*/} d
$b:r ${b%.*} b
$b:e ${b##*.} xxx

(source comp.unix.shell -- http://groups.google.com/group/comp.unix.questions/browse_frm/thread/9127ba46689a0c13/2384610dc72c6ba8?lnk=gst&q=variable+expansions&rnum=2#2384610dc72c6ba8)