27-JAN-2011: The day "/proc" died r1 (See the current copy)

Solaris'd...

The setup:

 Date: Thu, 27 Jan 2011 14:14:49
 From: "UNIX Manager"
 To: UNIX_SA_GROUP
 Subject: Milky Way District UNIX Server

 Someone take a look into whats going on with Milky Way District UNIX Server server today. The server is up but is acting really strange, I was not able to get a response from ps or ptree commands.  Please note, there was not any server consolidation work at this site last night.

 --
 A. Manager
 Enterprise Unix & Storage Systems Manager
 Spacely Sprockets, Inc.

---

 # ls -ln /proc & echo $!
 20023
 # mdb -k
 > 0t20023::pid2proc
 302765d6078
 > 302765d6078::walk thread
 300e3d5e100
 > 300e3d5e100::findstack -v
 stack pointer for thread 300e3d5e100: 2a10d548d71
 [ 000002a10d548d71 cv_wait+0x38() ]
   000002a10d548e21 pr_p_lock+0x80(0, 60032d48030, 60032d58600, 300b84af968, ff000000, 18f8218)
   000002a10d548ed1 prgetattr+0x2d4(30025ce8240, 2a10d549998, 149, ffffffffffffffef, 300bd1a8da8, 0)
   000002a10d548f91 fop_getattr+0x18(30025ce8240, 2a10d549998, 0, 3025fb17d00, 2a10d549ad8, 1362e00)
   000002a10d549041 cstat64_32+0x1c(30025ce8240, ffbffb20, 0, 3025fb17d00, 3fff, 3c00)
   000002a10d549221 cstatat64_32+0x5c(ffffffffffd19553, 26578, 1000, ffbffb20, 1000, 0)
   000002a10d5492e1 syscall_trap32+0xcc(26578, ffbffb20, ffffffffffffffff, 27f68, 6c, 1b)
 >
 > 302765d6078 $p
 debugger context set to proc 302765d6078
 Segmentation Fault
 #
 # mdb -p 20023
 mdb: failed to initialize /lib/libc_db.so.1: libthread_db call failed unexpectedly
 mdb: warning: debugger will only be able to examine raw LWPs
 Loading modules: [ ld.so.1 libc.so.1 libavl.so.1 ]
 > 26578::print -i char*
 0x26578 "/proc/26936"
 >
 # echo '::ps -f' | mdb -k | grep 26936
 S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
 R  26133  19481  26131  19469      0 0x4a004000 0000030276c7c4a0 grep 26936
 R  26936  26935  26935  26935   1000 0x4a014000 00000300b84af968
 # echo '::ps -fz' | mdb -k | grep 26936
 S    PID   PPID   PGID    SID  ZONE    UID      FLAGS             ADDR NAME
 R  26936  26935  26935  26935     3   1000 0x4a014000 00000300b84af968
 # mdb -k
 Loading modules: [ unix genunix specfs dtrace ufs sd mpt px md ldc ip hook neti sctp arp usba fcp fctl emlxs qlc lofs zfs ssd random crypto fcip logindmux ptm nfs ipc ]
 > 00000300b84af968::walk thread
 3000f654d40
 > 3000f654d40::findstack -v
 stack pointer for thread 3000f654d40: 2a109f38a61
 [ 000002a109f38a61 cv_wait+0x38() ]
   000002a109f38b11 txg_wait_open+0x54(6003f983d20, 16533a, 0, 6003f983d64, 6003f983d66, 6003f983d18)
   000002a109f38bc1 zfs_putapage+0x1e0(6004d6fae40, 5b, 2a109f39570, 2a109f39568, 400, 10a6ac0)
   000002a109f38cb1 zfs_putpage+0x1b8(3002836dd40, b7c1a000, 0, 400, 30114c15318, 7000f9c9d80)
   000002a109f38d81 fop_putpage+0x1c(3002836dd40, 0, b7c1a000, 400, 30114c15318, 7b262c7c)
   000002a109f38e31 zfs_delmap+0x6c(3002836dd40, 0, 10a6800, 30114c15318, b7c1a000, b)
   000002a109f38ef1 zfs_shim_delmap+0x3c(3002836dd40, 30114c15318, 1, 1, f, b)
   000002a109f38fc1 fop_delmap+0x40(3002836dd40, 30114c15318, 1, f, b7c1a000, b)
   000002a109f39091 segvn_unmap+0x180(3017e8a0bd0, fffffffec0000000, b7c1a000, b7c1a000, f, 600439d6b78)
   000002a109f39181 as_unmap+0xe4(300ee744d20, 3017e8a0bd0, 300de51bd88, b7c1a000, 1, 1)
   000002a109f39231 munmap+0x78(1fff, b7c1a000, 10a6800, 300b84af968, 300b84af968, fffffffec0000000)
   000002a109f392e1 syscall_trap+0xac(fffffffec0000000, b7c1a000, 0, 8, 10012c4d0, 10012c460)
 >