FatMin

Recovering from failed vxevac

If you have ever evacuated disks in Veritas, every so often this will happen to hang.  Usually you terminate your session or who knows what.  Kinda like Joe Girardi's willingness to sacrifice outs for no good reason every time the Yankees hottest hitter is at the plate.  It happens, you can't explain it, you move on.  Back to technology - vxtask list shows no tasks, but you get errors trying to rerun the failed evac. 

 

For example:

Plex %5 in volume rman is locked by another utility

Plex rman-01 in volume rman is locked by another utility

Subdisk rman_7_tmp-01 in plex rman-01 is locked by another utility

vxprint -hf is our best friend, as it shows you any flags that are set

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   ATT1    -

pl %5           rman   ENABLED  11719399168 -     TEMPRM   SDMVTMP -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

We can see that we have flags set on the temporary plex (from the failed evac), the subdisk for the temporary plex, the main plex, the subdisk in the main plex, as well as the volume itself.  We need to clear flags to be able to finish re-start our evac.  I will also cut the lines on the vxprint that don't change for the purpose of shortening this post.

vxmend -g rman_dg clear all rman %5

So we cleared the volume and temp plex flags, here's the vxprint -htf output afterwards

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl %5           rman    ENABLED  11719399168 -     TEMPRM   -       -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

So now with the flags cleared we can remove the temporary plex

vxplex -g rman_dg -o rm dis %5

 

And once again our new vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

Great, now down to two flags, the one on the plex and the one on the source disk of our original evac.  Clearing flags from subdisks is a lot trickier than clearing flags from volumes and plexes.  Because the tutil0 flga is already set, we will need to force the clear.  We clear by setting it to "".

vxedit -g rman_dg -f set tutil0="" rman_6_tmp-01

 

Once again, vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

 

And lastly, we clear the flag on the plex.  Why in this order?   Because I'm writing this up after I fixed my issues.  In the interest of not editing vxprint outputs, it's like this.  In retrospect, this could have been cleared with the first one we ran in the beginning.

vxmend -g rman_dg clear all rman rman-01

 

And finally, the way a vxprint -htf should look when all is healthy.

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   -       -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

 

At this point, feel free to proceed with your evac again.  If you're wondering what the putil and tutil fields are, here is what I found courtesy of Symantec:

http://www.symantec.com/business/support/index?page=content&id=TECH15609

 

Guest Authored By: @momkvi

 

ZFS Cheat Sheets and Quick Commands

Solaris is not dead, at least not yet, which is a good thing Solaris 10 included ZFS. What is ZFS you ask, well its a filesystem and logical volume manager in one. ZFS filesystems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices(vdevs), which are themselves constructed of block devices: files, hard drives or partitions, or entire drives, with the last being the recommended usage.

Since I do not use ZFS commands that often, I find myself forgetting them all the time. Below is a list of sites that I have found helpful.

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

http://www.lildude.co.uk/zfs-cheatsheet/

http://hell.jedicoder.net/?p=45

http://pr.zoinks.org/home/technology/tech-wiki/zfs-cheat-sheet

Solaris 10 and Net-SNMP

Apparently Net-SNMP is now included in Solaris 10 by default, however it is not enabled by default. To use it you must first disable Sun's SNMP

#svcadm disable svc:/application/management/snmpdx:default

Net-SNMP is part of Solaris' System Management Agent. More about that here:
Solaris System Management Agent Developer's Guide

Net-SNMP daemon is located at /usr/sfw/sbin/snmpd, and its configuration file is located in /etc/sma/snmp.

You will need to use svcadm to start and stop Net-SNMP:

#svcadm enable svc:/application/management/sma:default
#svcadm disable svc:/application/management/sma:default

Apparently there are two logs that you can reference for Net-SNMP

SMA's log is located here -- /var/svc/log/application-management-sma\:default.log.
However the default log file of snmpd is /var/log/snmpd.log.

You also have an init script in /etc/init.d called  init.sma

Solaris Package Management Cheat Sheet

Basic Package Management Commands Show description about <pkg>	pkginfo -l <pkg>
Install <pkg-file>	pkgadd -d <pkg-file>
Remove <pkg>	pkgrm <pkg>
Find package which owns <file>	pkgchk -l -p <file>
List files in installed <pkg>	pkgchk -l <pkg>|awk '/^Pathname/ {print $2}'
List files in <pkg-file>	pkgchk -l -d <pkg-file>|awk '/^Pathname/ {print $2}'
List all installed packages	pkginfo
Verify integrity of installed files from <pkg>	pkgchk <pkg>
Verify integrity of install files from all packages	pkginfo|awk '{print $2}'|xargs pkgchk

ZFS Quick Start

ZFS is a filesystem, designed by Sun, and implemented in Solaris 10. Unlike traditional filesystems, which reside on single disk devices and thus require a volume manager to use more than one device, ZFS filesystems are built on top of virtual storage pools called zpools, which elimiate the need to create individuall volumes. When creating a filesystem, you do not need to specify a size, as filesystems will grow automatically withing the zpool . When new storage is added, all file systems within the pool can immediately use the additional space without additional work. Most importantly ZFS is simple and pretty easy to administer. See my simple notes below which outline how to get started.

1. First create your pool. I called mine database_pool

-bash-3.00# zpool create database_pool c1t14d0s6 c1t13d0s6 c1t12d0s6

2. Now admire your new pool. Yours may differ in size :)

-bash-3.00# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
database_pool          50.2G   60.5K   50.2G     0%  ONLINE     -

3. Create your volumes. In this example I create two similar to what we will be doing in Philly.
-bash-3.00# zfs create database_pool/apdru
-bash-3.00# zfs create database_pool/apdi2

4. Take a look at your new mounts
-bash-3.00# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
database_pool        51867648      25 51867511     1%    /database_pool
database_pool/db1  51867648      24 51867511     1%    /database_pool/db1
database_pool/db2 51867648      24 51867511     1%    /database_pool/db2

5. Make the directory structure for your new volumes if the above path is not where you want them mounted.
-bash-3.00# mkdir -p /db1/v1
-bash-3.00# mkdir -p /db2/v1

6. Mount your new volumes where you want them to go using zfs set. Note there is no leading / on the pool name

-bash-3.00# zfs set mountpoint= /db1/v1 database_pool/db1
-bash-3.00# zfs set mountpoint= /db2/v1/apdru database_pool/db2

7. Admire your work. You do not have to create a filesystem, its already there and each mount point/volume will grow as needed up to the total capacity of the pool. So no need to worry about sizing each volume for each database.


-bash-3.00# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
database_pool        51867648      25 51867498     1%    /database_pool
database_pool/apdi2  51867648      24 51867498     1%    /db1/v1
database_pool/apdru  51867648      24 51867498     1%    /db2/v1

Solaris Fair Share Scheduling and Projects

Solaris FSS is a method of resource sharing and allocation based on the importance of a process. Specifically a SHARE is a portion of a CPUs resources. In the instance below I was asked to setup FSS and Projects so that two low priority test databases could run along side production databases without any risk of impact.

First off its work nothing that FSS is not always installed by default, you need to ensure that you have the package, SUNWfss.

pkginfo SUNWfss

You are looking for the response below

system      SUNWfss Fair Share Schedule system      SUNWfss Fair Share Schedule

 Ok, since I have several databases running on each box, I am going to make this easy on myself and create two NIS groups, one that the production database users will belong to and one that the non-production, or low priority databases will belong to. The two groups that I created are ora-high and ora-low. This should be self explanitory. Once created in NIS and given access to the box, the xext step is to add ourprojects.

The two examples below define projects by groups.

#projadd -c "Oracle high priority project" group.ora-high

#projadd -c "Oracle low priority project" group.ora-low

 

Now view your new projects.

#projects -l

 

Use the command below to see what projects a user id is part of.

#id -p <userid>

 Now that the projects are created, go ahead and change the system scheduling to use Solaris FSS. This change will require a reboot.

Changing a Solaris Server's IP Address

The following steps may be used to change the IP address of a Solaris system.

1. Change the host's IP in /etc/hosts for the change to take effect after reboot. If you are using Solaris 10, you must also change the host's IP in /etc/inet/ipnodes for the change to take effect after reboot.

2. Run ifconfig interface ip_address netmask broadcast_address for the IP address change to take effect immediately. The netmask and broadcast_address should be specified if you are using variable length subnet masks (VLSM), but may be omitted otherwise.

3. If you are using variable length subnet masks (VLSM), add the host's network number and subnet mask to /etc/netmasks.

If the new IP address places the system on a different network:

4. Place the host's new default gateway in /etc/defaultrouter, place new netmask in /etc/netmasks

5. Run route add default new_gateway for the new default gateway to take effect immediately.

6. Run route delete default old_gateway to delete the old default gateway from the routing table.

Verify Solaris 10 Multipathing/Configure SAN Disk

                                  
I was attempting to troubleshoot issues as a user was complaining about slow performance on a SAN disk. First thing that I did was check to ensure that there were not any performance issues on any disks that might have been causing this users issues


A quick iostat verified that everything was looking fine
iostat -cxzn 1

 

This box is running Veritas so lets check out the disks. Vxdisk list shows one Sun6140 disk.

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
Disk_0       auto:none       -            -            online invalid
Disk_1       auto:none       -            -            online invalid
SUN6140_0_1  auto:cdsdisk    diskname_dg02  diskname_dg online nohotuse

Luxadm is an utility, which discovers FC devices (luxadm probe), shut downs devives (luxadm shutown_device ...) runs a firmware upgrade (luxadm download_firmware ...) and many other things. In this instance I use luxadm to get the true device name for my disk

# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
  Node WWN:200600a0b829a7a0  Device Type:Disk device
    Logical Path:/dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2

I then run a luxadm on the device. Below you can see that I do indeed have two paths to the device.
1 controller = one path, 2 controllers = 2 paths

# luxadm display /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
  Vendor:               SUN
  Product ID:           CSM200_R
  Revision:             0619
  Serial Num:           SG71009283
  Unformatted capacity: 12288.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x1
    Maximum prefetch:   0x1
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
  /devices/scsi_vhci/ssd@g600a0b800029a7a000000dc747a8168a:c,raw
   Controller           /devices/pci@1f,4000/SUNW,qlc@5,1/fp@0,0
    Device Address              203700a0b829a7a0,1
    Host controller port WWN    210100e08bb370ab
    Class                       secondary
    State                       STANDBY
   Controller           /devices/pci@1f,4000/SUNW,qlc@5/fp@0,0
    Device Address              203600a0b829a7a0,1
    Host controller port WWN    210000e08b9370ab
    Class                       primary
    State                       ONLINE

Had I only had one path I would have run cfgadm. I would have seen that one of the fc-fabric devices would have been unconfigured. I then could have used cfgadm to configure it and enable my mulitpathing

# cfgadm
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c1                             scsi-bus     connected    unconfigured unknown
c2                             fc-fabric    connected    configured   unknown
c3                             fc-fabric    connected    configured   unknown

MPXIO Primer

Solaris I/O multipathing gives you the ability to set up multiple redundant paths to a storage system and gives you the benefits of load balancing and failover.

Need to enable MPXIO

Solaris 10 is the easier, because the mpxio capability is built-in. You just need to turn it on!

To enable it, edit the file /kernel/drv/fp.conf file. At the end it should say:

mpxio-disable="yes";

Just change yes to no and it will be enabled:

mpxio-disable="no";

Before multipathing, you should see two copies of each disk in format. Afterwards, you'll just see the one copy.

It assigns the next available controller ID, and makes up some horrendously long target number. For example:

Filesystem kbytes used avail capacity Mounted on /dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5 697942398 20825341 670137634 4% /test

Solaris Soft and Hard Limits

Hard limits are a kernel-configurable item, and users can't exceed them.

Soft limits are the user defaults, and users can change that using the 
ulimit program or the limit/unlimit builtins.

Basically, soft limits can be changed to anything up to the hard limit.
  
Think of soft limits as the warning barrier. When a user reaches 
the soft limit they will get an warning message but are still allowed 
to use more space up to the hard limit. Also, you can configure the 
system to set expiration times for users who have exceeded their soft limit.

You can set both soft and hard limits. The system will not allow a user
to exceed his or her hard limit. However, a system administrator may 
set a soft limit (sometimes referred to as a quota), which the user can temporarily
exceed. The soft limit must be less than the hard limit. 

Use ulimit -a to check soft limits, and ulimit -Ha to check hard limits.

These values will display as open files.

You can set these values by placing the following entries in /etc/system.

This will require a reboot

set rlim_fd_max=8192
set rlim_fd_cur=256 

rlim_fd_max is the hard limit, and rlim_fd_cur is the current limit (or soft limit)

Solaris Network Connection Performance Part I

Monitoring Solaris Ethernet Performance

Netstat -i

This option is used to diagnose the network problems when  the connectivity is there but  it is slow in response .

Values to look at:

    * Collisions (Collis)
    * Output packets (Opkts)
    * Input errors (Ierrs)
    * Input packets (Ipkts)

Network collision rate = Output collision counts / Output packets

Network-wide collision rate greater than 10 percent  will indicate

    *  Overloaded network,
    *  Poorly configured network,
    *  Hardware problems. 

Input Packet Error Rate = Ierrs / Ipkts.

If the input error rate is high (over 0.25 percent), the host is dropping packets. Hub/switch cables etc needs to be checked for potential problems. 

Name  Mtu  Net/Dest   Address    Ipkts     Ierrs Opkts     Oerrs Collis Queue
lo0   8232 loopback   localhost  43523390  0     43523390  0     0      0
hme0  1500 <hostname> <hostname> 561847305 886   483621617 0     0      0
qfe0  1500 <hostname> <hostname> 13480886  1     1521743   0     0      0

You can also use the following to determine the the ethernet interface link status,and speed on your interface. In the example below I am running these on hme0

ndd -set /dev/hme instance 0
ndd -get /dev/hme link_status
ndd -get /dev/hme link_mode
ndd -get /dev/hme link_speed

If you have only one ethernet interface, you can leave out the instance command. Otherwise, you can specify the hme instance number there. The results of the next three commands are either 1 or 0. In each case, the value means:

link_status:  0=down 1=up
link_mode:   0=half duplex  1=full duplex
link_speed:  0=10Mbps  1=100Mbps

You can also use the following netstat command. Replace ce0 with your ethernet adapter

>netstat -k ce0 | egrep 'link_speed|link_status|link_duplex'
lp_cap_asmpause 1 lp_cap_pause 0 link_T4 0 link_speed 1000
link_duplex 2 link_asmpause 0 link_pause 0 link_up 1 mac_mtu 1522

How to read this junk:
link_up - 0 down, 1 up
link_speed - speed in Mbit/s
link_duplex - 1 half duplex, 2 full duplex, 0 down