FatMin

Recovering from failed vxevac

If you have ever evacuated disks in Veritas, every so often this will happen to hang.  Usually you terminate your session or who knows what.  Kinda like Joe Girardi's willingness to sacrifice outs for no good reason every time the Yankees hottest hitter is at the plate.  It happens, you can't explain it, you move on.  Back to technology - vxtask list shows no tasks, but you get errors trying to rerun the failed evac. 

 

For example:

Plex %5 in volume rman is locked by another utility

Plex rman-01 in volume rman is locked by another utility

Subdisk rman_7_tmp-01 in plex rman-01 is locked by another utility

vxprint -hf is our best friend, as it shows you any flags that are set

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   ATT1    -

pl %5           rman   ENABLED  11719399168 -     TEMPRM   SDMVTMP -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

We can see that we have flags set on the temporary plex (from the failed evac), the subdisk for the temporary plex, the main plex, the subdisk in the main plex, as well as the volume itself.  We need to clear flags to be able to finish re-start our evac.  I will also cut the lines on the vxprint that don't change for the purpose of shortening this post.

vxmend -g rman_dg clear all rman %5

So we cleared the volume and temp plex flags, here's the vxprint -htf output afterwards

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl %5           rman    ENABLED  11719399168 -     TEMPRM   -       -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

So now with the flags cleared we can remove the temporary plex

vxplex -g rman_dg -o rm dis %5

 

And once again our new vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

Great, now down to two flags, the one on the plex and the one on the source disk of our original evac.  Clearing flags from subdisks is a lot trickier than clearing flags from volumes and plexes.  Because the tutil0 flga is already set, we will need to force the clear.  We clear by setting it to "".

vxedit -g rman_dg -f set tutil0="" rman_6_tmp-01

 

Once again, vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

 

And lastly, we clear the flag on the plex.  Why in this order?   Because I'm writing this up after I fixed my issues.  In the interest of not editing vxprint outputs, it's like this.  In retrospect, this could have been cleared with the first one we ran in the beginning.

vxmend -g rman_dg clear all rman rman-01

 

And finally, the way a vxprint -htf should look when all is healthy.

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   -       -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

 

At this point, feel free to proceed with your evac again.  If you're wondering what the putil and tutil fields are, here is what I found courtesy of Symantec:

http://www.symantec.com/business/support/index?page=content&id=TECH15609

 

Guest Authored By: @momkvi

 

Resolving SCSI Reservation Conflicts/Locks in Vsphere 4.0

A few days ago we got hit with a ton of alerts which indicated that a handful of VMs were down, then up, and down again. This cycle continued several times.

At first, after a bit of digging through logs, we thought that the issue was related to scsi reservation errors, but we were already compliant with the best practices for 3PAR mentioned here. So we dug deeper and found that we were in fact suffering from SCSI locks. Go here for more information.

 According to VMware...

"The second category involves acquisition of locks. These are locks related to VMFS specific meta-data (called cluster locks) and locks related to files (including directories). Operations in the second category occur much more frequently than operations in the first category. The following are examples of VMFS operations that require locking metadata:

• Creating a VMFS datastore
• Expanding a VMFS datastore onto additional extents
• Powering on a virtual machine
• Acquiring a lock on a file
• Creating or deleting a file
• Creating a template
• Deploying a virtual machine from a template
• Creating a new virtual machine
• Migrating a virtual machine with VMotion
• Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk

To resolve a SCSI Lock, log into each of your ESX boxes and run the following command. 

# esxcfg-info | egrep -B5 "s Reserved|Pending

Look for the output below, as the host that has "Pending Reservation" value greater than one is causing the lock.

|----Pending Reservations................ 1

Now reset the lun.

vmkfstools --lock lunreset /vmfs/devices/disks/vml.02000000006001c230d8abfe000ff76c198ddbc13e50455243

Veritas Enterprise Administrator Not Displaying Objects/Agents

Its been a while since I have done much with Veritas Storage Foundation, so I was at a bit of a loss after firing up the VEA gui on a fresh install on Centos 5.4, and not seeing any of the agents that I was used to seeing. A quick check on the command line showed that the Storage Agent was in fact running. A vxdisk list displayed my disks without issue, but the GUI was blank except for the server name.

A quick google search lead me to this Symantec KB article...

http://seer.entsupport.symantec.com/docs/302156.htm.

While this article was helpful it did not solve my issues as I mentioned above, my storage agent was in fact running without issue. However I was able to verify similar errors in my vxsis.log

:
Thu Mar 27 12:40:03 2008:3342:get_objects_by_type: Error in GetObjects : 0xc1000039
Thu Mar 27 12:40:03 2008:3342:database::HandleILCacheRequest:Error in get_objects_by_type() : 0xc1000039
Thu Mar 27 12:40:03 2008:3342:rpc_object_fetch: Could not fetch objects

So I started searching on the errors above, and ran into this post on the Symatec Forum, where one of the posters sugguest that the /etc/host file might be the culprit.

Sure enough a quick check of my ./etc/hosts revealed that my servers ip and FQDN were not present. I corrected and then followed the steps below.

First I stopped VEA Service

> /opt/VRTS/bin/vxsvcctrl stop    

Then I reconfigured Veritas

>/opt/VRTS/install/installsf -configure

I restarted my VEA GUI and reconnected and found all was as it should be.

Raid Levels Explained and Simplified

As a Systems Administrator, I deal with Raid 1(mirroring) pretty much exclusively. Hell nowadays when building a server the server automatically mirrors your Operating System disks for you, which means that you do not even need to understand what is happening behind the scenes. You just pop your two drives in your server and go. However the world of the San Administrator is much more complicated.

First off its important to know that RAID stands for either "Redundant Array of Independent Disks", or more commonly  "Redundant Array of Inexpensive Disks". Either way you slice it (pun intended) the basic idea of RAID is to combine multiple hard disks to either increase performance or increase redundancy. 

Before I get started its important to introduce the term LUN. A LUN is a logical disk that consists of raw physical disk space. LUNs are created as a basic part of the storage provisioning process. They are presented across a SAN to a server as a single physical disk.

Note that the title of this article is "Raid Levels Explained and Simplified", and when I say Simplified I mean it. I am going to give a brief overview of most of the common RAID levels and then present a weakness and strength. Scroll down to the bottom of the article for links to more in depth articles and web pages. 

RAID 0: Striped...No Fault Tolerance

OK, in my opinion, and in the opinion of many other, RAID 0 is not even RAID, because there is no redundancy. If a disk fails, you are toast. Basically your take a slice of two disk or more disks and create a LUN. For example lets say that you as the Sysadmin request 1 80GB disk from your local SAN Admin. In the scenario below your SAN guru would carve 8 10GB blocks and present them in order (block 1,2,3,4,5,6,7,8) to you as a single LUN. RAID 0 provides good read and write performance. In the end RAID 0 is striping which is the most important thing that you probably need to know about it.