FatMin

X11 No Worky - Worky

Yikes, was banging my head against a wall trying to get X11 forwarding working on my RHEL 6.1 box. Every time I ssh'd in and attempted to pop an xwindow back to my fedora desktop, I was greeted with the error message below.

# Error: no display specified

Ok, I thought to myself -- X11 forwarding is not enabled in sshd -- lets go fix it. However, when I checked /etc/ssh/sshd_config, I found that X11 forwarding was enabled. See the config item below, these needs to exist, and not set to no.

X11Forwarding yes

Ok so lets check what the display variable is set to. So I ran the command below, which retured absolutely nothing, which meant that its not set to anything at all.

# echo $DISPLAY

So I check my local box and check to make sure that forwarding is enabled. Which it is

#grep X11 /etc/ssh/ssh_config
ForwardX11 yes

After I bit of research on google - I found a bugzilla that sugguested I make sure that the following rpm was installed - xorg-x11-xauth.x86_64.  It wasn't..

yum install xorg-x11-xauth.x86_64

Then logout and back in again. You will be now be greeted by the line below.

/usr/bin/xauth:  creating new authority file /root/.Xauthority

Now check your display variable.

# echo $DISPLAY
localhost:10.0

Ah, much better

 

 

 

RHEL- Find UUID of Hard Disks

Anyone who has added and removed multiple disks from a RedHat server knows very well that your disks may not always enumerate exactly the same way after a reboot. You then have to resort to mounting up your filesystems to a temporary mount point to see exactly whats in them, and were they really need to be mounted.

Want to know if UUIDS are being used on your linux box, well just cat /etc/fstab, and if you see somthing similar to what's below, then you are using UUIDS, instead of traditional disk device names.

_{UUID=3fa4603e-9874-4f47-ae1c-3f7715a54238 /                       ext4    defaults,user_xattr        1 1
UUID=483c34a4-b3ec-4860-854f-b9e5b6a6efac /boot                   ext4    defaults        1 2
UUID=7cab6648-b3f3-4aaa-bb2b-d32b78156aab /var                    ext4    defaults        1 2
UUID=7b05f0a9-18d5-42e5-b259-78ba3a8cc1b7 swap                    swap    defaults        0 0}

One way of mapping device UUIDS back to device names is the blkid command -- usage and output below.

_{[root@fedora15 ~]# blkid
/dev/sda1: UUID="483c34a4-b3ec-4860-854f-b9e5b6a6efac" TYPE="ext4"
/dev/sda2: LABEL="_Fedora-15-x86_6" UUID="3fa4603e-9874-4f47-ae1c-3f7715a54238" TYPE="ext4"
/dev/sda3: UUID="7b05f0a9-18d5-42e5-b259-78ba3a8cc1b7" TYPE="swap"
/dev/sdb1: UUID="7cab6648-b3f3-4aaa-bb2b-d32b78156aab" TYPE="ext4"
/dev/sdb2: UUID="rJfNaK-e3Xp-n3qm-4aXM-BfKF-g7sg-Kwm33a" TYPE="LVM2_member"
/dev/sdc1: LABEL="raid1" UUID="34ed4ffd-cc4a-4b40-892f-6d7714fe7f4e" TYPE="ext3"
/dev/mapper/vm_vg-v1: UUID="2d96aa43-b5b8-4185-8014-323ad8a07a0d" TYPE="ext4"}

You can also do an ls on the following directory and get the same information.

_{[root@fedora15 ~]# ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Sep 25 21:41 2d96aa43-b5b8-4185-8014-323ad8a07a0d -> ../../dm-0
lrwxrwxrwx 1 root root 10 Sep 25 20:51 34ed4ffd-cc4a-4b40-892f-6d7714fe7f4e -> ../../sdc1
lrwxrwxrwx 1 root root 10 Sep 25 20:51 3fa4603e-9874-4f47-ae1c-3f7715a54238 -> ../../sda2
lrwxrwxrwx 1 root root 10 Sep 25 20:51 483c34a4-b3ec-4860-854f-b9e5b6a6efac -> ../../sda1
lrwxrwxrwx 1 root root 10 Sep 25 20:51 7b05f0a9-18d5-42e5-b259-78ba3a8cc1b7 -> ../../sda3
lrwxrwxrwx 1 root root 10 Sep 25 20:51 7cab6648-b3f3-4aaa-bb2b-d32b78156aab -> ../../sdb1}

If you are so inclined, you can also get the UUID and filesystem label (if there is one) with the tune2fs command

tune2fs -l /dev/sda1

_{Filesystem volume name:   <none>
Last mounted on:          /boot
Filesystem UUID:          483c34a4-b3ec-4860-854f-b9e5b6a6efac
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)}

_{....truncated...}

Oh and if you are running Fedora 15 and are wondering what the heck rootfs is (as I was), here is an explanation. More junk that I hope never gets into RHEL.

Oh and if you are really bored, and would like to know more about what a UUID is go here

 

 

e1000 device eth4 does not seem to be present, delaying initialization

The moral of the story today is beware of servers that have had a long history of changing networking configurations and network interfaces, as you never know what you are going to run into.

While working on a RHEL 5.4 box and attempting to make few minor changes to eth4 and up the interface, I was smacked with the error below.

_{e1000 device eth4 does not seem to be present, delaying initialization}

This was very much suck.

Ifconfig-eth4 contained what you see below

_{DEVICE=eth4
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.13.11
NETMASK=255.255.255.0}

So first i hit up /etc/modprobe.conf and verifed that the correct driver module was being aliased for the interface.

_{alias eth4 e1000}

Ok check. Modprobe.conf looks good.

Now re-initialize driver

_{modprobe eth100}

Then onto check lspci to make sure that the driver initialized all the interfaces. Note that there are two different versions of the 82546EB driver, which tells me that the driver was only initialized for two of the interfaces initially.

_{2a:01.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
2a:01.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
2f:02.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
2f:02.1 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)}

Turned my attention back to the ifconfig-eth4 and the fact that there was not a MAC address defined in the config file. Grabbed the mac address out of /etc/sysconfig/hwconf and added it to the ifconfig-eth4 config file

_{class: NETWORK
bus: PCI
detached: 0
device: eth4
driver: e1000
desc: "Intel Corporation 82546GB Gigabit Ethernet Controller"
network.hwaddr: 00:04:23:cb:89:2e
vendorId: 8086
deviceId: 1079
subVendorId: 8086
subDeviceId: 1179
pciType: 1
pcidom:    0
pcibus: 2f
pcidev:  2
pcifn:  0}

Then ifup works. Day is saved.

RHEL6 - How To Configure an NFS Server

Do you know whats almost as delicious as a tasty waffle with pure AAA Vermont maple syrup smothered all over it. Neither do I.

However I did run across a very well written blog post on how to configure an NFS server on RHEL6.

Wierd, is it a NFS server or an NFS server, the latter sounds and looks better but i dont think its correct.

Anyway link below:

http://aaronwalrath.wordpress.com/2011/03/18/configure-nfs-server-v3-and-v4-on-scientific-linux-6-and-red-hat-enterprise-linux-rhel-6/

Meld -- Graphical Merge and Diff Tool for Linux

Meld is a very nice tool for those who cant stand to use diff to compare the differenced between two text files. If you have either the rpmforge repo or the epel repo configured you can install it via yum. Note: I am using Centos 5.4 which is a bit dated, so you might find a newer version in your repo depending on the OS that you are running.

Anyway, once installed you can compare two files like so

_{>meld syslog.conf syslog.conf.orig}

This  will pop up a side by side comparison of the two files in an Xwindow, so make sure that you have forwarding enabled. You might need to ssh -X to your server.

Meld is not new by any means - the project's sourceforge page shows that its the latest meld version (1.5) was release. The version that I installed via my rpmforge repo is even older (1.1.5). Again I am running Centos 5.4.

Meld can also be used to merge two files together and perform a diff on directory structures, however just having a nice looking diff is enough for me. 

http://meld.sourceforge.net/

 

Recovering from failed vxevac

If you have ever evacuated disks in Veritas, every so often this will happen to hang.  Usually you terminate your session or who knows what.  Kinda like Joe Girardi's willingness to sacrifice outs for no good reason every time the Yankees hottest hitter is at the plate.  It happens, you can't explain it, you move on.  Back to technology - vxtask list shows no tasks, but you get errors trying to rerun the failed evac. 

 

For example:

Plex %5 in volume rman is locked by another utility

Plex rman-01 in volume rman is locked by another utility

Subdisk rman_7_tmp-01 in plex rman-01 is locked by another utility

vxprint -hf is our best friend, as it shows you any flags that are set

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   ATT1    -

pl %5           rman   ENABLED  11719399168 -     TEMPRM   SDMVTMP -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

We can see that we have flags set on the temporary plex (from the failed evac), the subdisk for the temporary plex, the main plex, the subdisk in the main plex, as well as the volume itself.  We need to clear flags to be able to finish re-start our evac.  I will also cut the lines on the vxprint that don't change for the purpose of shortening this post.

vxmend -g rman_dg clear all rman %5

So we cleared the volume and temp plex flags, here's the vxprint -htf output afterwards

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl %5           rman    ENABLED  11719399168 -     TEMPRM   -       -

sd rman_6-01 %5         ENABLED  1953232896 9766166272 -    SDMVDST -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

So now with the flags cleared we can remove the temporary plex

vxplex -g rman_dg -o rm dis %5

 

And once again our new vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - SDMVSRC -

 

Great, now down to two flags, the one on the plex and the one on the source disk of our original evac.  Clearing flags from subdisks is a lot trickier than clearing flags from volumes and plexes.  Because the tutil0 flga is already set, we will need to force the clear.  We clear by setting it to "".

vxedit -g rman_dg -f set tutil0="" rman_6_tmp-01

 

Once again, vxprint -htf

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   SDMV1   -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

 

And lastly, we clear the flag on the plex.  Why in this order?   Because I'm writing this up after I fixed my issues.  In the interest of not editing vxprint outputs, it's like this.  In retrospect, this could have been cleared with the first one we ran in the beginning.

vxmend -g rman_dg clear all rman rman-01

 

And finally, the way a vxprint -htf should look when all is healthy.

v  rman    fsgen        ENABLED  15625864960 -     ACTIVE   -       -

pl rman-01 rman    ENABLED  15625864960 -     ACTIVE   -       -

sd rman_1-01 rman-01 ENABLED 1953234688 0     -        -       -

sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 -   -       -

sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 -   -       -

sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 -   -       -

sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 -   -       -

sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 - -     -

sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 - -    -

sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 -  -       -

 

At this point, feel free to proceed with your evac again.  If you're wondering what the putil and tutil fields are, here is what I found courtesy of Symantec:

http://www.symantec.com/business/support/index?page=content&id=TECH15609

 

Guest Authored By: @momkvi

 

MegaRaid Cards Via CLI Part II

MegaRAID® is LSI's line of SATA/SAS Storage Controller. MegaCLI64 is LSI's command line raid controller management software for these controllers.

You can get a bit more info on it if you check out my original Megraid blog post here.

Anyway here are some basic commands. Some of these are mentioned in my original post.

The command below gives you detailed configuration and settings info on your controller.

_{./MegaCli64 -AdpAllInfo -aALL}

Use the command below to show all Logical Drives.

_{./MegaCli64 -LDInfo -LALL -aALL}

Use the command below to list all Physical Drives.

_{./MegaCli64 -PDlist -aALL}

Use the command below to display detailed adapter and device info.

_{./MegaCli64 -AdpAlILog -aALL}

Use the command below to offline a drive.

_{./MegaCli64 -PDOffline -PhysDrv [E:S] -aN}

Mark a drive as missing.

_{./MegaCli64 -PdMarkMissing -physdrv 8:1 -a0}

Prepare a drive for removal.

_{./MegaCli64 -PdPrpRmv -physdrv 8:1 -a0}

Replace a missing drive.

_{./MegaCli64 -PdReplaceMissing -physdrv 8:1 -array0}

 

Tracking Down I/O Hotspots in Linux w/ Veritas

Got a heads of from a DBA today stating that one of the their database servers was running hot from an i/o perspective. So in order to troubleshoot jumped on the server and ran an iostat

_{>iostat -d -x 5 3}

Based on the output below i was able to determine which disks where the most utilized by looking at the last column which is %utilized. Two other columns to take note of are await and svctime,where await time is the average response time in ms for an i/o request to the device, including any time spent waiting in a queue. Svctime is the average time it took to service a request after it was sent to the disk and out of the queue. In this case service times are low so i can pretty much rule out SAN issues.

_{VxVM65519     0.00     0.00 230.80 20.80  5753.60   556.80    25.08     1.77    6.94   3.50  99.08
VxVM65516     0.00     0.00 257.60 21.80  6022.40   467.20    23.23     1.68    6.02   3.01  93.22
VxVM65515     0.00     0.00 265.80 18.80  6563.20   364.80    24.34     1.42    4.89   2.69  94.58
VxVM65513     0.00     0.00 233.20 24.00  6032.00   969.60    27.22     1.41    5.47   2.99  88.92
VxVM65493     0.00     0.00 308.80 21.00  7590.40   944.00    25.88     1.74    5.27   2.52  98.14
VxVM65492     0.00     0.00 262.80 20.40  6502.40   716.80    25.49     1.76    6.10   2.94  99.22}

In this case svctimes are ok so i know that the disks are performing well, however i do know that the hot disks are under Veritas control, so i take a quick look at the disk groups on the box

_{>vxdg list}

Then do i vxstat on each disk group individually to look for the ones with high number or reads and rights. Then i can advise the dbas which specific database is causing the load on the box.

_{>vxstat -g example_dg -i 1}

Redirect Linux Console to HP ILO via SSH

Sometimes the ILO web interface is painfully slow to use when you are just trying to get on the console of a server. To get around this I have been taking advantage of the fact that you can SSH to the ILO and access the server console from there.

In order to do this you must add the following to the bottom of /etc/inittab.

S1:2345:respawn:/sbin/agetty 115200 ttyS1 vt100

Then you must add ttyS1 to the bottom of /etc/securetty. If you don't you will not be able to log in as root from the serial console. So don't forget this step.

Then ssh into the ilo console -- and run the following command.

</>hpiLO-> VSP

Note that this works on the ILO2, and possibly on the ILO3. It does not work on ILO1.

 

 

 

 

RedHat Performance Tuning: The Man Pages

This my first post following my week long Redhat Monitoring and Performance Tuning Course (RH442). This course was rather intense and dealt mostly with concepts and theories rather and was not at all a how-to on performance tuning. I am going to do my best to document some of what I learned.  Lets start with the documentation...

Man Pages, are the extensive documentation that comes installed by default on a linux system. They are seperated into 9 books or sections.

1. General Commands
2. System Calls
3. C Library Functions
4. Special FIles and Drivers
5. FIle Formats
6. Games/Screensavers
7. Protocols
8. System Admin Commands and Daemons

To access a particular book you specify the book name after the man command. For example:

>man 1 printf

which is different than

>man 3 printf

To find all the man pages related to printf, you run a the following

>man -k printf

Which gives you the output below listing all man pages related to printf. These specific ones are all on the the printf command specifically. Similarly you can use the apropos command and get the same output

To see only the man pages for printf run the command below

>man -f printf

Which will return only what you see below.

printf (1)           - format and print data
printf (1p)          - write formatted output
printf (3)           - formatted output conversion
printf (3p)          - print formatted output