So a user calls and says they aren't working and you wonder, did the virtual server go down, does the virtual server have any active members.  So here are some tips on troubleshooting.  If you have any yourself, please add them.You may want to just jump right into the logs and look to see if they are disconnected.  SSH into the box and run:

Quote

[wildweaselmi@usmi1slbdv03:Active:In Sync] ~ # zgrep app1 /var/log/ltm..gz

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 notice mcpd: 01070727:5: Pool /Development/pool.dev-app1direct.zahsystems.com.8201 member /Development/MI1SVDWB406:8201 monitor status up.

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 notice mcpd: 01070727:5: Pool /Development/pool.dev-app1directadmin.zahsystems.com.8201 member /Development/MI1SVDWB406:8201 monitor status up.

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 notice mcpd: 01071681:5: SNMP_TRAP: Virtual /Development/vs.si.dev-app1direct.zahsystems.com.https has become available

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm: 01010221:3: Pool /Development/pool.dev-app1direct.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm: 01010221:3: Pool /Development/pool.dev-app1directadmin.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm1: 01010221:3: Pool /Development/pool.dev-app1direct.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm1: 01010221:3: Pool /Development/pool.dev-app1directadmin.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm2: 01010221:3: Pool /Development/pool.dev-app1direct.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm2: 01010221:3: Pool /Development/pool.dev-app1directadmin.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm3: 01010221:3: Pool /Development/pool.dev-app1direct.zahsystems.com.8201 now has available members

/var/log/ltm.1.gz:Jul 17 01:06:04 usmi1slbdv03 err tmm3: 01010221:3: Pool /Development/pool.dev-app1directadmin.zahsystems.com.8201 now has available members

  Don't forget to try and search the logs without .gz at the end since those are your more recent messages.  When I want to search for a specific date and possibly key word like "was down for" I would run something like this

zgrep uszah1lvpap36 /var/log/ltm | zgrep "Jun 13" | zgrep "was down for"

 

Capture using tcpdump

First make sure an instance isn't already running on the appliance

ps cax | grep tcpdump

Next you should make sure you aren't going to kill appliance by over stressing it so make sure the appliance isn't already stressed out and run top(look at current CPU and Memory Utilization)

To help not stress the appliance you should not use interface 0.0 (which means capture traffic over all interfaces). You should instead be specific with your interface. Run ifconfig and use VLAN as interface to capture traffic (note interface is case sensitive)

Here is an example of capturing traffic on all interfaces

tcpdump -nni 0.0 -s0 host VirtualServerIPaddress -w /var/tmp/wildweaselmi.cap

Now decrypt and put contents in a readable txt file

ssldump -Aed -nr /var/tmp/dennis.cap -k /config/filestore/files_d/Common_d/certificate_key_d/:Common:crt.wildcard.thezah.com.2048.key_70273_1 > /var/tmp/wildweaselmi.txt

Show me all nodes on LTM using CLI

tmsh list /ltm node /partition1/*

or

tmsh list ltm node all

or even better...

tmsh list /ltm node /Partition1/* | grep -B 1 -A 4 10.40.91

Show me what pools a node is in using the CLI

tmsh list ltm pool /Partition/* one-line | grep '10.40.91.14'

Show me what virtual server a pool belongs to using CLI

tmsh list ltm virtual /Production/* | grep -B 5 'pool.tidalwave.zahsystems.com'

Sometimes you have multiple nodes that you would like to capture traffic on since you aren't positive which node (if any) is having a problem

Example

tcpdump -vv -i 0.0 -nn -p -e "(port 80 or port 443) and ((host 172.17.30.100 or host 172.17.30.101 or host 172.17.30.102) and ((host 172.17.31.200 or host 172.17.31.201 or host 172.17.31.202))"

If capturing for Wireshark

tcpdump -vv -i 0.0:nnn -p -s0 -w /var/tmp/appname1.cap "(host 10.40.91.14 or host 10.40.91.15 or host 10.40.91.16 or host 10.40.91.27)"

Just so you know you don't have to seperate the options so instead group them all together.

tcpdump -vvnni 0.0 -p -s0 -w /var/tmp/appname1.cap "(host 10.40.91.14 or host 10.40.91.15 or host 10.40.91.16 or host 10.40.91.27)"

It's best to use the specific interface (0.0 is every interface). Do a ifconfig to determine what interface you want to use.

Here are some other tips to troubleshooting on the F5 LTM

tcpdump for SSLv3

tcpdump -i any 'tcp port 443 and host 10.40.64.141 and tcp[((tcp[12]>>4)*4)+9:2]=0x0300' -w /var/tmp/staging.cap

SSL Handshake Failure

tmsh modify svs db bigd.debug ( value enable )

tail -f /var/log/bigdlog [grep 'return ssl false'

tmsh modify svs db bigd.debug ( value disable )

tmsh list svs db bigd.debug

F5 ssldump

Directions to run TCPDUMP and then decrypt into text file

First run the tcpdump to get the trace saves as a cap file

Then run ssldump to decrypt the cap file

ssldump -Aed -nr /var/tmp/staging.cap -k /config/filestore/files_d/Common_d/certificate_key_d/:Common:crt.any.any.zahsystems.com.2048.key_16532_1 > /var/tmp/staging.cap.txt

ssldump -Aed -nr /var/tmp/capture.cap -k /config/ssl/ssl.key/www-ssl.key

Directions to run SSLDUMP and look at handshake

Run TCPDUMP

tcpdump -vv -s0 -ni 0.0 host 64.74.98.99 -w /var/tmp/staging.cap

Then run SSLDUMP to see handshake

ssldump -nr /var/tmp/staging.cap

To write to file

ssldump -nr /var/tmp/staging.cap > /var/tmp/staging.cap.txt

Health Monitor Testing

Checks the server for file and passes host header info, -k bypasses ssl cert mismatch

curl --verbose --header 'Host: grc-mod.zahsystems.com' 'https://server01.nao.global.zahsystems.com/sa-health/f5chk.html' -k

Passes host and gives user info

curl -fNs --ntlm -k -v --user nao\dennis:superman https://hostname.com -H "Host: zahsystems.com"

Also don't forget to use https://ihealth.f5.com

Generate the qkview file by running

tmsh run /util qkview

To view the list of qkview command line options, type the following command:

qkview -h

For example, to reduce the performance burden of qkview, you can run qkview at the lowest possible priority by typing the following command:

nice -n 19 qkview

Note: For heavily-loaded systems, qkview may take a long time to finish running when using nice -n 19.

BIG-IP iHealth Diagnostics take advantage of the technical knowledge of experienced F5 engineers to assist you in implementing the best practices for your system. Using advanced diagnostics, iHealth can determine when your system is operating outside of normal levels so you can take the necessary steps to improve performance. BIG-IP iHealth Diagnostics also check for security issues and recommend patches or software updates. F5 recommends upload a qkview file to the BIG-IP iHealth system at least once a month, since F5 updates the BIG-IP iHealth known issues and best practices on a weekly basis.

The prioritized results provide tailored feedback about configuration issues or code defects, and provide a description of the issue, recommendations for resolution, and a link to further information in the AskF5TM Knowledge Base.

This customized diagnostic information enables you to take the recommended action, and in many cases, can help you resolve common configuration issues without the need to contact F5 Technical Support. If you require assistance from F5 Technical Support, your BIG-IP iHealth data will be available to F5 engineers for faster resolution.