How to troubleshoot when LUNs are missing from ESXi Server Environment

Problem Description

LUNs are reported as missing in ESXi Server Storage Adapter.
Common symptoms encountered are:

• CHAP Secret Mismatch
• Access Control List(ACL) was not properly set to LUN
• Masking the LUN Path from ESXi Host through HBA Adapter. This may prevent ESXi Host from accessing storage devices

Troubleshooting LUNs

We have to undertake some preliminary, pre-requisite troubleshooting steps to ensure that LUN missing is not due to the above mentioned three scenarios:
Link Connectivity b/w ESXi Host and Storage Subsystem

1. Link Connectivity verification using vmkping command

2. Available of ESXi HBA Adapter [ in online state] using esxcli iscsi adapter list command

Note: Ensure that the HOST ID on the array for the LUN (on ESX it shows up under LUN ID) is less than 255. Any LUN that has a HOST ID greater than 255 may not be shown under ESXi Host Storage Adapters.

i. Available device list from ESXi HBA Adapter [using esxcli storage core device list command – This should not go beyond 254+2[ local device and CD Drive ]

Now that we’re done with the preliminary steps, we can move on troubleshooting in case of the scenarios as mentioned above.

Case 1: CHAP Secret Mismatch

Check for CHAP Secret configuration from Storage Sub System and ensure that the same settings are configured from ESXi/ESX Hosts too. In case it differs, LUN Discover operation will fail, leading to LUN Missing issue.
Trouble Shooting Steps:
a. tail -f /var/log/syslog.log

Note: If you see the message like “Login failed” or “Login Authentication failed”, please go ahead and verify the CHAP Secret Settings on both Storage Subsystem and ESXi Host Storage Adapter.

Chap Secret Verification CLI Commands on ESXi Host:

b.esxcli iscsi adapter auth chap get –adapter vmhba37

c. esxcli iscsi adapter auth chap set –adapter vmhba37 –direction=uni –authname=”chap” –level=required –secret=”password”

Chap Secret Verification through vCenter Client on ESXi Host:

• Navigate to: Configuration tab -> ‘Storage adapters’
• Select the software adapter
• Click property and click chap button

Case 2: Access Control List (ACL) was not properly set to LUN

On the Storage Subsystem, please ensure that the LUN IQNs and access control list (ACL) set to allow the ESXi/ESX Host HBAs to access the iSCSI Targets.
Trouble Shooting Steps:
1.Run the following commands
a. vol –list (ClientName cmdlet) //This may vary among hardware vendors

b. Vol –info test1 //This may vary among hardware vendors

Note :
1.‘*’ in Initiator Group means unrestricted access
2.Specific IQN value of Initiator must be added into the LUN iSCSI Initiator Group Creation:
Adding Initiator to the Initiator Group:

ACL Update on LUN

Case 3: Masking the LUN Path from ESXi Host through HBA Adapter

Note: Ensure that there is no claim rule set for ESXi Host, which affects LUN Mapping.
Trouble Shooting Steps
1.Run the following commands in the ESXi Host
a) esxcli storage core claimrule list

b) esxcli storage core claimrule remove –rule 101
Note: Remove unwanted rules using above command
Troubleshooting the network issue using tcpdump with the below mentioned arguments:

• “tcpdump-uw -w /tmp/t.cap port 3260 -i vmk1”

Happy Troubleshooting..!