Citrix NetScaler VPX appliance fails to boot with the error: “can’t load ‘kernel’”

Problem

You’ve recently upgraded a NetScaler VPX appliance to version 11.0.63.16.nc and have been running the upgrade for a few months but noticed that the appliance fails to boot into the kernel after a restart halting at the following:

FreeBSD/x86 bootstrap loader, Revision NS1.2

Loading /boot/defaults/loader.conf

Unable to load a kernel!

can’t load ‘kernel’

Type ‘?’ for a list of commands, ‘help’ for more detailed help.

OK

You proceed with a restore of the virtual machine from a backup solution such as Veeam but noticed that all of the backups boot into the same error.

You attempt to try and find the /boot/defaults/loader.conf directory in the disks but it does not appear to exist:

lsdev -v

set currdev=disk1s1b

ls

set currdev=disk1s1d

ls

set currdev=disk1s1e

ls

You are able to locate the /nsconfig directory on one of the disks:

… and able to list the contents with the command ls nsconfig:

… but missing the other directories such as:

1. /var
2. /netscaler

*Reference: http://support.citrix.com/article/CTX200418

Solution

One of the ways to resolve this issue is to either restore from a backed up virtual machine that would actually boot properly or redeploy a new appliance then restore configuration files.  If you do not have either then the following steps can be used in an attempt to bring the appliance back to a bootable state:

Execute lsdev -v to list the available drives:

From the drives listed, we need to find the one that contains the nsinstall directory and in this example, the list is disk1s1e so execute the following to set the context of the prompt to that drive:

set currdev=disk1s1e

Use the ls command to list the directories:

Note the nsinstall directory in the screenshot above. Execute the ls nsinstall/ command to list the contents:

Note that the following 2 directories are listed:

1. 10.5.54.9.nc
2. 11.0.63.16.nc

This appliance was deployed as version 10.5.54.9.nc initially then upgraded several times to version 11.0.63.16.nc and what appears to have happened is that the boot configuration has gone missing.  Execute the commands:

ls 10.5.54.9.nc

ls 11.0.63.16.nc

Note that both directories contain quite a few files. The file we’re interested in is the NetScaler kernel file that is named ns-{version}-{build}.gz. Since both directories contain this file, we’ll attempt to load the newer 11.0.63.16.nc kernel file by executing the following:

unload

load disk1s1e:/nsinstall/11.0.63.16.nc/ns-11.0-63.16

With the kernel successfully loaded, proceed to type boot to boot into the kernel:

With the appliance booted and accessible, if you don’t have a configuration backup, proceed to get it now through the CLI or GUI:

What you’ll notice at this point is that if we attempt to restart the appliance, we’ll encounter the same error message:

can’t load ‘kernel’

… we faced earlier and this is because the boot configuration has been lost.  The next step to ensure that this boot configuration stays persistent is to reinstall the NetScaler binaries by executing the following:

shell

cd /var/nsinstall/

ls

cd 11.0.63.16.nc

./installns

You should now see the installation begin:

Proceed to reboot the appliance once the install has completed:

The appliance should now be able to boot into the OS and operate as it should:

Note that there is a good chance that:

1. The licenses on the appliance will need to be regenerated so log onto the web portal and see if the NetScaler is now labeled as NetScaler VPX (1)
2. SSL Certificates may be missing so they would need to get re-created (the files will still be there)
3. Various NetScaler features may be disabled so they will need to get re-enabled