Centralize ESXi Core Dumps

In my environment the majority of the hosts boot from SD cards, so persistent log storage is a big deal. I recently ran into a PSOD issue. VMWare requested the coredumps, which of course, I did not have. Thankfully we were able to sort out my issue via the PSOD screenshot I took from the console.

Obviously wanting to avoid this scenario in the future, I set off to find the best way to keep this from happening again. Interestingly, I found an easy way to do this. There is no direct way via GUI or even advanced host settings. Eventually I was able to get it setup. Below is my method. Enjoy.

Setup vCenter

Note: We are running VCSA 6.5 in my environment so the core dump location will be different than if you are using a windows based vCenter server.

First we setup the esxi dump collector on the vCenter server. While logged into the vCenter web interface with an administrator account, click on the Administration link in the vCenter home menu. Next under the Deployment section, click the System Configuration link. From here choose Services under the system configuration header on the left side of the screen. Finally choose “VMware vSphere ESXi Dump Collector.”

From here, click the Manage tab. Then click the pencil icon to change the startup type and set it to run automatically. Next hit the green play button. I can confirm this will not in anyway affect the vCenter server itself, so no worries about affecting production. I kept the defaults for port and size. As noted, if you try to change either of these two settings, it does require a vCenter reboot.

That’s it! Now the vCenter server is ready. Next we need to change the setting on the esxi hosts. I was surprised to find that this must be done at the command line via esxcli. There is no way via advanced settings on the host. The process is straight forward but heavy with administrative overhead. You must ssh to each host and run the following commands. You can also use Host Profiles but that is beyond the scope of this particular post.

Setup ESXi Hosts
### Gets the current coredump configuration ###
esxcli system coredump network get

### Sets server address and port to send kernel dump to ###
### vmk0 is management network in my environment
esxcli system coredump network set -v vmk0 -i vCenter IP -o 6500

### Enables sending of coredumps to vCenter server ###
esxcli system coredump network set -e true

### Shows new core dump configuration ###
esxcli system coredump network get

It now lists the settings compared to the first time we ran it.

 ### Sends test coredump to vCenter Server ###
esxcli system coredump network check

OK, we are done with the host setup. Now to confirm the last step on the vCenter server. Ssh to the vCenter server and check the following log file:

/var/log/vmware/netdumper/netdumper.log

You should see similar entries:

“Posting back a status check reply to…”    SUCCESS!!!

Quick credit to @lamw for this https://blogs.vmware.com/vsphere/2012/12/network-core-dump-collector-check-with-esxcli-5-1.html. This article lays out the steps of this setup in great detail.

Automate it

Having gotten it to work on one host, I now had to figure out how to get it working on the rest of my hosts with less typing and less time. Enter powercli. Borrowing a lot from this guy’s technique, I used the Get-EsxCli and a foreach loop to apply the above settings to each host. At the same time, I did a tail -f on the log file to witness the fruits of my labor. It was a good feeling knowing I had saved myself so much work. So without further adieu: