When running a development kernel (eg: FreeBSD-CURRENT), such as a kernel under
extreme conditions (eg: very high load averages, tens of thousands of connections,
exceedingly high number of concurrent users, hundreds of jail(8)s, etc.), or
using a new feature or device driver on FreeBSD-STABLE (eg: PAE), sometimes a kernel will panic. In the event that it does,
this chapter will demonstrate how to extract useful information out of a crash.
A system reboot is inevitable once a kernel panics. Once a system is rebooted, the
contents of a system's physical memory (RAM) is lost,
as well as any bits that are on the swap device before the panic. To preserve the bits in
physical memory, the kernel makes use of the swap device as a temporary place to store
the bits that are in RAM across a reboot after a crash. In doing this, when FreeBSD boots
after a crash, a kernel image can now be extracted and debugging can take place.
Note: A swap device that has been configured as a dump device still acts as a
swap device. Dumps to non-swap devices (such as tapes or CDRWs, for example) are not
supported at this time. A ``swap device'' is synonymous with a ``swap partition.''
To be able to extract a usable core, it is required that at least one swap partition
be large enough to hold all of the bits in physical memory. When a kernel panics, before
the system reboots, the kernel is smart enough to check to see if a swap device has been
configured as a dump device. If there is a valid dump device, the kernel dumps the
contents of what is in physical memory to the swap device.
Before the kernel will dump the contents of its physical memory to a dump device, a
dump device must be configured. A dump device is specified by using the dumpon(8) command to
tell the kernel where to save kernel crash dumps. The dumpon(8) program must
be called after the swap partition has been configured with swapon(8). This is
normally handled by setting the dumpdev variable in rc.conf(5) to the path
of the swap device (the recommended way to extract a kernel dump).
Alternatively, the dump device can be hard-coded via the dump clause in the config(5) line of a
kernel configuration file. This approach is deprecated and should be used only if a
kernel is crashing before dumpon(8) can be
executed.
Tip: Check /etc/fstab or swapinfo(8) for a list
of swap devices.
Important: Make sure the dumpdir specified in rc.conf(5) exists
before a kernel crash!
# mkdir /var/crash
# chmod 700 /var/crash
Also, remember that the contents of /var/crash is sensitive
and very likely contains confidential information such as passwords.
Once a dump has been written to a dump device, the dump must be extracted before the
swap device is mounted. To extract a dump from a dump device, use the savecore(8) program.
If dumpdev has been set in rc.conf(5), savecore(8) will be
called automatically on the first multi-user boot after the crash and before the swap
device is mounted. The location of the extracted core is placed in the rc.conf(5) value dumpdir, by default /var/crash and will
be named vmcore.0.
In the event that there is already a file called vmcore.0 in
/var/crash (or whatever dumpdev is
set to), the kernel will increment the trailing number for every crash to avoid
overwriting an existing vmcore (eg: vmcore.1). While debugging, it is highly likely that you will want
to use the highest version vmcore in /var/crash when searching for the right vmcore.
Tip: If you are testing a new kernel but need to boot a different one in order
to get your system up and running again, boot it only into single user mode using the
-s flag at the boot prompt, and then perform the following
steps:
# fsck -p
# mount -a -t ufs # make sure /var/crash is writable
# savecore /var/crash /dev/ad0s1b
# exit # exit to multi-user
This instructs savecore(8) to extract
a kernel dump from /dev/ad0s1b and place the contents in /var/crash. Don't forget to make sure the destination directory /var/crash has enough space for the dump. Also, don't forget to
specify the correct path to your swap device as it is likely different than /dev/ad0s1b!
The recommended, and certainly the easiest way to automate obtaining crash dumps is to
use the dumpdev variable in rc.conf(5).
