Best way to setup a VirtualBox to avoid system crashes on a reboot

We now use permanent VirtualBox OSes to run part of our infrastructure.

We ran in all sorts of issues with those, but one thing for sure is that you do not want your OS hard drive to fill up because a (re)boot may not be possible after that. This is because the drive is likely to go bust if completely full.

You should, of course, have tools to let you know how filled up your hard drive is to make sure to catch such issues early. In our case, we had a Jira instance that filled up the drive mainly because we made one backup of the database per day. One day, one of those backup ended that instance as it was now completely full. So full that the Linux Operating system decided to auto-umount and re-mount the drive as a Read-Only instance.

Once in that state, though, it is unlikely that you will be able to reboot properly. The shutdown will generate error after error and not really go anywhere and if you just kill that process, a reboot is likely to fail big time.

So we came to the conclusion that the best way to be safe is to have two separate drives or at least two separate partitions for your system. The first partition will be for the OS. That way you will always be able to reboot if required. The second will be your /home and /var. If you use a software that uses a different directory (it should not, but everyone does not follow rules... a while back, Jira would use /opt as a read/write folder), then move that other folder to another drive too.

The advantage here is that your OS won't get busted. If the second drive gets full, the app stops working, but the OS runs like a charm. You can then umount the second drive, work on it in various ways, and re-mount it once fixed. For example, maybe you had a large tarball somewhere, or a movie you don't need which is using many Gb of space.

Another advantage to having two drives: when you do an upgrade of the operating system to the next major LTS version, you can simply replace the OS drive. Your data drive will already be ready with most everything, possibly even your application (if not installed with packages, then your application may be sitting in a folder such as /var/www).

WARNING:

As shown above, the root directory must remain on the OS drive. This way you can always run commands as the root user and it will work as expected.

However, you must make sure that if you create backups of your application, you do NOT put those backups under /root. This is always a bad idea anyway. The /root directory should nearly never be used but for very small files as required to do repairs and such. All the other files should be in a user's /home directory instead or under various places such as /var and /tmp.