no space left on device

Morgan    Oct 13 7:23PM 2017

I went to one of my machines where the cron backups are not happening, to find out why.

I executed

./vertical backup

I got

Vertical Backup 1.1.2
Licensed to <name>
Storage set to /vmfs/volumes/datastore2/vbackup
Listing all virtual machines
Backing up CentOS7Minimal, id: 3, vmx path:      /vmfs/volumes/datastore1/CentOS7Minimal/CentOS7Minimal.vmx, guest os: centos7_64Guest
Last backup at revision 7 found
Virtual machine CentOS7Minimal is powered on
Removing all snapshots of CentOS7Minimal
Creating a new virtual machine snapshot for CentOS7Minimal
Failed to save     '/vmfs/volumes/datastore2/vbackup/chunks/35/c5/ffaf3464ed0f8b3da3b2537431aca0cfa0c97a0f0a8b7d473ab5699f5ce7.e602af7b': No space left on device
Removing all snapshots of CentOS7Minimal

So then I did a df to check space on the datastore2:

VMFS-5     930.2G 151.4G    778.9G  16% /vmfs/volumes/datastore2

It looks like it has plenty of space.

What else could be wrong?


gchen    Oct 13 9:40PM 2017

It could be that your esxi was out of inodes. Please follow the instructions in this article to see if this is the case:

Morgan    Oct 15 12:21AM 2017

It does not appear out of inodes on root:

> stat -f /
File: "/"
    ID: 100000000 Namelen: 127     Type: visorfs
Block size: 4096      
Blocks: Total: 423400     Free: 234579     Available: 234579
Inodes: Total: 655360     Free: 649056

However, the numbers reported on the datastores are not useful:

> stat -f /vmfs/volumes/datastore2
  File: "/vmfs/volumes/datastore2"
    ID: 365500583c8eaf8 Namelen: 127     Type: vmfs
Block size: 1048576   
Blocks: Total: 952576     Free: 797020     Available: 797020
Inodes: Total: 2147483647 Free: 2147483647
> stat -f /vmfs/volumes/datastore1
  File: "/vmfs/volumes/datastore1"
    ID: 12e1f6e41ec0bc60 Namelen: 127     Type: vmfs
Block size: 1048576   
Blocks: Total: 221184     Free: 112952     Available: 112952
Inodes: Total: 2147483647 Free: 2147483647

I believe it may be out of inodes due to the vertical backups. I can do some pruning, but this is a concern if my storage is already out of inodes. So far, I only have less than 10 days of backups for four VM's in the storage on datastore2. Is it normal to run out of inodes so quickly?

Morgan    Oct 15 1:46AM 2017

Allright, I did a file count on the affected device (datastore 2) and it came out to exactly 130000 files.

I've seen references that vmfs5 is limited to 130k.

So apparently it's reached the limit.

I have to figure out what to do next. So far in my searches, I can't find whether migrating to vmfs6 increases the number of inodes available on a datastore or not.

If so, that's probably the easiest route long term.

For now, I'm doing a massive pruning to get this machine back working.

Morgan    Oct 15 1:56PM 2017

Allright - I decided that this is too much of a hassle doing backups directly to VMFS partitions.

So now I've set it up to backup to a linux partition of one of my virtual machines, which will then be copied offsite using duplicacy.

So far, this seems to be working much more smoothly (and faster!!!).

It would be very helpful to have a more extensive guide covering backup scenarios/configurations, so that problems like this coming from trial-and-error (which are time wasting), are reduced. Not all of us are experienced esxi sysadmins, and I for one can't really afford the time that it's taken me to get everything working.

Though my time is limited, I may be able to write something up about my configuration, especially if e.g. there were some reward for doing so.

Thanks Morgan

gchen    Oct 15 9:51PM 2017

Sure, if you can write such a guide to share your experiences using Vertical Backup, I can extend all your licenses for one year for free!

Morgan    Oct 16 3:26PM 2017

okay - thanks! I will see if I can find some time soon - this week is very busy!

Log in to comment
Copyright © Acrosync LLC 2017