Morgan Oct 13 7:23PM 2017
I went to one of my machines where the cron backups are not happening, to find out why.
I executed
./vertical backup
I got
Vertical Backup 1.1.2
Licensed to <name>
Storage set to /vmfs/volumes/datastore2/vbackup
Listing all virtual machines
Backing up CentOS7Minimal, id: 3, vmx path: /vmfs/volumes/datastore1/CentOS7Minimal/CentOS7Minimal.vmx, guest os: centos7_64Guest
Last backup at revision 7 found
Virtual machine CentOS7Minimal is powered on
Removing all snapshots of CentOS7Minimal
Creating a new virtual machine snapshot for CentOS7Minimal
Failed to save '/vmfs/volumes/datastore2/vbackup/chunks/35/c5/ffaf3464ed0f8b3da3b2537431aca0cfa0c97a0f0a8b7d473ab5699f5ce7.e602af7b': No space left on device
Removing all snapshots of CentOS7Minimal
So then I did a df to check space on the datastore2:
VMFS-5 930.2G 151.4G 778.9G 16% /vmfs/volumes/datastore2
It looks like it has plenty of space.
What else could be wrong?
Thanks
gchen Oct 13 9:40PM 2017
It could be that your esxi was out of inodes. Please follow the instructions in this article to see if this is the case: https://kb.vmware.com/kb/1007638
Morgan Oct 15 12:21AM 2017
It does not appear out of inodes on root:
> stat -f /
File: "/"
ID: 100000000 Namelen: 127 Type: visorfs
Block size: 4096
Blocks: Total: 423400 Free: 234579 Available: 234579
Inodes: Total: 655360 Free: 649056
However, the numbers reported on the datastores are not useful:
> stat -f /vmfs/volumes/datastore2
File: "/vmfs/volumes/datastore2"
ID: 365500583c8eaf8 Namelen: 127 Type: vmfs
Block size: 1048576
Blocks: Total: 952576 Free: 797020 Available: 797020
Inodes: Total: 2147483647 Free: 2147483647
> stat -f /vmfs/volumes/datastore1
File: "/vmfs/volumes/datastore1"
ID: 12e1f6e41ec0bc60 Namelen: 127 Type: vmfs
Block size: 1048576
Blocks: Total: 221184 Free: 112952 Available: 112952
Inodes: Total: 2147483647 Free: 2147483647
I believe it may be out of inodes due to the vertical backups. I can do some pruning, but this is a concern if my storage is already out of inodes. So far, I only have less than 10 days of backups for four VM's in the storage on datastore2. Is it normal to run out of inodes so quickly?
Morgan Oct 15 1:46AM 2017
Allright, I did a file count on the affected device (datastore 2) and it came out to exactly 130000 files.
I've seen references that vmfs5 is limited to 130k.
So apparently it's reached the limit.
I have to figure out what to do next. So far in my searches, I can't find whether migrating to vmfs6 increases the number of inodes available on a datastore or not.
If so, that's probably the easiest route long term.
For now, I'm doing a massive pruning to get this machine back working.
Morgan Oct 15 1:56PM 2017
Allright - I decided that this is too much of a hassle doing backups directly to VMFS partitions.
So now I've set it up to backup to a linux partition of one of my virtual machines, which will then be copied offsite using duplicacy.
So far, this seems to be working much more smoothly (and faster!!!).
It would be very helpful to have a more extensive guide covering backup scenarios/configurations, so that problems like this coming from trial-and-error (which are time wasting), are reduced. Not all of us are experienced esxi sysadmins, and I for one can't really afford the time that it's taken me to get everything working.
Though my time is limited, I may be able to write something up about my configuration, especially if e.g. there were some reward for doing so.
Thanks Morgan
gchen Oct 15 9:51PM 2017
Sure, if you can write such a guide to share your experiences using Vertical Backup, I can extend all your licenses for one year for free!
Morgan Oct 16 3:26PM 2017
okay - thanks! I will see if I can find some time soon - this week is very busy!