Backup stops: No space left on device

tvh_solutions    Apr 13 2:40PM 2018

I use ESXi 6.5U1. Backing up to a B2 bucket, backups fail if we try to backup more than one virtual machine in a row.

Command line is: vertical backup --exclude 'Napp-IT*' --email --threads 32

--threads 32 gives the most upload speed. Reducing threads does not change the errors.

There are no network outages during backup. I see no problems on the backblaze console. From my testing, the problems seem to always occur during backup of the second powered-on VM. Storage is via NFS, there is plenty of free storage (75%). Also, the ramdisks do not seem to fill up (although, for the latter, I have no output right at the time of the abort)

2018-04-13 02:45:02.643394 INFO PROGRAM_VERSION Vertical Backup 1.1.5
2018-04-13 02:45:03.028537 INFO LICENSE_INFO Trial license expires on 2018-04-20
2018-04-13 02:45:09.878426 INFO STORAGE_CREATE Storage set to b2://van-hees-vm-backups
2018-04-13 02:45:11.998846 INFO SNAPSHOT_GETALLVM Listing all virtual machines
2018-04-13 02:45:12.190802 INFO BACKUP_VM Backing up Solaris SAN1, id: 111, vmx path: /vmfs/volumes/SSD/Solaris SAN1/Solaris SAN1.vmx, guest os: solaris11_64Guest
2018-04-13 02:45:12.378573 INFO BACKUP_PREV Last backup at revision 2 found
2018-04-13 02:45:12.952198 INFO SNAPSHOT_POWER Virtual machine Solaris SAN1 is powered off
2018-04-13 02:45:12.952323 INFO SNAPSHOT_REMOVE Removing all snapshots of Solaris SAN1
2018-04-13 02:45:15.893177 INFO BACKUP_UPLOAD Uploaded file /vmfs/volumes/SSD/Solaris SAN1/Solaris SAN1.vmdk
2018-04-13 02:45:15.895619 INFO BACKUP_UPLOAD Uploading file Solaris SAN1-flat.vmdk
2018-04-13 02:45:15.896359 INFO RESTORE_THREAD Using 32 uploading threads
2018-04-13 02:46:29.450089 INFO BACKUP_UPLOAD Uploaded file Solaris SAN1-flat.vmdk 556.87MB/s 00:01:13
2018-04-13 02:46:30.021988 INFO BACKUP_UPLOAD Uploaded file Solaris SAN1.vmx
2018-04-13 02:46:35.738060 INFO BACKUP_DONE Backup Solaris SAN1@fast at revision 3 has been successfully completed
2018-04-13 02:46:35.738138 INFO BACKUP_STATS Total 40967 chunks, 40963.01M bytes; 128 new, 124.01M bytes, 68.49M uploaded
2018-04-13 02:46:35.738165 INFO BACKUP_TIME Total backup time: 00:01:21
2018-04-13 02:46:35.739604 INFO BACKUP_VM Backing up Win Server 2016 Secondary, id: 114, vmx path: /vmfs/volumes/NFSFast/Win Server 2016 Secondary/Win Server 2016 Secondary.vmx, guest os: windows9Server64Guest
2018-04-13 02:46:35.942196 INFO BACKUP_PREV Last backup at revision 2 found
2018-04-13 02:46:36.660627 INFO SNAPSHOT_POWER Virtual machine Win Server 2016 Secondary is powered on
2018-04-13 02:46:36.660854 INFO SNAPSHOT_REMOVE Removing all snapshots of Win Server 2016 Secondary
2018-04-13 02:46:37.859242 INFO SNAPSHOT_CREATE Creating a new virtual machine snapshot for Win Server 2016 Secondary
2018-04-13 02:46:51.639065 INFO BACKUP_UPLOAD Uploaded file /vmfs/volumes/NFSFast/Win Server 2016 Secondary/Win Server 2016 Secondary.vmdk
2018-04-13 02:46:51.639951 INFO BACKUP_UPLOAD Uploading file Win Server 2016 Secondary-flat.vmdk
2018-04-13 02:46:51.641243 INFO RESTORE_THREAD Using 32 uploading threads
2018-04-13 02:58:06.061491 INFO BACKUP_UPLOAD Uploaded file Win Server 2016 Secondary-flat.vmdk 60.73MB/s 00:11:14
2018-04-13 02:58:06.063016 INFO BACKUP_UPLOAD Uploaded file Win Server 2016 Secondary.vmx
2018-04-13 02:58:19.731861 INFO BACKUP_DONE Backup Win Server 2016 Secondary@fast at revision 3 has been successfully completed
2018-04-13 02:58:19.731931 INFO BACKUP_STATS Total 40967 chunks, 40963.01M bytes; 2632 new, 2630.00M bytes, 1207.66M uploaded
2018-04-13 02:58:19.731959 INFO BACKUP_TIME Total backup time: 00:11:29
2018-04-13 02:58:19.733926 INFO SNAPSHOT_REMOVE Removing all snapshots of Win Server 2016 Secondary
2018-04-13 02:58:21.276001 INFO BACKUP_VM Backing up Sophos, id: 115, vmx path: /vmfs/volumes/NFSFast/Sophos/Sophos.vmx, guest os: sles12_64Guest
2018-04-13 02:58:21.464798 INFO BACKUP_LISTING No previous backup found.  Listing existing chunks
2018-04-13 02:59:45.073078 INFO SNAPSHOT_POWER Virtual machine Sophos is powered on
2018-04-13 02:59:45.073327 INFO SNAPSHOT_REMOVE Removing all snapshots of Sophos
2018-04-13 03:00:39.753467 INFO SNAPSHOT_CREATE Creating a new virtual machine snapshot for Sophos
2018-04-13 03:00:42.564397 INFO BACKUP_UPLOAD Uploaded file /vmfs/volumes/NFSFast/Sophos/Sophos.vmdk
2018-04-13 03:00:43.269040 INFO BACKUP_UPLOAD Uploaded file /vmfs/volumes/NFSFast/Sophos/Sophos_2.vmdk
2018-04-13 03:00:43.269797 INFO BACKUP_UPLOAD Uploading file Sophos-flat.vmdk
2018-04-13 03:00:43.270351 INFO RESTORE_THREAD Using 32 uploading threads
2018-04-13 03:01:13.165002 WARNING B2_CALL Failed to send request post https://api001.backblazeb2.com/b2api/v1/b2_list_file_names: ('Connection aborted.', BadStatusLine("''",))
2018-04-13 03:01:13.167896 WARNING B2_CALL Failed to send request post https://api001.backblazeb2.com/b2api/v1/b2_list_file_names: ('Connection aborted.', BadStatusLine("''",))
2018-04-13 03:01:13.183989 WARNING B2_CALL Failed to send request post https://api001.backblazeb2.com/b2api/v1/b2_list_file_names: ('Connection aborted.', BadStatusLine("''",))
2018-04-13 03:01:13.189305 WARNING B2_CALL Failed to send request post https://api001.backblazeb2.com/b2api/v1/b2_list_file_names: ('Connection aborted.', BadStatusLine("''",))
2018-04-13 03:01:35.992520 INFO BACKUP_UPLOAD Uploaded file Sophos-flat.vmdk 77.69MB/s 00:00:52
2018-04-13 03:01:35.992589 INFO BACKUP_UPLOAD Uploading file Sophos_2-flat.vmdk
2018-04-13 03:01:35.993695 INFO RESTORE_THREAD Using 32 uploading threads
2018-04-13 03:01:35.997267 INFO SNAPSHOT_REMOVE Removing all snapshots of Sophos
2018-04-13 03:01:36.165296 INFO COMMAND_FAILURE Failed to run command '/bin/vim-cmd vmsvc/snapshot.removeall 115': No space left on device
2018-04-13 03:01:36.165638 ERROR COMMAND_FAILURE Failed to run command '/bin/vim-cmd vmsvc/snapshot.removeall 115': No space left on device

esxcli system visorfs ramdisk list:

Ramdisk Name  System  Include in Coredumps   Reserved      Maximum       Used  Peak Used   Free  Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point                
------------  ------  --------------------  ---------  -----------  ---------  ---------  -----  -------------  --------------  ----------------  -----------  ---------------------------
root            true                  true  32768 KiB    32768 KiB   2000 KiB   2000 KiB   93 %           93 %           26624              5088         5073  /                          
etc             true                  true  28672 KiB    28672 KiB    408 KiB    440 KiB   98 %           98 %            4096              1024          648  /etc                       
opt             true                  true      0 KiB    32768 KiB      0 KiB      0 KiB  100 %            0 %            8192              1024           25  /opt                       
var             true                  true   5120 KiB    49152 KiB    392 KiB    396 KiB   99 %           92 %            8192               672          653  /var                       
tmp            false                 false   2048 KiB   262144 KiB  19536 KiB  39420 KiB   92 %            0 %            8192               256           80  /tmp                       
iofilters      false                 false      0 KiB    32768 KiB      0 KiB      0 KiB  100 %            0 %           10240                32            1  /var/run/iofilters         
hostdstats     false                 false      0 KiB  2102272 KiB   6348 KiB   6352 KiB   99 %            0 %            8192                32            5  /var/lib/vmware/hostd/stats
snmptraps      false                 false      0 KiB     1024 KiB      0 KiB      4 KiB  100 %            0 %            8192                32            1  /var/spool/snmp

df -h

Filesystem   Size   Used Available Use% Mounted on
NFS          1.0T 277.6G    785.4G  26% /vmfs/volumes/ZFS_SSD
VMFS-6     119.0G  66.9G     52.1G  56% /vmfs/volumes/NVMe
VMFS-6     999.8G 826.5G    173.3G  83% /vmfs/volumes/iSCSI
VMFS-6     260.8G  41.4G    219.3G  16% /vmfs/volumes/Datastore
vfat       249.7M 156.4M     93.3M  63% /vmfs/volumes/41626ce0-22067b1a-05bd-5d6d6ea5d533
vfat       249.7M 155.6M     94.2M  62% /vmfs/volumes/d6c39fe3-542b13dc-ef0a-e85e5a70813b
vfat       285.8M 232.2M     53.6M  81% /vmfs/volumes/5870d04e-78805b40-4538-0cc47a86750c


gchen    Apr 13 9:46PM 2018

This error actually means out of memory. This is similar to the other issue reported last week: https://www.verticalbackup.com/issue?id=5726683906703360.

I'll be working on the new version on Monday and will send you a new 30-day trial license.


gchen    Apr 17 10:12PM 2018

I uploaded a new version 1.2.0 at http://acrosync.com/esxi/vertical.

This version should be much more memory efficient than previous versions.

If you log into your customer page you should also see a 30-day license there. You can activate it by running vertical license activate and then entering the license code.


Log in to comment
Copyright © Acrosync LLC 2017