Backup of Large VM has errors & says no backup BUT space on Backup Server is used for Backup

LawcMnnn    Jul 16 12:09PM 2017

I tried backing up a VM server (we use as a database) that is quite large (around 800 GBs). The backup process took at least a day and half. At the end I saw this error message:

Creating a new virtual machine snapshot for MYDB Uploaded file /vmfs/volumes/VM2-SSD/MYDB/MYDB.vmdk Uploaded file /vmfs/volumes/VM2-SSD/MYDB/MYDB_3.vmdk Uploading file MYDB_3-flat.vmdk Uploaded file MYDB_3-flat.vmdk 6.62MB/s 1 day 06:05:43
Uploading file MYDB-flat.vmdk Uploaded file MYDB-flat.vmdk 13.52MB/s 01:40:57
Uploaded file MYDB.vmx Uploaded file MYDB.vmxf Removing all snapshots of MYDB Command '/bin/vim-cmd vmsvc/snapshot.removeall 6' times out Exception in thread Thread-12: Traceback (most recent call last): File "", line 801, in __bootstrap_inner File "", line 1073, in run File "", line 31, in runCommandTimeout File "", line 46, in LOG_ERROR File "", line 80, in log VerticalBackupError: COMMAND_RUN Command '/bin/vim-cmd vmsvc/snapshot.removeall 6' times out

Remove All Snapshots: Command '/bin/vim-cmd vmsvc/snapshot.removeall 6' returned -9 /opt/verticalbackup # df -h


/opt/verticalbackup # vertical list Vertical Backup 1.0.4 Storage set to sftp://user@

No backup is listed? But the space on backup volume has been allocated for this large backup so something is there? What should I do?

gchen    Jul 16 11:18PM 2017

The default timeout for removing the snapshots is 1200 seconds. Obviously that was not enough in your case. So the backup file didn't get a chance to be uploaded, while all chunks that had been uploaded would remain in the storage.

The real issue here is that the backup file should be uploaded first before removing the snapshots. It doesn't matter if the final snapshot removal takes too long. It will likely be done when the next backup is to be started.

So I made a change to move the uploading of the backup file to before the snapshot removal and uploaded a new version 1.0.5. You can download it from

LawcMnnn    Jul 17 10:59AM 2017

Ok thanks will download! What should I do after downloading the latest version? Just run another backup and it will remove the previous backup attempt?

Currently I have space on the backup drive that is allocated for the backup for this backup but not sure how to free up space?

gchen    Jul 17 12:26PM 2017

Right, just run another backup which will be much faster since many chunks have been uploaded by the previous backup attempt. However, some chunks uploaded by the previous backup attempt will become unreferenced due to changes in the disk files, so you'll need to clean up these chunks after the backup is finished.

This command will print out all unreferenced chunks:

vertical prune -d --exclusive --exhaustive

Without the -d option it will remove all unreferenced chunks (assuming no backup to the same storage is in progress):

vertical prune --exclusive --exhaustive 

LawcMnnn    Jul 17 2:30PM 2017

My network connection died while running backup and was wondering how to check the status of the running backup?

"Another backup job is in progress" when I type in vertical backup VM

gchen    Jul 17 7:26PM 2017

You can run pgrep vertical to see if the backup process is running, pkill vertical to kill it.

If pgrep vertical doesn't report any process while you still get "Another backup job is in progress" then you can manually remove the file .vertical/plock.

LawcMnnn    Jul 17 8:15PM 2017

I see the process.. do you know if the output of the process gets written to a log file?

gchen    Jul 17 9:51PM 2017

No, it doesn't log to a file unless you redirected stdout to a file when you started the program. You can run strace -p pid -- if the backup is running it will print out tons of messages.

LawcMnnn    Jul 18 1:26AM 2017

Thanks for your response.. Now whenever I execute a vertical command I get the following error (was trying to do a vertical list):

File "", line 8, in File "/tmp/pip-build-VI6Ne5/pyinstaller/PyInstaller/loader/", line 389, in load_module File "", line 23, in File "/tmp/pip-build-VI6Ne5/pyinstaller/PyInstaller/loader/", line 389, in load_module File "", line 18, in OSError: [Errno 28] No space left on device Failed to execute script vertical_main

But I have space available on the vmware server and the data backup server:

VMFS-5 953.0G 894.2G 58.8G 94% /vmfs/volumes/VM2-SSD VMFS-5 1.8T 1.3T 493.0G 73% /vmfs/volumes/VM2_Data vfat 4.0G 22.8M 4.0G 1% /vmfs/volumes/52721af7-72e1bda9-b39b-a0369f294c78 vfat 249.7M 158.8M 90.9M 64% /vmfs/volumes/8808c774-0879865b-3767-e2c4cfe15f86 vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/c6cee871-7b0506fd-f898-2a5e09b59206 vfat 285.8M 191.3M 94.6M 67% /vmfs/volumes/52721aee-dd51aa53-fe0e-a0369f294c78

VMFS-5 volume is getting close to being full but not there yet, not sure what the issue is.

gchen    Jul 18 9:38AM 2017

Your /tmp dir is full. There might be some folders similar to /tmp/pip-build-VI6Ne5 which were left by previous Vertical Backup processes not exiting cleanly. You can remove these folders to free up space.

LawcMnnn    Jul 18 1:59PM 2017

The /tmp dir is empty actually..

I'm getting new issues when running the backup:

Storage set to sftp://user@ Listing all virtual machines Backing up MYDB, id: 6, vmx path: /vmfs/volumes/VM2-SSD/MYDB/MYDB.vmx, guest os: centos64Guest No previous backup found Virtual machine MYDB is powered on Removing all snapshots of MYDB Remove All Snapshots: Remove all snapshots failed Command '/bin/vim-cmd vmsvc/snapshot.removeall 6' returned 1

The backup is not running. I killed the previous vertical process as I had no idea on what it was doing.

gchen    Jul 18 3:34PM 2017

You can do a manual snapshot consolidation in vSphere Client (Right client the VM -> Snapshot -> Consolidate) and wait until it's completed and then start a backup again.

LawcMnnn    Jul 18 4:38PM 2017

Is this something I can do from the command line? kinda shuffling vmware hosts around the 3 host essential license :(

gchen    Jul 18 5:03PM 2017

I think vsphere client is free, for example

LawcMnnn    Jul 18 7:05PM 2017

Ok was able to fix the issue by running the following command manually:

vim-cmd vmsvc/snapshot.removeall [VMID]

Log in to comment
Copyright © Acrosync LLC 2017