Snapshot timeout

Lars    Jan 12 9:32PM 2018

Hi,

I am testing backup of an average VM (140 GB of virtual disks), but without success.

Shortly after the backup is started, it fails:

./vertical backup WS2008 Vertical Backup 1.1.5 Trial license expires on 2018-02-11 Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket Listing all virtual machines Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest Last backup at revision 1 found Virtual machine WS2008 is powered on Removing all snapshots of WS2008 Creating a new virtual machine snapshot for WS2008 Command '/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-13-02-01-36 'Created by Vertical Backup 1.1.5' 0 1' times out Create Snapshot: Command '/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-13-02-01-36 'Created by Vertical Backup 1.1.5' 0 1' returned -9 Removing all snapshots of WS2008 #

When I checked the system, I found out that the snapshot task was still running, finished successfuly after minutes later:

vim-cmd vimsvc/task_list (ManagedObjectReference) [ 'vim.Task:haTask-1-vim.VirtualMachine.removeAllSnapshots-117620809', 'vim.Task:haTask-1-vim.VirtualMachine.removeAllSnapshots-117620872', 'vim.Task:haTask-1-vim.VirtualMachine.createSnapshot-117620833' ] #

Is there something wrong with my configuration (ESXi 6.5) or is vertical simply too impatient?

Thanks, Lars


gchen    Jan 13 2:20PM 2018

The timeout for removing all snapshots is 20 minutes which I think is enough for most cases. But, if you haven't done snapshot consolidation recently, the delta disk file can be very large and removing all snapshots may take a long time.

Now that the snapshot removing task has successfully finished, the timeout should not happen when you run the backup command again.


Lars    Jan 13 5:52PM 2018

I have tried it several times. There is no existing snapshot on the VM.

When I run the same command as seen in the log by hand, it lasts 3-5 minutes and finishes just fine. Vertical consistently fails during snapshot creation, even though - as seen in the ESXi logs - the snapshot succeeds. The point is that vertical fails when the snapshot is still running. The execution time until it fails is cca 2 minutes, definitelly not 20 or even more.


gchen    Jan 14 12:04AM 2018

Can you try this new build at http://acrosync.com/esxi/vertical (sha256: e42e52c125f5268b50a5c68ce631ff39bda02a423720e89bf13d89efd44aa920)?

The timeout for creating a new snapshot has been increased from 2 minutes to 10 minutes. This is only a temporary fix -- ideally there should be an option to set these timeout values.


Lars    Jan 14 5:04AM 2018

Hi, it did not help:

time ./vertical\ (2) backup WS2008 Vertical Backup 1.1.5 Trial license expires on 2018-02-11 Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket Listing all virtual machines Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest Last backup at revision 1 found Virtual machine WS2008 is powered on Removing all snapshots of WS2008 Creating a new virtual machine snapshot for WS2008 Command '/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-14-09-56-03 'Created by Vertical Backup 1.1.5' 0 1' times out Create Snapshot: Command '/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-14-09-56-03 'Created by Vertical Backup 1.1.5' 0 1' returned -9 Removing all snapshots of WS2008 Command exited with non-zero status 2 real 2m 23.44s user 0m 0.09s sys 0m 0.00s

sha 256sum vertical* e42e52c125f5268b50a5c68ce631ff39bda02a423720e89bf13d89efd44aa920 vertical (2)

The total execution time was less than 3 minutes, there is still some problem with the timeout limit. If you plan to introduce timeout option, please consider an option to disable timeout completely if the timeout value is zero.

Thanks, Lars


gchen    Jan 14 2:08PM 2018

Sorry my bad. There are two places where a new snapshot can be created and I only changed the timeout in one of them.

The new build at http://acrosync.com/esxi/vertical (sha256: d0fa9bebca2f7ae40902c6e10ef6a4b6427a0ffddfab08ac83db6e6a1dbffb0c) should fix this.


Lars    Jan 14 7:29PM 2018

I am sorry, not working again.

This time it does create the snapshot, but does not detect the successful creation, then removes it again (visible in ESXi log files) and fails.

time ./vertical\ (3) backup WS2008 Vertical Backup 1.1.5 Trial license expires on 2018-02-11 Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/jtk-vertical/ Listing all virtual machines Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest Last backup at revision 1 found Virtual machine WS2008 is powered on Removing all snapshots of WS2008 Creating a new virtual machine snapshot for WS2008 Backup can't proceed when there is no snapshot and an attempt to create a new one failed Removing all snapshots of WS2008 Command exited with non-zero status 2 real 2m 51.69s user 0m 0.09s sys 0m 0.00s


gchen    Jan 15 5:49PM 2018

Can you manually run the snapshot creation command, and then check under the vm directory if there is a delta disk file for each disk?


Lars    Jan 16 9:36AM 2018

Confirmed:

vim-cmd vmsvc/snapshot.create 1 20180116 test 0 1

command succeeded, delta file created:

-rw------- 1 root root 16945152 Jan 16 14:34 WS2008-000001-delta.vmdk


gchen    Jan 17 8:04PM 2018

I added a 10 second delay after the snapshot is created. Hope that is enough for Vertical Backup to detect the new snapshot.

The latest build can be downloaded from the same link with a sha256 of 11a9459a2a93047d16fe5e9399186c33f383e8843ac19b601527cc543eb897bf.


Lars    Jan 18 2:12AM 2018

Hi,

I am sorry to say that your last change did not help again. The same behaviour, vertical creates a snapshot, then fails to detect it and at the end it removes it again.:

Vertical Backup 1.1.5 Trial license expires on 2018-02-11 Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket Listing all virtual machines Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest Last backup at revision 1 found Virtual machine WS2008 is powered on Removing all snapshots of WS2008 Creating a new virtual machine snapshot for WS2008 Backup can't proceed when there is no snapshot and an attempt to create a new one failed Removing all snapshots of WS2008

Just an idea: the VM includes two disks, one regular and one independent-persistent, which is unaffected by snapshots. Could this be the reason?

Thanks, Lars


gchen    Jan 18 9:20PM 2018

Vertical Backup can't back up independent-persistent disks, so you will have to exclude that disk from the backup:

./vertical backup WS2008 --exclude-disk "*disk_name*"

The exclude-disk argument is the full path to the disk file but you can use wildcards instead to avoid specifying the entire path.


Lars    Jan 22 4:27AM 2018

Yup, now it works, thanks.

Can we expect some improvements in the documentation and error detection / messages?

Thanks again for your support, testing resumed :-) Lars


Log in to comment
Copyright © Acrosync LLC 2017