Droolio Jun 10 2:02PM 2018
Hi,
Started testing Vertical Backup this weekend and my first VM backup went fine (if a little slow; 6MB/s, which I don't know if is normal with a Windows/cygwin SFTP storage guest on the same host, but I'll investigate that for later...).
My second test was a 2012 R2 Exchange 2013 Server VM but which errored out on snapshot creation:
2018-06-10 17:27:49.513474 INFO SNAPSHOT_POWER Virtual machine CDSMail is powered on
2018-06-10 17:27:49.513547 INFO SNAPSHOT_REMOVE Removing all snapshots of CDSMail
2018-06-10 17:27:49.513607 TRACE COMMAND_RUN Running command: /bin/vim-cmd vmsvc/snapshot.removeall 4
2018-06-10 17:27:49.919228 TRACE COMMAND_OUTPUT Remove All Snapshots:
2018-06-10 17:27:50.925136 INFO SNAPSHOT_CREATE Creating a new virtual machine snapshot for CDSMail
2018-06-10 17:27:50.925221 TRACE COMMAND_RUN Running command: /bin/vim-cmd vmsvc/snapshot.create 4 2018-06-10-17-27-50 'Created by Vertical Backup 1.1.5' 0 0
2018-06-10 17:27:51.733424 TRACE COMMAND_OUTPUT Create Snapshot:
2018-06-10 17:27:51.737969 ERROR SNAPSHOT_PRESERVE Backup can't proceed when there is no snapshot and an attempt to create a new one failed
2018-06-10 17:27:51.738155 INFO SNAPSHOT_REMOVE Removing all snapshots of CDSMail
2018-06-10 17:27:51.738288 TRACE COMMAND_RUN Running command: /bin/vim-cmd vmsvc/snapshot.removeall 4
2018-06-10 17:27:52.956279 TRACE COMMAND_OUTPUT Remove All Snapshots:
The snapshots are created just fine, it's just that Vertical Backup seems to think they weren't. I can create snapshots for this VM in the vSphere Client, and with the vim-cmd vmsvc/snapshot.create command above. It works without issue, but verticalbackup goes no further.
There are no existing snapshots and I've already ran a consolidation as mentioned in other posts.
The backup completes OK if I exclude one of the disks (it has 2; 120GB OS and a 160GB thinly provisioned data disk). If I exclude the second disk, it runs and the OS disk is backed up. The second disk seems problematic. I've also tried --no-quiesce, doesn't make a difference.
Here's a (slightly sanitised) copy of the VMs vmware.log file, but I can't find anything wrong.
Any help is appreciated.
gchen Jun 11 7:36PM 2018
Your second disk is likely an independent disk which doesn't support the snapshot operation. So you'll need to exclude that disk (or run the backup when the VM is powered off).
Droolio Jun 12 7:29AM 2018
It's definitely not marked down as an Independent disk in vSphere Client - although it's all greyed out at the moment because the VM is running, it's also unticked.
What I'll try to do tonight is power off the VM, remove and re-add the disk to the VM, to see if that helps.
Droolio Jun 12 4:15PM 2018
So I tried removing the .vmdk from the VM (and in fact the OS .vmdk as well, along with the SCSI adapter) and re-added them back. No change. I then tried making a manual snapshot in vSphere Client and running vertical with --preserve-snapshots. This is what I got:
ERROR SNAPSHOT_PRESERVE Backup can't proceed when there is no snapshot and the --preserve-snapshot option is on
It seems vertical just isn't seeing the snapshots when that second .vmdk is included.
gchen Jun 12 8:39PM 2018
After you create a snapshot for the VM, do you see any *-delta.vmdk file for the disk in question? Each of these *-delta.vmdk files corresponds to a snapshot, so if there isn't one then that disk does not have a snapshot.
Droolio Jun 12 9:37PM 2018
The delta files definitely exist.
gchen Jun 12 10:56PM 2018
Can you check if those delta vmdk files are referenced first by CDSMail.vmx? For example, CDSMail-000002.vmdk rather than CDSMail.vmdk' should be listed in CDSMail.vmx.
Droolio Jun 12 10:57PM 2018
Problem solved!
I looked at the logs for the successful backup where I excluded the problematic disk (with --exclude-disk "CDSMail_1"), I spotted this:
2018-06-09 23:33:34.106863 INFO SNAPSHOT_REMOVE Removing all snapshots of CDSMail
2018-06-09 23:33:35.523462 INFO BACKUP_EXCLUDE Disk file '/vmfs/volumes/datastore1/CDSMail/CDSMail_1.vmdk' is excluded from backup
2018-06-09 23:33:35.524240 INFO BACKUP_EXCLUDE Disk file '/vmfs/volumes/datastore1/CDSMail/CDSMail_1.vmdk' is excluded from backup
2018-06-09 23:33:35.524324 INFO SNAPSHOT_CREATE Creating a new virtual machine snapshot for CDSMail
2018-06-09 23:33:53.608046 INFO BACKUP_EXCLUDE Disk file '/vmfs/volumes/datastore1/CDSMail/CDSMail_1.vmdk' is excluded from backup
2018-06-09 23:33:53.609416 INFO BACKUP_EXCLUDE Disk file '/vmfs/volumes/datastore1/CDSMail/CDSMail_1-000001.vmdk' is excluded from backup
Notice the triple mention of CDSMail_1.vmdk. I thought that was odd, so I looked closer at the .vmx and finally found the issue:
scsi1.virtualDev = "lsisas1068"
scsi1.present = "FALSE"
scsi1:0.deviceType = "scsi-hardDisk"
scsi1:0.fileName = "CDSMail_1.vmdk"
scsi1:0.present = "FALSE"
scsi1:0.redo = ""
scsi1.pciSlotNumber = "-1"
scsi1.sasWWID = "50 05 05 60 70 cd 3c 70"
scsi0.virtualDev = "lsisas1068"
scsi0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "CDSMail-000001.vmdk"
scsi0:0.present = "TRUE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.fileName = "CDSMail_1-000001.vmdk"
scsi0:1.present = "TRUE"
scsi0:1.redo = ""
scsi0:0.redo = ""
scsi0.pciSlotNumber = "160"
scsi0.sasWWID = "50 05 05 60 70 cd 3d 70"
A second, stale, scsi1 bus.
Notice the first mentioned .vmdk doesn't have a delta, because according to ESXi the disk nor the bus is supposed to be present and doesn't show up in the vSphere Client. Is Vertical Backup seeing those scsi1 entries, seeing a .vmdk without a delta and then stopping?
I fixed the .vmx without needing to edit it manually, by simply moving the second disk to scsi1:0 and then back to scsi0:1. I then had to add a dummy disk to scsi1:0 and remove it, in order to remove the unused scsi1 bus. All those scsi1 entries are now gone from the .vmx and the backup is running as I type this at nearly 5am. :)
Given how these stale entries hang around inside the .vmx, it might be a good idea for Vertical Backup to do some sanity checks and skip where disk.present = false?
Edit: fixed formatting :)
gchen Jun 14 9:41AM 2018
Yes, it definitely needs to check disk.present = false. Thank you for getting to the bottom of this!
