Lee Tickett Sep 1 5:28AM 2017
Any ideas?
[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical backup ts1 Vertical Backup 1.1.1 Licensed to Tickett Enterprises Limited; expires on 2018-08-30 Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup Listing all virtual machines Backing up ts1, id: 2, vmx path: /vmfs/volumes/SSD/ts1/ts1.vmx, guest os: windows9_64Guest Last backup at revision 4 found Virtual machine ts1 is powered on Removing all snapshots of ts1 Creating a new virtual machine snapshot for ts1 Failed to upload 'vmfs/volumes/HDD/backup/chunks/4b/4e/04f8729463bf910bc5e9a7802f964638d98ca503cc9cbf10b4d9e7f23b22': Failure Removing all snapshots of ts1 [root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup]
gchen Sep 1 10:30AM 2017
Running out of space? Failure is a very generic SFTP error that can caused by several different reasons. The most likely I can think of is that the server runs of the storage space to save the upload chunk.
Lee Tickett Sep 2 4:14AM 2017
That was my first idea. Unfortunately loads of space;
[root@localhost:~] df -h Filesystem Size Used Available Use% Mounted on VMFS-5 465.5G 1019.0M 464.5G 0% /vmfs/volumes/SSD VMFS-5 1.8T 107.2G 1.7T 6% /vmfs/volumes/HDD vfat 285.8M 205.9M 80.0M 72% /vmfs/volumes/59786810-48e9eb79-b2b0-00 25900c529e vfat 249.7M 143.8M 105.9M 58% /vmfs/volumes/fe0638e5-1b6dd0ba-1a9c-dc 094911ea19 vfat 249.7M 147.8M 102.0M 59% /vmfs/volumes/d27798e9-7daa00be-dcee-5d 9fd8598c30
I just tried backing up another VM which also failed so I don't think it's an issue with this specific VM.
What's next?
Lee Tickett Sep 2 4:17AM 2017
Hmmm...
[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical benchmark Vertical Backup 1.1.1 Test directory: /vmfs/volumes/4c2e0db0-c42d9b1e-ed3c-0025903c61d8 Creating a 1,024M test file with random data Write time: 12.808, speed: 79.950MB/s Read time: 6.734, speed: 152.065MB/s Creating another 1,024M test file with random data Write time: 12.857, speed: 79.648MB/s Read and hash time: 9.128, speed: 112.179MB/s Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup Failed to upload 'vmfs/volumes/HDD/backup/bench/fbdbdedd7ac8e2e8f3f9aaf5c1616ca0015fc5715fbeb2909094dba5893ef9cf': Failure
Lee Tickett Sep 2 4:23AM 2017
Very confusing...
[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] scp ./vertical backup@172.17.0.100:/vmfs/volumes/HDD/backup Password: scp: /vmfs/volumes/HDD/backup/vertical: No space left on device
Lee Tickett Sep 2 4:27AM 2017
Also tried the scp as root and I get the same error.
Trying to upload through the web interface it appears to've hung on 0%. Something very strange is going on.
gchen Sep 2 9:29PM 2017
Is /vmfs/volumes/HDD/backup a mount point that is actually a ram disk?
Lee Tickett Sep 3 2:12AM 2017
Nope. It's one of my datastores (a 2TB spinning disk).
Lee Tickett Sep 4 8:49AM 2017
I fear I may need to request a refund. I'm not convinced the product is mature/stable enough. I've now got a different error;
[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical backup dc1 ts1 --threads 4 Vertical Backup 1.1.1 Licensed to Tickett Enterprises Limited; expires on 2018-08-30 Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup Listing all virtual machines Backing up dc1, id: 1, vmx path: /vmfs/volumes/SSD/dc1/dc1.vmx, guest os: windows9Server64Guest No previous backup found Virtual machine dc1 is powered on Removing all snapshots of dc1 Creating a new virtual machine snapshot for dc1 Uploaded file /vmfs/volumes/SSD/dc1/dc1.vmdk Uploading file dc1-flat.vmdk Using 4 uploading threads Uploaded file dc1-flat.vmdk 5.70MB/s 02:59:45 Uploaded file dc1.vmx Uploaded file dc1.vmxf Backup dc1@hv1 at revision 1 has been successfully completed Total 61449 chunks, 61444.52M bytes; 36895 new, 36890.52M bytes, 21027.88M uploaded Total backup time: 02:59:51 Removing all snapshots of dc1 Backing up ts1, id: 2, vmx path: /vmfs/volumes/SSD/ts1/ts1.vmx, guest os: windows9_64Guest No previous backup found Virtual machine ts1 is powered on Removing all snapshots of ts1 Creating a new virtual machine snapshot for ts1 Uploaded file /vmfs/volumes/SSD/ts1/ts1.vmdk Uploading file ts1-flat.vmdk Using 4 uploading threads Uploaded file ts1-flat.vmdk 6.35MB/s 02:41:13 Uploaded file ts1.vmx Uploaded file ts1.vmxf Backup ts1@hv1 at revision 1 has been successfully completed Total 61449 chunks, 61444.52M bytes; 26472 new, 26468.52M bytes, 17750.72M uploaded Total backup time: 02:41:20 Removing all snapshots of ts1 Exception in thread Thread-7 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):
gchen Sep 4 8:49PM 2017
No problem with the refund but this is apparently a bug in the shutdown process. I believe the backup had been successfully completed before the error occurred. I'll fix the bug anyway.
How did you fix the out-of-space error?
Another note, in my own experience ESXi hosts are generally slow when used as SFTP servers. It may be much faster to run a dedicated linux box, physical or virtual, as SFTP servers.
Lee Tickett Sep 5 2:07AM 2017
Great, thanks I will persevere for a little longer then (especially if you are adding the "exclude disks" option too).
I am curious why ESXis implementation of sftp might be slow. I will probably avoid using a dedicated linux box (vm or otherwise) as my bottleneck should be bandwidth going forward (at present the source and destination are both in my lab, but they will be deployed to a site with a 10Mbit uplink).
Can we sign up anywhere to receive notifications of new releases?
Lee Tickett Sep 5 7:20AM 2017
Oh, and unfortunately i couldn't fix the space issue. I had to destroy the datastore and recreate (format).
I may consider moving to S3 depending on speed and cost.
gchen Sep 5 9:04PM 2017
I would suggest wasabi.com, a new S3-compatible provider that is faster and cheaper than Amazon S3.