Question on "restore" copying VM backup to another Host.
LawcMnnn Jul 19 2:59AM 2017
https://verticalbackup.com/issue?id=5657382461898752
In this thread you mentioned moving another VM backup from one host to another. We are interested in moving 1 Large VM from Host1 to Host2 with minimal downtime for the moved VM. (We could use VCenter but would need to shut down the host and copy it over).
Currently we are backing up the VM on Host1. We would like to restore to new Host2. Can this be done while VM is still running on Host1. Then shutdown VM on Host1 and bring up VM on Host2?
Also would it be possible to restore VM to Host2 and make it a clone of the original VM running on Host1 but VM2 on Host2? Again with minimal downtime?
Thanks again for your helpful responses!
gchen Jul 19 9:17AM 2017
I think the easiest way to to create a VM with the same name and same disk size on Host2, and use the same host id to initialize on Host2:
vertical init host1 s3://mybucket # note that the host id is host1 although this runs on host2
vertical restore vmname -r 1 vmname-flat.vmdk # restore only the disk file
If you take this route then make sure you only do backup on host1 and restore on host2.
You can also use the approach recommended in https://verticalbackup.com/issue?id=5657382461898752, but you'll need to manually create a sparse disk file first otherwise the disk won't be thin-provisioned.
Yes, restore can be done while VM is running on Host1, and this will give you a clone of the original VM on host2.
LawcMnnn Jul 19 3:34PM 2017
Thanks that is helpful.
Just wanted to clarify.. IF I wanted to CLONE and create a new VM from the original VM to another VMHost should I do the following:
Currently have DB2 (Mysql DB SLave replicating from DB1) on HOST1 and my goal is to clone DB2 to HOST2 and have it become DB3.
Create on HOST2 DB3 with Same HD spacing and configuration as DB2.
On HOST2: vertical init host1 s3://mybucket
vertical restore DB2 -r 1 DB2-flat.vmdk
I think I'm missing something here:
How could I power up and change network configurations to avoid IP conflicts?
Is there a way to also add delta changes from the last backup to restore to the new clone after creating the initial restore?
Thanks!
gchen Jul 19 6:42PM 2017
The steps are correct except that the VM on HOST2 must be named DB2 too. If you don't want to use the same name then you can restore the disk file to a different path and then copy over the restored disk file.
Also you need to specify the path for the disk file like this:
./vertical restore -r 13 vm-test vmfs/volumes/datastore1/vm-test/vm-test-flat.vmdk
The network configurations do not matter -- you can also change them after the new VM boots up.
Once you have the two machines running side by side you can still back up the original and then apply the latest backup to the clone (which must be shut down for the restore operation).
LawcMnnn Jul 20 2:03AM 2017
So I went the route of keeping the same name on HOST2 as I have set it up on HOST1:
This restore command did not work for me: vertical restore MYDB@vm2 -r 2 MYDB-flat.vmdk
This worked but with error: vertical restore -r 2 -f MYDB@vm2 MYDB file MYDB-flat.vmdk
ERROR: Removing all snapshots of MYDB Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/8e/07/8b735c03990ff577b1b14d5b38bd666bd729a5a2a0b0c6d8d3dfc426aaa5.23547951': No space left on device Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/63/b0/371bc149ee5cc734ac84b38af2ab80d0d17efbf01eb3f0bf589ce5700799.166d3f85': No space left on device Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/bf/ad/5c6b9efda2910d1a16f196deea04b1f581ec4bc575821a20428c7a303e36.286f05ad': No space left on device Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/e5/44/d852c8e3cf33eb7b3b9aa5b822bdd7817d7d70a07b59ff605e3ca2fdda10.ba35847b': No space left on device
Backup MYDB@vm2 at revision 1 has been successfully restored to /vmfs/volumes/DatastoreHDD/MYDB Total 0 chunks, 0 bytes; 0 new, 0 bytes, 0 downloaded Total restore time: 00:00:06
What's weird is I get this message "No space left on device" but no where on any of the hosts is storage space being full is an issue. There is plenty of HD space on all hosts.
What do you recommend?
gchen Jul 20 9:45AM 2017
Those messages were just warnings. The real problem was that the full path of the disk file wasn't specified. Run ./vertical list -f to list all files in the backup, and then pass the full path to the store command. In my test case it is vmfs/volumes/datastore1/vm-test/vm-test-flat.vmdk.
It ran out of space because /opt was on a ramdisk with very limited space. This command lists info about all ramdisks:
esxcli system visorfs ramdisk list
You can use the --tmp-dir option of the init command to move the cache directory to a directory under the datastore.
That being said, these messages are indeed very annoying and I'll fix it in the next release.
LawcMnnn Jul 20 12:42PM 2017
Looks like that did the trick. It seems to be working now. I get the following message:
The destination directory '/vmfs/volumes/VM2-SSD/MYDB' does not exist; use '/vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB' instead
The volumes are different. I don't think that would be an issue?
Also the restore (this is the first time)... for roughly 800GB could take hours. Will it be faster if after the initial restore is done I do a second restore on a newer backup?
gchen Jul 20 1:12PM 2017
That could be an issue. Is it possible to create a symlink '/vmfs/volumes/VM2-SSD/MYDB' to make it point to the directory where the disk file for the new VM is? Otherwise, you're not writing directly to the VM but instead creating a new disk file which must be added to the new VM manually.
Yes, the second restore should be much faster than the initial one.
LawcMnnn Jul 20 1:51PM 2017
So the new directory (VM) is at:
/vmfs/volumes/DatastoreHDD/MYDB
The files in the directory are as follows:
drwxr-xr-x 1 root root 1120 Jul 19 22:46 . drwxr-xr-t 1 root root 1960 Jul 19 22:14 .. -rw------- 1 root root 826781204480 Jul 19 22:14 MYDB-flat.vmdk -rw------- 1 root root 470 Jul 19 22:14 MYDB.vmdk -rw-r--r-- 1 root root 0 Jul 19 22:14 MYDB.vmsd -rwxr-xr-x 1 root root 3546 Jul 19 19:02 MYDB.vmx -rw-r--r-- 1 root root 3425 Apr 7 2016 MYDB.vmxf drwxr-xr-x 1 root root 420 Jul 19 22:46 vmfs
The restore is happening at:
/vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB
Currently the files in the VM2-SSD (Restoring VM) dir is the following:
-rw-r--r-- 1 root root 18874368 Jul 19 22:46 MYDB-flat.vmdk -rw-r--r-- 1 root root 115674710016 Jul 20 10:40 MYDB_3-flat.vmdk
What exactly should I sym link?
ln -s /vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB /vmfs/volumes/DatastoreHDD/
or the files within the new volumes to the old:
ln -s /vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB/* /vmfs/volumes/DatastoreHDD/MYDB/
Thanks again for your guidance!
gchen Jul 20 3:31PM 2017
The standard way would be to create a symlink VM2-SSD that points to DatastoreHDD:
cd /vmfs/volumes
ln -s DatastoreHDD VM2-SSD
However, that doesn't seem to work on ESXi. The workaround is to make /vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB points to the actual VM directory:
mkdir -p /vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD
cd /vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD
ln -s /vmfs/volumes/DatastoreHDD/MYDB MYDB
LawcMnnn Jul 20 5:02PM 2017
Running into another issue with the restore.
The backedup DB is around 800GBs. We have a volume that is 1.1 TB. So we created the HOST2 MYDB on that volume and we should have roughly 300GB to spare.
The issue is that the restore is not done and has an error of "No space left on device". It filled up the 1.1 TB volume (while still performing the restore).
Here is where it was at before failing:
The destination directory '/vmfs/volumes/VM2-SSD/MYDB' does not exist; use '/vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB' instead Downloading ***********************----------------------------------------------------------------------------------------------------------- 12.29MB/s 13:19:18 17.8% Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/2c/23/fab2390f0591ab89c96e8037eba04f91e9c7f109096b5ae4afbb817fa727.b78d39e3': No space left on device Downloading 12.29MB/s 13:19:17 17.8% Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/24/45/acb03695995c87f3487c5bb54849ca97bb19d0b96ef8447afcc027a3bcb5.e12b5a17': No space left on device Downloading 12.29MB/s 13:19:17 17.8% Failed to save '/opt/verticalbackup/.verticalbackup/cache/chunks/12/3f/fd29242c5fda0f41363ab8f7f4c453a15945f9ddd438c57d77fd086c71db.53753ddc': No space left on device Downloading 12.29MB/s 13:19:17 17.8% Downloading 12.29MB/s 13:19:17 17.8% Downloading 12.29MB/s 13:19:16 17.8% Downloading 12.29MB/s 13:19:16 17.8% Downloading 12.29MB/s 13:19:15 17.8% Downloading 12.29MB/s 13:19:15 17.8% Downloading 12.29MB/s 13:19:16 17.8% ..... Downloading 12.65MB/s 10:06:13 35.8% Downloading 12.65MB/s 10:06:13 35.8% Downloading 12.65MB/s 10:06:13 35.8% Downloading 12.65MB/s 10:06:13 35.8% Failed to write to '/vmfs/volumes/DatastoreHDD/MYDB/vmfs/volumes/VM2-SSD/MYDB/MYDB_3-flat.vmdk': No space left on device Chunk 06a4e3a01f50ec33ff788069edf6dedcbd13a0d78d1e14aacedd22e5d24118d6 cannot be found in the storage
~ # df -h Filesystem Size Used Available Use% Mounted on VMFS-5 364.5G 117.2G 247.3G 32% /vmfs/volumes/datastore1SSD VMFS-5 744.5G 476.6G 267.9G 64% /vmfs/volumes/DatastoreSSD VMFS-5 1.1T 1.1T 0.0B 100% /vmfs/volumes/DatastoreHDD vfat 4.0G 28.6M 4.0G 1% /vmfs/volumes/589ca7db-3f1b3ee5-858d-f8bc12125240 vfat 249.7M 174.7M 75.0M 70% /vmfs/volumes/adc8217a-eeee1720-fd15-bf13a3282df7 vfat 249.7M 174.7M 75.0M 70% /vmfs/volumes/c9b0847e-e8f0e14f-0c70-806d5bebbc00 vfat 285.8M 193.0M 92.9M 68% /vmfs/volumes/589ca7db-f6c1547b-f8ba-f8bc12125240
gchen Jul 20 7:07PM 2017
That was probably because restore wasn't writing to the same disk file. It looks like the disk file for the new VM is called MYDB-flat.vmdk while it is MYDB_3-flat.vmdk for the original.
You can remove the incomplete MYDB_3-flat.vmdk and temporarily rename MYDB-flat.vmdk to MYDB_3-flat.vmdk before the restore, and change back to MYDB-flat.vmdk after it is done.
Or you can take the other approach of restoring the vm to a directory under the datastore:
vertical init host2 s3://mybucket # now you can use host2 as the host id
mkdir -p /vmfs/volumes/DatastoreHDD/restore
vertical restore /vmfs/volumes/DatastoreHDD/restore -r 1 --restore-from MYDB@host1
You can then create the new VM from the disk file under /vmfs/volumes/DatastoreHDD/restore.
Of course, to go this route you need to first delete the MYDB vm on host2 to free up space.
LawcMnnn Jul 22 11:16AM 2017
The restore ran for a while. If I want to refresh the restore with a newer backup how can I "refresh" the data without starting over the process? or update the delta change?
LawcMnnn Jul 22 12:06PM 2017
Actually getting an error: ~ # vertical backup MYDB Creating a new virtual machine snapshot for MYDB Create Snapshot: Create snapshot failed Command '/bin/vim-cmd vmsvc/snapshot.create 6 2017-07-22-17-05-45 'Created by Vertical Backup 1.0.5' 0 1' returned 1 Removing all snapshots of MYDB
gchen Jul 22 12:40PM 2017
You can run the vmsvc/snapshot.create command manually to see if it gives you the same error. If so, do a snapshot consolidation from the vSphere Client. The vmware.log file under the VM folder may have more information.
When you run the second restore, just make sure it writes to the same file as the first restore. It will just download delta changes if it is the same file.
LawcMnnn Jul 22 2:35PM 2017
I think I have a space issue. This is the files in the DB directory ..I see some swap files and delta files.. can any of these files be deleted safely?
LawcMnnn Jul 22 2:36PM 2017
drwxr-xr-x 1 root root 3780 Jul 22 16:25 . drwxr-xr-t 1 root root 1400 Apr 6 2016 .. -rw-r--r-- 1 root root 13 Jul 19 14:00 MYDB-aux.xml -rw------- 1 root root 0 Apr 7 2016 MYDB-f9ec6894.vswp -rw------- 1 root root 85899345920 Jul 22 19:28 MYDB-flat.vmdk -rw------- 1 root root 8684 Jul 22 19:21 MYDB.nvram -rw------- 1 root root 516 Jul 22 16:25 MYDB.vmdk -rw-r--r-- 1 root root 77 Jul 22 16:25 MYDB.vmsd -rwxr-xr-x 1 root root 3552 Jul 22 16:25 MYDB.vmx -rw------- 1 root root 0 Apr 7 2016 MYDB.vmx.lck -rwxr-xr-x 1 root root 3563 Apr 7 2016 MYDB.vmx.old -rw-r--r-- 1 root root 3425 Jul 22 17:00 MYDB.vmxf -rwxr-xr-x 1 root root 3552 Jul 22 16:25 MYDB.vmx~ -rw------- 1 root root 122206679040 Jul 22 16:25 MYDB_3-000001-delta.vmdk -rw------- 1 root root 317 Jul 22 16:25 MYDB_3-000001.vmdk -rw------- 1 root root 12785676288 Jul 22 19:28 MYDB_3-000002-delta.vmdk -rw------- 1 root root 324 Jul 22 16:25 MYDB_3-000002.vmdk -rw------- 1 root root 751619276800 Jul 22 19:27 MYDB_3-flat.vmdk -rw------- 1 root root 470 Jul 22 16:25 MYDB_3.vmdk -rw-r--r-- 1 root root 178733 Apr 7 2016 vmware-16.log -rw-r--r-- 1 root root 306048 Apr 7 2016 vmware-17.log -rw-r--r-- 1 root root 163917 Apr 7 2016 vmware-18.log -rw-r--r-- 1 root root 308007 Apr 7 2016 vmware-19.log -rw-r--r-- 1 root root 163130 Apr 7 2016 vmware-20.log -rw-r--r-- 1 root root 61444 Apr 7 2016 vmware-21.log -rw-r--r-- 1 root root 579843 Jul 22 17:29 vmware.log -rw------- 1 root root 126877696 Apr 7 2016 vmx-MYDB-4193020052-1.vswp
gchen Jul 22 4:12PM 2017
Don't delete any files there. Those delta.vmdk files are snapshot files which should go way after you do a snapshot consolidation. You can try running 'vim-cmd vmsvc/snapshot.removeall` but that doesn't seem as reliable as doing it from the vSphere client.
gchen Jul 22 4:18PM 2017
By the way this forum supports the standard markdown syntax, so if you enclose multiple line text with ```, it will preserve newlines, like this:
```
line 1
line 2
line 3
```
Log in to comment