Out of memory during backup

misterbean    Apr 4 6:46PM 2018

Vertical backup 1.1.5 on ESXI 6.5, vertical runs out of memory running backup. Tried "--no-chunk-listing", did not help.

|----Total Memory......5.00 MB

                                         |----Touched...........1.12 MB
                                         |----Total Minimum (base)1.22 MB
                                         |----Total Maximum (base)1.22 MB
                                         |----Effective Minimum.1.74 MB
                                         |----Target Allocation.1.22 MB
                                         |----Total Memory......777.00 MB
                                         |----Reserved Memory...773.00 MB
                                         |----COW...............28.04 MB
                                         |----Mapped............749.25 MB
                                         |----Touched...........68.80 MB
                                         |----Total Minimum (base)773.62 MB
                                         |----Total Maximum (base)773.62 MB
                                         |----Effective Minimum.773.86 MB
                                         |----Target Allocation.773.62 MB

./vertical backup --no-chunk-listing fs Vertical Backup 1.1.5 Trial license expires on 2018-05-01 Storage set to sftp://esxibackup@192.168.24.6/esxi Listing all virtual machines Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest No previous backup found Virtual machine fs is powered on Removing all snapshots of fs Creating a new virtual machine snapshot for fs Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk Uploading file data2-flat.vmdk Uploaded file data2-flat.vmdk 197.91MB/s 00:25:52
Uploading file fs-flat.vmdk Uploaded file fs-flat.vmdk 183.46MB/s 00:01:51
Uploading file data2-000001-delta.vmdk Uploaded file data2-000001-delta.vmdk 12.92MB/s 00:00:01
Uploading file data-flat.vmdk Uploaded file data-flat.vmdk 172.00MB/s 01:51:45
Uploading file mm2-flat.vmdk Uploaded file mm2-flat.vmdk 215.09MB/s 00:25:23
Uploading file data2-000002-delta.vmdk Uploaded file data2-000002-delta.vmdk 87.93MB/s 00:00:00
Uploading file mm-flat.vmdk Uploading ***************************************************--------------------------- 101.54MB/s 01:27:00 66.3% Failed to read file '/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-flat.vmdk': MemoryError()


gchen    Apr 4 9:25PM 2018

How big is the mm-flat disk?


misterbean    Apr 6 2:21PM 2018

mm-flat.vmdk is approx 1.65TB


gchen    Apr 7 8:37PM 2018

I'll release a new version next week that should use much less memory. I'll let you know when it is ready.


gchen    Apr 13 9:44PM 2018

Sorry I hasn't been able to work on the new version this week. Should be able to get it done early next week. I'll send you a new 30-day trial license when the new version is out.


gchen    Apr 17 10:11PM 2018

I uploaded a new version 1.2.0 at http://acrosync.com/esxi/vertical.

This version should be much more memory efficient than previous versions.

If you log into your customer page you should also see a 30-day license there. You can activate it by running vertical license activate and then entering the license code.


misterbean    Apr 19 1:31PM 2018

Hi.. Thank you for the updated version. Unfortunately, it also seems to fail with a memory error:

                                    |----Total Memory......24.00 MB
                                         |----Reserved Memory...2.00 MB
                                         |----Touched...........1.12 MB
                                         |----Total Minimum (base)1.22 MB
                                         |----Total Maximum (base)1.22 MB
                                         |----Effective Minimum.2.20 MB
                                         |----Target Allocation.1.22 MB
                                         |----Total Memory......773.00 MB
                                         |----Reserved Memory...751.00 MB
                                         |----COW...............27.33 MB
                                         |----Mapped............720.88 MB
                                         |----Touched...........68.03 MB
                                         |----Total Minimum (base)750.18 MB
                                         |----Total Maximum (base)750.18 MB
                                         |----Effective Minimum.751.62 MB
                                         |----Target Allocation.750.18 MB

[root@localhost:/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/verticalbackup] ./vertical backup fs Vertical Backup 1.2.0 Trial license expires on 2018-05-01 Storage set to sftp://esxibackup@192.168.24.6/esxi Listing all virtual machines Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest No previous backup found Virtual machine fs is powered on Removing all snapshots of fs Creating a new virtual machine snapshot for fs Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2-000003.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-000001.vmdk Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk Uploaded file /vmfs/volumes/datastore1/fs/fs_1-000001.vmdk Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk Uploading file data2-flat.vmdk Uploaded file data2-flat.vmdk 196.56MB/s 00:26:02
Uploading file data2-000003-delta.vmdk Uploaded file data2-000003-delta.vmdk 35.16MB/s 00:00:03
Uploading file fs-flat.vmdk Uploaded file fs-flat.vmdk 157.69MB/s 00:02:09
Uploading file data2-000001-delta.vmdk Uploaded file data2-000001-delta.vmdk 95.27MB/s 00:00:00
Uploading file data-flat.vmdk Uploaded file data-flat.vmdk 164.25MB/s 01:57:02
Uploading file fs_1-flat.vmdk Uploaded file fs_1-flat.vmdk 204.75MB/s 01:25:21
Uploading file fs_1-000001-delta.vmdk Uploaded file fs_1-000001-delta.vmdk 91.37MB/s 00:00:00
Uploading file mm2-flat.vmdk Uploaded file mm2-flat.vmdk 206.25MB/s 00:26:28
Uploading file data2-000002-delta.vmdk Uploaded file data2-000002-delta.vmdk 77.13MB/s 00:00:00
Uploading file mm-flat.vmdk Uploading ******************************************************************************- 74.84MB/s 00:02:44 99.2%Exception in thread Thread-26: Traceback (most recent call last): File "threading.py", line 801, in __bootstrap_inner File "threading.py", line 754, in run File "vertical_snapshot.py", line 529, in uploadChunkThread File "vertical_utils.py", line 391, in encryptAESGCM MemoryError


gchen    Apr 19 9:17PM 2018

You can try to increase the chunk size by initializing the storage this way:

./vertical init -c 4M hostid sftp://esxibackup@192.168.24.6/esxi

This should reduce the size of the chunk list in memory. The downside is that the deduplication will become less efficient with a larger chunk size. However, I think the 4MB chunk size is reasonable for disks larger than 1TB (this is the default average chunk size in Duplicacy).


Log in to comment
Copyright © Acrosync LLC 2017