Danny Jan 13 9:47PM 2018
Hi,
For some reason upload to mounted drive is limited to 5Mb/s.
My setup:
ls /vmfs/volumes/
311e72e2-4cae7b69-be37-66b745ef2970 59940e55-4c6c944c-9a3a-38d547ca552a ARCHIVE1 S-SSD2 59940661-89cae3f8-c5a9-38d547ca552a 5a078c6c-374971af-9907-38d547ca552a P-SSD1 b330efe4-9ff20e61-02e6-65cfcc9491b9 59940d62-f9eabb98-83fe-38d547ca552a 5a178628-1e3337e4-a519-38d547ca552a S-SSD1
As you can see from here I have 2 physical SSD and 1 HDD drives mounted to ESXI as storages. ESXI itself installed to a USB stick. ESXI version: 6.5.0 Update 1 (Build 6765664) - free license
I installed vertical backup to storage ARCHIVE1:
mkdir /vmfs/volumes/ARCHIVE1/verticalbackup cd /vmfs/volumes/ARCHIVE1/verticalbackup wget http://verticalbackup.com/esxi/vertical chmod u+x vertical
initialized it:
./vertical init esxi /vmfs/volumes/ARCHIVE1/backups/
and started backup of vm-1 with thin provisioned 100GB HDD:
./vertical backup vm-1
The files of vm-1 are placed on SSD1, the backups folder is on ARCHIVE1. Physically these storages on the same machine. I'd expect quite fast speed but instead I see 5MB/s tops.
Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s 06:25:00 6.Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s 06:25:02 6.Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s ...and so on...
Here's the result of the benchmark run:
./vertical benchmark
Vertical Backup 1.1.5 Test directory: /vmfs/volumes/59940d62-f9eabb98-83fe-38d547ca552a Creating a 1,024M test file with random data Write time: 10.439, speed: 98.097MB/s Read time: 3.422, speed: 299.231MB/s Creating another 1,024M test file with random data Write time: 11.264, speed: 90.908MB/s Read and hash time: 6.559, speed: 156.124MB/s Storage set to /vmfs/volumes/ARCHIVE1/backups/ Upload time: 189.294, speed: 5.410MB/s Download time: 56.614, speed: 18.087MB/s
ARCHIVE1 is not used by anything else except the backup tool. What's wrong with it?
gchen Jan 14 12:08AM 2018
Storage set to /vmfs/volumes/ARCHIVE1/backups/
Upload time: 189.294, speed: 5.410MB/s
Download time: 56.614, speed: 18.087MB/s
In the upload test, 1024 random files with a size of 1M bytes are uploaded. This reflects the default size of chunks uploaded when backing up a virtual machine. So I think there is something wrong with the writing speed of that disk.
Danny Jan 14 10:51AM 2018
Upgraded ESXI from ESXi-6.5.0-20171004001-standard (Build 6765664) to ESXi-6.5.0-20171201001s-standard (Build 7273056).
./vertical benchmark
Vertical Backup 1.1.5 Test directory: /vmfs/volumes/59940d62-f9eabb98-83fe-38d547ca552a Creating a 1,024M test file with random data Write time: 5.281, speed: 193.888MB/s Read time: 3.256, speed: 314.493MB/s Creating another 1,024M test file with random data Write time: 5.335, speed: 191.927MB/s Read and hash time: 5.896, speed: 173.688MB/s Storage set to /vmfs/volumes/ARCHIVE1/backups/ Deleting old remote test files
Upload time: 92.026, speed: 11.127MB/s Download time: 14.169, speed: 72.270MB/s
I tried to move around 30-50Gb vmdk files from SSDs to HDD back and forth, the speed seems to be normal. According to the benchmark results write/read speeds are good: 193.888MB/s / 314.493MB/s, upload/download speeds are crap. Any thoughts?
gchen Jan 15 5:47PM 2018
Can you run ./vertical benchmark --datastore /vmfs/volumes/ARCHIVE1/backups/ to test the write/read speeds of the HDD?
Danny Jan 16 6:52PM 2018
@gchen ./vertical benchmark /vmfs/volumes/ARCHIVE1/backups/
Vertical Backup 1.1.5
Test directory: /vmfs/volumes/ARCHIVE1/backups/
Creating a 1,024M test file with random data
Write time: 20.812, speed: 49.203MB/s
Read time: 6.064, speed: 168.860MB/s
Creating another 1,024M test file with random data
Write time: 22.423, speed: 45.668MB/s
Read and hash time: 6.311, speed: 162.245MB/s
Storage set to /vmfs/volumes/ARCHIVE1/backups/
Upload time: 90.062, speed: 11.370MB/s
Download time: 14.081, speed: 72.725MB/s
gchen Jan 17 8:13PM 2018
So the speed of writing a 1G file is 49MB/s, while the speed of writing 1024 1M files is 11MB/s. There may be some issue with your HDD? What format is that disk? Can you try copying many small files to it to see what the speed is?
Danny Jan 22 4:58AM 2018
@gchen, It's brand new Western Digital 4TB Intellipower SATA 6Gb/s 64 MB Cache 3.5-Inch NAS Desktop Hard Disk Drive - Red (WD40EFRX). I converted it into ESXi's storage drive. May be my usb flash is dying and it causes such side effects, will try to replace it.