Public cloud measurement report

Public cloud performance measurement report

Silviya Ivanova Blog Leave a Comment

Introduction

In these public cloud performance tests, StorPool’s aim was to assess the block storage offerings of a number of public clouds, including Amazon AWS, Digital Ocean, DreamHost and OVH vs. a number of StorPool-based public cloud offerings. To the best of our ability we’ve selected VM instance types and everything else in the configurations to be identical because we are comparing storage systems/offerings, not other aspects.

The tests were performed between November 2018 and February 2019 by StorPool.

All systems under test are in production clusters. All but one of the systems (StorPool BCP) are behind generally available public cloud offerings, so our results are easily reproducible.

The StorPool BCP (standing for Best Current Practice of design) system is a production cluster of a StorPool customer, which is approximately 70% full with a high density shared hosting use-case. This cluster uses 2x 25Gnetworking for storage and public Internet access. In the StorPool cluster, the VMs were running on a hypervisor host separate from the storage nodes, to emulate a hyper-scale deployment (not hyperconverged).

We performed 4 types of tests:

  1. PGBENCH – a database benchmark, perhaps closest to “application performance”
  2. Sysbench/mysql – a second database benchmark for control of PGBENCH results
  3. fio – a set synthetic benchmarks — random reads/writes, sequential reads/writes, latency measurements
  4. rsync – copying files and syncing, simulating rapid deployment and backup workloads

Between these tests we consider PGBENCH and Sysbench/mysql to be somewhat representative of most real-world transactional workloads (including most web applications), and the remaining tests are somewhat further away from being representative of real-world applications.

The Virtual machines

7 virtual machines with identical parameters were procured from all tested clouds. Medium sized VMs with 16 GB RAM and with dedicated CPUs were selected to represent a medium-sized database server (the system in the heart of many web applications).

* – the VM procured from OVH had 30 GB RAM instead of the required 16GB. In our test this was adjusted for by instructing the Linux kernel to ignore all but 16GB of the memory.

The storage volumes

We procured block storage volumes, associated with each VM, attached to the VMs as virtual disks. The goal here was to give each provider a fair chance with their SSD-based block storage offering. In Digital Ocean and AWS we procured larger volumes to ensure a higher IOPS limit. In OVH and DreamHost we could not find information about per-GB performance so we went with 200GB size, which is enough for our tests. In the StorPool-based clouds our customers don’t use IOPS limits to allow each VM to receive high performance for short periods of time, so we went with 200GB size as well. This is typical usage for a StorPool system.

PGBENCH Large DB

  • Simulate a large OLTP (transactional) database — most web and mobile apps
  • “On-Disk Test” settings from Postgresql Wiki
  • dataset is 64GB, i.e. 4x RAM
  • Number of threads (-j parameter) is one-half of the number of clients
  • The number of clients (-c parameter) is varied between 1 (simulating light load and lowest latency) and 32 (simulating more than maximum recommended production load for an 8 vCPU database). – Reference for test scenario and parameters https://wiki.postgresql.org/wiki/Pgbenchtesting#Memory_vs._Disk_Performance

Presented as two charts at different zoom levels to make sure very different results are clearly visible.

This shows extreme differences in performance between the block storage offerings. We decided to investigate what the differences are caused by, so we ran some experiments as pairs of tests.

The first pair we ran was StorPool BCP unlimited (default) vs StorPool BCP with a 20k per-volume IOPS limit configured in StorPool. This was to determine how an IOPS limit influences application performance. Here it is presented against a test on AWS EBS 10k, which we consider the reference.

In these 3 tests the VMs are with 32 GB RAM by exception, otherwise identical conditions, like the rest of the document.

The result is presented in the following chart:


The conclusion we draw from this experiment is that on the same storage system an IOPS limit generally controls how many TPS we can get at large latency (suitable for batch processing), but does not directly affect latency under a fixed transactional load. For example if you have a load of 2,000 TPS, the StorPool BCP configuration would deliver half the latency of the $333 per month AWS EBS volume, at a fraction of the price.

In the second pair we compare AWS gp2 with a 3,000 IOPS limit (which is $100 per month) against DigitalOcean’s block storage offering (also for $100 per month). This is again presented alongside AWS gp2 with a 10,000 IOPS limit for reference.

In these 3 tests the VMs are with 32 GB RAM by exception, otherwise identical conditions, like the rest of the document.

From this experiment we draw the conclusion that IOPS ratings don’t really mean much on their own. A volume which is rated for 3,000 IOPS handily beats Digital Ocean’s 7.5-10k IOPS volume. For example at 500 TPS, the AWS service completes transactions for 2.0 ms while the DigitalOcean service handles the same workload with 8.1 ms latency, a 4 times difference.

Comparing against the AWS 10k IOPS service (which is $333 per month) seems pointless here.

PGBENCH Small DB

  • Simulate a small OLTP (transactional) database — most web and mobile apps
  • “Mostly Cached” settings from Postgresql Wiki
  • dataset is 14.4GB, i.e. 0.9x RAM
  • Number of threads (-j parameter) is one-half of number of clients
  • The number of clients (-c parameter) is varied between 1 (simulating light load and lowest latency) and 32 (simulating more than maximum recommended production load for a 8 vCPU database).
  • Reference for test scenario and parameters https://wiki.postgresql.org/wiki/Pgbenchtesting#Memory_vs._Disk_Performance

Again presented as two zoom levels.

Even when a database is smaller than RAM, the performance of the storage system has huge influence over its performance.

Sysbench/mysql

We ran Sysbench with a Mysql database, for control over the PGBENCH results. This shows that the results and extreme differences are reproducible with a completely different benchmark and database stack.

The database sizes are 50GB and 5GB respectively.

These tests were added to the mix after we had finished with testing at DreamHost.

*Tests not run at DreamHost
*Tests not run at DreamHost
*Tests not run at DreamHost
*Tests not run at DreamHost

FIO tests

We ran the usual suite of synthetic benchmarks, which show the envelope of the tested services.

These are:

  • random 4k queue depth 1 – for latency under light load
  • random 4k queue large queue depth – for IOPS throughput
  • sequential workload with large queue depth – for MB/s throughput

FIO latency

  • Transactional workload
  • Random read; random write
  • Block size 4k
  • Queue depth 1
  • Simulates small transactional workload
  • The poor results on these tests of DigitalOcean, OVH and DreamHost, especially for writes are the most likely reason PGBENCH and Sysbench/mysql are getting so low results on these services.

FIO IOPS

  • Parallel random workload
  • Random read/write 50/50 mix
  • Block size 4k
  • Queue depth 64Get the full report and detailed cloud performance comparison by downloading the document from the link bellow. Explore the measurements and benchmark easily where your service stays on the market.

FIO MB/s

  • Streaming workload
  • Sequential read; Sequential write
  • Block size 128k
  • Queue depth 64

RSYNC copy

  • ext4 filesystem
  • Linux kernel source (4.17.13) times 50
  • 3.12M files
  • 46 GB data
  • destination directory is empty
  • clear cache, then copy files to destination directory
  • approximates deployment workload readdir(), sequential reads, sequential writes
*Tests not run at eApps

RSYNC sync

  • ext4 filesystem
  • Linux kernel source (4.17.13) times 50
  • 3.12M files
  • 46 GB
  • source and destination directories identical. Each holding 3M files.
  • clear cache, then sync files to destination directory
  • approximates backup workload – dominated by readdir() and stat()

Book an Online Meeting

If you want to know more about StorPool and how we achieve this impressive levels of performance book a short meeting with one of our experts.

Leave a Reply

Your email address will not be published. Required fields are marked *