In January 2023 we did a major test between different cloud providers. We looked at the technical performance of their cloud servers. Exactly one year later, in January 2024, we will do this test again. We are curious to see if any notable changes have happened!

We can divide cloud providers into three different types: hyperscalers (Google Cloud, Amazon AWS ...), the mid-segment (DigitalOcean, Linode ...) and local players (TransIP, Combell ...). In this article we again compare a number of cloud providers from the mid-range segment. For more information about the different types of providers, read this blog article.

TL;DR? Jump to the conclusion at the very bottom!

What we compare

Once again in this blog post we make a purely technical comparison based on benchmarks of comparable virtual machines and their monthly price.

We did a benchmark of cloud servers from these providers:

• Hetzner (Germany)
• DigitalOcean (United States)
• Linode (United States)
• Vultr (United States)
• OVH (France)
• UpCloud (Finland)
• Scaleway (France)

A newcomer to this list is the French Scaleway. This time we ran the benchmarks on three different configurations:

• Cloud servers with a shared vCPU of 2 cores and 4 GB of RAM.
• Cloud servers with a shared vCPU of 4 cores and 8 GB of RAM.
• Cloud servers with a dedicated vCPU of 4 cores and 16 GB of RAM.

We included a cloud server from UpCloud in the January 2023 benchmark. That was incorrect, as their cloud servers all work with a dedicated vCPU. This time we have not included a cloud server from UpCloud in the benchmarks with a shared vCPU.

What are we not comparing?

When you look for a cloud provider, you obviously not only look at the performance of their cloud servers and the price, but also at their broader offering. For example, do you also need block storage? Or managed databases or Kubernetes? Equally important is the accessibility and willingness of the helpdesk. These items are rather personal and some are difficult to measure. In this comparison we only focus on the performance of these cloud servers.

Cost price

The graph below contains an overview of the cost per month for each cloud server. USD was converted to EUR for comparison.

In the first graph we compare the prices of the providers from June 2022, January 2023 and January 2024. The small differences between DigitalOcean and Vultr have more to do with the exchange rate (as of January 9, 2023) than a real price drop.

The progress of the benchmark

We ran the benchmark on an Ubuntu 22.04 LTS installation on which we installed all updates. For the benchmarks we used Phoronix Test Suite v10.8.4. Phoronix performed each test three times. If a standard deviation greater than 2.5% was observed between tests, an additional test was performed until the standard deviation fell below 2.5%, with a maximum of 40 tests. As a result, the average value was recorded each time.

We ran the benchmarks below, with a focus on the system, a single and multi-core CPU, the RAM and the storage space.

• pts/apache (focus on the systeem). This is a test of the Apache HTTPD web server. This Apache HTTPD web server benchmark test profile makes use of the Golang "Bombardier" program for facilitating the HTTP requests over a fixed period time with a configurable number of concurrent clients.
• pts/hint (focus on single core CPU). This test runs the U.S. Department of Energy's Ames Laboratory Hierarchical INTegration (HINT) benchmark.
• pts/compress-7zip (focues on multi core CPU). This is a test of 7-Zip compression/decompression with its integrated benchmark feature.
• pts/stream (focus on the memory). This is a benchmark of Stream, the popular system memory (RAM) benchmark.
• pts/postmark (focus on the storage). This is a test of NetApp's PostMark benchmark designed to simulate small-file testing similar to the tasks endured by web and mail servers. This test profile will set PostMark to perform 25,000 transactions with 500 files simultaneously with the file sizes ranging between 5 and 512 kilobytes.

Results

Benchmark: pts/apache

Focus on the system. A higher score is better.

Shared vCPU 2 cores, 4 GB ram

Shared vCPU 4 cores, 8 GB ram

Dedicated vCPU 4 cores, 16 GB ram

This benchmark failed on the OVH cloud server and therefore has no results. We started this benchmark several times, but it failed every time (all other benchmarks on the OVH server succeeded, by the way).

Benchmark: pts/hint

Focus on single core CPU. A higher score is better.

Shared vCPU 2 cores, 4 GB ram

Shared vCPU 4 cores, 8 GB ram

Dedicated vCPU 8 cores, 16 GB ram

Benchmark: pts/compression-7zip

Focus on multi core CPU. A higher score is better.

Shared vCPU 2 cores, 4 GB ram

Shared vCPU 4 cores, 8 GB ram

Dedicated vCPU 8 cores, 16 GB ram

Benchmark: pts/stream

Focus on the memory. A higher score is better.

Shared vCPU 2 cores, 4 GB ram

Shared vCPU 4 cores, 8 GB ram

Dedicated vCPU 8 cores, 16 GB ram

Benchmark: pts/postmark

Focus on the storage. A higher score is better. Each VM used ext4 as its file system.

Shared vCPU 2 cores, 4 GB ram

Shared vCPU 4 cores, 8 GB ram

Dedicated vCPU 8 cores, 16 GB ram

Global overview

It is not our intention to identify "the best" and "the worst" provider based on these tests.

We have made a percentage comparison of each benchmark. We have made a total percentage distribution of all those scores to arrive at a final comparison. You can see this in the graphs below.

Shared vCPU 2 cores, 4 GB ram

Vultr and Scaleway are the outliers when we look at (Apache) system performance. We also see this in the multi-core CPU benchmark (7-zip). When it comes to the performance of the underlying storage, Scaleway leaves everyone else far behind.

But that comes with a price. Linode and Scaleway are by far the most expensive. Price-performance ratio compared, DigitalOcean and Vultr are on the expensive side. Linode and OVH are worth their money. Hetzner scores similarly for a much lower cost price.

Shared vCPU 4 cores, 8 GB ram

The situation is completely different if we upgrade the servers to 4 shared cores and 8 GB of RAM. The big winner here is Hetzner; both in terms of technical performance and price.

In the general system benchmark (Apache), Hetzner is the leader, closely followed by Digital Ocean and Linode. Hetzner is also the leader in the multi-core benchmark (7-zip). If we look at the memory benchmark (Stream), Hetzner takes the cake again.

Price-performance ratio, Digital Ocean, Linode and Vultr in particular are very expensive. Only with Hetzner you get a lot for the price.

Dedicated vCPU 8 cores, 16 GB ram

Note: the Linode server only had 8 GB of ram.

Vultr was in the middle of the pack in the 'shared vCPU' tests, but does boast a dedicated vCPU - albeit at a high price. OVH scores very low, but the 'Apache' benchmark failed time and time again, meaning no results could be achieved.

In the Apache benchmark, Hetzner, Vultr and Scaleway color the top three. Vultr especially earns a lot of points on the Stream benchmark (RAM memory).

If we look at the price-performance ratio, Digital Ocean, OVH, UpCloud and Scaleway are expensive. Linode and Vultr are worth their price (but Linode only had half RAM!). Hetzner is a real leader for its low price.

Conclusion

Although all these cloud providers supply similar servers in terms of specifications, we still see major differences in both performance and corresponding cost.

Just like last year, Hetzner is once again making a very good move from a price-performance perspective.

Want to try it yourself?

Would you like to try one of these cloud providers? With the (affiliate) links below you will already receive a starting budget (and we can do these benchmarks again next year 😉).

• Hetzner - €20
• DigitalOcean - $200, te be used within 60 days
• Vultr - $100, te be used within 14 days
• UpCloud - €25