Use Internet2 SiteID

Already have an Internet2 SiteID?
Sign in here.

Internet2 SiteID


Cloud vs. Datacenter Costs for High Performance Computing (HPC): A Real World Example

Jun 19, 2017, by Sara Jeanes
Tags: 100, Amazon Web Services, Frontpage News, Google Cloud Platform, Internet2 NET+

By Boyd Wilson, Sara Jeanes, and Amy Cannon


After addressing the 5 Things to Consider When Deploying High Performance Computing (HPC) to the Cloud, it may be helpful to consider cost and benefits of running HPC workloads in the cloud versus running them on premise. This post will discuss costs for an on prem data center in the Southeast, and instances running in NET+ Amazon Web Services (AWS) and NET+ Google Cloud Platform (GCP), currently in NET+ Service Validation. This is the second of two posts on the HPC in the Cloud BoF from Internet2 Global Summit.

On Prem Model

These on prem numbers use pricing from a datacenter in South Carolina. The cost of power is variable by state and provider. These numbers include commercial off-the-shelf (COTS) servers, power, power equipment, and a free data center building.

One server with 36 cores across 2 CPU, 64 GB Ram, rack kit, cabling, PDU and 5 year warranty is $14,375 or $0.0091 per core-hr. This includes no UPS or Generator, no electrons, no cooling, no sysadmin, no netadmin, no building. It also assumes a 5 year refresh, which is longer than most 3-4 year refresh cycles). Each server consumes ½ kW of power.

Power and Cooling
Power and power equipment here is defined as including everything to ensure clean, uninterrupted electron flow to a rack including maintenance and emergency generation. This includes a UPS, generator, and transfer switch which total $0.13 each kilowatt-hour (kW h).

Electron charges are $0.08 kW h server and half again as much for AC units ($0.04 kW h) to cool servers totaling $0.12 kW h.

That brings the total for UPS, generator, electrons, and cooling to $0.25 kW h.

If each server uses ½ kW with an assumes 100% capacity data center then $0.125 / 36 cores is $0.0035 core-hr for power and cooling. This also assumes a completely free building.

Networking is approximately $2500 month per 10G Port or $0.057 hr. This divided by 36 cores equals $0.0016 per core-hr.

Adding up the above values we get:

  • $0.014 per core hour if power and cooling is 100% utilized and servers are 100% utilized
  • $0.021 per core hour if power and cooling is 50% utilized and servers are 100% utilized
  • $0.023 per core hour is the most likely scenario, where power and cooling is 50% utilized and servers are 85% utilized. This allows time for benchmarks, upgrades, and expects offline nodes. Over the lifetime of power and cooling equipment in the data center, it generally goes from mostly empty to a maximum load  of 80% capacity recommended by UPS and generator manufacturers.

All three of these assume a free building, and no labor for systems or network administration.

Cloud Model (NET+ Amazon Web Services (AWS))

AWS offers a Compute Optimized instance, the C4.8xlarge that currently offers 36 cores and 60GB Ram. The version number (the “4” in “C4”) will increment as new hardware is added.

Pricing for an On Demand instance is $1.591 per instance or $0.0419 per core hour. The latest CPU can be used as soon as they are released. GPUs are pluggable.

If buying services in a upfront model is preferred, a Reserved 3 year instance is $0.852 per instance or $0.0237 per core hour.

Spot Instances are instanced that can be reclaimed on 5 minutes notice and are allocated on a bid system. Current Spot pricing in Oregon is $0.60 per instance or $0.0167 per core hour. Current Spot pricing in Ohio is $0.39 per instance or $0.0108 per core hour. Spot instances can be interrupted, so checkpoint or small jobs with restart capability are best suited for these instances. NET+ AWS currently offers a 3% discount that is not included in the above pricing

Cloud Model (NET+ Google Cloud Platform (GCP))

GCP offers the N1 High-CPU machine type including a N1-highcpu-32, with 32 cores and 28.8GB Ram.

On Demand pricing is $1.1344 per instance or $0.03545 per core hour.

Running an instance continuously will net a sustained use discount, bring the instance price down to $0.79408 per instance or $0.024815 per core hour.

Preemptible instances can be interrupted at any time, but typically 2-3% of instances are typically preempted in any given hour and instances will run for no longer than 24 hours. Preemptible instanced are $0.2400 per instance or $0.0075 per core hour. GCP is currently undergoing NET+ Service Validation and may see additional discounts in the future.

Note: Azure does not currently offer an interruptible or preemptable instance type.


Cost (Per Core Hour)

On Prem (P/C 100% utilized and servers 100% utilized)


On Prem (P/C 50% utilized and servers 100% utilized)


On Prem (P/C 50% utilized and servers 85% utilized)


AWS On Demand


AWS Reserved 3 yr


AWS Spot Instance


GCP On Demand


GCP with Sustained Use Discount


GCP Preemptable Instance



You can run cost comparisons between cloud and on prem but running a cluster 24/7 is never going to beat the cost of running one for 4 hours, even if it is many times the size. NET+ AWS and GCP offer advantages that can never be provisioned on premise. On demand compute completely changes the complexion of the question. One example of this is Andrew V. Sutherland, a computational number theorist and Principal Research Scientist at MIT who ran 220,000 cores on GCP to make 10^17 calculations and then dismissed the cluster when he was through with it. Next time he plans to build a 400,000 core cluster…

For more information on NET+ Amazon Web Services (AWS) and NET+ Google Cloud Platform (GCP), email