Research Targets Computational Experiments in the Cloud

By Nicole Hemsoth

July 27, 2011

Conducting computational experiments often requires extensive but oftentimes sporadic compute power. The need for vast resources for shorter bursts of high activity makes building statistical models of systems and related experiments seem like an ideal fit for cloud computing. However, questions remain about the practicality of using cloud resources, especially public clouds, for such work.

A team from St. Andrews University recently tackled a series of questions about how suitable cloud computing and virtualized hardware is for running time-sensitive experiments. When approaching the matter, Ian Gent and Lars Kotthoff imagined that CPU time would not be reliable and that they would not be able to confirm consistent results. For Kotthoff and Gent, it was especially critical to be able to ensure reproducability, thus this was a primary component of their methodology.

The outcome of their investigation demonstrated that while their suspicions were correct in cases where there were several virtual machines running on the same physical hardware, “there is no inherent variation introduced by using virtualized hardware compared to non-virtualized hardware.”

These findings were worth investigating further, so we spent some time talking with Lars Kotthoff from St. Andrews University. Specifically, we wanted to find out about the reliability and reproducability of results of key experiments and how these were priced, especially against the only other alternative–physical hardware.

Some of Kotthoff’s research into the viability of running time sensitive applications on Amazon’s cloud has come from Amazon Web Services. The company has also backed other research projects, including his work on prototyping a system based on the Minion constraint solver that automatically splits and distributes large problems across different machines.

On that note, it should come as no surprise that Kotthoff and Gent were conducting their “experiment on experiments” using Amazon Web Services.

HPCc: Can you describe what led up to this study? What were some of the perceived challenges for running scientific (and computational) experiments on virtualized hardware and were these expected problems present?

Kotthoff: The initial motivation for this study was that we realized that it took us a long time to set up experiments, run them, evaluate them and make sure that they are reproducible. Especially the last point was a particular headache for us because we use sets of machines with different hardware and configurations. We also observed large variations in runtime of the same experiment on the same
machine because of the way CPU caches are shared among several cores.

The way we tried to tackle both of these problems was by running experiments several times and then using statistical methods to evaluate them. This has two main repercussions. First, we are going to need more resources to run experiments. Second, it becomes less important to run them on a set of machines
with the same hardware and software configuration. This is how we had the idea to run experiments in the cloud.

We had previously observed that the clocks of virtual machines running on desktop virtualisation software such as VMWare or VirtualBox were very unreliable. This is what prompted us to do this investigation.

HPCc: Along those lines, can an you provide something of a quick synopsis of your main findings?

Kotthoff:  We found that only for the small types of virtual machines in the Amazon cloud getting reliable CPU timings was a significant problem. For larger types, the reliability becomes as good as on physical hardware or even better. We do of course not know the actual configuration of the hardware of the cloud computing provider, but it is not hard to guess that for the smaller virtual machine type more are running on the same physical hardware and therefore the virtualization overhead is higher and affects the reported CPU times in the virtual machine. The largest type of virtual machine probably corresponds to a physical machine and introduces no measurable additional variability.

HPCc: Describe the reliability and reproducibility angles to your research—what we the outcomes and what does this mean for researchers considering using virtualized hardware?

Kotthoff: Reliability and reproducibility of results is essential for our research. If people compare different methods, there is almost always an empirical evaluation as well. The justification for new research is almost always that it decreases the time needed to solve a problem. If however the measured times are not reliable, the research findings are jeopardised.

This is especially crucial when implementing complex systems that solve artificial intelligence problems. During the implementation, many design decisions have to be made and algorithms have to be chosen. These choices are usually justified by empirical evaluation – either during the implementation itself or based on results reported in the literature.

Our results show that there is no inherent disadvantage to using virtualized resources for such experiments as long as the results are carefully evaluated and statistical methods used to identify outliers and establish confidence in the conclusions drawn from the results.

We believe that using a cloud to run experiments would, in terms of reproducibility of results, take our research to the next level. We can package the entire environment of the experiment as a virtual machine image, make it available to the public and reference it in publications. This makes reproducing our results much easier for other people.

Currently, we describe the hardware and sometimes the software used in experiments, but unless other people happen to have the same hardware, there is a significant barrier to being able to reproduce the exact experiments we ran.

HPCc: Do you think resources that are tailored for computationally intensive workloads like Amazon’s recent HPC offering, Cluster Compute Instances, will solve some of the problems you discuss?

Kotthoff:  That certainly seems to be case from what we saw in our experiments. It might however be “accidental” as virtual machines tailored for computationally intensive workloads tend to run exclusively on a physical machine. If many of those virtual machines with high specifications would run on for example on a single supercomputer, the original problem of a large variability in CPU times might arise again.

HPCc: With the above question in mind, what problems are going to remain semi-permanent hurdles for CPU-heavy workloads in the cloud? Is there hope on the horizon that it will someday compete with native/physical hardware?

Kotthoff:  In my opinion, the major problem at the moment is the cost associated with running large computational experiments in the cloud. The largest type of virtual machine in the Amazon cloud is, in terms of variability of reported CPU times, as good as physical hardware. The cost of running it is relatively high though. For a research group that regularly runs large-scale computational experiments, it would almost always be cheaper to buy physical hardware themselves.

I should point out that we are not so much concerned with the actual performance we get as long as the variability is low. While the virtual machine instances we used did not seem to be significantly slower than physical hardware in general, we did not evaluate this aspect.

HPCc: In what ways (or cases) is it not a good idea to run acomputationally-intensive task on cloud-based resources?

Kotthoff: If the task cannot easily be parallelised and takes a long time to run. Cloud computing has disadvantages over physical hardware in this scenario; for example virtual machines might be shut down without warning and there is no way to access the disk of the terminated instance to recover results. Using cloud-based resources is also a question of the available funds; if there is only a small amount of money available, it might be better to buy and run the experiment on physical hardware.

HPCc: Taking this in another direction, what is the best type of case for running CPU-intensive tasks on cloud resources?

Kotthoff: The best case is probably running a lot of experiments or experiments that can be distributed across many virtual machines easily when there are not enough physical resources available, e.g. impending paper deadline. Provided that the experiment can be made to run in a virtual machine, it is easy to distribute it across virtual machines in the cloud.

You can find out more about Gent and Kotthoff’s experiments here.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire