HPC in the Cloud: Elite HPC and the Cloud Culture Clash

Dedicated to Covering Enterprise & Scientific Large Scale Cloud Computing

HPC in the Cloud >> Blogs

Blog: Behind the Cloud

July 01, 2010

Elite HPC and the Cloud Culture Clash

We have case studies. We have examples of HPC in action in the clouds in one form or another. But for a still vast majority of large supercomputing centers—the terra firma for traditional HPC—cloud discussions are but fluff. They revolve around business models. And for the lofty aims of the supercomputing elite, business models have nothing to do with the world they live in.

That resistance is rooted in solid arguments now--no question about it--but as the technologies that underlie cloud, including the development of low-latency boosts to cloud power evolve, these arguments are bound to weaken, leaving only a culture question. And this will probably coincide with the arrival of a new group of thinkers who were weaned on the cloud in its infancy via everything-imaginable-as-a-service. That means any of us under what, say 25?

To quote an unnamed director of a supercomputing site following an off-the-cuff discussion at International Supercomputing Conference this year, “what everyone’s forgetting is that we [supercomputing centers and large research centers] have no incentive to have anything to with clouds. They offer no real benefit outside of cost—and that’s not even assured. It’s just a business model. It has no value and we have no reason to consider it since what we have now performs well. Give me a reason to think it is going to revolutionize what we’re doing and I’ll be glad to take a look, but it has nothing to offer, at least not yet.”

Fair enough.

When it comes to clouds and HPC, at least in the scientific computing arena, the key question almost invariably comes down to the matter of performance, which has led to overarching question, “if cost isn’t the issue, why would I ever bother to experiment with the cloud when standard HPC has been providing the compute resources needed to begin with.” This question most often stems from research and academia where, true enough, the performance is predictable and there are no concerns about virtualization slowdowns and bottlenecks and the same security concerns that have been around since the cluster was first unleashed are essentially the same.

In short, It is hard to imagine a bright future for HPC in the cloud when the single greatest concern for HPC is rooted in performance--and the single most-discussed flaw with cloud is performance. The two most important goals are at complete odds with one another. At least in the present.

It’s hard to disagree with the points here when we're putting this in the context of the present--where is the incentive for big HPC users if the performance question still hasn’t been broadly answered? To refine this question a little further, where is the incentive for HPC users who already have invested in their own clusters since the smaller outfits who have ready-made HPC on demand via the cloud are often already in simply because of cost?

Even more importantly, if the incentive comes, who is to say that it will ever mean much considering that so many jobs are dependent on the world just as it is, thank you very much.

What these questions embody is substance, but underlying all of this is the issue of culture. Outside of performance, service-level worries, and the other mess of cloud issues we are all well aware of, all the incentive in the world still might not be enough. This makes me wonder what will happen when this fresh new crop of graduating PhD students comes trickling forth from Berkeley, MIT and other universities that are looking to clouds in varying ways will change the culture. Actually, I don’t really wonder…do you?

It was interesting that the unnamed source quoted at the beginning did clarify his position by inserting the “not yet” as it might mean that there is a glimmer of interest—that is, of course, if the performance roadblocks are lifted. Or it might have just been wishful thinking that he was actually considering the possibilities somewhere in the back of his mind. But probably not.

Posted by Nicole Hemsoth - July 1 @ 12:02PM, Eastern Daylight Time

(Digg, Technorati, more)

Discussion

There are 2 discussion items posted.

Consistent with what I've been seeing
Submitted by rgillen on 07/01/2010 - 11:35AM

This article resonates well with my thoughts and some of the internal cultural road blocks I've encountered. That being said, there is - among some of us - a valuing of other metrics (besides raw performance). Things such as developer productivity and availability, and reducing the overall time to insight. It will be interesting to see how the industry evolves, and how super computing giants such as ourselves adapt, but I believe the cloud computing paradigm holds some interesting options for the future of science and am glad to be a small part of that environment.

Post #1

Submitted by Badri on 07/07/2010 - 2:08AM

I agree with the arguments. I think there are two reasons why people may indeed subscribe to HPC in the cloud:
(a) If I am an occasional HPC user, it makes sense.
(b) Infrastructure clouds allow the possibility of running the program in my own machine images (implying libraries, OS environment etc)

Lastly, SRIOV and other technologies are improving performance, so a wider range of apps may become feasible to run on the cloud and in virtualised environments.

Post #2

Nicole Hemsoth

Nicole Hemsoth is the managing editor of HPC in the Cloud and will discuss a range of overarching issues related to HPC-specific cloud topics in posts, which will appear several times per week in Behind the Cloud.

More Nicole Hemsoth

Feature Articles

On Par for 2 Billion: 3PAR Discusses Cloud Strategy

Storage vendor 3PAR has been at the heart of an intense bidding war between HP and Dell due to its unique refinements and developments in virtualized storage platform concepts. Thin provisioning and a focus on the needs of large-scale enterprises and cloud providers have catapulted the company into the public eye but as 3PAR's Craig Nunes discusses with HPC in the Cloud, the cloud strategy has been consistent since 1999--even if the world is just taking notice now.
Read More...

Busting the "Cloud in a Box" Myth

The concept of private clouds is gaining traction and due to the buzz, more enterprises are taking a much closer look at the possibility—if they haven’t taken steps to virtualize some or all of their infrastructure already. For those who have not yet made the transition, a lack of understanding of the complex process behind private cloud implementation is at the core of hesitancy, therefore vendors are looking for ways to convince users to fear not, the private cloud is not only within reach—but simple to step into.
Read More...

Achieving Ultra High Performance in the Cloud

Companies in competitive domains, such as financial services, create large data repositories containing significant amounts of data collected from daily operations. Using supercomputers to analyze these massive datasets might yield the highest level of performance, but this is prohibitively expensive. Using proprietary, custom-built HPC atop cloud environments is also a viable option--although one that does come with a series of drawbacks that must be mitigated to achieve critical performance levels.
Read More...

Read more HPC in the Cloud features...

Around the Web

Application Flexibility and the New Enterprise Architecture

Aug 31 | Application delivery strategies must be shaped with flexibility in mind as the number of platforms delivering core applications is bound to change with time. Since a greater number of devices and platforms are entering the infrastructure mix, those who do not adapt quickly face being locked into strategies that do not mesh well with new developments. Read more...

A Maturation Stage for Storage Virtualization

Aug 27 | Storage virtualization has been gaining momentum as it moves from concept to practice but some suggest the offerings in this realm have not matured sufficiently and require a longer maturation process before wider adoption occurs. Read more...

HP Seeks to Boost Cloud Service Automation Capability

Aug 27 | Although it was lost in the chaos of the 3PAR bidding war between, HP announced news that it acquired cloud service automation firm Stratavia to bolster its cloud management offering and further its strategy in the arena. Read more...

NASA CIO Weighs in on Nebula, Public Clouds

Aug 26 | In an interview from the NASA IT Summit last week, the agency's CIO, Linda Cureton weighs in on developments with Nebula platform and the adoption of the open source code by other agencies looking to the cloud. Read more...

Facing the Realities of Private Cloud Building

Aug 24 | While private clouds are getting far more attention than they received at the beginning of the cloud buzz boom, the realities of the complexities of actual building them--not to mention the financial and time investments--are often overlooked. Read more...

Read more headlines...

Featured Whitepapers

Increase IT Performance from the Enterprise to the Cloud with WAN Optimization

Aug 30 | | Enterprises face a paradox today: while workers become increasingly distributed, IT infrastructure is rapidly consolidating. Virtualization has made it possible to create consolidated, elastic pools.

Grid Computing in the Cloud White Paper

May 14 | | Empower business users, scientists and researchers with their own grid computing infrastructure in the cloud.

View the White Paper Library

Multimedia

Webcast: Storm Clouds - Security Issues in the Cloud and How to Address Them

This Webinar will highlight the four critical areas of concern when securing cloud infrastructure services and managed enterprise applications.

Webcast: Confronting HPC Cloud Computing Security Concerns

Escalating energy and operational costs of building and maintaining data centers are forcing enterprises to adopt cloud computing models. But are Infrastructure as a Service (IaaS) solutions like IBM's Computing on Demand (CoD) really cost effective? Join the discussion as industry experts discuss how you can exploit cloud computing for maximum ROI.