April 12, 2010
When Tom Tabor invited me to join HPC in the Cloud I saw, on the one hand, a great opportunity to contribute my undying optimism about clouds becoming mainstream (one day) and providing on-demand, scalable computing for everyone via the Internet. On the other hand, this is a great opportunity to discuss the many roadblocks our community still has to remove before clouds can become mainstream.
Then I remembered our community’s many years of effort in working hard to make grids mainstream. Looking back, I see that the roadblocks we faced for grids were indeed very similar to the ones we face with clouds today. To be honest, sometimes I think that we barely made progress towards our grid mainstream dream, at least in terms of resolving issues like security, user-friendliness, the wider acceptance of the grid paradigm, the disparate methods handling different administrative domains, releasing control, achieving reliability, and coming up with the right (set of) standards for interoperability. Sure, if we look closer, the research community has resolved many of these issues to the point that we now see research grids today in production around the world (i.e. DEISA, the Distributed European Infrastructure for Supercomputing Applications, with its UNICORE middleware, and EGEE, Grid5000, TeraGrid and its successor eXtreme Digital (XD), and Naregi, and hundreds of application specific Grids like NEESgrid, BIRN, GEONgrid, LEAD, nanoHUB, ServoGrid, GeneGrid, and MyGrid). Still, it is worth noting that most of these are special purpose research grids.
To make a long story short, for HPC in the Cloud, my primary task does not lie in simply extolling the many benefits of clouds (which are quite obvious to many already), rather I see my main task to be examining the nature of the many roadblocks to mainstream cloud adoption. The best place to start is to search for success stories from you, our fellow community members, that demonstrate how you are clearing these hurdles. A word of caution, however--if you think this task of mine (and of yours) is an easy one, just think of the following issues:
- The process of retrieving data from one cloud and move them into another cloud, and back to your desktop system, in a reliable and secure way.
- The fulfilment of (e.g. government) requirements for security, privacy, data protection, and the archiving risks associated with the cloud.
- The compliance with existing legal and regulatory frameworks and current policies (established far before the digital age) that impose antiquated (and sometimes even conflicting) rules about how to correctly deal with information and knowledge.
- The process of setting up a service level agreement.
- Migrating your applications from their existing environments into the cloud.
And for that matter…
- Do we all agree on the same security requirements; do we need a checklist, or do we need a federated security framework?
- Do our existing identity, access management, audit and monitoring strategies still hold for the clouds?
- What cloud deployment model would you have to choose: private, public, hybrid, or federated cloud?
- How much does the virtualization layer of the cloud affect application performance (i.e. trade-off between abstraction versus control)?
- How will clouds affect performance of high-throughput versus high-performance computing applications?
- What type of application needs what execution model to provide useful abstractions in the cloud, such as for data partitioning, data streaming, and parameter sweep algorithms?
- How do we handle large scientific workflows for complex applications that may be deployed as a set of virtual machines, virtual storage and virtual networks to support different functional components?
- What are common best practices and standards needed to achieve portability and interoperability for cloud applications and environments ?
- And last but not least, how can (and will) organizations like DMTF and OGF help us with our cloud standardization requirements?
So now you can see that all these roadblocks (and the process of barrier removal) will keep us very busy over the next couple of years. The end result of all of this hard work? The realization of the mainstream dream for clouds.
Wolfgang Gentzsch is Advisor to the EU project DEISA, the Distributed European Infrastructure for Supercomputing Applications, a member of the Board of Directors of the Open Grid Forum, and a contributing editor to HPC in the Cloud. Read more thoughts from Wolfgang Gentzsch on the topic of the grid and clouds here.
Posted by Wolfgang Gentzsch - April 11, 2010 @ 9:46 PM, Pacific Daylight Time
Independent HPC consultant for cluster, grid, and cloud computing, and for data and compute-intensive applications, and General Chair of the ISC Cloud Conference.
No Recent Blog Comments
The ever-growing complexity of scientific and engineering problems continues to pose new computational challenges. Thus, we present a novel federation model that enables end-users with the ability to aggregate heterogeneous resource scale problems. The feasibility of this federation model has been proven, in the context of the UberCloud HPC Experiment, by gathering the most comprehensive information to date on the effects of pillars on microfluid channel flow.
Large-scale, worldwide scientific initiatives rely on some cloud-based system to both coordinate efforts and manage computational efforts at peak times that cannot be contained within the combined in-house HPC resources. Last week at Google I/O, Brookhaven National Lab’s Sergey Panitkin discussed the role of the Google Compute Engine in providing computational support to ATLAS, a detector of high-energy particles at the Large Hadron Collider (LHC).
Frank Ding, engineering analysis & technical computing manager at Simpson Strong-Tie, discussed the advantages of utilizing the cloud for occasional scientific computing, identified the obstacles to doing so, and proposed workarounds to some of those obstacles.
May 23, 2013 |
The study of climate change is one of those scientific problems where it is almost essential to model the entire Earth to attain accurate results and make worthwhile predictions. In an attempt to make climate science more accessible to smaller research facilities, NASA introduced what they call ‘Climate in a Box,’ a system they note acts as a desktop supercomputer.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/02/2012 | AMD | Developers today are just beginning to explore the potential of heterogeneous computing, but the potential for this new paradigm is huge. This brief article reviews how the technology might impact a range of application development areas, including client experiences and cloud-based data management. As platforms like OpenCL continue to evolve, the benefits of heterogeneous computing will become even more accessible. Use this quick article to jump-start your own thinking on heterogeneous computing.