August 09, 2012
Cloud technologies have become integral to a number of services including video streaming, file sharing and social media to name a few. But when it comes to HPC applications, benefits shared by the previous examples don't always translate. This leaves some HPC users preferring private clusters sans virtualization, or straight-up supercomputers. Yet there have been quite a few high-profile accounts of users running HPC workloads on public clouds like Amazon Web Services and Microsoft Azure. Does this mean it's a good idea to run HPC applications in the cloud? Cluster Monkey's Thomas Eadline addresses this question, and points out which HPC applications are best suited for today's multi-purpose clouds.
In the HPC space, users prefer the following features:
While cloud systems are based on a different set of characteristics:
Eadline notes that where the two spaces overlap is in the need for scalability and redundancy though hardware independence, but otherwise these are very distinct worlds. HPC users have limited choice regarding what hardware they can use, and if they decide to use a service like Amazon, their resources will be virtualized. Furthermore, applications that lean heavily on a system's interconnect and call for high levels of I/O would suffer serious performance degradation.
Despite these differences, some HPC applications are good candidates for cloud. Also known as embarrassingly parallel (EP), these programs do not require high interconnect performance, and can run on 10 Gigabit Ethernet or even Gigabit Ethernet. Of these EP applications, there is a subset of programs that don't require especially high I/O rates. This group is typically well-suited for traditional cloud implementations.
Earlier this year, Cycle Computing ran one such application on Amazon's EC2 infrastructure. The program, called Glide, was used to test various potential cancer drugs for pharmaceutical firm Schrödinger and their research partner Nimbus Discovery. A 50,000-core virtual cluster named "Naga" was spun up for three hours and used the application to test 21 million drug compounds. The project processed over twelve-and-a-half years of compute work in just three hours. Total cost? $4,900. By comparison, had the end users built their own facility and purchased a comparable supercomputer, it would have cost between $20-30 million.
While Cycle's example demonstrates the extreme savings an HPC user can experience with cloud services, not all applications can be handled in the same fashion on a service like Amazon, but AWS is not the only cloud in town.
Some infrastructure providers like Penguin Computing, Softlayer and Zunicore focus on creating HPC-friendly cloud environments. These companies offer bare metal services, which allow end users to access resources without virtualization layers. Although these features are attractive for HPC applications, users need to analyze their requirements carefully before migrating to the cloud.
Frank Ding, engineering analysis & technical computing manager at Simpson Strong-Tie, discussed the advantages of utilizing the cloud for occasional scientific computing, identified the obstacles to doing so, and proposed workarounds to some of those obstacles.
The private industry least likely to adopt public cloud services for data storage are financial institutions. Holding the most sensitive and heavily-regulated of data types, personal financial information, banks and similar institutions are mostly moving towards private cloud services – and doing so at great cost.
In this week's hand-picked assortment, researchers explore the path to more energy-efficient cloud datacenters, investigate new frameworks and runtime environments that are compatible with Windows Azure, and design a uniﬁed programming model for diverse data-intensive cloud computing paradigms.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/02/2012 | AMD | Developers today are just beginning to explore the potential of heterogeneous computing, but the potential for this new paradigm is huge. This brief article reviews how the technology might impact a range of application development areas, including client experiences and cloud-based data management. As platforms like OpenCL continue to evolve, the benefits of heterogeneous computing will become even more accessible. Use this quick article to jump-start your own thinking on heterogeneous computing.