HPC in the Cloud: IU Team Sets Data Transfer Record at SC11

Dedicated to covering high-end cloud computing
in science, industry and the datacenter

Translation Disclaimer

Subscribe | Sign In

November 22, 2011

IU Team Sets Data Transfer Record at SC11

Tiffany Trader

Let's face it, the success of any cloud solution is only as strong as the underlying network; this is especially true when it comes to running HPC workloads. The communications network becomes even more important if we're to make make significant inroads with the running of data-intensive workloads. For that to happen, we're going to need bigger pipes and more robust protocols and standards. For the bleeding-edge in networking, what better place to turn than the annual supercomputing conference?

Scinet booth, SC11 On the heels of SC11, which seemingly packed the entire HPC community into the Seattle Convention Center last week, Indiana University is announcing the results of its Scinet Research Sandbox entry, called "The Data Superconductor: An HPC cloud using data-intensive scientific applications, Lustre-WAN and OpenFlow over 100Gb Ethernet." As a key component of SCinet, the SRS program gives researchers with innovative network approaches a chance to test out their ideas in the unique environment of the SCinet networks. The 100 Gbps environment provided by SCinet, ESnet, and Internet2 is ten times faster than the current standard, and, of course, many thousands of times faster than public Internet speeds.

The IU researchers set out to address a major concern of data-intensive research, which is how to transfer massive amounts of data to supercomputing facilities for analysis. With collaborators from Brocade, Ciena, Data Direct Networks, IBM, Internet2, Whamcloud and ZIH, the IU team created two compute clusters with Lustre file systems, one in Indianapolis and and the other Seattle, connected by a 2,300 mile 100Gbps link. The series of demonstrations, performed during the conference, achieved a throughput of 96 Gbps for network benchmarks and 6.5 GB/s using IOR, a standard file system benchmark, while running a mix of eight real-world applications returned a result of 5.2 GB/s.

IU believes these results are record-worthy, saying that "this appears to be the fastest data transfer ever achieved with a 100Gbps network at a distance of thousands of miles."

Stephen Simms, manager of the High Performance File Systems group at Indiana University, examines the implications for researchers as we enter the age of big data:

"100 Gigabit per second networking combined with the capabilities of the Lustre file system could enable dramatic changes in data-intensive computing. Lustre's ability to support distributed applications, and the production availability of 100 gigabit networks connecting research universities in the US, will provide much needed and exciting new avenues to manage, analyze, and wrest knowledge from the digital data now being so rapidly produced."

Share Options

Subscribe

Subscribe to HPC in the Cloud

Discussion

There are 2 discussion items posted.

Unit issues
Submitted by malallen on Nov 23, 2011 @ 1:27 PM EST

The units of the IOR test results and application performance should be gigabytes per second, as opposed to gigabits per second.

Post #1

Good catch
Submitted by Tiffany on Nov 23, 2011 @ 1:57 PM EST

Thanks for catching the notation error; it's been corrected.

Post #2

Join the Discussion

Become a Registered User Today!

Registered Users Log in join the Discussion

RSS Feeds

Feeds by Topic

Feeds by Content Type

Subscribe to All Content

All

May 18, 2012

MicroTech Unveils Advanced Virtual Environment for Private Cloud Computing

May 17, 2012

May 16, 2012

May 15, 2012

May 14, 2012

May 11, 2012

AWS, SAP Announce Certification of AWS Infrastructure for SAP Business All-in-One Solutions

May 10, 2012

May 09, 2012

Feature Articles

Cloud Services Satisfy a Higher Calling

Higher education involves many collaborative projects that lend themselves to cloud services, however often those services are not tailored to the uniqueness of an academic environment. That's where the Internet2 NET+ project comes in. By partnering with 16 major cloud providers, the networking consortium is seeking to expedite the delivery of cloud services and by doing so advance research and innovation in the United States.
Read more...

Read more HPC in the Cloud features...

Appro White Paper: Enabling Performance-per-Watt Gains in HPC

04/05/2012 | Appro | Designed to meet the growing global demand for HPC solutions, Appro's Xtreme-X™ Supercomputer delivers superior performance-per-watt and reduced I/O latency while bringing significant flexibility to HPC workload configurations including capacity, hybrid, data intensive and capability computing.

Exploring the Potential of Heterogeneous Computing

04/02/2012 | AMD | Developers today are just beginning to explore the potential of heterogeneous computing, but the potential for this new paradigm is huge. This brief article reviews how the technology might impact a range of application development areas, including client experiences and cloud-based data management. As platforms like OpenCL continue to evolve, the benefits of heterogeneous computing will become even more accessible. Use this quick article to jump-start your own thinking on heterogeneous computing.