October 25, 2011
Public-private collaborative demo leveraged new ESnet Advanced Networking Initiative (ANI) testbed
SOUTH SAN FRANCISCO, BERKELEY AND FREMONT, Calif., Oct. 25 — Researchers at the Department of Energy's Energy Sciences Network (ESnet), Orange Silicon Valley and Bay Microsystems have collaborated to showcase the world's first 40 Gbps (gigabits per second) wide area network demonstration of RDMA over InfiniBand. RDMA, or Remote Direct Memory Access, is an alternative model to TCP/IP for bulk data transfer designed to significantly improve the utilization of available bandwidth resources and the processing efficiency of the end hosts – and does so without the need for any special network tuning.
While RDMA over InfiniBand is not a new concept, this experiment marks the first time that it has been demonstrated at 40 Gbps data rates over long-haul distances, in this case approximately 200 miles. The demonstration showed that using RDMA moves data at up to 96 percent of the peak capacity of the network as opposed to other network technologies where utilization can drop significantly over longer distances.
"The volume of science data that needs to be shared among global research sites is reaching levels beyond what current 10 Gbps networks and traditional network protocols can handle," said Brian Tierney, ESnet Staff Scientist and lead for the Advanced Networking Initiative (ANI). "RDMA communication opens new possibilities for researchers by enabling quick and efficient data transfer for 'big data' applications. It does so by eliminating many of the typical network performance bottlenecks and helps users make much more efficient use of the WAN infrastructure – this helps to ensure that scientific progress is unimpeded even as data needs grow."
The long-haul RDMA demonstration is part of a research project being conducted at Brookhaven National Laboratory (BNL) in New York and leverages ESnet's ANI 100 Gbps testbed. The testbed is being made available to researchers both in the public and private sector as a first of its kind platform to test different and potentially disruptive approaches to new network protocols and architectures in a greater than 10 Gbps network environment. The testbed is currently available in the Long Island, New York region but will soon have a national footprint to enable experimentation at even greater distances.
"While RDMA over extended distances has been tested and shown to scale up to 10Gbps data rates, this is the first time RDMA over distance has been proven to work at full bandwidth for 40Gbps data rates" said Eric Dube, Senior ProductManager of Systems at Bay Microsystems, Inc. "Gaining access to a 40 Gbps wide area optical circuit is very costly and had prohibited this kind of research in the past. Using the ANI testbed, we are now able to prove these concepts in a live network environment setting the stage for deploying scalable RDMA-enabled applications over 100G networks. This is especially important as more geographically dispersed datacenters and science sites will require this type of bandwidth and capability.
RDMA works by enabling network adapters to transfer data to or from application memory, eliminating the overhead of copying data between application memory and the data buffers in the operating system. When an application performs an RDMA operation, the application data is delivered directly to the network eliminating the additional latency the transfer would incur if the data were to be copied into the operating system memory buffers. This dramatically increases the application-messaging rate and increases the utilization of available network bandwidth.
Bay Microsystems provided its IBEx G40 40 Gbps InfiniBand WAN acceleration platform as well as provided input on the network design, bring-up, and overall support during the testing effort. Bay specializes in providing secure, high-performance networking solutions to commercial enterprises, government agencies and service providers. In particular, Bay delivers a portfolio of wide area network acceleration solutions to datacenters to maximize the utilization of their computation, storage, and interconnect assets, and increase their security and reliability.
"Applications that combine RDMA with InfiniBand achieve additional benefits including reliable data transfer, sustained high bandwidth utilization, and reduced CPU overhead, all critical items to consider when ‘Big Data' Cloud computing, high frequency trading, and high performance computing applications are deployed across a WAN infrastructure," Dube said.
Orange is an international operator for mobile and broadband Internet services and, under the brand Orange Business Services, is one of the world leaders in providing telecommunication services to multinational companies. Orange Silicon Valley participated in the project to explore RDMA's potential for meeting the growing challenge of the company and its enterprise customers. Orange Silicon Valley is a U.S. subsidiary of France Telecom-Orange.
Soumik Sinharoy, project manager at Orange Silicon Valley, said "Orange believes that the high bandwidth efficiency that comes with RDMA would be beneficial considering the significant increase in traffic anticipated across our enterprise and consumer markets. Orange is also using this opportunity to evaluate how increased efficiency could help us increase the utilization of our existing assets."
By using the IBEx G40 40 Gbps InfiniBand WAN acceleration platform and Infinera's DTN switched DWDM system along with high-performance InfiniBand-connected hosts provided by BNL, researchers were able to extend 4X InfiniBand QDR outside the datacenter and transport it on a 40 Gbps wavelength over the 370.36 km fiber loop of ESnet's wide area network circuit from BNL in Upton, NY to Manhattan, NY. The IBEx G40, provides enhanced buffering and credit management allowing all native InfiniBand protocols to be extended over SONET OC-768/SDH STM-256 or ITU-T G.709 OTU3 at 40 Gbps data dates over thousands of miles.
The proof is in the results where application testing produced near theoretical line rate performance across the WAN circuit with RDMA-enabled application communication providing 96 percent link efficiency. These improvements can be attributed to the lossless nature of InfiniBand allowing for reliable transport when used as a wide area network technology and the reduced protocol overhead capabilities of RDMA.
Further demonstrations will be presented at the SC11 conference being held Nov. 12-18 in Seattle. Researchers will build a 40 Gbps transcontinental network circuit enabling RDMA application data transfers over InfiniBand at near line-rate performance.
France Telecom-Orange is one of the world's leading telecommunications operators with 170,000 employees worldwide, including 102,000 employees in France, and sales of 22.6 billion euros in the first semester 2011. Present in 35 countries, the Group had a customer base of 217.3 million customers at 30 June 2011, including 143 million customers under the Orange brand, the Group's single brand for internet, television and mobile services in the majority of countries where the company operates. At 30 June 2011, the Group had 158.3 million mobile customers and 14 million broadband internet (ADSL, fibre) customers worldwide. Orange is one of the main European operators for mobile and broadband internet services and, under the brand Orange Business Services, is one of the world leaders in providing telecommunication services to multinational companies.
With its industrial project, "conquests 2015", Orange is simultaneously addressing its employees, customers and shareholders, as well as the society in which the company operates, through a concrete set of action plans. These commitments are expressed through a new vision of human resources for employees; through the deployment of a network infrastructure upon which the Group will build its future growth; through the Group's ambition to offer a superior customer experience thanks in particular to improved quality of service; and through the acceleration of international development
France Telecom (NYSE:FTE) is listed on Euronext Paris (compartment A) and on the New York Stock Exchange. For more information (on the internet and on your mobile): www.orange.com, www.orange-business.com, www.orange-innovation.tv. (Orange and any other Orange product or service names included in this material are trademarks of Orange Brand Services Limited, Orange France or France Telecom.)
About Bay Microsystems, Inc.
Bay Microsystems was founded in 2000 to deliver secure high performance networking solutions to commercial enterprises, government agencies, and service providers. Bay delivers a portfolio of wide area network acceleration solutions that transform how global datacenters are architected by substantially increasing the performance of the applications they support, maximizing the utilization of their computation, storage, and interconnect assets, and increasing their security and reliability.
ESnet provides the high-bandwidth, reliable connections that link scientists at national laboratories, universities and other research institutions, enabling them to work together on some of the world's most important scientific challenges including energy, climate science, and the origins of the universe. Funded by the U.S. Department of Energy's Office of Science, and managed and operated by the ESnet team at Lawrence Berkeley National Laboratory (Berkeley Lab), ESnet provides scientists with access to unique DOE research facilities and computing resources, as well as to scientific collaborators including research and education networks around the world.
Researchers from the Suddhananda Engineering and Research Centre in Bhubaneswar, India developed a job scheduling system, which they call Service Level Agreement (SLA) scheduling, that is meant to achieve acceptable methods of resource provisioning similar to that of potential in-house systems. They combined that with an on-demand resource provisioner to ensure utilization optimization of virtual machines.
Experimental scientific HPC applications are continually being moved to the cloud, as covered here in several capacities over the last couple of weeks. Included in that rundown, Co-founder and CEO of CloudSigma Robert Jenkins penned an article for HPC in the Cloud where he discussed the emergence of cloud technologies to supplement research capabilities of big scientific initiatives like CERN and ESA (the European Space Agency)...
When considering moving excess or experimental HPC applications to a cloud environment, there will always be obstacles. Were that not the case, the cost effectiveness of cloud-based HPC would rule the high performance landscape. Jonathan Stewart Ward and Adam Barker of the University of St. Andrews produced an intriguing report on the state of cloud computing, paying a significant amount of attention to the problems facing cloud computing.
Jun 19, 2013 |
Ruan Pethiyagoda, Cameron Boehmer, John S. Dvorak, and Tim Sze, trained at San Francisco’s Hack Reactor, an institute designed for intense fast paced learning of programming, put together a program based on the N-Queens algorithm designed by the University of Cambridge’s Martin Richards, and modified it to run in parallel across multiple machines.
Jun 17, 2013 |
With that in mind, Datapipe hopes to establish themselves as a green-savvy HPC cloud provider with their recently announced Stratosphere platform. Datapipe markets Stratosphere as a green HPC cloud service and in doing so partnering with Verne Global and their Icelandic datacenter, which is known for its propensity in green computing.
Jun 12, 2013 |
Cloud computing is gaining ground in utilization by mid-sized institutions who are looking to expand their experimental high performance computing resources. As such, IBM released what they call Redbooks, in part to assist institutions’ movement of high performance computing applications to the cloud.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/02/2012 | AMD | Developers today are just beginning to explore the potential of heterogeneous computing, but the potential for this new paradigm is huge. This brief article reviews how the technology might impact a range of application development areas, including client experiences and cloud-based data management. As platforms like OpenCL continue to evolve, the benefits of heterogeneous computing will become even more accessible. Use this quick article to jump-start your own thinking on heterogeneous computing.