July 18, 2011
This week Microsoft unveiled Project Daytona, which is a community technical preview release of their own MapReduce framework designed and built for their own Windows Azure cloud offering.
Microsoft claims that their implementation of an interative MapReduce runtime on their cloud will allow researchers to capture the scalability of the cloud without sacrificing the performance and compute horsepower needed to analyze large data sets. In their statement today, Microsoft said that Project Daytona will allow researchers to “analyze data sets on gigabytes or terabytes of data and run large-scale machine learning algorithms on dozens or hundreds of compute cores.”
According to softies behind Project Daytona, one of the most persistent complaints they received from researchers was that they needed a data analysis and processing framework robust enough to handle the increasingly large data sets they were contending with. Like those in the enterprise space, these researchers were finding it difficult to find meaning in their massive pools of data using simplified tools. As many who are familiar with the MapReduce framework, however, using it to create parallelized code capable of uncovering hidden value in data isn’t light work—it took expertise that many domain specialists don’t possess.
Arguably, one of Microsoft’s strong suits has historically been their ability to take programmatic headaches, condense them behind a simple user interface and let the experts handle the drama. This is what they are hoping to do with Project Daytona on the software level while letting a bevy of instructional materials, sample code, and easy interfaces make the rest a bit more palatable for the average research group.
In terms of the key benefits, Microsoft says that their strong point with Project Daytona is that it was purpose-built for the cloud. For their cloud. The virtual machines, they say, “irrespective of infrastructure as a service (IaaS) or platform as a service (PaaS), introduce unique challenges and architectural tradeoffs for implementing a scale-out computation framework such as Project Daytona. Out of these, the most crucial are network communications between virtual machines (VMs) and the non-persistent disks of VMs. We have tuned the scheduling, network communications scheduling, and the fault tolerance logic of Project Daytona to suit this situation.”
Additionally, they point to the fact that this has been designed specifically with cloud storage services in mind via their defined stream-based data access layer for cloud data sources (which only include Windows Azure blob storage, although they claim more will be on the way at an undefined moment in the future). This means they will be able to partition data dynamically while supporting parallel reads, leaving the data inside the memory or in local non-persistent disks with backups relegated to the blobs. This means Project Daytona can gobble data without the overhead hit and overcome failures without a crisis, relying on their Azure storage services and without the need for a distributed file system.
If you shed the messy data movement issues, the scalability of the cloud for projects that are data-instensive can be quite a selling point. Instant provisioning makes the process seamless and users can define, shut down and otherwise manipulate virtual machines as needed. As Microsoft says in its closing message about the release, “Project Daytona lets you focus on your data exploration; without having to worry about acquiring compute capacity or time-consuming hardware setup and management.”
While Roger Barga and his team at the eXtreme Computing Group at Microsoft Research plan to fine-tune the offering in coming months, in many ways the release comes ready-made for new developers to climb on board without extensive parallel programming knowledge—always a spot of welcome news for scientific researchers who want to trade in their system admin skills for more time to get back to their focused work.
The project was borne out of the eXtreme Computing Group’s Cloud Research Engagement Program, which is a broad initiative that Microsoft hopes will “change the paradigm for scholarly and scientific research by extending the power of the computer into the cloud.” The group behind the effort is working on next generation cloud computing technologies in conjunction with the researchers who will be making the most use of such developments.
Other noteworthy projects that have spun out of their cloud program include the European research effort, VENUS-C Cloud Infrastructure, which is being tested and deployed on Azure with the promise of bringing new applications for bioinformatics, engineering, earth sciences and healthcare, among others, to a new class of users via the cloud.
Microsoft is blending what appears to be a dwindling presence in traditional HPC into its cloud offering with a host of announcements this year that cater to the HPC crowd yet provide the cloud as the backbone of the research. Even a quick glance down this list of initiatives that are research-oriented should give you an idea that Microsoft is serious about making its Azure the cloud of choice for technical and scientific computing.
On that note, the effort is in many ways geared toward appeasing the two key words that are the tip of every tech person’s tongue these days; big data and cloud computing. While Project Daytona is something of a latecomer to the big data cloud party given the Hadoop/MapReduce play Amazon made earlier this year, it is nonetheless unique in terms of its focus. Instead of appealing directly to the enterprise data mining professional, this community release offering is set apart by its emphasis on the researcher or scientist.
This is not to say that MapReduce on Microsoft’s cloud couldn’t aim to please the commercial data pro—it’s more a matter of messaging. Only time will tell what kind of adoption this will get once it gets off the ground and the bugs exterminated, of course, but it’s noteworthy in its target market focus.
Jun 19, 2013 |
Ruan Pethiyagoda, Cameron Boehmer, John S. Dvorak, and Tim Sze, trained at San Francisco’s Hack Reactor, an institute designed for intense fast paced learning of programming, put together a program based on the N-Queens algorithm designed by the University of Cambridge’s Martin Richards, and modified it to run in parallel across multiple machines.
Jun 17, 2013 |
With that in mind, Datapipe hopes to establish themselves as a green-savvy HPC cloud provider with their recently announced Stratosphere platform. Datapipe markets Stratosphere as a green HPC cloud service and in doing so partnering with Verne Global and their Icelandic datacenter, which is known for its propensity in green computing.
Jun 12, 2013 |
Cloud computing is gaining ground in utilization by mid-sized institutions who are looking to expand their experimental high performance computing resources. As such, IBM released what they call Redbooks, in part to assist institutions’ movement of high performance computing applications to the cloud.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/02/2012 | AMD | Developers today are just beginning to explore the potential of heterogeneous computing, but the potential for this new paradigm is huge. This brief article reviews how the technology might impact a range of application development areas, including client experiences and cloud-based data management. As platforms like OpenCL continue to evolve, the benefits of heterogeneous computing will become even more accessible. Use this quick article to jump-start your own thinking on heterogeneous computing.