Cloud, HPC And Open Technologies Converge To Fuel Research, Innovation
By Shel Waggener and Steve Wolff
With a mission that is focused on scientific discovery and advancement, universities and research institutes have long been at the forefront of technology innovation, embracing open standards and ease of collaboration. The academy has always understood that when hundreds of researchers contribute to a shared purpose and work together to solve problems in open and transparent ways, everyone benefits. As the pace of innovation accelerates in an open standards development environment, the diversity of potential solutions increases. Over time, the combination of scholarly scrutiny and testing against scientific applications helps to surface the best solutions without otherwise limiting the choices available to researchers.
The rapid growth and adoption of cloud technologies presents new opportunities and challenges for both researchers and service providers. The massive scale of cloud-based solutions—that must accommodate the crossing of borders, legal jurisdictions, and changing data management practices – requires that the academy engage in a conversation that brings our open community together with providers. This ensures that the full benefits of the cloud can be realized.
Recognizing the potential role of cloud-based open computing technologies for the research community, a group of 30 stakeholders and decision-makers from 20 leading academic institutions and industry partners convened on August 22. They shared their views on how open computing solutions can best support existing and emerging use cases across a range of research disciplines and high performance workloads. The event was hosted by Argonne National Labs and jointly sponsored by Notre Dame, Internet2 and Rackspace.
The participants discussed a variety of use cases and experiences with big data and high-performance computing and the ways in which these efforts have introduced new challenges and new frustrations. Say you’re an academic or researcher who needs time on a supercomputer; traditional queuing models oftentimes result in having to wait for months to get approved, and even then, you may only get a limited time window. So, if something with your software is not working when it is your turn, you’re out of luck.
Through the Internet2 NET+ initiative, the research and education community – consumers and service providers alike – are coming together to identify and develop solutions for some of the most complex computing challenges facing the community. It is exciting to see Internet2 and member campuses, MIT, Notre Dame and University of Texas-San Antonio, working together with research institutions such as Argonne, Fermilab and CERN, and providers of open cloud computing solutions such as Rackspace and others using the high-performance Internet2 Network to open up new areas for research and high-performance computing. Through such efforts, the community is able to collectively support institutional research and teaching missions, including not only scientific, research computing, but also university operations. Throughout this process, they are ensuring that their security, compliance and accessibility obligations are met.
The Internet2 NET+ initiative is designed by the research university community to engage in the development of solutions that allow for consistent and community supported approaches. Working with commercial providers helps to remove barriers to the research and education community to leverage the cloud through joint design of provisioning, and federated identity solutions and services. It also provides a context for addressing some of the real problems posed by the utilization of cloud based services to support research. For researchers, this means greater access to affordable services and a sustainable environment for scientific and scholarly exploration. The cloud changes the computing equation and redefines the user experience with services by adding on-demand, utility and self-service capabilities to computing infrastructure.
The cloud is quickly evolving into one of the most flexible models for scientific computation and big data. Cloud adoption will help accelerate the changes in high-performance computing. We’ve seen the change developing over the past couple of years, as open technologies like the Open Compute Project and OpenStack in particular, democratize access to mass commodity hardware and software. Now, top research institutes such as CERN, Argonne National Laboratory, Notre Dame, University of Texas at San Antonio and MIT have chosen to build their high-performance clouds on OpenStack. By embracing open standards and collaboration, university researchers are at the forefront of innovation and contribute to a shared purpose that benefits everyone.
The event organizers – Narayan Desai (Argonne National Lab), Paul Brenner (Notre Dame University), Khalil Yazdi (Internet2) and Paul Rad (Rackspace) – welcomed participants (pictured below) and laid out a vision and an agenda for the workshop. Some of the key points that emerged during discussions in the morning include:
- Public clouds were not designed to address the requirements of research communities.
- Public clouds appear inexpensive until you factor in the costs of networking and data transfers. Consequently, technical computing often requires hybrid and purpose built community clouds.
- Provisioning a private cloud for average loads and bursting for peak loads appears to provide the most economic model, but access to data sets and the proximity of compute resources can be critical for big data workflows, e.g. in high-energy physics.
- By working together as a community, we can develop scaled-up solutions more appropriately. For example, a community-owned open cloud might include a number of federated universities and research labs.
- Within five to 10 years, large supercomputer systems will converge with cloud provider systems – so continuing and organized effort will likely lead to better outcomes in the long term.
In the afternoon sessions, as part of a “Cloud Best Practices” discussion, Notre Dame’s Paul Brenner, Argonne’s Narayan Desai and UT-San Antonio’s Raj Boppana delivered in-depth presentations on a wide range of research and university OpenStack use cases for High Performance Cloud.
- Case Study No. 1: Lessons Learned Running a Technical Cloud – Narayan Desai
- Case Study No. 2: Bridging campus, lab, and commercial research infrastructure with an open cloud for high energy physics – Paul Brenner
- Case Study No. 3: OpenStack-based High Performance Cloud – Rajendra Boppana
They shared lessons learned and presented the findings and gaps that point the way forward for compute- and data-intensive applications – and stimulated an animated and thought-provoking discussion among the participants around both the challenges identified and potential next steps for the workshop series.
Workshops like the one held at the Argonne National Lab play an important role in bringing together stakeholders to explore technology trends and to cooperatively accelerate their development. Internet2 is excited to be involved in this cutting edge work.
Listen to the following video as workshop participants discuss the promise of cloud computing and the power of the research and education community coming together to harness cloud technologies to the benefit of researchers.
At the end of the session, the community identified two immediate incubation projects, with several other possible projects noted:
- Big-Data Reachback for Cloud Bursting Scientific Applications such as high energy physics led by Notre Dame, Internet2, Rackspace, UT-San Antonio, MIT, and Cycle Computing
- Big-Data Scale out storage architecture led by Argonne, University of Chicago and Nimbus Services
The teams are planning to develop blueprints and detailed service descriptions, draft plans for a continuing collaborative effort and will identify regular communication channels for these projects so that all workshop participants and other interested researchers and service providers are kept informed of progress and emerging developments. They will likely get back together at SC13 and WCSC 2013 in San Antonio Texas.
If you are interested in learning more about this initiative and/or in participating in future workshop sessions, please email us at: OpenCloud@internet2.edu.