Partly Cloudy

Partly Cloudy 2019

Join us for a one-day conference discussing the challenges of building and supporting Hybrid Clouds powering scientific research. 

We are IT infrastructure professionals managing Compute / Network / Storage in support of research activities; this is a peer-led and peer-funded event — no vendor support, no vendor marketing. Come spend a day in Seattle sharing your experiences, posing the hard questions, and kicking around current solutions.

Cloud adoption in the research space lags the commercial space: Why? What makes research difficult to port to the Cloud? Where have we been successful? Where have we struggled? What lessons can we learn from each other about what works and what doesn’t?  What lessons from our experience can the Cloud providers glean in order to acquire not only more of our business but also to acquire a competitive edge in expanding their offerings to their commercial customers?

Meeting Deliverable

The Partly Cloudy steering committee intends to produce a white paper, post-conference, aimed at Cloud providers, encapsulating how we answer these questions. Help us build the content of this white paper.

Date:
Thursday-Friday, October 17-18, 2019
Location:
Fred Hutch
1100 Fairview Ave N, Seattle WA 98109

Agenda

 

Thursday, October 17

6 – 9 p.m.            Welcome Dinner & Networking


Friday, October 18

8:30 – 9:00           Welcome & Opening Remarks
                                Bob Robbins, Scientist & Educator
9:00 – 9:30           Workflow Managers in a Cloudy World
                                Sam Minot, Staff Scientist, Fred Hutch
9:30 – 10:00         Cloud Billing and Cost Controls for Research at Scale 
                                Lukas Karlson, Cloud Architect, Broad Institute
10:00 – 10:45       Break
10:45 – 11:15       Chargebacks, Cost Control, and Serverless in Pharma
                                Lance Smith, IT Director, Celgene
11:15 – 11:45       Community Development
                                 Kate Hertweck, Bioinformatics Training Director, Fred Hutch  
11:45 – 12:15       Bursts, Forecasting, and Storm Avoidance: Utilization of Cloud Platforms 
                                 Shane Sanders + Sandeep Namburi, CyberInfrastructure Manager + Cloud                                         Analyst, Jackson Labs
12:15 – 1:15          Lunch
1:15 – 2:15            Breakout sessions 
2:15 – 2:30            Break
2:30 – 4:00            Plenary Session

  • What lessons will you take home?
  • What changes to Cloud services would increase our adoption?

Presentation Descriptions


Workflow Managers in a Cloudy World

How scientists are using the Public Cloud without really trying – opportunities / challenges of workflow managers, technical requirements for implementation, progress to date, and next steps. 


Cloud Billing and Cost Controls for Research at Scale

How the Broad Institute manages billing and cost controls for a wide variety of research projects with different funding sources across multiple public clouds


Chargebacks & Serverless in Pharma

Chargebacks and cost control across numerous budgets lay the foundation for R&D using the Cloud; on the technical side, deploying and favoring serverless over containers has enabled greater agility leading to increased Cloud adoption within the private sector. 


Community Development

The nature of academic research presents specific challenges when adopting or supporting new technologies. Initiating a conversion to cloud computing represents one of the biggest stumbling blocks for academic researchers who have primarily used local computing resources throughout their careers. What are the conceptual and technical skills necessary for successful adoption of cloud computing? How can we manage expectations related to usage of cloud-based resources? How do you assess what your community needs to implement cloud-based options, and then respond with relevant resources? In this talk, I’ll discuss my experiences developing computational training for a large, heterogeneous community of researchers, including common impediments to both technological and cultural change, and suggest ways to facilitate community building as a way to alleviate these challenges.


Bursts, Forecasting, and Storm Avoidance: Utilization of Cloud Platforms

Current state of our use of various cloud platforms in support of research, education, and controlled access data projects, with a particular glance at how we use both on-prem and cloud resources to accommodate seasonal bursts in HPC demand.

About Partly Cloudy

Scientific research faces challenges when it looks to the Cloud. Yes, elasticity is attractive and scaling beyond what we can host on-prem is tantalizing. However, Cloud economics dictate that increased resource consumption incurs increased costs, which suits the retail world neatly: more people buying your widgets leads to more revenue which pays the growing Cloud bill. But consumption of compute / network / storage in research tends to be poorly correlated with a revenue stream; this mismatch acts as an early barrier to leveraging the Cloud to support research activities.

Retooling our expertise to move from traditional programming techniques to Cloud-native paradigms (e.g. POSIX file semantics to object storage manipulation) requires retraining both IT and scientific staff. The Cloud comes with security defaults which are hard to replicate on-prem: how will we evolve our security models, handle compliance, and communicate risk in the Hybrid world? Capping costs and charge-backs schemes require evolution. And of course, the heterogeneity of the Hybrid Cloud user base requires additional flexibility: what are the challenges here?

Researchers vary in their financial resources — some can afford to trade $$ for time-to-completion, bursting into the Cloud to reduce wall-time; others are conserving their $ and want to live within the constraints of the on-prem systems. Some researchers rely on data sets already stored in the Cloud; others produce petabytes of images using custom instruments housed at their institution — these profiles experience different cost / performance trade-offs when comparing on-prem and Cloud hosted resources.

For IT infrastructure staff, Hybrid Cloud pushes us to support dual-stacks for the foreseeable future, and pushes us to understand our costs more clearly, both on-prem and sky-based, so that we can more effectively trade-off where to host which effort.

Unlike startups or retailers, IT infrastructure in a research environment tends to support multiple and sometimes conflicting goals: stability for customer A, agility & speed for customer B, regulatory compliance for customer C. Also, we want to anticipate rapid changes in requirements: “Oh, I forgot to mention: now I need this site to be protected and users need to login”.

Looking to the future, we predict a Partly Cloudy sky from zenith to horizon — i.e. we see the Hybrid Cloud as the research outfit’s default state.  This immediately raises the questions: when to host in the Cloud, when to host on-prem, and how to migrate workloads between the two. Will on-premise become more cloudy now that platforms such as Kubernetes provide standard interfaces to all three Hyperscalers? (AWS, Azure & Google). How will we help our researchers meet the developing standards for reproducible software stacks?

Speakers are drawn entirely from our peers: infrastructure engineers, product managers, and management supporting research, offering their perspectives on these challenges. We structure the conference as follows: Welcome dinner & networking Thursday evening. On Friday, speakers in the morning, small-group break-out after lunch, finishing with a plenary session in which we synthesize what we have learned. 

Last Modified, September 10, 2019