An LSST.org project

Rubin’s Hybrid On Premises-Cloud Data Access Center

  • William O'Mullane
  • Yusra AlSayyad
  • James Chiang
  • Richard Dubois
  • Frossie Economou
  • Fabio Hernandez
  • Flora Huang
  • Tim Jenness
  • Kian-Tat Lim
  • Yee-Ting Li
  • Fritz Mueller
  • Dan Speck
  • Stephen Pietrowicz
  • Wei Yang

Abstract

Cloud computing offers unparalleled flexibility, a constantly increasing set of 'Infrastructure As A Service' capabilities, resource elasticity and security isolation. One of the most significant barriers in astronomy to wholesale adoption of cloud infrastructures is the cost for hot storage of large datasets - particularly for Rubin, a Big Data project sized at 0.5 Exabytes (500 Petabytes) over the duration of its 10-year mission. We are planning to reconcile this with a “hybrid” model where user-facing services are deployed on Google Cloud with the majority of data holdings residing in our on-premises Data Facility at SLAC. We discuss the opportunities, status, risks, and technical challenges of this approach.

Download