Scientists in general, and astronomers in particular, have been at the forefront when it comes to dealing with large amounts of data. These days, the “Big Data” community, as it is known, includes almost every scientific endeavor — and even you.
In fact, Big Data is not just about extremely large collections of information hidden in databases inside archives like the Barbara A. Mikulski Archive for Space Telescopes. Big Data includes the hidden data you carry with you all the time in now-ubiquitous smart phones: calendars, photographs, SMS messages, usage information and records of our current and past locations. As we live our lives, we leave behind us a “data exhaust” that tells something about ourselves.
Does the universe contain some hidden data, data that is there in plain sight but has yet to be investigated? If so, what’s in the cosmos’ data exhaust?
In late 1995, the Hubble Space Telescope took hundreds of exposures of a seemingly empty patch of sky near the constellation of Ursa Major (the Big Dipper). The Hubble Deep Field (HDF), as it is known, uncovered a mystifying collection of about 3,000 galaxies at various stages of their evolution. Most of the galaxies were faint, and from them we began to learn a story about our Universe that had not been told before.
At the time, I was a young graduate student at the Ohio State University. I still remember very vividly how mesmerized I was by what our universe was telling us with just one image. I remember calculating the approximate total number of galaxies in the visible universe, assuming that the HDF was a representative patch of our universe: 100 billion. I ended up using that image for my first paper as a graduate student, looking for distant quasars in the HDF.
The HDF represented a tremendous achievement for science in ways that are still reverberating today. It initiated one of the many legacies of the Hubble Telescope: deep images showing infant galaxies in the early universe.
However, the HDF was also instrumental for a generation of young astronomers in another significant way. For the first time, an observation was deemed so important in order to address basic questions about the structure and evolution of the universe that it needed to be made available immediately to the astronomical community around the world. This, as well as the underlying science, was a game changer. Typically observations are released to the astronomical community after a proprietary period — typically 6 months or a year. This gives the astronomers who requested the observation time to perform their investigations. In this case, the HDF team took the unusual step of both swiftly preparing the observations for scientific study and releasing them without delay, thus allowing students and researchers alike to dive immediately into the science of the observation. The success of this decision paved the way for future observations to be released with similar speed.
So was the HDF unique? Were we just lucky to observe a crowded but faint patch of sky? To address this question, and determine if indeed the HDF was a “lucky shot,” in 2004 Hubble took a million-second-long exposure in a similarly “empty” patch of sky: The Hubble Ultra Deep Field (HUDF). The result was even more breathtaking. Containing an estimated 10,000 galaxies, the HUDF revealed glimpses of the first galaxies as they emerge from the so-called “dark ages” — the time shortly after the Big Bang when the first stars reheated the cold, dark universe. As with the HDF, the HUDF data was made immediately available to the community, and has spawned hundreds of publications and several follow-up observations.
Many more examples exist of “deep fields,” and in all cases it seems that if we look closely at an unobserved portion of our universe we discover more and more of its “data exhaust,” pointing us to the signatures of its origin.
The Hubble Deep Field started a revolution in the way we look at our universe but also in the way we access information. This trend exists to this day. Just a few days ago the European Southern Observatory (ESO) released the widest deep view of the sky ever made using infrared light. Once again an unremarkable patch of sky comes to life and reveals more than 200,000 galaxies!
This trend is not about to end. Over the next decade, astronomy will undergo dramatic changes. Missions like the Panoramic Survey Telescope and Rapid Response System (PanSTARRS) and the Large Synoptic Survey Telescope (LSST) will be able to survey the whole sky in just a few days, creating a 3D map of the universe. I personally cannot wait to see what we will find!