Description of reduced data products in the ESO Science archive
(Status: April 2024)
The ESO Phase 3 archive provides access to reduced and calibrated data products. All these data are stored in standard FITS format files and can be queried via the ESO Archive Science Portal, a graphical interface, or programmatically.
The data are either provided by external users such as the ESO Public Survey consortia and the ESO Large Programme teams, or produced by ESO using science grade reduction pipelines.
Here we give an overview of the types of data currently provided. For details see the ESO Science Data Product Standard and the release descriptions provided for each data collection available at the following URLs, from the community http://eso.org/rm/publicAccess#/dataReleases and the in-house processed products http://www.eso.org/sci/observing/phase3/data streams.html.
Calibrated images, flux maps and source lists
The Phase 3 archive started being populated with the VISTA and VST public imaging surveys products which consist of reduced, photometrically and astrometrically calibrated images. The VISTA/VIRCAM and VST/OmegaCAM instruments with their mosaic CCDs produce non-contiguous images of the sky in a single exposure. The gaps between the single detectors of OmegaCAM are much smaller than those of VIRCAM. Those non-contiguous images, called “pawprints”, are stored in multi-extension FITS files, where each extension corresponds to one of the 16 VIRCAM or 32 OmegaCAM images produced by the single detector chips.
The building blocks of the VISTA surveys are the “tile” images. The tile is a combined image of the field of view with no gaps (see Fig. 1,2). Tile images resulting from the coaddition of several individual images are termed as deep tiles (e.g. UltraVISTA, VIDEO). Tile images are provided as single-extension FITS files. The VST surveys follow a similar approach of combining multiple exposures. However, the resulting contiguous images are either re-gridded as single-extension tile images (e.g. KiDS) or stored as multi-extension FITS files (e.g. ATLAS, VPHAS+). The tabular data of sources extracted from imaging data products are named “source lists”. Typically, each image has a corresponding single-band source list.
High Acuity Wide field K-band Imager (HAWK-I) near infrared images and source lists were also added to the portfolio of reduced images in the ESO Science archive, which are produced as part of the ESO in-house reprocessing and also as science products from ESO Programmes, see for example the deep HAWK-I images for the Hubble Frontier Fields. The HAWK-I products include multi-extension stack exposures (4 detector chips), mosaiced tiles and source lists, plus variance and confidence maps as ancillary files. VIMOS products have a similar structure.
Ancillary data products are associated to scientific data products without being directly searchable via the query interface. For imaging data, weight maps giving the statistical significance of each pixel are associated to the images (see Fig. 2, right panel; Fig. 3, upper right panel). Other optional data products, sometimes associated to imaging data, are gain maps specifying the number of electrons that contributed to each pixel value of the image or masks to mark corrupted regions of the image (Fig. 3, lower panel). These ancillary data are usually stored in separate FITS files and automatically delivered together with the selected science files for download.
Alternatively, especially in the case of instruments with a small field of view, like ERIS, the ancillary files are provided within the same FITS file containing the imaging data, but in different extensions. The type of data stored per extension is identified by the use of the HDU class keyword definition mechanism.
Flux maps at a wavelength of 870 μm of 420 sq. deg from the APEX Telescope
are also available from the ESO archive. The variance maps, source lists and associated catalogues are also provided.
In the case of SPHERE, the data provided are from the high-contrast mode, taken with IRDIS. The imager produces two images for the left and right channel, containing the same astrophysical signal. The data are stored in a FITS file with a primary HDU containing just header keywords, no data, and two Image HDU in the extensions for the two IRDIS channels.
Calibrated spectra and data cubes
Spectroscopic data are served as one-dimensional reduced and wavelength calibrated spectra in FITS binary table format (Fig. 4). Information associated to science spectra is stored within separate columns of the same FITS-file as the science spectra themselves. Each data array (i.e. wavelength, flux, error, quality...) is provided in a single cell of the one row contained in the binary table. Wavelengths can be calibrated using wavelength lists measured in dry air (at standard temperature and pressure) or in vacuum, according to the value captured by the unified content descriptor for the wavelength array in the product’s header of the product.
Reduced spectra are produced by the Public Spectroscopic Surveys, or for selected instruments are also produced in house by ESO using science grade pipelines.
Newly obtained spectra are automatically reduced shortly after the observation. They are immediately accessible to the PIs via the user portal account and released to the public after the proprietary period.
An important addition to the ESO in house processed products are the integral field spectroscopy reduced data cubes, for instruments like MUSE and KMOS. Each data cube has a three-dimensional image with two spatial and one spectral coordinates, in its first extension; in the second extension contains the error information. The products are wavelength (in vacuum) and flux calibrated, and each of them come with an associated additional fits file that contains the two-dimensional white-light collapsed data cube image. See Fig. 5 for an example of a three-dimensional calibrated cube and the corresponding white-light image, with additional colour coding of the two-dimensional velocity field of the ionized gas in a galaxy.
Interferometric products
Interferometric calibrated data products obtained with PIONIER at the VLTI interferometer, include (squared) visibility amplitudes and closure phases, together with accompanying information on stations, targets, and wavelengths, that are stored in binary extension tables. Data must be in OIFITS format, and the primary HDU must not contain any data. Additional data products are going to to be delivered by the VLTI instruments GRAVITY and MATISSE.
Catalogues
Source catalogues may consist of sources extracted and band-merged from imaging data products, redshifts and additional physical parameter from spectra, or weak lensing shear measurements (e.g. KiDS release 3.1). These catalogues can be queried for single or multiple sources via the Phase 3 catalogue query interface (http://www.eso.org/qi/) by performing a cone or box search, or programmatically. Alternatively, the catalogues can also be downloaded partly or as a whole from the Phase 3 main query form of the ESO archive, together with the image or spectral products from which the measurements are extracted.
Simple catalogues (covering small patches of the sky or refering to target catalogues of spectroscopic public surveys) are stored in one single binary table. Catalogues that are either larger or referring to surveys covering large contiguous areas on the sky in a regular pattern, are stored in multiple files. For imaging surveys, each data file contains the extracted band-merged catalogue of one corresponding survey tile (see Fig. 6, lower panel).
In addition to the multi-band source catalogues, imaging surveys with multi-epoch observations provide multi-epoch catalogues (such as light curves) and catalogues of variable objects (e.g. VVV, VMC and PESSTO, see Fig. 6, upper panel).
