ESO Archive Frequently Asked Questions

Getting Data (CalSelector)

• Can I customize the association by specifying the nature and number of calibration files?

No, at this stage it is not possible to do so. Associations between raw science and raw and static calibrations are designed to reproduce the calibration plan of the relevant instrument mode. You can find the calibration plans as part of the instrument documentation (https://www.eso.org/sci/facilities/paranal/instruments/instrumentName/doc, e.g. https://www.eso.org/sci/facilities/paranal/instruments/hawki/doc).

• How do I access the user documentation in addition to these FAQs?

Detailed user documentation and a table summarizing the tool's coverage per instrument are available.

• How do I trigger the CalSelector tool, i.e. the association of science and calibrations?

As described in Detailed user documentation there are 3 ways to trigger the CalSelector tool: via an archive request, programmatically, or via the new Download Portal. Please note however that, under certain circumstances, CalSelector behaves erratically on request that contain a mixture of science and non-science (e.g. acquisition images) files. We would strongly encourage you to include only science files in the query (e.g. by selecting Science as Category in the main query interface) and let CalSelector associate the other relevant files.

• I am denied access to calibrations, which should always be public. Why is that?

The basic idea of the CalSelector tool is to return calibrations relevant to science files. As such, if you do not have access to some science files, for example because they are still proprietary to another user, the corresponding calibrations in the dataset are also not downloadable. Of course, if the same calibrations are also needed for other science file(s) you do have access to, then they will be downloadable as part of those datasets.

• In some cases the association is marked as incomplete, but it seems to me that all the necessary files are there. Could you, please, advise?

In addition to the files strictly needed for processing, the CalSelector attempts to include in the dataset also additional files that can be of general interest (e.g. acquisition images, through slit images, standard stars in different filters than the science to compute photometric color terms, etc.). If these "secondary" associations are incomplete, the whole dataset is marked as incomplete. More information on the missing files is provided in the corresponding "messages" section of the xml file included in each dataset.

• My request is split over several pages and the “Select All” button allows me to select for download only the files listed in one single page. Is there a way to download the entire request at once?

Indeed, requests via the old way consisting of more than 1,000 files are split among different pages. In this case a “Download Request” button allows downloading the entire request at once, without having to thumb through the different pages.

• What do all those icons in the request page mean (old way)?

• The icons to the left of a dataset name indicate the completeness of the association between science and calibration files.

  • the association is complete (green tickmark, xml attribute complete="true": all of the intended files were successfully associated)
  • the association is incomplete (orange tickmark, xml attribute complete="false": some of the intended files are missing because, e.g., they have not yet been acquired at the telescope)
  • the association is empty (yellow tickmark: e.g. the input science files are outside of the tool's applicability period).

• The icons in the rightmost column (“Access”) indicate whether the individual files are accessible to you (green tickmark) or not (e.g. because the file is still under proprietary period of another user; red no access sign).

• What do file types MASSOC and RASSOC mean?

MASSOC and RASSOC are, respectively, masters and raw files that are not needed for processing, but contain useful additional information. Typical examples are acquisition or through slit images.

• What is the content of the association package?

A package, or more properly a dataset, contains a set of raw science files that should be processed together (e.g. a jitter sequence in the infrared, or a nodding sequence in spectroscopy), the raw (e.g. bias and flat frames, arc exposures, etc.) and static (e.g. arc line lists, standard star lists, etc.) calibrations needed to process them, an xml representation of the calibration cascade and, if available, excerpts from the observing night log.

• What is the time coverage of the CalSelector?

The time coverage of CalSelector varies by instrument and is summarized here.

• What is the xml file that comes in each dataset?

The relation among files, i.e. which files are needed to calibrate what, is represented as an xml tree. Xml is a rather common format nowadays and there are plenty of readers (and editors) available. For example, popular web browsers can be used to render xml either natively (e.g. Google Chrome or Firefox), or with a dedicated plugin (e.g. Safari). Popular scripting languages like python and idl have xml parsers.

The xml returned by CalSelector is a nested series of <association/> elements, each of which contains a list of mainFiles and the corresponding associatedFiles. This latter contains, in turn, another association and so on and so forth until the level of files that do not need calibrations is reached. 

Each fits file listed in the xml is assigned a type, which is suitable to be provided as input to the ESO pipelines (for more information on the pipelines, please refer to the relevant User Manuals).

A sample xml file is available.

• When do associations become available?

Associations based on uncertified calibrations typically become available within two hours of acquisition at the telescope. They are, then, progressively replaced by associations containing certified calibrations as soon as the calibrations themselves are checked for quality (typically after two working days). In order to access these improved associations you need to re-run CalSelector by resubmitting the archive request.

• Why are sometimes more science files associated than requested?

When appropriate, the tool tries to complete an observing template, so as to provide all the necessary files to be processed together (e.g. on-target/off-target offsets to compute the background, or a science-calibrator sequence for VLTI).

• Why are there fewer xml files than datasets as the result of my request?

Under certain circumstances CalSelector does not return an xml file if no association could be established for the input science files (normally it should return an xml file in all cases). The requested input science files are, however, correctly made available for download.

• Why does my query with CalSelector return a lot of incomplete associations?

It is probably because your request contains a mixture of science and non-science (e.g. acquisition images) files, which occasionally causes CalSelector to behave erratically. We would strongly encourage you to include only science files in the query (e.g. by selecting Science as Category in the main query interface) and let CalSelector associate the other relevant files.

• Why is the category of a file not displayed for some files?

The file category is not assigned if, for whatever reason, an association could not be established.

• Why is the number of files requested different from the number of files downloaded?

During the Calselector process, some files may be associated several times to different datasets, but they will be downloaded only once. This is the reason why you may observe such differences between the number of requested files and the number of (unique) files available for download. In a future release, only the number of (unique) files available for download will be displayed.

Please note that when using the old way to download raw data, night log files are not included when the size to be displayed on the web interface is computed whereas they are included in the list of files available for download. For instance, for a given request, the number of files is 200 on the web page but 212 in the notification as 12 night logs are available.