Data processing of X-Shooter data: Frequently asked questions

General

• Where can I find the science-ready data products available for X-Shooter, and more information about their processing?

Answer: The X-shooter data can be queried under

http://archive.eso.org/wdb/wdb/adp/phase3_spectral/form?collection_name=XSHOOTER.

The Release Description, with information on the processing and on known limitations, comes with the downloaded data and can also be found at http://www.eso.org/qc/PHOENIX/XSHOOTER/processing.html.

• Are there any known problems with X-Shooter data?

Answer: The quality control group keeps a list of know problems at

http://www.eso.org/observing/dfo/quality/XSHOOTER/qc/problems/problems_xshooter.html

• What is the best way to reduce X-Shooter data?

Answer: We recommend to run the ESO pipeline with a Reflex workflow. Pipelines and tutorials how to reduce the data are available at http://www.eso.org/sci/software/pipelines/.

Pipeline

• I heard about problems with the X-shooter wavelength calibration. Where can I find more information?

Answer: Users have reported wavelength shifts in both the telluric lines for VIS and NIR data and in the sky lines of NIR data.

A report analysing these problems can be found at

http://www.eso.org/sci/facilities/paranal/instruments/xshooter/doc/XS_wlc_shift_150615.pdf

• Processing of my NIR daytime calibration data fails - why?

Answer: In the NIR arm all arc lamp and flat field calibrations are taken with the lamp switched on (ON frames) and off (OFF frames). The OFF frames are needed to correct for the dark current. If they are not present the recipes xsh_predict, xsh_orderpos, xsh_2dmap, xsh_mflat and xsh_wavecal will fail.

• The MERGE2D spectrum shows lots of bad pixels. Why?

Answer: The X-shooter pipeline flags bad pixels that are detected during the processing and records them in the QUAL extension. It interpolates across those bad pixels selected by the decode-bp parameter only in the extraction to the MERGE1D spectrum. There is no interpolation across bad pixels in the MERGE2D spectrum.

• Negative pixel values in the 2D spectrum image are not being ignored by the standard extraction procedure. How can I ignore them?

Answer: The X-Shooter pipeline flags but does not ignore those pixels that are negative in its algorithms including standard extraction. This is because a reset anomaly can cause all pixels in a raw image to be negative even though the data is perfectly valid. If a reset anomaly has not occurred in your data, and negative pixels are causing a problem, then simply add 2097152 to the value of the parameter decode-bp to start treating negative pixels in the raw data as bad pixels by ignoring them.

Reflex workflow

• All my STARE data are processed as individual files - how can I have them stacked by the pipeline?

Answer: You need to edit the OCA rules, which are stored in

<install_dir>/share/esopipes/xshoo-<version>/reflex/xsh_wkf.oca


(or if you want to process all data in STARE mode edit 


<install_dir>/share/esopipes/xshoo-<version>/reflex/xsh_wkf_stare.oca).
There you need to look for SLIT data for the lines 


select execute(ACTION_SCI_SLIT_STARE_UVB) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_UVB"
group by DET.READ.CLOCK,SEQ.ARM,INS.OPTI3.NAME,INS.OPTI2.NAME,ARCFILE;


select execute(ACTION_SCI_SLIT_STARE_VIS) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_VIS"
group by DET.READ.CLOCK,SEQ.ARM,INS.OPTI4.NAME,INS.OPTI2.NAME,ARCFILE;


select execute(ACTION_SCI_SLIT_STARE_NIR) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_NIR"
group by DET.DIT,SEQ.ARM,INS.OPTI5.NAME,INS.OPTI2.NAME,ARCFILE;


select execute(ACTION_SCI_SLIT_STARE_JH_NIR) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_JH_NIR"
group by DET.DIT,SEQ.ARM,INS.OPTI5.NAME,INS.OPTI2.NAME,ARCFILE;


and for IFU data for the lines


select execute(ACTION_SCI_IFU_STARE_UVB) from inputFiles where RAW.TYPE=="SCI_IFU_STARE_UVB"
group by DET.READ.CLOCK,INS.OPTI2.NAME,SEQ.ARM,ARCFILE;


select execute(ACTION_SCI_IFU_STARE_VIS) from inputFiles where RAW.TYPE=="SCI_IFU_STARE_VIS"
group by DET.READ.CLOCK,INS.OPTI2.NAME,SEQ.ARM,ARCFILE;


select execute(ACTION_SCI_IFU_STARE_NIR) from inputFiles where RAW.TYPE=="SCI_IFU_STARE_NIR"
group by DET.DIT,INS.OPTI2.NAME,SEQ.ARM,ARCFILE;


and replace


ARCFILE


by


TPL.START as (TPL_A,tpl)


so that


select execute(ACTION_SCI_SLIT_STARE_UVB) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_UVB"


group by 

DET.READ.CLOCK,SEQ.ARM,INS.OPTI3.NAME,INS.OPTI2.NAME,ARCFILE;


changes to


select execute(ACTION_SCI_SLIT_STARE_UVB) from inputFiles where RAW.TYPE=="SCI_SLIT_STARE_UVB"


group by 

DET.READ.CLOCK,SEQ.ARM,INS.OPTI3.NAME,INS.OPTI2.NAME,TPL.START as (TPL_A,tpl);



In this way the pipeline will merge the multiple exposures taken in a single template and will process the combined frame. If instead you want to stack all data obtained within one OB replace TPL.START by OBS.START. We recommend to apply the modification in a new oca file, named e.g. xsh_wkf_modif.oca and then to load this file into the DataOrganiser.

• What is the meaning and use of the parameter UseResponseFlat in the FlatStrategy actor? 

Answer: The transmission/reflection behaviour of the dichroics, that separate the wavelength regions for the three X-shooter arms, sometimes varies on short time scales. In consequence the flat fields for the standard star and for the science target may differ in the overlap regions. Therefore the workflow is set to use by default the flat field taken for the science data also for the standard star (UseResponseFlat=false). However, there may also be cases, where this setting results in spurious features in the flux calibrated data (see the X-Shooter Reflex tutorial for more details).

In such cases one should also check the results of the flux calibration using the standard star flat field to calibrate the standard star observation (UseResponseFlat=true).

• The IFU slice object traces show a not negligible shift. How can I improve the corresponding object traces alignment?

Answer: From time to time the IFU slice traces do not overlap. In such a case the alignment of the IFU slices may be improved by providing (in the sof) the optional static table tagged as IFU_CFG_TAB_<ARM> (ARM = UVB, VIS, NIR).

If the offset between slices is constant with wavelength the corresponding coefficients S_LOW_OFF or S_UPP_OFF need to be corrected in this table. The correction is defined by the shift one wants to correct multiplied by the size of the spatial bin one has chosen to reduce the data (by default 0.16/0.16/0.21 arcsec/pixel for UVB/VIS/NIR data, respectively).

If the offset between the slices varies with wavelength also the other coefficients provided in that table need to be changed. The coefficients controlling the slope are W_UPP_COEF1 and W_LOW_COEF1. The coefficients controlling the curvature are W_UPP_COEF2 and W_LOW_COEF2.

If the user has to change these parameters we recommend to first optimise the offset, then the slopes (judging if it needs to be increased or decreased and, in order to make small variations, starting from the second significant digit of the default value) and similarly eventually changing the curvature coefficients.

This correction can be determined by aligning the traces of the telluric standard observation usually taken in the same night. The table optimised this way should then be used as input of the (usually low signal to noise) object data reduction.

• I activated the "Use CalSelector Associations" option in the DataOrganizer. Now I get an error even though the Datasets were marked as complete. Why?

Answer: This happens when the OCA rules to define the Datasets differ between Reflex and calSelector. We are currently working to homogenise the two tools and sets of rules. Until this has been achieved we recommend not to use this option in the DataOrganizer.

Processing problems

• My NODDING data are not processed.

Answer: NODDING DataSets must have at least two nodding positions to be processed successfully. If a template is aborted the user may receive incomplete data, i.e. with only one nodding position. The DataOrganiser in the current release will not mark such DataSets as incomplete and the recipe xsh_scired_slit_nod will fail. In this case please use for the DataOrganiser the OCA file xsh_wkf_stare.oca instead of the default one xsh_wkf.oca. This is done by rightclicking on the DataOrganiser, selecting Configure Actor, replacing the name of the OCA file in the first line of the pop-up window and clicking on Commit. This change will result in the pipeline processing the NODDING data individually with the xsh_scired_slit_stare recipe.

Please note that the default parameters for xsh_scired_slit_stare assume that the object is positioned at the center of the slit, which usually not true for NODDING data. So you may have to adjust the extraction parameters accordingly.

• My OFFSET data are not processed.

Answer: Offset mode data containing only object or only sky data cannot be processed by the xsh_scired_slit_offset recipe. The DataOrganiser in the current release will mark such DataSets as incomplete. If you attempt to process such datasets with xsh_scired_slit_offset the workflow will ignore them. Instead please use for the DataOrganiser the OCA file xsh_wkf_stare.oca instead of the default one xsh_wkf.oca. This is done by right-clicking on the DataOrganiser, selecting Configure Actor, replacing the name of the OCA file in the first line of the pop-up window and clicking on Commit.

This change will result in the pipeline processing the offset mode data individually with the xsh_scired_slit_stare recipe.

• Can I process OFFSET or NODDING mode data as STARE mode data?

Answer: Some users may prefer to reduce OFFSET mode or NODDING mode data like STARE data, i.e. determine the sky background from the frame itself. To do so with the Reflex workflow use for the DataOrganiser the OCA file xsh_wkf_stare.oca instead of the default one xsh_wkf.oca. This is done by right-clicking on the DataOrganiser, selecting Configure Actor, replacing the name of the OCA file in the first line of the pop-up window and clicking on Commit. This change will result in the pipeline processing the data individually with the xsh_scired_slit_stare recipe.

Please note that the default parameters for xsh_scired_slit_stare assume that the object is positioned at the center of the slit, which usually not true for NODDING data. So you may have to adjust the extraction parameters accordingly.

Flux calibration

• The response curve leaves residual instrument features in my science.

Answer: Changes in the spectral energy distribution between the observations of the flat field for the flux standard star and that for the science data can cause problems in flux calibrations. This can be avoided by flat-fielding the flux standard star spectrum with the same flat field as is used for the science data. This can be done setting UseResponseFlat to false in the Flat Strategy actor of Reflex.

• My flux-calibrated data do not agree with independent photometric measurements.

Answer: In order to have a true absolute flux calibration several requirements need to be fulfilled:

• The fraction of the total flux of an object that is contained in the slit depends on the shape of the object, the width and orientation of the slit, and the seeing. Absolute flux calibration requires that all the flux of both the object and the standard star has been collected.

• The flux that arrives at the telescope depends on the transparency of the sky. Absolute flux calibration requires the same transparency for the observations of the target and the standard star.


A change between the flux standard observation and the science object observations of any of the parameters mentioned above will change the flux scale in the final spectrum. To compare the flux calibrated spectrum to other measurements, differences in slit losses and atmospheric conditions have to be taken into account.

With respect to the faction of flux contained within the slit one should keep in mind that the flux standard stars are observed with a 5'' wide slit, while science data are typically observed with slit widths of 0.8'' to 1''. For a point source a slit width of 0.8''/1.0'' used with a seeing of 0.8'' means that some 33%/24%, respectively, of the target flux are lost. This results in a too low flux for the flux calibrated spectrum of the target. The slit losses usually vary between the three X-Shooter arms, as the seeing varies with wavelength and slit widths cannot be chosen arbitrarily.

If the standard star or the target or both are observed under non-photometric conditions (e.g. CLR or THN) their observed flux will be lower than it should be. If the standard star is observed under photometric conditions but the science target is not the flux in the flux calibrated target spectrum will be too low. The opposite happens if the target is observed under photometric conditions but the standard star is not. CLR/THN conditions allow for transparency variations of 10%/20%, respectively. Master response curves are derived from the upper envelope of individual response curves and should thus correspond to flux standard stars observed under photometric conditions.

• I cannot find flux standard stars with the same readout mode and binning as my science data.

Answer: X-SHOOTER flux standard stars (VIS/UVB) are always acquired in 100k high gain 1x1 read-mode. The pipeline applies corrections for differences in gain and binning during the flux calibration of the science data.

• Why do I see only a few peaks in the three display windows that appear after running the xsh_respon_slit_<mode> recipe in Reflex?

Answer: These peaks mess up the automatic scaling of the plots. You can pan (move) the spectrum and zoom in-out in the GUI. This will allow you to see the spectra and the response curve.

• How can I improve the flux calibration of my spectra?

Answer: If you need an absolute flux calibration make sure to use both science and spectrophotometric standard star observations obtained under photometric conditions. In addition, your science data (like your flux standard stars) must use a wide slit (5 arcsec) to avoid slit losses.

Problems at the edges of the wavelength ranges may be solved by using different flat fields. By default Reflex uses the same flat field for science data and flux standard stars (UseResponseFlat set to false).

If your flux standard stars were observed in OFFSET mode, sky subtraction in the VIS and NIR may be bad. Reducing them in STARE mode may help.

Sky correction and spectra extraction

• The sky in my NIR SLIT STARE data is not well subtracted.

Answer: The K-band region of the NIR spectra often causes problems for the sky correction. Here using sky-method = BSPLINE may be better than MEDIAN,  but requires some fiddling with the break points (see pipeline manual for more details).

• Some of my extracted NIR STARE spectra show a jump at about 2.3μm - why and how can I correct it?

Answer: The fit of the sky background used by the X-shooter pipeline relies on the assumption that there is no gradient in flux along the slit. For the reddest part of the NIR spectra this assumption is not fulfilled and therefore the sky subtraction does not work well. The jump you see is caused by a bad sky correction and shows up most prominently for sky-dominated observations, i.e. sources with little flux compared to the flux of the sky background in that wavelength range.

If this wavelength region is important for your analysis you need to process the data without sky subtraction and then use independent software tools to fit and subtract the sky from the rectified, wavelength calibrated 2-dimensional frame (MERGE2D). 

• I get inconsistent results from different absorption/emission lines in my NIR spectrum - why? 

Answer: The NIR detector is read in a way that flux is extrapolated once it leaves the normal regime. This is no problem as long as observing conditions stay constant. If for instance the transparency varies a linear extrapolation will no longer give correct results. You can find pixels which have extrapolated flux by looking for the value 1048576 or larger in the QUAL extension. Adding this number to the decode-bp parameter will cause these pixels to be treated as bad. For more details see this report.

• How is the sky spectrum for OFFSET data produced?

Answer: The product sky frame is generated as follows:

• All the input sky frames are averaged
• The master flat is applied
• The frame is re-sampled and an order by order 2D frame is generated
• This 2D frame is merged

• How can I improve the sky correction of my STARE data?

Answer: Depending on the observations (for instance in crowded fields), you may have to change the windows in the slit where the sky is determined (sky-position1/2 and sky-hheight1/2). By default the pipeline uses all parts of the slit not covered by the object extraction window (localize-slit-position and localize-slit-hheight) and not masked by sky-slit-edges-mask to fit the sky.

Alternatively you can try the BSPLINE method and vary its parameters.

• Why are some of the sky lines in my STARE data not corrected at all?

Answer: Bright sky lines have very steep slopes and may be flagged as cosmic ray hits. In that case they will be ignored during the sky fitting. This problem can be solved by switching off the cosmic ray detection, i.e. setting removecrhsingle-niter to 0. Due to the brighter sky lines at longer wavelengths the effect is mostly found for NIR data. 

• Is there an optimal extraction in nodding mode?

Answer: The optimal extraction is not available in the nodding mode, only in stare mode. If you really need to do so, you can process the data in stare mode instead of nodding mode.

• How can I improve spectrum extraction?

Answer: You can set the extraction method to LOCALIZATION and choose an appropriate extraction position (localize-slit-position) and half slit height (localize-slit-hheight) via the python GUI.

• My extracted 1-dimensional NIR spectra show many spikes although the raw signal looks very good. What happened?

Answer: For data observed with very good seeing, steep slopes in the spatial profile of the spectra result in many spurious cosmic ray detections. This happens most often for NIR data, because they have a coarser spatial sampling (0.21 arcsec/pixel vs. 0.16 arcsec for UVB/VIS data), but due to their longer wavelengths better seeing than the bluer wavelength ranges.

The spurious cosmics may in turn cause spikes as can be seen in the plot (black spectrum). If you process the data with decode-bp = decode-bp - 32 (effectively treating the cosmic ray detections as good pixels) the spikes caused by spurious comics should vanish (red spectrum).

• Why does the extracted NODDING spectrum not change if I set localize-method to MANUAL and vary localize-slit-position? 

Answer: These parameters have no effect unless you also change extract-method from NOD to LOCALIZATION.

• My spectrum of a bright source looks much noisier than I expected - why?

Answer: Bright spectra observed in very good seeing have very steep slopes and may be flagged as cosmic ray hits. During the extraction the pipeline then creates a large number of spikes in the regions flagged as cosmic ray hits. This problem can be solved by switching off the cosmic ray detection, i.e. setting removecrhsingle-niter to 0.
The figure shows an example of this effect and its correction. Due to the better seeing at longer wavelengths the effect is mostly found for NIR data. 

• What is the best way to reduce UVB/VIS NODDING data?

Answer: For UVB and VIS data the sky background is much lower than for NIR data. Therefore the subtraction of the NODDING frames from each other may add unnecessary noise to the target's spectrum. If the sky background is comparable or lower than the target's flux you will usually get better results if you reduce the NODDING data with the xsh_scired_slit_stare recipe. This recipe fits the sky background and therefore does not introduce additional noise with the sky correction.

To do so with the Reflex workflow use for the DataOrganiser the OCA file xsh_wkf_stare.oca instead of the default one xsh_wkf.oca. This is done by right-clicking on the DataOrganiser, selecting Configure Actor, replacing the name of the OCA file in the first line of the pop-up window and clicking on Commit. This change will result in the pipeline processing the data individually with the xsh_scired_slit_stare recipe.

Please note that the default parameters for xsh_scired_slit_stare assume that the object is positioned at the center of the slit, which usually not true for NODDING data. So you may have to adjust the extraction parameters accordingly.

• The sky subtraction in my NODDING data is really bad - what could be the reason?

Answer: The NODDING observing technique relies on a constant sky background along the slit, i.e. no spatial variations are present in the sky background. If this is not true (for instance in crowded fields, extended objects, star-forming regions) the sky will be different at different positions and therefore not cancel out.

There is no way to subtract such sky background properly within the X-shooter pipeline. The best way is to process the data as STARE (see http://www.eso.org/sci/data-processing/faq/can-i-process-offset-or-nodding-mode-data-as-stare-mode-data.html) without sky subtraction and then use independent software tools to fit and subtract the sky from the rectified, wavelength calibrated 2-dimensional frame (MERGE2D).

• My data have spatially variable sky background - how can I subtract it properly?

Answer: There is no way to subtract such sky background properly within the X-shooter pipeline. All processing strategies rely on a spatially constant sky background. The best way is to process the data as STARE (see http://www.eso.org/sci/data-processing/faq/can-i-process-offset-or-nodding-mode-data-as-stare-mode-data.html) without sky subtraction and then use independent software tools to fit and subtract the sky from the rectified, wavelength calibrated 2-dimensional frame (MERGE2D).

• My MERGE1D spectrum shows a gap although the MERGE2D spectrum does not.

Answer: During the 1-dimensional extraction the local spatial profile (along the cross-order direction) of the spectrum is determined by collapsing the 2-dimensional spectrum along the dispersion axis. The stdextract-interp-hsize parameter defines the half size of the region across which the spectrum is collapsed.

This parameter affects the interpolation of flagged pixels - if all columns are affected by bad pixels the result is set to 0, which causes the gap. In this case one needs to increase the default value to an optimal value of

stdextract-interp-hsize = size_of_gap[nm]/(2*size_of_pixel[nm])+1,

where size_of_pixel is given by the value of rectify-bin-lambda (by default 0, corresponding to 0.02nm for UVB and VIS data and to 0.06nm for NIR data, recorded in the CDELT1 FITS keyword of the products).

• On the MERGE2D spectrum I see blobs with intensity zero. Why? 

Answer: During resampling the pipeline uses a CPL routine which returns a parameter to indicate the confidence for the resampled pixel intensity. If the area used for resampling contains at least one pixel flagged as bad the pipeline flags the resulting resampled pixel as "MISSING DATA" (using code 524288), and sets the pixel flux to zero. This behaviour is currently mostly seen for BSPLINE sky subtraction where sky line residuals are flagged.

Post-Processing

• How can I correct telluric absorption lines in the Reflex science end products?

Answer: There are basically 2 ways:

You select the zones to correct, you normalize both the science and the std star spectra to the continuum, then you divide one by the other. This should remove the telluric lines. Be careful, depending on the telluric std star, you may have stellar lines as well in the std spectrum.

You do a fit of the lines on the telluric std star spectrum with an atmosphere model, this gives you a first guess that you will apply on the science spectrum and on this one you will adjust the residuals to remove again more properly the telluric lines.


A software tool to  fit and correct the telluric lines directly in your science spectra without the needs of using an observed telluric standard star is available at http://www.eso.org/sci/software/pipelines//skytools/.