Skip to main content

Mini guide to installing/upgrading ESO pipelines via install scripts - Knowledgebase / Data processing and analysis software resources - ESO Operations Helpdesk

Mini guide to installing/upgrading ESO pipelines via install scripts

Authors list

Mini guide to installing and/or upgrading ESO pipelines via install scripts


Before starting...

Each ESO pipeline Source kit file contains all the software required to build that pipeline (except 'standard' softwares like compilers that need to be installed as pre-requisities).

The ESO pipeline install script methods allow to install pipelines for multiple instruments together making use of a common installation of common components (such as CPL, esorex, cfitsio, etc). The drawback though is that install script installations can not be easily managed, i.e. adding, removing or upgrading pipelines is not supported. Once an install script installation is completed, the only way to add, remove or upgrade a pipeline (or any of the common components) is to redo the complete installation.

Package manager installations on the other hand allow easy adding, removing and upgrading of the pipelines and all components. So if you are using Linux under one of our supported distributions or any version of macOS then we strongly recommend that you use our package manager installation method (MacPorts or RPM). 

After following the instructions on those pages, the pipelines should "just work out of the box" via esoreflex or esorex (Gasgano is not supported).

If you are not using macOS or Linux under one of our supported distributions, or do not have administrative privileges on your computer and can not get someone with administrative privileges to install the software for you, then the next best solution is indeed the install_esoreflex script method.

Before starting an install script installation, please insure that you have all pre-requisities installed.

Upgrading via install scripts

The install scripts do not support "upgrading" an existing esoreflex/pipeline installation in place. This also applies to adding or removing an instrument pipeline. Instead one must re-install the complete esoreflex/pipeline software stack for all the pipelines one requires.

So to "upgrade" simply follow the relevant section below and install the new version/new set of pipelines you require.

If you've followed the instructions below before, removing the current/old is easy, just remove the entire "INSTALL_DIR" of the current/old installation. If you have enough disk space, you may want to wait till the new installation is successfully installed, before removing the current/old one.

esoreflex and esorex

We recommend using esoreflex for processing ESO data, especially if you are a new or non expert user. Instructions to install the pipelines with Gasgano are provided below.

For this method you only need to "manually" download the install_esoreflex script itself. Everything else will then be downloaded by the install script as needed when you run it.

We recommend that you install the software as follows:

bash
 INSTALL_DIR=${HOME}/pipelines/reflex/$(date +%Y-%m-%d)
 mkdir -pv ${INSTALL_DIR}
 cd ${INSTALL_DIR}
 curl -O https://ftp.eso.org/pub/dfs/reflex/install_esoreflex
 env -i PATH=$(getconf PATH) bash ./install_esoreflex  

This will install the software in a self-contained, isolated way (making it easy to remove or re-install) and also avoid issues with with other installed software (e.g. *conda).

If you are planning to use esoreflex, it is ok during installation if you system python does not provide all the required modules, but you will need to either install those required python modules for your system python, or create a (dedicated) python environment for esoreflex and configure esoreflex to use that environment, please see  here.

Gasgano

In case you feel you need Gasgano, we recommend installing via the following method:

Double check you have installed the software pre-requisities, in particular a Java 1.8 JDK (e.g. java-1.8.0-opnjdk-devel RPM systems, openjdk-8-jdk on Ubuntu systems, openjdk8-temurin/openjdk8-zulu from MacPorts on macOS x86_64/aarm64).

Download

bash
INSTALL_DIR=${HOME}/pipelines/gasgano/$(date +%Y-%m-%d)
mkdir -pv "${INSTALL_DIR}"
cd ${INSTALL_DIR}
curl -O https://www.eso.org/sci/software/pipelines/install_pipelinekit 

You will then need to download the "Source kit" files (third column of the table under the 'Instrument Pipeline Packages' section here) for each pipeline you wish to install into the above directory.

Build

Once you have done so then install as follows, depending on your OS:

RPM based...

env -i PATH=$(getconf PATH) JAVA_HOME=/etc/alternatives/java_sdk_1.8.0_openjdk bash ./install_pipelinekit ${INSTALL_DIR} *-kit-*.tar.gz

APT based...

env -i PATH=$(getconf PATH) JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bash ./install_pipelinekit ${INSTALL_DIR} *-kit-*.tar.gz

macOS x86_64...

env -i PATH=/opt/local/bin:$(getconf PATH) JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk8-temurin/Contents/Home bash ./install_pipelinekit ${INSTALL_DIR} *-kit-*.tar.gz

macOS aarm64 (a.k.a. M1 a.k.a. Apple Silicon)...

env -i PATH=/opt/local/bin:$(getconf PATH) JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk8-zulu/Contents/Home FC=gfortran-mp-devel bash ./install_pipelinekit ${INSTALL_DIR} *-kit-*.tar.gz

This will install the software in a self-contained isolated way (making it easy to remove (rm -fr ${INSTALL_DIR}) or re-install) and also avoid issues with with other installed software (e.g. *conda).

Modify installed gasgano script

Before running gasgano, the gasgano script needs to be modified:

Linux

sed -i -e 's/^unset JAVA_HOME/#&/' ${HOME}/gasgano/bin/gasgano 

macOS

sed -i '' -e 's/^unset JAVA_HOME/#&/' ${HOME}/gasgano/bin/gasgano 

Run

Appropriate environment variables (JAVA_HOME & CPL_DIR) then need to be set for run time:

env -i PATH=$(getconf PATH) JAVA_HOME=/etc/alternatives/java_sdk_1.8.0_openjdk CPLDIR=$(pwd)/install ${HOME}/gasgano/bin/gasgano 

RPM based...

env -i PATH=$(getconf PATH) JAVA_HOME=/etc/alternatives/java_sdk_1.8.0_openjdk CPLDIR=$(pwd)/install ${HOME}/gasgano/bin/gasgano 

APT based...

env -i PATH=$(getconf PATH) JAVA_HOME=//usr/lib/jvm/java-8-openjdk-amd64 CPLDIR=$(pwd)/install ${HOME}/gasgano/bin/gasgano 

macOS x86_64...

JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk8-temurin/Contents/Home CPLDIR=$(pwd)/install ${HOME}/gasgano/bin/gasgano 

macOS aarm64 (a.k.a. M1 a.k.a. Apple Silicon)...

JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk8-zulu/Contents/Home CPLDIR=$(pwd)/install ${HOME}/gasgano/bin/gasgano 

JAVA_HOME and CPL_DIR could be set in your login scripts, or you could create an alias for the above command.