<!-- Glossary substitutions make it easier to refer to pluralised / -->
<!-- capitalised glossary terms. -->
<!-- Links -->

# The workflow

## An overview of the workflow

The [RTW](../glossary.html.md#term-RTW) performs the following steps:

`install_env_file`
: * **Description:**
    Copies the environment file for [ESMValTool](../glossary.html.md#term-ESMValTool), based on the `SITE`
    provided
  * **Runs on:**
    Localhost
  * **Executes:**
    The `cp` command from the [Rose](../glossary.html.md#term-Rose) app
  * **Details:**
    Runs once at the start of the workflow

`get_esmval`
: * **Description:**
    Either clones the latest versions of [ESMValTool](../glossary.html.md#term-ESMValTool) and [ESMValCore](../glossary.html.md#term-ESMValCore) from
    GitHub, or gets the latest container image from DockerHub and converts to
    a singularity image, depending on `SITE`
  * **Runs on:**
    Localhost, or `COMPUTE` on JASMIN
  * **Executes:**
    The `clone_latest_esmval.sh` script (if cloning), or a
    `singularity build` command (if getting container) from the [Rose](../glossary.html.md#term-Rose) app
  * **Details:**
    Runs at the start of each cycle

`configure`
: * **Description:**
    Creates the [ESMValTool](../glossary.html.md#term-ESMValTool) user configuration file and validates it
  * **Runs on:**
    Localhost
  * **Executes:**
    The `configure.py` script from the [Rose](../glossary.html.md#term-Rose) app
  * **Details:**
    Runs each cycle after `get_esmval` has completed

`process`
: * **Description:**
    Runs the requested recipes using [ESMValTool](../glossary.html.md#term-ESMValTool)
  * **Runs on:**
    `COMPUTE`, which depends on the `SITE`
  * **Executes:**
    The [ESMValTool](../glossary.html.md#term-ESMValTool) command line script from the [Rose](../glossary.html.md#term-Rose) app
  * **Details:**
    Runs each cycle for every recipe defined in the [RTW](../glossary.html.md#term-RTW) after `configure`
    has completed

`compare`
: * **Description:**
    Compares the output from the `process` job with [KGOs](../glossary.html.md#term-KGOs)
  * **Runs on:**
    `COMPUTE`, which depends on the `SITE`
  * **Executes:**
    The `esmvaltool develop compare` command
    from the [Rose](../glossary.html.md#term-Rose) app
  * **Details:**
    Runs each cycle for every recipe defined in the [RTW](../glossary.html.md#term-RTW) after `process`
    has completed

`housekeeping`
: * **Description:**
    Removes the logs and data (including recipe outputs)
    from two days or more before the current cycle
  * **Runs on:**
    Localhost
  * **Executes:**
    The [`rose_prune`](https://metomi.github.io/rose/doc/html/api/built-in/rose_prune.html) application
  * **Details:**
    Runs at the end of each cycle (except the first cycle)

## Design considerations

### Portability

The [RTW](../glossary.html.md#term-RTW) is portable; site-specific information can be found in the `site`
and `opt` directories within the [RTW](../glossary.html.md#term-RTW). The files required are:

<a id="site-recipes-file"></a>

`site/<site>/recipes.jinja`
: Contains all the recipes run at the `SITE`

#### HINT
* The file uses the [Jinja2](https://jinja.palletsprojects.com/en/stable/) templating language,
  which has a similar syntax to Python
* Jinja2 gives [Cylc](../glossary.html.md#term-Cylc) many powerful features;
  [Cylc Jinja2](https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/jinja2.html) provides more information

`site/<site>/runtime.cylc`
: Contains task definitions specific to the `SITE`, for example, `COMPUTE`

`site/<site>/env-file`
: Contains details on how to set up the environment for ESMValTool at the
  `SITE`

`opt/rose-suite-<site>.conf`
: Contains configuration items specific to the `SITE`, including `SITE`
  and where to find the required datasets

### Metadata

The [RTW](../glossary.html.md#term-RTW) uses Rose metadata. Every item defined in the suite configuration
file (`rose-suite.conf`) will have an entry in the main metadata
configuration file (`meta/rose-meta.conf`).

### Resources

The resources used by the `process` jobs
are defined in the `site/<site>/recipes.jinja` file,
allowing the jobs to be configured by `SITE` as well as by recipe.
This ensures only the required resources are requested
when running each of the `process` jobs.
