<a id="episode-4-configuration"></a>

# Configuration

## Configuring ESMValTool via YAML files

ESMValTool provides a set of predefined configuration files.
These include the files specifying the default configuration values,
but also machine-specific files that include data sources for various
HPC systems.

To show all available files, run

```bash
esmvaltool config list
```

All configuration files are [YAML files](https://yaml.org/spec/1.2/spec.html).

To customize your configuration via YAML files, you can copy one of the existing files.
For example, to copy the file containing the default values for many options, run

```bash
esmvaltool config copy defaults/config-user.yml
```

The default configuration file will be downloaded to the default location:
`~/.config/esmvaltool/config-user.yml`, where `~` is the
path to your home directory. Note that files and directories starting with a
period are “hidden”, to see the `.config` directory in the terminal use
`ls -la ~`. Note, if a configuration file by that name already exists in the
default location, the `config copy` command will not update the file as
ESMValTool will not overwrite the file. You will have to move the file first if
you want an updated copy of the default user configuration file.

We run a text editor called `nano` to have a look inside the configuration file
and then modify it if needed:

```bash
nano ~/.config/esmvaltool/config-user.yml
```

If `nano` does not work on your system, or if you prefer a different editor,
any other editor can be used, e.g. `vim`.

This file contains the information for:

- Output settings
- Destination directory
- Auxiliary data directory
- Number of tasks that can be run in parallel
- …

## Destination directory

The example configuration file contains the option `output_dir`, which is the
rootpath where ESMValTool will store its output folders containing e.g. figures,
data, logs, etc. With every run, ESMValTool automatically generates a new output
folder determined by recipe name, and date and time using the format:
YYYYMMDD_HHMMSS.

## Output settings

Additionally you can configure the output settings that
inform ESMValTool about your preference for output.
Most of these settings are fairly self-explanatory.

## Other settings

## Customizing your configuration

By default, configuration files are read from the directory `~/.config/esmvaltool`.
This can be changed via the `ESMVALTOOL_CONFIG_DIR` environment variable.
In addition another custom configuration directory can be specified via the
`--config_dir` command line argument.
We will learn how to do this in the next lesson on recipes.

It is possible to have several configuration files with different purposes, for
example: `dask_options.yml`, `data_sources.yml`.
In this case, ESMValTool searches for all YAML files within each of the
configuration directories and merges them together. How this is done is explained
in the [Configuration section](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#yaml-files)
in the ESMValTool documentation.

To show the final configuration that is actually used when running ESMValTool, you can use

```bash
esmvaltool config show
```

## Rootpath to input data

ESMValTool uses several categories (in ESMValTool, these are referred to as projects)
for input data based on their source (e.g.
CMIP6, CMIP5, obs4mips, OBS6, OBS). For example, CMIP is used for a dataset from
the Climate Model Intercomparison Project whereas OBS may be
used for an observational dataset.
More information about the projects used in ESMValTool is available in the
[documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/find_data.html).
The `data` section for each project in the configuration
files defines sources of input data. The easiest way to get started with these is to
copy one of the example configuration files and tailor it to your needs.

When using ESMValTool on your own machine, the recommended setup can
be obtained by running the command

```bash
esmvaltool config copy data-local-esmvaltool.yml
```

After the file `data-local-esmvaltool.yml` has been copied to your configuration
directory `~/.config/esmvaltool/`, you can update the `rootpath` and the
`dirname_template` to match your file locations. The `rootpath` specifies the
directories where ESMValTool will look for input data of the specific project. The
`dirname_template` setting describes the file structure for each project.

If you are working on a HPC system, there are also several configurations
for popular HPC systems available that you can use instead, e.g. JASMIN, DKRZ,
ETH, and IPSL. To list the available example files, run the command:

```bash
esmvaltool config list data-hpc
```

To load the configuration suitable for the HPC system at DKRZ, run:

```bash
esmvaltool config copy data-hpc-dkrz.yml
```

It is also possible to ask ESMValTool to download climate model data as needed. When
running ESMValTool you can automatically download the files required to run a recipe
from ESGF for the projects CMIP3, CMIP5, CMIP6, CORDEX, and obs4MIPs. For this,
copy the appropriate configuration file by running

```bash
esmvaltool config copy data-intake-esgf.yml
```

Additionally, it is necessary to configure
[intake-esgf](https://intake-esgf.readthedocs.io/en/latest/configure).
For this you need to copy the file `conf.yaml` (see below) into the directory `~/.config/intake-esgf` and
update the `local_cache` and `esg_dataroot` with your desired download directory in this intake-esgf
configuration file. The updated file should look like this:

### conf.yaml

```yaml
additional_df_cols: []
break_on_error: true
confirm_download: false
default_open_kwargs:
  chunks: auto
  compat: override
  coords: minimal
  data_vars: minimal
default_project: CMIP7
download_db: ~/.config/intake-esgf/download.db
esg_dataroot:
- /p/css03/esgf_publish
- /eagle/projects/ESGF2/esg_dataroot
- /global/cfs/projectdirs/m3522/cmip6/
- /glade/campaign/collections/cmip.mirror
globus_indices:
  ESGF2-US-1.5-Catalog: true
local_cache:
- ~/.esgf/
logfile: ~/.config/intake-esgf/esgf.log
num_threads: 6
print_log_on_error: false
requests_cache:
  cache_name: intake-esgf/requests-cache.sqlite
  expire_after: 3600
  use_cache_dir: true
slow_download_threshold: 0.5
solr_indices:
  esg-dn1.nsc.liu.se: false
  esgf-data.dkrz.de: false
  esgf-node.ipsl.upmc.fr: false
  esgf.ceda.ac.uk: false
  esgf.nci.org.au: false
stac_indices:
  discovery.east.esgf.io: true
  discovery.west.esgf.io: true
```
