<a id="new-dataset"></a>

# Making a new dataset

If you are contributing a new dataset, please have a look at
[Writing a CMORizer script for an additional dataset](../develop/dataset.html.md#new-cmorizer) for how to do so.
Please always create separate pull requests for CMORizer scripts, even when
introducing a new dataset or updating an existing dataset with a new recipe.

If you are updating a CMORizer script to support a different dataset version,
please have a look at [Support for multiple versions of a dataset](multiple_dataset_versions.html.md#dataset-versions) for how to handle multiple
dataset versions.

<a id="dataset-documentation"></a>

## Dataset documentation

The documentation required for a CMORizer script is the following:

- Make sure that the new dataset is added to the list of
  [Supported datasets for which a CMORizer script is available](../input.html.md#supported-datasets) and to the file [datasets.yml](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/datasets.yml).
- The code documentation should contain clear instructions on how to obtain
  the data.
- A BibTeX file named `<dataset>.bibtex` defining the reference for the new
  dataset should be placed in the directory `esmvaltool/references/`, see
  [Adding references](diagnostic.html.md#adding-references) for detailed instructions.

For more general information on writing documentation, see [Documentation](code_documentation.html.md#documentation).

<a id="dataset-test"></a>

## Testing

When contributing a new script, add an entry for the CMORized data to
[recipes/examples/recipe_check_obs.yml](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/examples/recipe_check_obs.yml)
and run the recipe, to make sure the CMOR checks pass without warnings or errors.

To test a pull request for a new CMORizer script:

1. Download the data following the instructions included in the script and
   place it in the directory specified by the `--original-data-dir` flag
   followed by a subdirectory `Tier1`, `Tier2` or `Tier3`, followed by
   a subdirectory with the same name as the dataset.
2. If available, use the downloading script by running
   `esmvaltool data download --original-data-dir </path/to/save/data> <dataset>`
3. Run the cmorization by running `esmvaltool data format --original-data-dir </path/to/save/data> <dataset>`
4. Copy the resulting data to the directory specified in your
   [configuration](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#config-data-sources) for the `OBS`
   (for CMIP5 compliant data) or `OBS6` (for CMIP6 compliant data) project.
5. Run `recipes/examples/recipe_check_obs.yml` with the new dataset to check that
   the data can be used

<a id="dataset-sanity-check"></a>

## Scientific sanity check

When contributing a new dataset, we expect that the numbers and units of the
dataset look physically meaningful.
The scientific reviewer needs to check this.

## Data availability

Once your pull request has been approved by the reviewers, ask a member of
[@OBS-maintainers](https://github.com/orgs/ESMValGroup/teams/obs-maintainers)
to add the new dataset to the data pool at DKRZ and CEDA-Jasmin.
This team is in charge of merging CMORizer pull requests.

<a id="dataset-checklist"></a>

## Detailed checklist for reviews

This (non-exhaustive) checklist provides ideas for things to check when
reviewing pull requests for new or updated CMORizer scripts.

### Dataset description

Check that new dataset has been added to the table of observations defined in
the ESMValTool guide user’s guide in section [Obtaining input data](../input.html.md#inputdata)
(generated from `doc/sphinx/source/input.rst`).
Check that the new dataset has also been added to the file [datasets.yml](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/datasets.yml).

### BibTeX info file

Check that a BibTeX file, i.e. `<dataset>.bibtex` defining the reference for
the new dataset has been created in `esmvaltool/references/`.

### recipe_check_obs.yml

Check that new dataset has been added to the testing recipe
`esmvaltool/recipes/examples/recipe_check_obs.yml`

### Downloader script

If present, check that the new downloader script
`esmvaltool/cmorizers/data/downloaders/datasets/<dataset>.py`
meets standards.
This includes the following items:

* Code quality checks
  1. Code quality
  2. No Codacy errors reported

### CMORizer script

Check that the new CMORizer script
`esmvaltool/cmorizers/data/formatters/datasets/<dataset>.{py,ncl}`
meets standards.
This includes the following items:

* In-code documentation (header) contains
  1. Download instructions
  2. Reference(s)
* Code quality checks
  1. Code quality (e.g. no hardcoded pathnames)
  2. No Codacy errors reported

### Config file

If present, check config file `<dataset>.yml` in
`esmvaltool/cmorizers/data/cmor_config/` for correctness.
Use `yamllint` to check for syntax errors and common mistakes.

### Run downloader script

If available, make sure the downloader script is working by running
`esmvaltool data download --config_file <config-file> <dataset>`

### Run CMORizer

Make sure CMORizer is working by running
`esmvaltool data format --config_file <config-file> <dataset>`

### Check output of CMORizer

After successfully running the new CMORizer, check that:

* Output contains (some) valid values (e.g. not only nan or zeros)
* Metadata is defined properly

Run `esmvaltool/recipes/examples/recipe_check_obs.yml` for new dataset.

### RAW data

Contact the team in charge of ESMValTool data pool ([@OBS-maintainers](https://github.com/orgs/ESMValGroup/teams/obs-maintainers)) and
request to copy RAW data to RAWOBS/Tier2 (Tier3).

### CMORized data

Contact the team in charge of ESMValTool data pool ([@OBS-maintainers](https://github.com/orgs/ESMValGroup/teams/obs-maintainers)) and
request to

* Merge the pull request
* Copy CMORized dataset to OBS/Tier2 (Tier3)
* Set file access rights for new dataset
