<a id="contributing-code-docs"></a>

# Contributing code and documentation

If you would like to contribute a new diagnostic and recipe or a new feature,
please discuss your idea with the development team before getting started, to
avoid double work and/or disappointment later.
A good way to do this is to open an
[issue on GitHub](https://github.com/ESMValGroup/ESMValTool/issues).
This is also a good way to get help with the implementation.

We value the time you invest in contributing and strive to make the process as
easy as possible.
If you have suggestions for improving the process of contributing, please do
not hesitate to propose them, for example by starting a discussion on our
[discussions page](https://github.com/ESMValGroup/ESMValTool/discussions).

## Getting started

See [Install from source](../quickstart/installation.html.md#install-from-source) for instructions on how to set up a development
installation.

New development should preferably be done in the
[ESMValTool](https://github.com/ESMValGroup/ESMValTool)
GitHub repository.
However, for scientists requiring confidentiality, private repositories are
available, see [Moving work from the private to the public repository](private_repository.html.md#private-repository) for more information.
The default git branch is `main`. Use
this branch to create a new feature branch from and make a pull request
against.
This
[page](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow)
offers a good introduction to git branches, but it was written for
BitBucket while we use GitHub, so replace the word BitBucket by GitHub
whenever you read it.
The first time you open a pull request,
you will need to agree to our contributor license agreement:
see [Contributor License Agreement](contributor_license_agreement.html.md#contributor-license-agreement).

It is recommended that you open a [draft pull
request](https://github.blog/2019-02-14-introducing-draft-pull-requests/)
early, as this will cause [CircleCI to run the unit tests](#tests),
[Codacy to analyse your code](#code-quality), and
[readthedocs to build the documentation](#documentation).
It’s also easier to get help from other developers if
your code is visible in a pull request.

Please review the results of the automatic checks below your pull request.
If one of the tests shows a red cross instead of a green checkmark, please click
the `Details` link behind the failing check and try to solve the issue.
Ask [@ESMValGroup/tech-reviewers](https://github.com/orgs/ESMValGroup/teams/tech-reviewers) for help if you do not know how to fix the
failing check.
Note that this kind of automated checks make it easier to
[review code](review.html.md#reviewing), but they are not flawless.
Preferably Codacy code quality checks pass, however a few remaining hard to
solve Codacy issues are still acceptable.
If you suspect Codacy may be wrong, please ask by commenting on your pull
request.

<a id="pull-request-checklist"></a>

## Checklist for pull requests

To clearly communicate up front what is expected from a pull request, we have
the following checklist.
Please try to do everything on the list before requesting a review.
If you are unsure about something on the list, please ask the
[@ESMValGroup/tech-reviewers](https://github.com/orgs/ESMValGroup/teams/tech-reviewers) or [@ESMValGroup/science-reviewers](https://github.com/orgs/ESMValGroup/teams/science-reviewers) for help
by commenting on your (draft) pull request or by starting a new
[discussion](https://github.com/ESMValGroup/ESMValTool/discussions).

In the ESMValTool community we use
[pull request reviews](review.html.md#reviewing) to ensure all code and
documentation contributions are of good quality.
The icons indicate whether the item will be checked during the
[🛠 Technical review](review.html.md#technical-review) or
[🧪 Scientific review](review.html.md#scientific-review).

### All pull requests

- 🛠 [The pull request has a descriptive title](#descriptive-pr-title)
- 🛠 [AI policy](ai_policy.html.md#ai-policy) has been followed and any AI usage has been disclosed
- 🛠 Code is written according to the [code quality guidelines](#code-quality)
- 🛠 [Documentation]() is available
- 🛠 The [list of authors](#authors) is up to date
- 🛠 Changed dependencies are [added or removed correctly](#dependencies)
- 🛠 The [checks shown below the pull request](#pull-request-checks) are successful

If a pull request introduces a change that causes a recipe to
no longer run successfully (*breaking change*), or which results in scientifically
significant changes in results (*science change*), additional requirements
defined in the [backward compatibility policy](backward_compatibility.html.md#backward-compatibility-policy) apply.
These include in particular:

- 🛠 Instructions for the release notes to assist *recipe
  developers* to adapt their recipe in light of the *backward-incompatible change*
  available.
- 🛠 If applicable, instructions for *recipe developers* working on *user
  recipes* to enable them to adapt their code related to
  *backward-incompatible changes* available (see [ESMValTool_Tutorial: issue
  #263](https://github.com/ESMValGroup/ESMValTool_Tutorial/issues/263))
  available.
- 🛠 Core development team tagged to notify them of the
  *backward-incompatible change*, and give at least
  2 weeks for objections to be raised before merging to the main
  branch. If a strong objection is raised the backward-incompatible
  change should not be merged until the objection is resolved.
- 🛠 Information required for the “*backward-incompatible changes*”
  section in the PR  that introduces the *backward-incompatible change*
  available.

### New or updated recipe and/or diagnostic

See [Making a new diagnostic or recipe](diagnostic.html.md#new-diagnostic) for detailed instructions.

- 🧪 [Recipe runs successfully](diagnostic.html.md#testing-recipes)
- 🧪 [Recipe and diagnostic documentation](diagnostic.html.md#recipe-documentation) is available
- 🧪 [Figure(s) and data](diagnostic.html.md#diagnostic-output) look as expected from literature
- 🛠 [Provenance information](diagnostic.html.md#recording-provenance) has been added

### New or updated data reformatting script

See [new dataset](dataset.html.md#new-dataset) for detailed instructions.

- 🛠 [Dataset documentation](dataset.html.md#dataset-documentation) is available
- 🛠 The dataset has been [added to the CMOR check recipe](dataset.html.md#dataset-test)
- 🧪 Numbers and units of the data look [physically meaningful](dataset.html.md#dataset-sanity-check)

<a id="descriptive-pr-title"></a>

## Pull request title

The title of a pull request should clearly describe what the pull request changes.
If you need more text to describe what the pull request does, please add it in
the description.
The titles of pull requests are used to compile the [Changelog](../changelog.html.md#changelog), therefore
it is important that they are easy to understand for people who are not
familiar with the code or people in the project.
Descriptive pull request titles also makes it easier to find back what was
changed when, which is useful in case a bug was introduced.

<a id="code-quality"></a>

## Code quality

To increase the readability and maintainability or the ESMValTool source
code, we aim to adhere to best practices and coding standards.
For code in all languages, it is highly recommended that you split your code up
in functions that are short enough to view without scrolling, e.g. no more than
50 lines long.

We include checks for Python, R, NCL, and yaml files, most of which are
described in more detail in the sections below.
This includes checks for invalid syntax and formatting errors.
[Pre-commit](../utils/utils.html.md#pre-commit) is a handy tool that can run all of these checks automatically
just before you commit your code.
It knows which tool to run for each filetype, and therefore provides
a convenient way to check your code.

To enable pre-commit checks (these will be run every time you commit something),
go to the `ESMValTool` directory and run:

```bash
pre-commit install
```

just once, after you have cloned the ESMValTool repository.

If you have installed the pre-commit checks, but for some reason need to skip
them (not recommended), use the `-n` flag:

```bash
git commit -n
```

To run the automatic code formatting and basic code quality checks, use the command:

```bash
pre-commit run -a
```

### Python

The standard document on best practices for Python code is
[PEP8](https://www.python.org/dev/peps/pep-0008/) and there is
[PEP257](https://www.python.org/dev/peps/pep-0257/) for code documentation.
We make use of
[numpy style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html)
to document Python functions that are visible on
[readthedocs](https://docs.esmvaltool.org).

To automatically format and check if code adheres to the standard, we use
[ruff](https://docs.astral.sh/ruff/). Automatic formatting and basic code
quality checks are enforced through pre-commit. In addition to these basic
quality checks, we use ruff to make recommendations on how your code could be
even better.

To automatically format your code and perform basic quality checks, run:

```bash
pre-commit run -a
```

To perform the additional code quality checks and automatically fix many of
the reported issues, run:

```bash
ruff check --fix esmvaltool/diag_scripts/your_diagnostic/your_script.py
```

Each code quality check performed by ruff is called a `rule` and has a unique
code, e.g. `RET504`. For a detailed explanation of the purpose of these rules,
you can visit the
[rules page on the ruff website](https://docs.astral.sh/ruff/rules) or
run the command:

```bash
ruff rule RET504
```

where `RET504` is an arbitrary example of a rule code.

When you make a pull request, adherence to the Python development best practices
is checked in two ways:

1. As part of the unit tests, pre-commit is run by
   [CircleCI](https://app.circleci.com/pipelines/github/ESMValGroup/ESMValTool),
   see the section on [Tests]() for more information.
2. [Codacy](https://app.codacy.com/gh/ESMValGroup/ESMValTool/pullRequests)
   is a service that runs ruff (and other code quality tools) on changed
   files and reports the results.
   Click on the ‘Codacy Static Code Analysis’ check entry below your pull request
   and then click the ‘Resolve’ button or the
   ‘View more details on Codacy Production’ link to see the results of the static
   code analysis done by [Codacy](https://app.codacy.com/gh/ESMValGroup/ESMValTool/pullRequests).
   If you need to log in, you can do so using your GitHub account.
   If Codacy says that there are issues, but you cannot see them on the Codacy
   webpage for some reason, please run the command `ruff check` on your own
   computer to see what the issues are.

A pull request should preferably not introduce new code quality issues.
However, we understand that there is a limit to how much time can be spent on
polishing code, so up to 10 new issues reported by Codacy/ruff is still an acceptable
amount.
However, a pull request where the CircleCI/pre-commit tests are failing cannot be merged.

#### NOTE
The automatic code quality checks by ruff are really helpful to
improve the quality of your code, but they are not flawless.
If you suspect ruff or Codacy may be wrong or you just need help,
please ask the [@ESMValGroup/tech-reviewers](https://github.com/orgs/ESMValGroup/teams/tech-reviewers) by commenting on your pull request.

### NCL

#### WARNING
Writing new NCL code is not recommended because the
[NCL interpreter](https://github.com/NCAR/ncl) is no longer maintained.

Because there is no standard best practices document for NCL, we use
[PEP8](https://www.python.org/dev/peps/pep-0008/) for NCL code as
well, with some minor adjustments to accommodate for differences in the
languages. The most important difference is that for NCL code the
indentation should be 2 spaces instead of 4.
Use the command `nclcodestyle /path/to/file.ncl` to check if your code
follows the style guide.
More information on the `nclcodestyle` command can be found
[here](../utils/utils.html.md#nclcodestyle).

### R

Best practices for R code are described in [The tidyverse style
guide](https://style.tidyverse.org/). We currently don’t check adherence
to this, but we strongly encourage any new R diagnostic to be checked against the
style guide by using [lintr](https://cran.r-project.org/web/packages/lintr/index.html).
Please use [styler](https://styler.r-lib.org/) to automatically format your code according
to this style guide. To run these tools, uncomment these lines:

```yaml
  # - repo: https://github.com/lorenzwalthert/precommit/  # Checks for R
  #   rev: 'v0.4.2'
  #   hooks:
  #     - id: style-files  # styler
  #     - id: lintr
```

and run the commands:

```bash
pre-commit run style-files --files esmvaltool/diag_scripts/your_script.R
pre-commit run lintr --files esmvaltool/diag_scripts/your_script.R
```

These pre-commit hooks have been disabled by default because over the past few
years all diagnostics that have been contributed were written in Python.
However, if R contributions pick up again, we would be happy to enable these
pre-commit hooks again.
In the future we would also like to make use of
[goodpractice](https://cran.r-project.org/web/packages/goodpractice/index.html)
to assess the quality of R code.

### YAML

Please use [yamllint](https://yamllint.readthedocs.io) to check that your
YAML files do not contain mistakes.
`yamllint` checks for valid syntax, common mistakes like key repetition and
cosmetic problems such as line length, trailing spaces, wrong indentation, etc.
When the tool complains about the maximum line length or too many spaces, please
use your own best judgement about whether solving the issue will make your
recipe more readable.

### Any text file

A generic tool to check for common spelling mistakes is
[codespell](https://pypi.org/project/codespell/).

<a id="documentation"></a>

## Documentation

The documentation lives on [docs.esmvaltool.org](https://docs.esmvaltool.org)
and is built using [Sphinx](https://www.sphinx-doc.org).
There are two main ways of adding documentation:

1. As written text in the directory
   [doc/sphinx/source](https://github.com/ESMValGroup/ESMValTool/tree/main/doc/sphinx/source).
   When writing
   [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html)
   (`.rst`) files, please try to limit the line length to 80 characters and
   always start a sentence on a new line.
   This makes it easier to review changes to documentation on GitHub.
2. As docstrings or comments in code.
   For Python code, the
   [docstrings](https://www.python.org/dev/peps/pep-0257/)
   of Python modules, classes, and functions
   that are mentioned in
   [doc/sphinx/source/api](https://github.com/ESMValGroup/ESMValTool/tree/main/doc/sphinx/source/api)
   are used to generate documentation.
   This results in the [ESMValTool Code API Documentation](../api/esmvaltool.html.md#api).

<a id="doc-howto"></a>

### What should be documented

See also [Recipe and diagnostic documentation](diagnostic.html.md#recipe-documentation) and [Dataset documentation](dataset.html.md#dataset-documentation).

Any code documentation that is visible on [docs.esmvaltool.org](https://docs.esmvaltool.org)
should be well written and adhere to the standards for documentation for the
respective language.
Note that there is no need to write extensive documentation for functions that
are not visible in the online documentation.
However, a short description in the docstring helps other contributors to
understand what a function is intended to do and and what its capabilities are.
For short functions, a one-line docstring is usually sufficient, but more
complex functions might require slightly more extensive documentation.

### How to build and view the documentation

Whenever you make a pull request or push new commits to an existing pull
request, [readthedocs](https://app.readthedocs.org/projects/esmvaltool/) will
automatically build the documentation.
The link to the documentation will be shown in the list of checks below your
pull request, click ‘Details’ behind the check
`docs/readthedocs.org:esmvaltool` to preview the documentation.
If all checks were successful, you may need to click ‘Show all checks’ to see
the individual checks.

To build the documentation on your own computer, go to the directory where the
repository was cloned and run

```default
sphinx-build doc/sphinx/source/ doc/sphinx/build/
```

or

```default
sphinx-build -Ea doc/sphinx/source/ doc/sphinx/build/
```

to build it from scratch.

Alternatively, the pixi task `doc` can be used:

```bash
pixi run doc
```

Make sure that your newly added documentation builds without warnings or
errors and looks correctly formatted.
[readthedocs](https://app.readthedocs.org/projects/esmvaltool/) will build the documentation with the command

```bash
sphinx-build -W doc/sphinx/source/ doc/sphinx/build/
```

to catch mistakes that can be detected automatically.

The configuration file for [Sphinx](https://www.sphinx-doc.org) is
[doc/sphinx/source/conf.py](https://github.com/ESMValGroup/ESMValTool/blob/main/doc/sphinx/source/conf.py)
and the configuration file for ReadTheDocs is
[.readthedocs.yaml](https://github.com/ESMValGroup/ESMValTool/blob/main/.readthedocs.yaml).

When reviewing a pull request, always check that the documentation checks
shown below the pull request were successful.
Successful checks have a green ✓ in front, a ❌ means the test job failed.

<a id="esmvalcore-documentation-integration"></a>

### Integration with the ESMValCore documentation

The [ESMValCore documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/index.html)
is hosted as a
[subproject](https://docs.readthedocs.io/en/stable/subprojects.html)
of the ESMValTool documentation on readthedocs.
To link to a section from the ESMValCore documentation from the reStructuredText
(`.rst`) files, use the usual `:ref:` but prefix the reference with
`esmvalcore:`.
For example, `:ref:`esmvalcore:recipe`` to link to
[The recipe format](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/index.html#recipe).

<a id="tests"></a>

## Tests

To check various aspects of the recipes and code, there tests available in the
[tests](https://github.com/ESMValGroup/ESMValTool/tree/main/tests)
directory.

Whenever you make a pull request or push new commits to an existing pull
request, these tests will be run automatically on [CircleCI](https://app.circleci.com/pipelines/github/ESMValGroup/ESMValTool).
The results appear at the bottom of the pull request.
Click on ‘Details’ for more information on a specific test job.
To see some of the results on CircleCI, you may need to log in.
You can do so using your GitHub account.

To run the tests on your own computer, go to the directory where the repository
is cloned and run the command `pytest` or `pixi run test`.

Have a look at [Testing recipes](diagnostic.html.md#testing-recipes) for information on testing recipes.

Every night, more extensive tests are run to make sure that problems with the
installation of the tool are discovered by the development team before users
encounter them.
These nightly tests have been designed to mimic the installation procedures
described in the documentation, e.g. in the [Installation](../quickstart/installation.html.md#install) chapter.
The nightly tests are run using both CircleCI and GitHub Actions, the
result of the tests ran by CircleCI can be seen on the
[CircleCI project page](https://app.circleci.com/pipelines/github/ESMValGroup/ESMValTool?branch=main)
and the result of the tests ran by GitHub Actions can be viewed on the
[Actions tab](https://github.com/ESMValGroup/ESMValTool/actions)
of the repository.

The configuration of the tests run by CircleCI can be found in the directory
[.circleci](https://github.com/ESMValGroup/ESMValTool/blob/main/.circleci),
while the configuration of the tests run by GitHub Actions can be found in the
directory
[.github/workflows](https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows).

When reviewing a pull request, always check that all test jobs on [CircleCI](https://app.circleci.com/pipelines/github/ESMValGroup/ESMValTool) were
successful.
Successful test jobs have a green ✓ in front, a ❌ means the test job failed.

<a id="authors"></a>

## List of authors

If you make a contribution to ESMValTool and you would like to be listed as an
author (e.g. on [Zenodo](https://zenodo.org/record/4562215)), please add your
name to the list of authors in `CITATION.cff` and generate the entry for the
`.zenodo.json` file by running the commands

```default
pip install cffconvert
cffconvert --infile CITATION.cff --format zenodo --outfile .zenodo.json
```

Presently, this method unfortunately discards entries communities
and grants from that file; please restore them manually.

Note that authors of recipes and/or diagnostics also need to be added to the file
[esmvaltool/config-references.yml](https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/config-references.yml),
see [Recording provenance](diagnostic.html.md#recording-provenance) for more information.

<a id="dependencies"></a>

## Dependencies

We follow the guidelines set out in [SPEC-0](https://scientific-python.org/specs/spec-0000/)
regarding the supported Python versions and dependencies.

Before considering adding a new dependency, carefully check that the
[license](https://book.the-turing-way.org/reproducible-research/licensing/)
of the dependency you want to add and any of its dependencies are
[compatible](https://book.the-turing-way.org/reproducible-research/licensing/licensing-compatibility/)
with the
[Apache 2.0](https://github.com/ESMValGroup/ESMValTool/blob/main/LICENSE/)
license that applies to the ESMValTool.
Permissively licensed dependencies (e.g. MIT, BSD, Apache 2.0) can be added,
as can dependencies licensed under a weak copyleft license such as the LGPL or
MPL 2.0.
Because the R software ecosystem largely consists of packages licensed under
the GPL family of licenses, R dependencies licensed under the GPL version 3 or
the GPL version 2 or later can also be added.
The combination of ESMValTool with a GPL-licensed dependency can only be
distributed under the terms of the GPL version 3.
Do not add dependencies licensed under the GPL version 2 only, because the Free
Software Foundation considers it incompatible with Apache 2.0, or under
licenses that are not open source, for example because they restrict
commercial use or redistribution.
See this [statement](https://www.apache.org/licenses/GPL-compatibility.html)
by the authors of the Apache 2.0 license for more information.
Please note that, for historical reasons, some existing functionality relies on
dependencies that do not meet these requirements, including dependencies
licensed under the GPL version 2 only and dependencies with licenses that
restrict commercial use or redistribution.

When adding or removing dependencies, please consider applying the changes in
the following two locations in `pyproject.toml`:

- `[project.optional-dependencies]`
  the `all` item contains all dependencies needed to run the diagnostics
  and CMORizer scripts, with the names used on [PyPI](https://pypi.org/).
- `[tool.pixi.feature.all]`
  the same list of dependencies as under `[project.optional-dependencies]`,
  but with the names used on [conda-forge](https://conda-forge.org/).

It is strongly preferred that those two lists are kept in sync, apart from
differences in how packages are named. Run the command `pixi lock --no-install`
after making changes to the dependencies to update the `pixi.lock` file, which
is used to make sure that the same versions of packages are installed for
all ESMValTool developers.

There is also a minimal set of dependencies that are required to run the example
recipe `examples/recipe_python.yml`. Please do not add new dependencies to
this list unless they are really needed to support the core functionality of
ESMValTool. These can be used by users who want to run ESMValTool with a minimal
set of dependencies and want to manage which dependencies of diagnostics and/or
CMORizer scripts get installed themselves. The minimal dependencies are listed
in the following two locations in `pyproject.toml`:

- `dependencies`
  contains a minimal set of dependencies, those required to run the example
  recipe, with the names used on [PyPI](https://pypi.org/)
- `[tool.pixi.dependencies]`
  the same list of dependencies as under `dependencies`, but with the names
  used on [conda-forge](https://conda-forge.org/)

It is strongly preferred that those two lists are kept in sync, apart from
differences in how packages are named.

In addition to the core dependencies mentioned above, there are further optional
dependencies for diagnostics that need R or NCL. These are listed in the sections
`[tool.pixi.feature.r]` and `[tool.pixi.feature.ncl]` respectively.

There are also three feature groups for development dependencies:

- `[tool.pixi.feature.dev]` contains tools that are useful for development
- `[tool.pixi.feature.doc]` contains tools that are needed to build the documentation
- `[tool.pixi.feature.test]` contains tools that are needed to run the tests

Note that packages may have a different name on
[conda-forge](https://conda-forge.org/) than on PyPI or CRAN.

#### TIP
When reviewing a pull request where dependencies are added or removed, always
check that the changes have been applied to both the PyPI and the conda-forge
dependencies and that the `pixi.lock` file has been updated by running
`pixi lock` after the changes were made.

<a id="pull-request-checks"></a>

## Pull request checks

To check that a pull request is up to standard, several automatic checks are
run when you make a pull request.
Read more about it in the [Tests]() and [Documentation]() sections.
Successful checks have a green ✓ in front, a ❌ means the check failed.

If you need help with the checks, please ask the technical reviewer of your pull
request for help.
Ask [@ESMValGroup/tech-reviewers](https://github.com/orgs/ESMValGroup/teams/tech-reviewers) if you do not have a technical reviewer yet.

If the checks are broken because of something unrelated to the current
pull request, please check if there is an open issue that reports the problem
and create one if there is no issue yet.
You can attract the attention of the [@ESMValGroup/esmvaltool-coreteam](https://github.com/orgs/ESMValGroup/teams/esmvaltool-coreteam) by
mentioning them in the issue if it looks like no-one is working on solving the
problem yet.
The issue needs to be fixed in a separate pull request first.
After that has been merged into the `main` branch and all checks are green
again on the `main` branch, merge it into your own branch to get the tests
to pass.

When reviewing a pull request, always make sure that all checks were successful.
If the Codacy check keeps failing, please run prospector locally.
If necessary, ask the pull request author to do the same and to address the
reported issues.
See the section on [code_quality]() for more information.
Never merge a pull request with failing CircleCI or readthedocs checks.
