D4Science
/
d4science_copernicus_notebooks
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

shared download implementation

This commit is contained in:
Alfredo Oliviero 2024-11-07 17:33:51 +01:00
parent 11f338495e
commit 29361b8b6b
18 changed files with 1701 additions and 3077 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -1,5 +1,6 @@
# Created by https://www.toptal.com/developers/gitignore/api/python,jupyternotebooks,macos,visualstudiocode,emacs # Created by https://www.toptal.com/developers/gitignore/api/python,jupyternotebooks,macos,visualstudiocode,emacs
# Edit at https://www.toptal.com/developers/gitignore?templates=python,jupyternotebooks,macos,visualstudiocode,emacs # Edit at https://www.toptal.com/developers/gitignore?templates=python,jupyternotebooks,macos,visualstudiocode,emacs
tutorials/data
### Emacs ### ### Emacs ###
# -*- mode: gitignore; -*- # -*- mode: gitignore; -*-

105
README.md
View File

@ -16,7 +16,7 @@ Locally clone the repository and copy them in your JupyterLab instance.
## Built With ## Built With
* [Copernicus CDSAPI](https://cds.climate.copernicus.eu/how-to-api/) - the Copernicus Climate Data Store (CDS) Application Program Interface (API) client * [Copernicus CDSAPI](https://cds.climate.copernicus.eu/how-to-api/) * the Copernicus Climate Data Store (CDS) Application Program Interface (API) client
* [python](https://python.org/) * [python](https://python.org/)
* [d4science](https://www.d4science.org/) * [d4science](https://www.d4science.org/)
* [d4science_copernicus_cds](https://code-repo.d4science.org/D4Science/d4science_copernicus_cds) * [d4science_copernicus_cds](https://code-repo.d4science.org/D4Science/d4science_copernicus_cds)
@ -32,27 +32,27 @@ These notebooks have been fixed (the official ones do not work with the new Cope
To test the notebooks in the D4Science JupyterLab environment, follow these steps: To test the notebooks in the D4Science JupyterLab environment, follow these steps:
1. **Access D4Science JupyterLab** 1. **Access D4Science JupyterLab**
- Log in to the D4Science portal with your credentials. * Log in to the D4Science portal with your credentials.
- Navigate to the JupyterLab section. * Navigate to the JupyterLab section.
- If existing, select the specific VM for Copernicus to have the dependencies pre-installed. * If existing, select the specific VM for Copernicus to have the dependencies pre-installed.
2. **Upload Notebooks** 2. **Upload Notebooks**
- Upload the tutorial notebooks to your JupyterLab workspace. * Upload the tutorial notebooks to your JupyterLab workspace.
3. **Install Required Libraries** 3. **Install Required Libraries**
- If not using the specific VM, open a terminal within JupyterLab. * If not using the specific VM, open a terminal within JupyterLab.
- Install the required libraries by running: * Install the required libraries by running:
```sh ```sh
pip install -r requirements_tutorial.txt pip install -r requirements_tutorial.txt
``` ```
4. **Configure CDS API Key** 4. **Configure CDS API Key**
- Open and run the `config_auth_cds.ipynb` notebook. * Open and run the `config_auth_cds.ipynb` notebook.
- Follow the instructions to configure your CDS API key. * Follow the instructions to configure your CDS API key.
5. **Run the Notebooks** 5. **Run the Notebooks**
- Open the tutorial notebooks in JupyterLab. * Open the tutorial notebooks in JupyterLab.
- Execute the cells to run the tutorials. * Execute the cells to run the tutorials.
Alternatively, you can execute the following command inside a notebook to install the required libraries: Alternatively, you can execute the following command inside a notebook to install the required libraries:
```sh ```sh
@ -63,49 +63,58 @@ Alternatively, you can execute the following command inside a notebook to instal
To test the notebooks locally on Visual Studio Code, follow these steps: To test the notebooks locally on Visual Studio Code, follow these steps:
1. **Install Visual Studio Code** ### 1. prerequisites
- Download and install Visual Studio Code from [here](https://code.visualstudio.com/).
2. **Install Necessary Extensions for Notebooks** * **pyenv**
- Open Visual Studio Code. * macos:
- Go to the Extensions view by clicking on the Extensions icon in the Activity Bar on the side of the window or by pressing `Ctrl+Shift+X`.
- Search for and install the following extensions:
- Python
- Jupyter
3. **Create and Activate a Virtual Environment** ```sh
- Open a terminal in Visual Studio Code by selecting `Terminal` > `New Terminal` from the top menu. brew install pyenv
- Create a virtual environment by running: brew install pyenv-virtualenv
```sh ```
python -m venv venv
```
- Activate the virtual environment:
- On Windows:
```sh
.\venv\Scripts\activate
```
- On macOS and Linux:
```sh
source venv/bin/activate
```
4. **Install Requirements** * **Visual Studio Code**
- Install the [required packages for tutorials](./requirements_tutorial.txt) by running: * * Download and install Visual Studio Code from [here](https://code.visualstudio.com/).
* Visual studio extensions for python and Jupyter Notebooks
* Open Visual Studio Code.
* Go to the Extensions view by clicking on the Extensions icon in the Activity Bar on the side of the window or by pressing `Ctrl+Shift+X`.
* Search for and install the following extensions:
* Python
* Jupyter
### 2. create a virtualenv
```sh
pyenv install 3.10.12
pyenv virtualenv 3.10.12 venv
pyenv activate venv
```
1. **Install Necessary Extensions for Notebooks**
* Open Visual Studio Code.
* Go to the Extensions view by clicking on the Extensions icon in the Activity Bar on the side of the window or by pressing `Ctrl+Shift+X`.
* Search for and install the following extensions:
* Python
* Jupyter
2. **Install Requirements**
* Install the [required packages for tutorials](./requirements_tutorial.txt) by running:
```sh ```sh
pip install -r requirements_tutorial.txt pip install -r requirements_tutorial.txt
``` ```
5. **Register on Copernicus Climate Data Store** 3. **Register on Copernicus Climate Data Store**
- Go to [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). * Go to [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/).
- Register for an account and create an API key. * Register for an account and create an API key.
6. **Open and Run `config_auth_cds.ipynb`** 4. **Open and Run `config_auth_cds.ipynb`**
- In Visual Studio Code, open and exec the [config_auth_cds.ipynb](./config_auth_cds.ipynb) notebook. * In Visual Studio Code, open and exec the [config_auth_cds.ipynb](./config_auth_cds.ipynb) notebook.
- Follow the instructions in the notebook to configure your CDS API key. * Follow the instructions in the notebook to configure your CDS API key.
7. **Run the Tutorial Notebooks** 5. **Run the Tutorial Notebooks**
- Open the [tutorial notebooks](./tutorials/) in Visual Studio Code. * Open the [tutorial notebooks](./tutorials/) in Visual Studio Code.
- Run the cells in each notebook to execute the tutorials. * Run the cells in each notebook to execute the tutorials.
## Change log ## Change log
@ -113,15 +122,15 @@ See [CHANGELOG.md](./CHANGELOG.md)
## Authors ## Authors
* **Alfredo Oliviero** ([ORCID]( https://orcid.org/0009-0007-3191-1025)) - [ISTI-CNR Infrascience Group](https://www.isti.cnr.it/People/A.Oliviero) * **Alfredo Oliviero** ([ORCID]( https://orcid.org/0009-0007-3191-1025)) * [ISTI-CNR Infrascience Group](https://www.isti.cnr.it/People/A.Oliviero)
## Maintainers ## Maintainers
* **Alfredo Oliviero** ([ORCID]( https://orcid.org/0009-0007-3191-1025)) - [ISTI-CNR Infrascience Group](https://www.isti.cnr.it/People/A.Oliviero) * **Alfredo Oliviero** ([ORCID]( https://orcid.org/0009-0007-3191-1025)) * [ISTI-CNR Infrascience Group](https://www.isti.cnr.it/People/A.Oliviero)
## License ## License
This project is licensed under the EUPL V.1.1 License - see the [LICENSE.md](./LICENSE.md) file for details. This project is licensed under the EUPL V.1.1 License * see the [LICENSE.md](./LICENSE.md) file for details.
## About the gCube Framework ## About the gCube Framework

240
tutorials/01_reanalysis/01x01_reanalysis-climatology.ipynb
View File

@ -22,6 +22,22 @@
"5. View time series and analyse trends" "5. View time series and analyse trends"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"01x01_reanalysis-climatology\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -56,7 +72,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -83,7 +99,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"cds_datadir will create a folder in our workspace, under cds_dataDir, with current timestamp and custom label" "cds_datadir will create a folder in our workspace, under DATADIR, with current timestamp and custom label"
] ]
}, },
{ {
@ -92,10 +108,37 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"reanalisys_climatology\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)\n",
"CDS_DATA"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -122,7 +165,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -216,42 +259,44 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client(url=URL, key=KEY)\n", "c = cdsapi.Client()"
"c.retrieve(\n", ]
" 'reanalysis-era5-single-levels-monthly-means',\n", },
" {\n", {
" 'product_type': 'monthly_averaged_reanalysis',\n", "cell_type": "code",
" 'variable': '2m_temperature',\n", "execution_count": null,
" 'year': [\n", "metadata": {},
" '1979', '1980', '1981',\n", "outputs": [],
" '1982', '1983', '1984',\n", "source": [
" '1985', '1986', '1987',\n", "\n",
" '1988', '1989', '1990',\n", "# we generate years in a smarter way than the original tutorial\n",
" '1991', '1992', '1993',\n", "start_year = 1979\n",
" '1994', '1995', '1996',\n", "end_year = 2020\n",
" '1997', '1998', '1999',\n", "years = ['%02d' % (y) for y in range(start_year, end_year +1)]\n",
" '2000', '2001', '2002',\n", "months = ['%02d' % (mnth) for mnth in range(1, 13)]\n",
" '2003', '2004', '2005',\n", "\n",
" '2006', '2007', '2008',\n", "# t2m = os.path.join(DATADIR, 'era5_monthly_t2m_eur.nc')\n",
" '2009', '2010', '2011',\n", "file_t2m = os.path.join(CDS_DATA, 'era5_monthly_t2m_eur_%d_%d.nc' % (start_year, end_year))\n",
" '2012', '2013', '2014',\n", "\n",
" '2015', '2016', '2017',\n", "area = [ 72, -25, 34, 40,] # maxlat, minlon, minlat, maxlon\n",
" '2018', '2019', '2020',\n", "\n",
" ],\n", "if not os.path.exists(file_t2m):\n",
" 'month': [\n", " print(f'Downloading {file_t2m}')\n",
" '01', '02', '03',\n", " c.retrieve(\n",
" '04', '05', '06',\n", " 'reanalysis-era5-single-levels-monthly-means',\n",
" '07', '08', '09',\n", " {\n",
" '10', '11', '12',\n", " 'product_type': 'monthly_averaged_reanalysis',\n",
" ],\n", " 'variable': '2m_temperature',\n",
" 'time': '00:00',\n", " 'year': years,\n",
" 'area': [\n", " 'month': months,\n",
" 72, -25, 34,\n", " 'time': '00:00',\n",
" 40,\n", " 'area': area,\n",
" ],\n", " 'data_format': 'netcdf_legacy',\n",
" 'data_format': 'netcdf_legacy',\n", " },\n",
" },\n", " file_t2m\n",
" f'{DATADIR}era5_monthly_t2m_eur.nc')" " )\n",
"else:\n",
" print(f'{file_t2m} already exists')"
] ]
}, },
{ {
@ -263,15 +308,6 @@
"Now that we have downloaded the data, we can inspect it. We have requested the data in NetCDF format. This is a commonly used format for array-oriented scientific data. To read and process this data we will make use of the [Xarray](http://xarray.pydata.org/en/stable/) library. Xarray is an open source project and Python package that makes working with labelled multi-dimensional arrays simple and efficient. We will read the data from our NetCDF file into an [xarray.Dataset](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.html)." "Now that we have downloaded the data, we can inspect it. We have requested the data in NetCDF format. This is a commonly used format for array-oriented scientific data. To read and process this data we will make use of the [Xarray](http://xarray.pydata.org/en/stable/) library. Xarray is an open source project and Python package that makes working with labelled multi-dimensional arrays simple and efficient. We will read the data from our NetCDF file into an [xarray.Dataset](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.html)."
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"t2m = f'{DATADIR}era5_monthly_t2m_eur.nc'"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
@ -279,7 +315,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# Create Xarray Dataset\n", "# Create Xarray Dataset\n",
"ds = xr.open_dataset(t2m)" "ds = xr.open_dataset(file_t2m)"
] ]
}, },
{ {
@ -679,7 +715,9 @@
"ax.legend(handles, labels)\n", "ax.legend(handles, labels)\n",
"ax.grid(linestyle='--')\n", "ax.grid(linestyle='--')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}Eur_monthly_t2m_clim.png')" "# fig.savefig(f'{DATADIR}Eur_monthly_t2m_clim.png')\n",
"path_image = os.path.join(DATADIR, 'Eur_monthly_t2m_clim.png')\n",
"fig.savefig(path_image)"
] ]
}, },
{ {
@ -743,7 +781,10 @@
"cbar = plt.colorbar(im,fraction=0.05, pad=0.04)\n", "cbar = plt.colorbar(im,fraction=0.05, pad=0.04)\n",
"cbar.set_label('temperature anomaly') \n", "cbar.set_label('temperature anomaly') \n",
"\n", "\n",
"fig.savefig(f'{DATADIR}ERA5_Europe_2020-08_anomaly.png')" "# fig.savefig(f'{DATADIR}ERA5_Europe_2020-08_anomaly.png')\n",
"path_image = os.path.join(DATADIR\n",
" , 'ERA5_Europe_2020-08_anomaly.png')\n",
"fig.savefig(path_image)"
] ]
}, },
{ {
@ -763,41 +804,26 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c.retrieve(\n", "arc_file = os.path.join(CDS_DATA, 'era5_monthly_t2m_Arc_%d_%d.nc' % (start_year, end_year))\n",
" 'reanalysis-era5-single-levels-monthly-means',\n", "\n",
" {\n", "area = [ 90, -180, 66.55, 180,]\n",
" 'product_type': 'monthly_averaged_reanalysis',\n", "\n",
" 'variable': '2m_temperature',\n", "if not os.path.exists(arc_file):\n",
" 'year': [\n", " print(f'Downloading {arc_file}')\n",
" '1979', '1980', '1981',\n", " c.retrieve(\n",
" '1982', '1983', '1984',\n", " 'reanalysis-era5-single-levels-monthly-means',\n",
" '1985', '1986', '1987',\n", " {\n",
" '1988', '1989', '1990',\n", " 'product_type': 'monthly_averaged_reanalysis',\n",
" '1991', '1992', '1993',\n", " 'variable': '2m_temperature',\n",
" '1994', '1995', '1996',\n", " 'year': years,\n",
" '1997', '1998', '1999',\n", " 'month': months,\n",
" '2000', '2001', '2002',\n", " 'time': '00:00',\n",
" '2003', '2004', '2005',\n", " 'area': area,\n",
" '2006', '2007', '2008',\n", " 'data_format': 'netcdf_legacy',\n",
" '2009', '2010', '2011',\n", " },\n",
" '2012', '2013', '2014',\n", " arc_file)\n",
" '2015', '2016', '2017',\n", "else:\n",
" '2018', '2019', '2020',\n", " print(f'{arc_file} already exists')"
" ],\n",
" 'month': [\n",
" '01', '02', '03',\n",
" '04', '05', '06',\n",
" '07', '08', '09',\n",
" '10', '11', '12',\n",
" ],\n",
" 'time': '00:00',\n",
" 'area': [\n",
" 90, -180, 66.55,\n",
" 180,\n",
" ],\n",
" 'data_format': 'netcdf_legacy',\n",
" },\n",
" f'{DATADIR}era5_monthly_t2m_Arc.nc')"
] ]
}, },
{ {
@ -813,8 +839,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"Arc_file = f'{DATADIR}era5_monthly_t2m_Arc.nc'\n", "Arc_ds = xr.open_dataset(arc_file)\n",
"Arc_ds = xr.open_dataset(Arc_file)\n",
"Arc_da = Arc_ds['t2m']" "Arc_da = Arc_ds['t2m']"
] ]
}, },
@ -860,7 +885,9 @@
"cbar = fig.colorbar(im, fraction=0.04, pad=0.07)\n", "cbar = fig.colorbar(im, fraction=0.04, pad=0.07)\n",
"cbar.set_label('° C')\n", "cbar.set_label('° C')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}ERA5_Arctic_t2m_Jan1979.png')" "# fig.savefig(f'{DATADIR}ERA5_Arctic_t2m_Jan1979.png')\n",
"path_image = os.path.join(DATADIR, 'ERA5_Arctic_t2m_Jan1979.png')\n",
"fig.savefig(path_image)\n"
] ]
}, },
{ {
@ -1008,7 +1035,9 @@
" ylabel='Temperature Anomaly',\n", " ylabel='Temperature Anomaly',\n",
" title='\\nYearly anomalies of Arctic air temperature (reference period 1991-2020)\\n',\n", " title='\\nYearly anomalies of Arctic air temperature (reference period 1991-2020)\\n',\n",
" )\n", " )\n",
"plt.savefig(f'{DATADIR}Arctic_t2m_anom_bar.png')" "# plt.savefig(f'{DATADIR}Arctic_t2m_anom_bar.png')\n",
"path_image = os.path.join(DATADIR, 'Arctic_t2m_anom_bar.png')\n",
"plt.savefig(path_image)\n"
] ]
}, },
{ {
@ -1065,7 +1094,9 @@
"fig.suptitle('\\nArctic seasonal averages of 2m air temp - 1979-2020\\n', fontsize=16)\n", "fig.suptitle('\\nArctic seasonal averages of 2m air temp - 1979-2020\\n', fontsize=16)\n",
"fig.tight_layout(pad=1.0)\n", "fig.tight_layout(pad=1.0)\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}Arctic_seasonal.png')" "# fig.savefig(f'{DATADIR}Arctic_seasonal.png')\n",
"path_image = os.path.join(DATADIR, 'Arctic_seasonal.png')\n",
"fig.savefig(path_image)"
] ]
}, },
{ {
@ -1074,6 +1105,29 @@
"source": [ "source": [
"Note the difference in variability of the seasonal average of air temperature in the Arctic: mean summer temperatures seem to be more constant compared to the other seasons." "Note the difference in variability of the seasonal average of air temperature in the Arctic: mean summer temperatures seem to be more constant compared to the other seasons."
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -1092,7 +1146,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

343
tutorials/01_reanalysis/01x02_reanalysis-temp-record.ipynb
View File

@ -8,6 +8,16 @@
"# Tutorial on July 2023 record-breaking global surface temperatures using climate data from C3S" "# Tutorial on July 2023 record-breaking global surface temperatures using climate data from C3S"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"id": "17f078b3",
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"01x02_reanalysis-temp-record\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "d3cbbf9e", "id": "d3cbbf9e",
@ -66,7 +76,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -88,16 +98,16 @@
"source": [ "source": [
"URL, KEY = cds_get_credentials()\n", "URL, KEY = cds_get_credentials()\n",
"print(\"URL\", URL)\n", "print(\"URL\", URL)\n",
"print (\"KEY\", KEY)\n", "print(\"KEY\", KEY)\n",
"\n", "\n",
"APIKEY = KEY # this tutorial uses a different variable name...\n", "APIKEY = KEY # this tutorial uses a different variable name...\n",
"\n", "\n",
"print (\"APIKEY\", APIKEY)" "print(\"APIKEY\", APIKEY)"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ae061938", "id": "e7da67c5",
"metadata": {}, "metadata": {},
"source": [ "source": [
"cds_datadir will create a folder in our workspace, under cds_dataDir, with current timestamp and custom label" "cds_datadir will create a folder in our workspace, under cds_dataDir, with current timestamp and custom label"
@ -106,14 +116,43 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "1b982ef2", "id": "68bc5703",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"reanalysis_temp-record\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"id": "1c6498bf",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4263a56f",
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"id": "0586a17a",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "91801237", "id": "91801237",
@ -259,14 +298,14 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"import cdsapi\n",
"import cartopy.crs as ccrs\n",
"import matplotlib.pyplot as plt\n",
"import xarray as xr\n",
"import numpy as np\n",
"import urllib3\n", "import urllib3\n",
"urllib3.disable_warnings()\n", "urllib3.disable_warnings()\n",
"\n", "\n",
"import numpy as np\n",
"import xarray as xr\n",
"import matplotlib.pyplot as plt\n",
"import cartopy.crs as ccrs\n",
"import cdsapi\n",
"\n", "\n",
"plt.style.use('bmh')" "plt.style.use('bmh')"
] ]
@ -300,14 +339,6 @@
"At the end of the download form, select **\"Show API request\"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below) ..." "At the end of the download form, select **\"Show API request\"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below) ..."
] ]
}, },
{
"cell_type": "markdown",
"id": "d1d8042c",
"metadata": {},
"source": [
"<center><img src=\"img/reanalysis-temp-record-001.png\" /></center>"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "1925d01d", "id": "1925d01d",
@ -325,49 +356,45 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client(url=URL, key=APIKEY)\n", "c = cdsapi.Client()"
"c.retrieve(\n", ]
" 'reanalysis-era5-single-levels-monthly-means',\n", },
" {\n", {
" 'data_format': 'netcdf_legacy',\n", "cell_type": "code",
" 'product_type': 'monthly_averaged_reanalysis',\n", "execution_count": null,
" 'variable': '2m_temperature',\n", "id": "c69c69cd",
" 'year': [\n", "metadata": {},
" '1940', '1941', '1942',\n", "outputs": [],
" '1943', '1944', '1945',\n", "source": [
" '1946', '1947', '1948',\n", "\n",
" '1949', '1950', '1951',\n", "# we generate years in a smarter way than the original tutorial\n",
" '1952', '1953', '1954',\n", "start_year = 1940\n",
" '1955', '1956', '1957',\n", "end_year = 2023\n",
" '1958', '1959', '1960',\n", "years = ['%02d' % (y) for y in range(start_year, end_year + 1)]\n",
" '1961', '1962', '1963',\n", "# months = ['%02d' % (mnth) for mnth in range(1, 13)]\n",
" '1964', '1965', '1966',\n", "area = [90, -180, -90, 180] # maxlat, minlon, minlat, maxlon\n",
" '1967', '1968', '1969',\n", "\n",
" '1970', '1971', '1972',\n", "# t2m = os.path.join(DATADIR, 'era5_monthly_t2m_eur.nc')\n",
" '1973', '1974', '1975',\n", "filename = os.path.join(\n",
" '1976', '1977', '1978',\n", " CDS_DATA, 't2m_global_july_%d-%d.nc' % (start_year, end_year))\n",
" '1979', '1980', '1981',\n", "\n",
" '1982', '1983', '1984',\n", "if not os.path.exists(filename):\n",
" '1985', '1986', '1987',\n", " print(\"Downloading \", filename)\n",
" '1988', '1989', '1990',\n", " c.retrieve(\n",
" '1991', '1992', '1993',\n", " 'reanalysis-era5-single-levels-monthly-means',\n",
" '1994', '1995', '1996',\n", " {\n",
" '1997', '1998', '1999',\n", " 'data_format': 'netcdf_legacy',\n",
" '2000', '2001', '2002',\n", " 'product_type': 'monthly_averaged_reanalysis',\n",
" '2003', '2004', '2005',\n", " 'variable': '2m_temperature',\n",
" '2006', '2007', '2008',\n", " 'year': years,\n",
" '2009', '2010', '2011',\n", " 'month': '07',\n",
" '2012', '2013', '2014',\n", " 'time': '00:00',\n",
" '2015', '2016', '2017',\n", " 'area': area, # maxlat, minlon, minlat, maxlon\n",
" '2018', '2019', '2020',\n", " },\n",
" '2021', '2022', '2023',\n", " filename\n",
" ],\n", " )\n",
" 'month': '07',\n", "else:\n",
" 'time': '00:00',\n", " print(\"File exists\", filename)"
" 'area': [90, -180, -90, 180], # maxlat, minlon, minlat, maxlon\n",
" },\n",
" f'{DATADIR}t2m_global_july_1940-2023.nc'\n",
")"
] ]
}, },
{ {
@ -387,7 +414,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"ds = xr.open_dataset(f'{DATADIR}t2m_global_july_1940-2023.nc')" "ds = xr.open_dataset(filename)"
] ]
}, },
{ {
@ -658,7 +685,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"t2m_ref_per = da_celsius.sel(time=slice('1991-01-01', '2020-12-31')).mean(dim='time')" "t2m_ref_per = da_celsius.sel(time=slice(\n",
" '1991-01-01', '2020-12-31')).mean(dim='time')"
] ]
}, },
{ {
@ -785,36 +813,38 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# create the figure panel and the map using the Cartopy PlateCarree projection\n", "# create the figure panel and the map using the Cartopy PlateCarree projection\n",
"fig, ax = plt.subplots(1, 1, figsize = (16, 8), subplot_kw={'projection': ccrs.PlateCarree()})\n", "fig, ax = plt.subplots(1, 1, figsize=(16, 8), subplot_kw={\n",
" 'projection': ccrs.PlateCarree()})\n",
"\n", "\n",
"# Plot the data\n", "# Plot the data\n",
"im = plt.pcolormesh(\n", "im = plt.pcolormesh(\n",
" anom.longitude, \n", " anom.longitude,\n",
" anom.latitude, \n", " anom.latitude,\n",
" anom, \n", " anom,\n",
" cmap='RdBu_r',\n", " cmap='RdBu_r',\n",
" vmin=-12, \n", " vmin=-12,\n",
" vmax=12\n", " vmax=12\n",
") \n", ")\n",
"\n", "\n",
"# Set the figure title, add lat/lon grid and coastlines\n", "# Set the figure title, add lat/lon grid and coastlines\n",
"ax.set_title('Temperature anomaly for July 2023 (with respect to 1991-2020 July mean)', fontsize=16)\n", "ax.set_title(\n",
" 'Temperature anomaly for July 2023 (with respect to 1991-2020 July mean)', fontsize=16)\n",
"ax.gridlines(\n", "ax.gridlines(\n",
" draw_labels=True, \n", " draw_labels=True,\n",
" linewidth=1, \n", " linewidth=1,\n",
" color='gray', \n", " color='gray',\n",
" alpha=0.5, \n", " alpha=0.5,\n",
" linestyle='--'\n", " linestyle='--'\n",
") \n", ")\n",
"ax.coastlines(color='black')\n", "ax.coastlines(color='black')\n",
"\n", "\n",
"# Specify the colorbar and set a label for the colorbar\n", "# Specify the colorbar and set a label for the colorbar\n",
"cbar = plt.colorbar(im, fraction=0.05, pad=0.04)\n", "cbar = plt.colorbar(im, fraction=0.05, pad=0.04)\n",
"cbar.set_label('Temperature anomaly') \n", "cbar.set_label('Temperature anomaly')\n",
"\n", "\n",
"# Show or save the figure, uncomment the line/s in case of need\n", "# Show or save the figure, uncomment the line/s in case of need\n",
"#fig.show() # not needed in a notebook inline context\n", "fig.show() \n",
"#fig.savefig('near_sfc_t2m_anomaly_july2023.png') # not needed in a notebook inline context" "fig.savefig(os.path.join(DATADIR, 'near_sfc_t2m_anomaly_july2023.png')) "
] ]
}, },
{ {
@ -904,7 +934,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"t2m_global_ref_per = t2m_global.sel(time=slice('1991-01-01', '2020-12-31')).mean(dim='time')\n", "t2m_global_ref_per = t2m_global.sel(time=slice(\n",
" '1991-01-01', '2020-12-31')).mean(dim='time')\n",
"print(f'{t2m_global_ref_per.values:.2f} \\N{DEGREE SIGN}C')" "print(f'{t2m_global_ref_per.values:.2f} \\N{DEGREE SIGN}C')"
] ]
}, },
@ -963,36 +994,36 @@
"\n", "\n",
"# we use the t2m_clim variable to plot a horizontal line\n", "# we use the t2m_clim variable to plot a horizontal line\n",
"# with the global climate normal value (1991-2020)\n", "# with the global climate normal value (1991-2020)\n",
"ax.axhline(y=t2m_clim, xmin = 0.05, xmax = 0.95, color='black')\n", "ax.axhline(y=t2m_clim, xmin=0.05, xmax=0.95, color='black')\n",
"\n", "\n",
"print (t2m_global.time)\n", "print(t2m_global.time)\n",
"# In order to highlight years above or below the global climate value\n", "# In order to highlight years above or below the global climate value\n",
"# we use two times the fill_between method that fills the area between two lines\n", "# we use two times the fill_between method that fills the area between two lines\n",
"# The first one is for values below the global climate normal in 'mediumslateblue' color\n", "# The first one is for values below the global climate normal in 'mediumslateblue' color\n",
"ax.fill_between(\n", "ax.fill_between(\n",
" global_time, # t2m_global.time, \n", " global_time, # t2m_global.time,\n",
" t2m_global, \n", " t2m_global,\n",
" t2m_clim, \n", " t2m_clim,\n",
" where=(t2m_global < t2m_clim), \n", " where=(t2m_global < t2m_clim),\n",
" color='mediumslateblue', \n", " color='mediumslateblue',\n",
" alpha=0.3,\n", " alpha=0.3,\n",
" interpolate=True\n", " interpolate=True\n",
")\n", ")\n",
"\n", "\n",
"# The second one is for values above the global climate normal in 'tomato' color\n", "# The second one is for values above the global climate normal in 'tomato' color\n",
"ax.fill_between(\n", "ax.fill_between(\n",
" global_time, # t2m_global.time, \n", " global_time, # t2m_global.time,\n",
" t2m_global, \n", " t2m_global,\n",
" t2m_clim, \n", " t2m_clim,\n",
" where=(t2m_global > t2m_clim), \n", " where=(t2m_global > t2m_clim),\n",
" color='tomato', \n", " color='tomato',\n",
" alpha=0.3,\n", " alpha=0.3,\n",
" interpolate=True\n", " interpolate=True\n",
")\n", ")\n",
"\n", "\n",
"# Show or save the figure, uncomment the line/s in case of need\n", "# Show or save the figure, uncomment the line/s in case of need\n",
"#fig.show() # not needed in a notebook inline context\n", "fig.show() \n",
"#fig.savefig('near_sfc_t2m_global_avg_july.png') # not needed in a notebook inline context" "fig.savefig(os.path.join(DATADIR, 'near_sfc_t2m_global_avg_july.png'))"
] ]
}, },
{ {
@ -1068,7 +1099,8 @@
"fig, ax = plt.subplots(figsize=(16, 6))\n", "fig, ax = plt.subplots(figsize=(16, 6))\n",
"\n", "\n",
"# We create a label for the 'x' axis using each year\n", "# We create a label for the 'x' axis using each year\n",
"xlabel = t2m_global_sorted.time.values.astype('datetime64[Y]').astype(int) + 1970 # years\n", "xlabel = t2m_global_sorted.time.values.astype(\n",
" 'datetime64[Y]').astype(int) + 1970 # years\n",
"\n", "\n",
"# We create also the location for each of the labels in the 'x' axis\n", "# We create also the location for each of the labels in the 'x' axis\n",
"xpos = np.arange(len(xlabel))\n", "xpos = np.arange(len(xlabel))\n",
@ -1080,7 +1112,7 @@
"ax.bar(xpos, y, color='tomato')\n", "ax.bar(xpos, y, color='tomato')\n",
"\n", "\n",
"# We place all the labels for the 'x' axis rotated 90º\n", "# We place all the labels for the 'x' axis rotated 90º\n",
"#ax.set_xticks(xpos, xlabel, rotation=90)\n", "# ax.set_xticks(xpos, xlabel, rotation=90)\n",
"ax.set_xticks(xpos)\n", "ax.set_xticks(xpos)\n",
"ax.set_xticklabels(xlabel, rotation=90)\n", "ax.set_xticklabels(xlabel, rotation=90)\n",
"\n", "\n",
@ -1094,16 +1126,16 @@
"ax.set_ylim(t2m_global_sorted.min() - 0.25, t2m_global_sorted.max() + 0.25)\n", "ax.set_ylim(t2m_global_sorted.min() - 0.25, t2m_global_sorted.max() + 0.25)\n",
"\n", "\n",
"# we add a horizontal line to plot the climate normal.\n", "# we add a horizontal line to plot the climate normal.\n",
"#ax.plot(range(len(clim_repeated)), clim_repeated, '-.', linewidth=0.5, color='black')\n", "# ax.plot(range(len(clim_repeated)), clim_repeated, '-.', linewidth=0.5, color='black')\n",
"ax.axhline(y=t2m_clim, xmin = 0.05, xmax = 0.95, color='black')\n", "ax.axhline(y=t2m_clim, xmin=0.05, xmax=0.95, color='black')\n",
"\n", "\n",
"# This function is very interesting as it does magic to prettify the final result :-)\n", "# This function is very interesting as it does magic to prettify the final result :-)\n",
"# https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.tight_layout.html\n", "# https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.tight_layout.html\n",
"fig.tight_layout()\n", "fig.tight_layout()\n",
"\n", "\n",
"# Show or save the figure, uncomment the line/s in case of need\n", "# Show or save the figure, uncomment the line/s in case of need\n",
"#fig.show() # not needed in a notebook inline context\n", "fig.show()\n",
"#fig.savefig('near_sfc_t2m_global_avg_july_sorted_barplot.png') # not needed in a notebook inline context" "fig.savefig(os.path.join(DATADIR, 'near_sfc_t2m_global_avg_july_sorted_barplot.png'))"
] ]
}, },
{ {
@ -1141,39 +1173,34 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c.retrieve(\n", "# we generate years in a smarter way than the original tutorial\n",
" 'reanalysis-era5-single-levels-monthly-means',\n", "start_year_reanalysys = 1991\n",
" {\n", "end_year_reanalysys = 2023\n",
" 'data_format': 'netcdf_legacy',\n", "\n",
" 'variable': 'sea_surface_temperature',\n", "years = ['%02d' % (y) for y in range(start_year_reanalysys, end_year_reanalysys +1)]\n",
" 'product_type': 'monthly_averaged_reanalysis',\n", "months = ['%02d' % (mnth) for mnth in range(1, 13)]\n",
" 'year': [\n", "area = [ 60, -40, 0, 0, ]\n",
" '1991', '1992', '1993',\n", "\n",
" '1994', '1995', '1996',\n", "# t2m = os.path.join(DATADIR, 'era5_monthly_t2m_eur.nc')\n",
" '1997', '1998', '1999',\n", "filename_reanalysis = os.path.join(CDS_DATA, 'sst_(NAtl)_monthly_%d-%d.nc' % (start_year_reanalysys, end_year_reanalysys))\n",
" '2000', '2001', '2002',\n", "\n",
" '2003', '2004', '2005',\n", "if not os.path.exists(filename_reanalysis):\n",
" '2006', '2007', '2008',\n", " print(\"Downloading \", filename_reanalysis)\n",
" '2009', '2010', '2011',\n", " c.retrieve(\n",
" '2012', '2013', '2014',\n", " 'reanalysis-era5-single-levels-monthly-means',\n",
" '2015', '2016', '2017',\n", " {\n",
" '2018', '2019', '2020',\n", " 'data_format': 'netcdf_legacy',\n",
" '2021', '2022', '2023',\n", " 'variable': 'sea_surface_temperature',\n",
" ],\n", " 'product_type': 'monthly_averaged_reanalysis',\n",
" 'month': [\n", " 'year': years,\n",
" '01', '02', '03',\n", " 'month': months,\n",
" '04', '05', '06',\n", " 'time': '00:00',\n",
" '07', '08', '09',\n", " 'area': area,\n",
" '10', '11', '12',\n", " },\n",
" ],\n", " filename_reanalysis\n",
" 'time': '00:00',\n", " )\n",
" 'area': [\n", "else:\n",
" 60, -40, 0,\n", " print(\"File exists\", filename_reanalysis)"
" 0,\n",
" ],\n",
" },\n",
" f'{DATADIR}sst_(NAtl)_monthly_1991-2023.nc'\n",
")"
] ]
}, },
{ {
@ -1191,7 +1218,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"sst_ds = xr.open_dataset(f'{DATADIR}sst_(NAtl)_monthly_1991-2023.nc')" "sst_ds = xr.open_dataset(filename_reanalysis)"
] ]
}, },
{ {
@ -1370,20 +1397,24 @@
"source": [ "source": [
"fig, ax = plt.subplots(figsize=(16, 6))\n", "fig, ax = plt.subplots(figsize=(16, 6))\n",
"\n", "\n",
"ax.set_facecolor('white') # For this plot we want a white axes background \n", "ax.set_facecolor('white') # For this plot we want a white axes background\n",
"\n", "\n",
"# Thin line for each year from 1991 to 2020\n", "# Thin line for each year from 1991 to 2020\n",
"for year in range(1991, 2021):\n", "for year in range(1991, 2021):\n",
" ax.plot(range(1, 13), sst_ts.sel(time=str(year)), color='lightgrey', lw=0.5)\n", " ax.plot(range(1, 13), sst_ts.sel(\n",
" time=str(year)), color='lightgrey', lw=0.5)\n",
"\n", "\n",
"# Climatological average\n", "# Climatological average\n",
"ax.plot(range(1, 13), avg_per_month_clim, '--', color='grey', lw=3, label='Clim')\n", "ax.plot(range(1, 13), avg_per_month_clim,\n",
" '--', color='grey', lw=3, label='Clim')\n",
"\n", "\n",
"# Monthly average for 2022\n", "# Monthly average for 2022\n",
"ax.plot(range(1, 13), sst_ts.sel(time='2022'), color='orange', lw=3, label='2022')\n", "ax.plot(range(1, 13), sst_ts.sel(time='2022'),\n",
" color='orange', lw=3, label='2022')\n",
"\n", "\n",
"# Monthly average for 2023\n", "# Monthly average for 2023\n",
"ax.plot(range(1, 13), sst_ts.sel(time='2023'), color='black', lw=5, label='2023')\n", "ax.plot(range(1, 13), sst_ts.sel(time='2023'),\n",
" color='black', lw=5, label='2023')\n",
"\n", "\n",
"# Customise the plot\n", "# Customise the plot\n",
"ax.set_xlabel('month')\n", "ax.set_xlabel('month')\n",
@ -1392,8 +1423,8 @@
"fig.tight_layout()\n", "fig.tight_layout()\n",
"\n", "\n",
"# Show or save the figure, uncomment the line/s in case of need\n", "# Show or save the figure, uncomment the line/s in case of need\n",
"#fig.show() # not needed in a notebook inline context\n", "fig.show() \n",
"#fig.savefig('SST_NorthAtlantic_avg.png') # not needed in a notebook inline context" "fig.savefig(os.path.join(DATADIR, 'SST_NorthAtlantic_avg.png')) "
] ]
}, },
{ {
@ -1403,6 +1434,32 @@
"source": [ "source": [
"Notice the dramatic increase in SST over the North Atlantic in 2023 compared to previous years!" "Notice the dramatic increase in SST over the North Atlantic in 2023 compared to previous years!"
] ]
},
{
"cell_type": "markdown",
"id": "f1a9f520",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "e5eeea37",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad3a4f61",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -1421,7 +1478,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

159
tutorials/01_reanalysis/01x03_reanalysis-heatwave.ipynb
View File

@ -7,6 +7,15 @@
"# Analysis of September 2020 European Heatwave using ERA5 Climate Reanalysis Data from C3S" "# Analysis of September 2020 European Heatwave using ERA5 Climate Reanalysis Data from C3S"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"01x03_reanalysis-heatwave\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -56,7 +65,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -94,10 +103,36 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"reanalysis_heatwave\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -124,7 +159,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -205,7 +240,6 @@
"- Geographical area: `North: 51`, `East: 4`, `South: 50`, `West: 3`\n", "- Geographical area: `North: 51`, `East: 4`, `South: 50`, `West: 3`\n",
"- Format: `NetCDF`\n", "- Format: `NetCDF`\n",
"\n", "\n",
"![logo](./img/Notebook3_data.png)\n",
"\n", "\n",
"At the end of the download form, select **\"Show API request\"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below) ...\n", "At the end of the download form, select **\"Show API request\"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below) ...\n",
"\n", "\n",
@ -220,58 +254,46 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client(url=URL, key=KEY)\n", "c = cdsapi.Client()"
"c.retrieve(\n", ]
" 'reanalysis-era5-single-levels',\n", },
" {\n", {
" 'product_type': 'reanalysis',\n", "cell_type": "code",
" 'data_format': 'netcdf_legacy',\n", "execution_count": null,
" 'variable': '2m_temperature',\n", "metadata": {},
" 'year': [\n", "outputs": [],
" '1979', '1980', '1981',\n", "source": [
" '1982', '1983', '1984',\n", "\n",
" '1985', '1986', '1987',\n", "# filename = f'{DATADIR}NFrance_hourly_Sep.nc'\n",
" '1988', '1989', '1990',\n", "\n",
" '1991', '1992', '1993',\n", "start_year = 1979\n",
" '1994', '1995', '1996',\n", "end_year = 2020\n",
" '1997', '1998', '1999',\n", "# if exists, we use our cached file\n",
" '2000', '2001', '2002',\n", "filename = os.path.join(CDS_DATA, f'NFrance_hourly_Sep_{start_year}-{end_year}.nc')\n",
" '2003', '2004', '2005',\n", "\n",
" '2006', '2007', '2008',\n", "years = [str(year) for year in range(start_year, end_year + 1)]\n",
" '2009', '2010', '2011',\n", "month = '09'\n",
" '2012', '2013', '2014',\n", "days = ['%2d' % day for day in range(1, 30 + 1)]\n",
" '2015', '2016', '2017',\n", "hours = ['%02d:00' % hour for hour in range(0, 23 + 1)]\n",
" '2018', '2019', '2020',\n", "area = [51, 3, 50, 4] \n",
" ],\n", "\n",
" 'month': '09',\n", "if not os.path.exists(filename):\n",
" 'day': [\n", " print(\"Downloading data\", filename)\n",
" '01', '02', '03',\n", " c.retrieve( #\n",
" '04', '05', '06',\n", " 'reanalysis-era5-single-levels',\n",
" '07', '08', '09',\n", " {\n",
" '10', '11', '12',\n", " 'product_type': 'reanalysis',\n",
" '13', '14', '15',\n", " 'data_format': 'netcdf_legacy',\n",
" '16', '17', '18',\n", " 'variable': '2m_temperature',\n",
" '19', '20', '21',\n", " 'year': years,\n",
" '22', '23', '24',\n", " 'month': month,\n",
" '25', '26', '27',\n", " 'day': days,\n",
" '28', '29', '30',\n", " 'time': hours,\n",
" ],\n", " 'area': area\n",
" 'time': [\n", " },\n",
" '00:00', '01:00', '02:00',\n", " filename)\n",
" '03:00', '04:00', '05:00',\n", "else:\n",
" '06:00', '07:00', '08:00',\n", " print(\"File exists\", filename)"
" '09:00', '10:00', '11:00',\n",
" '12:00', '13:00', '14:00',\n",
" '15:00', '16:00', '17:00',\n",
" '18:00', '19:00', '20:00',\n",
" '21:00', '22:00', '23:00',\n",
" ],\n",
" 'area': [\n",
" 51, 3, 50,\n",
" 4,\n",
" ],\n",
" },\n",
" f'{DATADIR}NFrance_hourly_Sep.nc')"
] ]
}, },
{ {
@ -289,7 +311,6 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"filename = f'{DATADIR}NFrance_hourly_Sep.nc'\n",
"# Create Xarray Dataset\n", "# Create Xarray Dataset\n",
"ds = xr.open_dataset(filename)" "ds = xr.open_dataset(filename)"
] ]
@ -564,7 +585,8 @@
"ax.legend(handles, labels)\n", "ax.legend(handles, labels)\n",
"ax.grid(linestyle='--')\n", "ax.grid(linestyle='--')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}Max_t2m_clim_Sep_Lille.png')" "# fig.savefig(f'{DATADIR}Max_t2m_clim_Sep_Lille.png')\n",
"fig.savefig(os.path.join(DATADIR, 'Max_t2m_clim_Sep_Lille.png'))"
] ]
}, },
{ {
@ -687,7 +709,8 @@
"ax.set_ylabel('Accumulated days')\n", "ax.set_ylabel('Accumulated days')\n",
"ax.set_xlabel('Maximum 2m temperature (° C)')\n", "ax.set_xlabel('Maximum 2m temperature (° C)')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}Hist_max_t2m_mid-Sep_1979-2019.png')" "# fig.savefig(f'{DATADIR}Hist_max_t2m_mid-Sep_1979-2019.png')\n",
"fig.savefig(os.path.join(DATADIR, 'Hist_max_t2m_mid-Sep_1979-2019.png'))"
] ]
}, },
{ {
@ -703,6 +726,22 @@
"source": [ "source": [
"<hr>" "<hr>"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -721,7 +760,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

152
tutorials/02_observation/02x01_erb-outgoing-longwave-radiation.ipynb
View File

@ -7,6 +7,15 @@
"# Calculation of global distribution and timeseries of Outgoing Longwave Radiation (OLR) using NOAA/NCEI HIRS data" "# Calculation of global distribution and timeseries of Outgoing Longwave Radiation (OLR) using NOAA/NCEI HIRS data"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"02x01_erb-outgoing-longwave-radiation\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -66,7 +75,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -102,10 +111,36 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"erb-outgoing-longwave-radiation\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -173,31 +208,6 @@
"This tutorial is in the form of a [Jupyter notebook](https://jupyter.org/). You will not need to install any software for the training as there are a number of free cloud-based services to create, edit, run and export Jupyter notebooks such as this. Here are some suggestions (simply click on one of the links below to run the notebook):" "This tutorial is in the form of a [Jupyter notebook](https://jupyter.org/). You will not need to install any software for the training as there are a number of free cloud-based services to create, edit, run and export Jupyter notebooks such as this. Here are some suggestions (simply click on one of the links below to run the notebook):"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<style>\n",
"td, th {\n",
" border: 1px solid white;\n",
" border-collapse: collapse;\n",
"}\n",
"</style>\n",
"<table align=\"left\">\n",
" <tr>\n",
" <th>Run the tutorial via free cloud platforms: </th>\n",
" <th><a href=\"https://kaggle.com/kernels/welcome?src=https://github.com/ecmwf-projects/copernicus-training-c3s/blob/main/ecv-notebooks/erb-outgoing-longwave-radiation.ipynb\">\n",
" <img src = \"https://kaggle.com/static/images/open-in-kaggle.svg\" alt = \"Kaggle\"></th>\n",
" <th><a href=\"https://mybinder.org/v2/gh/ecmwf-projects/copernicus-training-c3s/main?labpath=ecv-notebooks/erb-outgoing-longwave-radiation.ipynb\">\n",
" <img src = \"https://mybinder.org/badge.svg\" alt = \"Binder\"></th>\n",
" <th><a href=\"https://colab.research.google.com/github/ecmwf-projects/copernicus-training-c3s/blob/main/ecv-notebooks/erb-outgoing-longwave-radiation.ipynb\">\n",
" <img src = \"https://colab.research.google.com/assets/colab-badge.svg\" alt = \"Colab\"></th>\n",
" </tr>\n",
"</table>\n",
"\n",
"<br><br>"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -295,9 +305,9 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# Filename for the zip file downloaded from the CDS\n", "# Filename for the zip file downloaded from the CDS\n",
"download_zip_file = os.path.join(DATADIR, 'olr-monthly_v02r07.zip')\n", "download_zip_file = os.path.join(CDS_DATA, 'olr-monthly_v02r07.zip')\n",
"# Filename for the netCDF file which contain the merged contents of the monthly files.\n", "# Filename for the netCDF file which contain the merged contents of the monthly files.\n",
"merged_netcdf_file = os.path.join(DATADIR, 'olr-monthly_v02r07_197901_202207.nc')" "merged_netcdf_file = os.path.join(CDS_DATA, 'olr-monthly_v02r07_197901_202207.nc')"
] ]
}, },
{ {
@ -367,33 +377,51 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"client = cdsapi.Client(url=URL, key=KEY)\n", "client = cdsapi.Client()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n", "\n",
"zip_files = []\n", "zip_files = []\n",
"\n", "\n",
"for year in range (1979, 2023):\n", "start_year = 1979\n",
"end_year = 2023\n",
"\n",
"# years = ['%02d' % (y) for y in range(start_year, end_year)]\n",
"months = ['%02d' % (mnth) for mnth in range(1, 13)]\n",
"OLR_DATA = os.path.join(CDS_DATA, 'OLR')\n",
"os.makedirs(OLR_DATA, exist_ok=True)\n",
"\n",
"dataset = 'satellite-earth-radiation-budget'\n",
"for year in range (start_year, end_year + 1):\n",
" # Filename for the zip file downloaded from the CDS\n", " # Filename for the zip file downloaded from the CDS\n",
" download_zip_file_year = os.path.join(DATADIR, '%d_olr-monthly_v02r07.zip' % year)\n", " download_zip_file_year = os.path.join(OLR_DATA, '%d_olr-monthly_v02r07.zip' % year)\n",
" zip_files.append(download_zip_file_year)\n", " zip_files.append(download_zip_file_year)\n",
" \n",
" # Filename for the netCDF file which contain the merged contents of the monthly files.\n",
"\n", "\n",
" dataset = 'satellite-earth-radiation-budget'\n", " if not os.path.exists(download_zip_file_year): \n",
" request = {\n", " # Filename for the netCDF file which contain the merged contents of the monthly files.\n",
" \"product_family\": \"hirs\",\n", " print(\"Downloading \", download_zip_file_year)\n",
" \"origin\": \"noaa_ncei\",\n", " request = {\n",
" 'data_format': 'netcdf_legacy',\n", " \"product_family\": \"hirs\",\n",
" \"variable\": [\"outgoing_longwave_radiation\"],\n", " \"origin\": \"noaa_ncei\",\n",
" \"climate_data_record_type\": \"thematic_climate_data_record\",\n", " 'data_format': 'netcdf_legacy',\n",
" \"time_aggregation\": \"monthly_mean\",\n", " \"variable\": [\"outgoing_longwave_radiation\"],\n",
" 'year': [year],\n", " \"climate_data_record_type\": \"thematic_climate_data_record\",\n",
" 'month': ['%02d' % (mnth) for mnth in range(1, 13)],\n", " \"time_aggregation\": \"monthly_mean\",\n",
" \"version\": [\"2_7_reprocessed\"]\n", " 'year': [year],\n",
" }\n", " 'month': months,\n",
" \"version\": [\"2_7_reprocessed\"]\n",
" }\n",
"\n", "\n",
" target = download_zip_file_year\n", " target = download_zip_file_year\n",
"\n", " client.retrieve(dataset, request, target)\n",
" client.retrieve(dataset, request, target)\n" " else:\n",
" print(\"File exists\", download_zip_file_year)"
] ]
}, },
{ {
@ -613,7 +641,7 @@
" fontsize=20, pad=25)\n", " fontsize=20, pad=25)\n",
"\n", "\n",
"# and save the figure\n", "# and save the figure\n",
"fig1.savefig('./Example_1_HIRS_olr_mean.png', dpi=500, bbox_inches='tight')" "fig1.savefig(os.path.join(DATADIR, 'Example_1_HIRS_olr_mean.png'), dpi=500, bbox_inches='tight')"
] ]
}, },
{ {
@ -735,7 +763,7 @@
"dates_rng = pd.date_range(dateStart, date_End, freq='2YS')\n", "dates_rng = pd.date_range(dateStart, date_End, freq='2YS')\n",
"plt.xticks(dates_rng, [dtz.strftime('%Y') for dtz in dates_rng], rotation=45)\n", "plt.xticks(dates_rng, [dtz.strftime('%Y') for dtz in dates_rng], rotation=45)\n",
"\n", "\n",
"plt.savefig('./Example_2_olr_timeserie_Globe.png', dpi=500, bbox_inches='tight')" "plt.savefig(os.path.join(DATADIR, 'Example_2_olr_timeserie_Globe.png'), dpi=500, bbox_inches='tight')"
] ]
}, },
{ {
@ -809,7 +837,7 @@
"plt.xticks(dates_rng, [dtz.strftime('%Y') for dtz in dates_rng], rotation=45)\n", "plt.xticks(dates_rng, [dtz.strftime('%Y') for dtz in dates_rng], rotation=45)\n",
"\n", "\n",
"# Save figure to the disk \n", "# Save figure to the disk \n",
"plt.savefig('./Example_3_olr_timeserie_Arctic.png', dpi=500, bbox_inches='tight')" "plt.savefig(os.path.join(DATADIR, 'Example_3_olr_timeserie_Arctic.png'), dpi=500, bbox_inches='tight')"
] ]
}, },
{ {
@ -850,7 +878,23 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"[Back to top the Page](#page-top)" "<hr>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
] ]
} }
], ],
@ -870,7 +914,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
}, },
"toc": { "toc": {
"base_numbering": 1, "base_numbering": 1,

225
tutorials/03_climate_projection/03x01_projections-cmip6.ipynb
View File

@ -7,6 +7,15 @@
"# Plot an Ensemble of CMIP6 Climate Projections" "# Plot an Ensemble of CMIP6 Climate Projections"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"03x01_projections-cmip6\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@ -98,7 +107,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -134,7 +143,9 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"projections-cmip6\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
@ -142,7 +153,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"this tutorial needs additional dependencies" "to avoid to download already downloaded data, we put them in a local folder"
] ]
}, },
{ {
@ -151,7 +162,15 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"# !pip install --upgrade xarray zarr dask fsspec\n" "CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
] ]
}, },
{ {
@ -174,7 +193,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -324,14 +343,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"# new apis requires arrays of values for years and months instead of dates ranges (was date': '1850-01-01/2014-12-31',)\n",
"\n", "\n",
"\n", "c = cdsapi.Client()"
"start_year = 1850\n",
"end_year = 2014\n",
"\n",
"years = ['%04d' % (x) for x in range(start_year, end_year+1)]\n",
"months = ['%02d' % (x) for x in range(1, 13)]"
] ]
}, },
{ {
@ -342,40 +355,34 @@
"source": [ "source": [
"# DOWNLOAD DATA FOR HISTORICAL PERIOD\n", "# DOWNLOAD DATA FOR HISTORICAL PERIOD\n",
"\n", "\n",
"c = cdsapi.Client()\n",
"\n",
"for j in models:\n",
" c.retrieve(\n",
" 'projections-cmip6',\n",
" {\n",
" 'download_format': 'zip',\n",
" 'data_format': 'netcdf_legacy',\n",
" 'temporal_resolution': 'monthly',\n",
" 'experiment': 'historical',\n",
" 'level': 'single_levels',\n",
" 'variable': 'near_surface_air_temperature',\n",
" 'model': f'{j}',\n",
" # 'date': '1850-01-01/2014-12-31',\n",
" \"year\": years,\n",
" \"month\": months\n",
" },\n",
" f'{DATADIR}cmip6_monthly_1850-2014_historical_{j}.zip')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# new apis requires arrays of values for years and months instead of dates ranges (was date': '1850-01-01/2014-12-31',)\n", "# new apis requires arrays of values for years and months instead of dates ranges (was date': '1850-01-01/2014-12-31',)\n",
"\n", "start_year = 1850\n",
"\n", "end_year = 2014\n",
"start_year = 2015\n",
"end_year = 2100\n",
"\n", "\n",
"years = ['%04d' % (x) for x in range(start_year, end_year+1)]\n", "years = ['%04d' % (x) for x in range(start_year, end_year+1)]\n",
"months = ['%02d' % (x) for x in range(1, 13)]" "months = ['%02d' % (x) for x in range(1, 13)]\n",
"\n",
"for model in models:\n",
" filename = os.path.join(CDS_DATA, f'cmip6_monthly_{start_year}-{end_year}_{model}.zip')\n",
" if not os.path.exists(filename):\n",
" print(f\"Downloading {filename}\")\n",
" c.retrieve(\n",
" 'projections-cmip6',\n",
" {\n",
" 'download_format': 'zip',\n",
" 'data_format': 'netcdf_legacy',\n",
" 'temporal_resolution': 'monthly',\n",
" 'experiment': 'historical',\n",
" 'level': 'single_levels',\n",
" 'variable': 'near_surface_air_temperature',\n",
" 'model': f'{model}',\n",
" # 'date': '1850-01-01/2014-12-31',\n",
" \"year\": years,\n",
" \"month\": months\n",
" },\n",
" filename)\n",
" else:\n",
" print(f\"File {filename} already exists\")"
] ]
}, },
{ {
@ -386,26 +393,45 @@
"source": [ "source": [
"# DOWNLOAD DATA FOR FUTURE SCENARIOS\n", "# DOWNLOAD DATA FOR FUTURE SCENARIOS\n",
"\n", "\n",
"c = cdsapi.Client()\n", "# new apis requires arrays of values for years and months instead of dates ranges (was date': '1850-01-01/2014-12-31',)\n",
"\n", "\n",
"for i in experiments[1:]:\n", "start_year = 2015\n",
" for j in models:\n", "end_year = 2100\n",
" c.retrieve(\n", "\n",
" 'projections-cmip6',\n", "years = ['%04d' % (x) for x in range(start_year, end_year+1)]\n",
" {\n", "months = ['%02d' % (x) for x in range(1, 13)]\n",
" 'download_format': 'zip',\n", "\n"
" 'data_format': 'netcdf_legacy',\n", ]
" 'temporal_resolution': 'monthly',\n", },
" 'experiment': f'{i}',\n", {
" 'level': 'single_levels',\n", "cell_type": "code",
" 'variable': 'near_surface_air_temperature',\n", "execution_count": null,
" 'model': f'{j}',\n", "metadata": {},
" # 'date': '1850-01-01/2014-12-31',\n", "outputs": [],
" \"year\": years,\n", "source": [
" \"month\": months\n", "for experiment in experiments[1:]:\n",
" },\n", " for model in models:\n",
" f'{DATADIR}cmip6_monthly_2015-2100_{i}_{j}.zip'\n", " filename = os.path.join(CDS_DATA, f'cmip6_monthly_{start_year}-{end_year}_{experiment}_{model}.zip')\n",
" )" " if not os.path.exists(filename):\n",
" print(f\"Downloading {filename}\")\n",
" c.retrieve(\n",
" 'projections-cmip6',\n",
" {\n",
" 'download_format': 'zip',\n",
" 'data_format': 'netcdf_legacy',\n",
" 'temporal_resolution': 'monthly',\n",
" 'experiment': f'{experiment}',\n",
" 'level': 'single_levels',\n",
" 'variable': 'near_surface_air_temperature',\n",
" 'model': f'{model}',\n",
" # 'date': '1850-01-01/2014-12-31',\n",
" \"year\": years,\n",
" \"month\": months\n",
" },\n",
" filename\n",
" )\n",
" else:\n",
" print(f\"File {filename} already exists\")"
] ]
}, },
{ {
@ -428,10 +454,13 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"cmip6_zip_paths = glob(f'{DATADIR}*.zip')\n", "cmip6_zip_paths = glob(os.path.join(DATADIR, '*.zip'))\n",
"for j in cmip6_zip_paths:\n", "print (cmip6_zip_paths)\n",
" with zipfile.ZipFile(j, 'r') as zip_ref:\n", "\n",
" zip_ref.extractall(f'{DATADIR}')" "for model in cmip6_zip_paths:\n",
" with zipfile.ZipFile(model, 'r') as zip_ref:\n",
" print(f'{cmip6_zip_paths} will be extracted to {DATADIR}')\n",
" zip_ref.extractall(DATADIR)"
] ]
}, },
{ {
@ -450,9 +479,11 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"cmip6_nc = list()\n", "cmip6_nc = list()\n",
"cmip6_nc_rel = glob(f'{DATADIR}tas*.nc')\n", "\n",
"for i in cmip6_nc_rel:\n", "cmip6_nc_rel = glob(os.path.join(DATADIR, 'tas*.nc'))\n",
" cmip6_nc.append(os.path.basename(i))" "\n",
"for experiment in cmip6_nc_rel:\n",
" cmip6_nc.append(os.path.basename(experiment))"
] ]
}, },
{ {
@ -514,7 +545,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"ds = xr.open_dataset(f'{DATADIR}{cmip6_nc[0]}')\n", "ds = xr.open_dataset(os.path.join(DATADIR,cmip6_nc[0]))\n",
"ds" "ds"
] ]
}, },
@ -697,7 +728,7 @@
"# Function to aggregate in geographical lat lon dimensions\n", "# Function to aggregate in geographical lat lon dimensions\n",
"\n", "\n",
"def geog_agg(fn):\n", "def geog_agg(fn):\n",
" ds = xr.open_dataset(f'{DATADIR}{fn}')\n", " ds = xr.open_dataset( os.path.join(DATADIR,fn))\n",
" exp = ds.attrs['experiment_id']\n", " exp = ds.attrs['experiment_id']\n",
" mod = ds.attrs['source_id']\n", " mod = ds.attrs['source_id']\n",
" da = ds['tas']\n", " da = ds['tas']\n",
@ -711,7 +742,10 @@
" da_yr = da_yr.expand_dims('model')\n", " da_yr = da_yr.expand_dims('model')\n",
" da_yr = da_yr.assign_coords(experiment=exp)\n", " da_yr = da_yr.assign_coords(experiment=exp)\n",
" da_yr = da_yr.expand_dims('experiment')\n", " da_yr = da_yr.expand_dims('experiment')\n",
" da_yr.to_netcdf(path=f'{DATADIR}cmip6_agg_{exp}_{mod}_{str(da_yr.year[0].values)}.nc')" " \n",
" # da_yr.to_netcdf(path=f'{DATADIR}cmip6_agg_{exp}_{mod}_{str(da_yr.year[0].values)}.nc')\n",
" filename = f'cmip6_agg_{exp}_{mod}_{str(da_yr.year[0].values)}.nc'\n",
" da_yr.to_netcdf(path=os.path.join(DATADIR, filename))"
] ]
}, },
{ {
@ -727,10 +761,10 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"for i in cmip6_nc:\n", "for experiment in cmip6_nc:\n",
" try:\n", " try:\n",
" geog_agg(i)\n", " geog_agg(experiment)\n",
" except: print(f'{i} failed')" " except: print(f'{experiment} failed')"
] ]
}, },
{ {
@ -755,7 +789,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"data_ds = xr.open_mfdataset(f'{DATADIR}cmip6_agg*.nc')\n" "# data_ds = xr.open_mfdataset(f'{DATADIR}cmip6_agg*.nc')\n",
"data_ds = xr.open_mfdataset(os.path.join(DATADIR, 'cmip6_agg*.nc'))\n"
] ]
}, },
{ {
@ -873,11 +908,11 @@
"fig, ax = plt.subplots(1, 1, figsize = (16, 8))\n", "fig, ax = plt.subplots(1, 1, figsize = (16, 8))\n",
"\n", "\n",
"colours = ['black','red','green','blue']\n", "colours = ['black','red','green','blue']\n",
"for i in np.arange(len(experiments)):\n", "for experiment in np.arange(len(experiments)):\n",
" ax.plot(data_50.year, data_50[i,:], color=f'{colours[i]}', \n", " ax.plot(data_50.year, data_50[experiment,:], color=f'{colours[experiment]}', \n",
" label=f'{data_50.experiment[i].values} 50th quantile')\n", " label=f'{data_50.experiment[experiment].values} 50th quantile')\n",
" ax.fill_between(data_50.year, data_90[i,:], data_10[i,:], alpha=0.1, color=f'{colours[i]}', \n", " ax.fill_between(data_50.year, data_90[experiment,:], data_10[experiment,:], alpha=0.1, color=f'{colours[experiment]}', \n",
" label=f'{data_50.experiment[i].values} 10th and 90th quantile range')\n", " label=f'{data_50.experiment[experiment].values} 10th and 90th quantile range')\n",
"\n", "\n",
"ax.set_xlim(1850,2100)\n", "ax.set_xlim(1850,2100)\n",
"ax.set_title('CMIP6 annual global average temperature (1850 to 2100)')\n", "ax.set_title('CMIP6 annual global average temperature (1850 to 2100)')\n",
@ -887,7 +922,8 @@
"ax.legend(handles, labels)\n", "ax.legend(handles, labels)\n",
"ax.grid(linestyle='--')\n", "ax.grid(linestyle='--')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}CMIP6_annual_global_tas.png')" "# fig.savefig(f'{DATADIR}CMIP6_annual_global_tas.png')\n",
"fig.savefig(os.path.join(DATADIR, 'CMIP6_annual_global_tas.png'))"
] ]
}, },
{ {
@ -911,6 +947,29 @@
"<p></p>\n", "<p></p>\n",
"<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>" "<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -929,7 +988,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

1087
tutorials/03_climate_projection/03x01_projections-cmip6_parallel.ipynb
View File

File diff suppressed because it is too large Load Diff

177
tutorials/03_climate_projection/03x02_projections-cordex.ipynb
View File

@ -1,5 +1,10 @@
{ {
"cells": [ "cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -7,6 +12,15 @@
"# Analysis of Projected versus Historical Climatology with CORDEX Data" "# Analysis of Projected versus Historical Climatology with CORDEX Data"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"03x02_projections-cordex\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -111,7 +125,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -147,10 +161,36 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"projections-cordex\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -171,7 +211,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -266,25 +306,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client()\n", "c = cdsapi.Client()"
"\n",
"c.retrieve(\n",
" 'projections-cordex-domains-single-levels',\n",
" {\n",
" 'download_format': 'zip',\n",
" 'data_format': 'netcdf_legacy',\n",
" 'domain': 'africa',\n",
" 'experiment': 'historical',\n",
" 'horizontal_resolution': '0_44_degree_x_0_44_degree',\n",
" 'temporal_resolution': 'daily_mean',\n",
" 'variable': '2m_air_temperature',\n",
" 'gcm_model': 'cccma_canesm2',\n",
" 'rcm_model': 'cccma_canrcm4',\n",
" 'ensemble_member': 'r1i1p1',\n",
" 'start_year': ['1971', '1976', '1981', '1986', '1991', '1996'],\n",
" 'end_year': ['1975', '1980', '1985', '1990', '1995', '2000'],\n",
" },\n",
" f'{DATADIR}1971-2000_cordex_historical_africa.zip')"
] ]
}, },
{ {
@ -293,23 +315,61 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c.retrieve(\n", "\n",
" 'projections-cordex-domains-single-levels',\n", "filename_historical = os.path.join(CDS_DATA, '1971-2000_cordex_historical_africa.zip')\n",
" {\n", "\n",
" 'download_format': 'zip',\n", "if not os.path.exists(filename_historical):\n",
" 'data_format': 'netcdf_legacy',\n", " print(\"Downloading\", filename_historical)\n",
" 'domain': 'africa',\n", " c.retrieve(\n",
" 'experiment': 'rcp_4_5',\n", " 'projections-cordex-domains-single-levels',\n",
" 'horizontal_resolution': '0_44_degree_x_0_44_degree',\n", " {\n",
" 'temporal_resolution': 'daily_mean',\n", " 'download_format': 'zip',\n",
" 'variable': '2m_air_temperature',\n", " 'data_format': 'netcdf_legacy',\n",
" 'gcm_model': 'cccma_canesm2',\n", " 'domain': 'africa',\n",
" 'rcm_model': 'cccma_canrcm4',\n", " 'experiment': 'historical',\n",
" 'ensemble_member': 'r1i1p1',\n", " 'horizontal_resolution': '0_44_degree_x_0_44_degree',\n",
" 'start_year': ['2071', '2076', '2081', '2086', '2091', '2096'],\n", " 'temporal_resolution': 'daily_mean',\n",
" 'end_year': ['2075', '2080', '2085', '2090', '2095', '2100'],\n", " 'variable': '2m_air_temperature',\n",
" },\n", " 'gcm_model': 'cccma_canesm2',\n",
" f'{DATADIR}2071-2100_cordex_rcp_4_5_africa.zip')" " 'rcm_model': 'cccma_canrcm4',\n",
" 'ensemble_member': 'r1i1p1',\n",
" 'start_year': ['1971', '1976', '1981', '1986', '1991', '1996'],\n",
" 'end_year': ['1975', '1980', '1985', '1990', '1995', '2000'],\n",
" },\n",
" filename_historical)\n",
"else:\n",
" print(\"File already exists\", filename_historical)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"filename_future = os.path.join(CDS_DATA, '2071-2100_cordex_rcp_4_5_africa.zip')\n",
"\n",
"if not os.path.exists(filename_future):\n",
" print(\"downloading\", filename_future)\n",
" c.retrieve(\n",
" 'projections-cordex-domains-single-levels',\n",
" {\n",
" 'download_format': 'zip',\n",
" 'data_format': 'netcdf_legacy',\n",
" 'domain': 'africa',\n",
" 'experiment': 'rcp_4_5',\n",
" 'horizontal_resolution': '0_44_degree_x_0_44_degree',\n",
" 'temporal_resolution': 'daily_mean',\n",
" 'variable': '2m_air_temperature',\n",
" 'gcm_model': 'cccma_canesm2',\n",
" 'rcm_model': 'cccma_canrcm4',\n",
" 'ensemble_member': 'r1i1p1',\n",
" 'start_year': ['2071', '2076', '2081', '2086', '2091', '2096'],\n",
" 'end_year': ['2075', '2080', '2085', '2090', '2095', '2100'],\n",
" },\n",
" filename_future)\n",
"else:\n",
" print(\"File already exists\", filename_future)"
] ]
}, },
{ {
@ -334,7 +394,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"cordex_zip_paths = glob(f'{DATADIR}*.zip')" "cordex_zip_paths = glob(os.path.join(CDS_DATA, '*.zip'))"
] ]
}, },
{ {
@ -372,7 +432,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"ds_1971_2000_historical = xr.open_mfdataset(f'{DATADIR}*CanESM2_historical*.nc')\n", "ds_1971_2000_historical = xr.open_mfdataset(os.path.join(DATADIR, \"*CanESM2_historical*.nc\"))\n",
"ds_1971_2000_historical" "ds_1971_2000_historical"
] ]
}, },
@ -449,7 +509,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"ds_2071_2100_projection = xr.open_mfdataset(f'{DATADIR}*CanESM2_rcp45*.nc')" "ds_2071_2100_projection = xr.open_mfdataset(os.path.join(DATADIR, \"*CanESM2_rcp45*.nc\"))"
] ]
}, },
{ {
@ -574,7 +634,9 @@
"cbar.set_label('\\nTemperature difference (Kelvin/Celsius)\\n')\n", "cbar.set_label('\\nTemperature difference (Kelvin/Celsius)\\n')\n",
"\n", "\n",
"# Save the figure\n", "# Save the figure\n",
"fig.savefig(f'{DATADIR}TAS_1971-2000_2071-2100_june.png', dpi=300)" "# fig.savefig(f'{DATADIR}TAS_1971-2000_2071-2100_june.png', dpi=300)\n",
"image_path = os.path.join(DATADIR, 'TAS_1971-2000_2071-2100_june.png')\n",
"fig.savefig(image_path, dpi=300)"
] ]
}, },
{ {
@ -608,7 +670,9 @@
"cbar = plt.colorbar(im,fraction=0.046, pad=0.04)\n", "cbar = plt.colorbar(im,fraction=0.046, pad=0.04)\n",
"cbar.set_label('\\nTemperature difference (Kelvin/Celsius)\\n')\n", "cbar.set_label('\\nTemperature difference (Kelvin/Celsius)\\n')\n",
"\n", "\n",
"fig.savefig(f'{DATADIR}TAS_1971-2000_2071-2100.png', dpi=300)" "# fig.savefig(f'{DATADIR}TAS_1971-2000_2071-2100.png', dpi=300)\n",
"filename = os.path.join(DATADIR, 'TAS_1971-2000_2071-2100.png')\n",
"fig.savefig(filename, dpi=300)\n"
] ]
}, },
{ {
@ -627,6 +691,29 @@
"<p></p>\n", "<p></p>\n",
"<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>" "<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -645,7 +732,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

205
tutorials/04_seasonal_forecast/04x01_sf-anomalies.ipynb
View File

@ -16,6 +16,16 @@
"# Seasonal Forecast Anomalies" "# Seasonal Forecast Anomalies"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"CDS_PROJECT = \"04x01_sf-anomalies\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@ -140,7 +150,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -176,7 +186,9 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"sf-anomalies\")\n", "import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
@ -184,7 +196,17 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"this tutorial needs additional dependencies" "to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
] ]
}, },
{ {
@ -219,7 +241,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# Install CDS API for downloading data from the CDS\n", "# Install CDS API for downloading data from the CDS\n",
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -239,7 +261,7 @@
"source": [ "source": [
"# Install cfgrib to enable us to read GRIB format files\n", "# Install cfgrib to enable us to read GRIB format files\n",
"# !conda install -c conda-forge cfgrib -y\n", "# !conda install -c conda-forge cfgrib -y\n",
"!pip install cfgrib ecCodes" "# !pip install cfgrib ecCodes"
] ]
}, },
{ {
@ -258,6 +280,15 @@
"### Load packages" "### Load packages"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# !pip install ipywidgets"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
@ -432,68 +463,77 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client()\n", "c = cdsapi.Client()"
"\n",
"# Hindcast data request\n",
"c.retrieve(\n",
" 'seasonal-monthly-single-levels',\n",
" {\n",
" 'format': 'grib',\n",
" 'originating_centre': 'ecmwf',\n",
" 'system': '5',\n",
" 'variable': 'total_precipitation',\n",
" 'product_type': 'monthly_mean',\n",
" 'year': [\n",
" '1993', '1994', '1995',\n",
" '1996', '1997', '1998',\n",
" '1999', '2000', '2001',\n",
" '2002', '2003', '2004',\n",
" '2005', '2006', '2007',\n",
" '2008', '2009', '2010',\n",
" '2011', '2012', '2013',\n",
" '2014', '2015', '2016',\n",
" ],\n",
" 'month': '05',\n",
" 'leadtime_month': [\n",
" '1', '2', '3',\n",
" '4', '5', '6',\n",
" ],\n",
" },\n",
" f'{DATADIR}/ecmwf_seas5_1993-2016_05_hindcast_monthly_tp.grib')\n",
"\n",
"# Forecast data request\n",
"c.retrieve(\n",
" 'seasonal-monthly-single-levels',\n",
" {\n",
" 'format': 'grib',\n",
" 'originating_centre': 'ecmwf',\n",
" 'system': '5',\n",
" 'variable': 'total_precipitation',\n",
" 'product_type': 'monthly_mean',\n",
" 'year': '2021',\n",
" 'month': '05',\n",
" 'leadtime_month': [\n",
" '1', '2', '3',\n",
" '4', '5', '6',\n",
" ],\n",
" },\n",
" f'{DATADIR}/ecmwf_seas5_2021_05_forecast_monthly_tp.grib')"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "code",
"metadata": { "execution_count": null,
"papermill": { "metadata": {},
"duration": 0.377411, "outputs": [],
"end_time": "2022-03-14T17:57:27.753996",
"exception": false,
"start_time": "2022-03-14T17:57:27.376585",
"status": "completed"
},
"tags": []
},
"source": [ "source": [
"<br>" "\n",
"start_year = 1993\n",
"end_year = 2016\n",
"years = [str(year) for year in range(start_year, end_year + 1)]\n",
"month = '05'\n",
"\n",
"filename_hindcast = os.path.join(CDS_DATA, f'ecmwf_seas5_{start_year}-{end_year}_{month}_hindcast_monthly_tp.grib')\n",
"\n",
"# Hindcast data request\n",
"if not os.path.exists(filename_hindcast):\n",
" print(f\"Downloading hindcast\", filename_hindcast)\n",
" c.retrieve(\n",
" 'seasonal-monthly-single-levels',\n",
" {\n",
" 'format': 'grib',\n",
" 'originating_centre': 'ecmwf',\n",
" 'system': '5',\n",
" 'variable': 'total_precipitation',\n",
" 'product_type': 'monthly_mean',\n",
" 'year': years,\n",
" 'month': month,\n",
" 'leadtime_month': [\n",
" '1', '2', '3',\n",
" '4', '5', '6',\n",
" ],\n",
" },\n",
" filename_hindcast)\n",
"else:\n",
" print(f\"File exists\", filename_hindcast)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"forecast_year = 2021\n",
"filename_forecast = os.path.join(CDS_DATA, f'ecmwf_seas5_{forecast_year}_{month}_forecast_monthly_tp.grib')\n",
"\n",
"# Forecast data request\n",
"if not os.path.exists(filename_forecast):\n",
" print(f\"Downloading forecast\", filename_forecast)\n",
" c.retrieve(\n",
" 'seasonal-monthly-single-levels',\n",
" {\n",
" 'format': 'grib',\n",
" 'originating_centre': 'ecmwf',\n",
" 'system': '5',\n",
" 'variable': 'total_precipitation',\n",
" 'product_type': 'monthly_mean',\n",
" 'year': forecast_year,\n",
" 'month': month,\n",
" 'leadtime_month': [\n",
" '1', '2', '3',\n",
" '4', '5', '6',\n",
" ],\n",
" },\n",
" filename_forecast)\n",
"else:\n",
" print(f\"File exists\", filename_forecast)\n"
] ]
}, },
{ {
@ -555,7 +595,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"ds = xr.open_dataset(f'{DATADIR}/ecmwf_seas5_1993-2016_05_hindcast_monthly_tp.grib', engine='cfgrib')\n", "ds = xr.open_dataset(filename_hindcast, engine='cfgrib')\n",
"ds" "ds"
] ]
}, },
@ -607,7 +647,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"ds_hindcast = xr.open_dataset(f'{DATADIR}/ecmwf_seas5_1993-2016_05_hindcast_monthly_tp.grib', engine='cfgrib', backend_kwargs=dict(time_dims=('forecastMonth', 'time')))\n", "ds_hindcast = xr.open_dataset(filename_hindcast, engine='cfgrib', backend_kwargs=dict(time_dims=('forecastMonth', 'time')))\n",
"ds_hindcast" "ds_hindcast"
] ]
}, },
@ -759,7 +799,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"seas5_forecast = xr.open_dataset(f'{DATADIR}/ecmwf_seas5_2021_05_forecast_monthly_tp.grib', engine='cfgrib', \n", "seas5_forecast = xr.open_dataset(filename_forecast, engine='cfgrib', \n",
" backend_kwargs=dict(time_dims=('forecastMonth', 'time')))\n", " backend_kwargs=dict(time_dims=('forecastMonth', 'time')))\n",
"seas5_forecast" "seas5_forecast"
] ]
@ -1134,7 +1174,9 @@
"fig.colorbar(im, cax=cbar_ax, orientation='horizontal', label='Total precipitation anomaly (mm)')\n", "fig.colorbar(im, cax=cbar_ax, orientation='horizontal', label='Total precipitation anomaly (mm)')\n",
"\n", "\n",
"# Save the figure\n", "# Save the figure\n",
"fig.savefig('{}/TotalPrecAnomalyForecastSAsia.png'.format(DATADIR) )" "# fig.savefig('{}/TotalPrecAnomalyForecastSAsia.png'.format(DATADIR) )\n",
"image_path = os.path.join(DATADIR, 'TotalPrecAnomalyForecastSAsia.png')\n",
"fig.savefig(image_path)"
] ]
}, },
{ {
@ -1602,7 +1644,9 @@
"ax.set_xlabel(os.linesep + 'Valid date', fontsize=12)\n", "ax.set_xlabel(os.linesep + 'Valid date', fontsize=12)\n",
"\n", "\n",
"# Save the figure\n", "# Save the figure\n",
"fig.savefig('{}/TotalPrecForecastHindcastAnomaliesSAsia.png'.format(DATADIR) )" "# fig.savefig('{}/TotalPrecForecastHindcastAnomaliesSAsia.png'.format(DATADIR) )\n",
"image_path = os.path.join(DATADIR, 'TotalPrecForecastHindcastAnomaliesSAsia.png')\n",
"fig.savefig(image_path)"
] ]
}, },
{ {
@ -1914,6 +1958,29 @@
"<p></p>\n", "<p></p>\n",
"<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>" "<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -1932,7 +1999,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

427
tutorials/04_seasonal_forecast/04x02_sf-verification.ipynb
View File

@ -2,15 +2,31 @@
"cells": [ "cells": [
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "fd5dbf9e", "id": "00f5035f",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"id": "74c186a0",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Seasonal Forecast Verification" "# Seasonal Forecast Verification"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"id": "07f6b26b",
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"04x02_sf-verification\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "88d5a2a8", "id": "a9e2ed65",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### About" "### About"
@ -18,7 +34,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ddb2c373", "id": "1dbc6097",
"metadata": {}, "metadata": {},
"source": [ "source": [
"This notebook provides a practical introduction on how to produce some verification metrics and scores for seasonal forecasts with data from the Copernicus Climate Change Service (C3S). C3S seasonal forecast products are based on data from several state-of-the-art seasonal prediction systems. In this notebook, as an example, we will focus on data produced by [CMCC SPSv3.5 system](https://confluence.ecmwf.int/display/CKB/Description+of+CMCC-CM2-v20191201+C3S+contribution), which is one of the forecasting systems available through C3S.\n", "This notebook provides a practical introduction on how to produce some verification metrics and scores for seasonal forecasts with data from the Copernicus Climate Change Service (C3S). C3S seasonal forecast products are based on data from several state-of-the-art seasonal prediction systems. In this notebook, as an example, we will focus on data produced by [CMCC SPSv3.5 system](https://confluence.ecmwf.int/display/CKB/Description+of+CMCC-CM2-v20191201+C3S+contribution), which is one of the forecasting systems available through C3S.\n",
@ -28,7 +44,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "07976f3e", "id": "79e7a545",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The notebook has the following outline:\n", "The notebook has the following outline:\n",
@ -49,7 +65,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "e82da11c", "id": "587b347d",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Please see here the full documentation of the [C3S Seasonal Forecast Datasets](https://confluence.ecmwf.int/display/CKB/C3S+Seasonal+Forecasts%3A+datasets+documentation). This notebook will use data from the CDS dataset [seasonal forecast monthly statistics on single levels](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=overview) (as opposed to multiple levels in the atmosphere)." "Please see here the full documentation of the [C3S Seasonal Forecast Datasets](https://confluence.ecmwf.int/display/CKB/C3S+Seasonal+Forecasts%3A+datasets+documentation). This notebook will use data from the CDS dataset [seasonal forecast monthly statistics on single levels](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=overview) (as opposed to multiple levels in the atmosphere)."
@ -57,7 +73,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "372e33e2", "id": "1fe7f66c",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-success\">\n", "<div class=\"alert alert-block alert-success\">\n",
@ -68,7 +84,109 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "e8f14f30", "id": "5f90cb12",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"id": "cc224654",
"metadata": {},
"source": [
"### d4science_copernicus_cds Library\n",
"\n",
"To request data from the Climate Data Store (CDS) programmatically using the CDS API, we will manage our authentication with the `d4science_copernicus_cds` library.\n",
"\n",
"The library prompts us to enter our credentials, which are then securely saved in our workspace. **This request is only made the first time**; afterward, the `get_credentials` function will automatically retrieve the credentials from the environment or workspace, eliminating the need to re-enter them in the Jupyter notebook.\n",
"\n",
"To obtain your API credentials:\n",
"1. Register or log in to the CDS at [https://cds.climate.copernicus.eu](https://cds.climate.copernicus.eu).\n",
"2. Visit [https://cds.climate.copernicus.eu/how-to-api](https://cds.climate.copernicus.eu/how-to-api) and copy the API key provided.\n",
"\n",
"The library will prompt you to enter:\n",
"- **URL**: The URL field is prefilled; simply press Enter to accept the default.\n",
"- **KEY**: Insert the obtained API key when prompted, then confirm saving your credentials by pressing \"y.\"\n",
"\n",
"Once saved, your credentials will be loaded automatically in future sessions, ensuring a seamless experience with the CDS API."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d44f6da2",
"metadata": {},
"outputs": [],
"source": [
"# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "08634391",
"metadata": {},
"outputs": [],
"source": [
"from d4science_copernicus_cds import cds_get_credentials, cds_datadir"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07053620",
"metadata": {},
"outputs": [],
"source": [
"URL, KEY = cds_get_credentials()\n",
"print(\"URL\", URL)\n",
"print (\"KEY\", KEY)"
]
},
{
"cell_type": "markdown",
"id": "4895bfbd",
"metadata": {},
"source": [
"cds_datadir will create a folder in our workspace, under cds_dataDir, with current timestamp and custom label"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0f84e1f5",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)"
]
},
{
"cell_type": "markdown",
"id": "71a2d395",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c7ce62f7",
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"id": "2d21f858",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### How to run this tutorial\n", "### How to run this tutorial\n",
@ -78,21 +196,31 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "7daf2174", "id": "15b15a9e",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install numpy pandas xarray xskillscore\n", "# !pip install numpy pandas xarray xskillscore\n",
"!pip install cdsapi\n", "# !pip install cdsapi\n",
"!pip install matplotlib cartopy\n", "# !pip install matplotlib cartopy\n",
"!pip install cfgrib" "# !pip install cfgrib"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5c56a698",
"metadata": {},
"outputs": [],
"source": [
"# pip install -U xskillscore"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "4596a877", "id": "42e128c5",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-danger\">\n", "<div class=\"alert alert-block alert-danger\">\n",
@ -103,7 +231,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "8a551fed", "id": "24b5206e",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Load packages" "### Load packages"
@ -112,7 +240,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "b692d9a8", "id": "c4d26b91",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -145,7 +273,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "cf201000", "id": "662df356",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 1. Request data from the CDS using CDS API" "## 1. Request data from the CDS using CDS API"
@ -153,7 +281,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "bf1f7910", "id": "9e007558",
"metadata": {}, "metadata": {},
"source": [ "source": [
"First we will set up some common variables to be used in this notebook to define the folder containing the data, the variables to be retrieved, the hindcast period and the [nominal start month](https://confluence.ecmwf.int/display/CKB/Summary+of+available+data#Summaryofavailabledata-nominal_start_date)." "First we will set up some common variables to be used in this notebook to define the folder containing the data, the variables to be retrieved, the hindcast period and the [nominal start month](https://confluence.ecmwf.int/display/CKB/Summary+of+available+data#Summaryofavailabledata-nominal_start_date)."
@ -162,13 +290,13 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "e72ed409", "id": "578e6385",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = './data'\n", "# DATADIR = './data'\n",
"\n", "\n",
"config = dict(\n", "config = dict(\n",
" list_vars = ['2m_temperature', ],\n", " list_vars = ['2m_temperature', ],\n",
@ -180,7 +308,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ebd62346-ba37-4e46-99bd-fd3f0d60c154", "id": "de42e329",
"metadata": {}, "metadata": {},
"source": [ "source": [
"After setting up this initial configuration variables, the existence of all the data folders will be checked and directories will be created if needed." "After setting up this initial configuration variables, the existence of all the data folders will be checked and directories will be created if needed."
@ -189,7 +317,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "4f09c485-88ae-4307-acba-cd4cfb6a8f02", "id": "2ac9689c",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -197,8 +325,11 @@
"source": [ "source": [
"import os\n", "import os\n",
"\n", "\n",
"SCOREDIR = DATADIR + '/scores'\n", "# SCOREDIR = DATADIR + '/scores'\n",
"PLOTSDIR = DATADIR + f'/plots/stmonth{config[\"start_month\"]:02d}'\n", "# PLOTSDIR = DATADIR + f'/plots/stmonth{config[\"start_month\"]:02d}'\n",
"\n",
"SCOREDIR = os.path.join(DATADIR, 'scores')\n",
"PLOTSDIR = os.path.join(DATADIR, f'plots/stmonth{config[\"start_month\"]:02d}')\n",
"\n", "\n",
"for directory in [DATADIR, SCOREDIR, PLOTSDIR]:\n", "for directory in [DATADIR, SCOREDIR, PLOTSDIR]:\n",
" # Check if the directory exists\n", " # Check if the directory exists\n",
@ -210,30 +341,30 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "8cb1220b", "id": "eb7293de",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The first step is to request data from the Climate Data Store programmatically with the help of the CDS API. Let us make use of the option to manually set the CDS API credentials. First, you have to define two variables: `CDSAPI_URL` and `CDSAPI_KEY` which build together your CDS API key. Below, you have to replace the `#########` with your personal CDS key. Please find [here](https://cds.climate.copernicus.eu/how-to-api) your personal CDS key." "~The first step is to request data from the Climate Data Store programmatically with the help of the CDS API. Let us make use of the option to manually set the CDS API credentials. First, you have to define two variables: `CDSAPI_URL` and `CDSAPI_KEY` which build together your CDS API key. Below, you have to replace the `#########` with your personal CDS key. Please find [here](https://cds.climate.copernicus.eu/how-to-api) your personal CDS key.~"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "2954b6ce", "id": "24093960",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"CDSAPI_URL = 'https://cds.climate.copernicus.eu/api'\n", "# CDSAPI_URL = 'https://cds.climate.copernicus.eu/api'\n",
"CDSAPI_KEY = '########################################'\n", "# CDSAPI_KEY = '########################################'\n",
"\n", "\n",
"c = cdsapi.Client(url=CDSAPI_URL, key=CDSAPI_KEY)" "c = cdsapi.Client()"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "587a8b90", "id": "90f23c32",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 1a. Retrieve hindcast data" "### 1a. Retrieve hindcast data"
@ -241,7 +372,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "c1476d6e", "id": "03c60e1b",
"metadata": { "metadata": {
"papermill": { "papermill": {
"duration": 0.42258, "duration": 0.42258,
@ -271,7 +402,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "a657d6c8", "id": "0f4cc59e",
"metadata": {}, "metadata": {},
"source": [ "source": [
"If you have not already done so, you will need to accept the **terms & conditions** of the data before you can download it. These can be viewed and accepted in the [CDS download page](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=download) by scrolling to the end of the download form." "If you have not already done so, you will need to accept the **terms & conditions** of the data before you can download it. These can be viewed and accepted in the [CDS download page](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=download) by scrolling to the end of the download form."
@ -279,7 +410,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "530dce10", "id": "45ca97eb",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-info\">\n", "<div class=\"alert alert-block alert-info\">\n",
@ -290,7 +421,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "617e960d", "id": "96ce56e8",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -303,29 +434,33 @@
"\n", "\n",
"\n", "\n",
"hcst_bname = '{origin}_s{system}_stmonth{start_month:02d}_hindcast{hcstarty}-{hcendy}_monthly'.format(**config)\n", "hcst_bname = '{origin}_s{system}_stmonth{start_month:02d}_hindcast{hcstarty}-{hcendy}_monthly'.format(**config)\n",
"hcst_fname = f'{DATADIR}/{hcst_bname}.grib'\n", "# hcst_fname = f'{DATADIR}/{hcst_bname}.grib'\n",
"hcst_fname = os.path.join(CDS_DATA, f'{hcst_bname}.grib')\n",
"\n",
"print(hcst_fname)\n", "print(hcst_fname)\n",
"\n", "\n",
"\n", "if not os.path.exists(hcst_fname):\n",
"c.retrieve(\n", " print(f'Downloading hindcast data to {hcst_fname}')\n",
" 'seasonal-monthly-single-levels',\n", " c.retrieve(\n",
" {\n", " 'seasonal-monthly-single-levels',\n",
" 'data_format': 'grib',\n", " {\n",
" 'originating_centre': config['origin'],\n", " 'data_format': 'grib',\n",
" 'system': config['system'],\n", " 'originating_centre': config['origin'],\n",
" 'variable': config['list_vars'],\n", " 'system': config['system'],\n",
" 'product_type': 'monthly_mean',\n", " 'variable': config['list_vars'],\n",
" 'year': ['{}'.format(yy) for yy in range(config['hcstarty'],config['hcendy']+1)],\n", " 'product_type': 'monthly_mean',\n",
" 'month': '{:02d}'.format(config['start_month']),\n", " 'year': ['{}'.format(yy) for yy in range(config['hcstarty'],config['hcendy']+1)],\n",
" 'leadtime_month': ['1', '2', '3','4', '5', '6'],\n", " 'month': '{:02d}'.format(config['start_month']),\n",
" },\n", " 'leadtime_month': ['1', '2', '3','4', '5', '6'],\n",
" hcst_fname)\n", " },\n",
" " " hcst_fname)\n",
"else:\n",
" print(f'Hindcast data already downloaded to {hcst_fname}') "
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "0b7fff68", "id": "47223479",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 1b. Retrieve observations data (ERA5)" "### 1b. Retrieve observations data (ERA5)"
@ -333,7 +468,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "49d34806", "id": "ed320073",
"metadata": { "metadata": {
"papermill": { "papermill": {
"duration": 0.42258, "duration": 0.42258,
@ -363,37 +498,43 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "fc1b28a1", "id": "86d99493",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"obs_fname = '{fpath}/era5_monthly_stmonth{start_month:02d}_{hcstarty}-{hcendy}.grib'.format(fpath=DATADIR,**config)\n", "# obs_fname = '{fpath}/era5_monthly_stmonth{start_month:02d}_{hcstarty}-{hcendy}.grib'.format(fpath=DATADIR,**config)\n",
"\n",
"obs_name = 'era5_monthly_stmonth{start_month:02d}_{hcstarty}-{hcendy}.grib'.format(**config)\n",
"obs_fname = os.path.join(CDS_DATA, obs_name)\n",
"print(obs_fname)\n", "print(obs_fname)\n",
"\n", "\n",
"\n", "if not os.path.exists(obs_fname):\n",
"c.retrieve(\n", " print(f'Downloading ERA5 monthly data to {obs_fname}')\n",
" 'reanalysis-era5-single-levels-monthly-means',\n", " c.retrieve(\n",
" {\n", " 'reanalysis-era5-single-levels-monthly-means',\n",
" 'product_type': 'monthly_averaged_reanalysis',\n", " {\n",
" 'variable': config['list_vars'],\n", " 'product_type': 'monthly_averaged_reanalysis',\n",
" # NOTE from observations we need to go one year beyond so we have available all the right valid dates\n", " 'variable': config['list_vars'],\n",
" # e.g. Nov.2016 start date forecast goes up to April 2017 \n", " # NOTE from observations we need to go one year beyond so we have available all the right valid dates\n",
" 'year': ['{}'.format(yy) for yy in range(config['hcstarty'],config['hcendy']+2)],\n", " # e.g. Nov.2016 start date forecast goes up to April 2017 \n",
" 'month': ['{:02d}'.format((config['start_month']+leadm)%12) if config['start_month']+leadm!=12 else '12' for leadm in range(6)],\n", " 'year': ['{}'.format(yy) for yy in range(config['hcstarty'],config['hcendy']+2)],\n",
" 'time': '00:00',\n", " 'month': ['{:02d}'.format((config['start_month']+leadm)%12) if config['start_month']+leadm!=12 else '12' for leadm in range(6)],\n",
" # We can ask CDS to interpolate ERA5 to the same grid used by C3S seasonal forecasts\n", " 'time': '00:00',\n",
" 'grid': '1/1',\n", " # We can ask CDS to interpolate ERA5 to the same grid used by C3S seasonal forecasts\n",
" 'area': '89.5/0.5/-89.5/359.5',\n", " 'grid': '1/1',\n",
" 'data_format': 'grib',\n", " 'area': '89.5/0.5/-89.5/359.5',\n",
" },\n", " 'data_format': 'grib',\n",
" obs_fname)" " },\n",
" obs_fname)\n",
"else:\n",
" print(f'ERA5 data already downloaded to {obs_fname}')"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7e09ca3a", "id": "b1e19f17",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 2. Compute deterministic and probabilistic products from the hindcast data" "## 2. Compute deterministic and probabilistic products from the hindcast data"
@ -401,7 +542,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "8ac3f5d0", "id": "6b94f3cc",
"metadata": {}, "metadata": {},
"source": [ "source": [
"In this section we will be calculating the different derived products we will be scoring later in the notebook. We will use the reference period defined in `config[\"hcstarty\"]`,`config[\"hcendy\"]` to calculate:\n", "In this section we will be calculating the different derived products we will be scoring later in the notebook. We will use the reference period defined in `config[\"hcstarty\"]`,`config[\"hcendy\"]` to calculate:\n",
@ -413,7 +554,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "fc9c79b8", "id": "cfc4b79c",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-info\">\n", "<div class=\"alert alert-block alert-info\">\n",
@ -426,7 +567,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "6f022052", "id": "5ce38886",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -440,7 +581,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "447eff8a", "id": "3a77c97e",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 2a. Anomalies" "### 2a. Anomalies"
@ -448,7 +589,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "000fe064", "id": "a87790fc",
"metadata": {}, "metadata": {},
"source": [ "source": [
"We will start opening the hindcast data GRIB file we downloaded in the previous section to load it into an `xarray.Dataset` object.\n", "We will start opening the hindcast data GRIB file we downloaded in the previous section to load it into an `xarray.Dataset` object.\n",
@ -464,7 +605,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "a8a45a7e", "id": "b85a5221",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -503,13 +644,15 @@
"anom_3m = anom_3m.assign_attrs(reference_period='{hcstarty}-{hcendy}'.format(**config))\n", "anom_3m = anom_3m.assign_attrs(reference_period='{hcstarty}-{hcendy}'.format(**config))\n",
"\n", "\n",
"print('Saving anomalies 1m/3m to netCDF files')\n", "print('Saving anomalies 1m/3m to netCDF files')\n",
"anom.to_netcdf(f'{DATADIR}/{hcst_bname}.1m.anom.nc')\n", "# anom.to_netcdf(f'{DATADIR}/{hcst_bname}.1m.anom.nc')\n",
"anom_3m.to_netcdf(f'{DATADIR}/{hcst_bname}.3m.anom.nc')\n" "anom.to_netcdf(os.path.join(DATADIR, f'{hcst_bname}.1m.anom.nc'))\n",
"# anom_3m.to_netcdf(f'{DATADIR}/{hcst_bname}.3m.anom.nc')\n",
"anom_3m.to_netcdf(os.path.join(DATADIR, f'{hcst_bname}.3m.anom.nc'))"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "978bf201", "id": "62af30c5",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 2b. Probabilities for tercile categories" "### 2b. Probabilities for tercile categories"
@ -517,7 +660,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "896f1ece", "id": "75fb94c0",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Before we start computing the probabilities it will be useful to create a function devoted to computing the boundaries of a given category (`icat`) coming from a given set of quantiles (`quantiles`) of an `xr.Dataset` object." "Before we start computing the probabilities it will be useful to create a function devoted to computing the boundaries of a given category (`icat`) coming from a given set of quantiles (`quantiles`) of an `xr.Dataset` object."
@ -526,7 +669,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "8faece60", "id": "721ed6ba",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -553,7 +696,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "706afa29", "id": "c64b2ceb",
"metadata": {}, "metadata": {},
"source": [ "source": [
"We will start by creating a list with the quantiles which will be the boundaries for our categorical forecast. In this example our product will be based on 3 categories ('below lower tercile', 'normal' and 'above upper tercile') so we will be using as category boundaries the values `quantiles = [1/3., 2/3.]`\n", "We will start by creating a list with the quantiles which will be the boundaries for our categorical forecast. In this example our product will be based on 3 categories ('below lower tercile', 'normal' and 'above upper tercile') so we will be using as category boundaries the values `quantiles = [1/3., 2/3.]`\n",
@ -564,7 +707,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "301a7262", "id": "3a4d0fba",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -591,12 +734,13 @@
" print(f'Concatenating {aggr} tercile probs categories')\n", " print(f'Concatenating {aggr} tercile probs categories')\n",
" probs = xr.concat(l_probs_hcst,dim='category') \n", " probs = xr.concat(l_probs_hcst,dim='category') \n",
" print(f'Saving {aggr} tercile probs netCDF files')\n", " print(f'Saving {aggr} tercile probs netCDF files')\n",
" probs.to_netcdf(f'{DATADIR}/{hcst_bname}.{aggr}.tercile_probs.nc')" " # probs.to_netcdf(f'{DATADIR}/{hcst_bname}.{aggr}.tercile_probs.nc')\n",
" probs.to_netcdf(os.path.join(DATADIR, f'{hcst_bname}.{aggr}.tercile_probs.nc'))"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "534ca412", "id": "45bb14c2",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 3. Compute deterministic and probabilistic scores" "## 3. Compute deterministic and probabilistic scores"
@ -604,7 +748,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "1cc0d476", "id": "6e5ee40f",
"metadata": {}, "metadata": {},
"source": [ "source": [
"In this section we will produce a set of metrics and scores consistent with the [guidance on forecast verification](https://library.wmo.int/doc_num.php?explnum_id=4886) provided by WMO. Those values can be used to describe statistics of forecast performance over a past period (the reference period defined in the variable `config`), as an indication of expectation of 'confidence' in the real-time outputs.\n", "In this section we will produce a set of metrics and scores consistent with the [guidance on forecast verification](https://library.wmo.int/doc_num.php?explnum_id=4886) provided by WMO. Those values can be used to describe statistics of forecast performance over a past period (the reference period defined in the variable `config`), as an indication of expectation of 'confidence' in the real-time outputs.\n",
@ -621,7 +765,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "e3ddd83f", "id": "bf7a97d4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 3a. Read observations data into a xr.Dataset" "### 3a. Read observations data into a xr.Dataset"
@ -629,7 +773,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "bb8b7755", "id": "c5093f24",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Before any score can be calculated we will need to read into an `xr.Dataset` object the observational data from the GRIB file we downloaded in the first section of this notebook (whose name was stored in variable `obs_fname`).\n", "Before any score can be calculated we will need to read into an `xr.Dataset` object the observational data from the GRIB file we downloaded in the first section of this notebook (whose name was stored in variable `obs_fname`).\n",
@ -645,7 +789,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "06c3f1ab", "id": "71bc4e03",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -674,7 +818,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "9e894315", "id": "0292bce6",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 3b. Compute deterministic scores" "### 3b. Compute deterministic scores"
@ -682,7 +826,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7f647452", "id": "6292be5a",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The score used here is temporal correlation (Spearman rank correlation) calculated at each grid point and for each leadtime.\n", "The score used here is temporal correlation (Spearman rank correlation) calculated at each grid point and for each leadtime.\n",
@ -696,7 +840,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "bb128ae2", "id": "3fd7c41d",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-info\">\n", "<div class=\"alert alert-block alert-info\">\n",
@ -708,7 +852,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "9cbd5cf6", "id": "423bda99",
"metadata": { "metadata": {
"scrolled": true "scrolled": true
}, },
@ -727,7 +871,8 @@
" print(f'Computing deterministic scores for {aggr}-aggregation')\n", " print(f'Computing deterministic scores for {aggr}-aggregation')\n",
"\n", "\n",
" # Read anomalies file\n", " # Read anomalies file\n",
" h = xr.open_dataset(f'{DATADIR}/{hcst_bname}.{aggr}.anom.nc' )\n", " # h = xr.open_dataset(f'{DATADIR}/{hcst_bname}.{aggr}.anom.nc' )\n",
" h = xr.open_dataset(os.path.join(DATADIR, f'{hcst_bname}.{aggr}.anom.nc'))\n",
" is_fullensemble = 'number' in h.dims\n", " is_fullensemble = 'number' in h.dims\n",
"\n", "\n",
" l_corr=list()\n", " l_corr=list()\n",
@ -746,13 +891,15 @@
" corr_pval=xr.concat(l_corr_pval,dim='forecastMonth')\n", " corr_pval=xr.concat(l_corr_pval,dim='forecastMonth')\n",
"\n", "\n",
" print(f'Saving to netCDF file correlation for {aggr}-aggregation') \n", " print(f'Saving to netCDF file correlation for {aggr}-aggregation') \n",
" corr.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.corr.nc')\n", " # corr.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.corr.nc')\n",
" corr_pval.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.corr_pval.nc')\n" " corr.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.corr.nc'))\n",
" # corr_pval.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.corr_pval.nc')\n",
" corr_pval.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.corr_pval.nc'))\n"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "59707591", "id": "6a0d422b",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 3c. Compute probabilistic scores for tercile categories" "### 3c. Compute probabilistic scores for tercile categories"
@ -760,7 +907,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ee4f0e84", "id": "2b958bfe",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The scores used here for probabilistic forecasts are the area under the Relative Operating Characteristic (ROC) curve, the Ranked Probability Score (RPS) and the Brier Score (BS). Note that both ROC and BS are appropriate for binary events (applied individually to each category), while RPS scores all categories together. \n", "The scores used here for probabilistic forecasts are the area under the Relative Operating Characteristic (ROC) curve, the Ranked Probability Score (RPS) and the Brier Score (BS). Note that both ROC and BS are appropriate for binary events (applied individually to each category), while RPS scores all categories together. \n",
@ -774,7 +921,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "23bc7c79", "id": "0c66e277",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-info\">\n", "<div class=\"alert alert-block alert-info\">\n",
@ -786,7 +933,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "e095e10f", "id": "6c3dcb7d",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -803,7 +950,8 @@
" print(f'Computing deterministic scores for {aggr}-aggregation')\n", " print(f'Computing deterministic scores for {aggr}-aggregation')\n",
" \n", " \n",
" # READ hindcast probabilities file\n", " # READ hindcast probabilities file\n",
" probs_hcst = xr.open_dataset(f'{DATADIR}/{hcst_bname}.{aggr}.tercile_probs.nc')\n", " # probs_hcst = xr.open_dataset(f'{DATADIR}/{hcst_bname}.{aggr}.tercile_probs.nc')\n",
" probs_hcst = xr.open_dataset(os.path.join(DATADIR, f'{hcst_bname}.{aggr}.tercile_probs.nc'))\n",
"\n", "\n",
" l_roc=list()\n", " l_roc=list()\n",
" l_rps=list()\n", " l_rps=list()\n",
@ -862,18 +1010,22 @@
" bs=xr.concat(l_bs,dim='forecastMonth')\n", " bs=xr.concat(l_bs,dim='forecastMonth')\n",
"\n", "\n",
" print('writing to netcdf rps')\n", " print('writing to netcdf rps')\n",
" rps.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rps.nc')\n", " # rps.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rps.nc')\n",
" rps.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.rps.nc'))\n",
" print('writing to netcdf bs')\n", " print('writing to netcdf bs')\n",
" bs.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.bs.nc')\n", " # bs.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.bs.nc')\n",
" bs.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.bs.nc'))\n",
" print('writing to netcdf roc')\n", " print('writing to netcdf roc')\n",
" roc.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.roc.nc')\n", " # roc.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.roc.nc')\n",
" roc.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.roc.nc'))\n",
" print('writing to netcdf rocss')\n", " print('writing to netcdf rocss')\n",
" rocss.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rocss.nc')" " # rocss.to_netcdf(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rocss.nc')\n",
" rocss.to_netcdf(os.path.join(SCOREDIR, f'{hcst_bname}.{aggr}.rocss.nc'))"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "735e4fc4", "id": "1411b8e9",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 4. Visualize verification plots" "## 4. Visualize verification plots"
@ -881,7 +1033,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7d4b6646", "id": "b935869f",
"metadata": {}, "metadata": {},
"source": [ "source": [
"After we have computed some metrics and scores for our seasonal forecast data, in this section some examples of visualization will be introduced." "After we have computed some metrics and scores for our seasonal forecast data, in this section some examples of visualization will be introduced."
@ -889,7 +1041,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "0671493a-278e-4ec6-9e70-772a58df4451", "id": "c0289771",
"metadata": {}, "metadata": {},
"source": [ "source": [
"<div class=\"alert alert-block alert-success\">\n", "<div class=\"alert alert-block alert-success\">\n",
@ -900,7 +1052,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "6a5464da", "id": "01117acd",
"metadata": { "metadata": {
"tags": [] "tags": []
}, },
@ -918,7 +1070,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "0fa38104", "id": "7fedbcf6",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -943,7 +1095,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7a41b4b9", "id": "7c9b63fa",
"metadata": {}, "metadata": {},
"source": [ "source": [
"#### Create plot titles base information\n", "#### Create plot titles base information\n",
@ -956,7 +1108,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "c025046f", "id": "4444e737",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -981,7 +1133,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "57bb0fa9", "id": "5a2a59a0",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 4a. Correlation" "### 4a. Correlation"
@ -990,7 +1142,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "e2839460", "id": "523d201a",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1043,7 +1195,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "2bc0e1e6", "id": "cb4a706b",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 4b. Ranked Probability Score (RPS)" "### 4b. Ranked Probability Score (RPS)"
@ -1052,12 +1204,13 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "dddc9970", "id": "3887913f",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"# READ scores xr.Datasets\n", "# READ scores xr.Datasets\n",
"rps = xr.open_dataset(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rps.nc')\n", "# rps = xr.open_dataset(f'{DATADIR}/scores/{hcst_bname}.{aggr}.rps.nc')\n",
"rps = xr.open_dataset(os.path.join(SCOREDIR), f'{hcst_bname}.{aggr}.rps.nc')\n",
"# RE-ARRANGE the DATASETS longitude values for plotting purposes\n", "# RE-ARRANGE the DATASETS longitude values for plotting purposes\n",
"rps = rps.assign_coords(lon=(((rps.lon + 180) % 360) - 180)).sortby('lon')\n", "rps = rps.assign_coords(lon=(((rps.lon + 180) % 360) - 180)).sortby('lon')\n",
"\n", "\n",
@ -1075,13 +1228,14 @@
" cb.ax.set_ylabel('RPS',fontsize=12)\n", " cb.ax.set_ylabel('RPS',fontsize=12)\n",
" plt.title(tit_line1 + f' {VARNAMES[var]}' + ' (tercile categories)\\n' + tit_line2,loc='left')\n", " plt.title(tit_line1 + f' {VARNAMES[var]}' + ' (tercile categories)\\n' + tit_line2,loc='left')\n",
" plt.tight_layout() \n", " plt.tight_layout() \n",
" figname = f'{DATADIR}/plots/stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.rps.png'\n", " # figname = f'{DATADIR}/plots/stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.rps.png'\n",
" figname = os.path.joint(PLOTSDIR, f'stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.rps.png')\n",
" plt.savefig(figname) \n" " plt.savefig(figname) \n"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "1aae8404", "id": "5407dcb9",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### 4c. Area under Relative Operating Characteristic (ROC) curve" "### 4c. Area under Relative Operating Characteristic (ROC) curve"
@ -1090,7 +1244,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "9223a873", "id": "d2db367e",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1114,22 +1268,41 @@
" cb.ax.set_ylabel('Area under ROC curve',fontsize=12)\n", " cb.ax.set_ylabel('Area under ROC curve',fontsize=12)\n",
" plt.title(tit_line1 + f' {VARNAMES[var]}' + f' ({CATNAMES[icat]})\\n' + tit_line2, loc='left')\n", " plt.title(tit_line1 + f' {VARNAMES[var]}' + f' ({CATNAMES[icat]})\\n' + tit_line2, loc='left')\n",
" plt.tight_layout() \n", " plt.tight_layout() \n",
" figname = f'{DATADIR}/plots/stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.category{icat}.roc.png'\n", " # figname = f'{DATADIR}/plots/stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.category{icat}.roc.png'\n",
" figname = os.path.join(PLOTSDIR, f'stmonth{config[\"start_month\"]:02d}/{hcst_bname}.{aggr}.fcmonth{fcmonth}.{var}.category{icat}.roc.png')\n",
" plt.savefig(figname) " " plt.savefig(figname) "
] ]
}, },
{
"cell_type": "markdown",
"id": "c5376451",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "f6f85ce2",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "826e08ab", "id": "598b1d21",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [] "source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
"kernelspec": { "kernelspec": {
"display_name": "Python 3 (ipykernel)", "display_name": "Python 3",
"language": "python", "language": "python",
"name": "python3" "name": "python3"
}, },
@ -1143,7 +1316,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.10.9" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

1169
tutorials/04_seasonal_forecast/04x02_sf-verification_NOT_WORKING.ipynb
View File

File diff suppressed because it is too large Load Diff

198
tutorials/05_climate_index/05x01_ci-windchill.ipynb
View File

@ -4,7 +4,22 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Wind Chill Index with Reanalysis Data\n", "# Wind Chill Index with Reanalysis Data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_PROJECT = \"05x01_ci-windchill\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n", "\n",
"#### About\n", "#### About\n",
"\n", "\n",
@ -58,7 +73,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -94,10 +109,35 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = cds_datadir(\"ci-windchill\")\n", "import os\n",
"DATADIR = cds_datadir(CDS_PROJECT)\n",
"print(DATADIR)" "print(DATADIR)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -124,7 +164,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install cdsapi" "# !pip install cdsapi"
] ]
}, },
{ {
@ -233,30 +273,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client(url=URL, key=KEY)\n", "c = cdsapi.Client()"
"c.retrieve(\n",
" 'reanalysis-uerra-europe-single-levels',\n",
" {\n",
" 'origin': 'uerra_harmonie',\n",
" 'variable': '10m_wind_speed',\n",
" 'year': [\n",
" '1989', '1990', '1991',\n",
" '1992', '1993', '1994',\n",
" '1995', '1996', '1997',\n",
" '1998', '1999', '2000',\n",
" '2001', '2002', '2003',\n",
" '2004', '2005', '2006',\n",
" '2007', '2008', '2009',\n",
" '2010', '2011', '2012',\n",
" '2013', '2014', '2015',\n",
" '2016', '2017', '2018',\n",
" ],\n",
" 'month': '01',\n",
" 'day': '15',\n",
" 'time': '06:00',\n",
" 'data_format': 'netcdf_legacy',\n",
" },\n",
" f'{DATADIR}UERRA_ws10m.nc')"
] ]
}, },
{ {
@ -265,30 +282,81 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"c = cdsapi.Client()\n", "\n",
"c.retrieve(\n", "fw = os.path.join(CDS_DATA, \"UERRA_ws10m.nc\")\n",
" 'reanalysis-uerra-europe-single-levels',\n", "\n",
" {\n", "#restore execution\n",
" 'origin': 'uerra_harmonie',\n", "if not os.path.exists(fw):\n",
" 'variable': '2m_temperature',\n", " pending_id = \"0a24c457-eb7c-47ae-87a3-d6973fb38adb\"\n",
" 'year': [\n", " print (\"trying to resume\", pending_id)\n",
" '1989', '1990', '1991',\n", " \n",
" '1992', '1993', '1994',\n", " c.client.download_results(pending_id, fw)\n",
" '1995', '1996', '1997',\n", "\n",
" '1998', '1999', '2000',\n", "if not os.path.exists(fw):\n",
" '2001', '2002', '2003',\n", " print(\"Downloading data\", fw)\n",
" '2004', '2005', '2006',\n", " c.retrieve(\n",
" '2007', '2008', '2009',\n", " 'reanalysis-uerra-europe-single-levels',\n",
" '2010', '2011', '2012',\n", " {\n",
" '2013', '2014', '2015',\n", " 'origin': 'uerra_harmonie',\n",
" '2016', '2017', '2018',\n", " 'variable': '10m_wind_speed',\n",
" ],\n", " 'year': [\n",
" 'month': '01',\n", " '1989', '1990', '1991',\n",
" 'day': '15',\n", " '1992', '1993', '1994',\n",
" 'time': '06:00',\n", " '1995', '1996', '1997',\n",
" 'data_format': 'netcdf_legacy',\n", " '1998', '1999', '2000',\n",
" },\n", " '2001', '2002', '2003',\n",
" f'{DATADIR}UERRA_t2m.nc')" " '2004', '2005', '2006',\n",
" '2007', '2008', '2009',\n",
" '2010', '2011', '2012',\n",
" '2013', '2014', '2015',\n",
" '2016', '2017', '2018',\n",
" ],\n",
" 'month': '01',\n",
" 'day': '15',\n",
" 'time': '06:00',\n",
" 'data_format': 'netcdf_legacy',\n",
" },\n",
" fw)\n",
"else:\n",
" print(\"File exists\", fw)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"ft = os.path.join(CDS_DATA, \"UERRA_t2m.nc\")\n",
"\n",
"if not os.path.exists(ft):\n",
" print(\"Downloading data\", ft)\n",
" c.retrieve(\n",
" 'reanalysis-uerra-europe-single-levels',\n",
" {\n",
" 'origin': 'uerra_harmonie',\n",
" 'variable': '2m_temperature',\n",
" 'year': [\n",
" '1989', '1990', '1991',\n",
" '1992', '1993', '1994',\n",
" '1995', '1996', '1997',\n",
" '1998', '1999', '2000',\n",
" '2001', '2002', '2003',\n",
" '2004', '2005', '2006',\n",
" '2007', '2008', '2009',\n",
" '2010', '2011', '2012',\n",
" '2013', '2014', '2015',\n",
" '2016', '2017', '2018',\n",
" ],\n",
" 'month': '01',\n",
" 'day': '15',\n",
" 'time': '06:00',\n",
" 'data_format': 'netcdf_legacy',\n",
" },\n",
" ft)\n",
"else:\n",
" print(\"File exists\", ft)"
] ]
}, },
{ {
@ -306,8 +374,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"fw = f'{DATADIR}UERRA_ws10m.nc'\n", "# fw = f'{DATADIR}UERRA_ws10m.nc'\n",
"ft = f'{DATADIR}UERRA_t2m.nc'\n", "# ft = f'{DATADIR}UERRA_t2m.nc'\n",
"\n", "\n",
"# Create Xarray Dataset\n", "# Create Xarray Dataset\n",
"dw = xr.open_dataset(fw)\n", "dw = xr.open_dataset(fw)\n",
@ -494,7 +562,9 @@
"cbar = plt.colorbar(im,fraction=0.04, pad=0.01)\n", "cbar = plt.colorbar(im,fraction=0.04, pad=0.01)\n",
"cbar.set_label('Wind Chill Index') \n", "cbar.set_label('Wind Chill Index') \n",
"\n", "\n",
"fig.savefig(f'{DATADIR}UERRA_wind_chill_index_midJan.png')" "# fig.savefig(f'{DATADIR}UERRA_wind_chill_index_midJan.png')\n",
"figname = os.path.join(DATADIR, \"UERRA_wind_chill_index_midJan.png\")\n",
"fig.savefig(figname)"
] ]
}, },
{ {
@ -528,6 +598,22 @@
"source": [ "source": [
"<hr>" "<hr>"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -546,7 +632,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,

35
tutorials/06_bias_correction/06x01 Getting Started.ipynb
View File

@ -350,11 +350,37 @@
"source": [ "source": [
"The variable tas_fut_debiased_ISIMIP now contains the bias corrected climate model data for the period 2065-2100. Congratulations!" "The variable tas_fut_debiased_ISIMIP now contains the bias corrected climate model data for the period 2065-2100. Congratulations!"
] ]
},
{
"cell_type": "markdown",
"id": "b934aa04",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "1b36fed5",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07823758",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
"kernelspec": { "kernelspec": {
"display_name": "Python 3 (ipykernel)", "display_name": "venv",
"language": "python", "language": "python",
"name": "python3" "name": "python3"
}, },
@ -368,12 +394,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.13" "version": "3.13.0"
},
"vscode": {
"interpreter": {
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
}
} }
}, },
"nbformat": 4, "nbformat": 4,

26
tutorials/06_bias_correction/06x02 Adjusting Debiasers.ipynb
View File

@ -528,6 +528,32 @@
"source": [ "source": [
"**How to know which parameters to change**: Some debiasers such as ISIMIP have many parameters to control the behavior, but oftentimes only a few are central. The documentation for each of the debiasers provides an indication of the central and required parameters and how they modify the debiasing behavior." "**How to know which parameters to change**: Some debiasers such as ISIMIP have many parameters to control the behavior, but oftentimes only a few are central. The documentation for each of the debiasers provides an indication of the central and required parameters and how they modify the debiasing behavior."
] ]
},
{
"cell_type": "markdown",
"id": "74086a7a",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "4663a1f9",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a61dc80c",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {

26
tutorials/06_bias_correction/06x03 Evaluation.ipynb
View File

@ -781,6 +781,32 @@
"\n", "\n",
"We hope this notebook provides helpful guidance for you to evaluate your own bias adjustment project and choose the methodology most suited to your application." "We hope this notebook provides helpful guidance for you to evaluate your own bias adjustment project and choose the methodology most suited to your application."
] ]
},
{
"cell_type": "markdown",
"id": "6c899e39",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "7d19c888",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "088f7888",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {

26
tutorials/06_bias_correction/06x04 Parallelization and Advanced Topics.ipynb
View File

@ -378,6 +378,32 @@
"source": [ "source": [
"The LinearScaling debiaser set up here includes a running window functionality. If this is not required then we could also subclass the `Debiaser` instead of `RunningWindowDebiaser` to set up a new debiaser." "The LinearScaling debiaser set up here includes a running window functionality. If this is not required then we could also subclass the `Debiaser` instead of `RunningWindowDebiaser` to set up a new debiaser."
] ]
},
{
"cell_type": "markdown",
"id": "bdf3e376",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "7ad18a61",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "695972d2",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {

177
tutorials/06_bias_correction/6x00 Download and Preprocess.ipynb
View File

@ -8,6 +8,17 @@
"# Sorting pre-requisits for ibicus: downloading and preprocessing data" "# Sorting pre-requisits for ibicus: downloading and preprocessing data"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"id": "4d227eed",
"metadata": {},
"outputs": [],
"source": [
"\n",
"CDS_PROJECT = \"06x00_data_download_and_preprocessing\""
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "46c1c6de", "id": "46c1c6de",
@ -57,7 +68,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "# !pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
}, },
{ {
@ -90,6 +101,16 @@
"---" "---"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"id": "fa0848be",
"metadata": {},
"outputs": [],
"source": [
"# !pip install scitools-iris"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
@ -127,14 +148,41 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "0c707ab7", "id": "0701fcf7",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"DATADIR = \"data_download_and_preprocessing\"\n", "import os\n",
"\n", "\n",
"if not os.path.exists(DATADIR):\n", "DATADIR = cds_datadir(CDS_PROJECT)\n",
" os.mkdir(DATADIR)" "print(DATADIR)"
]
},
{
"cell_type": "markdown",
"id": "0d9ba17b",
"metadata": {},
"source": [
"to avoid to download already downloaded data, we put them in a local folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "55fd3c03",
"metadata": {},
"outputs": [],
"source": [
"CDS_DATA = os.path.join(\"../data/\", CDS_PROJECT)\n",
"os.makedirs(CDS_DATA, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"id": "7bd0d91e",
"metadata": {},
"source": [
"use `CDS_DATA = DATADIR` if you want to avoid the use of shared download folder"
] ]
}, },
{ {
@ -213,9 +261,18 @@
"\n", "\n",
"# choose a historical period to extract\n", "# choose a historical period to extract\n",
"period_hist = '1979-01-01/2005-12-31' \n", "period_hist = '1979-01-01/2005-12-31' \n",
"start_year_hist = 1979\n",
"end_year_hist = 2005\n",
"years_hist = [str(year) for year in range(start_year_hist, end_year_hist+1)]\n",
"\n", "\n",
"# choose a future period to extract:\n", "# choose a future period to extract:\n",
"period_fut = '2050-01-01/2070-12-31'\n", "period_fut = '2050-01-01/2070-12-31'\n",
"start_year_fut = 2050\n",
"end_year_fut = 2070\n",
"years_fut = [str(year) for year in range(start_year_fut, end_year_fut+1)]\n",
"\n",
"months = [str(month).zfill(2) for month in range(1, 13)]\n",
"\n",
"\n", "\n",
"# choose a filename for the historical cm data\n", "# choose a filename for the historical cm data\n",
"fname_cm_hist = f\"cmip6_daily_1979-2015_ipsl_historical_{variable}.zip\"\n", "fname_cm_hist = f\"cmip6_daily_1979-2015_ipsl_historical_{variable}.zip\"\n",
@ -258,21 +315,37 @@
"source": [ "source": [
"# download historical climate model data\n", "# download historical climate model data\n",
"\n", "\n",
"c = cdsapi.Client(url=URL, key=KEY)\n", "c = cdsapi.Client()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d673417",
"metadata": {},
"outputs": [],
"source": [
"zipfile_fname_cm_hist = os.path.join(CDS_DATA, fname_cm_hist)\n",
"\n", "\n",
"c.retrieve(\n", "if not os.exists(zipfile_fname_cm_hist):\n",
" 'projections-cmip6',\n", " print(f\"Downloading {zipfile_fname_cm_hist}\")\n",
" {\n", " c.retrieve(\n",
" 'temporal_resolution': 'daily',\n", " 'projections-cmip6',\n",
" 'experiment': 'historical',\n", " {\n",
" 'level': 'single_levels',\n", " 'temporal_resolution': 'daily',\n",
" 'variable': variable,\n", " 'experiment': 'historical',\n",
" 'model': model,\n", " 'level': 'single_levels',\n",
" 'date': period_hist,\n", " 'variable': variable,\n",
" 'area': area,\n", " 'model': model,\n",
" 'format': 'zip',\n", " # 'date': period_hist,\n",
" },\n", " 'year': years_hist,\n",
" f'{DATADIR}/{fname_cm_hist}')" " 'month': months,\n",
" 'area': area,\n",
" 'format': 'zip',\n",
" },\n",
" zipfile_fname_cm_hist)\n",
"else:\n",
" print(f\"File {zipfile_fname_cm_hist} already exists\")"
] ]
}, },
{ {
@ -292,7 +365,7 @@
"source": [ "source": [
"import zipfile\n", "import zipfile\n",
"\n", "\n",
"with zipfile.ZipFile(f'{DATADIR}/{fname_cm_hist}', 'r') as zip_ref:\n", "with zipfile.ZipFile(zipfile_fname_cm_hist, 'r') as zip_ref:\n",
" zip_ref.extractall(DATADIR)" " zip_ref.extractall(DATADIR)"
] ]
}, },
@ -329,21 +402,27 @@
"source": [ "source": [
"# download future climate model data\n", "# download future climate model data\n",
"\n", "\n",
"c = cdsapi.Client(url=URL, key=KEY)\n", "zipfile_cm_future = os.path.join(CDS_DATA, fname_cm_future) \n",
"\n", "\n",
"c.retrieve(\n", "if not os.path.exists(zipfile_cm_future):\n",
" 'projections-cmip6',\n", " print(f\"Downloading {zipfile_cm_future}\")\n",
" {\n", " c.retrieve(\n",
" 'temporal_resolution': 'daily',\n", " 'projections-cmip6',\n",
" 'experiment': 'ssp5_8_5',\n", " {\n",
" 'level': 'single_levels',\n", " 'temporal_resolution': 'daily',\n",
" 'variable': variable,\n", " 'experiment': 'ssp5_8_5',\n",
" 'model': model,\n", " 'level': 'single_levels',\n",
" 'date': period_fut,\n", " 'variable': variable,\n",
" 'area': area,\n", " 'model': model,\n",
" 'format': 'zip',\n", " # 'date': period_fut,\n",
" },\n", " 'year': years_fut,\n",
" f'{DATADIR}/{fname_cm_future}')" " 'month': months,\n",
" 'area': area,\n",
" 'format': 'zip',\n",
" },\n",
" zipfile_cm_future)\n",
"else:\n",
" print(f\"File {zipfile_cm_future} already exists\")"
] ]
}, },
{ {
@ -363,7 +442,7 @@
"source": [ "source": [
"import zipfile\n", "import zipfile\n",
"\n", "\n",
"with zipfile.ZipFile(f'{DATADIR}/{fname_cm_future}', 'r') as zip_ref:\n", "with zipfile.ZipFile(zipfile_cm_future, 'r') as zip_ref:\n",
" zip_ref.extractall(DATADIR)" " zip_ref.extractall(DATADIR)"
] ]
}, },
@ -821,6 +900,32 @@
"debiased_cm_future_era5 = debiaser.apply(obs_era5, cm_hist, cm_future)\n", "debiased_cm_future_era5 = debiaser.apply(obs_era5, cm_hist, cm_future)\n",
"debiased_cm_future_ncep_doe = debiaser.apply(obs_ncep_doe, cm_hist, cm_future)" "debiased_cm_future_ncep_doe = debiaser.apply(obs_ncep_doe, cm_hist, cm_future)"
] ]
},
{
"cell_type": "markdown",
"id": "a7708b04",
"metadata": {},
"source": [
"<hr>"
]
},
{
"cell_type": "markdown",
"id": "c6af2e47",
"metadata": {},
"source": [
"Output files saved in: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a3ffaff3",
"metadata": {},
"outputs": [],
"source": [
"DATADIR"
]
} }
], ],
"metadata": { "metadata": {
@ -839,7 +944,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.12"
} }
}, },
"nbformat": 4, "nbformat": 4,