D4Science
/
d4science_copernicus_cds
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: D4Science:29a8aa136d463243cbad2d10c2bc808c84e911c3
Branches Tags
D4Science:main
...
compare: D4Science:f1ee02ab2b818dd3b0e6578536fc848752ae2b38
Branches Tags
D4Science:main

3 Commits
29a8aa136d ... f1ee02ab2b

Author SHA1 Message Date
Alfredo Oliviero f1ee02ab2b removed test notebook 2024-11-03 10:04:23 +01:00
Alfredo Oliviero 884d75fd2f src 2024-11-03 10:03:53 +01:00
Alfredo Oliviero a66b5a7af0 readme & co 2024-11-03 10:03:20 +01:00
11 changed files with 350 additions and 3128 deletions
Show Stats Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -1,5 +1,5 @@
# Changelog for "d4science_auth_cds"
## [1.0.0]
## [v1.0.0]
- first release

44
FUNDING.md
View File

@ -3,24 +3,26 @@
The projects leading to this software have received funding from a series of European Union programmes including:
- the Sixth Framework Programme for Research and Technological Development
- [DILIGENT](https://cordis.europa.eu/project/id/004260) (grant no. 004260).
- the Seventh Framework Programme for research, technological development and demonstration
- [D4Science](https://cordis.europa.eu/project/id/212488) (grant no. 212488);
- [D4Science-II](https://cordis.europa.eu/project/id/239019) (grant no.239019);
- [ENVRI](https://cordis.europa.eu/project/id/283465) (grant no. 283465);
- [iMarine](https://cordis.europa.eu/project/id/283644) (grant no. 283644);
- [EUBrazilOpenBio](https://cordis.europa.eu/project/id/288754) (grant no. 288754).
- the H2020 research and innovation programme
- [SoBigData](https://cordis.europa.eu/project/id/654024) (grant no. 654024);
- [PARTHENOS](https://cordis.europa.eu/project/id/654119) (grant no. 654119);
- [EGI-Engage](https://cordis.europa.eu/project/id/654142) (grant no. 654142);
- [ENVRI PLUS](https://cordis.europa.eu/project/id/654182) (grant no. 654182);
- [BlueBRIDGE](https://cordis.europa.eu/project/id/675680) (grant no. 675680);
- [PerformFISH](https://cordis.europa.eu/project/id/727610) (grant no. 727610);
- [AGINFRA PLUS](https://cordis.europa.eu/project/id/731001) (grant no. 731001);
- [DESIRA](https://cordis.europa.eu/project/id/818194) (grant no. 818194);
- [ARIADNEplus](https://cordis.europa.eu/project/id/823914) (grant no. 823914);
- [RISIS 2](https://cordis.europa.eu/project/id/824091) (grant no. 824091);
- [EOSC-Pillar](https://cordis.europa.eu/project/id/857650) (grant no. 857650);
- [Blue Cloud](https://cordis.europa.eu/project/id/862409) (grant no. 862409);
- [SoBigData-PlusPlus](https://cordis.europa.eu/project/id/871042) (grant no. 871042);
- [DILIGENT](https://cordis.europa.eu/project/id/004260) (grant no. 004260)
- the Seventh Framework Programme for research, technological development and demonstration
- [D4Science](https://cordis.europa.eu/project/id/212488) (grant no. 212488)
- [D4Science-II](https://cordis.europa.eu/project/id/239019) (grant no.239019)
- [ENVRI](https://cordis.europa.eu/project/id/283465) (grant no. 283465)
- [iMarine](https://cordis.europa.eu/project/id/283644) (grant no. 283644)
- [EUBrazilOpenBio](https://cordis.europa.eu/project/id/288754) (grant no. 288754)
- the H2020 research and innovation programme
- [SoBigData](https://cordis.europa.eu/project/id/654024) (grant no. 654024)
- [PARTHENOS](https://cordis.europa.eu/project/id/654119) (grant no. 654119)
- [EGI-Engage](https://cordis.europa.eu/project/id/654142) (grant no. 654142)
- [ENVRI PLUS](https://cordis.europa.eu/project/id/654182) (grant no. 654182)
- [BlueBRIDGE](https://cordis.europa.eu/project/id/675680) (grant no. 675680)
- [PerformFISH](https://cordis.europa.eu/project/id/727610) (grant no. 727610)
- [AGINFRA PLUS](https://cordis.europa.eu/project/id/731001) (grant no. 731001)
- [DESIRA](https://cordis.europa.eu/project/id/818194) (grant no. 818194)
- [ARIADNEplus](https://cordis.europa.eu/project/id/823914) (grant no. 823914)
- [RISIS 2](https://cordis.europa.eu/project/id/824091) (grant no. 824091)
- [EOSC-Pillar](https://cordis.europa.eu/project/id/857650) (grant no. 857650)
- [Blue Cloud](https://cordis.europa.eu/project/id/862409) (grant no. 862409)
- [SoBigData-PlusPlus](https://cordis.europa.eu/project/id/871042) (grant no. 871042)

36
README.md
View File

@ -8,25 +8,33 @@ v.1.0.0
## Installation
Install the library directly from the D4Science Git
Python: Install the library directly from the D4Science Git
```
```sh
pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git
```
Jupyter notebook: install using the `!` prefix
```sh
! pip install git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git
```
## Built With
* [Copernicus CDSAPI](https://cds.climate.copernicus.eu/how-to-api/) - the Copernicus Climate Data Store (CDS) Application Program Interface (API) client
* [python](https://python.org/ecli)
* [python](https://python.org/)
* [d4science](https://www.d4science.org/)
## Documentation
- **Centralized Authentication**: Configures the JupyterLab environment for automatic CDS authentication without the need to specify credentials in each notebook.
- **Secure Credential Storage**: Optionally saves credentials in a hidden configuration file, enabling silent authentication in future sessions.
- **Flexible Credential Management**: Provides functions to add, view, and remove credentials from the environment and configuration files.
* **Centralized Authentication**: Configures the JupyterLab environment for automatic CDS authentication without the need to specify credentials in each notebook.
* **Secure Credential Storage**: Optionally saves credentials in a hidden configuration file, enabling silent authentication in future sessions.
* **Flexible Credential Management**: Provides functions to add, view, and remove credentials from the environment and configuration files.
### Usage
Usage
Below are the main commands provided by the auth_cds library to manage CDS API authentication and configuration.
First, import the necessary functions:
@ -35,43 +43,43 @@ First, import the necessary functions:
from auth_cds import cds_authenticate, cds_remove_conf, cds_save_conf, cds_remove_env, cds_show_conf
```
Main Commands
Authenticate and get CDS client:
#### Main Commands
**Authenticate and get CDS client:**
This command authenticates and returns a CDS API client instance.
```python
client = cds_authenticate()
```
Save the current or new configuration:
**Save the current or new configuration:**
Saves the configuration to a default hidden file.
```python
cds_save_conf()
```
Remove environment variables for the CDS API:
**Remove environment variables for the CDS API:**
Clears the CDS API credentials from environment variables. To execute, remove the comment symbol (#).
```python
cds_remove_env()
```
Remove configurations from default paths:
**Remove configurations from default paths:**
Removes any saved configuration files. To execute, remove the comment symbol (#).
```python
cds_remove_conf()
```
Show current configuration from environment and files:
**Show current configuration from environment and files:**
Displays the credentials configured in the environment and any found in the saved configuration file.
```python
cds_show_conf()
```
## Change log
See [CHANGELOG.md](CHANGELOG.md).

450
auth_cds.ipynb
View File

@ -1,450 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Configuration of the JupyterLab Environment for CDS\n",
"\n",
"## 1. Install Dependencies\n",
"\n",
"Before we begin, we need to prepare our environment. This includes installing the Climate Data Store (CDS) Application Programming Interface (API) and importing the necessary Python libraries.\n",
"\n",
"### 1.1 Install the CDS API\n",
"\n",
"To install the CDS API, run the following command. We use an exclamation mark (`!`) to pass the command to the shell (rather than to the Python interpreter).\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already up-to-date: cdsapi>=0.7.2 in /opt/conda/lib/python3.8/site-packages (0.7.4)\n",
"Requirement already satisfied, skipping upgrade: cads-api-client>=1.4.7 in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (1.5.0)\n",
"Requirement already satisfied, skipping upgrade: requests>=2.5.0 in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (2.24.0)\n",
"Requirement already satisfied, skipping upgrade: tqdm in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (4.60.0)\n",
"Requirement already satisfied, skipping upgrade: multiurl>=0.3.2 in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (0.3.2)\n",
"Requirement already satisfied, skipping upgrade: attrs in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (24.2.0)\n",
"Requirement already satisfied, skipping upgrade: typing-extensions in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (4.12.2)\n",
"Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (2.10)\n",
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (2020.12.5)\n",
"Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (1.25.10)\n",
"Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (3.0.4)\n",
"Requirement already satisfied, skipping upgrade: pytz in /opt/conda/lib/python3.8/site-packages (from multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (2020.1)\n",
"Requirement already satisfied, skipping upgrade: python-dateutil in /opt/conda/lib/python3.8/site-packages (from multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (2.8.1)\n",
"Requirement already satisfied, skipping upgrade: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil->multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (1.15.0)\n"
]
}
],
"source": [
"!pip install -U 'cdsapi>=0.7.2'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Update Python Libraries\n",
"\n",
"The current `cdsapi` library requires updating the Python libraries `attrs` and `typing_extensions` in the D4Science JupyterHub environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: attrs in /opt/conda/lib/python3.8/site-packages (24.2.0)\n",
"Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.8/site-packages (4.12.2)\n"
]
}
],
"source": [
"!pip install attrs typing_extensions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.3 Check the Import of Required Libraries\n",
"\n",
"Now, let's import the various libraries needed to execute CDS functions."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# CDS API\n",
"import cdsapi\n",
"\n",
"# Libraries for working with multidimensional arrays\n",
"import numpy as np\n",
"import xarray as xr\n",
"\n",
"# Libraries for plotting and visualising data\n",
"import matplotlib.path as mpath\n",
"import matplotlib.pyplot as plt\n",
"import cartopy.crs as ccrs\n",
"from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER\n",
"import cartopy.feature as cfeature\n",
"\n",
"# Disable warnings for data download via API\n",
"import urllib3 \n",
"urllib3.disable_warnings()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Authenticate\n",
"\n",
"The `cdsapi` client allows you to request data from the Climate Data Store (CDS) programmatically via the CDS API. \n",
"\n",
"We will configure JupyterLab with your personal CDS API key so that you can **instantiate the client without needing to enter your credentials directly in the notebook**.\n",
"\n",
"### 2.1 Obtain Your CDS API Key\n",
"\n",
"First, you'll need your CDS API key. This key includes both your personal User ID and the CDS API key. To obtain it, register or log in to the CDS at [https://cds-beta.climate.copernicus.eu](https://cds-beta.climate.copernicus.eu). Then, visit [https://cds-beta.climate.copernicus.eu/how-to-api](https://cds-beta.climate.copernicus.eu/how-to-api) and copy the string of characters listed after \"key\".\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Configure the JupyterLab Environment across Notebooks\n",
"\n",
"The `auth_cds.py` library, located in the current folder, is a D4Science library specifically designed to enable the use of `cdsapi` within D4Science JupyterLab environments. This library prompts you to enter your credentials and sets them in the JupyterLab environment, so any instance of the `cdsapi` client can authenticate automatically without requiring configurations in the notebook itself.\n",
"\n",
"Additionally, the library offers an option to save your credentials. If you choose to save them, it creates a hidden file in your personal workspace. With this file in place, the authentication process will run automatically and silently on subsequent runs, loading credentials from the saved file instead of prompting for input.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To configure our JupyterLab environment, simply import the function, then, execute it to set up authentication.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"from auth_cds import cds_authenticate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When you run `cds_authenticate()`, the function will handle the authentication process as follows:\n",
"\n",
"1. **If your credentials are already configured** (either stored in environment variables or in a hidden configuration file in your workspace), `cds_authenticate()` will automatically detect and load them. This allows the `cdsapi` client to authenticate silently, without prompting you for input.\n",
"\n",
"2. **If your credentials are not yet configured**, `cds_authenticate()` will prompt you to enter them. You’ll be asked for your CDS API URL and key. After entering these details, the function will:\n",
" - Set the credentials in the JupyterLab environment, so the `cdsapi` client can authenticate in this session.\n",
" - Optionally, it will offer to save the credentials in a hidden configuration file in your workspace, so future sessions can load them automatically without additional input.\n",
"\n",
"This configuration ensures that your credentials are securely handled and easily reusable across sessions.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from file ./.cdsapirc: {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:54:37,276 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:54:37,277 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:54:37,277 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:54:37,278 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:54:37,278 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
},
{
"data": {
"text/plain": [
"<cads_api_client.legacy_api_client.LegacyApiClient at 0x7f6d2936d730>"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cds_authenticate()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Instantiate the CDS API Client \n",
"\n",
"From this point on, we can instantiate a new client simply by running:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:56:53,336 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:56:53,337 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:56:53,337 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:56:53,338 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:56:53,338 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
}
],
"source": [
"client = cdsapi.Client()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is no need to explicitly pass the credentials each time, as they are now configured within the JupyterLab environment itself. This means the credentials are not tied to any specific notebook.\n",
"\n",
"Once these commands have been executed—such as by running this notebook—we can open a new notebook, and without needing to import auth_cds, we can instantiate cdsapi.Client() in the same way"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Executing CDS tutorials\n",
"\n",
"once executed the cds_authenticate(), we can execute any notebook provided as tutotorial in https://ecmwf-projects.github.io/copernicus-training-c3s/intro.html simply \n",
"\n",
"* commenting or removing the `Enter your CDS API key` command\n",
"\n",
"```py\n",
"# URL = 'https://cds-beta.climate.copernicus.eu/api'\n",
"# KEY = \"db1f2085-6b8b-42e6-b832-625dfaf831a4\"\n",
"```\n",
"\n",
"* instanciating the client with \n",
"\n",
"```py\n",
"client = cdsapi.Client()\n",
"```\n",
"\n",
"instead of \n",
"```py\n",
"client = cdsapi.Client(URL, KEY)\n",
"```\n",
"\n",
"you can find all the patched versions of the tutorials in the tutorial folder\n",
"\n",
"just run this notebook before executing them to ensure that the JupyterLAB is configured"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Managing Credentials\n",
"\n",
"The `auth_cds` library provides flexible options for managing your CDS API credentials. Below are the primary commands you can use to handle authentication and configuration. First, import the necessary functions:\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"from auth_cds import cds_authenticate, cds_remove_conf, cds_save_conf, cds_remove_env, cds_show_conf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. **Authenticate and get CDS client**: \n",
" This command authenticates and returns a CDS API client instance. "
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:58:06,299 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:58:06,300 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:58:06,300 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:58:06,301 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:58:06,302 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from environment {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
}
],
"source": [
" client = cds_authenticate()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. **Remove configurations from default paths**: \n",
" Removes any saved configuration files. To execute, remove the comment symbol (`#`)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"cds_remove_conf()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. **Save the current or new configuration**: \n",
" Saves the configuration to a default hidden file. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from environment {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n",
"Saved Configuration file ./.cdsapirc\n"
]
}
],
"source": [
"cds_save_conf()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. **Remove environment variables for the CDS API**: \n",
" Clears the CDS API credentials from environment variables. To execute, remove the comment symbol (`#`)."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removed environment variable CDSAPI_URL\n",
"Removed environment variable CDSAPI_KEY\n"
]
}
],
"source": [
"cds_remove_env()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" 5.. **Show current configuration from environment and files**: \n",
" Displays the credentials configured in the environment and any found in the saved configuration file."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cds_show_conf()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

203
d4science_auth_cds/test_auth_cds.ipynb
View File

@ -1,203 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Enter your CDS API key\n",
"\n",
"We will request data from the Climate Data Store (CDS) programmatically with the help of the CDS API. Let us make use of the option to manually set the CDS API credentials. First, you have to define two variables: `URL` and `KEY` which build together your CDS API key. The string of characters that make up your KEY include your personal User ID and CDS API key. To obtain these, first register or login to the CDS (https://cds-beta.climate.copernicus.eu), then visit https://cds-beta.climate.copernicus.eu/how-to-api and copy the string of characters listed after \"key:\". Replace the `#########` below with this string."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install 'cdsapi>=0.7.2'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U cdsapi\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U attrs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U typing_extensions"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import auth_cds"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from file ./.cdsapirc\n"
]
},
{
"data": {
"text/plain": [
"{'url': 'https://cds-beta.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"auth_cds.show_credentials()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from environment CDSAPI_URL - CDSAPI_KEY\n",
"{'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n"
]
}
],
"source": [
"print(auth_cds.config_from_env())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(auth_cds.config_from_file())"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from environment CDSAPI_URL - CDSAPI_KEY\n",
"set envs CDSAPI_URL , CDSAPI_KEY\n"
]
},
{
"data": {
"text/plain": [
"{'url': 'https://cds-beta.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"auth_cds.authenticate()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-30 17:53:32,030 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-30 17:53:32,031 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-30 17:53:32,031 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-30 17:53:32,031 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-30 17:53:32,032 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
}
],
"source": [
"import cdsapi\n",
"client = cdsapi.Client()\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# auth_cds.removeCredentials()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

0
d4science_auth_cds/AuthCDS.py → d4science_copernicus_cds/AuthCDS.py
View File

0
d4science_auth_cds/__init__.py → d4science_copernicus_cds/__init__.py
View File

0
d4science_auth_cds/auth_cds.py → d4science_copernicus_cds/auth_cds.py
View File

0
d4science_auth_cds/result.json → d4science_copernicus_cds/result.json
View File

304
d4science_copernicus_cds/test_auth_cds.ipynb Normal file
View File

@ -0,0 +1,304 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Enter your CDS API key\n",
"\n",
"We will request data from the Climate Data Store (CDS) programmatically with the help of the CDS API. Let us make use of the option to manually set the CDS API credentials. First, you have to define two variables: `URL` and `KEY` which build together your CDS API key. The string of characters that make up your KEY include your personal User ID and CDS API key. To obtain these, first register or login to the CDS (https://cds-beta.climate.copernicus.eu), then visit https://cds-beta.climate.copernicus.eu/how-to-api and copy the string of characters listed after \"key:\". Replace the `#########` below with this string."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"zsh:1: /Users/Alfredo/works/copernicus/d4science_copernicus_cds/venv/bin/pip: bad interpreter: /Users/Alfredo/works/copernicus/d4science_auth_cds/venv/bin/python3.13: no such file or directory\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.12 -m pip install --upgrade pip\u001b[0m\n",
"\u001b[1;31merror\u001b[0m: \u001b[1mexternally-managed-environment\u001b[0m\n",
"\n",
"\u001b[31m×\u001b[0m This environment is externally managed\n",
"\u001b[31m╰─>\u001b[0m To install Python packages system-wide, try brew install\n",
"\u001b[31m \u001b[0m xyz, where xyz is the package you are trying to\n",
"\u001b[31m \u001b[0m install.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python library that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m use a virtual environment:\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m python3 -m venv path/to/venv\n",
"\u001b[31m \u001b[0m source path/to/venv/bin/activate\n",
"\u001b[31m \u001b[0m python3 -m pip install xyz\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python application that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m it may be easiest to use 'pipx install xyz', which will manage a\n",
"\u001b[31m \u001b[0m virtual environment for you. You can install pipx with\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m brew install pipx\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m You may restore the old behavior of pip by passing\n",
"\u001b[31m \u001b[0m the '--break-system-packages' flag to pip, or by adding\n",
"\u001b[31m \u001b[0m 'break-system-packages = true' to your pip.conf file. The latter\n",
"\u001b[31m \u001b[0m will permanently disable this error.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you disable this error, we STRONGLY recommend that you additionally\n",
"\u001b[31m \u001b[0m pass the '--user' flag to pip, or set 'user = true' in your pip.conf\n",
"\u001b[31m \u001b[0m file. Failure to do this can result in a broken Homebrew installation.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m Read more about this behavior here: <https://peps.python.org/pep-0668/>\n",
"\n",
"\u001b[1;35mnote\u001b[0m: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\n",
"\u001b[1;36mhint\u001b[0m: See PEP 668 for the detailed specification.\n"
]
}
],
"source": [
"!pip install 'cdsapi>=0.7.2'"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"zsh:1: /Users/Alfredo/works/copernicus/d4science_copernicus_cds/venv/bin/pip: bad interpreter: /Users/Alfredo/works/copernicus/d4science_auth_cds/venv/bin/python3.13: no such file or directory\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.12 -m pip install --upgrade pip\u001b[0m\n",
"\u001b[1;31merror\u001b[0m: \u001b[1mexternally-managed-environment\u001b[0m\n",
"\n",
"\u001b[31m×\u001b[0m This environment is externally managed\n",
"\u001b[31m╰─>\u001b[0m To install Python packages system-wide, try brew install\n",
"\u001b[31m \u001b[0m xyz, where xyz is the package you are trying to\n",
"\u001b[31m \u001b[0m install.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python library that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m use a virtual environment:\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m python3 -m venv path/to/venv\n",
"\u001b[31m \u001b[0m source path/to/venv/bin/activate\n",
"\u001b[31m \u001b[0m python3 -m pip install xyz\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python application that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m it may be easiest to use 'pipx install xyz', which will manage a\n",
"\u001b[31m \u001b[0m virtual environment for you. You can install pipx with\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m brew install pipx\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m You may restore the old behavior of pip by passing\n",
"\u001b[31m \u001b[0m the '--break-system-packages' flag to pip, or by adding\n",
"\u001b[31m \u001b[0m 'break-system-packages = true' to your pip.conf file. The latter\n",
"\u001b[31m \u001b[0m will permanently disable this error.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you disable this error, we STRONGLY recommend that you additionally\n",
"\u001b[31m \u001b[0m pass the '--user' flag to pip, or set 'user = true' in your pip.conf\n",
"\u001b[31m \u001b[0m file. Failure to do this can result in a broken Homebrew installation.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m Read more about this behavior here: <https://peps.python.org/pep-0668/>\n",
"\n",
"\u001b[1;35mnote\u001b[0m: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\n",
"\u001b[1;36mhint\u001b[0m: See PEP 668 for the detailed specification.\n"
]
}
],
"source": [
"!pip install -U cdsapi\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"zsh:1: /Users/Alfredo/works/copernicus/d4science_copernicus_cds/venv/bin/pip: bad interpreter: /Users/Alfredo/works/copernicus/d4science_auth_cds/venv/bin/python3.13: no such file or directory\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.12 -m pip install --upgrade pip\u001b[0m\n",
"\u001b[1;31merror\u001b[0m: \u001b[1mexternally-managed-environment\u001b[0m\n",
"\n",
"\u001b[31m×\u001b[0m This environment is externally managed\n",
"\u001b[31m╰─>\u001b[0m To install Python packages system-wide, try brew install\n",
"\u001b[31m \u001b[0m xyz, where xyz is the package you are trying to\n",
"\u001b[31m \u001b[0m install.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python library that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m use a virtual environment:\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m python3 -m venv path/to/venv\n",
"\u001b[31m \u001b[0m source path/to/venv/bin/activate\n",
"\u001b[31m \u001b[0m python3 -m pip install xyz\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you wish to install a Python application that isn't in Homebrew,\n",
"\u001b[31m \u001b[0m it may be easiest to use 'pipx install xyz', which will manage a\n",
"\u001b[31m \u001b[0m virtual environment for you. You can install pipx with\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m brew install pipx\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m You may restore the old behavior of pip by passing\n",
"\u001b[31m \u001b[0m the '--break-system-packages' flag to pip, or by adding\n",
"\u001b[31m \u001b[0m 'break-system-packages = true' to your pip.conf file. The latter\n",
"\u001b[31m \u001b[0m will permanently disable this error.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m If you disable this error, we STRONGLY recommend that you additionally\n",
"\u001b[31m \u001b[0m pass the '--user' flag to pip, or set 'user = true' in your pip.conf\n",
"\u001b[31m \u001b[0m file. Failure to do this can result in a broken Homebrew installation.\n",
"\u001b[31m \u001b[0m \n",
"\u001b[31m \u001b[0m Read more about this behavior here: <https://peps.python.org/pep-0668/>\n",
"\n",
"\u001b[1;35mnote\u001b[0m: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\n",
"\u001b[1;36mhint\u001b[0m: See PEP 668 for the detailed specification.\n"
]
}
],
"source": [
"!pip install -U attrs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U typing_extensions"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import auth_cds"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - None None\n",
"env is not configured\n",
"Configuration from file /Users/Alfredo/.cdsapirc: {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from environment: None\n",
"Configuration from file: {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n"
]
}
],
"source": [
"auth_cds.cds_show_conf()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - None None\n",
"env is not configured\n",
"Configuration from file /Users/Alfredo/.cdsapirc: {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"saving config to env\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
},
{
"data": {
"text/plain": [
"{'url': 'https://cds-beta.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"auth_cds.cds_authenticate()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-11-03 10:02:43,960 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-11-03 10:02:43,961 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-11-03 10:02:43,962 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-11-03 10:02:43,962 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-11-03 10:02:43,962 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
}
],
"source": [
"import cdsapi\n",
"client = cdsapi.Client()\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# auth_cds.removeCredentials()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

2439
reanalysis-climatology.ipynb
View File

File diff suppressed because one or more lines are too long