D4Science
/
d4science_copernicus_cds
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

src

This commit is contained in:
Alfredo Oliviero 2024-11-03 10:04:23 +01:00
parent 884d75fd2f
commit d9204dc320
8 changed files with 1 additions and 3438 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

450
auth_cds.ipynb
View File

@ -1,450 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Configuration of the JupyterLab Environment for CDS\n",
"\n",
"## 1. Install Dependencies\n",
"\n",
"Before we begin, we need to prepare our environment. This includes installing the Climate Data Store (CDS) Application Programming Interface (API) and importing the necessary Python libraries.\n",
"\n",
"### 1.1 Install the CDS API\n",
"\n",
"To install the CDS API, run the following command. We use an exclamation mark (`!`) to pass the command to the shell (rather than to the Python interpreter).\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already up-to-date: cdsapi>=0.7.2 in /opt/conda/lib/python3.8/site-packages (0.7.4)\n",
"Requirement already satisfied, skipping upgrade: cads-api-client>=1.4.7 in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (1.5.0)\n",
"Requirement already satisfied, skipping upgrade: requests>=2.5.0 in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (2.24.0)\n",
"Requirement already satisfied, skipping upgrade: tqdm in /opt/conda/lib/python3.8/site-packages (from cdsapi>=0.7.2) (4.60.0)\n",
"Requirement already satisfied, skipping upgrade: multiurl>=0.3.2 in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (0.3.2)\n",
"Requirement already satisfied, skipping upgrade: attrs in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (24.2.0)\n",
"Requirement already satisfied, skipping upgrade: typing-extensions in /opt/conda/lib/python3.8/site-packages (from cads-api-client>=1.4.7->cdsapi>=0.7.2) (4.12.2)\n",
"Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (2.10)\n",
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (2020.12.5)\n",
"Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (1.25.10)\n",
"Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests>=2.5.0->cdsapi>=0.7.2) (3.0.4)\n",
"Requirement already satisfied, skipping upgrade: pytz in /opt/conda/lib/python3.8/site-packages (from multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (2020.1)\n",
"Requirement already satisfied, skipping upgrade: python-dateutil in /opt/conda/lib/python3.8/site-packages (from multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (2.8.1)\n",
"Requirement already satisfied, skipping upgrade: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil->multiurl>=0.3.2->cads-api-client>=1.4.7->cdsapi>=0.7.2) (1.15.0)\n"
]
}
],
"source": [
"!pip install -U 'cdsapi>=0.7.2'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Update Python Libraries\n",
"\n",
"The current `cdsapi` library requires updating the Python libraries `attrs` and `typing_extensions` in the D4Science JupyterHub environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: attrs in /opt/conda/lib/python3.8/site-packages (24.2.0)\n",
"Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.8/site-packages (4.12.2)\n"
]
}
],
"source": [
"!pip install attrs typing_extensions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.3 Check the Import of Required Libraries\n",
"\n",
"Now, let's import the various libraries needed to execute CDS functions."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# CDS API\n",
"import cdsapi\n",
"\n",
"# Libraries for working with multidimensional arrays\n",
"import numpy as np\n",
"import xarray as xr\n",
"\n",
"# Libraries for plotting and visualising data\n",
"import matplotlib.path as mpath\n",
"import matplotlib.pyplot as plt\n",
"import cartopy.crs as ccrs\n",
"from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER\n",
"import cartopy.feature as cfeature\n",
"\n",
"# Disable warnings for data download via API\n",
"import urllib3 \n",
"urllib3.disable_warnings()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Authenticate\n",
"\n",
"The `cdsapi` client allows you to request data from the Climate Data Store (CDS) programmatically via the CDS API. \n",
"\n",
"We will configure JupyterLab with your personal CDS API key so that you can **instantiate the client without needing to enter your credentials directly in the notebook**.\n",
"\n",
"### 2.1 Obtain Your CDS API Key\n",
"\n",
"First, you'll need your CDS API key. This key includes both your personal User ID and the CDS API key. To obtain it, register or log in to the CDS at [https://cds-beta.climate.copernicus.eu](https://cds-beta.climate.copernicus.eu). Then, visit [https://cds-beta.climate.copernicus.eu/how-to-api](https://cds-beta.climate.copernicus.eu/how-to-api) and copy the string of characters listed after \"key\".\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Configure the JupyterLab Environment across Notebooks\n",
"\n",
"The `auth_cds.py` library, located in the current folder, is a D4Science library specifically designed to enable the use of `cdsapi` within D4Science JupyterLab environments. This library prompts you to enter your credentials and sets them in the JupyterLab environment, so any instance of the `cdsapi` client can authenticate automatically without requiring configurations in the notebook itself.\n",
"\n",
"Additionally, the library offers an option to save your credentials. If you choose to save them, it creates a hidden file in your personal workspace. With this file in place, the authentication process will run automatically and silently on subsequent runs, loading credentials from the saved file instead of prompting for input.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To configure our JupyterLab environment, simply import the function, then, execute it to set up authentication.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"from auth_cds import cds_authenticate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When you run `cds_authenticate()`, the function will handle the authentication process as follows:\n",
"\n",
"1. **If your credentials are already configured** (either stored in environment variables or in a hidden configuration file in your workspace), `cds_authenticate()` will automatically detect and load them. This allows the `cdsapi` client to authenticate silently, without prompting you for input.\n",
"\n",
"2. **If your credentials are not yet configured**, `cds_authenticate()` will prompt you to enter them. You’ll be asked for your CDS API URL and key. After entering these details, the function will:\n",
" - Set the credentials in the JupyterLab environment, so the `cdsapi` client can authenticate in this session.\n",
" - Optionally, it will offer to save the credentials in a hidden configuration file in your workspace, so future sessions can load them automatically without additional input.\n",
"\n",
"This configuration ensures that your credentials are securely handled and easily reusable across sessions.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from file ./.cdsapirc: {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:54:37,276 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:54:37,277 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:54:37,277 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:54:37,278 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:54:37,278 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
},
{
"data": {
"text/plain": [
"<cads_api_client.legacy_api_client.LegacyApiClient at 0x7f6d2936d730>"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cds_authenticate()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Instantiate the CDS API Client \n",
"\n",
"From this point on, we can instantiate a new client simply by running:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:56:53,336 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:56:53,337 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:56:53,337 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:56:53,338 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:56:53,338 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
}
],
"source": [
"client = cdsapi.Client()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is no need to explicitly pass the credentials each time, as they are now configured within the JupyterLab environment itself. This means the credentials are not tied to any specific notebook.\n",
"\n",
"Once these commands have been executed—such as by running this notebook—we can open a new notebook, and without needing to import auth_cds, we can instantiate cdsapi.Client() in the same way"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Executing CDS tutorials\n",
"\n",
"once executed the cds_authenticate(), we can execute any notebook provided as tutotorial in https://ecmwf-projects.github.io/copernicus-training-c3s/intro.html simply \n",
"\n",
"* commenting or removing the `Enter your CDS API key` command\n",
"\n",
"```py\n",
"# URL = 'https://cds-beta.climate.copernicus.eu/api'\n",
"# KEY = \"db1f2085-6b8b-42e6-b832-625dfaf831a4\"\n",
"```\n",
"\n",
"* instanciating the client with \n",
"\n",
"```py\n",
"client = cdsapi.Client()\n",
"```\n",
"\n",
"instead of \n",
"```py\n",
"client = cdsapi.Client(URL, KEY)\n",
"```\n",
"\n",
"you can find all the patched versions of the tutorials in the tutorial folder\n",
"\n",
"just run this notebook before executing them to ensure that the JupyterLAB is configured"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Managing Credentials\n",
"\n",
"The `auth_cds` library provides flexible options for managing your CDS API credentials. Below are the primary commands you can use to handle authentication and configuration. First, import the necessary functions:\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"from auth_cds import cds_authenticate, cds_remove_conf, cds_save_conf, cds_remove_env, cds_show_conf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. **Authenticate and get CDS client**: \n",
" This command authenticates and returns a CDS API client instance. "
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-31 14:58:06,299 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-31 14:58:06,300 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-31 14:58:06,300 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-31 14:58:06,301 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-31 14:58:06,302 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from environment {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
}
],
"source": [
" client = cds_authenticate()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. **Remove configurations from default paths**: \n",
" Removes any saved configuration files. To execute, remove the comment symbol (`#`)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"cds_remove_conf()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. **Save the current or new configuration**: \n",
" Saves the configuration to a default hidden file. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration from environment {'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n",
"Saved Configuration file ./.cdsapirc\n"
]
}
],
"source": [
"cds_save_conf()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. **Remove environment variables for the CDS API**: \n",
" Clears the CDS API credentials from environment variables. To execute, remove the comment symbol (`#`)."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removed environment variable CDSAPI_URL\n",
"Removed environment variable CDSAPI_KEY\n"
]
}
],
"source": [
"cds_remove_env()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" 5.. **Show current configuration from environment and files**: \n",
" Displays the credentials configured in the environment and any found in the saved configuration file."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cds_show_conf()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

168
d4science_auth_cds/AuthCDS.py
View File

@ -1,168 +0,0 @@
import os
import cdsapi
DEFAULT_URL = 'https://cds-beta.climate.copernicus.eu/api'
ENV_CDSAPI_URL = 'CDSAPI_URL'
ENV_CDSAPI_KEY = 'CDSAPI_KEY'
CONFIG_URL = 'url'
CONFIG_KEY = 'key'
class AuthCDS:
def __init__(self, verbose=False):
"""Initialize AuthCDS with optional verbose logging."""
self.verbose = verbose
self.CONFIG_FILE = ".cdsapirc"
self.CURRENT_CDSAPIRC = "./" + self.CONFIG_FILE
# Expands `~` to the home directory path
self.HOME_CDSAPIRC = os.path.expanduser("~/" + self.CONFIG_FILE)
self.DEFAULT_URL = DEFAULT_URL
# self.ENV_CDSAPI_URL = ENV_CDSAPI_URL
# self.ENV_CDSAPI_KEY = ENV_CDSAPI_KEY
# self.CONFIG_URL = CONFIG_URL
# self.CONFIG_KEY = CONFIG_KEY
def log(self, message, force=False):
"""Log messages if verbose mode is enabled."""
if self.verbose or force:
print(message)
def create_config(self, url, key):
"""Create a configuration dictionary."""
return {self.CONFIG_URL: url, self.CONFIG_KEY: key}
def config_from_env(self):
"""Retrieve configuration from environment variables."""
url = os.environ.get(ENV_CDSAPI_URL)
key = os.environ.get(ENV_CDSAPI_KEY)
self.log("ENV - %s %s" % ( url, key))
if url is None and key is None:
self.log("env is not configured")
return None
config = {}
config[CONFIG_URL] = url
config[CONFIG_KEY] = key
self.log(f"Configuration from environment {config}")
return config
def config_to_env(self, config):
"""Set configuration in environment variables."""
os.environ[ENV_CDSAPI_URL] = config[CONFIG_URL]
os.environ[ENV_CDSAPI_KEY] = config[CONFIG_KEY]
self.log(f"Set environment variables {
ENV_CDSAPI_URL}, {ENV_CDSAPI_KEY}")
def config_to_file(self, config, config_path=None):
"""Save configuration to a file."""
if not config_path:
config_path = self.HOME_CDSAPIRC
else:
# Expands `~` to the full home directory path
config_path = os.path.expanduser(config_path)
URL = config[CONFIG_URL]
KEY = config[CONFIG_KEY]
with open(config_path, 'w') as file:
file.write(f"url: {URL}\nkey: {KEY}\n")
self.log(f"Saved Configuration file {config_path}")
def config_to_home(self, config):
"""Save configuration to user home."""
self.config_to_file(config, self.HOME_CDSAPIRC)
def remove_config_from_home(self, config):
"""remove configuration from user home."""
self.remove_conf_credentials(config, from_home=True)
def config_from_file(self, scan_all=False, custom_path=None):
"""Retrieve configuration from a file."""
config = None
check_paths = [self.CURRENT_CDSAPIRC, self.HOME_CDSAPIRC]
if custom_path:
check_paths = [custom_path]
for path in check_paths:
if os.path.exists(path):
found_config = cdsapi.api.read_config(path)
self.log(f"Configuration from file {path}: {found_config}")
config = config or found_config
if not scan_all:
break
return config
def authenticate(self, set_environment=True):
"""Authenticate and set environment variables."""
config = self.config_from_env()
if not config:
config = self.config_from_file()
if not config:
config = self.query_credentials()
if set_environment and config:
self.log("saving config to env")
self.config_to_env(config)
return config
def query_credentials(self):
"""Prompt user for URL and key if no configuration is found."""
url = input(
f"Insert URL (default: {self.DEFAULT_URL}): ") or self.DEFAULT_URL
key = input("Insert your key: ")
config = self.create_config(url, key)
save_option = input("Save config? (y/n): ").lower() == "y"
if save_option:
self.config_to_file(config)
return config
def get_authenticated_client(self):
"""Return an authenticated CDS client."""
config = self.authenticate()
if not config:
self.log("Cannot obtain authentication")
return None
return cdsapi.Client(url=config[CONFIG_URL], key=config[CONFIG_KEY])
def show_credentials(self):
"""Display current configuration from env and files."""
config_env = self.config_from_env()
config_file = self.config_from_file(scan_all=True)
self.log(f"Configuration from environment: {config_env}")
self.log(f"Configuration from file: {config_file}")
return config_env, config_file
def remove_conf_credentials(self, from_home=True, from_current_path=True, custom_path=None):
"""Remove saved configurations from files."""
paths_removed = []
if from_home and os.path.exists(self.HOME_CDSAPIRC):
os.remove(self.HOME_CDSAPIRC)
paths_removed.append(self.HOME_CDSAPIRC)
self.log(f"Removed configuration {self.HOME_CDSAPIRC}")
if from_current_path and os.path.exists(self.CURRENT_CDSAPIRC):
os.remove(self.CURRENT_CDSAPIRC)
paths_removed.append(self.CURRENT_CDSAPIRC)
self.log(f"Removed configuration {self.CURRENT_CDSAPIRC}")
if custom_path and os.path.exists(custom_path):
os.remove(custom_path)
paths_removed.append(custom_path)
self.log(f"Removed configuration {custom_path}")
return paths_removed
def remove_env_credentials(self):
"""Remove authentication credentials from environment variables."""
removed_envs = []
if self.ENV_CDSAPI_URL in os.environ:
del os.environ[self.ENV_CDSAPI_URL]
self.log(f"Removed environment variable {self.ENV_CDSAPI_URL}")
if self.ENV_CDSAPI_KEY in os.environ:
del os.environ[self.ENV_CDSAPI_KEY]
self.log(f"Removed environment variable {self.ENV_CDSAPI_KEY}")
return removed_envs

0
d4science_auth_cds/__init__.py
View File

175
d4science_auth_cds/auth_cds.py
View File

@ -1,175 +0,0 @@
import argparse
from AuthCDS import AuthCDS
# Version of the library
__version__ = "1.0.0"
# Standalone function for authentication from Jupyter
def cds_help():
"""Provide instructions on how to use methods programmatically from Jupyter."""
help_message = '''
CDS Authentication Helper - Jupyter Usage Guide
Version: %s
Available Functions:
---------------------
- cds_authenticate(verbose=True): Perform authentication and return CDS client.
- cds_remove_conf(verbose=True): Remove saved configurations from default paths.
- cds_save_conf(config=None, verbose=True): Save configuration to the default path.
- cds_remove_env(verbose=True): Remove environment-based credentials for CDS.
- cds_show_conf(verbose=True): Display current configuration from environment and files.
Usage Examples:
---------------
1. Authenticate and get CDS client:
client = cds_authenticate()
2. Remove configurations from default paths:
cds_remove_conf()
3. Save the current or new configuration:
cds_save_conf()
4. Remove environment variables for CDS API:
cds_remove_env()
5. Show current configuration from environment and files:
cds_show_conf()
''' % __version__
print(help_message)
def main():
parser = argparse.ArgumentParser(
description="""Authenticate with CDS and configure .cdsapirc
Version: {__version__}""",
add_help=False
)
parser.add_argument(
"-v", "--verbose",
action="store_true",
help="Enable verbose mode for detailed logging"
)
parser.add_argument(
"-a", "--authenticate",
action="store_true",
help="Run authentication and set environment variables"
)
parser.add_argument(
"-s", "--save-config",
metavar="PATH",
type=str,
help="Save configuration to a specific file path (default: ~/.cdsapirc)"
)
parser.add_argument(
"-r", "--remove-config",
action="store_true",
help="Remove saved configurations from default paths"
)
parser.add_argument(
"-q", "--query-credentials",
action="store_true",
help="Prompt user to re-enter credentials and optionally save them"
)
parser.add_argument(
"-p", "--print-config",
action="store_true",
help="Print current configuration from environment and files"
)
parser.add_argument(
"-h", "--help",
action="store_true",
help="Display help information with usage examples"
)
args = parser.parse_args()
if not any(vars(args).values()):
print("CDS Authentication CLI")
print("Version: %s" % __version__)
print("----------------------")
print("Usage:")
print(" python auth_cds.py [OPTIONS]")
print("Options:")
print(" -v, --verbose Enable verbose mode for detailed logging")
print(" -a, --authenticate Run authentication and set environment variables")
print(" -s, --save-config PATH Save configuration to a specific file path")
print(" -r, --remove-config Remove saved configurations from default paths")
print(" -q, --query-credentials Prompt user to re-enter credentials")
print(" -p, --print-config Print current configuration from environment and files")
print(" -h, --help Display this help information with usage examples")
print("Run with -h or --help to see examples and additional information.")
return
if args.help:
cds_help()
return
auth = AuthCDS(verbose=args.verbose)
if args.authenticate:
client = auth.authenticate()
if client:
auth.log("Authenticated successfully")
else:
auth.log("Authentication failed")
if args.save_config:
config = auth.config_from_env() or auth.config_from_file()
if config:
auth.config_to_file(config, config_path=args.save_config)
auth.log(f"Configuration saved to {args.save_config}")
else:
auth.log("No configuration found to save")
if args.remove_config:
removed_paths = auth.remove_conf_credentials()
auth.log(f"Removed configurations from paths: {removed_paths}")
if args.query_credentials:
auth.query_credentials()
auth.log("Credentials queried and optionally saved")
if args.print_config:
env_config, file_config = auth.show_credentials()
auth.log(f"Environment Config: {env_config}", force=True)
auth.log(f"File Config: {file_config}", force=True)
def cds_authenticate(verbose=True):
"""Perform authentication and return the CDS client if successful."""
auth = AuthCDS(verbose=verbose)
conf = auth.authenticate()
return conf
def cds_get_client(verbose=True):
auth = AuthCDS(verbose=verbose)
return auth.get_authenticated_client()
def cds_remove_conf(verbose=True):
auth = AuthCDS(verbose=verbose)
auth.remove_conf_credentials()
def cds_save_conf(config=None, verbose=True):
auth = AuthCDS(verbose=verbose)
if config is None:
config = auth.authenticate()
if config is not None:
auth.config_to_file(config)
def cds_remove_env(verbose=True):
auth = AuthCDS(verbose=verbose)
auth.remove_env_credentials()
def cds_show_conf(verbose=True):
auth = AuthCDS(verbose=verbose)
auth.show_credentials()
if __name__ == "__main__":
main()

1
d4science_auth_cds/result.json
View File

@ -1 +0,0 @@
"~/.cdsapirc"

203
d4science_auth_cds/test_auth_cds.ipynb
View File

@ -1,203 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Enter your CDS API key\n",
"\n",
"We will request data from the Climate Data Store (CDS) programmatically with the help of the CDS API. Let us make use of the option to manually set the CDS API credentials. First, you have to define two variables: `URL` and `KEY` which build together your CDS API key. The string of characters that make up your KEY include your personal User ID and CDS API key. To obtain these, first register or login to the CDS (https://cds-beta.climate.copernicus.eu), then visit https://cds-beta.climate.copernicus.eu/how-to-api and copy the string of characters listed after \"key:\". Replace the `#########` below with this string."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install 'cdsapi>=0.7.2'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U cdsapi\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U attrs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -U typing_extensions"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import auth_cds"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from file ./.cdsapirc\n"
]
},
{
"data": {
"text/plain": [
"{'url': 'https://cds-beta.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"auth_cds.show_credentials()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from environment CDSAPI_URL - CDSAPI_KEY\n",
"{'url': 'https://cds-beta.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n"
]
}
],
"source": [
"print(auth_cds.config_from_env())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(auth_cds.config_from_file())"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"config from environment CDSAPI_URL - CDSAPI_KEY\n",
"set envs CDSAPI_URL , CDSAPI_KEY\n"
]
},
{
"data": {
"text/plain": [
"{'url': 'https://cds-beta.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"auth_cds.authenticate()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-10-30 17:53:32,030 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your \n",
"[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.\n",
"2024-10-30 17:53:32,031 WARNING [2024-09-26T00:00:00] Should you have not yet migrated from the old CDS system to the new CDS, please check our [informative page](https://confluence.ecmwf.int/x/uINmFw) for guidance.\n",
"2024-10-30 17:53:32,031 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.\n",
"2024-10-30 17:53:32,031 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**\n",
"2024-10-30 17:53:32,032 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the \"Show API request code\" tool on the dataset Download Form to check you are using the correct syntax for your API request.\n"
]
}
],
"source": [
"import cdsapi\n",
"client = cdsapi.Client()\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# auth_cds.removeCredentials()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

3
d4science_copernicus_cds/AuthCDS.py
View File

@ -51,8 +51,7 @@ class AuthCDS:
"""Set configuration in environment variables."""
os.environ[ENV_CDSAPI_URL] = config[CONFIG_URL]
os.environ[ENV_CDSAPI_KEY] = config[CONFIG_KEY]
self.log(f"Set environment variables {
ENV_CDSAPI_URL}, {ENV_CDSAPI_KEY}")
self.log(f"Set environment variables {ENV_CDSAPI_URL}, {ENV_CDSAPI_KEY}")
def config_to_file(self, config, config_path=None):
"""Save configuration to a file."""

2439
reanalysis-climatology.ipynb
View File

File diff suppressed because one or more lines are too long