D4Science
/
d4science_copernicus_notebooks
15
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: D4Science:fb573ed673e59cc10599a87029128bec262e8abb
Branches Tags
D4Science:main
D4Science:shared_download
...
compare: D4Science:7cf23cd1cc9c6f714394ce7b644be07d57cc331d
Branches Tags
D4Science:shared_download
D4Science:main

2 Commits
fb573ed673 ... 7cf23cd1cc

Author SHA1 Message Date
Alfredo Oliviero 7cf23cd1cc requirements_tutorials.txt 2024-12-02 14:50:45 +01:00
Alfredo Oliviero c9d037d07e configuration notebook 2024-12-02 14:49:01 +01:00
2 changed files with 221 additions and 71 deletions
Show Stats Expand all files Collapse all files

287
config_auth_cds.ipynb
View File

@ -4,7 +4,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### d4science_copernicus_cds Library - Setup and Example\n", "### d4science_copernicus_cds Library Setup and Example\n",
"\n", "\n",
"This Jupyter notebook will guide you through setting up all dependencies and configuring the environment to use the `d4science_copernicus_cds` library. It also provides a comprehensive example of the library's features and capabilities, helping you to manage Climate Data Store (CDS) API authentication and make programmatic requests from the CDS.\n", "This Jupyter notebook will guide you through setting up all dependencies and configuring the environment to use the `d4science_copernicus_cds` library. It also provides a comprehensive example of the library's features and capabilities, helping you to manage Climate Data Store (CDS) API authentication and make programmatic requests from the CDS.\n",
"\n", "\n",
@ -14,8 +14,8 @@
"\n", "\n",
"To begin, you’ll need your CDS API credentials. Follow these steps to obtain them:\n", "To begin, you’ll need your CDS API credentials. Follow these steps to obtain them:\n",
"\n", "\n",
"1. Register or log in to the CDS at [https://cds.climate.copernicus.eu](https://cds.climate.copernicus.eu).\n", "1. Register or log in to the CDS at [https://cds-beta.climate.copernicus.eu](https://cds-beta.climate.copernicus.eu).\n",
"2. Visit [https://cds.climate.copernicus.eu/how-to-api](https://cds-beta.climate.copernicus.eu/how-to-api) and copy the API key provided.\n", "2. Visit [https://cds-beta.climate.copernicus.eu/how-to-api](https://cds-beta.climate.copernicus.eu/how-to-api) and copy the API key provided.\n",
"\n", "\n",
"The library will prompt you to enter:\n", "The library will prompt you to enter:\n",
"- **URL**: The URL field is prefilled; simply press Enter to accept the default.\n", "- **URL**: The URL field is prefilled; simply press Enter to accept the default.\n",
@ -44,54 +44,57 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 1,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
"source": [
"!pip install \"cdsapi>=0.7.4\"\n",
"!pip install \"attrs>=24.0.0\"\n",
"!pip install \"typing_extensions>=4.0.0\""
]
},
{ {
"cell_type": "code", "name": "stdout",
"execution_count": null, "output_type": "stream",
"metadata": {}, "text": [
"outputs": [], "Collecting cdsapi\n",
"source": [ " Downloading cdsapi-0.7.5-py2.py3-none-any.whl (12 kB)\n",
"!pip install numpy matplotlib cartopy xarray netCDF4 cdsapi" "Collecting datapi (from cdsapi)\n",
" Downloading datapi-0.1.1-py3-none-any.whl (26 kB)\n",
"Requirement already satisfied: requests>=2.5.0 in /opt/conda/lib/python3.10/site-packages (from cdsapi) (2.31.0)\n",
"Requirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from cdsapi) (4.65.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (3.1.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (2.0.2)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (2024.7.4)\n",
"Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from datapi->cdsapi) (23.1.0)\n",
"Collecting multiurl>=0.3.2 (from datapi->cdsapi)\n",
" Using cached multiurl-0.3.3-py3-none-any.whl\n",
"Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from datapi->cdsapi) (4.6.2)\n",
"Requirement already satisfied: pytz in /opt/conda/lib/python3.10/site-packages (from multiurl>=0.3.2->datapi->cdsapi) (2023.3)\n",
"Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.10/site-packages (from multiurl>=0.3.2->datapi->cdsapi) (2.8.2)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil->multiurl>=0.3.2->datapi->cdsapi) (1.16.0)\n",
"Installing collected packages: multiurl, datapi, cdsapi\n",
"Successfully installed cdsapi-0.7.5 datapi-0.1.1 multiurl-0.3.3\n",
"Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (23.1.0)\n",
"Collecting attrs\n",
" Using cached attrs-24.2.0-py3-none-any.whl (63 kB)\n",
"Installing collected packages: attrs\n",
" Attempting uninstall: attrs\n",
" Found existing installation: attrs 23.1.0\n",
" Uninstalling attrs-23.1.0:\n",
" Successfully uninstalled attrs-23.1.0\n",
"Successfully installed attrs-24.2.0\n",
"Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.10/site-packages (4.6.2)\n",
"Collecting typing_extensions\n",
" Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)\n",
"Installing collected packages: typing_extensions\n",
" Attempting uninstall: typing_extensions\n",
" Found existing installation: typing_extensions 4.6.2\n",
" Uninstalling typing_extensions-4.6.2:\n",
" Successfully uninstalled typing_extensions-4.6.2\n",
"Successfully installed typing_extensions-4.12.2\n"
] ]
}, }
{ ],
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [ "source": [
"!pip install \"numpy>=1.16.5,<1.23.0\"\n", "!pip install -U cdsapi\n",
"!pip install pandas xarray \n", "!pip install -U attrs\n",
"!pip install cdsapi\n", "!pip install -U typing_extensions"
"!pip install matplotlib cartopy\n",
"!pip install cfgrib\n",
"!pip install xskillscore"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pip install pandas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install zarr dask fsspec"
] ]
}, },
{ {
@ -107,9 +110,43 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 2,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git\n",
" Cloning https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git to /tmp/pip-req-build-ezp3t9w8\n",
" Running command git clone --filter=blob:none --quiet https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git /tmp/pip-req-build-ezp3t9w8\n",
" warning: filtering not recognized by server, ignoring\n",
" Resolved https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git to commit 22ec9dd7e72830d0057adc7454aa7e864a86a23c\n",
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25hRequirement already satisfied: cdsapi>=0.7.2 in /opt/conda/lib/python3.10/site-packages (from d4science-copernicus-cds==1.0.0) (0.7.5)\n",
"Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from d4science-copernicus-cds==1.0.0) (24.2.0)\n",
"Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.10/site-packages (from d4science-copernicus-cds==1.0.0) (4.12.2)\n",
"Requirement already satisfied: datapi in /opt/conda/lib/python3.10/site-packages (from cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (0.1.1)\n",
"Requirement already satisfied: requests>=2.5.0 in /opt/conda/lib/python3.10/site-packages (from cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (2.31.0)\n",
"Requirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (4.65.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (3.1.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (2.0.2)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (2024.7.4)\n",
"Requirement already satisfied: multiurl>=0.3.2 in /opt/conda/lib/python3.10/site-packages (from datapi->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (0.3.3)\n",
"Requirement already satisfied: pytz in /opt/conda/lib/python3.10/site-packages (from multiurl>=0.3.2->datapi->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (2023.3)\n",
"Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.10/site-packages (from multiurl>=0.3.2->datapi->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (2.8.2)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil->multiurl>=0.3.2->datapi->cdsapi>=0.7.2->d4science-copernicus-cds==1.0.0) (1.16.0)\n",
"Building wheels for collected packages: d4science-copernicus-cds\n",
" Building wheel for d4science-copernicus-cds (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for d4science-copernicus-cds: filename=d4science_copernicus_cds-1.0.0-py3-none-any.whl size=12131 sha256=6ab25037c1292ce4574176dcbf903fc89ef4fefb1f35e2c4b05305a21b04b886\n",
" Stored in directory: /tmp/pip-ephem-wheel-cache-bqdnt94j/wheels/fc/f5/c5/7e21e683ac76148a67a1198259550e1ea8370aa32dad6e4d0e\n",
"Successfully built d4science-copernicus-cds\n",
"Installing collected packages: d4science-copernicus-cds\n",
"Successfully installed d4science-copernicus-cds-1.0.0\n"
]
}
],
"source": [ "source": [
"!pip install -U git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git" "!pip install -U git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git"
] ]
@ -125,7 +162,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 3,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -163,9 +200,21 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 4,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - None None\n",
"env is not configured\n",
"Configuration from file /home/jovyan/.cdsapirc: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"saving config to env\n",
"Set environment variables CDSAPI_URL, CDSAPI_KEY\n"
]
}
],
"source": [ "source": [
"client = cds_authenticate()" "client = cds_authenticate()"
] ]
@ -188,9 +237,34 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 5,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - https://cds.climate.copernicus.eu/api db1f2085-6b8b-42e6-b832-625dfaf831a4\n",
"Configuration from environment {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from file /home/jovyan/.cdsapirc: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from environment: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from file: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n"
]
},
{
"data": {
"text/plain": [
"({'url': 'https://cds.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'},\n",
" {'url': 'https://cds.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'})"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"cds_show_conf()" "cds_show_conf()"
] ]
@ -211,9 +285,18 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 6,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"URL https://cds.climate.copernicus.eu/api\n",
"KEY db1f2085-6b8b-42e6-b832-625dfaf831a4\n"
]
}
],
"source": [ "source": [
"URL, KEY = cds_get_credentials()\n", "URL, KEY = cds_get_credentials()\n",
"print(\"URL\", URL)\n", "print(\"URL\", URL)\n",
@ -237,9 +320,19 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 7,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - https://cds.climate.copernicus.eu/api db1f2085-6b8b-42e6-b832-625dfaf831a4\n",
"Configuration from environment {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Saved Configuration file /home/jovyan/.cdsapirc\n"
]
}
],
"source": [ "source": [
"cds_save_conf()" "cds_save_conf()"
] ]
@ -259,7 +352,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 8,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -281,7 +374,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 9,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -301,9 +394,34 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 10,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ENV - https://cds.climate.copernicus.eu/api db1f2085-6b8b-42e6-b832-625dfaf831a4\n",
"Configuration from environment {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from file /home/jovyan/.cdsapirc: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from environment: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n",
"Configuration from file: {'url': 'https://cds.climate.copernicus.eu/api', 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'}\n"
]
},
{
"data": {
"text/plain": [
"({'url': 'https://cds.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'},\n",
" {'url': 'https://cds.climate.copernicus.eu/api',\n",
" 'key': 'db1f2085-6b8b-42e6-b832-625dfaf831a4'})"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"cds_show_conf()" "cds_show_conf()"
] ]
@ -333,13 +451,38 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 11,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"datadir: %s /home/jovyan/cds_dataDir/out_2024_12_02_13_38_29_example/\n"
]
}
],
"source": [ "source": [
"datadir = cds_datadir(\"example\")" "datadir = cds_datadir(\"example\")"
] ]
}, },
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"datadir: %s ./out/out_2024_12_02_13_38_29_current_example/\n"
]
}
],
"source": [
"datadir_current = cds_datadir(\"current_example\", basepath=\"./out\")"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -367,9 +510,17 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 13,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"datadir: %s ./out/out_2024_12_02_13_38_29_current_example/\n"
]
}
],
"source": [ "source": [
"datadir_current = cds_datadir(\"current_example\", basepath=\"./out\")\n" "datadir_current = cds_datadir(\"current_example\", basepath=\"./out\")\n"
] ]
@ -384,7 +535,7 @@
], ],
"metadata": { "metadata": {
"kernelspec": { "kernelspec": {
"display_name": "Python 3", "display_name": "Python 3 (ipykernel)",
"language": "python", "language": "python",
"name": "python3" "name": "python3"
}, },
@ -398,7 +549,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.8.5" "version": "3.10.11"
} }
}, },
"nbformat": 4, "nbformat": 4,

1
requirements_tutorials.txt
View File

@ -13,4 +13,3 @@ cfgrib
zarr zarr
dask dask
fsspec fsspec
git+https://code-repo.d4science.org/D4Science/d4science_copernicus_cds.git