CatalogueRevamping
/
docker-ckan
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #48 from mjanez/ckan-2.9.8 

Improve documentation
This commit is contained in:
mjanez 2023-05-09 16:36:33 +02:00 committed by GitHub
parent 77a67768b2 b362ad8e2f
commit 3f96a0bca5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 211 additions and 129 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

340
README.md
View File

@ -11,7 +11,8 @@
<a href="#ckan-images">CKAN images</a> • <a href="#ckan-images">CKAN images</a> •
<a href="#extending-the-base-images">Extending guide</a> • <a href="#extending-the-base-images">Extending guide</a> •
<a href="#applying-patches">Applying patches</a> • <a href="#applying-patches">Applying patches</a> •
<a href="#ckan-docker-addons">Addons</a> <a href="#ckan-docker-addons">Addons</a> •
<a href="#ckan-docker-tips">Info & Backups</a> •
</p> </p>
**Requirements**: **Requirements**:
@ -32,7 +33,7 @@ Available components:
| CKAN Version | Type | Docker tag | Notes | | CKAN Version | Type | Docker tag | Notes |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| 2.9.8 | custom image | `ghcr.io/mjanez/ckan-spatial:ckan-2.9.8` | Stable version with CKAN 2.9.8 | | 2.9.8 | custom image | `ghcr.io/mjanez/ckan-spatial:ckan-2.9.8` | Stable version with CKAN 2.9.8 |
| master | custom image | `ghcr.io/mjanez/ckan-spatial:master` | Latest version. | | 2.9.8 | custom image | `ghcr.io/mjanez/ckan-spatial:master` | Latest version. Master branch. |
The non-CKAN images are as follows: The non-CKAN images are as follows:
* PostgreSQL: [Custom image](/postgresql/Dockerfile) based on official PostgreSQL image. Database files are stored in a named volume. * PostgreSQL: [Custom image](/postgresql/Dockerfile) based on official PostgreSQL image. Database files are stored in a named volume.
@ -96,76 +97,9 @@ To upgrade Docker Engine, first run sudo `apt-get update`, then follow the [inst
To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output
versions for client and server. versions for client and server.
### Docker. Basic commands >**Note**<br>
#### Linux post-install steps > Learn more about [Docker](#docker-basic-commands)/[Docker Compose](#docker-compose-basic-commands) basic commands.
[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user). >
#### Configure Docker to start on boot
```bash
sudo systemctl enable docker
# To disable this behavior, use disable instead.
sudo systemctl disable docker
```
#### Clear all Docker unused objects (images, containers, networks, local volumes)
```bash
docker system prune # Clear all
docker image prune # Clear unused images
docker container prune # Clear unused containers
docker volume prune # Clear unused volumes
docker network prune # Clear unused networks
```
### Docker Compose. Basic commands
More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/).
```bash
# Basic. All containers or specific container: <container>
## Starts existing containers for a service.
docker compose start <container>
## Restarts existing containers/container for a service.
docker compose restart <container>
## Stops running containers without removing them.
docker compose stop <container>
## Pauses running containers of a service.
docker compose pause <container>
## Unpauses paused containers of a service.
docker compose unpause <container>
# Display the logs of a container. Is it possible to retrieve only the last n seconds or other
docker logs [--since 60s] <container> -f
## Lists containers.
docker compose ps
## Remove all docker compose project
docker compose rm <container>
# Build.
## Builds, (re)creates, starts, and attaches to containers for a service.
docker compose [-f <docker compose-file>] up
## Build & up all the containers.
docker compose [-f <docker compose-file>] up -d --build
## To avoid using a cache of the previous build while creating a new image.
docker compose [-f <docker compose-file>] build --no-cache
## Build a project with a specific Docker Compose prefix.
docker compose [-f <docker compose-file>] -p <my_project> up -d --build
# Down
# Stops containers and removes containers, networks, volumes, and images created by up.
docker compose [-p <my_project>] down
```
## Install (build and run) CKAN plus dependencies ## Install (build and run) CKAN plus dependencies
### Base mode ### Base mode
@ -216,6 +150,11 @@ window for something else.
> * Or `docker compose up --build` to build & up the containers. > * Or `docker compose up --build` to build & up the containers.
> * Or `docker compose -f docker-compose.nginx.yml up -d --build` to use the NGINX version. > * Or `docker compose -f docker-compose.nginx.yml up -d --build` to use the NGINX version.
>**Note**<br>
> Learn more about configuring this ckan docker:
> - [Backup the CKAN Database](#ckan-backups)
> - [Configuring a docker compose service to start on boot](#docker-compose-configure-a-docker-compose-service-to-start-on-boot)
At the end of the container start sequence there should be 6 containers running (or 5 if use NGINX Docker Compose file) At the end of the container start sequence there should be 6 containers running (or 5 if use NGINX Docker Compose file)
After this step, CKAN should be running at {`APACHE_SERVER_NAME`}{`APACHE_CKAN_LOCATION`} and ckan-pycsw at {`APACHE_SERVER_NAME`}{`APACHE_PYCSW_LOCATION`}, i.e: http://localhost/catalog or http://localhost/csw After this step, CKAN should be running at {`APACHE_SERVER_NAME`}{`APACHE_CKAN_LOCATION`} and ckan-pycsw at {`APACHE_SERVER_NAME`}{`APACHE_PYCSW_LOCATION`}, i.e: http://localhost/catalog or http://localhost/csw
@ -229,63 +168,6 @@ After this step, CKAN should be running at {`APACHE_SERVER_NAME`}{`APACHE_CKAN_L
|2cdd25cea0de|ckan-docker-db |docker-entrypoint.s…|6 minutes ago |Up 4 minutes (healthy)|5432/tcp |db | | |2cdd25cea0de|ckan-docker-db |docker-entrypoint.s…|6 minutes ago |Up 4 minutes (healthy)|5432/tcp |db | |
|9cdj25dae6gr|ckan-docker-pycsw |docker-entrypoint.s…|6 minutes ago |Up 4 minutes (healthy)|8000/tcp |pycsw | | |9cdj25dae6gr|ckan-docker-pycsw |docker-entrypoint.s…|6 minutes ago |Up 4 minutes (healthy)|8000/tcp |pycsw | |
#### Configure a docker compose service to start on boot
>**Note**<br>
> Test on Debian.
To have Docker Compose run automatically when you reboot a machine, you can follow the steps below:
1. Create a systemd service file for Docker Compose. You can create a file named `ckan-docker-compose.service` in the `/etc/systemd/system/` folder with the following content:
```bash
[Unit]
Description=CKAN Docker Compose Application Service
Requires=docker.service
After=docker.service
[Service]
User=docker
Group=docker
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/path/to/project/ckan-docker/
ExecStart=/bin/docker compose up -d
ExecStop=/bin/docker compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
```
2. Replace `/path/to/project/ckan-docker/` with the path where your project's `docker-compose.yml` file is located and and check the path to the docker compose binary on execution and stop: `/bin/docker`. Also change the `User` / `Group` to execute the service.
3. Load the systemd service file with the following command:
```bash
sudo systemctl daemon-reload
```
4. Enables the service to start automatically when the machine boots up:
```bash
sudo systemctl enable ckan-docker-compose
```
5. You can now start the service with the following command:
```bash
sudo systemctl start ckan-docker-compose
```
6. If you want to stop or check the status of the service, use the following commands:
```bash
# Stop the service
sudo systemctl stop ckan-docker-compose
# Check the status
sudo systemctl status ckan-docker-compose
```
### Quick mode ### Quick mode
If you just want to test the package and see the general functionality of the platform, you can use the `ckan-spatial` image from the [Github container registry](https://github.com/mjanez/ckan-docker/pkgs/container/ckan-spatial): If you just want to test the package and see the general functionality of the platform, you can use the `ckan-spatial` image from the [Github container registry](https://github.com/mjanez/ckan-docker/pkgs/container/ckan-spatial):
@ -519,6 +401,206 @@ Available components:
* **pycsw**: The pycsw app. An [OARec](https://ogcapi.ogc.org/records) and [OGC CSW](https://opengeospatial.org/standards/cat) server implementation written in Python. * **pycsw**: The pycsw app. An [OARec](https://ogcapi.ogc.org/records) and [OGC CSW](https://opengeospatial.org/standards/cat) server implementation written in Python.
* **ckan2pycsw**: Software to achieve interoperability with the open data portals based on CKAN. To do this, ckan2pycsw reads data from an instance using the CKAN API, generates ISO-19115/ISO-19139 metadata using [pygeometa](https://geopython.github.io/pygeometa/), or a custom schema that is based on a customized CKAN schema, and populates a [pycsw](https://pycsw.org/) instance that exposes the metadata using CSW and OAI-PMH. * **ckan2pycsw**: Software to achieve interoperability with the open data portals based on CKAN. To do this, ckan2pycsw reads data from an instance using the CKAN API, generates ISO-19115/ISO-19139 metadata using [pygeometa](https://geopython.github.io/pygeometa/), or a custom schema that is based on a customized CKAN schema, and populates a [pycsw](https://pycsw.org/) instance that exposes the metadata using CSW and OAI-PMH.
## ckan-docker tips
### CKAN. Backups
PostgreSQL offers the command line tools [`pg_dump`](https://www.postgresql.org/docs/current/static/app-pgdump.html) and [`pg_restore`](https://www.postgresql.org/docs/current/static/app-pgrestore.html) for dumping and restoring a database and its content to/from a file.
### Backup service for db container
1. Create a new file called `ckan_backup_custom.sh` and open it in your preferred text editor.
2. Add the following code to the script, replacing the placeholders with your actual values:
```sh
#!/bin/bash
# Set the necessary variables
POSTGRESQL_CONTAINER_NAME="db"
DATABASE_NAME="ckan"
POSTGRES_USER="ckan"
POSTGRES_PASSWORD="your_postgres_password"
BACKUP_DIRECTORY="/path/to/your/backup/directory"
DATE=`date +%Y%m%d%H%M%S`
# Run the backup command
docker exec -e PGPASSWORD=$POSTGRES_PASSWORD $POSTGRESQL_CONTAINER_NAME pg_dump -U $POSTGRES_USER -Fc $DATABASE_NAME > $BACKUP_DIRECTORY/ckan_backup_$DATE.dump
```
3. Replace the following placeholders with your actual values:
- `your_postgres_password`: The password for the PostgreSQL user.
- `/path/to/your/backup/directory`: The path to the directory where you want to store the backup files.
>**Warning**<br>
> If you have changed the values of the PostgreSQL container, database or user, change them too.
4. Save and close the file.
5. Make the script executable:
```bash
chmod +x ckan_backup_custom.sh
```
6. Open the crontab for the current user:
```bash
crontab -e
```
7. Add the following line to schedule the backup to run daily at midnight (adjust the schedule as needed):
```sh
0 0 * * * /path/to/your/script/ckan_backup_custom.sh
```
>**Info**<br>
> Replace `/path/to/your/script` with the actual path to the `ckan_backup_custom.sh` script.
8. Save and close the file.
The cronjob is now set up and will backup your CKAN PostgreSQL database daily at midnight using the custom format. The backups will be stored in the specified directory with the timestamp in the filename.
### Restore a backup
If need to use a backup, restore it:
1. First clean the database. **Caution, this will delete all data from your CKAN database!**
```bash
docker exec -it ckan /bin/bash -c "export TERM=xterm; exec bash"
# Delete everything in the CKAN database, including the tables, to start from scratch
ckan -c $CKAN_INI db clean
```
2. After cleaning the database you must do either [initialize it](https://docs.ckan.org/en/2.9/maintaining/database-management.html#initialization) or import a previously created dump.
```bash
docker exec -e PGPASSWORD=$POSTGRES_PASSWORD $POSTGRESQL_CONTAINER_NAME pg_restore -U $POSTGRES_USER --clean --if-exists -d $DATABASE_NAME < /path/to/your/backup/directory/ckan.dump
```
### Docker. Basic commands
#### Linux post-install steps
[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).
#### Configure Docker to start on boot
```bash
sudo systemctl enable docker
# To disable this behavior, use disable instead.
sudo systemctl disable docker
```
#### Clear all Docker unused objects (images, containers, networks, local volumes)
```bash
docker system prune # Clear all
docker image prune # Clear unused images
docker container prune # Clear unused containers
docker volume prune # Clear unused volumes
docker network prune # Clear unused networks
```
### Docker Compose. Basic commands
More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/).
```bash
# Basic. All containers or specific container: <container>
## Starts existing containers for a service.
docker compose start <container>
## Restarts existing containers/container for a service.
docker compose restart <container>
## Stops running containers without removing them.
docker compose stop <container>
## Pauses running containers of a service.
docker compose pause <container>
## Unpauses paused containers of a service.
docker compose unpause <container>
# Display the logs of a container. Is it possible to retrieve only the last n seconds or other
docker logs [--since 60s] <container> -f
## Lists containers.
docker compose ps
## Remove all docker compose project
docker compose rm <container>
# Build.
## Builds, (re)creates, starts, and attaches to containers for a service.
docker compose [-f <docker compose-file>] up
## Build & up all the containers.
docker compose [-f <docker compose-file>] up -d --build
## To avoid using a cache of the previous build while creating a new image.
docker compose [-f <docker compose-file>] build --no-cache
## Build a project with a specific Docker Compose prefix.
docker compose [-f <docker compose-file>] -p <my_project> up -d --build
# Down
# Stops containers and removes containers, networks, volumes, and images created by up.
docker compose [-p <my_project>] down
```
### Docker Compose. Configure a docker compose service to start on boot
To have Docker Compose run automatically when you reboot a machine, you can follow the steps below:
1. Create a systemd service file for Docker Compose. You can create a file named `ckan-docker-compose.service` in the `/etc/systemd/system/` folder with the following content:
```bash
[Unit]
Description=CKAN Docker Compose Application Service
Requires=docker.service
After=docker.service
[Service]
User=docker
Group=docker
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/path/to/project/ckan-docker/
ExecStart=/bin/docker compose up -d
ExecStop=/bin/docker compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
```
2. Replace `/path/to/project/ckan-docker/` with the path where your project's `docker-compose.yml` file is located and and check the path to the docker compose binary on execution and stop: `/bin/docker`. Also change the `User` / `Group` to execute the service.
3. Load the systemd service file with the following command:
```bash
sudo systemctl daemon-reload
```
4. Enables the service to start automatically when the machine boots up:
```bash
sudo systemctl enable ckan-docker-compose
```
5. You can now start the service with the following command:
```bash
sudo systemctl start ckan-docker-compose
```
6. If you want to stop or check the status of the service, use the following commands:
```bash
# Stop the service
sudo systemctl stop ckan-docker-compose
# Check the status
sudo systemctl status ckan-docker-compose
```
[^1]: Official CKAN repo: https://github.com/ckan/ckan-docker-base [^1]: Official CKAN repo: https://github.com/ckan/ckan-docker-base
[^2]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html) [^2]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)