From 9af99bfba80c6abc3932fe523a6d6834c195a5a6 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Fri, 17 Mar 2023 12:17:23 +0000
Subject: [PATCH 05/34] Improve docker compose reference
---
README.md | 56 ++++++++++++++++++++++++++++++++++---------------------
1 file changed, 35 insertions(+), 21 deletions(-)
diff --git a/README.md b/README.md
index cd27b7d..a0ea333 100644
--- a/README.md
+++ b/README.md
@@ -96,26 +96,49 @@ docker network prune # Clear unused networks
More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/).
```bash
-# Starts existing containers for a service.
-docker compose start
+# Basic. All containers or specific container:
+## Starts existing containers for a service.
+docker compose start
-# Stops running containers without removing them.
-docker compose stop
+## Restarts existing containers/container for a service.
+docker compose restart
-# Pauses running containers of a service.
-docker compose pause
+## Stops running containers without removing them.
+docker compose stop
-# Unpauses paused containers of a service.
-docker compose unpause
+## Pauses running containers of a service.
+docker compose pause
-# Lists containers.
+## Unpauses paused containers of a service.
+docker compose unpause
+
+# Display the logs of a container. Is it possible to retrieve only the last n seconds or other
+docker logs [--since 60s] -f
+
+## Lists containers.
docker compose ps
-# Builds, (re)creates, starts, and attaches to containers for a service.
-docker compose up
+## Remove all docker compose project
+docker compose rm
+
+# Build.
+## Builds, (re)creates, starts, and attaches to containers for a service.
+docker compose [-f ] up
+
+## Build & up all the containers.
+docker compose [-f ] up -d --build
+
+## To avoid using a cache of the previous build while creating a new image.
+docker compose [-f ] build --no-cache
+
+## Build a project with a specific Docker Compose prefix.
+docker compose [-f ] -p up -d --build
+
+
+# Down
# Stops containers and removes containers, networks, volumes, and images created by up.
-docker compose down
+docker compose [-p ] down
```
## Install (build and run) CKAN plus dependencies
@@ -166,15 +189,6 @@ To start the containers:
See [CKAN Images](#ckan-images) for more details of what happens when using development mode.
-**docker/docker compose commands**
- * `docker compose [-f ] up -d --build`: to build & up all the containers.
- * `docker compose [-f ] build --no-cache`: to avoid using a cache of the previous build while creating a new image.
- * `docker compose -p up -d --build`: to build a project with a specific Docker Compose prefix.
- * `docker compose restart `: to restart only a specific container.
- * `docker compose rm`: to remove all docker compose project.
- * `docker compose [-p ] down`: to down all containers
- * `docker logs --since 60s -f `: to display the logs of a container for the last n seconds.
-
#### Create an extension
You can use the ckan [extension](https://docs.ckan.org/en/latest/extensions/tutorial.html#creating-a-new-extension) instructions to create a CKAN extension, only executing the command inside the CKAN container and setting the mounted `src/` folder as output:
From 64195b52e8d7e6c9056900b874bc1fd526d78b3a Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Fri, 17 Mar 2023 12:30:01 +0000
Subject: [PATCH 06/34] Add warning info about this repo
---
README.md | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/README.md b/README.md
index a0ea333..56f3047 100644
--- a/README.md
+++ b/README.md
@@ -33,8 +33,7 @@ The non-CKAN images are as follows:
The site is configured using environment variables that you can set in the `.env` file.
>**Warning**:
-> This is the **install from Docker Compose**. To see the install from source, check it out: [`ckan/doc/repository/installations/ckan_source.md`](https://github.com/OpenDataGIS/ckan/blob/16dbe5da5ca0fd28d595ba2049e4d47e52c40c0f/doc/repository/installations/ckan_source.md)
-
+>This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: CKAN documentation: Installation
### ckan-docker roadmap
Information about extensions installed in the `main` image. More info described in the [Extending the base images](#extending-the-base-images)
From 309078d3e546002de66bce3474063c64bd1af3b5 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Fri, 17 Mar 2023 12:31:35 +0000
Subject: [PATCH 07/34] Fix link to CKAN Docs
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 56f3047..51e1b34 100644
--- a/README.md
+++ b/README.md
@@ -33,7 +33,7 @@ The non-CKAN images are as follows:
The site is configured using environment variables that you can set in the `.env` file.
>**Warning**:
->This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: CKAN documentation: Installation
+>This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: [CKAN documentation: Installation](https://docs.ckan.org/en/latest/maintaining/installing/index.html).
### ckan-docker roadmap
Information about extensions installed in the `main` image. More info described in the [Extending the base images](#extending-the-base-images)
From de7d05e8de30ece81705c6bfcb254f8df1d70b35 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Fri, 17 Mar 2023 12:32:46 +0000
Subject: [PATCH 08/34] Update message ckan-pycsw
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 51e1b34..4466221 100644
--- a/README.md
+++ b/README.md
@@ -55,7 +55,7 @@ Information about extensions installed in the `main` image. More info described
| Extension | [ckanext-resourcedictionary](https://github.com/OpenDataGIS/ckanext-resourcedictionary) | main | Completed | ✔️ | ✔️ | Stable installation. This extension extends the default CKAN Data Dictionary functionality by adding possibility to create data dictionary before actual data is uploaded to datastore. |
| Extension | [ckanext-pages](https://github.com/ckan/ckanext-pages) | 0.5.1 | Completed | ✔️ | ✔️ | Stable installation. This extension gives you an easy way to add simple pages to CKAN. |
| Extension | [ckanext-pdfview](https://github.com/ckan/ckanext-pdfview) | 0.0.8 | Completed | ✔️ | ✔️ | Stable installation. This extension provides a view plugin for PDF files using an html object tag. |
-| Software | [docker-pycsw](https://github.com/mjanez/ckan-pycsw) | main | Completed standalone version | ✔️ | ❌ | Stable installation. PyCSW Endpoint of Open Data Portal with docker compose config. Harvest the CKAN catalogue in a CSW endpoint based on existing spatial datasets in the open data portal. |
+| Software | [docker-pycsw](https://github.com/mjanez/ckan-pycsw) | main | Ready to use standalone version | ✔️ | ❌ | Stable installation. PyCSW Endpoint of Open Data Portal with docker compose config. Harvest the CKAN catalogue in a CSW endpoint based on existing spatial datasets in the open data portal. |
## Environment: docker
From c61176e20cc612a6ba00b6e998e8f371e17d9c65 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 09:51:35 +0000
Subject: [PATCH 09/34] Remove datapusher
---
docker-compose.dev.yml | 7 -------
docker-compose.yml | 7 -------
samples/ckan.ini.example | 2 +-
3 files changed, 1 insertion(+), 15 deletions(-)
diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml
index 2b6a0f1..5fc9ea4 100755
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -32,13 +32,6 @@ services:
healthcheck:
test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:5000"]
- datapusher:
- container_name: ${DATAPUSHER_CONTAINER_NAME}
- image: ckan/ckan-base-datapusher:${DATAPUSHER_VERSION}
- restart: unless-stopped
- healthcheck:
- test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:8800"]
-
db:
container_name: ${POSTGRESQL_CONTAINER_NAME}
build:
diff --git a/docker-compose.yml b/docker-compose.yml
index 8548f0e..75b2ede 100755
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -43,13 +43,6 @@ services:
healthcheck:
test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:5000"]
- datapusher:
- container_name: ${DATAPUSHER_CONTAINER_NAME}
- image: ckan/ckan-base-datapusher:${DATAPUSHER_VERSION}
- restart: unless-stopped
- healthcheck:
- test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:8800"]
-
db:
container_name: ${POSTGRESQL_CONTAINER_NAME}
build:
diff --git a/samples/ckan.ini.example b/samples/ckan.ini.example
index 618582b..7ec88b5 100644
--- a/samples/ckan.ini.example
+++ b/samples/ckan.ini.example
@@ -149,10 +149,10 @@ ckan.harvest.mq.hostname = redis
# Define which views should be created by default
# (plugins must be loaded in ckan.plugins)
ckan.views.default_views = image_view text_view recline_view harvest ckan_harvester wmts_view geojson_view geo_view shp_view pdf_view
-ckanext.geoview.geojson.max_file_size = 100000000
ckanext.geoview.ol_viewer.formats = wms wfs geojson gml kml
ckanext.geoview.shp_viewer.srid = 3857
ckanext.geoview.shp_viewer.encoding = UTF-8
+ckanext.geoview.geojson.max_file_size = 100000000
# Customize which text formats the text_view plugin will show
ckan.preview.json_formats = json jsonld
From 4985c8befc9f1c33bb04b8a6f8dd80ea4555aed8 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 09:52:41 +0000
Subject: [PATCH 10/34] Add entrypoints - ckanext-dcat - previews -
ckanext-xloader
---
ckan/docker-entrypoint.d/setup_dcat.sh | 7 +++++++
ckan/docker-entrypoint.d/setup_preview.sh | 20 ++++++++++++++++++++
ckan/docker-entrypoint.d/setup_xloader.sh | 11 +++++++++++
3 files changed, 38 insertions(+)
create mode 100644 ckan/docker-entrypoint.d/setup_dcat.sh
create mode 100644 ckan/docker-entrypoint.d/setup_preview.sh
create mode 100644 ckan/docker-entrypoint.d/setup_xloader.sh
diff --git a/ckan/docker-entrypoint.d/setup_dcat.sh b/ckan/docker-entrypoint.d/setup_dcat.sh
new file mode 100644
index 0000000..9c8f5ae
--- /dev/null
+++ b/ckan/docker-entrypoint.d/setup_dcat.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Add ckanext-dcat settings to the CKAN config file
+echo "Loading ckanext-dcat settings in the CKAN config file"
+ckan config-tool $CKAN_INI \
+ "ckanext.dcat.base_uri = $CKANEXT__DCAT__BASE_URI" \
+ "ckanext.dcat.rdf.profiles = $CKANEXT__DCAT__RDF_PROFILES"
\ No newline at end of file
diff --git a/ckan/docker-entrypoint.d/setup_preview.sh b/ckan/docker-entrypoint.d/setup_preview.sh
new file mode 100644
index 0000000..246a1ed
--- /dev/null
+++ b/ckan/docker-entrypoint.d/setup_preview.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+#TODO: Correct views.
+
+# Add CKAN Resource views to the CKAN config file
+echo "Loading resource views in the CKAN config file"
+ckan config-tool $CKAN_INI \
+ "ckan.views.default_views = $CKAN__VIEWS__DEFAULT_VIEWS" \
+ "ckan.preview.json_formats = $CKAN__PREVIEW__JSON_FORMATS" \
+ "ckan.preview.xml_formats = $CKAN__PREVIEW__XML_FORMATS" \
+ "ckan.preview.text_formats = $CKAN__PREVIEW__TEXT_FORMATS" \
+ "ckan.preview.loadable = $CKAN__PREVIEW__LOADABLE"
+
+# Add CKAN Resource geoviews to the CKAN config file
+echo "Loading geoviews in the CKAN config file"
+ckan config-tool $CKAN_INI \
+ "ckanext.geoview.ol_viewer.formats = $CKANEXT__GEOVIEW__OL_VIEWER__FORMATS" \
+ "ckanext.geoview.shp_viewer.srid = $CKANEXT__GEOVIEW__SHP_VIEWER__SRID" \
+ "ckanext.geoview.shp_viewer.encoding = $CKANEXT__GEOVIEW__SHP_VIEWER__ENCODING" \
+ "ckanext.geoview.geojson.max_file_size = $CKANEXT__GEOVIEW__GEOJSON__MAX_FILE_SIZE"
\ No newline at end of file
diff --git a/ckan/docker-entrypoint.d/setup_xloader.sh b/ckan/docker-entrypoint.d/setup_xloader.sh
new file mode 100644
index 0000000..d34e245
--- /dev/null
+++ b/ckan/docker-entrypoint.d/setup_xloader.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# Add ckanext.xloader.api_token to the CKAN config file
+echo "Loading ckanext-xloader settings in the CKAN config file"
+ckan config-tool $CKAN_INI \
+ "ckanext.xloader.api_token = xxx" \
+ "ckanext.xloader.jobs_db.uri = $CKANEXT__XLOADER__JOBS__DB_URI"
+
+# Create ckanext-xloader API_TOKEN
+echo "Set up ckanext.xloader.api_token in the CKAN config file"
+ckan config-tool $CKAN_INI "ckanext.xloader.api_token = $(ckan -c $CKAN_INI user token add ckan_admin xloader | tail -n 1 | tr -d '\t')"
From bb48c03341de6113634ab6d91d38784c5faddef3 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 09:54:02 +0000
Subject: [PATCH 11/34] Update Dockerfile.dev - Use stable 2.9.7 - Add plugins
---
ckan/Dockerfile.dev | 37 +++++++++++++++++++++----------------
1 file changed, 21 insertions(+), 16 deletions(-)
diff --git a/ckan/Dockerfile.dev b/ckan/Dockerfile.dev
index 5098db1..a4f79c6 100644
--- a/ckan/Dockerfile.dev
+++ b/ckan/Dockerfile.dev
@@ -1,4 +1,4 @@
-FROM ckan/ckan-base:ckan-2.10.0-dev
+FROM ckan/ckan-base:2.9.7-dev
# Set up environment variables
ENV APP_DIR=/srv/app
@@ -11,7 +11,7 @@ RUN apk add geos-dev proj proj-util proj-dev
# Make sure both files are not exactly the same
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
- cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
+ cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
fi ;
# Install any extensions needed by your CKAN instance
@@ -31,9 +31,9 @@ RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
###TODO: XLoader ###
-#RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@master#egg=ckanext-xloader' && \
-# pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
-# pip3 install -U requests[security]
+RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@master#egg=ckanext-xloader' && \
+ pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
+ pip3 install -U requests[security]
### Harvester ###
RUN echo "ckan/ckanext-harvester" && \
@@ -58,12 +58,7 @@ RUN echo "ckan/ckanext-spatial" && \
### Scheming-GeoDCAT-AP extended version ###
RUN echo "mjanez/ckanext-scheming (GeoDCAT-AP extended version)" && \
- pip3 install -e git+https://github.com/mjanez/ckanext-scheming.git#egg=ckanext-scheming && \
- # update CKAN config with ckanext-scheming parameters
- ckan config-tool ${CKAN_INI} "scheming.dataset_schemas=''" && \
- ckan config-tool ${CKAN_INI} "scheming.group_schemas=''" && \
- ckan config-tool ${CKAN_INI} "scheming.organization_schemas=''" && \
- ckan config-tool ${CKAN_INI} "scheming.presets=''"
+ pip3 install -e git+https://github.com/mjanez/ckanext-scheming.git#egg=ckanext-scheming
### Pages ###
RUN echo "ckan/ckanext-pages" && \
@@ -78,6 +73,16 @@ RUN echo "mjanez/ckanext-dcat (GeoDCAT-AP extended version)" && \
pip3 install -e git+https://github.com/mjanez/ckanext-dcat.git#egg=ckanext-dcat && \
pip3 install -r https://raw.githubusercontent.com/mjanez/ckanext-dcat/master/requirements.txt
+###TODO: ckanext-visualize
+#https://github.com/keitaroinc/ckanext-visualize
+
+###TODO datapreview error
+# 2023-03-19 12:54:42,311 WARNI [ckan.lib.datapreview] Plugin for view t could not be found
+# 2023-03-19 12:54:42,313 WARNI [ckan.lib.datapreview] Plugin for view e could not be found
+# 2023-03-19 12:54:42,314 WARNI [ckan.lib.datapreview] Plugin for view x could not be found
+# 2023-03-19 12:54:42,315 WARNI [ckan.lib.datapreview] Plugin for view t could not be found
+# https://github.com/ckan/ckan/search?q=datapreview&type=issues
+
# ###TODO: SPARQL Interface - Adapt to Py3###
# RUN echo "mjanez/sparql (Fixed version)" && \
# pip3 install -e git+https://github.com/mjanez/ckanext-sparql.git#egg=ckanext-sparql && \
@@ -97,9 +102,9 @@ COPY docker-entrypoint.d/* /docker-entrypoint.d/
COPY patches ${APP_DIR}/patches
RUN for d in $APP_DIR/patches/*; do \
- if [ -d $d ]; then \
- for f in `ls $d/*.patch | sort -g`; do \
- cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
- done ; \
- fi ; \
+ if [ -d $d ]; then \
+ for f in `ls $d/*.patch | sort -g`; do \
+ cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
+ done ; \
+ fi ; \
done
\ No newline at end of file
From 9d99392ea639f46224b8e66d89d7994aff21c811 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 09:54:57 +0000
Subject: [PATCH 12/34] Update Dockerfile - Use stable 2.9.7 - Add stable
plugins
---
ckan/Dockerfile | 65 ++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 62 insertions(+), 3 deletions(-)
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 89b67a8..2e27be0 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -1,12 +1,71 @@
-FROM ckan/ckan-base:ckan-2.10.0
-#FROM ckan/ckan-base:dev-v2.10
+FROM ckan/ckan-base:2.9.7
# Set up environment variables
ENV APP_DIR=/srv/app
ENV TZ=UTC
RUN echo ${TZ} > /etc/timezone
+# Set up PROJ
+RUN apk add geos-dev proj proj-util proj-dev
+
# Make sure both files are not exactly the same
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
- cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
+ cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
fi ;
+
+# Extensions
+### XLoader - v0.12.2 ###
+RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@0.12.2#egg=ckanext-xloader' && \
+ pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
+ pip3 install -U requests[security]
+
+### Harvester - v1.5.0 ###
+RUN echo "ckan/ckanext-harvest" && \
+ pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@v1.5.0#egg=ckanext-harvest' && \
+ pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt
+
+### Geoview - v0.0.20 ###
+RUN echo "ckan/ckanext-geoview" && \
+ pip3 install -e 'git+https://github.com/ckan/ckanext-geoview.git@v0.0.20#egg=ckanext-geoview'
+
+### Spatial - v2.0.0 ###
+RUN echo "ckan/ckanext-spatial" && \
+ pip3 install -e 'git+https://github.com/ckan/ckanext-spatial.git@v2.0.0#egg=ckanext-spatial' && \
+ pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-spatial/bae9290395b252ee8e40056256fa694569d1d78b/requirements.txt
+
+### DCAT - master (GeoDCAT-AP extended version) ###
+RUN echo "mjanez/ckanext-dcat (GeoDCAT-AP extended version)" && \
+ pip3 install -e git+https://github.com/mjanez/ckanext-dcat.git#egg=ckanext-dcat && \
+ pip3 install -r https://raw.githubusercontent.com/mjanez/ckanext-dcat/master/requirements.txt
+
+### Scheming - master (GeoDCAT-AP extended version) ###
+RUN echo "mjanez/ckanext-scheming (GeoDCAT-AP extended version)" && \
+ pip3 install -e git+https://github.com/mjanez/ckanext-scheming.git#egg=ckanext-scheming
+
+### Resource dictionary (Fix version) - main ###
+RUN echo "mjanez/ckanext-resourcedictionary" && \
+ pip3 install -e 'git+https://github.com/mjanez/ckanext-resourcedictionary.git#egg=ckanext-resourcedictionary'
+
+### Pages - v0.5.1 ###
+RUN echo "ckan/ckanext-pages" && \
+ pip3 install -e git+https://github.com/ckan/ckanext-pages.git@v0.5.1#egg=ckanext-pages
+
+### PDFView - v0.0.8 ###
+RUN echo "ckan/ckanext-pdfview" && \
+ pip3 install -e git+https://github.com/ckan/ckanext-pdfview.git@0.0.8#egg=ckanext-pdfview
+
+# Used to configure the container environment by setting environment variables, creating users, running initialization scripts, .etc
+COPY docker-entrypoint.d/* /docker-entrypoint.d/
+
+# Apply any patches needed to CKAN core
+COPY patches ${APP_DIR}/patches
+
+RUN for d in $APP_DIR/patches/*; do \
+ if [ -d $d ]; then \
+ for f in `ls $d/*.patch | sort -g`; do \
+ cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
+ done ; \
+ fi ; \
+ done
+
+CMD ["/srv/app/start_ckan.sh"]
\ No newline at end of file
From 404f72b63ae88054c7c3abd32a23b9402de98079 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 09:55:59 +0000
Subject: [PATCH 13/34] Update README - Add info about plugins
---
.env | 16 +++++++++-------
README.md | 6 +++---
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git a/.env b/.env
index dfff212..6161cc7 100644
--- a/.env
+++ b/.env
@@ -3,7 +3,6 @@ NGINX_CONTAINER_NAME=nginx
REDIS_CONTAINER_NAME=redis
POSTGRESQL_CONTAINER_NAME=db
SOLR_CONTAINER_NAME=solr
-DATAPUSHER_CONTAINER_NAME=datapusher
CKAN_CONTAINER_NAME=ckan
WORKER_CONTAINER_NAME=ckan-worker
@@ -33,11 +32,10 @@ CKAN_SITE_ID=default
CKAN_SITE_URL=http://localhost:5000
CKAN_PORT=5000
CKAN_PORT_HOST=5000
-CKAN___BEAKER__SESSION__SECRET=39ASDK913r4jas0
+CKAN___BEAKER__SESSION__SECRET=CHANGE_ME
# See https://docs.ckan.org/en/latest/maintaining/configuration.html#api-token-settings
-CKAN___API_TOKEN__JWT__ENCODE__SECRET=string:WqVhDAXsF7E9FvzVnNqdsKpDT9wZbiu6hzxyMijv4FN3B
-CKAN___API_TOKEN__JWT__DECODE__SECRET=string:WqVhDAXsF7E9FvzVnNqdsKpDT9wZbiu6hzxyMijv4FN3B
-CKAN___DATAPUSHER__API_TOKEN=api_token
+CKAN___API_TOKEN__JWT__ENCODE__SECRET=string:CHANGE_ME
+CKAN___API_TOKEN__JWT__DECODE__SECRET=string:CHANGE_ME
CKAN_SYSADMIN_NAME=ckan_admin
CKAN_SYSADMIN_PASSWORD=test1234
CKAN_SYSADMIN_EMAIL=your_email@example.com
@@ -72,7 +70,7 @@ REDIS_VERSION=6
CKAN_REDIS_URL=redis://redis:6379/1
TEST_CKAN_REDIS_URL=redis://redis:6379/1
-# Datapusher
+# Datapusher - Deprecated by ckanext-xloader
DATAPUSHER_VERSION=0.0.19
CKAN_DATAPUSHER_URL=http://datapusher:8800
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
@@ -84,13 +82,17 @@ NGINX_PORT=80
NGINX_SSLPORT=443
# Extensions
-CKAN__PLUGINS="envvars stats text_view image_view recline_view resourcedictionary datastore datapusher harvest ckan_harvester spatial_metadata spatial_query spatial_harvest_metadata_api csw_harvester waf_harvester doc_harvester resource_proxy geo_view geojson_view wmts_view shp_view dcat dcat_rdf_harvester dcat_json_harvester dcat_json_interface scheming_datasets scheming_groups scheming_organizations pdf_view pages"
+CKAN__PLUGINS="envvars stats text_view image_view recline_view resourcedictionary datastore xloader harvest ckan_harvester spatial_metadata spatial_query spatial_harvest_metadata_api csw_harvester waf_harvester doc_harvester resource_proxy geo_view geojson_view wmts_view shp_view dcat dcat_rdf_harvester dcat_json_harvester dcat_json_interface scheming_datasets scheming_groups scheming_organizations pdf_view pages"
CKAN__HARVEST__MQ__TYPE=redis
CKAN__HARVEST__MQ__HOSTNAME=redis
CKAN__HARVEST__MQ__PORT=6379
CKAN__HARVEST__MQ__REDIS_DB=1
+# ckanext-xloader
+CKANEXT__XLOADER__API_TOKEN=api_token
+CKANEXT__XLOADER__JOBS__DB_URI=postgresql://ckan:ckan@db/ckan
+
# ckanext-dcat
CKANEXT__DCAT__BASE_URI=https://localhost:5000
CKANEXT__DCAT__RDF_PROFILES="euro_dcat_ap_2 euro_dcat_ap"
diff --git a/README.md b/README.md
index 4466221..52484ff 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
CKAN Docker Compose - Open Data & GIS
-
+
@@ -24,7 +24,6 @@ Available components:
* The CKAN images used are from the official CKAN [ckan-docker-base](https://github.com/ckan/ckan-docker-base) repo
The non-CKAN images are as follows:
-* DataPusher: CKAN's [pre-configured DataPusher image](https://github.com/ckan/ckan-docker-base/tree/main/datapusher).
* PostgreSQL: Official PostgreSQL image. Database files are stored in a named volume.
* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). Index data is stored in a named volume.
* Redis: standard Redis image
@@ -46,7 +45,8 @@ Information about extensions installed in the `main` image. More info described
|-------------|-----------------------------------------------------------------------------------------|-------------|------------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.10.0 | Completed | ✔️ | ✔️ | Stable installation for version 2.10.0 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.10.0 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
-| Core + | [Datapusher](https://github.com/mjanez/ckan-docker) | 0.0.19 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose.
**TODO**: Upgrade to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. A replacement for DataPusher. |
+| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
+| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ❌ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
From 8bc880f5e7f7551146fe4be5af072ec96651d0f9 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 11:01:51 +0100
Subject: [PATCH 14/34] Update README.md
---
README.md | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 52484ff..e4df69a 100644
--- a/README.md
+++ b/README.md
@@ -146,10 +146,17 @@ Use this if you are a maintainer and will not be making code changes to CKAN or
Copy the included `.env.example` and rename it to `.env`. Modify it depending on your own needs.
->*Note**:
+>**Note**:
> Please note that when accessing CKAN directly (via a browser) ie: not going through NGINX you will need to make sure you have "ckan" set up to be an alias to localhost in the local hosts file. Either that or you will need to change the `.env` entry for `CKAN_SITE_URL`
-Using the default values on the `.env.example` file will get you a working CKAN instance. There is a sysadmin user created by default with the values defined in `CKAN_SYSADMIN_NAME` and `CKAN_SYSADMIN_PASSWORD`(`ckan_admin` and `test1234` by default). This should be obviously changed before running this setup as a public CKAN instance.
+>**Warning**:
+> Using the default values on the `.env.example` file will get you a working CKAN instance. There is a sysadmin user created by default with the values defined in `CKAN_SYSADMIN_NAME` and `CKAN_SYSADMIN_PASSWORD`(`ckan_admin` and `test1234` by default). This should be obviously changed before running this setup as a public CKAN instance.
+
+Clone project
+```shell
+cd /path/to/my/project
+git clone https://github.com/mjanez/ckan-docker.git
+```
To build the images:
@@ -357,4 +364,4 @@ Available components:
[^1]: Development environment.
[^2]: Production environment.
[^3]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
-[^4]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
\ No newline at end of file
+[^4]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
From 79f0b37255efb7c859006b1ae009756cccf01202 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 15:36:55 +0000
Subject: [PATCH 15/34] ckan-docker 2.9.8
---
.env | 4 +-
.github/workflows/docker.yml | 47 +++++++++++++++
README.md | 4 +-
ckan/Dockerfile | 109 +++++++++++++++++++++++++++++++++--
ckan/Dockerfile.dev | 3 +-
ckan/setup/start_ckan.sh | 11 ++++
docker-compose.yml | 1 +
7 files changed, 170 insertions(+), 9 deletions(-)
create mode 100644 .github/workflows/docker.yml
diff --git a/.env b/.env
index 6161cc7..5c4a951 100644
--- a/.env
+++ b/.env
@@ -27,7 +27,7 @@ TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test
# CKAN core
-CKAN_VERSION=2.10.0
+CKAN_VERSION=2.9.8
CKAN_SITE_ID=default
CKAN_SITE_URL=http://localhost:5000
CKAN_PORT=5000
@@ -61,7 +61,7 @@ CKAN__RESOURCE_PROXY__TIMEOUT=10
CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view harvest ckan_harvester wmts_view geojson_view geo_view shp_view pdf_view"
# Solr
-SOLR_IMAGE_VERSION=2.10-spatial
+SOLR_IMAGE_VERSION=2.9-solr8-spatial
CKAN_SOLR_URL=http://solr:8983/solr/ckan
TEST_CKAN_SOLR_URL=http://solr:8983/solr/ckan
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
new file mode 100644
index 0000000..0eb6ca6
--- /dev/null
+++ b/.github/workflows/docker.yml
@@ -0,0 +1,47 @@
+name: Docker
+
+on:
+ push:
+ branches: [ 'ckan-2.9.8' ]
+ tags: [ 'v*.*.*' ]
+ pull_request:
+ branches: [ 'ckan-2.9.8' ]
+
+env:
+ REGISTRY: ghcr.io
+ IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+ docker:
+ runs-on: ubuntu-latest
+ steps:
+ -
+ name: Set up QEMU
+ uses: docker/setup-qemu-action@v2
+ -
+ name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v2
+ -
+ name: Checkout
+ uses: actions/checkout@v1
+ -
+ name: Login to registry
+ if: github.event_name != 'pull_request'
+ uses: docker/login-action@v2
+ with:
+ registry: ${{ env.REGISTRY }}
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ -
+ name: Extract Docker metadata
+ id: meta
+ uses: docker/metadata-action@v4
+ with:
+ images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+ -
+ name: Build and push
+ uses: docker/build-push-action@v3
+ with:
+ push: ${{ github.event_name != 'pull_request' }}
+ tags: ${{ steps.meta.outputs.tags }}
+ labels: ${{ steps.meta.outputs.labels }}
\ No newline at end of file
diff --git a/README.md b/README.md
index 52484ff..777174f 100644
--- a/README.md
+++ b/README.md
@@ -43,8 +43,8 @@ Information about extensions installed in the `main` image. More info described
| **Element** | **Description** | **version** | **Status** | **DEV**[^1] | **PRO**[^2] | **Remarks** |
|-------------|-----------------------------------------------------------------------------------------|-------------|------------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.10.0 | Completed | ✔️ | ✔️ | Stable installation for version 2.10.0 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
-| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.10.0 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
+| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.7 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
+| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ❌ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 2e27be0..9741f50 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -1,12 +1,113 @@
-FROM ckan/ckan-base:2.9.7
+FROM alpine:3.13
-# Set up environment variables
+# Internals, you probably don't need to change these
ENV APP_DIR=/srv/app
+ENV SRC_DIR=/srv/app/src
+ENV CKAN_INI=${APP_DIR}/ckan.ini
+ENV PIP_SRC=${SRC_DIR}
+ENV CKAN_STORAGE_PATH=/var/lib/ckan
+ENV GIT_URL=https://github.com/ckan/ckan.git
ENV TZ=UTC
RUN echo ${TZ} > /etc/timezone
+# CKAN version to build
+ENV GIT_BRANCH=ckan-2.9.8
+# Customize these on the .env file if needed
+ENV CKAN_SITE_URL=http://localhost:5000
+ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
-# Set up PROJ
-RUN apk add geos-dev proj proj-util proj-dev
+# UWSGI options
+ENV UWSGI_HARAKIRI=50
+
+WORKDIR ${APP_DIR}
+
+# Install necessary packages to run CKAN
+RUN apk add --no-cache tzdata \
+ git \
+ gettext \
+ postgresql-client \
+ python3 \
+ libxml2 \
+ libxslt \
+ musl-dev \
+ uwsgi \
+ uwsgi-http \
+ uwsgi-corerouter \
+ uwsgi-python \
+ py3-gevent \
+ uwsgi-gevent \
+ libmagic \
+ curl \
+ patch \
+ sudo && \
+ # Packages to build CKAN requirements and plugins
+ apk add --no-cache --virtual .build-deps \
+ postgresql-dev \
+ gcc \
+ make \
+ g++ \
+ autoconf \
+ automake \
+ libtool \
+ python3-dev \
+ libxml2-dev \
+ libxslt-dev \
+ linux-headers \
+ openssl-dev \
+ libffi-dev \
+ cargo \
+ # Set up PROJ
+ geos-dev \
+ proj \
+ proj-util \
+ proj-dev && \
+ # Create SRC_DIR
+ mkdir -p ${SRC_DIR} && \
+ # Install pip, supervisord and uwsgi
+ curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
+ python3 ${SRC_DIR}/get-pip.py && \
+ pip3 install supervisor && \
+ mkdir /etc/supervisord.d && \
+ #pip wheel --wheel-dir=/wheels uwsgi gevent && \
+ rm -rf ${SRC_DIR}/get-pip.py
+
+COPY setup/supervisord.conf /etc
+
+# Install CKAN
+RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
+ cd ${SRC_DIR}/ckan && \
+ cp who.ini ${APP_DIR} && \
+ pip3 install -r requirement-setuptools.txt && \
+ pip3 install --no-binary markdown -r requirements.txt && \
+ # Install CKAN envvars to support loading config from environment variables
+ pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
+ # Create and update CKAN config
+ ckan generate config ${CKAN_INI} && \
+ ckan config-tool ${CKAN_INI} "beaker.session.secret = " && \
+ ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
+
+# Create a local user and group to run the app
+RUN addgroup -g 92 -S ckan && \
+ adduser -u 92 -h /srv/app -s /bin/bash -D -G ckan ckan
+
+# Create local storage folder
+RUN mkdir -p ${CKAN_STORAGE_PATH} && \
+ chown -R ckan:ckan ${CKAN_STORAGE_PATH}
+
+COPY setup/prerun.py ${APP_DIR}
+COPY setup/start_ckan.sh ${APP_DIR}
+ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${APP_DIR}
+
+# Create entrypoint directory for children image scripts
+ONBUILD RUN mkdir /docker-entrypoint.d
+
+RUN chown ckan -R /srv/app
+
+RUN addgroup sudo && \
+ adduser ckan sudo
+
+EXPOSE 5000
+
+HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
# Make sure both files are not exactly the same
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
diff --git a/ckan/Dockerfile.dev b/ckan/Dockerfile.dev
index a4f79c6..1e7b26e 100644
--- a/ckan/Dockerfile.dev
+++ b/ckan/Dockerfile.dev
@@ -1,4 +1,5 @@
-FROM ckan/ckan-base:2.9.7-dev
+# syntax = edrevo/dockerfile-plus
+INCLUDE+ Dockerfile
# Set up environment variables
ENV APP_DIR=/srv/app
diff --git a/ckan/setup/start_ckan.sh b/ckan/setup/start_ckan.sh
index e1fb8e1..d6d437d 100755
--- a/ckan/setup/start_ckan.sh
+++ b/ckan/setup/start_ckan.sh
@@ -1,5 +1,16 @@
#!/bin/bash
+# Set up the Secret key used by Beaker and Flask
+# This can be overriden using a CKAN___BEAKER__SESSION__SECRET env var
+if grep -E "beaker.session.secret ?= ?$" ckan.ini
+then
+ echo "Setting beaker.session.secret in ini file"
+ ckan config-tool $CKAN_INI "beaker.session.secret=$(python3 -c 'import secrets; print(secrets.token_urlsafe())')"
+ JWT_SECRET=$(python3 -c 'import secrets; print("string:" + secrets.token_urlsafe())')
+ ckan config-tool $CKAN_INI "api_token.jwt.encode.secret=${JWT_SECRET}"
+ ckan config-tool $CKAN_INI "api_token.jwt.decode.secret=${JWT_SECRET}"
+fi
+
# Run the prerun script to init CKAN and create the default admin user
sudo -u ckan -EH python3 prerun.py
diff --git a/docker-compose.yml b/docker-compose.yml
index 75b2ede..b5d7b96 100755
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -26,6 +26,7 @@ services:
dockerfile: Dockerfile
args:
- TZ=${TZ}
+ image: ghcr.io/mjanez/ckan-docker:ckan-2.9.8
env_file:
- .env
depends_on:
From 2d0e9a34d7bab5d35373ea4bf06a4be6f0a3fa0f Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 16:40:05 +0100
Subject: [PATCH 16/34] Fix Dockerfile
---
ckan/Dockerfile | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 9741f50..4927e79 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -54,7 +54,7 @@ RUN apk add --no-cache tzdata \
linux-headers \
openssl-dev \
libffi-dev \
- cargo \
+ cargo && \
# Set up PROJ
geos-dev \
proj \
@@ -169,4 +169,4 @@ RUN for d in $APP_DIR/patches/*; do \
fi ; \
done
-CMD ["/srv/app/start_ckan.sh"]
\ No newline at end of file
+CMD ["/srv/app/start_ckan.sh"]
From 925835d93bfd23992942190027d3582f501724b2 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 16:44:54 +0100
Subject: [PATCH 17/34] Update docker.yml
---
.github/workflows/docker.yml | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
index 0eb6ca6..f2fbc4e 100644
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -21,6 +21,9 @@ jobs:
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
+ with:
+ context: ./ckan
+ file: ./ckan/Dockerfile
-
name: Checkout
uses: actions/checkout@v1
@@ -44,4 +47,4 @@ jobs:
with:
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
- labels: ${{ steps.meta.outputs.labels }}
\ No newline at end of file
+ labels: ${{ steps.meta.outputs.labels }}
From 5766a7f79406147a3501688c033e1e7004fd4408 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 16:51:07 +0100
Subject: [PATCH 18/34] Update docker.yml
---
.github/workflows/docker.yml | 36 +++++++++++++++++-------------------
1 file changed, 17 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
index f2fbc4e..688cf5f 100644
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -1,11 +1,11 @@
-name: Docker
+name: Build CKAN Docker
on:
push:
- branches: [ 'ckan-2.9.8' ]
- tags: [ 'v*.*.*' ]
+ branches: [ 'main' ]
+ tags: [ 'ckan-*.*.*' ]
pull_request:
- branches: [ 'ckan-2.9.8' ]
+ branches: [ 'main' ]
env:
REGISTRY: ghcr.io
@@ -15,36 +15,34 @@ jobs:
docker:
runs-on: ubuntu-latest
steps:
- -
- name: Set up QEMU
+ - name: Set up QEMU
uses: docker/setup-qemu-action@v2
- -
- name: Set up Docker Buildx
+
+ - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- with:
- context: ./ckan
- file: ./ckan/Dockerfile
- -
- name: Checkout
+
+ - name: Checkout
uses: actions/checkout@v1
- -
- name: Login to registry
+
+ - name: Login to registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- -
- name: Extract Docker metadata
+
+ - name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
- -
- name: Build and push
+
+ - name: CKAN Build and push
uses: docker/build-push-action@v3
with:
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
+ context: ./ckan
+ file: ./ckan/Dockerfile
From cd3dfa0985875ede8f28fd5713b2a08cefc3df5a Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 16:58:36 +0100
Subject: [PATCH 19/34] Fix Dockerfile
---
ckan/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 4927e79..49a621b 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -54,7 +54,7 @@ RUN apk add --no-cache tzdata \
linux-headers \
openssl-dev \
libffi-dev \
- cargo && \
+ cargo \
# Set up PROJ
geos-dev \
proj \
From 5fb6a14fb5bda6604d21897621dadaaf512a0316 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 16:54:17 +0000
Subject: [PATCH 20/34] Fix README
---
README.md | 38 ++++++++++++++++++++++++--------------
1 file changed, 24 insertions(+), 14 deletions(-)
diff --git a/README.md b/README.md
index 46b4554..fed77a9 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
CKAN Docker Compose - Open Data & GIS
-
+
@@ -20,19 +20,28 @@
## Overview
Contains Docker images for the different components of CKAN Cloud and a Docker compose environment (based on [ckan](https://github.com/ckan/ckan)) for development and testing Open Data portals.
+>**Warning**:
+>This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: [CKAN documentation: Installation](https://docs.ckan.org/en/latest/maintaining/installing/index.html).
+
Available components:
-* The CKAN images used are from the official CKAN [ckan-docker-base](https://github.com/ckan/ckan-docker-base) repo
+* CKAN custom image based on the official CKAN repo [ckan-docker-base](https://github.com/ckan/ckan-docker-base)
The non-CKAN images are as follows:
* PostgreSQL: Official PostgreSQL image. Database files are stored in a named volume.
-* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). Index data is stored in a named volume.
+* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). The index data is stored in a named volume and has a spatial schema. [^1]
* Redis: standard Redis image
* NGINX: latest stable nginx image that includes SSL and Non-SSL endpoints
+| Repository | Type | Docker tag | Size | Notes |
+| --- | --- | --- | --- | --- |
+| PostgreSQL 12.14 | base image | [`postgres/postgres:12-alpine`](https://hub.docker.com/layers/library/postgres/12-alpine/images/sha256-e07198f71e85eccd6c351bb4eb0b2ff833bef2dd6ea314f6442c2dc47535fbb7?context=explore) | 87.09 MB | Custom Dockerfile: [`postgresql/Dockerfile`](/postgresql/Dockerfile) |
+| Solr 8.11.1 | base image | [`ckan/ckan-solr:2.9-solr8-spatial`](https://registry.hub.docker.com/layers/ckan/ckan-solr/2.9-solr8-spatial/images/sha256-b5ee4979891c7dd1f10d2ac2cbdd4d80ff656879edb0f0493616be7b4cf8bc3a?context=explore) | 331.1 MB | - |
+| Redis 6.2.11 | base image | [`redis/redis:6`](https://hub.docker.com/layers/library/redis/6/images/sha256-e10c69dc5f9b27c47874060962fdaa26ec0df592205d6d21402945267081b434?context=explore) | 39.32 MB | - |
+| NGINX 1.22.1 | base image | [`nginx:stable-alpine`](https://hub.docker.com/layers/library/nginx/stable-alpine/images/sha256-ff2a5d557ca22fa93669f5e70cfbeefda32b98f8fd3d33b38028c582d700f93a?context=explore) | 9.74 MB | Custom Dockerfile: [`nginx/Dockerfile`](/nginx/Dockerfile) |
+
+
The site is configured using environment variables that you can set in the `.env` file.
->**Warning**:
->This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: [CKAN documentation: Installation](https://docs.ckan.org/en/latest/maintaining/installing/index.html).
### ckan-docker roadmap
Information about extensions installed in the `main` image. More info described in the [Extending the base images](#extending-the-base-images)
@@ -41,9 +50,9 @@ Information about extensions installed in the `main` image. More info described
> Switch branches to see the `roadmap` for other projects: [ckan-docker/branches](https://github.com/mjanez/ckan-docker/branches)
-| **Element** | **Description** | **version** | **Status** | **DEV**[^1] | **PRO**[^2] | **Remarks** |
+| **Element** | **Description** | **version** | **Status** | **DEV**[^2] | **PRO**[^3] | **Remarks** |
|-------------|-----------------------------------------------------------------------------------------|-------------|------------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.7 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
+| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.8 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ❌ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
@@ -51,7 +60,7 @@ Information about extensions installed in the `main` image. More info described
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-dcat](https://github.com/mjanez/ckanext-dcat) | 1.2.0 | Completed | ✔️ | ✔️ | Stable installation, include DCAT-AP 2.1 profile compatible with GeoDCAT-AP. |
-| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^3] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
+| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^4] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
| Extension | [ckanext-resourcedictionary](https://github.com/OpenDataGIS/ckanext-resourcedictionary) | main | Completed | ✔️ | ✔️ | Stable installation. This extension extends the default CKAN Data Dictionary functionality by adding possibility to create data dictionary before actual data is uploaded to datastore. |
| Extension | [ckanext-pages](https://github.com/ckan/ckanext-pages) | 0.5.1 | Completed | ✔️ | ✔️ | Stable installation. This extension gives you an easy way to add simple pages to CKAN. |
| Extension | [ckanext-pdfview](https://github.com/ckan/ckanext-pdfview) | 0.0.8 | Completed | ✔️ | ✔️ | Stable installation. This extension provides a view plugin for PDF files using an html object tag. |
@@ -248,7 +257,7 @@ ckan -c /srv/app/ckan.ini validation init-db
And then in our `Dockerfile.dev` file we install the extension and copy the initialization scripts:
```Dockerfile
-FROM ckan/ckan-base:2.9.7-dev
+FROM ckan/ckan-base:2.9.8-dev
RUN pip install -e git+https://github.com/frictionlessdata/ckanext-validation.git#egg=ckanext-validation && \
pip install -r https://raw.githubusercontent.com/frictionlessdata/ckanext-validation/master/requirements.txt
@@ -354,14 +363,15 @@ For more information please see [ckanext-envvars](https://github.com/okfn/ckanex
### ckan-pycsw
-[ckan-pycsw](https://github.com/mjanez/ckan-pycsw) is a docker compose environment (based on [pycsw](https://github.com/geopython/pycsw)) for development and testing with CKAN Open Data portals.[^4]
+[ckan-pycsw](https://github.com/mjanez/ckan-pycsw) is a docker compose environment (based on [pycsw](https://github.com/geopython/pycsw)) for development and testing with CKAN Open Data portals.[^5]
Available components:
* **pycsw**: The pycsw app. An [OARec](https://ogcapi.ogc.org/records) and [OGC CSW](https://opengeospatial.org/standards/cat) server implementation written in Python.
* **ckan2pycsw**: Software to achieve interoperability with the open data portals based on CKAN. To do this, ckan2pycsw reads data from an instance using the CKAN API, generates ISO-19115/ISO-19139 metadata using [pygeometa](https://geopython.github.io/pygeometa/), or a custom schema that is based on a customized CKAN schema, and populates a [pycsw](https://pycsw.org/) instance that exposes the metadata using CSW and OAI-PMH.
-[^1]: Development environment.
-[^2]: Production environment.
-[^3]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
-[^4]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
+[^1]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)
+[^2]: Development environment.
+[^3]: Production environment.
+[^4]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
+[^5]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
From e3ab3927369e86be7568447c15e9eff98647c8be Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Wed, 22 Mar 2023 17:03:33 +0000
Subject: [PATCH 21/34] Update ckanext-xloader
---
.env | 2 +-
README.md | 2 +-
ckan/docker-entrypoint.d/setup_xloader.sh | 3 +++
samples/.env.example | 24 ++++++++++++-----------
4 files changed, 18 insertions(+), 13 deletions(-)
diff --git a/.env b/.env
index 5c4a951..de58312 100644
--- a/.env
+++ b/.env
@@ -58,7 +58,7 @@ CKAN__RESOURCE_PROXY__MAX_FILE_SIZE=50048576
CKAN__RESOURCE_PROXY__CHUNK_SIZE=4096
## Default timeout for fetching proxied items
CKAN__RESOURCE_PROXY__TIMEOUT=10
-CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view harvest ckan_harvester wmts_view geojson_view geo_view shp_view pdf_view"
+CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view recline_grid_view recline_graph_view wmts_view webpage_view geojson_view geo_view shp_view pdf_view"
# Solr
SOLR_IMAGE_VERSION=2.9-solr8-spatial
diff --git a/README.md b/README.md
index fed77a9..bd4ae58 100644
--- a/README.md
+++ b/README.md
@@ -55,7 +55,7 @@ Information about extensions installed in the `main` image. More info described
| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.8 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
-| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ❌ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
+| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ✔️ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
diff --git a/ckan/docker-entrypoint.d/setup_xloader.sh b/ckan/docker-entrypoint.d/setup_xloader.sh
index d34e245..5f738c1 100644
--- a/ckan/docker-entrypoint.d/setup_xloader.sh
+++ b/ckan/docker-entrypoint.d/setup_xloader.sh
@@ -9,3 +9,6 @@ ckan config-tool $CKAN_INI \
# Create ckanext-xloader API_TOKEN
echo "Set up ckanext.xloader.api_token in the CKAN config file"
ckan config-tool $CKAN_INI "ckanext.xloader.api_token = $(ckan -c $CKAN_INI user token add ckan_admin xloader | tail -n 1 | tr -d '\t')"
+
+# Setup worker
+ckan config-tool $CKAN_INI jobs worker
\ No newline at end of file
diff --git a/samples/.env.example b/samples/.env.example
index 2a0e497..de58312 100644
--- a/samples/.env.example
+++ b/samples/.env.example
@@ -3,7 +3,6 @@ NGINX_CONTAINER_NAME=nginx
REDIS_CONTAINER_NAME=redis
POSTGRESQL_CONTAINER_NAME=db
SOLR_CONTAINER_NAME=solr
-DATAPUSHER_CONTAINER_NAME=datapusher
CKAN_CONTAINER_NAME=ckan
WORKER_CONTAINER_NAME=ckan-worker
@@ -28,16 +27,15 @@ TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test
# CKAN core
-CKAN_VERSION=2.10.0
+CKAN_VERSION=2.9.8
CKAN_SITE_ID=default
CKAN_SITE_URL=http://localhost:5000
CKAN_PORT=5000
CKAN_PORT_HOST=5000
-CKAN___BEAKER__SESSION__SECRET=39ASDK913r4jas0
+CKAN___BEAKER__SESSION__SECRET=CHANGE_ME
# See https://docs.ckan.org/en/latest/maintaining/configuration.html#api-token-settings
-CKAN___API_TOKEN__JWT__ENCODE__SECRET=string:WqVhDAXsF7E9FvzVnNqdsKpDT9wZbiu6hzxyMijv4FN3B
-CKAN___API_TOKEN__JWT__DECODE__SECRET=string:WqVhDAXsF7E9FvzVnNqdsKpDT9wZbiu6hzxyMijv4FN3B
-CKAN___DATAPUSHER__API_TOKEN = api_token
+CKAN___API_TOKEN__JWT__ENCODE__SECRET=string:CHANGE_ME
+CKAN___API_TOKEN__JWT__DECODE__SECRET=string:CHANGE_ME
CKAN_SYSADMIN_NAME=ckan_admin
CKAN_SYSADMIN_PASSWORD=test1234
CKAN_SYSADMIN_EMAIL=your_email@example.com
@@ -60,10 +58,10 @@ CKAN__RESOURCE_PROXY__MAX_FILE_SIZE=50048576
CKAN__RESOURCE_PROXY__CHUNK_SIZE=4096
## Default timeout for fetching proxied items
CKAN__RESOURCE_PROXY__TIMEOUT=10
-CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view harvest ckan_harvester wmts_view geojson_view geo_view shp_view pdf_view"
+CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view recline_grid_view recline_graph_view wmts_view webpage_view geojson_view geo_view shp_view pdf_view"
# Solr
-SOLR_IMAGE_VERSION=2.10-spatial
+SOLR_IMAGE_VERSION=2.9-solr8-spatial
CKAN_SOLR_URL=http://solr:8983/solr/ckan
TEST_CKAN_SOLR_URL=http://solr:8983/solr/ckan
@@ -72,7 +70,7 @@ REDIS_VERSION=6
CKAN_REDIS_URL=redis://redis:6379/1
TEST_CKAN_REDIS_URL=redis://redis:6379/1
-# Datapusher
+# Datapusher - Deprecated by ckanext-xloader
DATAPUSHER_VERSION=0.0.19
CKAN_DATAPUSHER_URL=http://datapusher:8800
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
@@ -84,13 +82,17 @@ NGINX_PORT=80
NGINX_SSLPORT=443
# Extensions
-CKAN__PLUGINS="envvars stats text_view image_view recline_view resourcedictionary datastore datapusher harvest ckan_harvester spatial_metadata spatial_query spatial_harvest_metadata_api csw_harvester waf_harvester doc_harvester resource_proxy geo_view geojson_view wmts_view shp_view dcat dcat_rdf_harvester dcat_json_harvester dcat_json_interface scheming_datasets scheming_groups scheming_organizations pdf_view pages"
+CKAN__PLUGINS="envvars stats text_view image_view recline_view resourcedictionary datastore xloader harvest ckan_harvester spatial_metadata spatial_query spatial_harvest_metadata_api csw_harvester waf_harvester doc_harvester resource_proxy geo_view geojson_view wmts_view shp_view dcat dcat_rdf_harvester dcat_json_harvester dcat_json_interface scheming_datasets scheming_groups scheming_organizations pdf_view pages"
CKAN__HARVEST__MQ__TYPE=redis
CKAN__HARVEST__MQ__HOSTNAME=redis
CKAN__HARVEST__MQ__PORT=6379
CKAN__HARVEST__MQ__REDIS_DB=1
+# ckanext-xloader
+CKANEXT__XLOADER__API_TOKEN=api_token
+CKANEXT__XLOADER__JOBS__DB_URI=postgresql://ckan:ckan@db/ckan
+
# ckanext-dcat
CKANEXT__DCAT__BASE_URI=https://localhost:5000
CKANEXT__DCAT__RDF_PROFILES="euro_dcat_ap_2 euro_dcat_ap"
@@ -114,6 +116,6 @@ CKANEXT__GEOVIEW__OL_VIEWER__FORMATS="wms wfs geojson gml kml"
CKANEXT__GEOVIEW__SHP_VIEWER__SRID=3857
CKANEXT__GEOVIEW__SHP_VIEWER__ENCODING=UTF-8
-#ckanext-sparql
+#TODO: ckanext-sparql - Not implemented yet
CKANEXT__SPARQL__ENDPOINT_URL=http://dbpedia.org/sparql
CKANEXT__SPARQL__HIDE_ENDPOINT_URL=False
\ No newline at end of file
From 531edd9e0bc34ee00fbc4ce3bfd891d69f88dc80 Mon Sep 17 00:00:00 2001
From: mnjnz <96422458+mjanez@users.noreply.github.com>
Date: Thu, 23 Mar 2023 09:59:13 +0100
Subject: [PATCH 22/34] Update actions - Updated build.yml to test only
auxiliary images. - Update docker.yml to build and upload the image to the
Github registry.
---
.github/workflows/build.yml | 11 +----
.github/workflows/docker.yml | 80 ++++++++++++++++++------------------
docker-compose.yml | 2 +-
3 files changed, 43 insertions(+), 50 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 95cde00..91123cf 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -1,4 +1,4 @@
-name: Build CKAN Docker
+name: Build CKAN Docker auxiliary images
on:
# Trigger the workflow on push or pull request,
@@ -45,12 +45,3 @@ jobs:
file: ./solr/Dockerfile
push: false
tags: kowhai/ckan-docker-solr:test-build-only
-
- - name: CKAN build
- uses: docker/build-push-action@v2
- with:
- context: ./ckan
- file: ./ckan/Dockerfile
- push: false
- tags: kowhai/ckan-docker-ckan:test-build-only
-
\ No newline at end of file
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
index 688cf5f..6c6c3a6 100644
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -1,48 +1,50 @@
-name: Build CKAN Docker
+name: Build & Push CKAN-Spatial Docker image
on:
- push:
- branches: [ 'main' ]
- tags: [ 'ckan-*.*.*' ]
- pull_request:
- branches: [ 'main' ]
+ # Trigger the workflow after build.yml,
+ # but only for the master branch
+ workflow_run:
+ workflows: ["Build CKAN Docker auxiliary images"]
+ branches: [master]
+ types:
+ - completed
env:
REGISTRY: ghcr.io
- IMAGE_NAME: ${{ github.repository }}
+ IMAGE_NAME: mjanez/ckan-spatial
-jobs:
- docker:
+
+jobs:
+ build:
runs-on: ubuntu-latest
- steps:
- - name: Set up QEMU
- uses: docker/setup-qemu-action@v2
+ steps:
+ - uses: actions/checkout@v2
+
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v1
- - name: Set up Docker Buildx
- uses: docker/setup-buildx-action@v2
+ - name: Set up QEMU
+ uses: docker/setup-qemu-action@v1
+
+ - name: Login to registry
+ if: github.event_name != 'pull_request'
+ uses: docker/login-action@v2
+ with:
+ registry: ${{ env.REGISTRY }}
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
- - name: Checkout
- uses: actions/checkout@v1
-
- - name: Login to registry
- if: github.event_name != 'pull_request'
- uses: docker/login-action@v2
- with:
- registry: ${{ env.REGISTRY }}
- username: ${{ github.actor }}
- password: ${{ secrets.GITHUB_TOKEN }}
-
- - name: Extract Docker metadata
- id: meta
- uses: docker/metadata-action@v4
- with:
- images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
-
- - name: CKAN Build and push
- uses: docker/build-push-action@v3
- with:
- push: ${{ github.event_name != 'pull_request' }}
- tags: ${{ steps.meta.outputs.tags }}
- labels: ${{ steps.meta.outputs.labels }}
- context: ./ckan
- file: ./ckan/Dockerfile
+ - name: Extract Docker metadata
+ id: meta
+ uses: docker/metadata-action@v4
+ with:
+ images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+
+ - name: CKAN Build and push
+ uses: docker/build-push-action@v3
+ with:
+ push: ${{ github.event_name != 'pull_request' }}
+ tags: ${{ steps.meta.outputs.tags }}
+ labels: ${{ steps.meta.outputs.labels }}
+ context: ./ckan
+ file: ./ckan/Dockerfile
diff --git a/docker-compose.yml b/docker-compose.yml
index b5d7b96..e51be94 100755
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -26,7 +26,7 @@ services:
dockerfile: Dockerfile
args:
- TZ=${TZ}
- image: ghcr.io/mjanez/ckan-docker:ckan-2.9.8
+ image: ghcr.io/mjanez/ckan-spatial:master
env_file:
- .env
depends_on:
From 58e9c9a2d50e3fd2d28767eb15d95b6826451db4 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 23 Mar 2023 10:34:40 +0100
Subject: [PATCH 23/34] Update PostgreSQL 15
---
README.md | 12 +++++++++---
postgresql/Dockerfile | 2 +-
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index bd4ae58..83e8cb5 100644
--- a/README.md
+++ b/README.md
@@ -34,7 +34,7 @@ The non-CKAN images are as follows:
| Repository | Type | Docker tag | Size | Notes |
| --- | --- | --- | --- | --- |
-| PostgreSQL 12.14 | base image | [`postgres/postgres:12-alpine`](https://hub.docker.com/layers/library/postgres/12-alpine/images/sha256-e07198f71e85eccd6c351bb4eb0b2ff833bef2dd6ea314f6442c2dc47535fbb7?context=explore) | 87.09 MB | Custom Dockerfile: [`postgresql/Dockerfile`](/postgresql/Dockerfile) |
+| PostgreSQL 15.2 | base image | [`postgres/postgres:15-alpine`](https://hub.docker.com/layers/library/postgres/15-alpine/images/sha256-53a02ecbe9d18ff6476e6651c34811da39f054424c725fc15d2b480fc3fab877?context=explore) | 89.74 MB | Custom Dockerfile: [`postgresql/Dockerfile`](/postgresql/Dockerfile) |
| Solr 8.11.1 | base image | [`ckan/ckan-solr:2.9-solr8-spatial`](https://registry.hub.docker.com/layers/ckan/ckan-solr/2.9-solr8-spatial/images/sha256-b5ee4979891c7dd1f10d2ac2cbdd4d80ff656879edb0f0493616be7b4cf8bc3a?context=explore) | 331.1 MB | - |
| Redis 6.2.11 | base image | [`redis/redis:6`](https://hub.docker.com/layers/library/redis/6/images/sha256-e10c69dc5f9b27c47874060962fdaa26ec0df592205d6d21402945267081b434?context=explore) | 39.32 MB | - |
| NGINX 1.22.1 | base image | [`nginx:stable-alpine`](https://hub.docker.com/layers/library/nginx/stable-alpine/images/sha256-ff2a5d557ca22fa93669f5e70cfbeefda32b98f8fd3d33b38028c582d700f93a?context=explore) | 9.74 MB | Custom Dockerfile: [`nginx/Dockerfile`](/nginx/Dockerfile) |
@@ -210,7 +210,13 @@ You can use the ckan [extension](https://docs.ckan.org/en/latest/extensions/tuto
docker compose -f docker compose.dev.yml exec ckan-dev /bin/sh -c "ckan generate extension --output-dir /srv/app/src_extensions"
-
+|CONTAINER ID |IMAGE |COMMAND|CREATED|STATUS|PORTS|NAMES|
+|------------|----------------------------------|--------------------|-------|-------|------|-----|
+|0217537f717e|ckan-docker-nginx |/docker-entrypoint.…|6 minutes ago |Up 4 minutes|81/tcp,0.0.0.0:81->80/tcp, 0.0.0.0:8443->443/tcp|nginx |
+|7b06ab2e060a|ghcr.io/mjanez/ckan-spatial:master|/srv/app/start_ckan…|6 minutes ago |Up 5 minutes (healthy)|0.0.0.0:5000->5000/tcp|ckan | |
+|1b8d9789c29a|redis:6 |docker-entrypoint.s…|6 minutes ago |Up 5 minutes (healthy)|6379/tcp |redis | |
+|7f162741254d|ckan/ckan-solr:2.9-solr8-spatial |docker-entrypoint.s…|6 minutes ago |Up 5 minutes (healthy)|8983/tcp |solr | |
+|2cdd25cea0de|ckan-docker-db |docker-entrypoint.s…|6 minutes ago |Up 5 minutes (healthy)|5432/tcp |db | |
The new extension files and directories are created in the `/srv/app/src_extensions/` folder in the running container. They will also exist in the local src/ directory as local `/src` directory is mounted as `/srv/app/src_extensions/` on the ckan container. You might need to change the owner of its folder to have the appropiate permissions.
@@ -374,4 +380,4 @@ Available components:
[^2]: Development environment.
[^3]: Production environment.
[^4]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
-[^5]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
+[^5]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
\ No newline at end of file
diff --git a/postgresql/Dockerfile b/postgresql/Dockerfile
index e912383..42ea06f 100755
--- a/postgresql/Dockerfile
+++ b/postgresql/Dockerfile
@@ -1,4 +1,4 @@
-FROM postgres:12-alpine
+FROM postgres:15-alpine
# Allow connections; we don't map out any ports so only linked docker containers can connect
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
From 8bd0b13b999d00e0bb50260944c58f7b87552a9a Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 23 Mar 2023 13:51:30 +0100
Subject: [PATCH 24/34] Update to ckanext-harvest 1.5.1
Fix url routing https://github.com/ckan/ckanext-harvest/pull/523
---
README.md | 2 +-
ckan/Dockerfile | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index bd4ae58..62ff06a 100644
--- a/README.md
+++ b/README.md
@@ -56,7 +56,7 @@ Information about extensions installed in the `main` image. More info described
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ✔️ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
-| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
+| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.1 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-dcat](https://github.com/mjanez/ckanext-dcat) | 1.2.0 | Completed | ✔️ | ✔️ | Stable installation, include DCAT-AP 2.1 profile compatible with GeoDCAT-AP. |
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 49a621b..1dff8d6 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -120,9 +120,9 @@ RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@0.12.2#egg=
pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
pip3 install -U requests[security]
-### Harvester - v1.5.0 ###
+### Harvester - v1.5.1 ###
RUN echo "ckan/ckanext-harvest" && \
- pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@v1.5.0#egg=ckanext-harvest' && \
+ pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@v1.5.1#egg=ckanext-harvest' && \
pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt
### Geoview - v0.0.20 ###
From 64b07daf630de7febdb0899e3521114a63f23ecc Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 23 Mar 2023 13:54:14 +0100
Subject: [PATCH 25/34] Update dev mode
---
ckan/Dockerfile.dev | 52 ++-------------
ckan/Dockerfile.spatial-vanilla | 112 ++++++++++++++++++++++++++++++++
2 files changed, 116 insertions(+), 48 deletions(-)
create mode 100644 ckan/Dockerfile.spatial-vanilla
diff --git a/ckan/Dockerfile.dev b/ckan/Dockerfile.dev
index 1e7b26e..0f72975 100644
--- a/ckan/Dockerfile.dev
+++ b/ckan/Dockerfile.dev
@@ -1,5 +1,5 @@
# syntax = edrevo/dockerfile-plus
-INCLUDE+ Dockerfile
+INCLUDE+ Dockerfile.spatial-vanilla
# Set up environment variables
ENV APP_DIR=/srv/app
@@ -25,54 +25,10 @@ RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
#
# For instance:
#
-### XLoader ###
-#RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@master#egg=ckanext-xloader' && \
-# pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
-# pip3 install -U requests[security]
-
-
###TODO: XLoader ###
-RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@master#egg=ckanext-xloader' && \
- pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
- pip3 install -U requests[security]
-
-### Harvester ###
-RUN echo "ckan/ckanext-harvester" && \
- pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest' && \
- pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt
-# will also require gather_consumer and fetch_consumer processes running (please see https://github.com/ckan/ckanext-harvest)
-
-### Resource dictionary-Fix version ###
-RUN echo "mjanez/ckanext-resourcedictionary" && \
- pip3 install -e git+https://github.com/mjanez/ckanext-resourcedictionary.git#egg=ckanext-resourcedictionary && \
- pip3 install -r ${APP_DIR}/src/ckanext-resourcedictionary/requirements.txt
-
-### Geoview ###
-RUN echo "ckan/ckanext-geoview" && \
- pip3 install -e git+https://github.com/ckan/ckanext-geoview.git#egg=ckanext-geoview
-
-### Spatial ### Check solr geoindexing: https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html // https://github.com/ckan/ckanext-spatial/issues/195
-#TODO: 2.10 support reqs: https://github.com/ckan/ckanext-spatial/pull/308
-RUN echo "ckan/ckanext-spatial" && \
- pip3 install -e git+https://github.com/ckan/ckanext-spatial.git#egg=ckanext-spatial && \
- pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-spatial/bae9290395b252ee8e40056256fa694569d1d78b/requirements.txt
-
-### Scheming-GeoDCAT-AP extended version ###
-RUN echo "mjanez/ckanext-scheming (GeoDCAT-AP extended version)" && \
- pip3 install -e git+https://github.com/mjanez/ckanext-scheming.git#egg=ckanext-scheming
-
-### Pages ###
-RUN echo "ckan/ckanext-pages" && \
- pip3 install -e git+https://github.com/ckan/ckanext-pages.git#egg=ckanext-pages
-
-### PDFView ###
-RUN echo "ckan/ckanext-pdfview" && \
- pip3 install -e git+https://github.com/ckan/ckanext-pdfview.git#egg=ckanext-pdfview
-
-### DCAT-GeoDCAT-AP extended version ###
-RUN echo "mjanez/ckanext-dcat (GeoDCAT-AP extended version)" && \
- pip3 install -e git+https://github.com/mjanez/ckanext-dcat.git#egg=ckanext-dcat && \
- pip3 install -r https://raw.githubusercontent.com/mjanez/ckanext-dcat/master/requirements.txt
+# RUN pip3 install -e 'git+https://github.com/ckan/ckanext-xloader.git@master#egg=ckanext-xloader' && \
+# pip3 install -r ${APP_DIR}/src/ckanext-xloader/requirements.txt && \
+# pip3 install -U requests[security]
###TODO: ckanext-visualize
#https://github.com/keitaroinc/ckanext-visualize
diff --git a/ckan/Dockerfile.spatial-vanilla b/ckan/Dockerfile.spatial-vanilla
new file mode 100644
index 0000000..2e4acb5
--- /dev/null
+++ b/ckan/Dockerfile.spatial-vanilla
@@ -0,0 +1,112 @@
+FROM alpine:3.13
+
+# Internals, you probably don't need to change these
+ENV APP_DIR=/srv/app
+ENV SRC_DIR=/srv/app/src
+ENV CKAN_INI=${APP_DIR}/ckan.ini
+ENV PIP_SRC=${SRC_DIR}
+ENV CKAN_STORAGE_PATH=/var/lib/ckan
+ENV GIT_URL=https://github.com/ckan/ckan.git
+ENV TZ=UTC
+RUN echo ${TZ} > /etc/timezone
+# CKAN version to build
+ENV GIT_BRANCH=ckan-2.9.8
+# Customize these on the .env file if needed
+ENV CKAN_SITE_URL=http://localhost:5000
+ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
+
+# UWSGI options
+ENV UWSGI_HARAKIRI=50
+
+WORKDIR ${APP_DIR}
+
+# Install necessary packages to run CKAN
+RUN apk add --no-cache tzdata \
+ git \
+ gettext \
+ postgresql-client \
+ python3 \
+ libxml2 \
+ libxslt \
+ musl-dev \
+ uwsgi \
+ uwsgi-http \
+ uwsgi-corerouter \
+ uwsgi-python \
+ py3-gevent \
+ uwsgi-gevent \
+ libmagic \
+ curl \
+ patch \
+ sudo && \
+ # Packages to build CKAN requirements and plugins
+ apk add --no-cache --virtual .build-deps \
+ postgresql-dev \
+ gcc \
+ make \
+ g++ \
+ autoconf \
+ automake \
+ libtool \
+ python3-dev \
+ libxml2-dev \
+ libxslt-dev \
+ linux-headers \
+ openssl-dev \
+ libffi-dev \
+ cargo \
+ # Set up PROJ
+ geos-dev \
+ proj \
+ proj-util \
+ proj-dev && \
+ # Create SRC_DIR
+ mkdir -p ${SRC_DIR} && \
+ # Install pip, supervisord and uwsgi
+ curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
+ python3 ${SRC_DIR}/get-pip.py && \
+ pip3 install supervisor && \
+ mkdir /etc/supervisord.d && \
+ #pip wheel --wheel-dir=/wheels uwsgi gevent && \
+ rm -rf ${SRC_DIR}/get-pip.py
+
+COPY setup/supervisord.conf /etc
+
+# Install CKAN
+RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
+ cd ${SRC_DIR}/ckan && \
+ cp who.ini ${APP_DIR} && \
+ pip3 install -r requirement-setuptools.txt && \
+ pip3 install --no-binary markdown -r requirements.txt && \
+ # Install CKAN envvars to support loading config from environment variables
+ pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
+ # Create and update CKAN config
+ ckan generate config ${CKAN_INI} && \
+ ckan config-tool ${CKAN_INI} "beaker.session.secret = " && \
+ ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
+
+# Create a local user and group to run the app
+RUN addgroup -g 92 -S ckan && \
+ adduser -u 92 -h /srv/app -s /bin/bash -D -G ckan ckan
+
+# Create local storage folder
+RUN mkdir -p ${CKAN_STORAGE_PATH} && \
+ chown -R ckan:ckan ${CKAN_STORAGE_PATH}
+
+COPY setup/prerun.py ${APP_DIR}
+COPY setup/start_ckan.sh ${APP_DIR}
+ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${APP_DIR}
+
+# Create entrypoint directory for children image scripts
+ONBUILD RUN mkdir /docker-entrypoint.d
+
+RUN chown ckan -R /srv/app
+
+RUN addgroup sudo && \
+ adduser ckan sudo
+
+EXPOSE 5000
+
+HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
+
+CMD ["/srv/app/start_ckan.sh"]
\ No newline at end of file
From 53d2d65cb10d22af216a6e132aa552090dc08d21 Mon Sep 17 00:00:00 2001
From: mnjnz <96422458+mjanez@users.noreply.github.com>
Date: Mon, 27 Mar 2023 12:26:51 +0200
Subject: [PATCH 26/34] Add custom images of base Dockerfiles
Now use ghcr.io/mjanez/ckan-base-spatial:master
---
README.md | 19 +++---
ckan/Dockerfile | 110 +------------------------------
ckan/Dockerfile.dev | 9 +--
ckan/Dockerfile.spatial-vanilla | 112 --------------------------------
4 files changed, 15 insertions(+), 235 deletions(-)
delete mode 100644 ckan/Dockerfile.spatial-vanilla
diff --git a/README.md b/README.md
index 62ff06a..c9c3e61 100644
--- a/README.md
+++ b/README.md
@@ -24,11 +24,11 @@ Contains Docker images for the different components of CKAN Cloud and a Docker c
>This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: [CKAN documentation: Installation](https://docs.ckan.org/en/latest/maintaining/installing/index.html).
Available components:
-* CKAN custom image based on the official CKAN repo [ckan-docker-base](https://github.com/ckan/ckan-docker-base)
+* CKAN custom image based on the official CKAN repo [ckan-docker-spatial](https://github.com/mjanez/ckan-docker-spatial)[^1]
The non-CKAN images are as follows:
* PostgreSQL: Official PostgreSQL image. Database files are stored in a named volume.
-* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). The index data is stored in a named volume and has a spatial schema. [^1]
+* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). The index data is stored in a named volume and has a spatial schema. [^2]
* Redis: standard Redis image
* NGINX: latest stable nginx image that includes SSL and Non-SSL endpoints
@@ -50,7 +50,7 @@ Information about extensions installed in the `main` image. More info described
> Switch branches to see the `roadmap` for other projects: [ckan-docker/branches](https://github.com/mjanez/ckan-docker/branches)
-| **Element** | **Description** | **version** | **Status** | **DEV**[^2] | **PRO**[^3] | **Remarks** |
+| **Element** | **Description** | **version** | **Status** | **DEV**[^3] | **PRO**[^4] | **Remarks** |
|-------------|-----------------------------------------------------------------------------------------|-------------|------------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.8 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
@@ -60,7 +60,7 @@ Information about extensions installed in the `main` image. More info described
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-dcat](https://github.com/mjanez/ckanext-dcat) | 1.2.0 | Completed | ✔️ | ✔️ | Stable installation, include DCAT-AP 2.1 profile compatible with GeoDCAT-AP. |
-| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^4] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
+| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^5] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
| Extension | [ckanext-resourcedictionary](https://github.com/OpenDataGIS/ckanext-resourcedictionary) | main | Completed | ✔️ | ✔️ | Stable installation. This extension extends the default CKAN Data Dictionary functionality by adding possibility to create data dictionary before actual data is uploaded to datastore. |
| Extension | [ckanext-pages](https://github.com/ckan/ckanext-pages) | 0.5.1 | Completed | ✔️ | ✔️ | Stable installation. This extension gives you an easy way to add simple pages to CKAN. |
| Extension | [ckanext-pdfview](https://github.com/ckan/ckanext-pdfview) | 0.0.8 | Completed | ✔️ | ✔️ | Stable installation. This extension provides a view plugin for PDF files using an html object tag. |
@@ -370,8 +370,9 @@ Available components:
* **ckan2pycsw**: Software to achieve interoperability with the open data portals based on CKAN. To do this, ckan2pycsw reads data from an instance using the CKAN API, generates ISO-19115/ISO-19139 metadata using [pygeometa](https://geopython.github.io/pygeometa/), or a custom schema that is based on a customized CKAN schema, and populates a [pycsw](https://pycsw.org/) instance that exposes the metadata using CSW and OAI-PMH.
-[^1]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)
-[^2]: Development environment.
-[^3]: Production environment.
-[^4]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
-[^5]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
+[^1]: Official CKAN repo: https://github.com/ckan/ckan-docker-base
+[^2]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)
+[^3]: Development environment.
+[^4]: Production environment.
+[^5]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
+[^6]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
diff --git a/ckan/Dockerfile b/ckan/Dockerfile
index 1dff8d6..1b3552e 100644
--- a/ckan/Dockerfile
+++ b/ckan/Dockerfile
@@ -1,117 +1,13 @@
-FROM alpine:3.13
+FROM ghcr.io/mjanez/ckan-base-spatial:master
-# Internals, you probably don't need to change these
+# Set up environment variables
ENV APP_DIR=/srv/app
-ENV SRC_DIR=/srv/app/src
-ENV CKAN_INI=${APP_DIR}/ckan.ini
-ENV PIP_SRC=${SRC_DIR}
-ENV CKAN_STORAGE_PATH=/var/lib/ckan
-ENV GIT_URL=https://github.com/ckan/ckan.git
ENV TZ=UTC
RUN echo ${TZ} > /etc/timezone
-# CKAN version to build
-ENV GIT_BRANCH=ckan-2.9.8
-# Customize these on the .env file if needed
-ENV CKAN_SITE_URL=http://localhost:5000
-ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
-
-# UWSGI options
-ENV UWSGI_HARAKIRI=50
-
-WORKDIR ${APP_DIR}
-
-# Install necessary packages to run CKAN
-RUN apk add --no-cache tzdata \
- git \
- gettext \
- postgresql-client \
- python3 \
- libxml2 \
- libxslt \
- musl-dev \
- uwsgi \
- uwsgi-http \
- uwsgi-corerouter \
- uwsgi-python \
- py3-gevent \
- uwsgi-gevent \
- libmagic \
- curl \
- patch \
- sudo && \
- # Packages to build CKAN requirements and plugins
- apk add --no-cache --virtual .build-deps \
- postgresql-dev \
- gcc \
- make \
- g++ \
- autoconf \
- automake \
- libtool \
- python3-dev \
- libxml2-dev \
- libxslt-dev \
- linux-headers \
- openssl-dev \
- libffi-dev \
- cargo \
- # Set up PROJ
- geos-dev \
- proj \
- proj-util \
- proj-dev && \
- # Create SRC_DIR
- mkdir -p ${SRC_DIR} && \
- # Install pip, supervisord and uwsgi
- curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
- python3 ${SRC_DIR}/get-pip.py && \
- pip3 install supervisor && \
- mkdir /etc/supervisord.d && \
- #pip wheel --wheel-dir=/wheels uwsgi gevent && \
- rm -rf ${SRC_DIR}/get-pip.py
-
-COPY setup/supervisord.conf /etc
-
-# Install CKAN
-RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
- cd ${SRC_DIR}/ckan && \
- cp who.ini ${APP_DIR} && \
- pip3 install -r requirement-setuptools.txt && \
- pip3 install --no-binary markdown -r requirements.txt && \
- # Install CKAN envvars to support loading config from environment variables
- pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
- # Create and update CKAN config
- ckan generate config ${CKAN_INI} && \
- ckan config-tool ${CKAN_INI} "beaker.session.secret = " && \
- ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
-
-# Create a local user and group to run the app
-RUN addgroup -g 92 -S ckan && \
- adduser -u 92 -h /srv/app -s /bin/bash -D -G ckan ckan
-
-# Create local storage folder
-RUN mkdir -p ${CKAN_STORAGE_PATH} && \
- chown -R ckan:ckan ${CKAN_STORAGE_PATH}
-
-COPY setup/prerun.py ${APP_DIR}
-COPY setup/start_ckan.sh ${APP_DIR}
-ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${APP_DIR}
-
-# Create entrypoint directory for children image scripts
-ONBUILD RUN mkdir /docker-entrypoint.d
-
-RUN chown ckan -R /srv/app
-
-RUN addgroup sudo && \
- adduser ckan sudo
-
-EXPOSE 5000
-
-HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
# Make sure both files are not exactly the same
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
- cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
+ cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
fi ;
# Extensions
diff --git a/ckan/Dockerfile.dev b/ckan/Dockerfile.dev
index 0f72975..6f2d24b 100644
--- a/ckan/Dockerfile.dev
+++ b/ckan/Dockerfile.dev
@@ -1,18 +1,13 @@
-# syntax = edrevo/dockerfile-plus
-INCLUDE+ Dockerfile.spatial-vanilla
+FROM ghcr.io/mjanez/ckan-base-spatial:master
# Set up environment variables
ENV APP_DIR=/srv/app
ENV TZ=UTC
RUN echo ${TZ} > /etc/timezone
-# Set up PROJ
-# PROJ4 for ckanext-spatial: https://github.com/pyproj4/pyproj/issues/582 // https://github.com/nickumia-reisys/ckanext-geodatagov/commit/ed61ffdd118e21e7a1db482c75738c3f261aa180
-RUN apk add geos-dev proj proj-util proj-dev
-
# Make sure both files are not exactly the same
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
- cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
+ cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
fi ;
# Install any extensions needed by your CKAN instance
diff --git a/ckan/Dockerfile.spatial-vanilla b/ckan/Dockerfile.spatial-vanilla
deleted file mode 100644
index 2e4acb5..0000000
--- a/ckan/Dockerfile.spatial-vanilla
+++ /dev/null
@@ -1,112 +0,0 @@
-FROM alpine:3.13
-
-# Internals, you probably don't need to change these
-ENV APP_DIR=/srv/app
-ENV SRC_DIR=/srv/app/src
-ENV CKAN_INI=${APP_DIR}/ckan.ini
-ENV PIP_SRC=${SRC_DIR}
-ENV CKAN_STORAGE_PATH=/var/lib/ckan
-ENV GIT_URL=https://github.com/ckan/ckan.git
-ENV TZ=UTC
-RUN echo ${TZ} > /etc/timezone
-# CKAN version to build
-ENV GIT_BRANCH=ckan-2.9.8
-# Customize these on the .env file if needed
-ENV CKAN_SITE_URL=http://localhost:5000
-ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
-
-# UWSGI options
-ENV UWSGI_HARAKIRI=50
-
-WORKDIR ${APP_DIR}
-
-# Install necessary packages to run CKAN
-RUN apk add --no-cache tzdata \
- git \
- gettext \
- postgresql-client \
- python3 \
- libxml2 \
- libxslt \
- musl-dev \
- uwsgi \
- uwsgi-http \
- uwsgi-corerouter \
- uwsgi-python \
- py3-gevent \
- uwsgi-gevent \
- libmagic \
- curl \
- patch \
- sudo && \
- # Packages to build CKAN requirements and plugins
- apk add --no-cache --virtual .build-deps \
- postgresql-dev \
- gcc \
- make \
- g++ \
- autoconf \
- automake \
- libtool \
- python3-dev \
- libxml2-dev \
- libxslt-dev \
- linux-headers \
- openssl-dev \
- libffi-dev \
- cargo \
- # Set up PROJ
- geos-dev \
- proj \
- proj-util \
- proj-dev && \
- # Create SRC_DIR
- mkdir -p ${SRC_DIR} && \
- # Install pip, supervisord and uwsgi
- curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
- python3 ${SRC_DIR}/get-pip.py && \
- pip3 install supervisor && \
- mkdir /etc/supervisord.d && \
- #pip wheel --wheel-dir=/wheels uwsgi gevent && \
- rm -rf ${SRC_DIR}/get-pip.py
-
-COPY setup/supervisord.conf /etc
-
-# Install CKAN
-RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
- cd ${SRC_DIR}/ckan && \
- cp who.ini ${APP_DIR} && \
- pip3 install -r requirement-setuptools.txt && \
- pip3 install --no-binary markdown -r requirements.txt && \
- # Install CKAN envvars to support loading config from environment variables
- pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
- # Create and update CKAN config
- ckan generate config ${CKAN_INI} && \
- ckan config-tool ${CKAN_INI} "beaker.session.secret = " && \
- ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
-
-# Create a local user and group to run the app
-RUN addgroup -g 92 -S ckan && \
- adduser -u 92 -h /srv/app -s /bin/bash -D -G ckan ckan
-
-# Create local storage folder
-RUN mkdir -p ${CKAN_STORAGE_PATH} && \
- chown -R ckan:ckan ${CKAN_STORAGE_PATH}
-
-COPY setup/prerun.py ${APP_DIR}
-COPY setup/start_ckan.sh ${APP_DIR}
-ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${APP_DIR}
-
-# Create entrypoint directory for children image scripts
-ONBUILD RUN mkdir /docker-entrypoint.d
-
-RUN chown ckan -R /srv/app
-
-RUN addgroup sudo && \
- adduser ckan sudo
-
-EXPOSE 5000
-
-HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
-
-CMD ["/srv/app/start_ckan.sh"]
\ No newline at end of file
From f50edb467a305d94a8920262a55874b49752a98d Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Mon, 27 Mar 2023 17:53:35 +0200
Subject: [PATCH 27/34] Fix setup_xloader
Assign default qeue
---
ckan/docker-entrypoint.d/setup_xloader.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/ckan/docker-entrypoint.d/setup_xloader.sh b/ckan/docker-entrypoint.d/setup_xloader.sh
index 5f738c1..556de7a 100644
--- a/ckan/docker-entrypoint.d/setup_xloader.sh
+++ b/ckan/docker-entrypoint.d/setup_xloader.sh
@@ -11,4 +11,5 @@ echo "Set up ckanext.xloader.api_token in the CKAN config file"
ckan config-tool $CKAN_INI "ckanext.xloader.api_token = $(ckan -c $CKAN_INI user token add ckan_admin xloader | tail -n 1 | tr -d '\t')"
# Setup worker
-ckan config-tool $CKAN_INI jobs worker
\ No newline at end of file
+echo "Set up CKAN jobs worker"
+ckan config-tool $CKAN_INI jobs worker default
\ No newline at end of file
From 7db161188eaa8d210b7b818c5cbdde92a8fcd53c Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Mon, 27 Mar 2023 17:54:49 +0200
Subject: [PATCH 28/34] Remove datapusher - Datapusher out of date with
ckanext-xloader - Fix deprecated views
---
.env | 9 +--------
samples/.env.example | 9 +--------
2 files changed, 2 insertions(+), 16 deletions(-)
diff --git a/.env b/.env
index de58312..5437e53 100644
--- a/.env
+++ b/.env
@@ -58,7 +58,7 @@ CKAN__RESOURCE_PROXY__MAX_FILE_SIZE=50048576
CKAN__RESOURCE_PROXY__CHUNK_SIZE=4096
## Default timeout for fetching proxied items
CKAN__RESOURCE_PROXY__TIMEOUT=10
-CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view recline_grid_view recline_graph_view wmts_view webpage_view geojson_view geo_view shp_view pdf_view"
+CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view wmts_view geojson_view geo_view shp_view pdf_view"
# Solr
SOLR_IMAGE_VERSION=2.9-solr8-spatial
@@ -70,13 +70,6 @@ REDIS_VERSION=6
CKAN_REDIS_URL=redis://redis:6379/1
TEST_CKAN_REDIS_URL=redis://redis:6379/1
-# Datapusher - Deprecated by ckanext-xloader
-DATAPUSHER_VERSION=0.0.19
-CKAN_DATAPUSHER_URL=http://datapusher:8800
-CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
-DATAPUSHER_REWRITE_RESOURCES=True
-DATAPUSHER_REWRITE_URL=http://ckan:5000
-
# NGINX
NGINX_PORT=80
NGINX_SSLPORT=443
diff --git a/samples/.env.example b/samples/.env.example
index de58312..5437e53 100644
--- a/samples/.env.example
+++ b/samples/.env.example
@@ -58,7 +58,7 @@ CKAN__RESOURCE_PROXY__MAX_FILE_SIZE=50048576
CKAN__RESOURCE_PROXY__CHUNK_SIZE=4096
## Default timeout for fetching proxied items
CKAN__RESOURCE_PROXY__TIMEOUT=10
-CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view recline_grid_view recline_graph_view wmts_view webpage_view geojson_view geo_view shp_view pdf_view"
+CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view wmts_view geojson_view geo_view shp_view pdf_view"
# Solr
SOLR_IMAGE_VERSION=2.9-solr8-spatial
@@ -70,13 +70,6 @@ REDIS_VERSION=6
CKAN_REDIS_URL=redis://redis:6379/1
TEST_CKAN_REDIS_URL=redis://redis:6379/1
-# Datapusher - Deprecated by ckanext-xloader
-DATAPUSHER_VERSION=0.0.19
-CKAN_DATAPUSHER_URL=http://datapusher:8800
-CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
-DATAPUSHER_REWRITE_RESOURCES=True
-DATAPUSHER_REWRITE_URL=http://ckan:5000
-
# NGINX
NGINX_PORT=80
NGINX_SSLPORT=443
From 424035617f2f3a541f21e89b964eac2a11804a9e Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 30 Mar 2023 05:28:20 +0000
Subject: [PATCH 29/34] Fix setup worker
---
ckan/docker-entrypoint.d/setup_xloader.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ckan/docker-entrypoint.d/setup_xloader.sh b/ckan/docker-entrypoint.d/setup_xloader.sh
index 5f738c1..9773aac 100644
--- a/ckan/docker-entrypoint.d/setup_xloader.sh
+++ b/ckan/docker-entrypoint.d/setup_xloader.sh
@@ -11,4 +11,4 @@ echo "Set up ckanext.xloader.api_token in the CKAN config file"
ckan config-tool $CKAN_INI "ckanext.xloader.api_token = $(ckan -c $CKAN_INI user token add ckan_admin xloader | tail -n 1 | tr -d '\t')"
# Setup worker
-ckan config-tool $CKAN_INI jobs worker
\ No newline at end of file
+ckan config-tool $CKAN_INI jobs worker default
\ No newline at end of file
From d30bbad8e96bc7f0d326dcc454eafa71d992be56 Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 30 Mar 2023 05:28:30 +0000
Subject: [PATCH 30/34] Fix views
---
.env | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.env b/.env
index de58312..dcb1e46 100644
--- a/.env
+++ b/.env
@@ -58,7 +58,7 @@ CKAN__RESOURCE_PROXY__MAX_FILE_SIZE=50048576
CKAN__RESOURCE_PROXY__CHUNK_SIZE=4096
## Default timeout for fetching proxied items
CKAN__RESOURCE_PROXY__TIMEOUT=10
-CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view recline_grid_view recline_graph_view wmts_view webpage_view geojson_view geo_view shp_view pdf_view"
+CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view recline_view wmts_view geojson_view geo_view shp_view pdf_view"
# Solr
SOLR_IMAGE_VERSION=2.9-solr8-spatial
From a62b09923b1b114955a6943d534219877a2c50dd Mon Sep 17 00:00:00 2001
From: mjanez <96422458+mjanez@users.noreply.github.com>
Date: Thu, 30 Mar 2023 05:29:45 +0000
Subject: [PATCH 31/34] Update README.md
- Add info about postgres:15-alpine
- Update footnotes
- Update plugins
- Update containers info
---
README.md | 21 +++++++++++----------
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/README.md b/README.md
index 83e8cb5..d0bde5f 100644
--- a/README.md
+++ b/README.md
@@ -24,11 +24,11 @@ Contains Docker images for the different components of CKAN Cloud and a Docker c
>This is a **custom installation of Docker Compose** with specific extensions for spatial data and [GeoDCAT-AP](https://github.com/SEMICeu/GeoDCAT-AP)/[INSPIRE](https://github.com/INSPIRE-MIF/technical-guidelines) metadata [profiles](https://en.wikipedia.org/wiki/Geospatial_metadata). For official installations, please have a look: [CKAN documentation: Installation](https://docs.ckan.org/en/latest/maintaining/installing/index.html).
Available components:
-* CKAN custom image based on the official CKAN repo [ckan-docker-base](https://github.com/ckan/ckan-docker-base)
+* CKAN custom image based on the official CKAN repo [ckan-docker-spatial](https://github.com/mjanez/ckan-docker-spatial)[^1]
The non-CKAN images are as follows:
* PostgreSQL: Official PostgreSQL image. Database files are stored in a named volume.
-* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). The index data is stored in a named volume and has a spatial schema. [^1]
+* Solr: CKAN's [pre-configured Solr image](https://github.com/ckan/ckan-solr). The index data is stored in a named volume and has a spatial schema. [^2]
* Redis: standard Redis image
* NGINX: latest stable nginx image that includes SSL and Non-SSL endpoints
@@ -50,17 +50,17 @@ Information about extensions installed in the `main` image. More info described
> Switch branches to see the `roadmap` for other projects: [ckan-docker/branches](https://github.com/mjanez/ckan-docker/branches)
-| **Element** | **Description** | **version** | **Status** | **DEV**[^2] | **PRO**[^3] | **Remarks** |
+| **Element** | **Description** | **version** | **Status** | **DEV**[^3] | **PRO**[^4] | **Remarks** |
|-------------|-----------------------------------------------------------------------------------------|-------------|------------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Core | [CKAN](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation for version 2.9.8 (Production & Dev images) via Docker Compose based on [official images](https://github.com/ckan/ckan-docker-base)). Initial configuration, basic customisation and operation guide. |
| Core + | [Datastore](https://github.com/mjanez/ckan-docker) | 2.9.8 | Completed | ✔️ | ✔️ | Stable installation (Production & Dev images) via Docker Compose. |
| Core + | [~~Datapusher~~](https://github.com/mjanez/ckan-docker) | 0.0.19 | Deprecated | ❌ | ❌ | Updated to [xloader](https://github.com/ckan/ckanext-xloader), an express Loader - quickly load data into DataStore. |
| Extension | [ckanext-xloader](https://github.com/ckan/ckanext-xloader) | 0.12.2 | Completed | ✔️ | ✔️ | Stable installation, a replacement for DataPusher because it offers ten times the speed and more robustness |
-| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
+| Extension | [ckanext-harvest](https://github.com/ckan/ckanext-harvest) | 1.5.1 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-geoview](https://github.com/ckan/ckanext-geoview) | 0.0.20 | Completed | ✔️ | ✔️ | Stable installation. |
| Extension | [ckanext-spatial](https://github.com/ckan/ckanext-spatial) | 2.0.0 | Completed | ✔️ | ✔️ | Stable installation, necessary for the implementation of the Collector ([ogc_ckan](#recollector-ckan)) |
| Extension | [ckanext-dcat](https://github.com/mjanez/ckanext-dcat) | 1.2.0 | Completed | ✔️ | ✔️ | Stable installation, include DCAT-AP 2.1 profile compatible with GeoDCAT-AP. |
-| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^4] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
+| Extension | [ckanext-scheming](https://github.com/mjanez/ckanext-scheming) | 3.0.0 | WIP | ✔️ | ✔️ | Stable installation. Customised ckanext schema[^5] based on the [Spanish Metadata Core](https://datos.gob.es/es/doc-tags/nti-risp) with the aim of completing the minimum metadata elements included in the current datasets in accordance with [GeoDCAT-AP](https://semiceu.github.io/GeoDCAT-AP/releases/) and [INSPIRE](https://inspire.ec.europa.eu/about-inspire). |
| Extension | [ckanext-resourcedictionary](https://github.com/OpenDataGIS/ckanext-resourcedictionary) | main | Completed | ✔️ | ✔️ | Stable installation. This extension extends the default CKAN Data Dictionary functionality by adding possibility to create data dictionary before actual data is uploaded to datastore. |
| Extension | [ckanext-pages](https://github.com/ckan/ckanext-pages) | 0.5.1 | Completed | ✔️ | ✔️ | Stable installation. This extension gives you an easy way to add simple pages to CKAN. |
| Extension | [ckanext-pdfview](https://github.com/ckan/ckanext-pdfview) | 0.0.8 | Completed | ✔️ | ✔️ | Stable installation. This extension provides a view plugin for PDF files using an html object tag. |
@@ -376,8 +376,9 @@ Available components:
* **ckan2pycsw**: Software to achieve interoperability with the open data portals based on CKAN. To do this, ckan2pycsw reads data from an instance using the CKAN API, generates ISO-19115/ISO-19139 metadata using [pygeometa](https://geopython.github.io/pygeometa/), or a custom schema that is based on a customized CKAN schema, and populates a [pycsw](https://pycsw.org/) instance that exposes the metadata using CSW and OAI-PMH.
-[^1]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)
-[^2]: Development environment.
-[^3]: Production environment.
-[^4]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
-[^5]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
\ No newline at end of file
+[^1]: Official CKAN repo: https://github.com/ckan/ckan-docker-base
+[^2]: Contains fields needed for the [ckanext-spatial geo search](https://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html)
+[^3]: Development environment.
+[^4]: Production environment.
+[^5]: [ckan_geodcatap](https://github.com/mjanez/ckanext-scheming/blob/036b8c6503059e0d42b0eba180d5bd39205c64a3/ckanext/scheming/ckan_geodcatap.yaml), more info: https://github.com/mjanez/ckanext-scheming/pull/1
+[^6]: A fork of [COATNor/coat2pycsw](https://github.com/COATnor/coat2pycsw) that has been extended to meet the needs of harvesting GeoDCAT-AP metadata according to INSPIRE ISO19139.
\ No newline at end of file
From 3ffba6fe0dc1c941f84dfa0a48d9467a5f9e3e6c Mon Sep 17 00:00:00 2001
From: mnjnz <96422458+mjanez@users.noreply.github.com>
Date: Sun, 2 Apr 2023 01:30:43 +0200
Subject: [PATCH 32/34] Fix xloader setup
TODO: Setup worker background
---
ckan/docker-entrypoint.d/setup_xloader.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/ckan/docker-entrypoint.d/setup_xloader.sh b/ckan/docker-entrypoint.d/setup_xloader.sh
index 556de7a..8aa2569 100644
--- a/ckan/docker-entrypoint.d/setup_xloader.sh
+++ b/ckan/docker-entrypoint.d/setup_xloader.sh
@@ -10,6 +10,6 @@ ckan config-tool $CKAN_INI \
echo "Set up ckanext.xloader.api_token in the CKAN config file"
ckan config-tool $CKAN_INI "ckanext.xloader.api_token = $(ckan -c $CKAN_INI user token add ckan_admin xloader | tail -n 1 | tr -d '\t')"
-# Setup worker
-echo "Set up CKAN jobs worker"
-ckan config-tool $CKAN_INI jobs worker default
\ No newline at end of file
+#TODO: Setup worker background
+#echo "Set up CKAN jobs worker"
+#ckan -c $CKAN_INI jobs worker default
\ No newline at end of file
From 4702b86449d8b27de0b1006e96ce229fea8c809e Mon Sep 17 00:00:00 2001
From: mnjnz <96422458+mjanez@users.noreply.github.com>
Date: Sun, 2 Apr 2023 01:36:58 +0200
Subject: [PATCH 33/34] Fix .env.example
---
.env.example | 77 ----------------------------------------------------
1 file changed, 77 deletions(-)
delete mode 100644 .env.example
diff --git a/.env.example b/.env.example
deleted file mode 100644
index f31d1ee..0000000
--- a/.env.example
+++ /dev/null
@@ -1,77 +0,0 @@
-# Container names
-NGINX_CONTAINER_NAME=nginx
-REDIS_CONTAINER_NAME=redis
-POSTGRESQL_CONTAINER_NAME=db
-SOLR_CONTAINER_NAME=solr
-DATAPUSHER_CONTAINER_NAME=datapusher
-CKAN_CONTAINER_NAME=ckan
-WORKER_CONTAINER_NAME=ckan-worker
-
-# Host Ports
-CKAN_PORT_HOST=5000
-NGINX_PORT_HOST=81
-NGINX_SSLPORT_HOST=8443
-
-# CKAN databases
-POSTGRES_USER=ckan
-POSTGRES_PASSWORD=ckan
-DATASTORE_READONLY_USER=datastore_ro
-DATASTORE_READONLY_PASSWORD=datastore
-POSTGRES_HOST=db
-CKAN_SQLALCHEMY_URL=postgresql://ckan:ckan@db/ckan
-CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore
-CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore
-
-# Test database connections
-TEST_CKAN_SQLALCHEMY_URL=postgres://ckan:ckan@db/ckan_test
-TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
-TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test
-
-# CKAN core
-CKAN_VERSION=2.10.0
-CKAN_SITE_ID=default
-CKAN_SITE_URL=http://ckan:5000
-CKAN_PORT=5000
-CKAN_PORT_HOST=5000
-CKAN___BEAKER__SESSION__SECRET=CHANGE_ME
-# See https://docs.ckan.org/en/latest/maintaining/configuration.html#api-token-settings
-CKAN___API_TOKEN__JWT__ENCODE__SECRET=string:CHANGE_ME
-CKAN___API_TOKEN__JWT__DECODE__SECRET=string:CHANGE_ME
-CKAN_SYSADMIN_NAME=ckan_admin
-CKAN_SYSADMIN_PASSWORD=test1234
-CKAN_SYSADMIN_EMAIL=your_email@example.com
-CKAN_STORAGE_PATH=/var/lib/ckan
-CKAN_SMTP_SERVER=smtp.corporateict.domain:25
-CKAN_SMTP_STARTTLS=True
-CKAN_SMTP_USER=user
-CKAN_SMTP_PASSWORD=pass
-CKAN_SMTP_MAIL_FROM=ckan@localhost
-TZ=UTC
-
-# Solr
-SOLR_IMAGE_VERSION=2.9-solr8
-CKAN_SOLR_URL=http://solr:8983/solr/ckan
-TEST_CKAN_SOLR_URL=http://solr:8983/solr/ckan
-
-# Redis
-REDIS_VERSION=6
-CKAN_REDIS_URL=redis://redis:6379/1
-TEST_CKAN_REDIS_URL=redis://redis:6379/1
-
-# Datapusher
-DATAPUSHER_VERSION=0.0.19
-CKAN_DATAPUSHER_URL=http://datapusher:8800
-CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
-DATAPUSHER_REWRITE_RESOURCES=True
-DATAPUSHER_REWRITE_URL=http://ckan:5000
-
-# NGINX
-NGINX_PORT=80
-NGINX_SSLPORT=443
-
-# Extensions
-CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher"
-CKAN__HARVEST__MQ__TYPE=redis
-CKAN__HARVEST__MQ__HOSTNAME=redis
-CKAN__HARVEST__MQ__PORT=6379
-CKAN__HARVEST__MQ__REDIS_DB=1
From e0eb54f61f7fe0a62fdb471439d16a339e4aa915 Mon Sep 17 00:00:00 2001
From: mnjnz <96422458+mjanez@users.noreply.github.com>
Date: Sun, 2 Apr 2023 01:43:35 +0200
Subject: [PATCH 34/34] Remove solr build
---
.github/workflows/build.yml | 10 +---------
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 91123cf..6edb7d1 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -36,12 +36,4 @@ jobs:
context: ./postgresql
file: ./postgresql/Dockerfile
push: false
- tags: kowhai/ckan-docker-postgresql:test-build-only
-
- - name: Solr build
- uses: docker/build-push-action@v2
- with:
- context: ./solr
- file: ./solr/Dockerfile
- push: false
- tags: kowhai/ckan-docker-solr:test-build-only
+ tags: kowhai/ckan-docker-postgresql:test-build-only
\ No newline at end of file
-* [Overview](#overview)
-* [Installing Docker](#installing-docker)
-* [docker compose vs docker-compose](#docker-compose-vs-docker-compose)
-* [Install CKAN plus dependencies](#install-ckan-plus-dependencies)
-* [Development mode](#development-mode)
- * [Create an extension](#create-an-extension)
-* [CKAN images](#ckan-images)
- * [Extending the base images](#extending-the-base-images)
- * [Applying patches](#applying-patches)
-* [Debugging with pdb](#pdb)
-* [Datastore and Datapusher](#Datastore-and-datapusher)
-* [NGINX](#nginx)
-* [The ckanext-envvars extension](#envvars)
+