Updates to README, delete old directory
This commit is contained in:
Brett
parent
29df3f6d00
commit
ec9ec421f6
57
README.md
57
README.md
|
|
@ -3,8 +3,8 @@
|
||||||
|
|
||||||
* [Overview](#overview)
|
* [Overview](#overview)
|
||||||
* [docker compose vs docker-compose](#docker-compose-vs-docker-compose)
|
* [docker compose vs docker-compose](#docker-compose-vs-docker-compose)
|
||||||
* [Quick start](#quick-start)
|
* [Installing Docker](#installing-docker)
|
||||||
* [The ckanext-envvars extension](#envvars)
|
* [Install CKAN plus dependencies](#install-ckan-plus-dependencies)
|
||||||
* [Development mode](#development-mode)
|
* [Development mode](#development-mode)
|
||||||
* [Create an extension](#create-an-extension)
|
* [Create an extension](#create-an-extension)
|
||||||
* [CKAN images](#ckan-images)
|
* [CKAN images](#ckan-images)
|
||||||
|
|
@ -13,6 +13,7 @@
|
||||||
* [Debugging with pdb](#pdb)
|
* [Debugging with pdb](#pdb)
|
||||||
* [Datastore and Datapusher](#Datastore-and-datapusher)
|
* [Datastore and Datapusher](#Datastore-and-datapusher)
|
||||||
* [NGINX](#nginx)
|
* [NGINX](#nginx)
|
||||||
|
* [The ckanext-envvars extension](#envvars)
|
||||||
* [Known Issues](#known-issues)
|
* [Known Issues](#known-issues)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -38,7 +39,14 @@ All Docker Compose commands in this README will use the V2 version of Compose ie
|
||||||
used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for
|
used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for
|
||||||
more information.
|
more information.
|
||||||
|
|
||||||
## Quick start
|
## Installing Docker
|
||||||
|
|
||||||
|
Install Docker by following the following instructions: [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
|
||||||
|
|
||||||
|
To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output
|
||||||
|
versions for client and server.
|
||||||
|
|
||||||
|
## Install CKAN plus dependencies
|
||||||
|
|
||||||
Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs.
|
Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs.
|
||||||
|
|
||||||
|
|
@ -56,25 +64,8 @@ At the end of the container start sequence there should be 6 containers running
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## envvars
|
After this step, CKAN should be running at `CKAN_SITE_URL`.
|
||||||
|
|
||||||
The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images.
|
|
||||||
This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value.
|
|
||||||
|
|
||||||
For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way:
|
|
||||||
|
|
||||||
All uppercase
|
|
||||||
Replace periods ('.') with two underscores ('__')
|
|
||||||
Keys must begin with 'CKAN' or 'CKANEXT'
|
|
||||||
|
|
||||||
For example:
|
|
||||||
|
|
||||||
CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher"
|
|
||||||
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
|
|
||||||
|
|
||||||
These parameters can be added to the `.env` file
|
|
||||||
|
|
||||||
For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars)
|
|
||||||
|
|
||||||
## Development mode
|
## Development mode
|
||||||
|
|
||||||
|
|
@ -195,12 +186,32 @@ running the latest version of Datapusher.
|
||||||
|
|
||||||
## NGINX
|
## NGINX
|
||||||
|
|
||||||
* The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production.
|
The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production.
|
||||||
|
|
||||||
Creating the SSL cert and key files as follows:
|
Creating the SSL cert and key files as follows:
|
||||||
`openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=DE/ST=Berlin/L=Berlin/O=None/CN=localhost" -keyout ckan-local.key -out ckan-local.crt`
|
`openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=DE/ST=Berlin/L=Berlin/O=None/CN=localhost" -keyout ckan-local.key -out ckan-local.crt`
|
||||||
The `ckan-local.*` files will then need to be moved into the nginx/setup/ directory
|
The `ckan-local.*` files will then need to be moved into the nginx/setup/ directory
|
||||||
|
|
||||||
|
## envvars
|
||||||
|
|
||||||
|
The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images.
|
||||||
|
This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value.
|
||||||
|
|
||||||
|
For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way:
|
||||||
|
|
||||||
|
All uppercase
|
||||||
|
Replace periods ('.') with two underscores ('__')
|
||||||
|
Keys must begin with 'CKAN' or 'CKANEXT'
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
* `CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher"`
|
||||||
|
* `CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000`
|
||||||
|
|
||||||
|
These parameters can be added to the `.env` file
|
||||||
|
|
||||||
|
For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars)
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
* Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that.
|
Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that.
|
||||||
|
|
|
||||||
|
|
@ -1,66 +0,0 @@
|
||||||
### Please note that this re-purposed CKAN Docker repo is a WORK IN PROGRESS ###
|
|
||||||
### It should not be used to install CKAN via Docker until this page is updated ###
|
|
||||||
|
|
||||||
-------------------------------------
|
|
||||||
(From Adria)
|
|
||||||
Be limited in scope, and act as a base that users can extend to their own needs
|
|
||||||
Be opinionated, and provide one way to do things
|
|
||||||
Be automatically tested
|
|
||||||
-------------------------------------
|
|
||||||
|
|
||||||
should we have an nginx container included? maybe just instructions on how to include one
|
|
||||||
|
|
||||||
ARG should be used for sensitive variables so as to keep the values of these variables
|
|
||||||
out of the build and hence will not show with an "inspect image" command
|
|
||||||
|
|
||||||
Do we pre-build a CKAN image and use that (and extend) as the base image OR
|
|
||||||
just build it from scratch and while saving to storage what we need to
|
|
||||||
|
|
||||||
To use local storage for the ckan.ini file and the CKAN src code, do the following:
|
|
||||||
|
|
||||||
# docker-compose up -d --build
|
|
||||||
# mkdir local
|
|
||||||
# docker cp ckan:/srv/app/ckan.ini ./local/ckan.ini
|
|
||||||
# docker cp ckan:/srv/app/src ./local/src
|
|
||||||
Stop/Remove ckan container
|
|
||||||
Start ckan container as follows
|
|
||||||
Use a bind mount for the config file (ckan.ini)
|
|
||||||
# docker run -p 0.0.0.0:5000:5000 --net ckan-docker_default --hostname ckan --name ckan \
|
|
||||||
--env-file=./environ --mount type=bind,source=$(pwd)/local/ckan.ini,target=/srv/app/ckan.ini \
|
|
||||||
--mount type=bind,source=$(pwd)/local/src,target=/srv/app/src \
|
|
||||||
-d ckan-docker_ckan
|
|
||||||
|
|
||||||
Maybe include a script to replace docker-compose if required
|
|
||||||
|
|
||||||
Dev Mode (OKFN)
|
|
||||||
|
|
||||||
The differences between Docker dev abd Docker base is as the following:
|
|
||||||
|
|
||||||
docker-compose.dev.yml
|
|
||||||
solr: explicitly puts in ports (8983:8983)
|
|
||||||
db: Doesn't pass in environment and arg values
|
|
||||||
ckan: has extra volume bind mount (./src:/srv/app/src_extensions)
|
|
||||||
|
|
||||||
Dockerfile.dev
|
|
||||||
Takes the base image and
|
|
||||||
Adds a new directory (SRC_EXTENSIONS_DIR=/srv/app/src_extensions)
|
|
||||||
installs libffi-dev
|
|
||||||
installs dev-requirements.txt
|
|
||||||
runs different start script (start_ckan_development.sh) which installs any extension located in SRC_EXTENSIONS_DIR
|
|
||||||
runs a typical Dev install
|
|
||||||
pip install -r pip-requirements.txt
|
|
||||||
pip install -r requirements.txt
|
|
||||||
pip install -r dev-requirements.txt
|
|
||||||
python3 setup.py develop
|
|
||||||
ckan config-tool test.ini
|
|
||||||
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
|
|
||||||
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
|
|
||||||
ckan config-tool $SRC_DIR/ckan/test-core.ini \
|
|
||||||
"sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \
|
|
||||||
"ckan.datastore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \
|
|
||||||
"ckan.datastore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \
|
|
||||||
"solr_url = $TEST_CKAN_SOLR_URL" \
|
|
||||||
"ckan.redis.url = $TEST_CKAN_REDIS_URL"
|
|
||||||
|
|
||||||
All other steps are similar to the base Dockerfile
|
|
||||||
|
|
||||||
|
|
@ -1,71 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
#
|
|
||||||
# Create the network first
|
|
||||||
#
|
|
||||||
docker network create ckan-net
|
|
||||||
#
|
|
||||||
|
|
||||||
|
|
||||||
# Redis - Run only (image already built)
|
|
||||||
#
|
|
||||||
REDIS_CONTAINERNAME=redis
|
|
||||||
REDIS_HOSTNAME=redis
|
|
||||||
|
|
||||||
docker run -d -it --net ckan-net --name ${REDIS_CONTAINERNAME} --hostname ${REDIS_HOSTNAME} redis:latest
|
|
||||||
|
|
||||||
#
|
|
||||||
# Solr - Run/Build
|
|
||||||
#
|
|
||||||
SOLR_CONTAINERNAME=solr
|
|
||||||
SOLR_HOSTNAME=solr
|
|
||||||
SOLR_IMAGENAME=solr
|
|
||||||
|
|
||||||
cd solr
|
|
||||||
docker image build -t ${SOLR_IMAGENAME} .
|
|
||||||
docker run -d -it --net ckan-net --name ${SOLR_CONTAINERNAME} --hostname ${SOLR_HOSTNAME} \
|
|
||||||
-v solr_core:/opt/solr/server/solr ${SOLR_IMAGENAME}
|
|
||||||
cd -
|
|
||||||
|
|
||||||
#
|
|
||||||
# PostgreSQL (DB) - Run/Build
|
|
||||||
#
|
|
||||||
DB_CONTAINERNAME=db
|
|
||||||
DB_HOSTNAME=db
|
|
||||||
DB_IMAGENAME=db
|
|
||||||
|
|
||||||
cd postgresql
|
|
||||||
docker image build -t ${DB_IMAGENAME} .
|
|
||||||
docker run -d -it --net ckan-net --name ${DB_CONTAINERNAME} --hostname ${DB_HOSTNAME} --env-file=../environ \
|
|
||||||
-v pg_data:/var/lib/postgresql/data ${DB_IMAGENAME}
|
|
||||||
cd -
|
|
||||||
|
|
||||||
#
|
|
||||||
# CKAN - Run/Build
|
|
||||||
#
|
|
||||||
CKAN_CONTAINERNAME=ckan
|
|
||||||
CKAN_HOSTNAME=ckan
|
|
||||||
CKAN_IMAGENAME=ckan
|
|
||||||
|
|
||||||
cd ckan
|
|
||||||
docker image build -t ${CKAN_IMAGENAME} .
|
|
||||||
docker run -d -it --net ckan-net --name ${CKAN_CONTAINERNAME} --hostname ${CKAN_HOSTNAME} --env-file=../environ \
|
|
||||||
-v ckan_storage:/var/lib/ckan ${CKAN_IMAGENAME}
|
|
||||||
cd -
|
|
||||||
|
|
||||||
#
|
|
||||||
# Nginx - Run/Build
|
|
||||||
#
|
|
||||||
NGINX_CONTAINERNAME=nginx
|
|
||||||
NGINX_HOSTNAME=nginx
|
|
||||||
NGINX_IMAGENAME=nginx
|
|
||||||
|
|
||||||
cd nginx
|
|
||||||
docker image build -t ${NGINX_IMAGENAME} .
|
|
||||||
docker run -d -it --net ckan-net --name ${NGINX_CONTAINERNAME} --hostname ${NGINX_HOSTNAME} -p 0.0.0.0:80:80 ${NGINX_IMAGENAME}
|
|
||||||
cd -
|
|
||||||
|
|
||||||
#
|
|
||||||
# PostgreSQL client
|
|
||||||
#
|
|
||||||
docker run -d -it --net ckan-net --name postgres-client --hostname postgres-client -e POSTGRES_PASSWORD=ckan -p 5432:5432 postgres
|
|
||||||
|
|
@ -1,16 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Stop all containers
|
|
||||||
docker kill $(docker ps -q)
|
|
||||||
|
|
||||||
# Remove all stopped containers
|
|
||||||
docker rm -f $(docker ps -aq)
|
|
||||||
|
|
||||||
# Remove all docker images
|
|
||||||
docker rmi $(docker images -q)
|
|
||||||
|
|
||||||
# Remove all volumes
|
|
||||||
docker volume rm $(docker volume ls -q)
|
|
||||||
|
|
||||||
# Remove ckan-docker_default network
|
|
||||||
docker network rm ckan-docker_default
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
REDIS_CONTAINERNAME=redis
|
|
||||||
REDIS_HOSTNAME=redis
|
|
||||||
|
|
||||||
SOLR_CONTAINERNAME=solr
|
|
||||||
SOLR_HOSTNAME=solr
|
|
||||||
SOLR_IMAGENAME=solr
|
|
||||||
|
|
||||||
DB_CONTAINERNAME=db
|
|
||||||
DB_HOSTNAME=db
|
|
||||||
DB_IMAGENAME=db
|
|
||||||
POSTGRES_PASSWORD=ckan
|
|
||||||
DS_RO_PASS=datastore
|
|
||||||
PGDATA=/var/lib/postgresql/data/db
|
|
||||||
|
|
||||||
CKAN_CONTAINERNAME=ckan
|
|
||||||
CKAN_HOSTNAME=ckan
|
|
||||||
CKAN_IMAGENAME=ckan
|
|
||||||
|
|
||||||
CKAN_SQLALCHEMY_URL=postgresql://ckan:ckan@db/ckan
|
|
||||||
CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore
|
|
||||||
CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore
|
|
||||||
CKAN_SOLR_URL=http://solr:8983/solr/ckan
|
|
||||||
CKAN_REDIS_URL=redis://redis:6379/1
|
|
||||||
CKAN_SITE_URL=http://localhost
|
|
||||||
CKAN_MAX_UPLOAD_SIZE_MB=32
|
|
||||||
POSTGRES_PASSWORD=ckan
|
|
||||||
DS_RO_PASS=datastore
|
|
||||||
CKAN_PORT=5000
|
|
||||||
CKAN_DATAPUSHER_URL=http://datapusher:8800
|
|
||||||
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000
|
|
||||||
|
|
||||||
NGINX_CONTAINERNAME=nginx
|
|
||||||
NGINX_HOSTNAME=nginx
|
|
||||||
NGINX_IMAGENAME=nginx
|
|
||||||
|
|
@ -1,97 +0,0 @@
|
||||||
FROM alpine:3.7
|
|
||||||
|
|
||||||
# Internals, you probably don't need to change these
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_DIR=/srv/app/src
|
|
||||||
ENV CKAN_INI=${APP_DIR}/production.ini
|
|
||||||
ENV PIP_SRC=${SRC_DIR}
|
|
||||||
ENV CKAN_STORAGE_PATH=/var/lib/ckan
|
|
||||||
ENV GIT_URL=https://github.com/ckan/ckan.git
|
|
||||||
# CKAN version to build
|
|
||||||
ENV GIT_BRANCH=ckan-2.7.12
|
|
||||||
# Customize these on the .env file if needed
|
|
||||||
ENV CKAN_SITE_URL=http://localhost:5000
|
|
||||||
ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
|
|
||||||
|
|
||||||
WORKDIR ${APP_DIR}
|
|
||||||
|
|
||||||
# Install necessary packages to run CKAN
|
|
||||||
RUN apk add --no-cache tzdata \
|
|
||||||
git \
|
|
||||||
gettext \
|
|
||||||
postgresql-client \
|
|
||||||
python \
|
|
||||||
apache2-utils \
|
|
||||||
libxml2 \
|
|
||||||
libxslt \
|
|
||||||
musl-dev \
|
|
||||||
uwsgi \
|
|
||||||
uwsgi-http \
|
|
||||||
uwsgi-corerouter \
|
|
||||||
uwsgi-python \
|
|
||||||
py2-gevent \
|
|
||||||
uwsgi-gevent \
|
|
||||||
libmagic \
|
|
||||||
curl \
|
|
||||||
sudo && \
|
|
||||||
# Packages to build CKAN requirements and plugins
|
|
||||||
apk add --no-cache --virtual .build-deps \
|
|
||||||
postgresql-dev \
|
|
||||||
gcc \
|
|
||||||
make \
|
|
||||||
g++ \
|
|
||||||
autoconf \
|
|
||||||
automake \
|
|
||||||
libtool \
|
|
||||||
python-dev \
|
|
||||||
libxml2-dev \
|
|
||||||
libxslt-dev \
|
|
||||||
linux-headers
|
|
||||||
# Create SRC_DIR
|
|
||||||
RUN mkdir -p ${SRC_DIR} && \
|
|
||||||
# Install pip, supervisord and uwsgi
|
|
||||||
curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \
|
|
||||||
python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \
|
|
||||||
pip install supervisor && \
|
|
||||||
mkdir /etc/supervisord.d && \
|
|
||||||
#pip wheel --wheel-dir=/wheels uwsgi gevent && \
|
|
||||||
rm -rf ${SRC_DIR}/get-pip.py
|
|
||||||
|
|
||||||
COPY 2.7/setup/supervisord.conf /etc
|
|
||||||
|
|
||||||
# Install CKAN
|
|
||||||
RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
|
|
||||||
cd ${SRC_DIR}/ckan && \
|
|
||||||
cp who.ini ${APP_DIR} && \
|
|
||||||
# Workaround to solve https://github.com/psycopg/psycopg2/issues/594 in Alpine 3.7
|
|
||||||
sed -i -e "s/psycopg2==2.4.5/psycopg2==2.7.3.2/g" requirements.txt && \
|
|
||||||
pip install --no-binary :all: -r requirements.txt && \
|
|
||||||
# Install CKAN envvars to support loading config from environment variables
|
|
||||||
pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
|
|
||||||
# Create and update CKAN config
|
|
||||||
paster --plugin=ckan make-config ckan ${CKAN_INI} && \
|
|
||||||
paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
|
|
||||||
paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}"
|
|
||||||
|
|
||||||
# Create a local user and group to run the app
|
|
||||||
RUN addgroup -g 92 -S ckan && \
|
|
||||||
adduser -u 92 -h /srv/app -H -D -S -G ckan ckan
|
|
||||||
|
|
||||||
# Create local storage folder
|
|
||||||
RUN mkdir -p $CKAN_STORAGE_PATH && \
|
|
||||||
chown -R ckan:ckan $CKAN_STORAGE_PATH
|
|
||||||
|
|
||||||
# Copy setup files
|
|
||||||
COPY 2.7/setup ${APP_DIR}
|
|
||||||
RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf
|
|
||||||
|
|
||||||
# Create entrypoint directory for children image scripts
|
|
||||||
ONBUILD RUN mkdir /docker-entrypoint.d
|
|
||||||
|
|
||||||
RUN chown ckan -R /srv/app
|
|
||||||
|
|
||||||
EXPOSE 5000
|
|
||||||
|
|
||||||
HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan.sh"]
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
[uwsgi]
|
|
||||||
http-socket = :5000
|
|
||||||
uid = ckan
|
|
||||||
guid = ckan
|
|
||||||
plugins = python3
|
|
||||||
wsgi-file = /srv/app/wsgi.py
|
|
||||||
virtualenv = /srv/app
|
|
||||||
module = wsgi:application
|
|
||||||
master = true
|
|
||||||
processes = 5
|
|
||||||
pidfile = /tmp/%n.pid
|
|
||||||
harakiri = 50
|
|
||||||
max-requests = 5000
|
|
||||||
vacuum = true
|
|
||||||
callable = application
|
|
||||||
|
|
@ -1,197 +0,0 @@
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import subprocess
|
|
||||||
import psycopg2
|
|
||||||
import urllib2
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
|
|
||||||
ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini')
|
|
||||||
|
|
||||||
RETRY = 5
|
|
||||||
|
|
||||||
def update_plugins():
|
|
||||||
|
|
||||||
plugins = os.environ.get('CKAN__PLUGINS', '')
|
|
||||||
print('[prerun] Setting the following plugins in {}:'.format(ckan_ini))
|
|
||||||
print(plugins)
|
|
||||||
cmd = ['paster', '--plugin=ckan', 'config-tool',
|
|
||||||
ckan_ini, 'ckan.plugins = {}'.format(plugins)]
|
|
||||||
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
|
|
||||||
print '[prerun] Plugins set.'
|
|
||||||
|
|
||||||
|
|
||||||
def check_main_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_SQLALCHEMY_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db'
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_datastore_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db'
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_db_connection(conn_str, retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print '[prerun] Giving up after 5 tries...'
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print str(e)
|
|
||||||
print '[prerun] Unable to connect to the database, waiting...'
|
|
||||||
time.sleep(10)
|
|
||||||
check_db_connection(conn_str, retry=retry - 1)
|
|
||||||
else:
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def check_solr_connection(retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print '[prerun] Giving up after 5 tries...'
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
url = os.environ.get('CKAN_SOLR_URL', '')
|
|
||||||
search_url = '{url}/select/?q=*&wt=json'.format(url=url)
|
|
||||||
|
|
||||||
try:
|
|
||||||
connection = urllib2.urlopen(search_url)
|
|
||||||
except urllib2.URLError as e:
|
|
||||||
print str(e)
|
|
||||||
print '[prerun] Unable to connect to solr, waiting...'
|
|
||||||
time.sleep(10)
|
|
||||||
check_solr_connection(retry=retry - 1)
|
|
||||||
else:
|
|
||||||
eval(connection.read())
|
|
||||||
|
|
||||||
|
|
||||||
def init_db():
|
|
||||||
|
|
||||||
db_command = ['paster', '--plugin=ckan', 'db',
|
|
||||||
'init', '-c', ckan_ini]
|
|
||||||
print '[prerun] Initializing or upgrading db - start'
|
|
||||||
try:
|
|
||||||
subprocess.check_output(db_command, stderr=subprocess.STDOUT)
|
|
||||||
print '[prerun] Initializing or upgrading db - end'
|
|
||||||
except subprocess.CalledProcessError, e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print e.output
|
|
||||||
print '[prerun] Database not ready, waiting a bit before exit...'
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print e.output
|
|
||||||
raise e
|
|
||||||
|
|
||||||
|
|
||||||
def init_datastore_db():
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] Skipping datastore initialization'
|
|
||||||
return
|
|
||||||
|
|
||||||
datastore_perms_command = ['paster', '--plugin=ckan', 'datastore',
|
|
||||||
'set-permissions', '-c', ckan_ini]
|
|
||||||
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
cursor = connection.cursor()
|
|
||||||
|
|
||||||
print '[prerun] Initializing datastore db - start'
|
|
||||||
try:
|
|
||||||
datastore_perms = subprocess.Popen(
|
|
||||||
datastore_perms_command,
|
|
||||||
stdout=subprocess.PIPE)
|
|
||||||
|
|
||||||
perms_sql = datastore_perms.stdout.read()
|
|
||||||
# Remove internal pg command as psycopg2 does not like it
|
|
||||||
perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql)
|
|
||||||
cursor.execute(perms_sql)
|
|
||||||
for notice in connection.notices:
|
|
||||||
print notice
|
|
||||||
|
|
||||||
connection.commit()
|
|
||||||
|
|
||||||
print '[prerun] Initializing datastore db - end'
|
|
||||||
print datastore_perms.stdout.read()
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print '[prerun] Could not initialize datastore'
|
|
||||||
print str(e)
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError, e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print e.output
|
|
||||||
print '[prerun] Database not ready, waiting a bit before exit...'
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print e.output
|
|
||||||
raise e
|
|
||||||
finally:
|
|
||||||
cursor.close()
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def create_sysadmin():
|
|
||||||
|
|
||||||
name = os.environ.get('CKAN_SYSADMIN_NAME')
|
|
||||||
password = os.environ.get('CKAN_SYSADMIN_PASSWORD')
|
|
||||||
email = os.environ.get('CKAN_SYSADMIN_EMAIL')
|
|
||||||
|
|
||||||
if name and password and email:
|
|
||||||
|
|
||||||
# Check if user exists
|
|
||||||
command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini]
|
|
||||||
|
|
||||||
out = subprocess.check_output(command)
|
|
||||||
if 'User:None' not in re.sub(r'\s', '', out):
|
|
||||||
print '[prerun] Sysadmin user exists, skipping creation'
|
|
||||||
return
|
|
||||||
|
|
||||||
# Create user
|
|
||||||
command = ['paster', '--plugin=ckan', 'user', 'add',
|
|
||||||
name,
|
|
||||||
'password=' + password,
|
|
||||||
'email=' + email,
|
|
||||||
'-c', ckan_ini]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print '[prerun] Created user {0}'.format(name)
|
|
||||||
|
|
||||||
# Make it sysadmin
|
|
||||||
command = ['paster', '--plugin=ckan', 'sysadmin', 'add',
|
|
||||||
name,
|
|
||||||
'-c', ckan_ini]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print '[prerun] Made user {0} a sysadmin'.format(name)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
|
|
||||||
maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true'
|
|
||||||
|
|
||||||
if maintenance:
|
|
||||||
print '[prerun] Maintenance mode, skipping setup...'
|
|
||||||
else:
|
|
||||||
check_main_db_connection()
|
|
||||||
init_db()
|
|
||||||
update_plugins()
|
|
||||||
check_datastore_db_connection()
|
|
||||||
init_datastore_db()
|
|
||||||
check_solr_connection()
|
|
||||||
create_sysadmin()
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Run the prerun script to init CKAN and create the default admin user
|
|
||||||
sudo -u ckan -EH python prerun.py
|
|
||||||
|
|
||||||
# Run any startup scripts provided by images extending this one
|
|
||||||
if [[ -d "/docker-entrypoint.d" ]]
|
|
||||||
then
|
|
||||||
for f in /docker-entrypoint.d/*; do
|
|
||||||
case "$f" in
|
|
||||||
*.sh) echo "$0: Running init file $f"; . "$f" ;;
|
|
||||||
*.py) echo "$0: Running init file $f"; python "$f"; echo ;;
|
|
||||||
*) echo "$0: Ignoring $f (not an sh or py file)" ;;
|
|
||||||
esac
|
|
||||||
echo
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Set the common uwsgi options
|
|
||||||
UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L"
|
|
||||||
|
|
||||||
# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully
|
|
||||||
if [ $? -eq 0 ]
|
|
||||||
then
|
|
||||||
if [ "$PASSWORD_PROTECT" = true ]
|
|
||||||
then
|
|
||||||
if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ]
|
|
||||||
then
|
|
||||||
# Generate htpasswd file for basicauth
|
|
||||||
htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi with basicauth
|
|
||||||
sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS
|
|
||||||
else
|
|
||||||
echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi
|
|
||||||
sudo -u ckan -EH uwsgi $UWSGI_OPTS
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo "[prerun] failed...not starting CKAN."
|
|
||||||
fi
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
||||||
[unix_http_server]
|
|
||||||
file = /tmp/supervisor.sock
|
|
||||||
chmod = 0777
|
|
||||||
chown = nobody:nogroup
|
|
||||||
|
|
||||||
[supervisord]
|
|
||||||
logfile = /tmp/supervisord.log
|
|
||||||
logfile_maxbytes = 50MB
|
|
||||||
logfile_backups=10
|
|
||||||
loglevel = info
|
|
||||||
pidfile = /tmp/supervisord.pid
|
|
||||||
nodaemon = true
|
|
||||||
umask = 022
|
|
||||||
identifier = supervisor
|
|
||||||
|
|
||||||
[supervisorctl]
|
|
||||||
serverurl = unix:///tmp/supervisor.sock
|
|
||||||
|
|
||||||
[rpcinterface:supervisor]
|
|
||||||
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
|
||||||
|
|
||||||
[include]
|
|
||||||
files = /etc/supervisord.d/*.conf
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
[uwsgi]
|
|
||||||
route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd
|
|
||||||
|
|
@ -1,9 +0,0 @@
|
||||||
import os
|
|
||||||
from ckan.config.middleware import make_app
|
|
||||||
from ckan.cli import CKANConfigLoader
|
|
||||||
from logging.config import fileConfig as loggingFileConfig
|
|
||||||
config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini')
|
|
||||||
abspath = os.path.join(os.path.dirname(os.path.abspath(__file__)))
|
|
||||||
loggingFileConfig(config_filepath)
|
|
||||||
config = CKANConfigLoader(config_filepath).get_config()
|
|
||||||
application = make_app(config)
|
|
||||||
|
|
@ -1,95 +0,0 @@
|
||||||
FROM alpine:3.7
|
|
||||||
|
|
||||||
# Internals, you probably don't need to change these
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_DIR=/srv/app/src
|
|
||||||
ENV CKAN_INI=${APP_DIR}/production.ini
|
|
||||||
ENV PIP_SRC=${SRC_DIR}
|
|
||||||
ENV CKAN_STORAGE_PATH=/var/lib/ckan
|
|
||||||
ENV GIT_URL=https://github.com/ckan/ckan.git
|
|
||||||
# CKAN version to build
|
|
||||||
ENV GIT_BRANCH=ckan-2.8.9
|
|
||||||
# Customize these on the .env file if needed
|
|
||||||
ENV CKAN_SITE_URL=http://localhost:5000
|
|
||||||
ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
|
|
||||||
|
|
||||||
WORKDIR ${APP_DIR}
|
|
||||||
|
|
||||||
# Install necessary packages to run CKAN
|
|
||||||
RUN apk add --no-cache tzdata \
|
|
||||||
git \
|
|
||||||
gettext \
|
|
||||||
postgresql-client \
|
|
||||||
python \
|
|
||||||
apache2-utils \
|
|
||||||
libxml2 \
|
|
||||||
libxslt \
|
|
||||||
musl-dev \
|
|
||||||
uwsgi \
|
|
||||||
uwsgi-http \
|
|
||||||
uwsgi-corerouter \
|
|
||||||
uwsgi-python \
|
|
||||||
py2-gevent \
|
|
||||||
uwsgi-gevent \
|
|
||||||
libmagic \
|
|
||||||
curl \
|
|
||||||
sudo && \
|
|
||||||
# Packages to build CKAN requirements and plugins
|
|
||||||
apk add --no-cache --virtual .build-deps \
|
|
||||||
postgresql-dev \
|
|
||||||
gcc \
|
|
||||||
make \
|
|
||||||
g++ \
|
|
||||||
autoconf \
|
|
||||||
automake \
|
|
||||||
libtool \
|
|
||||||
python-dev \
|
|
||||||
libxml2-dev \
|
|
||||||
libxslt-dev \
|
|
||||||
linux-headers
|
|
||||||
# Create SRC_DIR
|
|
||||||
RUN mkdir -p ${SRC_DIR} && \
|
|
||||||
# Install pip, supervisord and uwsgi
|
|
||||||
curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \
|
|
||||||
python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \
|
|
||||||
pip install supervisor && \
|
|
||||||
mkdir /etc/supervisord.d && \
|
|
||||||
#pip wheel --wheel-dir=/wheels uwsgi gevent && \
|
|
||||||
rm -rf ${SRC_DIR}/get-pip.py
|
|
||||||
|
|
||||||
COPY 2.8/setup/supervisord.conf /etc
|
|
||||||
|
|
||||||
# Install CKAN
|
|
||||||
RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
|
|
||||||
cd ${SRC_DIR}/ckan && \
|
|
||||||
cp who.ini ${APP_DIR} && \
|
|
||||||
pip install --no-binary :all: -r requirements.txt && \
|
|
||||||
# Install CKAN envvars to support loading config from environment variables
|
|
||||||
pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
|
|
||||||
# Create and update CKAN config
|
|
||||||
paster --plugin=ckan make-config ckan ${CKAN_INI} && \
|
|
||||||
paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
|
|
||||||
paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}"
|
|
||||||
|
|
||||||
# Create a local user and group to run the app
|
|
||||||
RUN addgroup -g 92 -S ckan && \
|
|
||||||
adduser -u 92 -h /srv/app -H -D -S -G ckan ckan
|
|
||||||
|
|
||||||
# Create local storage folder
|
|
||||||
RUN mkdir -p $CKAN_STORAGE_PATH && \
|
|
||||||
chown -R ckan:ckan $CKAN_STORAGE_PATH
|
|
||||||
|
|
||||||
# Copy setup files
|
|
||||||
COPY 2.8/setup ${APP_DIR}
|
|
||||||
RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf
|
|
||||||
|
|
||||||
# Create entrypoint directory for children image scripts
|
|
||||||
ONBUILD RUN mkdir /docker-entrypoint.d
|
|
||||||
|
|
||||||
RUN chown ckan -R /srv/app
|
|
||||||
|
|
||||||
EXPOSE 5000
|
|
||||||
|
|
||||||
HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan.sh"]
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
[uwsgi]
|
|
||||||
http-socket = :5000
|
|
||||||
uid = ckan
|
|
||||||
guid = ckan
|
|
||||||
plugins = python3
|
|
||||||
wsgi-file = /srv/app/wsgi.py
|
|
||||||
virtualenv = /srv/app
|
|
||||||
module = wsgi:application
|
|
||||||
master = true
|
|
||||||
processes = 5
|
|
||||||
pidfile = /tmp/%n.pid
|
|
||||||
harakiri = 50
|
|
||||||
max-requests = 5000
|
|
||||||
vacuum = true
|
|
||||||
callable = application
|
|
||||||
|
|
@ -1,197 +0,0 @@
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import subprocess
|
|
||||||
import psycopg2
|
|
||||||
import urllib2
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
|
|
||||||
ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini')
|
|
||||||
|
|
||||||
RETRY = 5
|
|
||||||
|
|
||||||
def update_plugins():
|
|
||||||
|
|
||||||
plugins = os.environ.get('CKAN__PLUGINS', '')
|
|
||||||
print('[prerun] Setting the following plugins in {}:'.format(ckan_ini))
|
|
||||||
print(plugins)
|
|
||||||
cmd = ['paster', '--plugin=ckan', 'config-tool',
|
|
||||||
ckan_ini, 'ckan.plugins = {}'.format(plugins)]
|
|
||||||
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
|
|
||||||
print '[prerun] Plugins set.'
|
|
||||||
|
|
||||||
|
|
||||||
def check_main_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_SQLALCHEMY_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db'
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_datastore_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db'
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_db_connection(conn_str, retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print '[prerun] Giving up after 5 tries...'
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print str(e)
|
|
||||||
print '[prerun] Unable to connect to the database, waiting...'
|
|
||||||
time.sleep(10)
|
|
||||||
check_db_connection(conn_str, retry=retry - 1)
|
|
||||||
else:
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def check_solr_connection(retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print '[prerun] Giving up after 5 tries...'
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
url = os.environ.get('CKAN_SOLR_URL', '')
|
|
||||||
search_url = '{url}/select/?q=*&wt=json'.format(url=url)
|
|
||||||
|
|
||||||
try:
|
|
||||||
connection = urllib2.urlopen(search_url)
|
|
||||||
except urllib2.URLError as e:
|
|
||||||
print str(e)
|
|
||||||
print '[prerun] Unable to connect to solr, waiting...'
|
|
||||||
time.sleep(10)
|
|
||||||
check_solr_connection(retry=retry - 1)
|
|
||||||
else:
|
|
||||||
eval(connection.read())
|
|
||||||
|
|
||||||
|
|
||||||
def init_db():
|
|
||||||
|
|
||||||
db_command = ['paster', '--plugin=ckan', 'db',
|
|
||||||
'init', '-c', ckan_ini]
|
|
||||||
print '[prerun] Initializing or upgrading db - start'
|
|
||||||
try:
|
|
||||||
subprocess.check_output(db_command, stderr=subprocess.STDOUT)
|
|
||||||
print '[prerun] Initializing or upgrading db - end'
|
|
||||||
except subprocess.CalledProcessError, e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print e.output
|
|
||||||
print '[prerun] Database not ready, waiting a bit before exit...'
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print e.output
|
|
||||||
raise e
|
|
||||||
|
|
||||||
|
|
||||||
def init_datastore_db():
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print '[prerun] Skipping datastore initialization'
|
|
||||||
return
|
|
||||||
|
|
||||||
datastore_perms_command = ['paster', '--plugin=ckan', 'datastore',
|
|
||||||
'set-permissions', '-c', ckan_ini]
|
|
||||||
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
cursor = connection.cursor()
|
|
||||||
|
|
||||||
print '[prerun] Initializing datastore db - start'
|
|
||||||
try:
|
|
||||||
datastore_perms = subprocess.Popen(
|
|
||||||
datastore_perms_command,
|
|
||||||
stdout=subprocess.PIPE)
|
|
||||||
|
|
||||||
perms_sql = datastore_perms.stdout.read()
|
|
||||||
# Remove internal pg command as psycopg2 does not like it
|
|
||||||
perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql)
|
|
||||||
cursor.execute(perms_sql)
|
|
||||||
for notice in connection.notices:
|
|
||||||
print notice
|
|
||||||
|
|
||||||
connection.commit()
|
|
||||||
|
|
||||||
print '[prerun] Initializing datastore db - end'
|
|
||||||
print datastore_perms.stdout.read()
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print '[prerun] Could not initialize datastore'
|
|
||||||
print str(e)
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError, e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print e.output
|
|
||||||
print '[prerun] Database not ready, waiting a bit before exit...'
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print e.output
|
|
||||||
raise e
|
|
||||||
finally:
|
|
||||||
cursor.close()
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def create_sysadmin():
|
|
||||||
|
|
||||||
name = os.environ.get('CKAN_SYSADMIN_NAME')
|
|
||||||
password = os.environ.get('CKAN_SYSADMIN_PASSWORD')
|
|
||||||
email = os.environ.get('CKAN_SYSADMIN_EMAIL')
|
|
||||||
|
|
||||||
if name and password and email:
|
|
||||||
|
|
||||||
# Check if user exists
|
|
||||||
command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini]
|
|
||||||
|
|
||||||
out = subprocess.check_output(command)
|
|
||||||
if 'User:None' not in re.sub(r'\s', '', out):
|
|
||||||
print '[prerun] Sysadmin user exists, skipping creation'
|
|
||||||
return
|
|
||||||
|
|
||||||
# Create user
|
|
||||||
command = ['paster', '--plugin=ckan', 'user', 'add',
|
|
||||||
name,
|
|
||||||
'password=' + password,
|
|
||||||
'email=' + email,
|
|
||||||
'-c', ckan_ini]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print '[prerun] Created user {0}'.format(name)
|
|
||||||
|
|
||||||
# Make it sysadmin
|
|
||||||
command = ['paster', '--plugin=ckan', 'sysadmin', 'add',
|
|
||||||
name,
|
|
||||||
'-c', ckan_ini]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print '[prerun] Made user {0} a sysadmin'.format(name)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
|
|
||||||
maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true'
|
|
||||||
|
|
||||||
if maintenance:
|
|
||||||
print '[prerun] Maintenance mode, skipping setup...'
|
|
||||||
else:
|
|
||||||
check_main_db_connection()
|
|
||||||
init_db()
|
|
||||||
update_plugins()
|
|
||||||
check_datastore_db_connection()
|
|
||||||
init_datastore_db()
|
|
||||||
check_solr_connection()
|
|
||||||
create_sysadmin()
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Run the prerun script to init CKAN and create the default admin user
|
|
||||||
sudo -u ckan -EH python prerun.py
|
|
||||||
|
|
||||||
# Run any startup scripts provided by images extending this one
|
|
||||||
if [[ -d "/docker-entrypoint.d" ]]
|
|
||||||
then
|
|
||||||
for f in /docker-entrypoint.d/*; do
|
|
||||||
case "$f" in
|
|
||||||
*.sh) echo "$0: Running init file $f"; . "$f" ;;
|
|
||||||
*.py) echo "$0: Running init file $f"; python "$f"; echo ;;
|
|
||||||
*) echo "$0: Ignoring $f (not an sh or py file)" ;;
|
|
||||||
esac
|
|
||||||
echo
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Set the common uwsgi options
|
|
||||||
UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L"
|
|
||||||
|
|
||||||
# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully
|
|
||||||
if [ $? -eq 0 ]
|
|
||||||
then
|
|
||||||
if [ "$PASSWORD_PROTECT" = true ]
|
|
||||||
then
|
|
||||||
if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ]
|
|
||||||
then
|
|
||||||
# Generate htpasswd file for basicauth
|
|
||||||
htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi with basicauth
|
|
||||||
sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS
|
|
||||||
else
|
|
||||||
echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi
|
|
||||||
sudo -u ckan -EH uwsgi $UWSGI_OPTS
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo "[prerun] failed...not starting CKAN."
|
|
||||||
fi
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
||||||
[unix_http_server]
|
|
||||||
file = /tmp/supervisor.sock
|
|
||||||
chmod = 0777
|
|
||||||
chown = nobody:nogroup
|
|
||||||
|
|
||||||
[supervisord]
|
|
||||||
logfile = /tmp/supervisord.log
|
|
||||||
logfile_maxbytes = 50MB
|
|
||||||
logfile_backups=10
|
|
||||||
loglevel = info
|
|
||||||
pidfile = /tmp/supervisord.pid
|
|
||||||
nodaemon = true
|
|
||||||
umask = 022
|
|
||||||
identifier = supervisor
|
|
||||||
|
|
||||||
[supervisorctl]
|
|
||||||
serverurl = unix:///tmp/supervisor.sock
|
|
||||||
|
|
||||||
[rpcinterface:supervisor]
|
|
||||||
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
|
||||||
|
|
||||||
[include]
|
|
||||||
files = /etc/supervisord.d/*.conf
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
[uwsgi]
|
|
||||||
route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd
|
|
||||||
|
|
@ -1,9 +0,0 @@
|
||||||
import os
|
|
||||||
from ckan.config.middleware import make_app
|
|
||||||
from ckan.cli import CKANConfigLoader
|
|
||||||
from logging.config import fileConfig as loggingFileConfig
|
|
||||||
config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini')
|
|
||||||
abspath = os.path.join(os.path.dirname(os.path.abspath(__file__)))
|
|
||||||
loggingFileConfig(config_filepath)
|
|
||||||
config = CKANConfigLoader(config_filepath).get_config()
|
|
||||||
application = make_app(config)
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
FROM ckan/ckan-dev:testing-only.2.9
|
|
||||||
|
|
||||||
LABEL maintainer="brett@kowh.ai"
|
|
||||||
|
|
||||||
# Set up environment variables
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV TZ=UTC
|
|
||||||
RUN echo ${TZ} > /etc/timezone
|
|
||||||
|
|
||||||
# Make sure both files are not exactly the same
|
|
||||||
RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \
|
|
||||||
cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\
|
|
||||||
fi ;
|
|
||||||
|
|
||||||
# Install any extensions needed by your CKAN instance
|
|
||||||
# (Make sure to add the plugins to CKAN__PLUGINS in the .env file)
|
|
||||||
# For instance:
|
|
||||||
#RUN pip install -e git+https://github.com/ckan/ckanext-pages.git#egg=ckanext-pages && \
|
|
||||||
# pip install -e git+https://github.com/ckan/ckanext-dcat.git@v0.0.6#egg=ckanext-dcat && \
|
|
||||||
# pip install -r https://raw.githubusercontent.com/ckan/ckanext-dcat/v0.0.6/requirements.txt
|
|
||||||
|
|
||||||
# Clone the extension(s) your are writing for your own project in the `src` folder
|
|
||||||
# to get them mounted in this image at runtime
|
|
||||||
|
|
||||||
# Apply any patches needed to CKAN core or any of the built extensions (not the
|
|
||||||
# runtime mounted ones)
|
|
||||||
# See https://github.com/okfn/docker-ckan#applying-patches
|
|
||||||
|
|
||||||
COPY patches ${APP_DIR}/patches
|
|
||||||
|
|
||||||
RUN for d in ${APP_DIR}/patches/*; do \
|
|
||||||
if [ -d $d ]; then \
|
|
||||||
for f in `ls $d/*.patch | sort -g`; do \
|
|
||||||
cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
|
|
||||||
done ; \
|
|
||||||
fi ; \
|
|
||||||
done
|
|
||||||
|
|
@ -1,20 +0,0 @@
|
||||||
FROM ckan/ckan-base:testing-only.2.7
|
|
||||||
|
|
||||||
LABEL maintainer="brett@kowh.ai"
|
|
||||||
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions
|
|
||||||
|
|
||||||
# Install packages needed by the dev requirements
|
|
||||||
RUN apk add --no-cache libffi-dev
|
|
||||||
|
|
||||||
# Install CKAN dev requirements
|
|
||||||
RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt
|
|
||||||
|
|
||||||
# Create folder for local extensions sources
|
|
||||||
RUN mkdir $SRC_EXTENSIONS_DIR
|
|
||||||
|
|
||||||
COPY setup/start_ckan_development.sh ${APP_DIR}
|
|
||||||
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan_development.sh"]
|
|
||||||
|
|
@ -1,20 +0,0 @@
|
||||||
FROM ckan/ckan-base:testing-only.2.8
|
|
||||||
|
|
||||||
LABEL maintainer="brett@kowh.ai"
|
|
||||||
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions
|
|
||||||
|
|
||||||
# Install packages needed by the dev requirements
|
|
||||||
RUN apk add --no-cache libffi-dev
|
|
||||||
|
|
||||||
# Install CKAN dev requirements
|
|
||||||
RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt
|
|
||||||
|
|
||||||
# Create folder for local extensions sources
|
|
||||||
RUN mkdir $SRC_EXTENSIONS_DIR
|
|
||||||
|
|
||||||
COPY setup/start_ckan_development.sh ${APP_DIR}
|
|
||||||
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan_development.sh"]
|
|
||||||
|
|
@ -1,29 +0,0 @@
|
||||||
FROM ckan/ckan-base:testing-only.2.9
|
|
||||||
|
|
||||||
LABEL maintainer="brett@kowh.ai"
|
|
||||||
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions
|
|
||||||
|
|
||||||
# Install packages needed by the dev requirements
|
|
||||||
RUN apk add --no-cache libffi-dev
|
|
||||||
|
|
||||||
# Set up Python3 virtual environment
|
|
||||||
RUN cd ${APP_DIR} && \
|
|
||||||
source ${APP_DIR}/bin/activate
|
|
||||||
|
|
||||||
# Virtual environment binaries/scripts to be used first
|
|
||||||
ENV PATH=${APP_DIR}/bin:${PATH}
|
|
||||||
|
|
||||||
# Install CKAN dev requirements
|
|
||||||
# Will need to change this eventually - when CKAN 2.9 is out
|
|
||||||
# wget https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt
|
|
||||||
# RUN pip3 install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt
|
|
||||||
RUN pip3 install -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt
|
|
||||||
|
|
||||||
# Create folder for local extensions sources
|
|
||||||
RUN mkdir $SRC_EXTENSIONS_DIR
|
|
||||||
|
|
||||||
COPY setup/start_ckan_development.sh ${APP_DIR}
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan_development.sh"]
|
|
||||||
|
|
@ -1,82 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Install any local extensions in the src_extensions volume
|
|
||||||
echo "Looking for local extensions to install..."
|
|
||||||
echo "Extension dir contents:"
|
|
||||||
ls -la $SRC_EXTENSIONS_DIR
|
|
||||||
for i in $SRC_EXTENSIONS_DIR/*
|
|
||||||
do
|
|
||||||
if [ -d $i ];
|
|
||||||
then
|
|
||||||
|
|
||||||
if [ -f $i/pip-requirements.txt ];
|
|
||||||
then
|
|
||||||
pip install -r $i/pip-requirements.txt
|
|
||||||
echo "Found requirements file in $i"
|
|
||||||
fi
|
|
||||||
if [ -f $i/requirements.txt ];
|
|
||||||
then
|
|
||||||
pip install -r $i/requirements.txt
|
|
||||||
echo "Found requirements file in $i"
|
|
||||||
fi
|
|
||||||
if [ -f $i/dev-requirements.txt ];
|
|
||||||
then
|
|
||||||
pip install -r $i/dev-requirements.txt
|
|
||||||
echo "Found dev-requirements file in $i"
|
|
||||||
fi
|
|
||||||
if [ -f $i/setup.py ];
|
|
||||||
then
|
|
||||||
cd $i
|
|
||||||
python $i/setup.py develop
|
|
||||||
echo "Found setup.py file in $i"
|
|
||||||
cd $APP_DIR
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Point `use` in test.ini to location of `test-core.ini`
|
|
||||||
if [ -f $i/test.ini ];
|
|
||||||
then
|
|
||||||
echo "Updating \`test.ini\` reference to \`test-core.ini\` for plugin $i"
|
|
||||||
paster --plugin=ckan config-tool $i/test.ini "use = config:../../src/ckan/test-core.ini"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Set debug to true
|
|
||||||
echo "Enabling debug mode"
|
|
||||||
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
|
|
||||||
|
|
||||||
# Update the plugins setting in the ini file with the values defined in the env var
|
|
||||||
echo "Loading the following plugins: $CKAN__PLUGINS"
|
|
||||||
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
|
|
||||||
|
|
||||||
# Update test-core.ini DB, SOLR & Redis settings
|
|
||||||
echo "Loading test settings into test-core.ini"
|
|
||||||
ckan config-tool $SRC_DIR/ckan/test-core.ini \
|
|
||||||
"sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \
|
|
||||||
"ckan.datstore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \
|
|
||||||
"ckan.datstore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \
|
|
||||||
"solr_url = $TEST_CKAN_SOLR_URL" \
|
|
||||||
"ckan.redis_url = $TEST_CKAN_REDIS_URL"
|
|
||||||
|
|
||||||
# Run the prerun script to init CKAN and create the default admin user
|
|
||||||
sudo -u ckan -EH python prerun.py
|
|
||||||
|
|
||||||
# Run any startup scripts provided by images extending this one
|
|
||||||
if [[ -d "/docker-entrypoint.d" ]]
|
|
||||||
then
|
|
||||||
for f in /docker-entrypoint.d/*; do
|
|
||||||
case "$f" in
|
|
||||||
*.sh) echo "$0: Running init file $f"; . "$f" ;;
|
|
||||||
*.py) echo "$0: Running init file $f"; python "$f"; echo ;;
|
|
||||||
*) echo "$0: Ignoring $f (not an sh or py file)" ;;
|
|
||||||
esac
|
|
||||||
echo
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
|
|
||||||
# Start the development server with automatic reload
|
|
||||||
# Check the --reloader options sudo -u ckan -EH ckan -c $CKAN_INI run --reloader <TEXT>
|
|
||||||
sudo -u ckan -EH ckan -c $CKAN_INI run
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# These are packages that are required by ckan developers - for running ckan in debug mode, running ckan tests, building the docs and to pip-compile the requirements.in file.
|
|
||||||
|
|
||||||
beautifulsoup4==4.5.1
|
|
||||||
coveralls #Let Unpinned - Requires latest coveralls
|
|
||||||
docutils==0.12
|
|
||||||
factory-boy==2.1.1
|
|
||||||
Flask-DebugToolbar==0.10.1
|
|
||||||
freezegun==0.3.15
|
|
||||||
responses==0.10.6
|
|
||||||
mock==2.0.0
|
|
||||||
pycodestyle==2.5.0
|
|
||||||
pip-tools==2.0.2
|
|
||||||
pyfakefs==3.2
|
|
||||||
Sphinx==1.8.5
|
|
||||||
sphinx-rtd-theme==0.3.1
|
|
||||||
cookiecutter==1.6.0
|
|
||||||
|
|
||||||
# nose==1.3.0 # already in requirements.txt
|
|
||||||
pytest==4.6.5
|
|
||||||
pytest-split-tests==1.0.9
|
|
||||||
pytest-cov==2.7.1
|
|
||||||
pytest-freezegun==0.4.1
|
|
||||||
pytest-rerunfailures==8.0
|
|
||||||
|
|
||||||
towncrier==19.2.0
|
|
||||||
Loading…
Reference in New Issue