diff --git a/.env.template b/.env.example similarity index 100% rename from .env.template rename to .env.example diff --git a/README.md b/README.md index b324eca..b35a1b5 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ * [Overview](#overview) * [docker compose vs docker-compose](#docker-compose-vs-docker-compose) -* [Quick start](#quick-start) -* [The ckanext-envvars extension](#envvars) +* [Installing Docker](#installing-docker) +* [Install CKAN plus dependencies](#install-ckan-plus-dependencies) * [Development mode](#development-mode) * [Create an extension](#create-an-extension) * [CKAN images](#ckan-images) @@ -13,6 +13,7 @@ * [Debugging with pdb](#pdb) * [Datastore and Datapusher](#Datastore-and-datapusher) * [NGINX](#nginx) +* [The ckanext-envvars extension](#envvars) * [Known Issues](#known-issues) @@ -38,7 +39,14 @@ All Docker Compose commands in this README will use the V2 version of Compose ie used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for more information. -## Quick start +## Installing Docker + +Install Docker by following the following instructions: [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) + +To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output +versions for client and server. + +## Install CKAN plus dependencies Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs. @@ -56,25 +64,8 @@ At the end of the container start sequence there should be 6 containers running ![Screenshot 2022-12-12 at 10 36 21 am](https://user-images.githubusercontent.com/54408245/207012236-f9571baa-4d99-4ffe-bd93-30b11c4829e0.png) -## envvars +After this step, CKAN should be running at `CKAN_SITE_URL`. -The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images. -This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value. - -For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way: - - All uppercase - Replace periods ('.') with two underscores ('__') - Keys must begin with 'CKAN' or 'CKANEXT' - -For example: - - CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher" - CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000 - -These parameters can be added to the `.env` file - -For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) ## Development mode @@ -195,12 +186,32 @@ running the latest version of Datapusher. ## NGINX -* The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production. +The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production. Creating the SSL cert and key files as follows: `openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=DE/ST=Berlin/L=Berlin/O=None/CN=localhost" -keyout ckan-local.key -out ckan-local.crt` The `ckan-local.*` files will then need to be moved into the nginx/setup/ directory +## envvars + +The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images. +This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value. + +For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way: + + All uppercase + Replace periods ('.') with two underscores ('__') + Keys must begin with 'CKAN' or 'CKANEXT' + +For example: + + * `CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher"` + * `CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000` + +These parameters can be added to the `.env` file + +For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) + ## Known Issues -* Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that. +Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that. diff --git a/auxilliary stuff (delete before going live)/OLD.README.txt b/auxilliary stuff (delete before going live)/OLD.README.txt deleted file mode 100644 index 70d1f75..0000000 --- a/auxilliary stuff (delete before going live)/OLD.README.txt +++ /dev/null @@ -1,66 +0,0 @@ -### Please note that this re-purposed CKAN Docker repo is a WORK IN PROGRESS ### -### It should not be used to install CKAN via Docker until this page is updated ### - -------------------------------------- -(From Adria) -Be limited in scope, and act as a base that users can extend to their own needs -Be opinionated, and provide one way to do things -Be automatically tested -------------------------------------- - -should we have an nginx container included? maybe just instructions on how to include one - -ARG should be used for sensitive variables so as to keep the values of these variables -out of the build and hence will not show with an "inspect image" command - -Do we pre-build a CKAN image and use that (and extend) as the base image OR - just build it from scratch and while saving to storage what we need to - -To use local storage for the ckan.ini file and the CKAN src code, do the following: - -# docker-compose up -d --build -# mkdir local -# docker cp ckan:/srv/app/ckan.ini ./local/ckan.ini -# docker cp ckan:/srv/app/src ./local/src -Stop/Remove ckan container -Start ckan container as follows -Use a bind mount for the config file (ckan.ini) -# docker run -p 0.0.0.0:5000:5000 --net ckan-docker_default --hostname ckan --name ckan \ - --env-file=./environ --mount type=bind,source=$(pwd)/local/ckan.ini,target=/srv/app/ckan.ini \ - --mount type=bind,source=$(pwd)/local/src,target=/srv/app/src \ - -d ckan-docker_ckan - -Maybe include a script to replace docker-compose if required - -Dev Mode (OKFN) - -The differences between Docker dev abd Docker base is as the following: - -docker-compose.dev.yml - solr: explicitly puts in ports (8983:8983) - db: Doesn't pass in environment and arg values - ckan: has extra volume bind mount (./src:/srv/app/src_extensions) - -Dockerfile.dev - Takes the base image and - Adds a new directory (SRC_EXTENSIONS_DIR=/srv/app/src_extensions) - installs libffi-dev - installs dev-requirements.txt - runs different start script (start_ckan_development.sh) which installs any extension located in SRC_EXTENSIONS_DIR - runs a typical Dev install - pip install -r pip-requirements.txt - pip install -r requirements.txt - pip install -r dev-requirements.txt - python3 setup.py develop - ckan config-tool test.ini - ckan config-tool $CKAN_INI -s DEFAULT "debug = true" - ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS" - ckan config-tool $SRC_DIR/ckan/test-core.ini \ - "sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \ - "ckan.datastore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \ - "ckan.datastore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \ - "solr_url = $TEST_CKAN_SOLR_URL" \ - "ckan.redis.url = $TEST_CKAN_REDIS_URL" - - All other steps are similar to the base Dockerfile - diff --git a/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh b/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh deleted file mode 100755 index 791bb28..0000000 --- a/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/bin/bash - -# -# Create the network first -# -docker network create ckan-net -# - - -# Redis - Run only (image already built) -# -REDIS_CONTAINERNAME=redis -REDIS_HOSTNAME=redis - -docker run -d -it --net ckan-net --name ${REDIS_CONTAINERNAME} --hostname ${REDIS_HOSTNAME} redis:latest - -# -# Solr - Run/Build -# -SOLR_CONTAINERNAME=solr -SOLR_HOSTNAME=solr -SOLR_IMAGENAME=solr - -cd solr -docker image build -t ${SOLR_IMAGENAME} . -docker run -d -it --net ckan-net --name ${SOLR_CONTAINERNAME} --hostname ${SOLR_HOSTNAME} \ - -v solr_core:/opt/solr/server/solr ${SOLR_IMAGENAME} -cd - - -# -# PostgreSQL (DB) - Run/Build -# -DB_CONTAINERNAME=db -DB_HOSTNAME=db -DB_IMAGENAME=db - -cd postgresql -docker image build -t ${DB_IMAGENAME} . -docker run -d -it --net ckan-net --name ${DB_CONTAINERNAME} --hostname ${DB_HOSTNAME} --env-file=../environ \ - -v pg_data:/var/lib/postgresql/data ${DB_IMAGENAME} -cd - - -# -# CKAN - Run/Build -# -CKAN_CONTAINERNAME=ckan -CKAN_HOSTNAME=ckan -CKAN_IMAGENAME=ckan - -cd ckan -docker image build -t ${CKAN_IMAGENAME} . -docker run -d -it --net ckan-net --name ${CKAN_CONTAINERNAME} --hostname ${CKAN_HOSTNAME} --env-file=../environ \ - -v ckan_storage:/var/lib/ckan ${CKAN_IMAGENAME} -cd - - -# -# Nginx - Run/Build -# -NGINX_CONTAINERNAME=nginx -NGINX_HOSTNAME=nginx -NGINX_IMAGENAME=nginx - -cd nginx -docker image build -t ${NGINX_IMAGENAME} . -docker run -d -it --net ckan-net --name ${NGINX_CONTAINERNAME} --hostname ${NGINX_HOSTNAME} -p 0.0.0.0:80:80 ${NGINX_IMAGENAME} -cd - - -# -# PostgreSQL client -# -docker run -d -it --net ckan-net --name postgres-client --hostname postgres-client -e POSTGRES_PASSWORD=ckan -p 5432:5432 postgres diff --git a/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh b/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh deleted file mode 100755 index d1a8d1f..0000000 --- a/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -# Stop all containers -docker kill $(docker ps -q) - -# Remove all stopped containers -docker rm -f $(docker ps -aq) - -# Remove all docker images -docker rmi $(docker images -q) - -# Remove all volumes -docker volume rm $(docker volume ls -q) - -# Remove ckan-docker_default network -docker network rm ckan-docker_default diff --git a/auxilliary stuff (delete before going live)/environ b/auxilliary stuff (delete before going live)/environ deleted file mode 100644 index df4e7f3..0000000 --- a/auxilliary stuff (delete before going live)/environ +++ /dev/null @@ -1,34 +0,0 @@ -REDIS_CONTAINERNAME=redis -REDIS_HOSTNAME=redis - -SOLR_CONTAINERNAME=solr -SOLR_HOSTNAME=solr -SOLR_IMAGENAME=solr - -DB_CONTAINERNAME=db -DB_HOSTNAME=db -DB_IMAGENAME=db -POSTGRES_PASSWORD=ckan -DS_RO_PASS=datastore -PGDATA=/var/lib/postgresql/data/db - -CKAN_CONTAINERNAME=ckan -CKAN_HOSTNAME=ckan -CKAN_IMAGENAME=ckan - -CKAN_SQLALCHEMY_URL=postgresql://ckan:ckan@db/ckan -CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore -CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore -CKAN_SOLR_URL=http://solr:8983/solr/ckan -CKAN_REDIS_URL=redis://redis:6379/1 -CKAN_SITE_URL=http://localhost -CKAN_MAX_UPLOAD_SIZE_MB=32 -POSTGRES_PASSWORD=ckan -DS_RO_PASS=datastore -CKAN_PORT=5000 -CKAN_DATAPUSHER_URL=http://datapusher:8800 -CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000 - -NGINX_CONTAINERNAME=nginx -NGINX_HOSTNAME=nginx -NGINX_IMAGENAME=nginx diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile deleted file mode 100755 index 0904522..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile +++ /dev/null @@ -1,97 +0,0 @@ -FROM alpine:3.7 - -# Internals, you probably don't need to change these -ENV APP_DIR=/srv/app -ENV SRC_DIR=/srv/app/src -ENV CKAN_INI=${APP_DIR}/production.ini -ENV PIP_SRC=${SRC_DIR} -ENV CKAN_STORAGE_PATH=/var/lib/ckan -ENV GIT_URL=https://github.com/ckan/ckan.git -# CKAN version to build -ENV GIT_BRANCH=ckan-2.7.12 -# Customize these on the .env file if needed -ENV CKAN_SITE_URL=http://localhost:5000 -ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars - -WORKDIR ${APP_DIR} - -# Install necessary packages to run CKAN -RUN apk add --no-cache tzdata \ - git \ - gettext \ - postgresql-client \ - python \ - apache2-utils \ - libxml2 \ - libxslt \ - musl-dev \ - uwsgi \ - uwsgi-http \ - uwsgi-corerouter \ - uwsgi-python \ - py2-gevent \ - uwsgi-gevent \ - libmagic \ - curl \ - sudo && \ - # Packages to build CKAN requirements and plugins - apk add --no-cache --virtual .build-deps \ - postgresql-dev \ - gcc \ - make \ - g++ \ - autoconf \ - automake \ - libtool \ - python-dev \ - libxml2-dev \ - libxslt-dev \ - linux-headers - # Create SRC_DIR - RUN mkdir -p ${SRC_DIR} && \ - # Install pip, supervisord and uwsgi - curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \ - python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \ - pip install supervisor && \ - mkdir /etc/supervisord.d && \ - #pip wheel --wheel-dir=/wheels uwsgi gevent && \ - rm -rf ${SRC_DIR}/get-pip.py - -COPY 2.7/setup/supervisord.conf /etc - -# Install CKAN -RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \ - cd ${SRC_DIR}/ckan && \ - cp who.ini ${APP_DIR} && \ - # Workaround to solve https://github.com/psycopg/psycopg2/issues/594 in Alpine 3.7 - sed -i -e "s/psycopg2==2.4.5/psycopg2==2.7.3.2/g" requirements.txt && \ - pip install --no-binary :all: -r requirements.txt && \ - # Install CKAN envvars to support loading config from environment variables - pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \ - # Create and update CKAN config - paster --plugin=ckan make-config ckan ${CKAN_INI} && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" - -# Create a local user and group to run the app -RUN addgroup -g 92 -S ckan && \ - adduser -u 92 -h /srv/app -H -D -S -G ckan ckan - -# Create local storage folder -RUN mkdir -p $CKAN_STORAGE_PATH && \ - chown -R ckan:ckan $CKAN_STORAGE_PATH - -# Copy setup files -COPY 2.7/setup ${APP_DIR} -RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf - -# Create entrypoint directory for children image scripts -ONBUILD RUN mkdir /docker-entrypoint.d - -RUN chown ckan -R /srv/app - -EXPOSE 5000 - -HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1 - -CMD ["/srv/app/start_ckan.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini deleted file mode 100644 index 2361f36..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini +++ /dev/null @@ -1,15 +0,0 @@ -[uwsgi] -http-socket = :5000 -uid = ckan -guid = ckan -plugins = python3 -wsgi-file = /srv/app/wsgi.py -virtualenv = /srv/app -module = wsgi:application -master = true -processes = 5 -pidfile = /tmp/%n.pid -harakiri = 50 -max-requests = 5000 -vacuum = true -callable = application diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py deleted file mode 100755 index 7ebacfa..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py +++ /dev/null @@ -1,197 +0,0 @@ -import os -import sys -import subprocess -import psycopg2 -import urllib2 -import time -import re - -ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini') - -RETRY = 5 - -def update_plugins(): - - plugins = os.environ.get('CKAN__PLUGINS', '') - print('[prerun] Setting the following plugins in {}:'.format(ckan_ini)) - print(plugins) - cmd = ['paster', '--plugin=ckan', 'config-tool', - ckan_ini, 'ckan.plugins = {}'.format(plugins)] - subprocess.check_output(cmd, stderr=subprocess.STDOUT) - print '[prerun] Plugins set.' - - -def check_main_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_SQLALCHEMY_URL') - if not conn_str: - print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_datastore_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_db_connection(conn_str, retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - try: - connection = psycopg2.connect(conn_str) - - except psycopg2.Error as e: - print str(e) - print '[prerun] Unable to connect to the database, waiting...' - time.sleep(10) - check_db_connection(conn_str, retry=retry - 1) - else: - connection.close() - - -def check_solr_connection(retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - url = os.environ.get('CKAN_SOLR_URL', '') - search_url = '{url}/select/?q=*&wt=json'.format(url=url) - - try: - connection = urllib2.urlopen(search_url) - except urllib2.URLError as e: - print str(e) - print '[prerun] Unable to connect to solr, waiting...' - time.sleep(10) - check_solr_connection(retry=retry - 1) - else: - eval(connection.read()) - - -def init_db(): - - db_command = ['paster', '--plugin=ckan', 'db', - 'init', '-c', ckan_ini] - print '[prerun] Initializing or upgrading db - start' - try: - subprocess.check_output(db_command, stderr=subprocess.STDOUT) - print '[prerun] Initializing or upgrading db - end' - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - - -def init_datastore_db(): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] Skipping datastore initialization' - return - - datastore_perms_command = ['paster', '--plugin=ckan', 'datastore', - 'set-permissions', '-c', ckan_ini] - - connection = psycopg2.connect(conn_str) - cursor = connection.cursor() - - print '[prerun] Initializing datastore db - start' - try: - datastore_perms = subprocess.Popen( - datastore_perms_command, - stdout=subprocess.PIPE) - - perms_sql = datastore_perms.stdout.read() - # Remove internal pg command as psycopg2 does not like it - perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql) - cursor.execute(perms_sql) - for notice in connection.notices: - print notice - - connection.commit() - - print '[prerun] Initializing datastore db - end' - print datastore_perms.stdout.read() - except psycopg2.Error as e: - print '[prerun] Could not initialize datastore' - print str(e) - - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - finally: - cursor.close() - connection.close() - - -def create_sysadmin(): - - name = os.environ.get('CKAN_SYSADMIN_NAME') - password = os.environ.get('CKAN_SYSADMIN_PASSWORD') - email = os.environ.get('CKAN_SYSADMIN_EMAIL') - - if name and password and email: - - # Check if user exists - command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini] - - out = subprocess.check_output(command) - if 'User:None' not in re.sub(r'\s', '', out): - print '[prerun] Sysadmin user exists, skipping creation' - return - - # Create user - command = ['paster', '--plugin=ckan', 'user', 'add', - name, - 'password=' + password, - 'email=' + email, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Created user {0}'.format(name) - - # Make it sysadmin - command = ['paster', '--plugin=ckan', 'sysadmin', 'add', - name, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Made user {0} a sysadmin'.format(name) - - -if __name__ == '__main__': - - maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true' - - if maintenance: - print '[prerun] Maintenance mode, skipping setup...' - else: - check_main_db_connection() - init_db() - update_plugins() - check_datastore_db_connection() - init_datastore_db() - check_solr_connection() - create_sysadmin() \ No newline at end of file diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh deleted file mode 100755 index d1c6726..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Set the common uwsgi options -UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L" - -# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully -if [ $? -eq 0 ] -then - if [ "$PASSWORD_PROTECT" = true ] - then - if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ] - then - # Generate htpasswd file for basicauth - htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi with basicauth - sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS - else - echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..." - exit 1 - fi - else - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi - sudo -u ckan -EH uwsgi $UWSGI_OPTS - fi -else - echo "[prerun] failed...not starting CKAN." -fi diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf deleted file mode 100755 index a3f6671..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf +++ /dev/null @@ -1,23 +0,0 @@ -[unix_http_server] -file = /tmp/supervisor.sock -chmod = 0777 -chown = nobody:nogroup - -[supervisord] -logfile = /tmp/supervisord.log -logfile_maxbytes = 50MB -logfile_backups=10 -loglevel = info -pidfile = /tmp/supervisord.pid -nodaemon = true -umask = 022 -identifier = supervisor - -[supervisorctl] -serverurl = unix:///tmp/supervisor.sock - -[rpcinterface:supervisor] -supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface - -[include] -files = /etc/supervisord.d/*.conf diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf deleted file mode 100755 index 6321d6d..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf +++ /dev/null @@ -1,2 +0,0 @@ -[uwsgi] -route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py deleted file mode 100644 index b37d80e..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py +++ /dev/null @@ -1,9 +0,0 @@ -import os -from ckan.config.middleware import make_app -from ckan.cli import CKANConfigLoader -from logging.config import fileConfig as loggingFileConfig -config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini') -abspath = os.path.join(os.path.dirname(os.path.abspath(__file__))) -loggingFileConfig(config_filepath) -config = CKANConfigLoader(config_filepath).get_config() -application = make_app(config) diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile deleted file mode 100755 index 09d8806..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile +++ /dev/null @@ -1,95 +0,0 @@ -FROM alpine:3.7 - -# Internals, you probably don't need to change these -ENV APP_DIR=/srv/app -ENV SRC_DIR=/srv/app/src -ENV CKAN_INI=${APP_DIR}/production.ini -ENV PIP_SRC=${SRC_DIR} -ENV CKAN_STORAGE_PATH=/var/lib/ckan -ENV GIT_URL=https://github.com/ckan/ckan.git -# CKAN version to build -ENV GIT_BRANCH=ckan-2.8.9 -# Customize these on the .env file if needed -ENV CKAN_SITE_URL=http://localhost:5000 -ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars - -WORKDIR ${APP_DIR} - -# Install necessary packages to run CKAN -RUN apk add --no-cache tzdata \ - git \ - gettext \ - postgresql-client \ - python \ - apache2-utils \ - libxml2 \ - libxslt \ - musl-dev \ - uwsgi \ - uwsgi-http \ - uwsgi-corerouter \ - uwsgi-python \ - py2-gevent \ - uwsgi-gevent \ - libmagic \ - curl \ - sudo && \ - # Packages to build CKAN requirements and plugins - apk add --no-cache --virtual .build-deps \ - postgresql-dev \ - gcc \ - make \ - g++ \ - autoconf \ - automake \ - libtool \ - python-dev \ - libxml2-dev \ - libxslt-dev \ - linux-headers - # Create SRC_DIR - RUN mkdir -p ${SRC_DIR} && \ - # Install pip, supervisord and uwsgi - curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \ - python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \ - pip install supervisor && \ - mkdir /etc/supervisord.d && \ - #pip wheel --wheel-dir=/wheels uwsgi gevent && \ - rm -rf ${SRC_DIR}/get-pip.py - -COPY 2.8/setup/supervisord.conf /etc - -# Install CKAN -RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \ - cd ${SRC_DIR}/ckan && \ - cp who.ini ${APP_DIR} && \ - pip install --no-binary :all: -r requirements.txt && \ - # Install CKAN envvars to support loading config from environment variables - pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \ - # Create and update CKAN config - paster --plugin=ckan make-config ckan ${CKAN_INI} && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" - -# Create a local user and group to run the app -RUN addgroup -g 92 -S ckan && \ - adduser -u 92 -h /srv/app -H -D -S -G ckan ckan - -# Create local storage folder -RUN mkdir -p $CKAN_STORAGE_PATH && \ - chown -R ckan:ckan $CKAN_STORAGE_PATH - -# Copy setup files -COPY 2.8/setup ${APP_DIR} -RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf - -# Create entrypoint directory for children image scripts -ONBUILD RUN mkdir /docker-entrypoint.d - -RUN chown ckan -R /srv/app - -EXPOSE 5000 - -HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1 - -CMD ["/srv/app/start_ckan.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini deleted file mode 100644 index 2361f36..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini +++ /dev/null @@ -1,15 +0,0 @@ -[uwsgi] -http-socket = :5000 -uid = ckan -guid = ckan -plugins = python3 -wsgi-file = /srv/app/wsgi.py -virtualenv = /srv/app -module = wsgi:application -master = true -processes = 5 -pidfile = /tmp/%n.pid -harakiri = 50 -max-requests = 5000 -vacuum = true -callable = application diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py deleted file mode 100755 index 7ebacfa..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py +++ /dev/null @@ -1,197 +0,0 @@ -import os -import sys -import subprocess -import psycopg2 -import urllib2 -import time -import re - -ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini') - -RETRY = 5 - -def update_plugins(): - - plugins = os.environ.get('CKAN__PLUGINS', '') - print('[prerun] Setting the following plugins in {}:'.format(ckan_ini)) - print(plugins) - cmd = ['paster', '--plugin=ckan', 'config-tool', - ckan_ini, 'ckan.plugins = {}'.format(plugins)] - subprocess.check_output(cmd, stderr=subprocess.STDOUT) - print '[prerun] Plugins set.' - - -def check_main_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_SQLALCHEMY_URL') - if not conn_str: - print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_datastore_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_db_connection(conn_str, retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - try: - connection = psycopg2.connect(conn_str) - - except psycopg2.Error as e: - print str(e) - print '[prerun] Unable to connect to the database, waiting...' - time.sleep(10) - check_db_connection(conn_str, retry=retry - 1) - else: - connection.close() - - -def check_solr_connection(retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - url = os.environ.get('CKAN_SOLR_URL', '') - search_url = '{url}/select/?q=*&wt=json'.format(url=url) - - try: - connection = urllib2.urlopen(search_url) - except urllib2.URLError as e: - print str(e) - print '[prerun] Unable to connect to solr, waiting...' - time.sleep(10) - check_solr_connection(retry=retry - 1) - else: - eval(connection.read()) - - -def init_db(): - - db_command = ['paster', '--plugin=ckan', 'db', - 'init', '-c', ckan_ini] - print '[prerun] Initializing or upgrading db - start' - try: - subprocess.check_output(db_command, stderr=subprocess.STDOUT) - print '[prerun] Initializing or upgrading db - end' - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - - -def init_datastore_db(): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] Skipping datastore initialization' - return - - datastore_perms_command = ['paster', '--plugin=ckan', 'datastore', - 'set-permissions', '-c', ckan_ini] - - connection = psycopg2.connect(conn_str) - cursor = connection.cursor() - - print '[prerun] Initializing datastore db - start' - try: - datastore_perms = subprocess.Popen( - datastore_perms_command, - stdout=subprocess.PIPE) - - perms_sql = datastore_perms.stdout.read() - # Remove internal pg command as psycopg2 does not like it - perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql) - cursor.execute(perms_sql) - for notice in connection.notices: - print notice - - connection.commit() - - print '[prerun] Initializing datastore db - end' - print datastore_perms.stdout.read() - except psycopg2.Error as e: - print '[prerun] Could not initialize datastore' - print str(e) - - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - finally: - cursor.close() - connection.close() - - -def create_sysadmin(): - - name = os.environ.get('CKAN_SYSADMIN_NAME') - password = os.environ.get('CKAN_SYSADMIN_PASSWORD') - email = os.environ.get('CKAN_SYSADMIN_EMAIL') - - if name and password and email: - - # Check if user exists - command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini] - - out = subprocess.check_output(command) - if 'User:None' not in re.sub(r'\s', '', out): - print '[prerun] Sysadmin user exists, skipping creation' - return - - # Create user - command = ['paster', '--plugin=ckan', 'user', 'add', - name, - 'password=' + password, - 'email=' + email, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Created user {0}'.format(name) - - # Make it sysadmin - command = ['paster', '--plugin=ckan', 'sysadmin', 'add', - name, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Made user {0} a sysadmin'.format(name) - - -if __name__ == '__main__': - - maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true' - - if maintenance: - print '[prerun] Maintenance mode, skipping setup...' - else: - check_main_db_connection() - init_db() - update_plugins() - check_datastore_db_connection() - init_datastore_db() - check_solr_connection() - create_sysadmin() \ No newline at end of file diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh deleted file mode 100755 index d1c6726..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Set the common uwsgi options -UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L" - -# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully -if [ $? -eq 0 ] -then - if [ "$PASSWORD_PROTECT" = true ] - then - if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ] - then - # Generate htpasswd file for basicauth - htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi with basicauth - sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS - else - echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..." - exit 1 - fi - else - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi - sudo -u ckan -EH uwsgi $UWSGI_OPTS - fi -else - echo "[prerun] failed...not starting CKAN." -fi diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf deleted file mode 100755 index a3f6671..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf +++ /dev/null @@ -1,23 +0,0 @@ -[unix_http_server] -file = /tmp/supervisor.sock -chmod = 0777 -chown = nobody:nogroup - -[supervisord] -logfile = /tmp/supervisord.log -logfile_maxbytes = 50MB -logfile_backups=10 -loglevel = info -pidfile = /tmp/supervisord.pid -nodaemon = true -umask = 022 -identifier = supervisor - -[supervisorctl] -serverurl = unix:///tmp/supervisor.sock - -[rpcinterface:supervisor] -supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface - -[include] -files = /etc/supervisord.d/*.conf diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf deleted file mode 100755 index 6321d6d..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf +++ /dev/null @@ -1,2 +0,0 @@ -[uwsgi] -route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py deleted file mode 100644 index b37d80e..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py +++ /dev/null @@ -1,9 +0,0 @@ -import os -from ckan.config.middleware import make_app -from ckan.cli import CKANConfigLoader -from logging.config import fileConfig as loggingFileConfig -config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini') -abspath = os.path.join(os.path.dirname(os.path.abspath(__file__))) -loggingFileConfig(config_filepath) -config = CKANConfigLoader(config_filepath).get_config() -application = make_app(config) diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev b/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev deleted file mode 100644 index 6086f6e..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev +++ /dev/null @@ -1,37 +0,0 @@ -FROM ckan/ckan-dev:testing-only.2.9 - -LABEL maintainer="brett@kowh.ai" - -# Set up environment variables -ENV APP_DIR=/srv/app -ENV TZ=UTC -RUN echo ${TZ} > /etc/timezone - -# Make sure both files are not exactly the same -RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \ - cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\ - fi ; - -# Install any extensions needed by your CKAN instance -# (Make sure to add the plugins to CKAN__PLUGINS in the .env file) -# For instance: -#RUN pip install -e git+https://github.com/ckan/ckanext-pages.git#egg=ckanext-pages && \ -# pip install -e git+https://github.com/ckan/ckanext-dcat.git@v0.0.6#egg=ckanext-dcat && \ -# pip install -r https://raw.githubusercontent.com/ckan/ckanext-dcat/v0.0.6/requirements.txt - -# Clone the extension(s) your are writing for your own project in the `src` folder -# to get them mounted in this image at runtime - -# Apply any patches needed to CKAN core or any of the built extensions (not the -# runtime mounted ones) -# See https://github.com/okfn/docker-ckan#applying-patches - -COPY patches ${APP_DIR}/patches - -RUN for d in ${APP_DIR}/patches/*; do \ - if [ -d $d ]; then \ - for f in `ls $d/*.patch | sort -g`; do \ - cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \ - done ; \ - fi ; \ - done diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile deleted file mode 100755 index a421a59..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.7 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Install CKAN dev requirements -RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile deleted file mode 100755 index 6142a8d..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.8 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Install CKAN dev requirements -RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile deleted file mode 100755 index fc170ee..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile +++ /dev/null @@ -1,29 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.9 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Set up Python3 virtual environment -RUN cd ${APP_DIR} && \ - source ${APP_DIR}/bin/activate - -# Virtual environment binaries/scripts to be used first -ENV PATH=${APP_DIR}/bin:${PATH} - -# Install CKAN dev requirements -# Will need to change this eventually - when CKAN 2.9 is out -# wget https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt -# RUN pip3 install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt -RUN pip3 install -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh deleted file mode 100755 index 331266c..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh +++ /dev/null @@ -1,82 +0,0 @@ -#!/bin/bash - -# Install any local extensions in the src_extensions volume -echo "Looking for local extensions to install..." -echo "Extension dir contents:" -ls -la $SRC_EXTENSIONS_DIR -for i in $SRC_EXTENSIONS_DIR/* -do - if [ -d $i ]; - then - - if [ -f $i/pip-requirements.txt ]; - then - pip install -r $i/pip-requirements.txt - echo "Found requirements file in $i" - fi - if [ -f $i/requirements.txt ]; - then - pip install -r $i/requirements.txt - echo "Found requirements file in $i" - fi - if [ -f $i/dev-requirements.txt ]; - then - pip install -r $i/dev-requirements.txt - echo "Found dev-requirements file in $i" - fi - if [ -f $i/setup.py ]; - then - cd $i - python $i/setup.py develop - echo "Found setup.py file in $i" - cd $APP_DIR - fi - - # Point `use` in test.ini to location of `test-core.ini` - if [ -f $i/test.ini ]; - then - echo "Updating \`test.ini\` reference to \`test-core.ini\` for plugin $i" - paster --plugin=ckan config-tool $i/test.ini "use = config:../../src/ckan/test-core.ini" - fi - fi -done - -# Set debug to true -echo "Enabling debug mode" -ckan config-tool $CKAN_INI -s DEFAULT "debug = true" - -# Update the plugins setting in the ini file with the values defined in the env var -echo "Loading the following plugins: $CKAN__PLUGINS" -ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS" - -# Update test-core.ini DB, SOLR & Redis settings -echo "Loading test settings into test-core.ini" -ckan config-tool $SRC_DIR/ckan/test-core.ini \ - "sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \ - "ckan.datstore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \ - "ckan.datstore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \ - "solr_url = $TEST_CKAN_SOLR_URL" \ - "ckan.redis_url = $TEST_CKAN_REDIS_URL" - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Start supervisord -supervisord --configuration /etc/supervisord.conf & - -# Start the development server with automatic reload -# Check the --reloader options sudo -u ckan -EH ckan -c $CKAN_INI run --reloader -sudo -u ckan -EH ckan -c $CKAN_INI run diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt b/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt deleted file mode 100644 index 902fe5f..0000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt +++ /dev/null @@ -1,25 +0,0 @@ -# These are packages that are required by ckan developers - for running ckan in debug mode, running ckan tests, building the docs and to pip-compile the requirements.in file. - -beautifulsoup4==4.5.1 -coveralls #Let Unpinned - Requires latest coveralls -docutils==0.12 -factory-boy==2.1.1 -Flask-DebugToolbar==0.10.1 -freezegun==0.3.15 -responses==0.10.6 -mock==2.0.0 -pycodestyle==2.5.0 -pip-tools==2.0.2 -pyfakefs==3.2 -Sphinx==1.8.5 -sphinx-rtd-theme==0.3.1 -cookiecutter==1.6.0 - -# nose==1.3.0 # already in requirements.txt -pytest==4.6.5 -pytest-split-tests==1.0.9 -pytest-cov==2.7.1 -pytest-freezegun==0.4.1 -pytest-rerunfailures==8.0 - -towncrier==19.2.0