CatalogueRevamping
/
docker-ckan
14
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix caching of docker layers

This commit is contained in:
Marko Bocevski 2021-01-16 02:42:37 +01:00
parent 8940703616
commit d4a86326d8
3 changed files with 69 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
.github/workflows/master_merge.yml vendored
View File

@ -29,10 +29,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-9
/tmp/.buildx-cache-ubuntu-2-9
key: ${{ runner.os }}-buildx-2-9-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx ${{ runner.os }}-buildx-2-9
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -48,8 +50,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-9
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-9
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -65,8 +67,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-9
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-9
build-ckan-2-8: build-ckan-2-8:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -92,10 +94,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-8
/tmp/.buildx-cache-ubuntu-2-8
key: ${{ runner.os }}-buildx-2-8-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-2-8
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -111,8 +115,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-8
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-8
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -128,8 +132,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-8
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-8
build-ckan-2-7: build-ckan-2-7:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -155,10 +159,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-7
/tmp/.buildx-cache-ubuntu-2-7
key: ${{ runner.os }}-buildx-2-7-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-2-7
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -174,8 +180,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-7
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-7
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -191,8 +197,8 @@ jobs:
tags: | tags: |
keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-7
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-7
build-ckan-datapusher: build-ckan-datapusher:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -218,10 +224,10 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: /tmp/.buildx-cache-datapusher
key: ${{ runner.os }}-buildx-${{ github.sha }} key: ${{ runner.os }}-buildx-datapusher-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-datapusher
- name: Get docker tag for datapusher image - name: Get docker tag for datapusher image
id: datapusher id: datapusher
@ -237,5 +243,5 @@ jobs:
tags: | tags: |
keitaro/datapusher:${{ steps.datapusher.outputs.IMAGE_TAG }} keitaro/datapusher:${{ steps.datapusher.outputs.IMAGE_TAG }}
ghcr.io/keitaroinc/datapusher:${{ steps.datapusher.outputs.IMAGE_TAG }} ghcr.io/keitaroinc/datapusher:${{ steps.datapusher.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-datapusher
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-datapusher

58
.github/workflows/pr_checks.yml vendored
View File

@ -16,10 +16,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-9
/tmp/.buildx-cache-ubuntu-2-9
key: ${{ runner.os }}-buildx-2-9-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx ${{ runner.os }}-buildx-2-9
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -33,8 +35,8 @@ jobs:
file: ./images/ckan/2.9/Dockerfile file: ./images/ckan/2.9/Dockerfile
push: false push: false
tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-9
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-9
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -48,8 +50,8 @@ jobs:
file: ./images/ckan/2.9/Dockerfile.focal file: ./images/ckan/2.9/Dockerfile.focal
push: false push: false
tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-9
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-9
build-ckan-2-8: build-ckan-2-8:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -62,10 +64,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-8
/tmp/.buildx-cache-ubuntu-2-8
key: ${{ runner.os }}-buildx-2-8-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-2-8
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -79,8 +83,8 @@ jobs:
file: ./images/ckan/2.8/Dockerfile file: ./images/ckan/2.8/Dockerfile
push: false push: false
tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-8
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-8
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -94,8 +98,8 @@ jobs:
file: ./images/ckan/2.8/Dockerfile.focal file: ./images/ckan/2.8/Dockerfile.focal
push: false push: false
tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-8
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-8
build-ckan-2-7: build-ckan-2-7:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -108,10 +112,12 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: |
key: ${{ runner.os }}-buildx-${{ github.sha }} /tmp/.buildx-cache-alpine-2-7
/tmp/.buildx-cache-ubuntu-2-7
key: ${{ runner.os }}-buildx-2-7-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-2-7
- name: Get docker tag for Alpine image - name: Get docker tag for Alpine image
id: alpine id: alpine
@ -125,8 +131,8 @@ jobs:
file: ./images/ckan/2.7/Dockerfile file: ./images/ckan/2.7/Dockerfile
push: false push: false
tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.alpine.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-alpine-2-7
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-alpine-2-7
- name: Get docker tag for Ubuntu image - name: Get docker tag for Ubuntu image
id: ubuntu id: ubuntu
@ -140,8 +146,8 @@ jobs:
file: ./images/ckan/2.7/Dockerfile.focal file: ./images/ckan/2.7/Dockerfile.focal
push: false push: false
tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }} tags: keitaro/ckan:${{ steps.ubuntu.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-ubuntu-2-7
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-ubuntu-2-7
build-ckan-datapusher: build-ckan-datapusher:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -154,10 +160,10 @@ jobs:
- name: Cache Docker layers - name: Cache Docker layers
uses: actions/cache@v2 uses: actions/cache@v2
with: with:
path: /tmp/.buildx-cache path: /tmp/.buildx-cache-datapusher
key: ${{ runner.os }}-buildx-${{ github.sha }} key: ${{ runner.os }}-buildx-datapusher-${{ github.sha }}
restore-keys: | restore-keys: |
${{ runner.os }}-buildx- ${{ runner.os }}-buildx-datapusher
- name: Get docker tag for datapusher image - name: Get docker tag for datapusher image
id: datapusher id: datapusher
@ -171,5 +177,5 @@ jobs:
file: ./images/datapusher/Dockerfile file: ./images/datapusher/Dockerfile
push: false push: false
tags: keitaro/ckandatapusher:${{ steps.datapusher.outputs.IMAGE_TAG }} tags: keitaro/ckandatapusher:${{ steps.datapusher.outputs.IMAGE_TAG }}
cache-from: type=local,src=/tmp/.buildx-cache cache-from: type=local,src=/tmp/.buildx-cache-datapusher
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-datapusher

6
Readme.md
View File

@ -15,7 +15,11 @@ and Github Container Registry:
Looking to run CKAN on Kubernetes? Check out our [CKAN Helm Chart](https://github.com/keitaroinc/ckan-helm)! Looking to run CKAN on Kubernetes? Check out our [CKAN Helm Chart](https://github.com/keitaroinc/ckan-helm)!
## Overview ## Overview
All images are based on [Alpine Linux](https://alpinelinux.org/) and include only required extensions to start a CKAN instance. The docker images are built using a multi-stage docker approach in order to produce slim production grade docker images with the right libraries and configuration. This multi-stage approach allows us to build python binary wheels in the build stages that later on we install in the main stage. Images are provided in two flavors:
- [Alpine Linux](https://alpinelinux.org/) based images
- [Ubuntu Focal](https://ubuntu.com/) based images are the ones ending with `-focal` in the tag name
The Docker containers include only the required extensions to start a CKAN instance. The docker images are built using a multi-stage docker approach in order to produce slim production grade docker images with the right libraries and configuration. This multi-stage approach allows us to build python binary wheels in the build stages that later on we install in the main stage.
Directory layout: Directory layout:
- [compose](./compose) - contains a docker-compose setup allowing users to spin up a CKAN setup easily using [docker-compose](https://docs.docker.com/compose/) - [compose](./compose) - contains a docker-compose setup allowing users to spin up a CKAN setup easily using [docker-compose](https://docs.docker.com/compose/)