tuned generation for all environments and added local-site

This commit is contained in:
dcore94 2021-05-19 17:02:57 +02:00
parent 288482d5b6
commit 676cac24ec
23 changed files with 352 additions and 165 deletions

View File

@ -1,53 +1,55 @@
# Conductor Setup
**Conductor Setup** is composed of a set of ansible roles and a playbook named site.yaml useful for deploying a docker swarm running Conductor microservice orchestrator by [Netflix OSS](https://netflix.github.io/conductor/).
**Conductor Setup** is composed of a set of ansible roles and a playbook named site-*.yaml useful for deploying a docker swarm running Conductor microservice orchestrator by [Netflix OSS](https://netflix.github.io/conductor/).
Current setup is based on Conductor 3.0.4 version adapted by Nubisware S.r.l.
It uses the docker images on dockerhub:
- nubisware/conductor-server:3.0.4
- nubisware/conductor-ui-oauth2:3.0.4 (which has been improved with Oauth2 login in collaboration with Keycloak)
Besides the basic components Conductor itself (server and ui) and Elasticsearch 6.1.8, the repository can be configured to launch postgres or mysql persistence plus basic python based workers for running PyRest, PyMail, PyExec and PyShell in the same Swarm.
In addition a nginx based PEP can be executed to protect the conductor REST API server.
It is also possible to connect to an external postgres for stateful deployments.
## Structure of the project
The AutoDynomite Docker image script file is present in `dynomite` folder.
The Docker Compose Swarm files are present in the `stack` folder.
The folder roles contains the necessary roles for configuring the different configurations
There are 4 file for deploying to local, nw-cluster or D4SCience dev, pre and prod environments.
To run a deployment
`ansible-playbook site-X.yaml`
whereas
`ansible-playbook site-X.yaml -e dry=true`
only generates the files for the stack without actually deploying it.
The folder *local-site* contains a ready version for quickly launching a conductor instance with no replications (except workers), no auth in the Conductor UI and no PEP.
`docker stack deploy -c elasticsearch-swarm.yaml -c postgres-swarm.yaml -c conductor-swarm.yaml -c conductor-workers-swarm.yaml -c pep-swarm.yaml conductor`
When you have ensured that Postgres and Elasticsearch are running, execute:
`docker stack deploy -c conductor-swarm.yaml conductor`
This will create a local stack accessible through permissive pep at port 80. Please add two mappings for localhost in your /etc/hosts
`127.0.1.1 conductor-server conductor-ui`
and point your browser to http://conductor-ui.
## Built With
* [Ansible](https://www.ansible.com)
* [Docker](https://www.docker.com)
## Documentation
The provided Docker stack files provide the following configuration:
- 2 Conductor Server nodes with 2 replicas handled by Swarm
- 2 Conductor UI nodes with 2 replicas handled by Swarm
- 1 Elasticsearch node
- 1 Database node that can be postgres (default), mysql or mariadb
- 2 Optional replicated instances of PyExec worker running the tasks Http, Eval and Shell
- 1 Optional cluster-replacement service that sets up a networking environment (including on HAProxy LB) similar to the one available in production. By default it's disabled.
The default configuration is run with the command: `ansible-playbook site.yaml`
Files for swarms and configurations will be generated inside a temporary folder named /tmp/conductor_stack on the local machine.
In order to change destination folder use the switch: `-e target_path=anotherdir`
If you only want to review the generated files run the command `ansible-playbook site.yaml -e dry=true`
In order to switch between postgres and mysql specify the db on the proper variable: `-e db=mysql`
In order to connect to an external postgres withouth pulling up a service in the stack use -e db=remote-postgres
In order to skip worker creation specify the noworker variable: `-e noworker=true`
In order to enable the cluster replacement use the switch: `-e cluster_replacement=true`
If you run the stack in production behind a load balanced setup ensure the variable cluster_check is true: `ansible-playbook site.yaml -e cluster_check=true`
In order to avoid confusion of all variables a proper site.yaml has been created for prod environment. Run `ansible-playbook site-prod.yaml` and this should deploy a stack with prod setup.
Other setting can be fine tuned by checking the variables in the proper roles which are:
- *common*: defaults and common tasks
- *conductor*: defaults, templates and tasks for generating swarm files for replicated conductor-server and ui.
- *elasticsearch*: defaults, templates and task for starting in the swarm a single instance of elasticsearch
- *mysql*: defaults, template and tasks for starting in the swarm a single instance of mysql/mariadb
- *postgres*: defaults, templates and tasks for starting in the swarm a single instance of postgres
- *workers*: defaults and task for starting in the swarm a replicated instance of the workers for executing HTTP, Shell, Eval operations.
## Examples
The following example runs as user username on the remote hosts listed in hosts a swarm with 2 replicas of conductor server and ui, 1 postgres, 1 elasticsearch, 2 replicas of simple PyExec, an HAProxy that acts as load balancer.
`ansible-playbook -u username -i hosts site.yaml -e target_path=/tmp/conductor -e cluster_replacement=true`
Checkout the files site-X.yaml as a reference for different configurations.
## Change log

View File

@ -0,0 +1,13 @@
[common]
loglevel = info
#server =
threads = 1
pollrate = 1
[pymail]
server = smtp-relay.d4science.org
user = conductor_local
password =
protocol = starttls
port = 587

View File

@ -0,0 +1,23 @@
# Servers.
conductor.grpc-server.enabled=false
# Database persistence type.
conductor.db.type=postgres
conductor.postgres.jdbcUrl=jdbc:postgresql://postgresdb:5432/conductor
conductor.postgres.jdbcUsername=conductor
conductor.postgres.jdbcPassword=password
# Hikari pool sizes are -1 by default and prevent startup
conductor.postgres.connectionPoolMaxSize=10
conductor.postgres.connectionPoolMinIdle=2
# Elastic search instance indexing is enabled.
conductor.indexing.enabled=true
conductor.elasticsearch.url=http://elasticsearch:9200
workflow.elasticsearch.instanceType=EXTERNAL
workflow.elasticsearch.index.name=conductor
# Load sample kitchen sink workflow
loadSample=false

View File

@ -0,0 +1,45 @@
version: '3.6'
services:
conductor-server:
environment:
- CONFIG_PROP=conductor-swarm-config.properties
image: "nubisware/conductor-server:3.0.4"
networks:
- conductor-network
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
delay: 5s
window: 120s
configs:
- source: swarm-config
target: /app/config/conductor-swarm-config.properties
logging:
driver: "journald"
conductor-ui:
environment:
- WF_SERVER=http://conductor-server:8080/api/
- AUTH_CONFIG_PATH=/app/config/auth.config
image: "nubisware/conductor-ui-oauth2:3.0.4"
networks:
- conductor-network
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
delay: 5s
window: 120s
networks:
conductor-network:
configs:
swarm-config:
file: ./conductor-swarm-config.properties

View File

@ -0,0 +1,29 @@
version: '3.6'
services:
base:
environment:
CONDUCTOR_SERVER: http://conductor-server:8080/api
configs:
- source: base-config
target: /app/config.cfg
image: 'nubisware/nubisware-conductor-worker-py-base'
networks:
- conductor-network
deploy:
mode: replicated
replicas: 2
restart_policy:
condition: on-failure
delay: 5s
window: 120s
logging:
driver: "journald"
networks:
conductor-network:
configs:
base-config:
file: base-config.cfg

View File

@ -0,0 +1,28 @@
version: '3.6'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.15
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- transport.host=0.0.0.0
- discovery.type=single-node
- xpack.security.enabled=false
networks:
conductor-network:
aliases:
- es
logging:
driver: "journald"
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
networks:
conductor-network:

13
local-site/nginx.conf Normal file
View File

@ -0,0 +1,13 @@
load_module modules/ngx_http_js_module.so;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

View File

@ -0,0 +1,40 @@
upstream _conductor-server {
ip_hash;
server conductor-server:8080;
}
upstream _conductor-ui {
ip_hash;
server conductor-ui:5000;
}
server {
listen *:80;
listen [::]:80;
server_name conductor-server;
location / {
proxy_pass http://_conductor-server;
}
}
server {
listen *:80 default_server;
listen [::]:80 default_server;
server_name conductor-ui;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://_conductor-ui;
}
}

30
local-site/pep-swarm.yaml Normal file
View File

@ -0,0 +1,30 @@
version: '3.6'
services:
pep:
image: nginx:stable-alpine
networks:
- conductor-network
ports:
- "80:80"
deploy:
replicas: 1
restart_policy:
condition: on-failure
delay: 10s
window: 120s
configs:
- source: nginxconf
target: /etc/nginx/templates/default.conf.template
- source: nginxbaseconf
target: /etc/nginx/nginx.conf
networks:
conductor-network:
configs:
nginxconf:
file: ./nginx.default.conf
nginxbaseconf:
file: ./nginx.conf

View File

@ -0,0 +1,16 @@
version: '3.6'
services:
postgresdb:
image: postgres
environment:
POSTGRES_USER: "conductor"
POSTGRES_PASSWORD: "password"
POSTGRES_DB: "conductor"
networks:
- conductor-network
deploy:
replicas: 1
networks:
conductor-network:

View File

@ -1,15 +1,18 @@
$ANSIBLE_VAULT;1.1;AES256
64626566636365626537356334643266623431393062653538313362663664643538373137383732
3865313539323962666165386336373633303066353634350a633231666363356238326130373561
64336633343066323464343136336333613233396164623537623762323261383537633137363234
3534616537666436370a326331316232613839656436646164343236356233646233623430623665
64663265313964653063333133326636353162353532626364316433373030396434616434333631
32333936666339303963636438616164343063393364666332323831363833323131653666303534
62376162313737303036366532316163383434333130643363613166333433616331393636613635
30613132636261613165613136356638353532663634393431383739363636323961323538383566
62316261373262663335393632376366383031306563343531643632633234346531633164303038
62633039363961613538393832623039383237623663366430313238653030376263613032663437
38386635303332386630386133366232343966393761643635313833316536386634633563326639
38626435393963643866363663353834343333346139363565353161663737393166613938353562
62623661326237353163623138386432376531353864383036613931643164333633646431353162
3666373032663262623438353236626436303132306436326636
62366130363930353837376531653565316531653234366233663032386266346338356335623537
3765393265633163396330646365393865386130393661650a666264363165656539396365643465
35313238313135363736386661633833333736396236303861383061313366613235623731356336
3634376335626138370a646666343033316165343665633338316432636562323736626466376233
64633738356663666563643465363137636261643639643035633931386631383436353936613334
64333135643036336539313164386264643737636164613462646130393730393334626335333262
30373231353061376565336366353938356338643432633664306632366436383262636333643961
62613562666463633164313235366433616134613831393436303466366236323337323635616337
34383634613736343034626330303661663662633661383734633834373464313137656461356562
37336430633865656330623863396133613636316136613133633965353932333266663532356334
35333138316339353236623963383739663730313737303838396538666338316366636537643663
35366537353736343462383734663762393433666266303963306136626631653539396632326337
39326266316532623232643437323238313765653261343630636339633936356138646262346634
63363763306533363839386364646130396534383437366631343537303165326539393639613735
39393364616361393435643531363462393633343437393936613861353266356230353338616163
37373562393362356563623966313034653138616632336264343533363165313362306330386639
32356363653031656465623463373337643930386361393839613139623530363635

View File

@ -6,6 +6,9 @@ conductor_ui_image: nubisware/conductor-ui-oauth2:3.0.4
conductor_config: conductor-swarm-config.properties
conductor_config_template: "{{ conductor_config }}.j2"
conductor_ui_clientid: "conductor-ui"
conductor_ui_public_url: "http://conductor-ui"
#nw_cluster_conductor_ui_secret: in vault
#dev_conductor_ui_secret: in vault
#pre_conductor_ui_secret: in vault

View File

@ -3,11 +3,11 @@
"strategySettings": {
"authorizationURL": "{{ iam_host }}/auth/realms/d4science/protocol/openid-connect/auth",
"tokenURL": "{{ iam_host }}/auth/realms/d4science/protocol/openid-connect/token",
"clientID": "conductor-ui",
"clientID": "{{ conductor_ui_clientid }}",
"clientSecret": "{{ conductor_ui_secret }}",
"callbackURL": "http://conductor-ui/login/callback",
"callbackURL": "{{ conductor_ui_public_url }}/login/callback",
"logoutURL": "{{ iam_host }}/auth/realms/d4science/protocol/openid-connect/logout",
"logoutCallbackURL": "http://conductor-ui/logout/callback",
"logoutCallbackURL": "{{ conductor_ui_public_url }}/logout/callback",
"roles": [ "admin", "viewer" ]
},
"cookieSecret": "{{ conductor_ui_secret }}",

View File

@ -1,2 +1,6 @@
pep_port: 80
# hostnames to be used as vhosts
conductor_server_name: conductor-server
conductor_ui_server_name: conductor-ui
#pep_credentials: in vault

View File

@ -1,18 +1,24 @@
$ANSIBLE_VAULT;1.1;AES256
64326266376663626435303764383036326164336561303030633464333131373161336461326162
3630623962383434623834313737616435613966343637390a353562636535376539353538353061
31383933313734646661633661353836386266393565633830353137646431613431663236376137
6362613731386433370a323934373363383565323337373239666434353036333435613061666231
34346134313038366165343861316233326331393732353334303039616535633866366261623764
64653630353830396665363862633730396432633062363932636335643136613237373339613139
32386330396237396363383638653431663864333162303936663563313535343536376139343166
65316137326533306335643833353338376533633733393333623131316662386334653633353332
66363734636237363637303863323638393339373364356433666466643038343930616166396136
61666232356337613431316662353766393335306232616266363933653032656536386562373665
36306234636233313237623364613033313261393431633139343037623732646431663139383062
30396230326432376335303362356534613937306431636361663335376265363139366463656638
31386430393037306233663161333465616236383134623961343732383633386665333231363036
64346630633337643961653464613336623363303737626231326138633736656530653138326537
35386161656461313034343935353863333635376664386565393530633532613965646662363634
65396137646561353534373536616162353631383130363466356637643639323333643964323638
3535
63653037396633613264356337303461626364643463616264616333313065336263626665646233
3861663135613138333863343261373464326239303835650a643535633265653339376332663462
35306231383136623339313436343732666332333435383162366135386663363063376466636233
6233353263663839310a623233353138373734356465653965376132643137643738363430333861
63336132646562343639666334616633356631366535343561646434323130633135393535383061
38313337303261396364653663316462376337393837373038623266633831303564646539326665
30303065363335346538643436613030336163336535383665623533303535623064376539363062
33393137376263383335363632633836626137346663613934346136306436353230663934633637
32356234386161393937303563343931373939623737636466363936393438353666326663373038
66343339353430393065346237626434356462653330313064303166366239343636636661633438
38613863386666343638663762303531326531633062343132663462333137373062646339623961
35666164313962356139623839323161303131306132633139303463393661636165353566373561
37333963386332386635616332326239386639636434376232356465366131306366376464366433
33323839326366653261636665623136336564373333313135313661633536333837353163373334
32366532373239303263386565363236383036623333353662303031373335653032646166386262
33656266356164666130343135386263346533393533386166306666366137313231386434343434
31653633303133323031343566663834636565313235323863353963363633346264636339653463
34353834343836306633346638313066316162373239326435313532643764306461663965303236
31386331303334636636623035303236303265633839323963633066633932336335326561623334
34366565393434393131656564646132343964653637393739613837313561646238646631316265
32303865633862386162393161336533313465326632363463653831623961633039393932623633
63613730663131343463316436326437393931343566373533666638366631333264353939343862
306362633430393061666539616565383366

View File

@ -4,7 +4,8 @@ var config = {
"pep-credentials" : "{{ pep_credentials }}",
"hosts" : [
{
"host": "conductor-server",
"host": "{{ conductor_server_name }}",
"audience" : "conductor-server",
"allow-basic-auth" : true,
"pip" : [ { claim: "context", operator : "get-contexts" } ],
"paths" : [

View File

@ -19,11 +19,14 @@ server {
listen *:80;
listen [::]:80;
server_name conductor-server;
server_name {{ conductor_server_name }};
{% if conductor_server_name != conductor_ui_server_name %}
# When there is the possibility to separate vhosts for ui and apis as in local-site deployment forward also / to swagger docs
location / {
proxy_pass http://_conductor-server;
}
{% endif %}
location /api/ {
js_content pep.enforce;
@ -80,7 +83,7 @@ server {
listen *:80 default_server;
listen [::]:80 default_server;
server_name conductor-ui;
server_name {{ conductor_ui_server_name }};
location / {
proxy_set_header Host $host;

View File

@ -15,6 +15,10 @@ services:
{% endif %}
deploy:
replicas: 1
{% if infrastructure != 'local' %}
placement:
constraints: [node.role != worker]
{% endif %}
restart_policy:
condition: on-failure
delay: 10s

86
run.sh
View File

@ -1,86 +0,0 @@
#!/bin/bash
#
# The "directory/directory.yml" is the old way that we used to simplify jobs execution.
# The "directory/site.yml" is the syntax used by roles (from ansible version 1.2)
#
# Otherwise we can directly execute a single play (file)
#
PAR=50
TIMEOUT=15
PLAY=site.yml
HOSTS_DIR=.
ANSIBLE_HOSTS=
export TMPDIR=/var/tmp/${USER}
if [ ! -d ${TMPDIR} ] ; then
mkdir -p ${TMPDIR}
fi
if [ -f ./ansible.cfg ] ; then
export ANSIBLE_CONFIG="./ansible.cfg"
fi
# No cows!
export ANSIBLE_NOCOWS=1
export ANSIBLE_ERROR_ON_UNDEFINED_VARS=True
export ANSIBLE_HOST_KEY_CHECKING=False
export ANSIBLE_LIBRARY="/usr/share/ansible:./modules:../modules:$ANSIBLE_LIBRARY"
# Update the galaxy requirements
if [ -f requirements.yml ] ; then
ansible-galaxy install --ignore-errors -f -r requirements.yml
fi
PLAY_OPTS="-T $TIMEOUT -f $PAR"
if [ -f "$1" ] ; then
PLAY=$1
elif [ ! -f $PLAY ] ; then
echo "No play file available."
exit 1
fi
if [ -f "${PLAY}" ] ; then
MAIN="${PLAY}"
shift
elif [ -f "${PLAY}.yml" ]; then
MAIN="${PLAY}.yml"
shift
fi
if [ -f ${HOSTS_DIR}/hosts ] ; then
ANSIBLE_HOSTS=${HOSTS_DIR}/hosts
fi
if [ -f ${HOSTS_DIR}/inventory/hosts ] ; then
ANSIBLE_HOSTS=${HOSTS_DIR}/inventory/hosts
fi
if [ ! -z "$ANSIBLE_HOSTS" ] ; then
PLAY_OPTS="-i $ANSIBLE_HOSTS"
fi
#echo "Find vault encrypted files if any"
#if [ -d ./group_vars ] ; then
# VAULT_GROUP_FILES=$( find ./group_vars -name \*vault\* )
#fi
#if [ -d ./host_vars ] ; then
# VAULT_HOST_FILES=$( find ./host_vars -name \*vault\* )
#fi
#if [ -n "$VAULT_GROUP_FILES" ] || [ -n "$VAULT_HOST_FILES" ] ; then
# Vault requires a password.
# To encrypt a password for a user: python -c "from passlib.hash import sha512_crypt; print sha512_crypt.encrypt('<password>')"
# if [ -f ~/.conductor_ansible_vault_pass.txt ] ; then
# PLAY_OPTS="$PLAY_OPTS --vault-password-file=~/.conductor_ansible_vault_pass.txt"
# else
# echo "There are password protected encrypted files, we will ask for password before proceeding"
# PLAY_OPTS="$PLAY_OPTS --ask-vault-pass"
# fi
#fi
PLAY_OPTS="$PLAY_OPTS --ask-vault-pass"
# Main
ansible-playbook $PLAY_OPTS $MAIN $@
rm -f /tmp/passwordfile

View File

@ -12,8 +12,12 @@
pep_credentials: "{{ dev_pep_credentials }}"
conductor_ui_secret: "{{ dev_conductor_ui_secret }}"
conductor_auth: oauth2
conductor_server_name: conductor.dev.d4science.org
conductor_ui_server_name: conductor-ui.dev.d4science.org
conductor_ui_public_url: "http://{{ conductor_ui_server_name }}"
conductor_replicas: 2
conductor_ui_replicas: 2
roles:
- common
- databases

View File

@ -1,6 +1,5 @@
---
- hosts: localhost
#- hosts: nw_cluster_infra
- hosts: nw_cluster_infra
vars_files:
- roles/workers/defaults/smtp.yaml
- roles/pep/defaults/pep_credentials.yaml
@ -13,6 +12,9 @@
pep_credentials: "{{ nw_cluster_pep_credentials }}"
conductor_ui_secret: "{{ nw_cluster_conductor_ui_secret }}"
conductor_auth: oauth2
conductor_server_name: conductor-dev.int.d4science.net
conductor_ui_server_name: conductorui-dev.int.d4science.net
conductor_ui_public_url: "http://{{ conductor_ui_server_name }}"
conductor_replicas: 2
conductor_ui_replicas: 2
roles:

View File

@ -1,6 +1,5 @@
---
- hosts: localhost
#- hosts: pre_infra
- hosts: pre_infra
vars_files:
- roles/workers/defaults/smtp.yaml
- roles/pep/defaults/pep_credentials.yaml
@ -13,6 +12,9 @@
pep_credentials: "{{ pre_pep_credentials }}"
conductor_ui_secret: "{{ pre_conductor_ui_secret }}"
conductor_auth: oauth2
conductor_server_name: conductor.pre.d4science.org
conductor_ui_server_name: conductor-ui.pre.d4science.org
conductor_ui_public_url: "http://{{ conductor_ui_server_name }}"
conductor_replicas: 2
conductor_ui_replicas: 2
roles:

View File

@ -1,6 +1,5 @@
---
- hosts: localhost
#- hosts: prod_infra
- hosts: prod_infra
vars_files:
- roles/workers/defaults/smtp.yaml
- roles/pep/defaults/pep_credentials.yaml
@ -13,6 +12,9 @@
pep_credentials: "{{ prod_pep_credentials }}"
conductor_ui_secret: "{{ prod_conductor_ui_secret }}"
conductor_auth: oauth2
conductor_server_name: conductor.d4science.org
conductor_ui_server_name: conductor-ui.d4science.org
conductor_ui_public_url: "http://{{ conductor_ui_server_name }}"
conductor_replicas: 2
conductor_ui_replicas: 2
roles: