Skip to content
Snippets Groups Projects
Commit 14fb4bc8 authored by Hinrikus Wolf's avatar Hinrikus Wolf
Browse files

Merge branch 'unify_uwsgi_python' into 'master'

Unify uwsgi python

See merge request !3
parents dcb2d1c0 6f58dfaf
No related branches found
No related tags found
1 merge request!3Unify uwsgi python
Showing
with 91 additions and 587 deletions
# Webservices
## Roles
- `acmetool`, manages let's encrypt certificates via acmetool
- `debian-repository`, installs the software reprepro (very basic)
- `lehrpreis`, teaching award nomination web app
- `meckerkasten`, mimimi web app
- `mediawiki`, setups mediawiki with php-fpm
- `php-fpm`, setups php-fpm pools
- `protokollsystem`, protocols, meetings, and everything management web app (TM)
- `schildergenerator`, sign generator web app
- `sentry`, error collection api and web app
- `sso`, single signon web app
- `uwsgi-php`, deprecated
- `uwsgi-python`, unified web app deployment role for python-based web apps
- `wahlhelfer`, setups wahlhelfer registration and scheduling web app
- `wahlsystem`, setups an election management system
- `webserver`, configures nginx
- `wordpress`, setups wordpresses via an overlayfs technology
*Note:* Some roles are still undocumented.
## uwsgi-python
The idea of the role is to have only one role for all the python-based web apps. Instead of a role (and group) for every single app.
We distinguish between apps and instances. An app is the actual program, which can be instaciated more than once, which we call the instance.
The role iterates over the list `webapps` (in the host vars) and import app and instance specific variable files.
An app specific task snippet is imported if it configured in the variable files (see below).
We place the instance specific variable file under <inventory_dir>/vars/<instance>.yml. The app specific variable file is <role_dir>/vars/<app>.yml.
In the latter file there should be sensible defaults for _every_ variable used within the role.
This role needs the patch from this [pull-request](https://github.com/ansible/ansible/pull/39762) or ansible 2.6 or later.
### Variables
- `app_name`: the name of the app
- `app_user`: the user which runs the app---will be created
- `app_group`: the user's group which runs the app---will be created
- `app_home`: home directory of the user, usually `/var/www/<app_user>`
- `app_path`: the path where the program can be found, usually same as `app_home`
- `app_python_version`: the app's python version as number (2|3)
- `app_venv`: the absolut path to the virtualenv, which shall be used for the app
- `app_program`: the python executable, which lies in `app_path`
- `app_callable`: uwsgi variable <callable>, mostly app
- `app_command`: commandline arguments for the app
- `app_mountpoint`: the path in the URI, if the app shall be reachable under https://www.example.com/app, usually /
- `app_db_name`: name of the db for the app, can be `""` if not any database is needed, usally same as `app_user`
- `app_db_type`: db type: (postgres|mysql|sqlite)
- `app_additional_software`: list of software, which is additionally needed by the app e.g. LaTeX. If no additional software is needed it is `[]`
- `app_deploy_key`: path to a ssh-key which is needed to get the software usaally, "{{ inventory_dir }}/files/deploy-keys/<app_name>"
- `app_git_url`: git url of the project
- `app_git_version`: git version usually `HEAD`
- `app_config_file`: config file, usally `config.py`
- `app_secret_config`: it might be useful to have the cookie's key in seperate config (true|false)
### How to create a new app
Create the following files
* `uwsgi-python/vars/<app_name>.yml`
Define the variable in it from above and the variables, you need for you config file. We recommend to define them as defaults (and as documentation)
* `uwsgi-python/templates/apps/<app_name>.j2`
That's your config template.
* `uwsgi-python/tasks/apps/<app_name>.j2`
If you need some additional tasks, you can define them in this file
If you have done this, you can create an instance of the app. Note, you have to define for every app an instance altough you only want to deploy it once.
### How to create a new app's instance
Create for every instance <inventory_dir>/vars/<instance_name>.yml a variable in which you can override the defaults from uwsgi-python/vars/<app_name>.yml
Define in your host_vars/<host> the dictionary `webapps`
webapps:
- instance: <instancen_name>
app: <app_name>
app_vars: "<instance_name>.yml" <-- path to variable file from above
! Note, it very important, to add the handler
- name: restart uwsgi instance <instance_name>
service: name="uwsgi@<instance_name>" state=restarted
---
# file: roles/lehrpreis/handlers/main.yml
- name: reload systemd service files
command: systemctl daemon-reload
- name: restart uwsgi for lehrpreis
service: name=lehrpreis state=restarted enabled=yes
- name: create tmpfiles
command: systemd-tmpfiles --create
- name: recompile translations
command: ./bin/pybabel compile -d translations
args:
chdir: "{{ lehrpreis_web_root }}"
---
# file:roles/lehrpreis/meta/main.yml
dependencies:
- { role: uwsgi-python, uwsgi_name: "lehrpreis", uwsgi_user: "{{lehrpreis_user}}", uwsgi_group: "{{lehrpreis_group}}", uwsgi_path: "{{lehrpreis_web_root}}", uwsgi_home: "{{lehrpreis_web_root}}", uwsgi_program: "lehrpreis.py", uwsgi_callable: "app", uwsgi_command: "", uwsgi_python: 3, uwsgi_db: "postgres" }
---
# file: roles/lehrpreis/tasks/main.yml
- name: ensure the deploy key is available
copy:
src: "{{ lehrpreis_deploy_key }}"
dest: /root/.ssh/lehrpreis
owner: root
group: root
mode: 0600
tags:
- lehrpreis
- webservices
# https://github.com/ansible/ansible/issues/27699
- name: ensure git module is able to clone
command: mount -o remount,exec /tmp
tags:
- lehrpreis
- webservices
- name: ensure we have the program
git:
repo: git@git.fsmpi.rwth-aachen.de:studi-systeme/lehrpreis.git
dest: "{{ lehrpreis_web_root }}"
key_file: /root/.ssh/lehrpreis
version: HEAD
notify:
- recompile translations
- restart uwsgi for lehrpreis
tags:
- lehrpreis
- webservices
- name: ensure git module is not able to clone anymore
command: mount -o remount,noexec /tmp
tags:
- lehrpreis
- webservices
- name: ensure we have a virtualenv
pip:
requirements: "{{ lehrpreis_web_root }}/requirements.txt"
virtualenv: "{{ lehrpreis_web_root }}"
virtualenv_python: python3
notify:
- restart uwsgi for lehrpreis
tags:
- lehrpreis
- webservices
- name: ensure we have our config
template:
src: config.py.j2
dest: "{{ lehrpreis_web_root }}/config.py"
owner: "{{ lehrpreis_user }}"
group: "{{ lehrpreis_group }}"
mode: 0640
notify:
- restart uwsgi for lehrpreis
tags:
- lehrpreis
- webservices
- name: ensure we have our secret config
template:
src: secret_config.py.j2
dest: "{{ lehrpreis_web_root }}/secret_config.py"
owner: "{{ lehrpreis_user }}"
group: "{{ lehrpreis_group }}"
mode: 0600
force: no
notify:
- restart uwsgi for lehrpreis
tags:
- lehrpreis
- webservices
- name: ensure we have our branded logo
copy:
src: "{{ lehrpreis_branding_logo_src }}"
dest: "{{ lehrpreis_web_root }}/static/images/{{ lehrpreis_branding_logo }}"
owner: "{{ lehrpreis_user }}"
group: "{{ lehrpreis_group }}"
mode: 0644
notify:
- restart uwsgi for lehrpreis
when: lehrpreis_branding_logo and lehrpreis_branding_logo_src
tags:
- lehrpreis
- webservices
- name: ensure the unit file exists
template:
src: lehrpreis.service.j2
dest: "/etc/systemd/system/lehrpreis.service"
owner: root
group: root
mode: 0644
notify:
- reload systemd service files
- restart uwsgi for lehrpreis
tags:
- lehrpreis
- webservices
- meta: flush_handlers
- name: ensure the service is enabled
service:
name: lehrpreis
enabled: yes
state: started
tags:
- lehrpreis
- webservices
[Unit]
Description=Lehrpreis-Nominierungs-WebApp
After=network.target
[Service]
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/apps-available/lehrpreis.ini
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target
secret_key = '{{ (2**2048)|random }}'
---
# file: meckerkasten/defaults/main.yml
meckerkasten_web_root: /var/www/meckerkasten
meckerkasten_name: meckerkasten
meckerkasten_user: meckerkasten
meckerkasten_group: meckerkasten
meckerkasten_admins: [['Admins', 'admin@example.com']]
meckerkasten_sender: 'meckerkasten@example.com'
meckerkasten_mail_host: 'mail.example.invalid'
meckerkasten_allowed_hosts: ['example.com']
meckerkasten_ldap_cert: ''
---
# file: meckerkasten/handlers/main.yml
- name: reload systemd service files
command: systemctl daemon-reload
- name: restart uwsgi for meckerkasten
service:
name: "{{ meckerkasten_name }}"
state: restarted
enabled: yes
- name: create tmpfiles
command: systemd-tmpfiles --create
---
# file: meckerkasten/meta/main.yml
dependencies:
- { role: uwsgi-python, uwsgi_name: "{{meckerkasten_name}}", uwsgi_user: "{{meckerkasten_user}}", uwsgi_group: "{{meckerkasten_group}}", uwsgi_path: "{{meckerkasten_web_root}}/program", uwsgi_home: "{{meckerkasten_web_root}}", uwsgi_program: "meckerkasten/wsgi.py", uwsgi_callable: "application", uwsgi_command: "runserver", uwsgi_db: "sqlite", uwsgi_python: 2 }
---
# file: meckerkasten/tasks/main.yml
- name: ensure the deploy key is available
copy:
src: "{{ meckerkasten_deploy_key }}"
dest: /root/.ssh/meckerkasten
owner: root
group: root
mode: 0600
tags:
- meckerkasten
- webservices
# https://github.com/ansible/ansible/issues/27699
- name: ensure fucking git module is able to clone
command: mount -o remount,exec /tmp
tags:
- meckerkasten
- webservices
- name: ensure we have the program
git:
repo: git@git.fsmpi.rwth-aachen.de:studi-systeme/meckerkasten.git
dest: "{{ meckerkasten_web_root }}"
key_file: /root/.ssh/meckerkasten
version: HEAD
notify:
- restart uwsgi for meckerkasten
tags:
- meckerkasten
- webservices
- name: ensure fucking git module is not able to clone anymore
command: mount -o remount,noexec /tmp
tags:
- meckerkasten
- webservices
- name: ensure we have a virtualenv
pip:
requirements: "{{ meckerkasten_web_root }}/requirements.txt"
virtualenv: "{{ meckerkasten_web_root }}"
virtualenv_python: python2
notify:
- restart uwsgi for meckerkasten
tags:
- meckerkasten
- webservices
- name: ensure we have our config
template:
src: settings.py.j2
dest: "{{ meckerkasten_web_root }}/meckerkasten/settings.py"
owner: "{{ meckerkasten_user }}"
group: "{{ meckerkasten_group }}"
mode: 0640
notify:
- restart uwsgi for meckerkasten
tags:
- meckerkasten
- webservices
- name: ensure the unit file exists
template:
src: meckerkasten.service.j2
dest: "/etc/systemd/system/{{ meckerkasten_name }}.service"
owner: root
group: root
mode: 0644
notify:
- reload systemd service files
- restart uwsgi for meckerkasten
tags:
- meckerkasten
- webservices
- meta: flush_handlers
- name: ensure the service is enabled
service:
name: "{{ meckerkasten_name }}"
enabled: yes
state: started
tags:
- meckerkasten
- webservices
[Unit]
Description=Meckerkasten
After=network.target
[Service]
{% if meckerkasten_ldap_cert %}
Environment=LDAPTLS_CACERT={{ meckerkasten_ldap_cert }}
{% endif %}
Environment=MECKERKASTEN_WEB_ROOT={{meckerkasten_web_root}}/
Environment=MECKERKASTEN_WEB_SUBDIR=meckerkasten
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/apps-available/{{meckerkasten_name}}.ini
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target
---
# file: protokollsystem/defaults/main.yml
protokolle_web_root: /var/www/protokollsystem
protokolle_name: protokollsystem
protokolle_user: protokolle
protokolle_group: protokolle
protokolle_celery_concurrency: 4
protokolle_ldap_cert: ''
protokolle_mail: True
protokolle_mail_from: 'sender@example.com'
protokolle_mail_host: 'mail.example.com:25'
protokolle_mail_user: ''
protokolle_mail_password: ''
protokolle_mail_tls: False # 'tls' or 'starttls'
protokolle_celery_broker: 'redis://localhost:6379/0'
protokolle_url_root: 'protokolle.example.com'
protokolle_url_proto: 'https'
protokolle_url_path: '/'
protokolle_printing: True
protokolle_printing_server: 'printsrv.example.com:631'
protokolle_printing_user: 'protokolle'
protokolle_printing_printers:
- printer: kopierer
options: ["ColorModel=Gray", "KCStaple=Center", "KCPunch=2HoleEUR", "Duplex=DuplexNoTumble"]
protokolle_etherpad: True
protokolle_etherpad_url: 'https://pad.example.com'
protokolle_wiki: True
protokolle_wiki_type: MEDIAWIKI
protokolle_wiki_api: 'https://wiki.example.com/wiki/api.php'
protokolle_wiki_anonymous: False
protokolle_wiki_user: ''
protokolle_wiki_password: ''
protokolle_wiki_domain: ''
protokolle_calendar: True
protokolle_calendar_url: 'https://user:password@groupware.example.com/SOGo/dav/user/Calendar/personal'
protokolle_admin_mail: 'admin@example.com'
protokolle_admin_group: 'protokolladmin'
protokolle_latex_local_templates: '' # local-templates
protokolle_latex_logo_template: '' # logo.tex
protokolle_latex_geometry: '' # bottom=1.6cm,top=1.6cm,inner=2.5cm,outer=1.0cm,footskip=1.0cm,headsep=0.6cm
protokolle_latex_pagestyle: '' # fancy
protokolle_latex_packages: [] # ["[absolute]{textpos}", "{fancyheadings}"]
protokolle_latex_header_footer: None # True
protokolle_logos: []
protokolle_auth_max_duration: 86400
protokolle_auth_backends:
- type: ADManager
host: auth.example.com
domain: EXAMPLE
user_dn: 'cn=users,dc=example,dc=com'
group_dn: 'dc=example,dc=com'
ca_cert: ''
obsolete: False
protokolle_auth_obsoletion_warning: ''
---
# file: protokollsystem/handlers/main.yml
- name: reload systemd service files
command: systemctl daemon-reload
- name: restart uwsgi for protokollsystem
service:
name: "{{ item }}"
state: restarted
with_items:
- "{{ protokolle_name }}"
- "{{ protokolle_name }}-celery"
- name: create tmpfiles
command: systemd-tmpfiles --create
---
# file: protokollsystem/meta/main.yml
dependencies:
- { role: texlive }
- { role: cups-client }
- { role: uwsgi-python, uwsgi_name: "{{protokolle_name}}", uwsgi_user: "{{protokolle_user}}", uwsgi_group: "{{protokolle_group}}", uwsgi_path: "{{protokolle_web_root}}", uwsgi_home: "{{protokolle_web_root}}", uwsgi_program: "server.py", uwsgi_callable: "app", uwsgi_command: "runserver", uwsgi_db: "postgres", uwsgi_python: 3, uwsgi_mules: 1, uwsgi_options: ["enable-threads"] }
---
# file: protokollsystem/tasks/main.yml
- name: ensure we have all required software and fonts
apt:
name: "{{ item }}"
state: present
with_items:
- python3-virtualenv
- virtualenv
- libxml2-dev
- libxslt-dev
- fontconfig
- tex-gyre
tags:
- protokollsystem
- webservices
- name: ensure the deploy key is available
copy:
src: "{{ protokolle_deploy_key }}"
dest: /root/.ssh/protokolle
owner: root
group: root
mode: 0600
tags:
- protokollsystem
- webservices
# https://github.com/ansible/ansible/issues/27699
- name: ensure fucking git module is able to clone
command: mount -o remount,exec /tmp
tags:
- protokollsystem
- webservices
- name: ensure we have the program
git:
repo: git@git.fsmpi.rwth-aachen.de:protokollsystem/proto3.git
dest: "{{ protokolle_web_root }}"
accept_hostkey: True # TODO remove this
key_file: /root/.ssh/protokolle
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure fucking git module is not able to clone anymore
command: mount -o remount,noexec /tmp
tags:
- protokollsystem
- webservices
- name: ensure we have a virtualenv
pip:
requirements: "{{ protokolle_web_root }}/requirements.txt"
virtualenv: "{{ protokolle_web_root }}/"
virtualenv_python: python3
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure we have our config
template:
src: config.py.j2
dest: "{{ protokolle_web_root }}/config.py"
owner: "{{ protokolle_user }}"
group: "{{ protokolle_group }}"
mode: 0640
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: check our config
command: "{{protokolle_web_root}}/bin/python {{protokolle_web_root}}/configproxy.py check --log-level warning"
args:
chdir: "{{protokolle_web_root}}"
become: yes
become_user: "{{protokolle_user}}"
changed_when: no
tags:
- protokollsystem
- webservices
- name: ensure data model upgrades are applied
command: "{{protokolle_web_root}}/bin/python {{protokolle_web_root}}/server.py db upgrade"
args:
chdir: "{{protokolle_web_root}}"
become: yes
become_user: "{{protokolle_user}}"
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure we have our local templates
copy:
src: "{{ protokolle_local_templates }}"
dest: "{{ protokolle_web_root }}/"
owner: "{{ protokolle_user }}"
group: "{{ protokolle_group }}"
mode: 0644
when: protokolle_local_templates|default('') != ''
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure one local template is the default
file:
src: "{{ protokolle_web_root }}/{{ protokolle_latex_local_templates }}/{{ protokolle_local_templates_default }}/{{ item.path }}"
dest: "{{ protokolle_web_root }}/{{ protokolle_latex_local_templates }}/{{ item.path }}"
state: link
with_filetree: "{{ protokolle_local_templates }}/{{ protokolle_local_templates_default }}"
when: protokolle_local_templates|default(False) and protokolle_local_templates_default|default(False)
notify:
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure the unit file exists
template:
src: protokollsystem.service.j2
dest: "/etc/systemd/system/{{ protokolle_name }}.service"
owner: root
group: root
mode: 0644
notify:
- reload systemd service files
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- name: ensure the celery unit file exists
template:
src: celery.service.j2
dest: "/etc/systemd/system/{{ protokolle_name }}-celery.service"
owner: root
group: root
mode: 0644
notify:
- reload systemd service files
- restart uwsgi for protokollsystem
tags:
- protokollsystem
- webservices
- meta: flush_handlers
- name: ensure the services are enabled
service:
name: "{{ item }}"
enabled: yes
state: started
with_items:
- "{{ protokolle_name }}"
- "{{ protokolle_name }}-celery"
tags:
- protokollsystem
- webservices
[Unit]
Description={{ protokolle_name }}-Celery
After=network.target
[Service]
User={{ protokolle_user }}
Group={{ protokolle_group }}
WorkingDirectory={{ protokolle_web_root }}
Environment=VIRTUAL_ENV="{{ protokolle_web_root }}"
ExecStart={{ protokolle_web_root }}/bin/celery -A server.celery worker --loglevel=DEBUG --concurrency={{ protokolle_celery_concurrency }}
Restart=always
[Install]
WantedBy=multi-user.target
---
# file: schildergenerator/defaults/main.yml
schilder_web_root: /var/www/schilder
schilder_name: schilder
schilder_user: schilder
schilder_group: schilder
schilder_printsrv: printsrv.fsmpi.rwth-aachen.de
schilder_printers:
- description: "1 - Kopierer"
name: "Kopierer"
schilder_lproptions:
- "-Fa4g"
- "-N1"
- "-o fitplot"
schilder_templates_url: ''
Flask
Flask-Genshi
Genshi
docutils
---
# file: schildergenerator/handlers/main.yml
- name: reload systemd service files
command: systemctl daemon-reload
- name: restart uwsgi for schilder
service:
name: "{{ schilder_name }}"
state: restarted
- name: create tmpfiles
command: systemd-tmpfiles --create
---
# file: schildergenerator/meta/main.yml
dependencies:
- { role: texlive }
- { role: cups-client }
- { role: uwsgi-python, uwsgi_name: "{{schilder_name}}", uwsgi_user: "{{schilder_user}}", uwsgi_group: "{{schilder_group}}", uwsgi_path: "{{schilder_web_root}}/program", uwsgi_home: "{{schilder_web_root}}", uwsgi_program: "schilder.py", uwsgi_callable: "app", uwsgi_command: "", uwsgi_db: "", uwsgi_python: 2, uwsgi_mules: 0 }
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment