The task
Imagine you have a bunch of middle-sized projects not big enough to have dedicated DevOps. These projects are pretty similar, for instance, they use either mysql or postgres, uwsgi, django, grunt or brunch for frontend, etc. The common mistake here is to let devs do the provisioning and deployment. You'll have site somehow provisioned with a bunch of child mistakes not connected to continuous integration thing with an outdated fabric script that does only deployment (updates to newer version) and one guy that knows (probably) how to setup everything from scratch.
The right way to do the job is to share one-two DevOps between projects. But in order to support 10-20 projects they need to be really effective. Fortunately there is such a thing as
Salt Formulas. For instance, if all or several projects use uwsgi one can write uwsgi formula and reuse it (this is similar to code reuse: minimizes time to write and support the code).
Thus we end up with something like this. Salt states in XXX.sls:
include:
- deploy_key
- django
- django.collectstatic
- django.compilemessages
- pil
- django.sorl
- mysql.client
- nginx
- uwsgi
- time
- compass
- npm
- grunt
- grunt.bower
And
salt pillar that configures formulas:
username: XXX
django:
git_url: ssh://git@XXX
settings: XXX.settings
chdir: /home/XXX/XXX/
requirements: /home/XXX/XXX/requirements.txt
nginx:
server_name: 'XXX.com *.XXX.com'
auth_basic_user_file: /home/XXX/htpasswd
uwsgi:
chdir: /home/XXX/XXX/
env: DJANGO_SETTINGS_MODULE=XXX.settings
compass:
compass_version: 1.0.0.alpha.19
sass_version: 3.3.6
npm:
chdir: /home/XXX/XXX/frontend
grunt:
chdir: /home/XXX/XXX/frontend
And this is almost enough to automatically deploy project XXX with SaltStack having all those formulas written beforehand. Adding a new project means gathering (with include) and configuring formulas and sometimes write new ones.
But here comes the problem. Some formulas depend on other formulas and one needs to enforce the right order of execution. For instance, uwsgi service needs to watch changes in source, thus one need to append to XXX.sls something like
extend:
uwsgi_service:
service:
- watch:
- git: source
Cons. First of all you'll need to write this in every project that uses uwsgi formula. Secondly this breaks encapsulation: this information belongs to the formula itself and may not be exposed, in case something changes in the formula you'll need to change/test ALL projects's using it.
The solution
We'll use White-board pattern that is common to devs working in OSGi/Spring world. See
OSGI White-Board pattern with a sample. Imagine you need some job A to be done, you do not know who can do this and you simply put an advertisement on whiteboard (service registry) that you need someone to do this job A (need some service that implements interface A). From the other side someone who is looking for the job and can do A, B and C (implements interfaces A, B and C) puts an advertisement on whiteboard what he cans. OSGi does the rest. With SaltStack one can do the same. In our example uwsgi formula knows that it needs source code to be present and restart upon its changes. Thus the formula declares this dependency, excerpt from uwsgi/init.sls:
uwsgi_service:
service.running:
- name: {{ uwsgi.service }}
- watch:
- file: {{ uwsgi.ini }}
- git: source
- require:
- file: /etc/init/{{ uwsgi.service }}.conf
- pip: uwsgi
Uwsgi formula doesn't know who provides the sources. One of formulas provides them, in our case this is django formula, excerpt from django/init.sls:
source:
git.latest:
- name: {{ django.git_url }}
- rev: {{ django.git_rev }}
- target: {{ django.target }}
- force: true
- user: {{ django.username }}
- submodules: true
- force_checkout: true
- force_reset: true
- require:
- pkg: django_pkgs
Some other project can use uwsgi formula without django formula but with some other formula that implements 'git: source' state. In this way one can drastically decrease amount of explicit dependencies. But of course beware making a lot of contracts between formulas, this will complicate their standalone usage and influence resulting flexibility. Thus we'll keep some deps in XXX.sls that are unique to this project.
The resulting XXX.sls:
include:
- deploy_key
- django
- django.collectstatic
- django.compilemessages
- pil
- django.sorl
- mysql.client
- nginx
- uwsgi
- time
- compass
- npm
- grunt
- grunt.bower
extend:
django_prerequisites:
cmd:
- watch:
- file: local_settings.py
- require:
- cmd: mysql_db
uwsgi_service:
service:
- watch:
- file: local_settings.py
collectstatic:
cmd:
- require:
- cmd: grunt
grunt:
cmd:
- require:
- gem: compass
{% from "django/map.jinja" import django with context %}
local_settings.py:
file.managed:
- name: {{ django.chdir }}/XXX/local_settings.py
- source: salt://XXX/local_settings.py
- mode: 644
- user: {{ django.username }}
- group: {{ django.username }}
mysql_db:
cmd.run:
- name: {{ django.virtualenv }}/bin/python manage.py syncdb --noinput --settings={{ django.settings }} && {{ django.virtualenv }}/bin/python manage.py migrate --delete-ghost-migrations --settings={{ django.settings }}
- user: {{ django.username }}
- cwd: {{ django.chdir }}
- require:
- virtualenv: virtualenv
- file: local_settings.py
- git: source
htpasswd:
file.managed:
- name: {{ django.home }}/htpasswd
- source: salt://XXX/htpasswd
- mode: 644
- user: {{ django.username }}
- group: {{ django.username }}
And yes, you'll need only 63 lines of code (+26 lines of pillar configuration) to deploy django, pil, solr, mysql, nginx, uwsgi, compile frontend with grunt/compass, collect static files, compile messages for translations, etc.
