Wednesday, August 27, 2014

Multisite with SaltStack

You have a site being deployed by SaltStack but you need to add multisiteness to it. This can mean you need to run multiple uwsgi instances (one per domain) which run wsgi app with different settings, have multiple nginx configurations, different users (one per domain) etc, etc. The problem is you cannot simply include nginx or uwsgi formula twice with different parameters just because salt doesn't support it. One has following options:
  1. Use loops in jinja templates. But every formula needs to support it, pillar data is complicated is this case
  2. Use multiple salt-minion instances (one per domain), see minionswarm.py on how to implement this. Most appealing if you need to setup on one node unrelated components, for instance, web service and database, minion in this case corresponds to the component but not to the node
  3. Use grains in pillar templates to change pillar data according to grains data, see Role-based infrastructure. Best for multisites because they differ a little
Pillar data example:
{% set site_id = grains['site_id'] %}

username: XXX
site_id: {{ site_id }}

django:
  git_url: git@XXX.git
  settings: settings_{{ site_id }}
  chdir: /home/XXX/XXX
  media_root: /home/XXX/uploads

nginx:
  server_name: {{ site_id }}.com
  media_root: /home/XXX/uploads
  media_location: /uploads

uwsgi:
  chdir: /home/XXX/XXX
  env: DJANGO_SETTINGS_MODULE=settings_{{ site_id }}
  harakiri: 90
  pythonpath: /home/XXX/XXX
And salt states doesn't change. From jenkins one can run
salt XXX grains.setval site_id site1
salt XXX state.highstate pillar="{git_rev: '$GIT_COMMIT'}"
salt XXX grains.setval site_id site2
salt XXX state.highstate pillar="{git_rev: '$GIT_COMMIT'}"
salt XXX file.blockreplace /home/XXX/static/version.txt "#-- jenkins info below this line ---" "#-- jenkins info above this line ---" "BUILD_NUMBER $BUILD_NUMBER BUILD_ID $BUILD_ID" True

Whiteboard pattern in SaltStack

The task
Imagine you have a bunch of middle-sized projects not big enough to have dedicated DevOps. These projects are pretty similar, for instance, they use either mysql or postgres, uwsgi, django, grunt or brunch for frontend, etc. The common mistake here is to let devs do the provisioning and deployment. You'll have site somehow provisioned with a bunch of child mistakes not connected to continuous integration thing with an outdated fabric script that does only deployment (updates to newer version) and one guy that knows (probably) how to setup everything from scratch.
The right way to do the job is to share one-two DevOps between projects. But in order to support 10-20 projects they need to be really effective. Fortunately there is such a thing as Salt Formulas. For instance, if all or several projects use uwsgi one can write uwsgi formula and reuse it (this is similar to code reuse: minimizes time to write and support the code).
Thus we end up with something like this. Salt states in XXX.sls:
include:
  - deploy_key
  - django
  - django.collectstatic
  - django.compilemessages
  - pil
  - django.sorl
  - mysql.client
  - nginx
  - uwsgi
  - time
  - compass
  - npm
  - grunt
  - grunt.bower
And salt pillar that configures formulas:
username: XXX

django:
  git_url: ssh://git@XXX
  settings: XXX.settings
  chdir: /home/XXX/XXX/
  requirements: /home/XXX/XXX/requirements.txt

nginx:
  server_name: 'XXX.com *.XXX.com'
  auth_basic_user_file: /home/XXX/htpasswd

uwsgi:
  chdir: /home/XXX/XXX/
  env: DJANGO_SETTINGS_MODULE=XXX.settings

compass:
  compass_version: 1.0.0.alpha.19
  sass_version: 3.3.6

npm:
  chdir: /home/XXX/XXX/frontend

grunt:
  chdir: /home/XXX/XXX/frontend
And this is almost enough to automatically deploy project XXX with SaltStack having all those formulas written beforehand. Adding a new project means gathering (with include) and configuring formulas and sometimes write new ones.
But here comes the problem. Some formulas depend on other formulas and one needs to enforce the right order of execution. For instance, uwsgi service needs to watch changes in source, thus one need to append to XXX.sls something like
extend:
  uwsgi_service:
    service:
      - watch:
        - git: source
Cons. First of all you'll need to write this in every project that uses uwsgi formula. Secondly this breaks encapsulation: this information belongs to the formula itself and may not be exposed, in case something changes in the formula you'll need to change/test ALL projects's using it.

The solution
We'll use White-board pattern that is common to devs working in OSGi/Spring world. See OSGI White-Board pattern with a sample. Imagine you need some job A to be done, you do not know who can do this and you simply put an advertisement on whiteboard (service registry) that you need someone to do this job A (need some service that implements interface A). From the other side someone who is looking for the job and can do A, B and C (implements interfaces A, B and C) puts an advertisement on whiteboard what he cans. OSGi does the rest. With SaltStack one can do the same. In our example uwsgi formula knows that it needs source code to be present and restart upon its changes. Thus the formula declares this dependency, excerpt from uwsgi/init.sls:
uwsgi_service:
  service.running:
    - name: {{ uwsgi.service }}
    - watch:
      - file: {{ uwsgi.ini }}
      - git: source
    - require:
      - file: /etc/init/{{ uwsgi.service }}.conf
      - pip: uwsgi
Uwsgi formula doesn't know who provides the sources. One of formulas provides them, in our case this is django formula, excerpt from django/init.sls:
source:
  git.latest:
    - name: {{ django.git_url }}
    - rev: {{ django.git_rev }}
    - target: {{ django.target }}
    - force: true
    - user: {{ django.username }}
    - submodules: true
    - force_checkout: true
    - force_reset: true
    - require:
      - pkg: django_pkgs
Some other project can use uwsgi formula without django formula but with some other formula that implements 'git: source' state. In this way one can drastically decrease amount of explicit dependencies. But of course beware making a lot of contracts between formulas, this will complicate their standalone usage and influence resulting flexibility. Thus we'll keep some deps in XXX.sls that are unique to this project.
The resulting XXX.sls:
include:
  - deploy_key
  - django
  - django.collectstatic
  - django.compilemessages
  - pil
  - django.sorl
  - mysql.client
  - nginx
  - uwsgi
  - time
  - compass
  - npm
  - grunt
  - grunt.bower

extend:
  django_prerequisites:
    cmd:
      - watch:
        - file: local_settings.py
      - require:
        - cmd: mysql_db
  uwsgi_service:
    service:
      - watch:
        - file: local_settings.py
  collectstatic:
    cmd:
      - require:
        - cmd: grunt
  grunt:
    cmd:
      - require:
        - gem: compass

{% from "django/map.jinja" import django with context %}

local_settings.py:
  file.managed:
    - name: {{ django.chdir }}/XXX/local_settings.py
    - source: salt://XXX/local_settings.py
    - mode: 644
    - user: {{ django.username }}
    - group: {{ django.username }}

mysql_db:
  cmd.run:
    - name: {{ django.virtualenv }}/bin/python manage.py syncdb --noinput --settings={{ django.settings }} && {{ django.virtualenv }}/bin/python manage.py migrate --delete-ghost-migrations --settings={{ django.settings }}
    - user: {{ django.username }}
    - cwd: {{ django.chdir }}
    - require:
      - virtualenv: virtualenv
      - file: local_settings.py
      - git: source

htpasswd:
  file.managed:
    - name: {{ django.home }}/htpasswd
    - source: salt://XXX/htpasswd
    - mode: 644
    - user: {{ django.username }}
    - group: {{ django.username }}
And yes, you'll need only 63 lines of code (+26 lines of pillar configuration) to deploy django, pil, solr, mysql, nginx, uwsgi, compile frontend with grunt/compass, collect static files, compile messages for translations, etc.