Django and Websockets. Setup for production

Django Channels is promising but not ready yet. This technology hasn't been included in Django 1.10 (see Django 1.10 release notes) as was promised earlier. Another reason is that daphne is not a reliable replacement at the moment for a bullet proof uwsgi or gunicorn.
A typical way to approach websockets is to use socket.io running inside a NodeJS. But having uwsgi this is not necessary as uwsgi offers websockets. It rather a low-level solution, in order to omit boilerplate it is advisory to use django-websocket-redis. Its setup can be a tricky thing especially if you targeting scalable solution for production. I've been using this approach in 2 projects in production and gathered minimal django application with websockets on board with production ready configuration. See details below. Full source code is available.

Ftp server with zero configuration

Log in to the server, pip install pyftpdlib and type in python interpreter

from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
authorizer = DummyAuthorizer()
authorizer.add_user('test', 'test37', '/some/dir/')
handler = FTPHandler
handler.authorizer = authorizer
server = FTPServer(('0.0.0.0', 5021), handler)
server.serve_forever()

Try it out

~$ ftp
ftp> open X.X.X.X 5021
Connected to X.X.X.X.
220 pyftpdlib 1.4.0 ready.
Name (X.X.X.X:adolgarev): test
331 Username ok, send password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> passive
Passive mode on.
ftp> ls
227 Entering passive mode (X,X,X,X,X,X).
150 File status okay. About to open data connection.
-rw-r--r--   1 4001     4001      3622358 Aug 20 08:17 xxx
drwxr-xr-x   3 4001     4001         4096 Jul 09 10:57 xxx
-rw-r--r--   1 4000     4000      3585261 Jul 09 12:49 xxx
drwxr-xr-x   3 4000     4000         4096 Aug 20 08:35 xxx
226 Transfer complete.

Even more, it is surprisingly fast, see comparison to other implementations.

Inversion of Control for 6 year old

If you can't explain it to a six year old, you don't understand it yourself. (Albert Einstein)

Despite release 1.0 of Spring was in 2003 the new wave of programmers seem to misuse the concept. Below we'll consider a small example to explain.

Python, scalable file uploading

Almost every solution suffers from dedicating a thread for particular client (like django) or despite having event loop on board sucking the whole file into the memory (like tornado). That makes impossible to handle either big amount of clients or big files. We also do not want to dedicate a file upload completely to some external entity while we want to make some auth checks before upload will be permitted (otherwise server can be flooded with unauthorized ingest traffic).
I've borrowed the idea below from Anatoly Mikhailov, see his post Nginx direct file upload without passing them through backend. Lets do this quickly.

Maintainable python code

The benefit of dynamically typed languages is the ease of writing the code but the cost of this is the problem with its understanding.
Such heavily typed languages as Haskell and Scala make use of comprehensive static type system and compiler that does all dirty job. At any given point of code one knows for sure that this particular variable is of this type and this function has such return type, etc. So basically one can understand what is going on. In python with its duck typing one can pass to function any object that adheres to some contract, this function can pass it further and further, add/remove some methods on the fly, etc., etc. So looking at some piece of code where you see variables and function applications one can barely understand it and lose track on what is going on. A static analyzer can help to some degree, see for instance PySonar, a Deep Static Analyzer for Python:

Treatment of Python’s dynamism. Static analysis for Python is hard because it has many dynamic features. They help make programs concise and flexible, but they also make automated reasoning about Python programs hard. Fortunately, some of these features can be reasonably handled. For example, function or class redefinition can be handled by inferring the effective scope of the old and new definitions. For code that are really undecidable, PySonar uses a universal honest answer: “I don’t know.” Well, not quite so. It attempts to report all known possibilities. For example, if a function is “conditionally defined” (e.g., defined differently in two branches of an if-statement) and the condition is undecidable, then PySonar gives it a union type which contains all possible types it can possibly have. By doing that, PySonar reduces false negative rates.

Sidenote, Scala has duck typing via structural types but their usage in general is not recommended because implementation uses reflection that is slow. But indeed Scala structural typing is type safe in contrast to python, see Structural typing vs. Duck typing.
There is an example of dynamically typed language that doesn't suffer from code readability problem - Erlang. One always knows what comes in and what comes out (and as a result what is in each line of code). It doesn't have some comprehensive type system except records aka structs in C. But it has Function Specifications and dialyzer. Unlike python when you call the function in Erlang you pass not some object that has incapsulated state and exposed behavior but just plain data, the input data format is defined in function spec along with return data format. One doesn't need to pass behavior because it is incapsulated in some other lightweight process pid of which you may pass within the data. Because of such elegant/specific implementation of incapsulation and polymorphism Erlang solves problem with readability.

So, while pysonar is promising can one still do something easier and better? Unit tests? Good to have but covering each function is too much. Docstrings? Too informal. After a while I hit the article Making Wrong Code Look Wrong. I ended up with simple idea: name each variable/function in a way everybody understands what type it has/returns (the same for function args).
Simple example. Having following information aside

user variable has type model.User
userid is a user id, has type int

It is easy to get idea what line below does indepedently on where in code you see it

user = user_by_userid(userid)

If you present this information on variables/functions in some formal way, IDEs/static analyzers can also put warnings on variables that do not have such spec and on expressions/statements that just look wrong (see again article by Joel Spolsky), navigate to type definitions, show variable/function descriptions upon hovering, etc.

Don't parse output from system utilities

One often can see that some system utility returns information he needs. Then he does a wrong thing: parses output from this utility. We'll do opposite. For instance, we'll get broadcast address

/sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:0A:CD:14:CD:77  
          inet addr:192.168.44.177  Bcast:192.168.44.255  Mask:255.255.255.0
          inet6 addr: fe80::20a:cdff:fe14:cd77/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:791580 errors:0 dropped:0 overruns:0 frame:0
          TX packets:381581 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:468681235 (446.9 Mb)  TX bytes:47469801 (45.2 Mb)
          Interrupt:18 Base address:0xc000

What does ifconfig do to get this info?

strace /sbin/ifconfig eth1
...
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
...
ioctl(4, SIOCGIFBRDADDR, {ifr_name="eth1", ifr_broadaddr={AF_INET, inet_addr("192.168.44.255")}}) = 0
...

strace shows that descriptor 4 is passed to ioctl. In python one can do the same

# get the constant beforehand
grep -R SIOCGIFBRDADDR /usr/include/*                         
/usr/include/bits/ioctls.h:#define SIOCGIFBRDADDR  0x8919  /* get broadcast PA address */


import fcntl, socket, struct
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_IP)
SIOCGIFBRDADDR = 0x8919
iface = struct.pack('256s', 'eth1')
info = fcntl.ioctl(s.fileno(), SIOCGIFBRDADDR, iface)
socket.inet_ntoa(info[20:24])
'192.168.44.255'

Why we get bytes from 20 to 24? One passes struct ifreq to ioctl (see netdevice(7)), IFNAMSIZ is 16, plus offsetof(struct sockaddr_in, sin_addr), this equals to 20, and plus unsigned long that is 4 bytes.

struct ifreq {
    char ifr_name[IFNAMSIZ]; /* Interface name */
    union {
        struct sockaddr ifr_addr;
        struct sockaddr ifr_dstaddr;
        struct sockaddr ifr_broadaddr;
        struct sockaddr ifr_netmask;
        struct sockaddr ifr_hwaddr;
        short           ifr_flags;
        int             ifr_ifindex;
        int             ifr_metric;
        int             ifr_mtu;
        struct ifmap    ifr_map;
        char            ifr_slave[IFNAMSIZ];
        char            ifr_newname[IFNAMSIZ];
        char *          ifr_data;
    };
};
struct sockaddr_in {
    short            sin_family;
    unsigned short   sin_port;
    struct in_addr   sin_addr;
    char             sin_zero[8];
};
struct in_addr {
    unsigned long s_addr;
};

Note, there are no holes in these structs. One can expect 4 byte hole before sin_addr on 64 bit systems, but there is no. Those structures are declared in a way that omits holes. A simple test to show

#include <stdio.h>
#include <stddef.h>

#include <netinet/in.h>


struct in_addr2 {
    unsigned long s_addr;
};
struct sockaddr_in2 {
    short            sin_family;
    unsigned short   sin_port;
    struct in_addr2  sin_addr;
    char             sin_zero[8];
};

int main(void) {
    printf("%d\n", offsetof(struct sockaddr_in, sin_addr));
    printf("%d\n", offsetof(struct sockaddr_in2, sin_addr));
    return 0;
}

gcc 1.c
./a.out
4
8

With the help of strace you can find out a lot about utilities, one more example

strace ps
...
open("/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
fcntl(5, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
getdents64(5, /* 211 entries */, 32768) = 5552
stat("/proc/1", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/1/stat", O_RDONLY)          = 6
read(6, "1 (init) S 0 1 1 0 -1 4202752 31"..., 1023) = 187
close(6)                                = 0
open("/proc/1/status", O_RDONLY)        = 6
read(6, "Name:\tinit\nState:\tS (sleeping)\nT"..., 1023) = 675
close(6)
stat("/proc/2", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
... 

So, the advice is to not parse output from utilities, it is not reliable thing to do, it is a subject to change. Use ioctl, sysctl, /proc, etc. to gather info you need. Use strace and others to find out what you need.

P.S. The funniest bug I've seen is when one mixed stdout and stderr and got every time new result upon parsing due to two streams are mixed in an unpredictable way. The other one was due to i18n feature.

Python, profiling

'Have you seen python web developers who do high load site development for living?'
'Oh, almost all of them do such things,' you say.
'Have you seen python devs who know how to debug?'
'A few,' you say.
'Have you seen python devs who profile things that supposed to be high loaded?'

Python has cProfile, C has kcachegrind, they look good together.

For instance, lets profile multithreaded wsgi app. Change your handler

def read(self, request, *args, **kwargs):
    ...
    return ...

To something like

def read(self, *args, **kwargs):
    import cProfile
    import uuid
    cProfile.runctx('self.read2(*args, **kwargs)', globals(), locals(),
        '/folder_with_stats/' + uuid.uuid4().get_hex())
    return self.__res

def read2(self, *args, **kwargs):
    self.__res = self.read3(*args, **kwargs)

def read3(self, request, *args, **kwargs):
    ...
    return ...

Then (high) load your app. Gather results from folder_with_stats

import os
import pstats
import time
from pyprof2calltree import convert

# Collect stats
p = None
for i in os.listdir('/folder_with_stats'):
    filename = '/folder_with_stats/' + i
    if not p:
        p = pstats.Stats(filename)
    else:
        p.add(filename)
    os.unlink(filename)
res = str(int(time.time())) + '.kgrind'
convert(p, res)

os.execlp('kcachegrind', res)

The main thing here to note is pyprof2calltree. The result

Signals and threads in python

The task: start a set of processes and wait till they terminate, if SIGTERM is received send same signal to all child processes and again wait till they terminate. (Ok, I'd go with sending signal to the process group, but however.)
The naive solution: use subprocess and threading modules, start thread per child process and communicate(), in main thread join() with others. Why naive? This doesn't work. Signal handler is not invoked. The documentation to signal module says:

Although Python signal handlers are called asynchronously as far as the Python user is concerned, they can only occur between the "atomic" instructions of the Python interpreter. This means that signals arriving during long calculations implemented purely in C (such as regular expression matches on large bodies of text) may be delayed for an arbitrary amount of time.

That is signal handler will be processed only after current "atomic" operation finishes. Unfortunately join() is one of such atomic operations. In other works signal handler will be invoked only after thread termination.
Another way is to use coroutines and select:

proc = subprocess.Popen(cmd,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        close_fds=True,
                        preexec_fn=preexec_fn)

fdesc = proc.stdout.fileno()
flags = fcntl.fcntl(fdesc, fcntl.F_GETFL)
fcntl.fcntl(fdesc, fcntl.F_SETFL, flags | os.O_NONBLOCK)

while True:
    try:
        dt = proc.stdout.read()
        if dt == '':
            break
        yield dt
    except IOError:
        # EWOULDBLOCK
        
        yield ''
        
        try:
            select.select([proc.stdout], [], [], 1)
        except select.error:
            # select.error: (4, 'Interrupted system call') - ignore it,
            # just call select again
            pass

while True:
    try:
        proc.wait()
        break
    except OSError:
        # OSError: [Errno 4] Interrupted system call
        continue

And the supervisor (left - an array of generators from coroutines)

while left:
    new_left = []
    
    for execute in left:
        try:
            execute.next()
            new_left.append(execute)
        except StopIteration:
            pass
        except Exception, e:
            err = e

    left = new_left

Also note that EINTR in general is not processed by python standard library, it is just forwarded up the stack as C does. In most cases if you are lucky it is enough to call interrupted routine again as in C (but C guarantees that this works, python doesn't).

Python, how to open TUN/TAP device

Just a note in case I forget

def open(n):
    TUNSETIFF = 0x400454ca
    IFF_TUN   = 0x0001
    IFF_TAP   = 0x0002
    TUNMODE = IFF_TAP
    MODE = 0
    DEBUG = 0
    f = os.open("/dev/net/tun", os.O_RDWR)
    ifs = ioctl(f, TUNSETIFF, struct.pack("16sH", "tap%d" % n, TUNMODE))
    #ifname = ifs[:16].strip("\x00")
    return f

And then as usual

f1 = open(1)
f2 = open(2)

p = os.read(f1, 65000)
os.write(f2, p)

os.close(f1)
os.close(f2)