Cgroups, limit memory

If some application may eat a lot of memory and cause swapping you can put it to sandbox
Test app memtest.c

#include <stdlib.h>
#include <stdio.h>


int main() {
    int     i;
    void   *p;

    p = NULL;

    for (i = 0; 1; i+= 4096) {
        p = malloc(4096);
        if (!p) {
            perror("malloc");
            break;
        }
        printf("allocated %d bytes\n", i);
    }

}

Create new group and add to it current shell process

mkdir /sys/fs/cgroup/memory/0
echo $$ > /sys/fs/cgroup/memory/0/tasks

Configure memory limit

echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes

Test

gcc memtest.c
./a.out
...
allocated 3997696 bytes
Killed

As you mentioned malloc didn't return NULL, app receives a signal and terminates because swap is not used and kernel kills one of the processes when there is no enough free memory left.