Celery is a great tool for background task processing in Django. We use it in a lot of the custom web apps we build at Caktus, and it's quickly becoming the standard for all variety of task scheduling work loads, from simple to highly complex.

Although rarely, sometimes a Celery worker may stop processing tasks and appear completely hung. In other words, issuing a restart command (through Supervisor) or kill (from the command line) doesn't immediately restart or shutdown the process. This can particularly be an issue in cases where you have a queue setup with only one worker (e.g., to avoid processing any of the tasks in this queue simultaneously), because then none of the new tasks in the queue will get processed. In these cases you may find yourself resorting to manually calling kill -9 <pid> on the process to get the queue started back up again.

We recently ran into this issue at Caktus, and in our case the stuck worker process wasn't processing any new tasks and wasn't showing any CPU activity in top. That seemed a bit odd, so I thought I'd make an attempt to discover what that process was actually doing at the time that it became non-responsive. Enter strace.

strace is a powerful command-line tool for inspecting running processes to determine what "system calls" they're making. System calls are low level calls to the operating system kernel that might involve accessing hard disks, the network, creating new processes, or other such operations. First, let's find the PID of the celery process we're interested in:

ps auxww|grep celery

You'll find the PID in the second column. For the purposes of this post let's assume that's 1234. You can inspect the full command in the process list to make sure you've identified the right celery worker.

Next, run strace on that PID as follows:

sudo strace -p 1234 -s 100000

The -p flag specifies the PID, and the -s flag specifies the size of the output. By default it's limited to 32 characters, which we found isn't very helpful if the system call being made includes a long string as an argument. You might see something like this:

Process 1234 attached - interrupt to quit
write(5, "ion id='89273' responses='12'>\n     ...", 37628

In our case, the task was writing what looked like some XML to file descriptor "5". The XML was much longer and at the end included what looked like a few attributes of a pickled Python object, but I've shortened it here for clarity's sake. You can see what "5" corresponds to by looking at the output of lsof:

sudo lsof|grep 1234

The file descriptor shows up in the "FD" column; in our version of strace, that happens to be the 4th column from the left. You'll see a bunch of files that you don't care about, and then down near the bottom, the list of open file descriptors:

python    1234   myuser    0r     FIFO                0,8      0t0    6593806 pipe
python    1234   myuser    1w     FIFO                0,8      0t0    6593807 pipe
python    1234   myuser    2w     FIFO                0,8      0t0    6593807 pipe
python    1234   myuser    3u     0000                0,9        0       4738 anon_inode
python    1234   myuser    4r     FIFO                0,8      0t0    6593847 pipe
python    1234   myuser    5w     FIFO                0,8      0t0    6593847 pipe
python    1234   myuser    6r      CHR                1,9      0t0       4768 /dev/urandom
python    1234   myuser    7r     FIFO                0,8      0t0    6593850 pipe
python    1234   myuser    8w     FIFO                0,8      0t0    6593850 pipe
python    1234   myuser    9r     FIFO                0,8      0t0    6593916 pipe
python    1234   myuser   10u     IPv4            6593855      0t0        TCP ip-10-142-126-212.ec2.internal:33589->ip-10-112-43-181.ec2.internal:amqp (ESTABLISHED)

You can see "5" corresponds to a pipe, which at least in theory ends up with a TCP connection to the amqp port on another EC2 server (host names are fictional).

RabbitMQ was operating properly and not reporting any errors, so our attention turned to the Celery task in question. Upon further examination, an object we were passing to the task included a long XML string as an attribute, which was being pickled and passed to RabbitMQ. Issues have been reported with long argument sizes in Celery before, and while it appears they should be supported, an easy workaround for us (and Celery's recommended approach) was to pass an ID for this object rather than the object itself, greatly reducing the size of the task's arguments and avoiding the risk of overwriting any object attributes.

While there may have been other ways to fix the underlying issue, strace and lsof were crucial in helping us figure out the problem. One might be able to accomplish the same thing with a lot of logging, but if your code is stuck in a system call and doesn't appear to be showing any noticeable CPU usage in top, strace can take you immediately to the root of the problem.

Last week everyone at Caktus stepped away from client work for a day and a half to focus on learning and experimenting. This was our fourth ShipIt day at Caktus, our first being almost exactly a year ago. Each time we all learn a ton, not only by diving head first into something new, but also by hearing the experiences of everyone else on the team.

DevOps: Provisioning with SaltStack & LXC+Vagrant

We have found SaltStack to be useful in provisioning servers. It is a Python based tool for spinning up and configuring servers with all of the services that are needed to run your web application. This work is a natural progression from the previous work that we have done at Caktus in deploying code in a consistent way to servers. SaltStack really shines with larger more complicated systems designed for high availability (HA) and scaling where each service runs on its own server. Salt will make sure that the setup is reproducible.

This is often important while testing ideas for HA and scaling. The typical cycle looks like:

• Work on states in Salt
  
• Run Salt through Fabric building the whole system, locally through vagrant, on a cloud provider, or on physical hardware.
  
• Pound on the system using benchmarking tools narrowing in on bottlenecks and single points of failure.
  
• Start fixing the problems you uncovered in your states starting the cycle over again.

Victor, Vinod, David, and Dan all worked on learning more about SaltStack through scratching different itches they’ve felt while working on client projects. Some of the particular issues folks looked at included understanding the differences between running Salt with and without a master, how to keep passwords safe on the server while sharing them internally on the development team, and setting up Halite, a new web interface for Salt.

In order to test these complicated server configurations, we often rely on Vagrant to run these full system configurations on developer’s laptops. This is the quickest way to start building out systems. The problem with this though is that our laptops are not as fast as the hardware that the services will eventually be provisioned on. In order to reduce the time waiting in the code-rebuild-test cycle, Scott our friendly system administrator delved into running Vagrant with LXC containers on the backend with our development laptops. LXC containers are more lightweight than VirtualBox virtual machines and can be created more quickly on the fly. This involved learning about how to upgrade the development laptop’s kernels on Ubuntu long term support image. Scott was successful and there was a response of “Oooh” and “Ahhh” from the developers when he demoed the speed of creating a new VM with LXC through vagrant.

Front-end Web + Visualizations: Realtime Web, Cellular Automata, Javascript MVC, & SVG Splines

Caktus has a growing team of front-end developers and folks interested in user interaction. There were a number of projects this ShipIt day that explored different tools for designing visualizations and user experiences.

Mark dove into WebRTC building a new project, rtc-demo with the actual demo hosted on Github static pages (note: it only works on recent Firefox builds so far). It’s neat that this is hosted on a static web assets host since the project does allow users to interact with one another. Red & Blue users go to the static site and create links that they share with one another and connect directly through their browsers without any server in between. Both users see a tic-tac-toe board and can alternate taking turns. The moves are relayed directly to their opponent. This exploration allowed Mark to evaluate some new technologies and their support in different browsers. He was able to play around with new modes of interaction on the web where the traditional server-client model can be challenged and hybrid peer-to-peer apps can be built.

Caleb has been taking a class on programming in C outside of work and wanted to continue to work on one of his personal projects during the ShipIt day. There’s something about pointer arithmetic and syntax that Caleb finds fascinating  Caleb extended some of the features of his cellular automata simulation engine, gameoflife. This included some of his rule files, initial states, and rule file engine. The result was experiments with the Wireworld simulation including a working XOR gate using simple cellular automata rules and Brian’s Brain which produces a visually interesting chaotic simulation. Caleb’s demo was a crowd pleaser with everyone enthralled by the puffers, spaceships, and gliders moving around and interacting on the screen.

Rebecca made some amazing usability improvements to our own django-timepiece. Timepiece is part of the suite of tools that we use internally for managing contracts, hours, timesheets, and schedules. Rebecca focused on some of the timesheet related workflows including verifying timesheets. She made these processes more fluid by building an API and making calls to it in using Backbone. This allowed users to do things like delete or edit entries on their timesheet before verifying it without leaving the page.

Wray was also decided to work on a front-end project. His project focused on building SVGs on the fly in raw Javascript without any supporting libraries. Wray started by showing us his passion for how to mathematically define curves and how useful the different definitions are from a user experience point of view as designer working in a vector drawing program. The easier it is for the designer to get from what they meant to draw to what is represented on the screen the better the resulting design will be. He showed a particular interest in the Spiro curve definition. Wray dug into this further by looking at the definition of curves supported by the SVG standard and built an interactive tool for drawing curves by editing the string defining the SVG element on the fly. The resulting project is still experimental at this point, but is an interesting exploration into what can be done with SVGs in Javascript without any supporting libraries.

Django Core

Karen and Tobias waded into some internals of bleeding edge of Django. In particular, Tobias offered up comments on a pull request related to ticket #21271. Tobias gave feedback to the original pull request author and Django Core committer Tim Graham on the difference between instance and class variables and when it’s appropriate to use class variables in Python.

Karen gave an illuminating talk on transitioning a particular client project to Django 1.6 from Django 1.5. Her slides are available to view online. She generalized the testing, debugging, and eventually fixing strategies she used while discussing the particular problems she encountered on the project. The strategy Karen used was running manage.py check to check for new settings that may have changed. This is a new management comment added in 1.6 and can be used from now on to ease upgrading versions.

She found the following issues when upgrading:

• UNUSABLE_PASSWORD was part of an internal API that changed. We used this internal in the project as sometimes you must, but special care must be taken when upgrading code that relies on unstable APIs.
  
• MAXIMUM_PASSWORD_LENGTH was used on this project after the recent Django security fix. Based on subsequent security updates, this is no longer needed and can be removed in 1.6.
  
• BooleanFields now do not automatically default to False. This was changed to encourage coder to be explicit and to increase the level of standards within the code.
  
• Django debug toolbar broke in 1.6! This is a shame, but will hopefully be updated by the time 1.6 comes out or soon after.

After this, Karen ran the project’s test suite and discovered the following remaining changes:

• The default session serializer used to be picked. Now it’s JSON.
• BoundField.label_tag now includes the form’s label_suffix.

Karen urged us to check out the Django 1.6 release notes and remember where you’re using unsupported internals and to check and write tests for that code particularly carefully. Also, she encouraged everyone to run python -Wall manage.py test to help expose more deprecation warnings and bugs.

Estimation Process Review

Ben and Daryl, our Project Managers at Caktus, worked up a full proposal and project schedule for an internal process of reworking our estimation process. We don’t shy away from large fixed bid projects and pride ourselves on meticulous estimates informed by careful client communication and requirements gathering. Our PMs wanted to help us look at this process and make it more formal as we grow to a larger group with lots of people leading the same process for different incoming projects.

Wrap Up

We had a great time with our latest ShipIt day. Each time we learn a ton from each other and by digging into some tough technical problems. We also learn a little bit more about what makes a good ShipIt day project. There were a number of crowd pleasers in this batch of projects and while some folks decided to try a completely new library or technology others decided to stay a bit closer to home by extending learning they did not have time look into more during a client project. We had a lot of fun and are looking forward to our next ShipIt day.

The following is a press release posted in partnership with our team at UNC  and Duke.

The web application development company Caktus Group has teamed up with researchers at the UNC Institute for Global Health & Infectious Diseases and the Duke Global Health Institute to develop a mobile phone app that may help patients better adhere to their medication regimens.

The new venture is funded by a $150,000 Phase I Small Business Innovation Research (SBIR) grant from the National Institutes of Health. The study team will develop a novel mobile phone game app with the goal of improving medication adherence among HIV-infected young black men who have sex with men. In the United States, this is the demographic with the highest number of new HIV infections. 

The Daily Dose app will utilize game mechanics and social networking features to improve adherence to HIV medication. In addition to scheduled medication reminders and adherence tracking, Daily Dose will use gaming features to create a compelling and engaging experience that will motivate and support behavior change. The app will also promote social interaction, which will allow users to share their successes and encourage others to maintain their adherence goals. Gamers will be awarded points for sharing their own successes and providing support and encouragement to their fellow gamers. The study team hopes this anonymous social network of peer support will drive patients to maintain their medication schedules. 

While Caktus Group leads the development of the game, the usability studies will be conducted by Lisa Hightow-Weidman, MD   associate professor of medicine at UNC and Sara LeGrand, PhD   assistant research professor at the Duke Global Health Institute. The researchers will organize focus groups and conduct a series of in-person interviews with potential users to get feedback on the app’s design throughout the development process. 

“Because the game will be designed, developed and refined based on consistent user feedback, we are confident that Daily Dose will be both engaging and effective,” said Tobias McNulty, principlal investigator and managing member at Caktus Group.

Ultimately, Daily Dose aims to improve drug adherence among a population disproportionately affected by HIV by developing an effective antiretroviral therapy adherence app tailored specifically for this group. Recent research has shown that treating HIV makes people less contagious and drastically reduces the spread of the virus to sexual partners. Through careful design and development, this new project hopes to improve patient outcomes and reduce the spread of HIV among a vulnerable population.

About Caktus Group

Caktus Consulting Group, LLC is a growing team of creative developers and designers based in Carrboro, North Carolina. The company was founded in August 2007 to serve the web needs of startups, researchers, health care organizations, and established businesses in the North Carolina Triangle region and beyond. Caktus Group’s specialty is creating web and mobile applications using Django, an open source web framework that is business friendly and easily customized. By listening carefully, Caktus Group develops products that clients consistently say are intuitive and fill needs they had not yet discovered.

About the UNC Institute for Global Health & Infectious Diseases 

Founded in 2007, the Institute for Global Health & Infectious Diseases at UNC  harnesses the resources of the University and its partners to solve local and global health problems, reduce the burden of disease, and inspire and train the next generation of leaders in global health.  

About the Duke Global Health Institute

The Duke Global Health Institute established in 2006, brings knowledge from every corner of Duke University to bear on the most important global health issues of our time. DGHI was established as a University-wide institute to coordinate, support, and implement Duke’s interdisciplinary research, education, and service activities related to global health. DGHI is committed to developing and employing new models of education and research that engage international partners and find innovative solutions to global health challenges. 

We are always looking for ways to make our tests run faster. That means writing tests which don't preform I/O (DB reads/writes, disk reads/writes) when possible. Django has a collection of TestCase subclasses for different use cases. The common TestCase handles the fixture loading and the creation the of TestClient. It uses the database transactions to ensure that the database state is reset for every test. That is it wraps each test in a transaction and rolls it back once the test is over. Any transaction management inside the test becomes a no-op. Since TestCase` overrides the transaction facilities, if you need to test the transactional behavior of a piece of code you can instead use TransactionTestCase. TransactionTestCase resets the database after the test runs by truncating all tables which is much slower than rolling back the transaction particularly if you have a large number of tables.

There is also SimpleTestCase which is the base class for the previous to classes. It has some additional assertions for testing HTML and overriding Django settings but doesn't manage the database state. If you are testing something that doesn't need to interact with the database such as form field/widget output, utility code or have mocked all of the database interactions you can save the overhead of the transaction by using SimpleTestCase.

Now what if you are running a set of tests which are only using SimpleTestCase or the base unittest.TestCase? Then you don't really need the test database creation at all. Depending on the backend you are using, the number of tables you have and the number of tests you are running the database creation can take many times longer than running the test themselves.

Our solution for this was to extend the default test runner. A quick examination of the build-in test runner reveals a solution.

def run_tests(self, test_labels, extra_tests=None, **kwargs):
    """
    Run the unit tests for all the test labels in the provided list.

    Test labels should be dotted Python paths to test modules, test
    classes, or test methods.

    A list of 'extra' tests may also be provided; these tests
    will be added to the test suite.

    Returns the number of tests that failed.
    """
    self.setup_test_environment()
    suite = self.build_suite(test_labels, extra_tests)
    old_config = self.setup_databases()
    result = self.run_suite(suite)
    self.teardown_databases(old_config)
    self.teardown_test_environment()
    return self.suite_result(suite, result)

The test suite is discovered before the test db is created. That means we can look at the set of tests which are going to be run and if none of them are using TransactionTestCase (TestCase is a subclass of TransactionTestCase) then we can skip the database creation/teardown. Here's what that looks like:

from django.test import TransactionTestCase
try:
    from django.test.runner import DiscoverRunner as BaseRunner
except ImportError:
    # Django < 1.6 fallback
    from django.test.simple import DjangoTestSuiteRunner as BaseRunner

from mock import patch


class NoDatabaseMixin(object):
    """
    Test runner mixin which skips the DB setup/teardown
    when there are no subclasses of TransactionTestCase to improve the speed
    of running the tests.
    """

    def build_suite(self, *args, **kwargs):
        """
        Check if any of the tests to run subclasses TransactionTestCase.
        """
        suite = super(NoDatabaseMixin, self).build_suite(*args, **kwargs)
        self._needs_db = any([isinstance(test, TransactionTestCase) for test in suite])
        return suite

    def setup_databases(self, *args, **kwargs):
        """
        Skip test creation if not needed. Ensure that touching the DB raises and
        error.
        """
        if self._needs_db:
            return super(NoDatabaseMixin, self).setup_databases(*args, **kwargs)
        if self.verbosity >= 1:
            print 'No DB tests detected. Skipping Test DB creation...'
        self._db_patch = patch('django.db.backends.util.CursorWrapper')
        self._db_mock = self._db_patch.start()
        self._db_mock.side_effect = RuntimeError('No testing the database!')
        return None

    def teardown_databases(self, *args, **kwargs):
        """
        Remove cursor patch.
        """
        if self._needs_db:
            return super(NoDatabaseMixin, self).teardown_databases(*args, **kwargs)
        self._db_patch.stop()
        return None


class FastTestRunner(NoDatabaseMixin, BaseRunner):
    """Actual test runner sub-class to make use of the mixin."""

There are a couple of things to note. Like the previous temporary MEDIA_ROOT runner, this is written as a mixin so that it can be combined with other test runner improvements. Second it uses mock to ensure that any attempts to connect to the database will fail. This idea is borrowed from Carl Meyer's Testing and Django Talk from PyCon 2012. To make use of this you would need to include this runner on your Python path and change the TEST_RUNNER setting to the full Python path to the FastTestRunner class.

With this in place if you had the follow tests

# myapp.tests.py
from django.test import TestCase, SimpleTestCase


class DbTestCase(TestCase):
    """Does something with the DB."""


class NoDbTestCase(SimpleTestCase):
    """Does something with the DB."""

in your app called myapp. If you were to run:

python manage.py test myapp

it would create the DB. However, if you run:

# For Django < 1.6
python manage.py test myapp.NoDbTestCase
# For Django 1.6+ with the DiscoverRunner
python manage.py test myapp.tests.NoDbTestCase

it would skip the test DB creation. Hooray for faster tests!

Django's logging configuration facilities, which arrived in version 1.3, have greatly eased (and standardized) the process of configuring logging for Django projects. When building complex and interactive web applications at Caktus, we've found that detailed (and properly configured!) logs are key to successful and efficient debugging. Another step in that process—which can be particularly useful in environments where you have multiple web servers—is setting up a centralized logging server to receive all your logs and make them available through an easily accessible web interface. There are a number useful tools to do this, but one we've found that works quite well is Graylog2. Installing and configuring Graylog2 is outside the scope of this post, but there are plenty of tutorials on how to do so accessible through your search engine of choice.

Once you have it setup, getting logs flowing to Graylog2 from Django is relatively straightforward. First, grab a copy of the graypy package from PyPI and add it to your requirements file:

pip install -U graypy

Next, add the following configuration inside the LOGGING['handlers'] dictionary in your settings.py, where graylog2.example.com is the hostname of your Graylog2 server:

LOGGING = {
    # ...
    'handlers': {
        # ...
        'gelf': {
            'class': 'graypy.GELFHandler',
            'host': 'graylog2.example.com',
            'port': 12201,
        },
    },
}

You'll most likely want to tell your project's top-level logger to send logs to the new gelf handler as well, like so:

LOGGING = {
    # ...
    'loggers': {
        # ...
        'projectname': {
            # mail_admins will only accept ERROR and higher
            'handlers': ['mail_admins', 'gelf'],
            'level': 'DEBUG',
        },
    },
}

With this configuration in place, log messages with a severity of DEBUG or greater that are sent to the projectname logger should begin flowing to Graylog2. You can easily test this by opening Django's python manage.py shell, grabbing the logger manually, and sending a log message:

import logging
logger = logging.getLogger('projectname')
logger.debug('testing message to graylog2')

You should see the message show up in Graylog2 almost immediately.

Now, this is all well and good, but if you want to use your Graylog2 server for multiple projects, you'll quickly find that all the log messages are interspersed and it can be difficult to tell what messages are coming from what projects. To address this issue, Graylog2 supports the concept of "streams," that is, filters that you can setup (which work only on incoming messages, not existing messages) to show messages that match only certain criteria. A simple solution here could be to filter on the hostname of the originating web servers, but this may not scale well in environments like Amazon Web Services' EC2 where you're often adding or removing web servers. As a better alternative, you can add metadata to log messages at the Python level prior to sending them to Graylog2 that will help you more easily identify the messages for different projects.

To do this, you need to use a feature of Python logging filters. While filters are most commonly used to filter out certain types of messages from being emitted altogether (as discussed in the Django documentation), they can also be used to modify the log records in transit and impart contextual metadata to be transmitted with the original message. To add this to our logging configuration, first create the following filter class in a Python module accessible from your project:

class StaticFieldFilter(logging.Filter):
    """
    Python logging filter that adds the given static contextual information
    in the ``fields`` dictionary to all logging records.
    """
    def __init__(self, fields):
        self.static_fields = fields

    def filter(self, record):
        for k, v in self.static_fields.items():
            setattr(record, k, v)
        return True

Next, we need to load this filter in our logging configuration and tell the gelf logger to pass records through it:

LOGGING = {
    # ...
    'filters': {
        # ...
        'static_fields': {
            '()': 'projectname.core.logfilters.StaticFieldFilter',
            'fields': {
                'project': 'projectname', # CHANGEME
                'environment': 'staging', # can be overridden in local_settings.py
            },
        },
    },
    'handlers': {
        # ...
        'gelf': {
            'class': 'graypy.GELFHandler',
            'host': 'graylog2.example.com',
            'port': 12201,
            'filters': ['static_fields'],
        },
        # ...
    },
}

The configuration under filters instantiates the StaticFieldFilter class and passes in the static fields that we want to attach to all of our log records. In this case, two fields are attached, a 'project' field with value 'projectname' and an 'environment' field with value 'staging'. The configuration for the gelf logger is the same, with the addition of the static_fields filter on the last line.

With these two items in place, you should be able to create streams via the Graylog2 web interface to trap and display records that match the combination of project and environment names that you're looking for.

Lastly, as an optional addition to this logging configuration, it may be desirable to filter out Django request objects from being sent to Graylog2. The request is added to log messages created by Django's exception handler and may contain sensitive information or in some cases may not be capable of being pickled (which is necessary to encode and send it with the log message). You can remove them from log messages with the following filter:

class RequestFilter(logging.Filter):
    """
    Python logging filter that removes the (non-pickable) Django ``request``
    object from the logging record.
    """
    def filter(self, record):
        if hasattr(record, 'request'):
            del record.request
        return True

and this corresponding filter configuration:

LOGGING = {
    # ...
    'filters': {
        # ...
        'django_exc': {
            '()': 'projectname.core.logfilters.RequestFilter',
        },
    },
    'handlers': {
        # ...
        'gelf': {
            'class': 'graypy.GELFHandler',
            'host': 'graylog2.example.com',
            'port': 12201,
            'filters': ['static_fields', 'django_exc'],
        },
        # ...
    },
}

With this configuration in place, you can have log messages flowing to Graylog2 from any number of project and server environment combinations, limited only by the resources of the log server itself.

Caktus is happy to be involved in this year’s DjangoCon hosted in Chicago.  We are pumped about the great lineup of speakers and can’t wait to see some of our old friends as well as meet some new folks. Beyond going to see the wonderful talks, Caktus is participating as a sponsor and Tobias McNulty will be speaking on scaling Django web apps. Come stop by our booth or see Tobias’ talk to connect with us.

Tobias’ talk "Scaling Your Write-Heavy Django App: A Case Study" will delve into some Django related scaling challenges we encountered while developing a write-heavy web app that would be deployed to millions of students and parents in the Chicago public school system. Through the lens of these specific problems he will show widely applicable solutions to forecasting web app loads, scaling, and automating the configuration so that it is completely repeatable.

We are proud to support the Django community through our sponsorship and involvement in DjangoCon. We’re all looking forward to the event and hope to see some of you there!

Back in March, at PyCon 2013, the PSF provided each attendee with a Raspberry Pi, a tiny credit-card sized computer meant to be paired with the Python programming language. The power and portability of the Raspberry Pi has stimulated an explosion of interest among hobbyists and educators. Their uses seem to be limited only by our collective imagination.

Along with that generous gift, the PSF contracted with Caktus to help tap into this collective imagination. Raspberry IO is a site dedicated to Python and Raspberry Pi enthusiasts. The goal of the site is to serve as a friendly place where anyone can learn and teach about using Python with a Raspberry Pi.

We are proud to announce that Raspberry IO is now an open source project. We've written tests and documentation to help you get started and contribute your ideas to the site. Please create an issue at the github repository.

We're excited about the possibilities for a site like this in the hands of the Raspberry Pi and Python communities.  If you have an interesting Raspberry Pi project, then we'd love for you to tell us about it! Razzy welcomes you!

The new custom user model configuration that arrived in Django makes it relatively straightforward to swap in your own model for the Django user model. In most cases, Django's built-in User model works just fine, but there are times when certain limitations (such as the length of the email field) require a custom user model to be installed. If you're starting out with a custom user model, setup and configuration are relatively straightforward, but if you need to migrate an existing legacy project (e.g., one that started out in Django 1.4 or earlier), there are a few gotchas that you might run into. We did this recently for one of our larger, long-term client projects at Caktus, and here's an outline of how we'd recommend tackling this issue:

1. First, assess any third party apps that you use to make sure they either don't have any references to the Django's User model, or if they do, that they use Django's generic methods for referencing the user model.

2. Next, do the same thing for your own project. Go through the code looking for any references you might have to the User model, and replace them with the same generic references. In short, you can use the get_user_model() method to get the model directly, or if you need to create a ForeignKey or other database relationship to the user model, you can settings.AUTH_USER_MODEL (which is just a string corresponding to the appname.ModelName path to the user model).

   Note that get_user_model() cannot be called at the module level in any models.py file (and by extension any file that a models.py imports), due to circular reference issues. Generally speaking it's easier to keep calls to get_user_model() inside a method whenever possible (so it's called at run time rather than load time), and use settings.AUTH_USER_MODEL in all other cases. This isn't always possible (e.g., when creating a ModelForm), but the less you use it at the module level, the fewer circular references you'll have to stumble your way through.

   This is also a good time to add the AUTH_USER_MODEL setting to your settings.py, just for the sake of explicitness:

# settings.py

AUTH_USER_MODEL = 'auth.User'

3. Now that you've done a good bit of the leg work, you can turn to actually creating the custom user model itself. How this looks is obviously up to you, but here's a sample that duplicates the functionality of Django's built-in User model, removes the username field, and extends the length of the email field

# appname/models.py

from django.db import models
from django.utils import timezone
from django.utils.http import urlquote
from django.utils.translation import ugettext_lazy as _
from django.core.mail import send_mail
from django.contrib.auth.models import AbstractBaseUser, PermissionsMixin

class CustomUser(AbstractBaseUser, PermissionsMixin):
    """
    A fully featured User model with admin-compliant permissions that uses
    a full-length email field as the username.

    Email and password are required. Other fields are optional.
    """
    email = models.EmailField(_('email address'), max_length=254, unique=True)
    first_name = models.CharField(_('first name'), max_length=30, blank=True)
    last_name = models.CharField(_('last name'), max_length=30, blank=True)
    is_staff = models.BooleanField(_('staff status'), default=False,
        help_text=_('Designates whether the user can log into this admin '
                    'site.'))
    is_active = models.BooleanField(_('active'), default=True,
        help_text=_('Designates whether this user should be treated as '
                    'active. Unselect this instead of deleting accounts.'))
    date_joined = models.DateTimeField(_('date joined'), default=timezone.now)

    objects = CustomUserManager()

    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = []

    class Meta:
        verbose_name = _('user')
        verbose_name_plural = _('users')

    def get_absolute_url(self):
        return "/users/%s/" % urlquote(self.email)

    def get_full_name(self):
        """
        Returns the first_name plus the last_name, with a space in between.
        """
        full_name = '%s %s' % (self.first_name, self.last_name)
        return full_name.strip()

    def get_short_name(self):
        "Returns the short name for the user."
        return self.first_name

    def email_user(self, subject, message, from_email=None):
        """
        Sends an email to this User.
        """
        send_mail(subject, message, from_email, [self.email])

Note that this duplicates all aspects of the built-in Django User model except the get_profile() method, which you may or may not need in your project. Unless you have third party apps that depend on it, it's probably easier simply to extend the custom user model itself with the fields that you need (since you're already overriding it) than to rely on the older get_profile() method. It is worth noting that, unfortunately, since Django does not support overriding model fields, you do need to copy all of this from the AbstractUser class within django.contrib.auth.models rather than simply extending and overriding the email field.

4. You might have noticed the Manager specified in the model above doesn't actually exist yet. In addition to the model itself, you need to create a custom manager that supports methods like create_user(). Here's a sample manager that creates users without a username (just an email):

# appname/models.py

from django.contrib.auth.models import BaseUserManager

class CustomUserManager(BaseUserManager):

    def _create_user(self, email, password,
                     is_staff, is_superuser, **extra_fields):
        """
        Creates and saves a User with the given email and password.
        """
        now = timezone.now()
        if not email:
            raise ValueError('The given email must be set')
        email = self.normalize_email(email)
        user = self.model(email=email,
                          is_staff=is_staff, is_active=True,
                          is_superuser=is_superuser, last_login=now,
                          date_joined=now, **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_user(self, email, password=None, **extra_fields):
        return self._create_user(email, password, False, False,
                                 **extra_fields)

    def create_superuser(self, email, password, **extra_fields):
        return self._create_user(email, password, True, True,
                                 **extra_fields)

5. If you plan to edit users in the admin, you'll most likely also need to supply custom forms for your new user model. In this case, rather than copying and pasting the complete forms from Django, you can extend Django's built-in UserCreationForm and UserChangeForm to remove the username field (and optionally add any others that are required) like so:

# appname/forms.py

from django.contrib.auth.forms import UserCreationForm, UserChangeForm

from appname.models import CustomUser

class CustomUserCreationForm(UserCreationForm):
    """
    A form that creates a user, with no privileges, from the given email and
    password.
    """

    def __init__(self, *args, **kargs):
        super(CustomUserCreationForm, self).__init__(*args, **kargs)
        del self.fields['username']

    class Meta:
        model = CustomUser
        fields = ("email",)

class CustomUserChangeForm(UserChangeForm):
    """A form for updating users. Includes all the fields on
    the user, but replaces the password field with admin's
    password hash display field.
    """

    def __init__(self, *args, **kargs):
        super(CustomUserChangeForm, self).__init__(*args, **kargs)
        del self.fields['username']

    class Meta:
        model = CustomUser

Note that in this case we do not use the generic accessors for the user model; rather, we import the CustomUser model directly since this form is tied to this (and only this) model. The benefit of this approach is that it also allows you to test your model via the admin in parallel with your existing user model, before you migrate all your user data to the new model.

6. Next, you need to create a new admin.py entry for your user model, mimicking the look and feel of the built-in admin as needed. Note that for the admin, similar to what we did for forms, you can extend the built-in UserAdmin class and modify only the attributes that you need to change, keeping the other behavior intact.

# appname/admin.py

from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from django.utils.translation import ugettext_lazy as _

from appname.models import CustomUser
from appname.forms import CustomUserChangeForm, CustomUserCreationForm

class CustomUserAdmin(UserAdmin):
    # The forms to add and change user instances

    # The fields to be used in displaying the User model.
    # These override the definitions on the base UserAdmin
    # that reference the removed 'username' field
    fieldsets = (
        (None, {'fields': ('email', 'password')}),
        (_('Personal info'), {'fields': ('first_name', 'last_name')}),
        (_('Permissions'), {'fields': ('is_active', 'is_staff', 'is_superuser',
                                       'groups', 'user_permissions')}),
        (_('Important dates'), {'fields': ('last_login', 'date_joined')}),
    )
    add_fieldsets = (
        (None, {
            'classes': ('wide',),
            'fields': ('email', 'password1', 'password2')}
        ),
    )
    form = CustomUserChangeForm
    add_form = CustomUserCreationForm
    list_display = ('email', 'first_name', 'last_name', 'is_staff')
    search_fields = ('email', 'first_name', 'last_name')
    ordering = ('email',)

admin.site.register(CustomUser, CustomUserAdmin)

7. Once you're happy with the fields in your model, use South to create the schema migration to create your new table:

python manage.py schemamigration appname --auto

8. This is a good point to pause, check out your user model via the admin, and make sure it looks and functions as expected. You should still see both user models at this point, because we haven't yet adjusted the AUTH_USER_MODEL setting to point to our new model (this is intentional). You may have to delete the migration file and repeat the previous step a few times if you don't get it quite right the first time.
9. Next, we need to write a data migration using South to copy the data from our old user model to our new user model. This is relatively straightforward, and you can get a template for the data migration as follows:

python manage.py datamigration appname --freeze otherapp1 --freeze otherapp2

Note that the --freeze arguments are optional and should be used only if you need to access the models of these other apps in your data migration. If you have foreign keys in these other apps to Django's built-in auth.User model, you'll likely need to include them in the data migration. Again, you can experiment and repeat this step until you get it right, deleting the incorrect migrations as you go.

10. Once you have the template for your data migration created, you can write the content for your migration. A simple forward migration to simply copy the users, maintaining primary key IDs (this has been verified to work with PostgreSQL but no other backend), might look something like this:

# appname/migrations/000X_copy_auth_user_data.py

class Migration(DataMigration):
    def forwards(self, orm):
        "Write your forwards methods here."
        for old_u in orm['auth.User'].objects.all():
            new_u = orm.CustomUser.objects.create(
                        date_joined=old_u.date_joined,
                        email=old_u.email and old_u.email or '%s@example.com' % old_u.username,
                        first_name=old_u.first_name,
                        id=old_u.id,
                        is_active=old_u.is_active,
                        is_staff=old_u.is_staff,
                        is_superuser=old_u.is_superuser,
                        last_login=old_u.last_login,
                        last_name=old_u.last_name,
                        password=old_u.password)
            for perm in old_u.user_permissions.all():
                new_u.user_permissions.add(perm)
            for group in old_u.groups.all():
                new_u.groups.add(group)

Since we ensure that the primary keys stay the same from one table to another, we can just adjust the foreign keys in our other models to point to this new custom user model, rather than needing to update each row in turn.

Note 1: This migration does not account for any duplicate emails that exist in the database, so if this is a problem in your case, you may need to write a separate migration first that resolves any such duplicates (and/or manually resolve them with the users in question).

Note 2: This migration does not update the content types for any generic relations in your database. If you use generic relations and one or more of them points to the old user model, you'll also need to update the content type foreign keys in these relations to reference the content type of the new user model.

11. Once you have this migration written and tested to your liking (and have resolved any duplicate user issues), you can run the migration and verify that it did what you expected via the Django admin.

python manage.py migrate

Needless to say, we recommend doing this testing on a local, development copy of the production database (using the same database server) rather than on a live production or even staging server. Once you have the entire process complete, you can test it on a staging (and finally on the production) server.

12. Before we switch to our new model, let's create a temporary initial migration for the auth app which we can later use as the basis for creating a migration to delete the obsolete auth_user and related tables. First, create a temporary module for auth migrations in your settings file:

SOUTH_MIGRATION_MODULES = {
    'auth': 'myapp.authmigrations',
}

Then, "convert" the auth app to South like so:

python manage.py convert_to_south auth

This won't do anything to your database, but it will create (and fake the run of) an initial migration for the auth app.

13. Let's review where we stand. You've (a) updated all your code to use the generic interface for accessing the user model, (b) created a new model and the corresponding forms to access it through the admin, and (c) written a data migration to copy your user data to the new model. Now it's time to make the switch. Open up your settings.py and adjust the AUTH_USER_MODEL setting to point to the appname.ModelName path of your new model:

# settings.py

AUTH_USER_MODEL = 'appname.CustomModel'

14. Since any foreign keys to the old auth.User model have now been updated to point to your new model, we can create migrations for each of these apps to adjust the corresponding database tables as follows:

python manage.py schemamigration --auto otherapp1

You'll need to repeat this for each of the apps in your project that were affected by the change in the AUTH_USER_MODEL setting. If this includes any third party apps, you may want to store those migrations in an app inside your project (rather than use the SOUTH_MIGRATION_MODULES setting) so as not to break future updates to those apps that may include additional migrations.

15. Additionally, at this time you'll likely want to create the previously-mentioned South migration to delete the old auth_user and associated tables from your database:

python manage.py schemamigration --auto auth

Since South is not actually intended to work with Django's contrib apps, you need to copy the migration this command creates into the app that contains your custom model, renumbering it along the way to make sure that it's run after your data migration to copy your user data. Once you have that created, be sure to delete the SOUTH_MIGRATION_MODULES setting from your settings file and remove the unneeded, initial migration from your file system.

16. Last but not least, let's run all these migrations and make sure that all the pieces work together as planned:

python manage.py migrate

Note: If you get prompted to delete a stale content type for auth | user, don't answer "yes" yet. Despite Django's belief to the contrary, this content type is not actually stale yet, because the South migration to remove the auth.User model has not yet run. If you accidentally answer "yes," you might see an error stating InternalError: current transaction is aborted, commands ignored until end of transaction block. No harm was done, just run the command again and answer "no" when prompted. If you'd like to remove the stale auth | user content type, just run python manage.py syncdb again after all the migrations have completed.

That completes the tutorial on creating (and migrating to) a custom user model with Django 1.5's new support for the same. As you can see it's a pretty involved process, even for something as simple as lengthening and requiring a unique email field in place of a username, so it's not something to be taken lightly. Nonetheless, some projects can benefit from investing in a custom user model that affords all the necessary flexibility. Hopefully this post will help you make that decision (and if needed the migration itself) with all the information necessary to make it go as smoothly as possible.

On Wednesday, Mark and Julia are teaming up to deliver their talk "Developers + Designers: Collaborating on your Open Source project " at OSCON 2013. We’re thrilled that their talk got accepted and know that they have a lot to contribute to the Open Source conversation which too often lacks the voice of designers. Julia and Mark will be discussing how it is important to start collaborating early on your ideas before writing code. We’re happy to have such adept developers and designers here at Caktus pushing each other to work outside of their strict job titles and collaborate to build projects that couldn’t be built anywhere else. If you are at OSCON this week, stop by and hear Mark and Julia’s talk. Or, if you miss their talk, they will be around all week and are excited about meeting new people.

When testing a Django application you often need to populate the test database with some sample data. The standard Django TestCase has support for fixture loading but there are a number of problems with using fixtures:

• First, they must be updated each time your schema changes.
• Second, they force you to hard-code dates which can create test failures when your date, which was “very far in the future” when the fixture was created, has now just passed.
• Third, fixtures are painfully slow to load. They are discovered, deserialized and the data inserted at the start of every test method. Then at the end of the test that transaction is rolled back. Many times you didn’t even use the data in the fixture.

What is the alternative to fixtures? It’s simple: your test cases should create the data they need. Let’s take a simple model:

# models.py
from django.db import models


class Thing(models.Model):
    name = models.CharField(max_length=100)
    description = models.CharField(max_length=100)

    def __unicode__(self):
        return self.name

Now we want to write some tests which need some Things in the database.

# tests.py
import random
import string

from django.test import TestCase

from .models import Thing


def random_string(length=10):
    return u''.join(random.choice(string.ascii_letters) for x in range(length))


class ThingTestCase(TestCase):

    def create_thing(self, **kwargs):
        "Create a random test thing."
        options = {
            'name': random_string(),
            'description': random_string(),
        }
        options.update(kwargs)
        return Thing.objects.create(**options)

    def test_something(self):
        # Get a completely random thing
        thing = self.create_thing()
        # Test assertions would go here

    def test_something_else(self):
        # Get a thing with an explicit name
        thing = self.create_thing(name='Foo')
        # Test assertions would go here

Instead of using a fixture we have a create_thing method to create a new Thing instance. In our tests we can get a new random thing object. In the cases where some of the fields need explicit values we can pass those into the creation. The tests create the exact number of Things that they need and any requirements about these instances is explicit in how they are created.

Writing these methods to create new instances can be rather repetitive. If you prefer, you can use something like Factory Boy to help you. Factory Boy is a Python port of a popular Ruby project called Factory Girl. It provides a declarative syntax for how new instances should be created. It also has helpers for common patterns such as sub-factories for foreign keys and other inter-dependencies. Rewriting our tests to use Factory Boy would look like this:

# tests.py
import random
import string

import factory
from django.test import TestCase

from .models import Thing


def random_string(length=10):
    return u''.join(random.choice(string.ascii_letters) for x in range(length))


class ThingFactory(factory.DjangoModelFactory):
    FACTORY_FOR = Thing

    name = factory.LazyAttribute(lambda t: random_string())
    description = factory.LazyAttribute(lambda t: random_string())


class ThingTestCase(TestCase):

    def test_something(self):
        # Get a completely random thing
        thing = ThingFactory.create()
        # Test assertions would go here

    def test_something_else(self):
        # Get a thing with an explicit name
        thing = ThingFactory.create(name='Foo')
        # Test assertions would go here

Here the create_thing method is removed in favor of the ThingFactory. Calling it is similar to how we were previously calling the method. One advantage of the ThingFactory is that we can also call ThingFactory.build() which will create an unsaved instance for tests where we don’t need the instance to be saved. Any time we can avoid a write to the database can save time for the test suite.

One handy pattern when working with models which have images is to have a test image file which is used by default. Let’s create a new model with an image.

# models.py
from django.db import models

class ImageThing(models.Model):
    name = models.CharField(max_length=100)
    image = models.ImageField(upload_to='images/')

    def __unicode__(self):
        return self.name

And a factory to test it

# tests.py
import os
import random
import string

import factory
from django.core.files import File

from .models import ImageThing

# Assumes there is a test.png next to our tests.py
TEST_IMAGE = os.path.join(os.path.dirname(__file__), 'test.png')


def random_string(length=10):
    return u''.join(random.choice(string.ascii_letters) for x in range(length))


class ImageThingFactory(factory.DjangoModelFactory):
    FACTORY_FOR = ImageThing

    name = factory.LazyAttribute(lambda t: random_string())
    image = factory.LazyAttribute(lambda t: File(open(TEST_IMAGE)))

Unlike the name this image isn’t random but by default all of your model instances will have images associated with them.

In summary, using fixtures for complex data structures in your tests is fraught with peril. They are hard to maintain and they make your tests slow. Creating model instances as they are needed is a cleaner way to write your tests which will make them faster and more maintainable. With tools like Factory Boy it can be very easy.

Do your mornings usually consist of reading design blogs and drinking coffee?  Are you obsessive about fonts? Are you constantly seeking out new inspiration to make your designs better?  As a Front-End Developer + Designer at Caktus, you’ll be able to put your passion for design and development to work by creating beautiful designs for complex websites. You will work closely with clients to bring their visions to life and help lead the branding and design process. Our designers take ownership of the user experience and design process from the beginning and collaborate with our development team to implement the vision. You should be passionate about the open source community and the philosophy behind it. If you’re interested in becoming part of and contributing to a creative, dynamic team, here’s an idea of what you’ll do:

1. Listen closely to clients and translate their needs into a comprehensive design and branding vision
   
2. Direct all the aspects of the design process, user experience, and front-end development on a diverse set of projects
3. Get your hands dirty with Django/HTML/CSS and run projects locally on your development laptop
4. Use your love of design and knowledge of user experience to create functional mockups and wireframes of proposed designs and user stories
5. Leverage your passion of the web to create dynamic, responsive-friendly prototypes and designs for small and large web apps
6. Use your ingenuity to create aesthetically pleasing designs that are also user-centric
7. Collaborate closely with developers and project managers to deliver sharp web applications on time and to dazzling results

For more information about this position, please see the full description on our site.

Caktus is proud to be a part of the launch of the PyCon 2014 in Montreal website. We were delighted to be selected as this year’s partner for software development and brand updates. In the past, we've enjoyed working with the Python Software Foundation as a collaborator for the Raspberry.io branding and development and are excited to help out with the PyCon website.

If you’ve ever written a test for a view or model with associated uploaded files you might have noticed a small problem with those files hanging around after the tests are complete. Since version 1.3, Django won’t delete the files associated with your model instances when they are deleted. Some work-arounds for this issue involve writing a custom delete for your model or using a post_delete signal handler. But even with those in place the files would not be deleted during tests because the model instances are not explicitly deleted at the end of the test case. Instead, Django simply rolls back the transaction and the delete method is never called nor are the signals fired. This can be quite an annoyance when running the tests repeatedly and watching your MEDIA_ROOT (or worse your S3 bucket) fill up with garbage data. More than annoyance, this introduces something you always want to avoid in unittests: global state.

One way to avoid this problem is to take a similar approach to how Django manages the database. That is, each time you run the tests, create a new MEDIA_ROOT and when the tests finish tear it down. This can all be done with a basic test runner class.

import shutil
import tempfile

from django.conf import settings
from django.test.simple import DjangoTestSuiteRunner


class TempMediaMixin(object):
    "Mixin to create MEDIA_ROOT in temp and tear down when complete."

    def setup_test_environment(self):
        "Create temp directory and update MEDIA_ROOT and default storage."
        super(TempMediaMixin, self).setup_test_environment()
        settings._original_media_root = settings.MEDIA_ROOT
        settings._original_file_storage = settings.DEFAULT_FILE_STORAGE
        self._temp_media = tempfile.mkdtemp()
        settings.MEDIA_ROOT = self._temp_media
        settings.DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'

    def teardown_test_environment(self):
        "Delete temp storage."
        super(TempMediaMixin, self).teardown_test_environment()
        shutil.rmtree(self._temp_media, ignore_errors=True)
        settings.MEDIA_ROOT = settings._original_media_root
        del settings._original_media_root
        settings.DEFAULT_FILE_STORAGE = settings._original_file_storage
        del settings._original_file_storage


class CusomTestSuiteRunner(TempMediaMixin, DjangoTestSuiteRunner):
    "Local test suite runner."

The current Django master (1.6 dev) has a new discovery test runner. You can also make this mixin work with this new runner.

# Requires Django 1.6+
from django.test.runner import DiscoveryRunner

class CusomTestSuiteRunner(TempMediaMixin, DiscoveryRunner):
    "Local test suite runner."

This code can go anywhere on your Python import path. You might choose to put this into a runner.py your project module, next to the settings.py and the root urls.py. In that case you would update your settings with

# Here {{ project_name }} would be replaced by the name of your project module
TEST_RUNNER = '{{ project_name }}.runner.CusomTestSuiteRunner'

This is written as a mixin so that you could add the same functionality to an existing test runner such as the one provided by django-jenkins, django-discovery-runner or django-celery. While this doesn’t entirely avoid the global state between tests, it does avoid problems between test runs and keeps the tests from filling up your local MEDIA_ROOT.