A planet of blogs from our members...

Caktus GroupCSS Grid, not Frameworks, are the Future

At the 2016 An Event Apart Conference in San Francisco, I peeked under the hood of a new technology that would finally address all the layout woes that we as designers and developers face: CSS Grid Layout Module. At first I was a little skeptical - except for Microsoft Edge, browser support for Grid is currently non-existent - however its official release is actually not that far off. Currently it is enabled behind a flag in Chrome and Firefox, or you can download the latest nightly or developer versions of Firefox or Safari. Here’s my brief synopsis of why I think CSS Grid is going to change the landscape of the web forever, and why I think it’s so important from a design and developer perspective.

Many website designs today are stuck in what I would call an aesthetic rut. That is, they are all comprised of similar design patterns (similar icons, sections, hero images, etc.) and are structured with common layout patterns. As many speakers at the conference pointed out, this gets boring, fast. The CSS Grid Layout Module is meant to address these concerns by implementing a dynamic method of creating elegant layouts easily, and across two dimensions. Where Flexbox only handled layout in one dimension at a time (either column or row direction), CSS Grid handles layout for columns and rows simultaneously. CSS Grid makes possible what we used to do in traditional print layout: the utilization of white space to create movement and depth, with very little code that is both responsive and easily adaptable to new content.

CSS Grid involves very little markup. A simple display: grid with its subset of attributes is all it takes. Rather than bore you with examples, check out this nifty guide. What used to comprise hundreds of lines of code wrapped in a framework (Bootstrap, Foundation, Skeleton) is now accomplished with a few lines, and presumably fewer dependencies mean an increase in performance and decrease in page load times. Grid is a great tool to prototype and design with, simply because you can now get up and running with no setup or dependencies - everything is baked into the browser.

The true power and beauty of Grid is that it allows for both complete control over layout placement, or you can let the browser do the work. You can specify column (or row) spacing, and have CSS Grid decide where to place your content. If you want to leverage more control over where your content goes on the page, you can specify where it goes with grid-column: start line/end line or grid-row: start line/end line, or a combination of both.

One of the most exciting things about CSS Grid is that we can use it now to prototype and plan for the future. My challenge to you, as designers and developers, to use it now so that when CSS Grid is released, not only will your project already take advantage of all the new and wonderful possibilities using CSS Grid, you will have also adapted the future friendly approach for your project. Need inspiration? Check out My reinterpretation of a Japanese Magazine Cover with CSS Grid.

Caktus GroupDjango Under the Hood 2016 Recap

Caktus was a proud sponsor of Django Under the Hood (DUTH) 2016 in Amsterdam this year. Organized by Django core developers and community members, DUTH is a highly technical conference that delves deep into Django.

Django core developers and Caktus Technical Manager Karen Tracey and CEO/Co-founder Tobias McNulty both flew to Amsterdam to spend time with fellow Djangonauts. Since not all of us could go, we wanted to ask them what Django Under the Hood was like.

Can you tell us more about Django Under the Hood?

Tobias: This was my first Django Under the Hood. The venue was packed. It’s an in-depth, curated talk series by invite-only speakers. It was impeccably organized. Everything is thought through. They even have little spots where you can pick up toothbrush and toothpaste.

Karen: I’ve been to all three. They sell out very quickly. Core developers are all invited, get tickets, and some funding depending on sponsorship. This is the only event where some costs are covered for core developers. DjangoCon EU and US have core devs going, but they attend it however they manage to get funds for it.

What was your favorite part of Django Under the Hood?

Tobias: The talks: they’re longer and more detailed than typical conference talks; they’re curated and confined to a single track so the conference has a natural rhythm to it. I really liked the talks, but also being there with the core team. Just being able to meet these people you see on IRC and mailing list, there’s a big value to that. I was able to put people in context. I’d met quite a few of the core team before but not all.

Karen: I don’t have much time to contribute to Django because of heavy involvement in cat rescue locally and a full time job, but this is a great opportunity to have at least a day to do Django stuff at the sprint and to see a lot of people I don’t otherwise have a chance to see.

All the talk videos are now online. Which talk do you recommend we watch first?

Karen: Depends on what you’re interested in. I really enjoyed the Instagram one. As someone who contributed to the Django framework, to see it used and scaled to the size of Instagram 500 million plus users is interesting.

Tobias: There were humorous insights, like the Justin Bieber effect. Originally they’d shared their database by user ID, so everybody on the ops team had memorized his user ID to be prepared in case he posted anything. At that scale, maximizing the number of requests they can serve from a single server really matters.

Karen: All the monitoring was interesting too.

Tobias: I liked Ana Balica’s testing talk. It included a history of testing in Django, which was educational to me. Django didn’t start with a framework for testing your applications. It was added as a ticket in the low thousands. She also had practical advice on how to treat your test suite as part of the application, like splitting out functional tests and unit tests. She had good strategies to make your unit tests as fast as possible so you can run them as often as needed.

What was your favorite tip or lesson?

Tobias: Jennifer Akullian gave a keynote on mental health that had a diagram of how to talk about feelings in a team. You try to dig into what that means. She talked about trying to destigmatize mental health in tech. I think that’s an important topic we should be discussing more.

Karen: I learned things in each of the talks. I have a hard time picking out one tip that sticks with me. I’d like to look into what Ana Balica said about mutation testing and learn more about it.

What are some trends you’re seeing in Django?

Karen: The core developers met for a half-day meeting the first day of the conference. We talked about what’s going on with DJango, what’s happened in the past year, what’s the future of Django. The theme was “Django is boring.”

Tobias: “Django is boring” because it is no longer unknown. It’s an established, common framework now used by big organizations like NASA, Instagram, Pinterest, US Senate, etc. At the start, it was a little known bootstrappy cutting edge web framework. The reasons why we hooked up with Django nine years ago at Caktus, like security and business efficacy, all of those arguments are ever so much stronger today. That can make it seem boring for developers but it’s a good thing for business.

Karen: It’s been around for awhile. Eleven years. A lot of the common challenges in Django have been solved. Not that there aren’t cutting edge web problems. But should you solve some problems elsewhere? For example, in third party, reusable apps like channels, REST framework.

Tobias: There was also recognition that Django is so much more than the software. It’s the community and all the packages around it. That’s what make Dango great.

Where do you see Django going in the future?

Karen: I hate those sorts of questions. I don’t know how to answer that. It’s been fun to see the Django community grow and I expect to see continued growth.

Tobias: That’s not my favorite question either. But Django has a role in fostering and continuing to grow the community it has. Django can set an example for open source communities on how to operate and fund themselves in sustainable ways. Django is experimenting with funding right now. How do we make open source projects like this sustainable without relying on people with full-time jobs volunteering their nights and weekends? This is definitely not a “solved problem,” and I look forward to seeing the progress Django and other open source communities make in the coming years.

Thank you to Tobias and Karen for sharing their thoughts.

Caktus GroupOn building relationships - Digital Project Management Summit Recap

Photo of Elizabeth speaking to DPM 2016 Summit by David Jordan.

When I first became a digital project manager, I struggled to find professional resources. There was a plethora of information available for traditional project management, but not much specifically for digital project management. Luckily, a colleague recommended the Digital PM Summit, sponsored by the Bureau of Digital.

It's one of the first, and still one of the only, professional conferences in the United States for digital project managers, and it’s grown every year. I initially attended the Summit three years ago in Austin, TX and it was an eye-opening, informative, and motivational experience. I met many people who did the same work that I did! I don't know where they were hiding before, but I was thankful to finally connect with them. It was such a relief to learn that others had the same challenges that I did, and that I was not alone.

I attended the Summit every year since, and this year, I was invited to speak. I was one of twenty-two expert speakers and I was thrilled about the opportunity to present on one of the most important aspects of digital project management—relationships. I’ve found that positive working relationships are key not only to project success, but also to my success, my team’s success, and our client’s success. As project managers, we must focus on process and logistics to deliver quality projects on time, and all of that involves people.

Investing in Relationships is Key to Project Success

Projects are always about people, no matter where you work, and no matter what the project involves. Building positive working relationships can be challenging, but I’ve found that the best project managers invest in their team, clients, and stakeholders’ success, and not just in the project’s success. While it is possible to launch a project that successfully meets its goals, if the people involved are miserable, was it really a success? After all, the project is not going to pat you on the back, but the people involved would.

I became a better project manager when I realized the importance of relationships, and when I recognized how much I could impact the people around me. Several years ago, as a brand new PM, I didn’t have the confidence that I do now, and it was difficult for me to take the lead. After a couple years, and thanks in large part to the Digital PM Summit, through which I learned skills that I could apply on the job, I became a more effective PM. I’m more flexible and adaptable, which is key to collaboration, and my interpersonal communication skills have improved.

The importance of collaboration and communication were key points within my Digital PM Summit presentation, “Think Outside the Project Management Triangle.” The Project Management Triangle, or Iron Triangle, is a widely-known model of the typical constraints of project management that impact project quality—resources (budget and workers), project scope (features and functionality), and schedule (time and prioritization)—these are all components that project managers must consider and work with. In my experience, the Triangle is too limiting and overlooks relationships. The Triangle is a good basic model, but the best PMs think outside of the Triangle to positively leverage relationships in order to balance resources, scope, and schedule.

Project Management Triangle

The Project Management Triangle, or Iron Triangle

Approximately fifty project managers attended my talk at the Digital PM Summit on October 13, 2016. By that point, I’d worked at Caktus for four weeks, which impacted my presentation because it was the first time I’ve worked with external clients and the first time I wasn’t a lone PM.

As an established, full service Django shop, Caktus includes a team of trained PMs who provide professional project management services to clients. Working with other PMs helped me feel more at home at Caktus, and I learned a lot from them in a few weeks. For example, the PM team taught me about the Agile Scrum process, which I was familiar with, but never practiced before. Scrum includes a product owner who serves as an extension of the client, championing the client’s goals and priorities to the development team. At Caktus, project managers also act as product owners. During the Digital PM Summit, some attendees were curious about how I made the shift from working in-house to working with external clients, and how the Scrum process impacted my transition. I was happy to inform them that while working with external clients is different from working in-house, there are still similarities, and that Scrum had been a refreshing change for me.

Unity in the Project Management Community Raises our Standards

It’s not unusual for a PM to be a lone wolf, like I was in my last job where I was connected with only one other digital PM who was in a different department. We quickly became friends and confidants based on our shared experiences. As a new digital PM, support from others is critical to success, and I’m glad the Bureau of Digital, which hosted the Digital PM Summit, provides a platform for project managers to connect and share their knowledge during and after the conference. I was honored to support their mission with my own presentation, and as it turned out, relationship building was a main theme during this year’s Summit.

The Bureau of Digital’s leaders, Brett Harned, Carl Smith, and Lori Averitt have increasingly focused on building a supportive community of professional project managers through events like the Summit. This year, the conference brought together 223 talented individuals, and the conversations have not stopped, thanks to Slack, Twitter, and LinkedIn. The attendees are still sharing tips, tools, and strategies with each other, and they’re forming Meetup groups. Since digital project management is still evolving and growing, conversations and collaboration among practitioners and experts is crucial to creating a greater shared understanding of best practices and to raise industry standards as well as recognition, helping all of us to better serve our clients and teams. I’m thrilled to work at a place like Caktus that recognizes the value of digital project management, and supports my engagement within the PM community.

The relationships I’ve made and the community support that I’ve received via the Digital PM Summit has been integral to my growth and success as a digital project manager. I would not be where I am today, and I certainly would not have presented at the Digital PM Summit, without support. What it comes down to is that no matter who you are or what your job is, none of us work or live in a bubble, and none of us are an island. We depend upon others. Perhaps Carl Smith, one of the conference organizers, said it best: "When you invest in others, they invest in you.”

Additional Links

Thinking Outside the Project Management Triangle

Caktus GroupRapidCon 2016: RapidPro Developer's Recap

Developer Erin Mullaney was just in Amsterdam for RapidCon, a UNICEF-hosted event for developers using RapidPro, an SMS tool built on Django. The teams that have worked on RapidPro and its predecessor RapidSMS have gotten to know each other virtually over the years. This marks the second time they’ve all come from across the globe to share learnings on RapidPro and to discuss its future.

RapidPro has the potential to transform how field officers build surveys, collect data, and notify populations. It allows users with no technical background to quickly build surveys and message workflows. With over 100% cell phone saturation in many developing regions, SMS presents a cheap, fast means of reaching many quickly.

Erin worked closely with UNICEF Uganda in the development of a data analytics and reporting tool called TracPro for RapidPro. The organizers invited her to speak about the tool with other RapidPro users.

How was the conference?

Erin: The conference was amazing and I was ecstatic to go. Meeting the folks who work at UNICEF for the first time was exciting because we normally only speak via audio over Skype. It was nice to see them in person. We had an evening event, so it was fun to get to know them better in a social atmosphere. It was also a great opportunity to get together with other technical people who are very familiar with RapidPro and to think about ways we could increase usage of this very powerful product.

What was your talk about?

Erin: The title of my talk was “TracPro: How it Works and What it Does”. TracPro is an open source dashboard for use with RapidPro. You can use it for activities like real-time monitoring of education surveys. Nyaruka originally built it for UNICEF and it’s now being maintained by Caktus.

I was one of two developers who worked on TracPro at Caktus. We worked to flesh out the data visualizations including bar charts, line charts over date ranges and maps. We also improved celery tasks and added other features like syncing more detailed contact data from RapidPro.

What do you hope your listeners came away with?

Erin: I delved into the code for how we synced data locally via Celery and the RapidPro API and how we did it in a way that is not server-intensive. I also had examples on how to build the visualizations. Both of those features were hopefully helpful for people thinking of building their own dashboards. Building custom dashboards in a short amount of time is really easy and fun. For example, it took a ShipIt Day I to build a custom RapidPro dashboard for PyCon that called the RapidPro API.

What did you learn about RapidCon?

Erin: People discussed the tools they were building. UNICEF talked about a new project, eTools, being used for monitoring. That sounds like an interesting project that will grow.

RapidPro has had exponential usage and growth and Nyaruka and UNICEF are working really hard to manage that. It was interesting to learn about the solutions Nyaruka is looking at to deal with incredibly large data sets from places with a ton of contacts. They’ll be erasing unnecessary data and looking at other ways to minimize these giant databases.

UNICEF is pretty happy with how RapidPro is working now and don’t expect to add too many new features to it. They’re looking ahead to managing dashboard tools like TracPro. So their focus is really on these external dashboards and building them out. The original RapidPro was really not for dashboards.

What was the best part of RapidCon for you?

Erin: It was pretty cool to be in a room and selected for this. I was one of only two women. Having them say “You have this knowledge that other developers don’t have” was rewarding. I felt like I had a value-add to this conference based on the past year and a half working on RapidPro-related projects.

Will you be sharing more of your RapidPro knowledge in the future?

Erin: So far, we’ve been the only one giving a talk about RapidPro, it seems. I gave a RapidPro talk at PyData Carolinas this year with Rebecca Muraya, Reach More People: SMS Data Collection with RapidPro and during a PyCon 2016 sponsor workshop. I’ve been encouraged to give this talk at more conferences and spread the word about RapidPro in order to get the word out further. I plan to submit it to a few 2017 conferences for sure!

Thank you Erin for sharing your experience with us!

To view another RapidPro talk Erin gave during PyData 2016 Carolinas, view the video here.

Tim HopperData Scientists Need More Automation

Many data scientists aren't lazy enough.

Whether we are managing production services or running computations on AWS machines, many data scientists are working on computers besides their laptops.

For me, this often takes the form of SSH-ing into remote boxes1, manually configuring the system with a combination of apt installs, Conda environments, and bash scripts.

To run my service or scripts, I open a tmux window, activate my virtual environement, and start the process.2

When I need to check my logs or see the output, I SSH back into each box, reconnect to tmux (after I remember the name of my session), and tail my logs. When running on multiple boxes, I repeat this process N times. If I need to restart a process, I flip through my tmux tabs until I find the correct process, kill it with a Ctrl-C, and use the up arrow to reload the last run command.

All of this works, of course. And as we all know, a simple solution that works can be preferable to a fragile solution that requires constant maintenance. That said, I suspect many of us aren't lazy enough. We don't spend enough time automating tasks and processes. Even when we don't save time by doing it, we may save mental overhead.

I recently introduced several colleagues to some Python-based tools that can help. Fabric is a "library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks." Fabric allows you to encapsulate sequences of commands as you might with a Makefile. It's killer feature is the ease with which it lets you execute those commands on remote machines over SSH. With Fabric, you could tail all the logs on all your nodes with a single command executed in your local terminal. There are a number of talks about Fabric on Youtube if you want to learn more. One of my colleagues reduced his daily workload by writing his system management tasks into a Fabric file.

Another great tool is Supervisor. If you run long running processes in tmux/screen/nohup, Supervisor might be for you. It allows you to define the tasks you want to run in an INI file and "provides you with one place to start, stop, and monitor your processes". Supervisor will log the stdout and stderr to a log location of your choice. It can be a little confusing to set up, but will likely make your life easier in the longer run.

A tool I want to learn but haven't is Ansible, "a free-software platform for configuring and managing computers which combines multi-node software deployment, ad hoc task execution, and configuration management". Unlike Chef and Puppet, Ansible doesn't require an agent on the systems you need to configure; it does all the configuration over SSH. You can use Ansible to configure your systems and install your dependencies, even Supervisor! Ansible is written in Python and, mercifully, doesn't require learning a Ruby-based DSL (as does Chef).

Recently I've been thinking that Fabric, Supervisor, and Ansible combined become a powerful toolset for management and configuration of data science systems. Each tool is also open source and can be installed in a few minutes. Each tool is well documented and offers helpful tutorials on getting started; however, learning to use them effectively may require some effort.

I would love to see someone create training materials on these tools (and others!) focused on how data scientists can take improve their system management, configuration, and operations. A screencast series may be the perfect thing. Someone please help data scientists be lazier, do less work, and reduce the mental overhead of dealing with computers!


  1. Thankfully I recently started taking better advantage of aliases in my ssh config

  2. When I have to do this on multiple machines, I'm occasionally clever enough to use tmux to broadcast the commands to multiple terminal windows. 

Caktus GroupCommon web site security vulnerabilities

I recently decided I wanted to understand better what Cross-Site Scripting and Cross-Site Request Forgery were, and how they compared to that classic vulnerability, SQL Injection.

I also looked into some ways that sites protect against those attacks.

Vulnerabilities

SQL Injection

SQL Injection is a classic vulnerability. It probably dates back almost to punch cards.

Suppose a program uses data from a user in a database query.

For example, the company web site lets users enter a name of an employee, free-form, and the site will search for that employee and display their contact information.

A naive site might build a SQL query as a string using code like this, including whatever the user entered as NAME:

"SELECT * FROM employees WHERE name LIKE '" + NAME + "'"

If NAME is "John Doe", then we get:

SELECT * FROM employees WHERE name LIKE 'John Doe'

which is fine. But suppose someone types this into the NAME field:

John Doe'; DROP TABLE EMPLOYEES;

then the site will end up building this query:

SELECT * FROM employees WHERE name LIKE 'John Doe'; DROP TABLE EMPLOYEES;'

which might delete the whole employee directory. It could instead do something less obvious but even more destructive in the long run.

This is called a SQL Injection attack, because the attacker is able to inject whatever they want into a SQL command that the site then executes.

Cross Site Scripting

Cross Site Scripting, or XSS, is a similar idea. If an attacker can get their Javascript code embedded into a page on the site, so that it runs whenever someone visits that page, then the attacker's code can do anything on that site using the privileges of the user.

For example, maybe an attacker posts a comment on a page that looks to users like:

Great post!

but what they really put in their comment was:

Great post!<script> do some nefarious Javascript stuff </script>

If the site displays comments by just embedding the text of the comment in the page, then whenever a user views the page, the browser will run the Javascript - it has no way to know this particular Javascript on the page was written by an attacker rather than the people running the site.

This Javascript is running in a page that was served by the site, so it can do pretty much anything the user who is currently logged in can do. It can fetch all their data and send it somewhere else, or if the user is particularly privileged, do something more destructive, or create a new user with similar privileges and send its credentials somewhere the bad guy can retrieve them and use them later, even after the vulnerability has been discovered and fixed.

So, clearly, a site that accepts data uploaded by users, stores it, and then displays it, needs to be careful of what's in that data.

But even a site that doesn't store any user data can be vulnerable. Suppose a site lets users search by going to http://example.com/search?q=somethingtosearchfor (Google does something similar to this), and then displays a page showing what the search string was and what the results were. An attacker can embed Javascript into the search term part of that link, put that link somewhere people might click on it, and maybe label it "Cute Kitten Pictures". When a user clicks the link to see the kittens, her browser visits the site and tries the search. It'll probably fail, but if the site embeds the search term in the results page unchanged (which Google doesn't do), the attacker's code will run.

Why is it called Cross-Site Scripting? Because it allows an attacker to run their script on a site they don't control.

CSRF

Cross Site Request Forgeries

The essence of a CSRF attack is a malicious site making a request to another site, the site under attack, using the current user's permissions.

That last XSS example could also be considered a CSRF attack.

As another, extreme example, suppose a site implemented account deletion by having a logged-in user visit (GET) /delete-my-account. Then all a malicious site would have to do is link to yoursite.com/delete-my-account and if a user who was logged into yoursite.com clicked the link, they'd make the /delete-my-account request and their account would be gone.

In a more sophisticated attack, a malicious site can build a form or make AJAX calls that do a POST or other request to the site under attack when a user visits the malicious site.

Protecting against vulnerabilities

Protections in the server and application

SQL Injection protection

Django's ORM, and most database interfaces I've seen, provide a way to specify parameters to queries directly, rather than having the programmer build the whole query as a string. Then the database API can do whatever is appropriate to protect against malicious content in the parameters.

XSS protection

Django templates apply "escaping" to all embedded content by default. This marks characters that ordinarily would be special to the browser, like "<", so that the browser will just display the "<" instead of interpreting it. That means if content includes "<SCRIPT>...</SCRIPT>", instead of the browser executing the "..." part, the user will just see "<SCRIPT>...</SCRIPT>" on the page.

CSRF protection

We obviously can't disable links to other sites - that would break the entire web. So to protect against CSRF, we have to make sure that another site cannot build any request to our site that would actually do anything harmful.

The first level of protection is simply making sure that request methods like GET don't change anything, or display unvalidated data. That blocks the simplest possible attack, where a simple link from another site causes harm when followed.

A malicious site can still easily build a form or make AJAX calls that do a POST or other request to the site under attack, so how do we protect against that?

Django's protection is to always include a user-specific, unguessable string as part of such requests, and reject any such request that doesn't include it. This string is called the CSRF token. Any form on a Django site that does a POST etc has to include it as one of the submitted parameters. Since the malicious site doesn't know the token, it cannot generate a malicious POST request that the Django site will pay any attention to.

Protections in the browser

Modern browsers implement a number of protections against these kinds of attacks.

"But wait", I hear you say. "How can I trust browsers to protect my application, when I have no control over the browser being used?"

I frequently have to remind myself that browser protections are designed to protect the user sitting in front of the browser, who for these attacks, is the victim, not the attacker. The user doesn't want their account hacked on your site any more than you do, and these browser protections help keep the attacker from doing that to the user, and incidentally to your site.

Same-origin security policy

All modern browsers implement a form of Same Origin Policy, which I'll call SOP. In some cases, it prevents a page loaded from one site from accessing resources on other sites, that is, resources that don't have the same origin.

The most important thing about SOP is that AJAX calls are restricted by default. Since an AJAX call can use POST and other data-modifying HTTP requests, and would send along the user's cookies for the target site, an AJAX call could do anything it wanted using the user's permissions on the target site. So browsers don't allow it.

What kind of attack does this prevent? Suppose the attacker sets up a site with lots of cute kitten pictures, and gets a user victim to access it. Without SOP, pages on that site could run Javascript that made AJAX calls (in the background) to the user's bank. Such calls would send along whatever cookies the user's browser had stored for the bank site, so the bank would treat them as coming from the user. But with SOP, the user's browser won't let those AJAX calls to another site happen. They can only talk to the attacker's own site, which doesn't do the attacker any good.

CSP

Content Security Policy (CSP)

CSP is a newer mechanism that browsers can use to better protect from these kinds of attacks.

If a response includes the CSP header, then by default the browser will not allow any inline javascript, CSS, or use of javascript "eval" on the page. This blocks many forms of XSS. Even if an attacker manages to trick the server into including malicious code on the page, the browser will refuse to execute it.

For example, if someone uploads a comment that includes a <script> tag with some Javascript, and the site includes that in the page, the browser just won't run the Javascript.

Conclusion

I've barely touched the surface on these topics here. Any web developer ought to have at least a general knowledge of common vulnerabilities, if only to know what areas might require more research on a given project.

A reasonable place to start is Django's Security Overview.

The OWASP Top Ten is a list of ten of the most commonly exploited vulnerabilities, with links to more information about each. The ones I've described here are numbers 1, 3, and 8 on the list, so you can see there are many more to be aware of.

Caktus GroupManaging multiple Python projects: Virtual environments

Even Python learning materials that get into very advanced language features rarely mention some practical things that would be very helpful to know as soon as you start working on more serious projects, like:

  • How to install packages written by others so that your code can use them, without just copying the files into your own project.
  • How to work on multiple projects on one computer that might depend on different packages, and even different versions of the same packages, without them interfering with each other.

The key concept that helps to manage all this is the "virtual environment".

A virtual environment is a way of giving each of your Python projects a separate and isolated world to run in, with its own version of Python and installed libraries. It’s almost like installing a completely separate copy of Python for each project to use, but it’s much lighter weight than that.

When you create a virtual environment named "foo", somewhere on your computer, a new directory named "foo" is created. There's a "bin" directory inside it, which contains a "python" executable. When you run that python executable, it will only have access to the python built-in libraries and any libraries that have been installed "inside" that virtual environment.

Using a Virtual Environment

When working at the command line, you can put the virtual environment's "bin" directory first on your PATH, what we call "activating" the environment, and from then on, anytime you run python, you'll be running in the environment.

(This is one advantage of starting your scripts with:

#!/usr/bin/env python

rather than:

#!/usr/bin/python

By using the "/usr/bin/env" version, you'll get the first copy of Python that's on your PATH, and if you've activated a virtual environment, your script will run in that environment.)

Virtual environments provide a "bin/activate" script that you can source from your shell to activate them, e.g.:

$ . /path/to/virtualenv/foo/bin/activate
$ python
... runs the python from virtual environment "foo"

(but be sure to notice that you have to source the activate script, using . or source, and running the script normally will not activate your virtual environment as you might expect).

After activating a virtual environment in a shell, a new command deactivate will become available, that will undo the activation.

Activation is just a convenience for use in an interactive shell. If you're scripting something, just invoke Python or whatever other script or executable you need directly from the virtual environment's bin directory. E.g.:

/path/to/env/bin/python myprogram.py

or:

/path/to/env/bin/script

Anything that's been installed in the virtual environment and has executables or scripts in the bin directory can be run from there directly and will run in the virtual environment.

Installing packages into a virtual environment

In an activated virtual environment, you'll have a command pip that you'll use to install, update, and remove packages. E.g.:

$ pip install requests==2.11.1

will make the requests package, version 2.11.1, available to Python programs running in that virtual environment (and no other). It works by installing it in the lib directory within the virtual environment, which will only be on the library path for Python when running under that virtual environment.

When the requests package releases version 2.12, you can upgrade to it with:

$ pip install -U requests==2.12

or just upgrade to the latest version with:

$ pip install -U requests

If you decide that your project no longer needs requests, you can remove it:

$ pip uninstall requests

In practice, we usually put a project's requirements in a file in the project's top directory named requirements.txt, one per line, and then use the -r option of pip to install them all at once. So we might have this in requirements.txt:

requests==2.12
psycopg2==2.6.1

and then install all that with this command:

$ pip install -r requirements.txt

Of course, pip has all sorts of options and other features, which you can read about in the Pip documentation.

Pip outside of virtual environments

Pip is automatically provided for you inside all virtual environments, but you can also install it system-wide and use it to install Python packages for the whole system. But there are some things to be careful of:

  • Be sure to be aware whether you're in a virtual environment or not; it's much more likely that you intend to install something in a virtual environment, but if you run pip outside of one, it'll happily install things to the whole system if you have permissions.
  • Using pip to install things to the whole system can create conflicts with Python packages provided by your operating system distribution. Pip will typically install things under /usr/local while OS packaging will install things under /usr, which can help you figure out which version of things you're getting.

Creating virtual environments

Python started including direct support for virtual environments in version 3.3, but for versions before that, including Python 2.7, you need to use a third-party tool called virtualenv to create them.

virtualenv still works for all versions of Python, and since I still need to deal with Python 2.7 on a regular basis, and don't want to try to remember the details of two different ways to create virtual environments, I just use virtualenv for everything. Plus, I still think virtualenv is simpler to use and more flexible.

Here's how simple using virtualenv is:

$ virtualenv -p /path/to/python /path/to/new/virtualenv

So if I wanted to create a virtual environment at $HOME/foo that was running Python 2.7, I could run:

$ virtualenv -p /usr/bin/python2.7 $HOME/foo

That just creates the virtual environment. When you're ready to use it, don't forget to activate it first.

Note that whenever virtualenv creates a new virtual environment, it automatically installs pip into it. This avoids the chicken-and-egg problem of how to install pip into the virtual environment so you can install things into the virtual environment.

Installing virtualenv

You can often install virtualenv using a system package, but it will likely be an old version. Since virtualenv bundles pip, that means all virtual environments you create with it will start with an old version of pip, and will complain at you until you upgrade pip to the latest. (That's easy - pip install -U pip - but gets tiresome.)

I prefer to just install the latest version of virtualenv, and ironically, the best way to install the latest version of virtualenv system-wide is using pip.

First, you'll want to install pip on your system - that is, globally. If your distribution provides a package to install pip, use that to avoid possibly breaking your system's installed Python.

I'll try to provide instructions for a variety of systems, but be warned that the only one I'm currently using is Ubuntu.

For Debian or Ubuntu, you'd use sudo apt-get install python-pip. For RPM-based distributions, yum -y install python-pip. For Arch, try pacman -S python-pip. On Mac OS X, brew install python should install pip too.

Only if your distribution does not provide a way to install pip, then you'll need to install it directly. First, securely download get-pip.py. You can click that link and save the file somewhere, or use wget from the command line:

$ wget https://bootstrap.pypa.io/get-pip.py

Then run get-pip.py, with system privileges:

$ sudo python get-pip.py
Collecting pip
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 932kB/s
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 8.1MB/s
Installing collected packages: pip, wheel
Successfully installed pip-8.1.2 wheel-0.29.0

Now that you have a system pip, you can install virtualenv:

$ sudo pip install virtualenv

and anytime you want to make sure you have the latest version of virtualenv:

$ sudo pip install -U virtualenv

Learning more

virtualenv and pip are just parts of the whole virtual environment ecosystem for Python. They're enough to get a lot done, but here are some other things you might find interesting at some point.

  • virtualenvwrapper provides shell shortcuts to make working with virtual environments easier. For example, instead of having to type . path/to/my-venv/bin/activate, you can just type workon my-venv.

  • If you want to make your own package installable by pip, see the Python Packaging User Guide.

    (And send a word of thanks to the people behind it, the Python Packaging Authority. For many years, the packaging side of Python was a poorly documented mess of different, conflicting, poorly documented tools (did I mention the poor documentation?). In the last couple of years, the PPA has brought order out of this chaos and the Python development community is so much better for it.)

Caktus GroupPresidential debate questions influenced by open source platform

During the past two presidential debates, moderators from ABC and Fox asked candidates Hillary Clinton and Donald Trump voter-submitted questions from PresidentialOpenQuestions.com. The site, created by the bipartisan Open Debate Coalition (ODC) with the support of Caktus Group, is built on top of the open source Django web framework.

“This coalition effort is a first-of-its-kind attempt to ensure moderators can ask questions that are not just submitted by the public, but voted on by the public to truly represent what Republican, Democratic, and Independent families are discussing around their dinner tables. Open Debates are the future,” said Lilia Tamm Dixon, Open Debate Coalition Director.

Voters using PresidentialOpenQuestions.com submitted over 15,000 questions and cast more than 3.6 million votes for their favorite submissions. The selected debate questions had an unprecedented audience. According to Nielsen Media, 66.5 million viewers watched the second debate and 71.6 million the third debate.

The ODC and Caktus teams continue to make improvements to the platform, readying new versions for use in political debates around the country. For national media coverage on the Open Debate Coalition and to learn more about their goals, see articles from The Atlantic, The Los Angeles Times, and Politico.

Tim HopperUnderstanding Probabilistic Topic Models By Simulation

I gave a talk last week at Research Triangle Analysts on understanding probabilistic topic models (specificly LDA) by using Python for simulation. Here's the description:

Latent Dirichlet Allocation and related topic models are often presented in the form of complicated equations and confusing diagrams. Tim Hopper presents LDA as a generative model through probabilistic simulation in simple Python. Simulation will help data scientists to understand the model assumptions and limitations and more effectively use black box LDA implementations.

You can watch the video on Youtube:

I gave a shorter version of the talk at PyData NYC 2015.

Caktus GroupShipIt Day Recap Q3 2016

This ShipIt day marks four years of ShipIt days at Caktus! We had a wide range of projects that people came together to build. Most importantly, we all had fun and learned through actively working on the projects. People explored new technologies and tools, and had a chance to dig a bit deeper into items that piqued their interest in their regular work.

React + Django = django-jsx

Calvin did some work inspired by a client project to create tools for working with React’s JSX DOM manipulation within Django projects. This bridge allows embedding of JSX in Django templates (even using Django template language syntax) to be compiled and then rendered on the page. Calvin released django-jsx up on Github and pypi, and is interested in feedback from people who use it.

Django ImageField compression

Dmitriy continued working on the TinyPNG compressed Django ImageField from the previous ShipIt Day. He’s shared his updates through the Github repository django_tinypng. This time Dmitriy worked on cleaning up the project in preparation for possibly submitting it to pypi. His work included adding documentation and a nice way to migrate pre-existing image fields in projects to the new compressed image field.

Python microservices with asyncio

Dan explored the asyncio capabilities of Python 3 via a long standing project of his. He had a previous project to control displaying videos. Issues came up when the player would lose connectivity and Dan wanted his program to be able to dynamically recover. Dan dug into the asyncio documentation head first, but was a bit overwhelmed by the scope of the library. Luckily, he found an excellent write up by Doug Hellmann in his Python Module of the Week series. Dan used what he learned to build an event loop, and focused on making his project more resilient to handle errors gracefully.

More Python microservices with asyncio

Mark created a set of microservices working together including a worker, a web server to handle webhook requests, a web hook generator, and a logging server. These services communicated together using rabbitmq and asyncio. The work that Mark did on this was a fun but relevant diversion from his prep for his upcoming All Things Open talk next week on RabbitMQ and the Advanced Message Queueing Protocol.

Microsoft Azure provisioning and deployment

David worked with Microsoft Azure and compared it with our standard provisioning and deployment practices. He learned about how configuration and management tools around Azure compare to those of other cloud providers. As a test case, David built a test application using our SaltStack based django-project-template and worked on getting the test application up and running to identify any pain points.

Elixir

Neil shared with the team his explorations into using Elixir. Elixir is a functional language built on top of Erlang’s virtual machine (VM), but without some of the peculiarities of the older Erlang language. The Erlang VM was developed with extreme fault tolerance in mind for creating telecom software (eg. the electronic phone switching system) that would never go down and could even be upgraded without downtime. Neil delved into this high availability mindset by creating a test project with worker processes handling data storage and supervisor processes in place to restart failed worker processes. Overall, Neil found the exploration useful in that understanding a wide range of programming language paradigms helps you to think through challenges in any language, in different ways.

Selenium testing

Rebecca and Alex both worked on updating or adding front-end tests to projects via Selenium. Rebecca looked at updating the tests in django-scribbler in preparation of an upgrade of the open source project to support Django 1.10. Alex looked into using information from Mark Lavin’s 2015 DjangoCon talk on front-end testing and amazing documentation to add front-end tests to an existing project.

Implicit biases

Charlotte F., Liza, and Sarah facilitated Caktus team members taking an implicit bias test from Harvard’s Project Implicit. Caktus team members participated by taking the test and anonymously shared their results. Charlotte and Liza reviewed the responses and compared them to the average responses for all project implicit respondents. This was a great way for team members to examine and become aware of implicit biases that may arise when interacting with people inside and outside of Caktus.

Team availability forecasts

Sarah worked on a new tool for forecasting team availability via a tool that uses project schedules in their native format so that they do not need to be reformatted for forecasting. In working on this project, Sarah had a chance to learn some new spreadsheet creation tools and practice laying out a sheet in a way that can be sustainably maintained.

UX design case study for cat adoption

Charlotte M. and Basia Coulter invited everyone at Caktus to participate as clients would in a UX discovery workshop. Alley Cats and Angels is a local animal rescue that Caktus is particularly fond of. Alley Cats and Angels has an internal database to track information about cats getting enrolled in its programs, foster homes, adoption applications, adopters etc. It also has a public-facing website where its programs are described and relevant application forms can be submitted, and where cats available for adoption are featured. But there is no automated communication between the database and the public-facing website, and not all important information is being tracked in the database. That results in significant overhead of manual processes required to keep all information in order, and to facilitate programs. Using a story mapping technique, Charlotte and Basia worked with Caktii to map out a web-based application that would allow for an integration of the internal database and the public-website, automation of critical processes, and more complete information tracking. They identified important user flows and functionality, and broke them down into individual user stories, effectively creating a backlog of tasks that could be prioritized and leveraged in a sprint-based development process. They also determined which features were necessary for the first iteration of the application to deliver client value. By doing so, they defined a version of the minimum viable product for the application. At the end of the workshop they took some time to sketch paper prototypes of selected features and screens. The result of the workshop was a comprehensive set of deliverables (user flows, backlog of user stories, minimum viable product, and paper prototypes) that could serve as starting point for application development.

Tim HopperUndersampled Radio Interview

I was flattered to be asked to be on a burgeoning data science podcast called Undersampled Radio. You can listen here. We recorded the interview on a Google Hangout, so you can also watch it here.

Caktus GroupDon't keep important data in your Celery queue

The Celery library (previous posts) makes it as easy to schedule a task to run later as calling a function. Just change:

send_welcome_email('dan@example.com')

to:

send_welcome_email.apply_async('dan@example.com')

and rely on Celery to run it later.

But this introduces a new point of failure for your application -- if Celery loses track of that task and never runs it, then a user will not get an email that we wanted to send them.

This could happen, for example, if the server hosting your Celery queue crashed. You could set up a hot backup server for your queue server, but it would be a lot of work.

It's simpler if you treat your Celery queue like a cache -- it's helpful to keep around, but if you lose it, your work will still get done.

[Ed: the code snippets in this post are just intended as little illustrations to go with the ideas in this post, and by no means should they be taken as examples of how to implement these ideas in production. ]

We can do that by changing our pattern for doing things in the background. The key idea is to keep the information about the work that needs to be done in the database, with the rest of our crucial data.

For example, instead of:

send_welcome_email.apply_async('dan@example.com')

we might add a needs_welcome_email Boolean field to our model, and write:

user.needs_welcome_email = True
user.save()

Now we know from our database that this user needs to get a welcome email, independently of Celery's queue.

Then we set up a periodic task to send any emails that need sending:

@task
def send_background_emails():
    for user in User.objects.filter(needs_welcome_email=True):
        send_welcome_email(user.email)
        user.needs_welcome_email = False
        user.save()

We can run that every 5 minutes, it'll be quite cheap to run if there's no work to do, and it'll take care of all the "queued" welcome emails that need sending.

If we want the user to get their email faster, we can just schedule another run of the background task immediately:

user.needs_welcome_email = True
user.save()
send_background_emails.apply_async()

And the user will get their email as fast as they would have before.

We will still want to run the background task periodically in case our queued task gets lost (the whole point of this), but it doesn't have to run as frequently since it will rarely have any work to do.

By the way, I learned this while doing a code review of some of my co-worker Karen's code. This ability to continue learning is one of my favorite benefits of working on a team.

Expiring tasks

Now that we've made this change, it opens up opportunities for more improvements.

Suppose we're scheduling our periodic task in our settings like this:

CELERYBEAT_SCHEDULE = {
    'process_new_work': {
        'task': 'tasks.send_background_emails',
        'schedule': timedelta(minutes=15),
    },
}

Every 15 minutes, celery will schedule another execution of our background task, and if all is well, it'll run almost immediately.

But suppose that our worker is unavailable for a while. (Maybe it lost connectivity temporarily.) Celery will keep on queuing our task every 15 minutes. If our worker is down for a day, then when it comes back, it'll see 24*4 = 96 scheduled executions of our task, and will have to run the task 96 times.

In our case, we're not scheduling our task all that frequently, and the task is pretty lightweight. But I've seen times when we had thousands of tasks queued up, and when the workers were able to resume, the server was brought to its knees as the workers tried to run them all.

We know that we only need to run our task once to catch up. We could manually flush the queue and let the next scheduled task handle it. But wouldn't it be simpler if Celery knew the tasks could be thrown away if not executed before the next one was scheduled?

In fact, we can do just that. We can add the expires option to our task when we schedule it:

CELERYBEAT_SCHEDULE = {
    'process_new_work': {
        'task': 'tasks.send_background_emails',
        'schedule': timedelta(minutes=15),
        'options': {
            'expires': 10*60,  # 10 minutes
        }
    },
}

That tells Celery that if the task hasn't been run within 10 minutes of when we schedule it, it's not worth running at all and to just throw it away. That's fine because we'll schedule another one in 5 more minutes.

So now what happens if our workers stop running? We continue adding tasks to the queue - but when we restart the worker, most of those tasks will have expired. Every time the worker comes to a task on the queue that is expired, it will just throw it away. This allows it to catch up on the backlog very quickly without wasting work.

Conclusion

As always, none of these techniques are going to be appropriate in all cases. But they might be handy to keep in your toolbox for the times when they might be helpful.

Caktus GroupPyData Carolinas 2016 Recap

We had a great time at the inaugural PyData Carolinas hosted nearby at IBM in the Research Triangle Park. People from Caktus presented a number of talks and the videos are now up online:

There were also many more fascinating talks about how people in and around North and South Carolina are using Python to do data analysis with Pandas, Jupyter notebooks, and more. It was a great event that brought together the strong communities around data and Python locally to celebrate their overlapping interests. We had a great time meeting folks and reconnecting with old friends at the after hours events hosted by MaxPoint and the Durham Convention & Visitors Bureau. Many thanks to all of the local organizers and sponsors who worked together to put on a great program and event. We can’t wait until the next one!

Caktus GroupHow to tell if you’re building great user experience

In web and software development, when we talk about user experience, we usually mean the experience you have when you are using a website or an application. Does it feel like you know where to click to get to your next destination? Do you know what to do in order to accomplish a task? Has anything you’ve clicked taken you to an unexpected page? Are you getting frustrated with or are you enjoying the website or app? Answers to these and similar questions are what describes the experience you’re having as a user. That’s user experience.

Nobody wants to deliver a bad experience. It’s a common sense approach to build applications that people will want to use, especially if your bottom line depends on the adoption and usage of the application you’re building. Ultimately, it’s not cutting edge technology that’ll make your application successful; it’s the people who will pay for your service or product. And people who use technology have increasingly higher expectations about the experience your product offers.

While there is a gut-level understanding around the idea that a happy customer is a customer more likely to pay, there seems to be less of a recognition that if a customer happens to be a user (of your application or website), it is the user you need to make happy.

“Numerous industry studies have stated that every dollar spent on UX brings in between $2 and $100 dollars in return.” Peter Eckert, FastDesign

Product Stakeholders, Designers, or Developers Usually Aren’t the Users

It turns out that building great user experience isn’t easy. But it is possible if you invest the time in understanding your users and their predicaments. One of the most common mistakes made in software design and development is forgetting that product stakeholders, designers, or developers usually are not the user segments for whom the application is being built. Any interface or application flow you design without understanding your target users’ pain points, their current workflows, and their needs and wants, will be but an assumption. And as humans we tend to assume that others would think and act as we do. So long as you let your assumptions go untested, you’re not really building for your users.

Methods to Build Great User Experience

There are methods that help forge an understanding of users and the contexts within which they function. You can employ these methods before any design and development work is commenced, during the product design and development processes, and after the product is built and released. For the purpose of this article, I’ve chosen to group those methods as follows:

  • Product discovery
  • Qualitative user research (user interviews, contextual inquiry)
  • Usability testing (testing your application with real users)
  • Quantitative user research (data analytics and user surveys)
  • Heuristic evaluation (UX review)

“While it's not realistic to use the full set of methods on a given project, nearly all projects would benefit from multiple research methods and from combining insights.” Christian Rohrer, Nielsen Norman Group

  • A new project should start with a discovery phase, during which you build a shared understanding with all stakeholders about the problem to be solved and the users for whom it is being solved.
  • Qualitative user research methods are used before product design and development begin, but they can continue into product development stages, and beyond. They allow you to ask users directly about their needs and pain points or to observe them in contexts in which they would be using your product.
  • A project in progress (and, of course, any existing project) always benefits from usability testing. Usability testing is an effective method to discover usability problems and to make adjustments consistent with user expectations. In-person usability testing sessions are relatively easy and inexpensive to set up. With three to five users per testing session, you have a great chance of discovering usability issues your application may be suffering from.
  • Quantitative user research methods give you insights into questions such as “who,” “what,” “where,” “how much,” “how many.” You can learn how many people visit your website, which pages they visit, who they are, and where they are coming from. You can also analyse user surveys and understand the distribution of users’ responses. All that data can help you identify your user segments and their behaviors.
  • Finally, in order to analyze an existing application against established UX conventions, you can conduct a heuristic evaluation. Heuristics.

For Best Results, Use a Combination of Methods

It would be difficult to include all user experience methods and tools on any given project. Time and budgets are always important factors to consider. A good approach is to leverage a combination of small number of different methods, for example a discovery workshop and a user survey before a new project begins, and usability testing with a dash of quantitative methods once development is underway. You can customize your tool set for each project as needed in order to make sure you build as good user experience as you possibly can.

Tim HopperSharing Your Side Projects Online

I gave a talk at Pydata Carolinas 2016 on Sharing Your Side Projects Online. Here's the abstract:

Python makes it easy to create small programs to handle all kinds of tasks, and tools like Github make it easy and free to share code with the world. However, simply adding a *.py to a Github repository (or worse: a zip file on your personal website) doesn't mean other Python programmers will be able to run and use your code.

For years, I've written one-off scripts and small programs to automate personal tasks and satisfy my curiosity. Until recently, I was never comfortable sharing this code online. In this talk, I will share good practices I've learned and developed for sharing my small projects online.

The talk will include tips on writing reusable scripts, the basics of Git and Github, the importance of READMEs and software licenses, and creation of reproducible Python environments with Conda.

Besides making your code more usable and accessible to others, the tips in this talk will help you make your Github profile a valuable component of your online résumé and open the door for others to improve your programs through Github pull requests.

The video is now online. I sincerely hope others find it valuable.

Caktus GroupPrinciples of good user experience (UX)

Google “UX principles” and you’ll end up with a search results page that offers:

  • 5 principles...
  • 31 fundamentals….
  • 10 principles…
  • Guidelines…
  • Basics….

So let’s get this out of the way: no single checklist will guarantee that you create a great user experience. Every project is different, and the development of every product should be tailored to the user segment the product is built for. But there are some principles that can guide design decisions no matter what and you will not go wrong if you follow them. Let’s talk about a few:

  • Consistency,
  • Content structure,
  • Affordances and signifiers, and
  • Feedback (it’s not opinions!).

Consistency

Consistency applies to a range of design contexts:

  • Visual design,
  • Typography,
  • User interface (UI) design,
  • Interactions design,
  • Markup (the HTML code written by developers),
  • Language (the copy written by content strategists),
  • And more.

Consistency means that we use the same devices—whether visual, typographic, interactive, or linguistic—to convey the same type of information across the entire website or application. If a green checkmark mean success, use the same shade of green, size, shape, and style, no matter where in the application it is used.

Content structure

Content structure example (Principles of good UX)

Content structure reflects the information architecture of the application. Every website or application has content that can be divided up into categories. Those categories have certain relationships to one another:

  • Some categories are like siblings, they reside at the same level of hierarchy;
  • Other categories have parent-child relationships, with parent categories containing the children within them.

Even on a single web page, it is not only important to divide content into smaller chunks and to categorize it, but also to convey visually the relationships between the various pieces.

Users do not read text on the screen word by word, but rather scan it to locate points of interest and drill deeper from there. For that reason, content on any interface must be scannable. Some techniques used to enhance user experience include:

  • Breaking up text into short paragraphs,
  • Using clearly distinguishable and semantic headings,
  • Displaying text as bulleted lists,
  • Using shorter line length, and
  • Aligning text to the left.

Affordances and signifiers

Example of door knobs with different affordances. (Principles of good UX)

Affordance is a property of an object that allows us to use that object in a specific way. A well-designed affordance is accompanied by signifiers that communicate to us, “I can be used in this way!”

Think about a lever door handle you know that you need to press down on it in order to open the door. A spherical knob, on the other hand, communicates a need to turn to accomplish the same outcome. The shape of these devices informs us about how they can be used. The same is true for websites and applications.

Users recognize a piece of underlined text as a hyperlink. In this case, linked text is an affordance and the underline is a signifier that communicates clickability. When we talk about affordances and signifiers of an interface, we’re talking about interface elements and visual cues that accompany them that help users understand that those elements are interactive and what kind of interaction to expect from them.

Feedback (not an opinion!)

Example of feedback (Principles of good UX)

Feedback is any way in which the system communicates to the user that an action has occurred and what outcome results from that action. Take, for example, a task of filling out and submitting a form online. It’s important that you design the form to communicate feedback to the user upon submission.

Form submission feedback could be provided as a text message, change of color, a subtle animation, or a combination of all of the above. No matter how the feedback is expressed, however, it must convey a meaning such as “You have been successful in submitting this form,” or “You have not been successful in submitting this form.”

Users need to know what happens when they interact with an interface. They need to understand what result has come out of their specific interaction with the system, and what they should do next based on that outcome.

Takeaway

Whether you follow five principles, ten principles, or thirty-one fundamentals, you have to test your application with actual users. Designers and developers make assumptions along the way, no matter how diligent they are in applying principles of good UX to product development. Only testing those assumptions against actual people, who are using the application, will help weed out any lingering problems that may compromise great user experience you want to offer to your users.

Caktus GroupWhat We’re Clicking - September 2016 Link Roundup

Every 30 days, we look over the most popular and talked about links that we’ve either shared on social media or amongst ourselves. This month, a lot of the favorites were from our peers on our blog post. Here’s our top five for September.

Insights into software development from a quality assurance (QA) pro

Our QA Analyst, Charlotte Foque, sheds light onto what exactly quality assurance is and shares with us the intricacies of doing it well.

NoSQL Databases: a Survey and Decision Guidance

Felix Gessert of Baqend shares an overview of the NoSQL landscape and various tradeoffs in this highly detailed article. The article demonstrates how difficult it can be to match your database to your work/queries.

Audiences, Outcomes, and Determining User Needs

“Every website needs an audience. And every audience needs a goal. Advocating for end-user needs is the very foundation of the user experience disciplines. We make websites for real people. Those real people are able to do real things. Everyone is happy.”

Digital development principles: a tech firm’s take on understanding ecosystems

Caktus UX Designer Basia Coulter and Strategist Tania Lee talk about ways to understand existing ecosystems and building consensus behind goals and solutions.

Creating Your Code Review Checklist

In this DZone article, Erik Dietrich presents not only a code review checklist, but a philosophy: automate the easy stuff, code review the important stuff.

Caktus GroupCaktus Group @ PyData Carolinas 2016

Tomorrow marks the official beginning of PyData Carolinas 2016 (though technically, the tutorials started today). This is the first time PyData has hosted a conference in our area. We’re especially proud of the way local leaders and members of meetups like TriPython, TechGirlz, Girl Develop It RDU, and PyLadies have worked in tandem to put this event together for the Python community.

Caktus will be at PyData tomorrow and Friday as a Silver sponsor. We’re glad to be in the company of esteemed sponsoring organizations like IBM, RENCI, Continuum Analytics, and the Python Software Foundation.

Come see us at the following PyData events and talks!

Wednesday, September 14th


7:00PM - 8:00PM
Evening Social at the 21c Museum Hotel
Join us after the tutorials with a social hosted by the Durham Convention Center. More details here.

Thursday, September 15th


8:30AM - 5:30PM
Caktus Booth
We’ll have a booth with giveaways for everyone plus a raffle. We’ll also have a display of OpenDataPolicingNC, a project Caktus CTO Colin Copeland helped lead; it received a White House nod for Code for Durham.

11:30AM - 12:10PM
Reach More People: SMS Data Collection with RapidPro (Room 1)
Erin Mullaney (Caktus) and Rebecca Muraya (TransLoc) will share how to use RapidPro, an open source SMS survey data collection app developed by UNICEF, to collect data. They’ll also show you how to use RapidPro’s API to create your own data visualizations.

11:30AM - 12:10PM
Python, C, C++, and Fortran Relationship Status: It’s Not That Complicated (Room 2)
Philip Semanchuk, a Caktus contractor, gives an overview of your many options for getting Python to call and exchange data with code written in a compiled language. The goal is to make attendees aware of choices they may not know they have, and when to prefer one over another.

6:30PM - 8:30PM
Drinks & Data (The Rickhouse, Durham)
We're looking forward to this event, hosted by MaxPoint. It overlooks the park where the Durham Bulls play.

Friday, September 16th


8:30AM - 5:30PM
Caktus Booth
Do stop on by to say hello! We’d love to learn more about the different projects you’re working on.

10:40AM - 11:20PM
Identifying Racial Bias in Policing Practices: Open Data Policing (Room 2)
Colin Copeland, Caktus Co-founder and CTO, Co-chief of Code for Durham, and a 2015 Triangle Business Journal 40 Under 40 awardee, will give a talk on OpenDataPolicingNC.com. His efforts were recognized via an invitation to the White House during Obama’s Police Data Initiative celebration. North Carolina developers and civil rights advocates used demographic data from nearly 20,000,000 unique NC traffic stops to create a digital tool for identifying race-based policing practices.

11:30AM - 12:10 PM
You Belong with Me: Scraping Taylor Swift Lyrics with Python and Celery (Room 1)
Mark Lavin, Caktus Technical Director, and Rebecca Conley, Caktus developer, will demonstrate the use of Celery in an application to extract all of the lyrics of Taylor Swift from the internet. Expect laughter and fun gifs.

12:30PM
Raffle drawing for a copy of Lightweight Django (O’Reilly)
We’ll contact the winner just in time for the book signing. Lightweight Django is co-authored by Caktus Technical Director Mark Lavin.

12:45 - 1:10PM
Book signing of Lightweight Django (O’Reilly) with Mark Lavin
Line up early! We only have limited copies to give away. Each time we’ve done a book signing, the line has been far longer than copies available. For those who aren’t able to get a copy of the book, we’ll have coupon cards for a discount from O’Reilly.

Can’t join us?

If you can’t join us at PyData Carolinas and there’s a talk of ours you want to see, we’ll have the slides available after the conference. You can also follow us on Twitter during PyData itself: @caktusgroup.

Caktus GroupWhat Web Analytics Can’t Tell You About User Experience

Is analytics data collected for a website, an application, or a game sufficient to understand what problems users encounter while interacting with it and what prevents their full engagement?

Why would you want to engage your client in a discovery workshop or your client’s users in user interviews, user surveys, or usability testing sessions, if you can simply look at the data gathered by an analytics tool, and tell with a high level of precision what’s working, and what is not?

“The biggest issue with analytics is that it can very quickly become a distracting black hole of “interesting” data without any actionable insight.” - Jennifer Cardello, Nielsen Norman Group

What metrics do you get out of data analytics?

Analytics tools track and assemble data from events that happen on an existing website or in an application.

The type of quantitative data you can collect with an analytics tools include:

  • Number of visits/sessions
  • Average duration of visits/sessions
  • Number of unique visitors
  • Average time on page
  • Percentage of new visits
  • Bounce rate
  • Sources of traffic
  • List of pages where users exit the website/application

It is an abundant source of information. Data analytics tell you what users do on your website and where—and on which pages—they do it.

So what’s missing from this picture?

While data analytics are incredibly powerful in identifying the “whats” and the “wheres” of your website/application’s traffic, they tell you nothing about the “why.” And without an answer to the “why,” you are a step away from misinterpretation.

Data analytics can be misleading if not supported by insights from qualitative user research

Let’s say you notice that an average visit time on a given page is high. You might be tempted to congratulate yourself for having created such an engaging experience that users spend several minutes on the page. But it is equally possible that the experience you have created is confusing. It takes users a lot of time to make sense of what they are looking at on the page, and they’re spending all that time in deep frustration.

Quantitative data can track user's journey through your website or application. They help you ask better questions, verify hypotheses about patterns of usage, and optimize the application’s performance to align with desired user behaviors.

What data analytics cannot do is identify usability issues. Usability issues and their causes are best diagnosed through usability testing.

Don’t take my word for it

UX professionals frequently report their own and their clients’ inability to draw conclusive answers from data analytics alone. Below are a few insights from conversations I’ve had with UX practitioners on the UX.guide Slack channel.

Christian Ress, co-founder at PlaytestCloud (a mobile game usability testing platform) says that customers often come to them because they spotted issues during soft-launch through their analytics. They see, for example, low interaction with certain features, retention issues, much higher number of attempts for certain game levels, but they do not understand what is causing those problems. It is through remote usability and playability testing sessions that the causes of the problems signaled by quantitative data can be discovered. Remote usability and playability testing involves recording players and prompting them to think out loud during all gameplay sessions.

David Sharek, the founder of UX.guide, finds the greatest challenge in data overload, when a lot of quantitative information is collected without a sufficient amount of time spent on defining the problem. David approaches an investigation into product performance and usability like any research experiment. He formulates a hypothesis and sets out to test it. The quantitative data he collects with an analytics tool Piwik helps him verify hypotheses about the “what” of user behavior. Then he drills deeper into the “why” by talking to users.

Vivien Chang, a UX designer at Brisbane, points out that quantitative methods are used to confirm or disconfirm working hypotheses about the usage patterns within an application, and they require a significant amount of data to do so. Qualitative methods, on the other hand, are tools to gain an understanding of underlying reasons for user actions and user’s motivations. In other words, you collect quantitative data to learn how people use your website or application. That information in itself gives you little or no insight into what problems users might be encountering in the process. To identify and counter usability issues, you should conduct qualitative studies such as usability testing.

What’s the secret sauce?

When you build a product such as a website or an application, you must pay attention to user experience. Your product’s success is not merely dependent on a cutting edge technology you may have employed; it depends on users (or customers) adopting the product. And increasingly sophisticated and savvy users won’t settle for a mediocre experience. You must give them the best experience you can.

How do you build a great experience? By taking strategic advantage of all the tools in your toolbox. You begin the journey by exploring the problem to be solved, understanding the users, and the broader context in which they function. Through discovery workshops, you build a shared understanding with all stakeholders and work together as a team to design a great solution. You monitor potential and actual usability pain points by testing the product iterations with users and adjusting the product’s design accordingly. You measure performance and monitor user behavior patterns with data analytics to further back up your product strategy decisions. Then you dig deeper to understand the causes of user actions by conducting more usability testing.

There you have it; the secret sauce to understanding the “what,” the “where,” and the “why” of user experience by tying together quantitative and qualitative user research methods.

Og MacielPodcasts I've Been Listening To Lately

Podcasts

For someone who has run his own podcast for several years (albeit not generating a lot of content lately), it took me quite some time to actually start listening to podcasts myself. Ironic, I know, but I guess the main reason behind this was because I was always reading code at work and eventually, no matter how hard I tried, I just couldn't pay attention to what was being said! No matter how interesting the topic being discussed was or how engaging the hosts (or hosts) were, my brain would be so focused on reading code that everything else just turned into white noise.

Well, fast forward a couple of years and I still am reading code (though not as much as I used to due to a new role), and I still have a hard time listening to podcast while at work... so I decided to only listen to them when I was not working. Simple, right? But it took me a while to change that for some reason.

Anyhow, I now listen to podcasts while driving (which I don't really do a lot of since I work from home 99.99% of the time) or when I go for walks, and after a while I have started following a handful of them which are now part of my weekly routine:

  • All The Books which provide me with an up to date list of suggestions for what books to read next. They're pretty regular with their episodes, so I can always count on listening about new books pretty much every week.
  • Book Riot for another dose of more news about books!
  • Hack the Entrepreneur to keep up with people who are making something about what they are passionate about.
  • Wonderland Podcast which I only started listening to a few weeks back but it has turned into one of my favorite.
  • Science Vs another new addition to my list, with entertaining takes at interesting topics such as 'the G-spot', 'Fracking', 'Gun Control' and 'Organic Food'.

Today I was introduced to Invisibilia and though I only listened to the first 10 minutes (I was giving the link during working hours, so no go for me), I'm already very interested and will follow it.

I do have other podcasts that I am still subscribed to, but these listed here are the ones that I am still following every episode. Maybe if I had to drive to work every day or went for walks more often, maybe then I would listen to more podcasts? Trust me though, I rather continue listening to only a small set of them than drive to work every day. Don't get me wrong, I love going to work, but that's 2 hours/day of my life that I rather spend at home :)

Caktus GroupDigital development principles: a tech firm’s take on understanding ecosystems

When we meet potential clients, we want to learn more about their software development needs. Beyond that, we’re deeply curious about the work they do, those involved, and the kind of impact they desire to make in the world.

Digital Principle 2, "Understand the Existing Ecosystem" embraces this idea. In many ways, the Digital Principles are an extension of conversations that are ongoing throughout the greater technology community. We're an Agile company, and one of the four propositions of the Agile manifesto reads, “Customer collaboration over contract negotiation.” No collaboration is complete without the inclusion of the user and relevant communities. We share here one of the methods we use to uphold Digital Principle 2.

A critical tool to building shared understanding: Discovery Workshop

We strive to reach a shared understanding of the existing ecosystem and to build consensus behind goals and solutions. To best produce for relevance and sustainability we collaborate with as many stakeholders as possible during discovery workshops. A discovery workshop is a method for all stakeholders to acknowledge existing assumptions. We then:

  • State hypotheses,
  • Brainstorm ideas, and
  • Prototype solutions for any set of problems that need solving.

The client’s software development needs remain front and center, but it is through the process of building a thorough and shared understanding of the ecosystem that we can arrive at tailored solutions that address the actual needs in a more impactful way.

We use a physical, participatory process that focuses on centering the human perspective and contextual environment with all the complexities in between. Human beings learn best by doing. We also tend to make assumptions without being fully aware of them. Within the collaborative environment of a discovery workshop, we get to:

  • Represent a range of perspectives,
  • Create the perfect setting to dig deep into the questions that are being asked,
  • Reframe the existing questions, and
  • Discover new questions that may have been missed before.

A discovery workshop allows all stakeholders to suspend the focus on building the product in order to think about creating experiences first. We use established and well-researched industry tools like journey maps (pictured above) to support our efforts.

Imagine you want to build a web application to support tracking climate change in a region for local communities. In order to build a solution that will enhance rather than impede the work of community actors, it is not enough to make a list of desirable features. You can never be sure the list is complete, unless you fully understand the current workflows with their pain points and unfulfilled needs, the benefits the proposed application is expected to generate, and the contexts in which it will be used.

Understanding the contexts in which applications are being used

In order to build a successful product, we need to have a deep understanding of the outcomes the product is expected to bring about. And for that to happen, we need to learn about all contexts in which the product will function.

  • Who will be using the product?
  • What level of comfort with technology do its potential users have?
  • Where will people be using it?
  • Will they be using the product exclusively to fulfill the need or will they also be using alternative ways to accomplish the same goals.
  • If the latter, will the two paths be competing with one another or will they be complimentary?
  • Does the product’s functionality need to be informed by external inputs or will it be entirely independent?

Using threat modeling to mitigate risks to users

Understanding the ecosystem is also a necessary part of threat modeling. Threat modeling requires understanding the physical and political spaces in addition to the digital touchpoints. Even with the best of intentions, data and assets can be co-opted and potentially used for harm. Understanding and planning risk mitigation strategies helps to improve the impact of the final product. These perspectives are critical for delivering the kinds of products that make a lasting impact on lives and have been an important aspect of our social impact work.

Change happens and this change can save time and resources

From our experience in conducting these workshops and closely partnering with our clients, it has been rare that the original pitch for a product did not change after having gone through the discovery workshop process. In some cases, we uncovered issues in business processes that needed to be addressed before a product could be useful or successful. In one case, for example, an original product pitch required a complex printing system. As we went through the discovery process, we determined that the necessary human resources required to maintain that printing system would be difficult to implement. We decided to take printing out of the priority feature set and thus saved the client time and money with this early discovery.

Digital Principle 2 reflects best practice application design

Investing time and energy into understanding the existing ecosystem is a core digital principle because of the incredible value it adds to outcome of any project.

Within the software industry, the importance of directly answering user needs by speaking to them and understanding their world, is a long-cherished standard. Seeing this same approach promulgated throughout ICT4D via the Digital Principles can help more projects scale up from pilot projects which, in turn, can positively impact more lives meaningfully.

Discovery workshops are a tool to uncover and dig into holistic and technical questions using industry tools and best practices. However, it is just the beginning of improving products and impact as ecosystems always change.

Caktus GroupInsights into software development from a quality assurance (QA) pro

Because quality assurance (QA) is all about creating a seamless application experience across any number of devices, it’s most successful when no one notices it. The craft and expertise behind the work of QA professionals, as a result, can sometimes feel hidden. Charlotte Fouque, Caktus’ QA Analyst, sheds light onto what exactly quality assurance is and the intricacies of doing it well.

How did you get into QA?

I came to QA because I speak French. I was doing language quality assurance testing, testing translations in social games. So I continued into software QA after that. I am very organized outside of work. I use Kanban in my personal life. It’s part of a natural impetus to create order out of chaos.

What is quality assurance?

QA for a web project means testing across selected devices and browsers to make sure the site works as intended. We hold the development team to their own definition of quality. Whatever they or the organization has set as their quality threshold, QA ensures that is being met.

QA also serves as a user advocate. We have to think about whether the application feels good, feels enjoyable for users. Things don’t always line up between design intentions and technical implementation, and we have to be able to call out anything that doesn’t make sense. QA will make sure that the user is getting the best experience possible.

When does QA play a role in software development?

At Caktus, QA is involved from the beginning. We’re involved in estimates before a contract is signed. And we’re involved from every step on a project from the very first sprint, to development, and release.

What’s a typical day of QA for you?

In a typical day, I attend all the scrum meetings in the morning for the teams I’m working with. I have a testing plan that adds on only whatever stories or group of tasks the teams are working on in that sprint. I write the test cases, or reproducible steps to test a feature, in the morning. A test case, for example, could be that the close button on the popup needs to highlight when a mouse hovers over it. This test case would go into a test matrix that contains intersections of all devices and browsers. There’s a pass/fail for each test matrix field, or each device to each browser. Then in the afternoon, I usually follow the testing plan— cross browser, cross device testing.

But no two days are ever the same in an Agile environment, and it’s never boring!

How do you highlight potential bugs and issues for the development team?

If I find an issue, I will submit a ticket that contains all the relevant information for the developer to reproduce and debug it. I try to be as concise as possible (no developer wants to have to read paragraphs of text to figure out what the problem is!) Basically, I try to be as helpful as I can be in answering any questions they have or helping them track down where the issue is occurring. I want the bug fixed just as much as they want to fix it.

I sometimes sit down with the team, or we bounce ideas off each other on how to make a feature better. In my role, I put myself in the user’s shoes in a way that’s hard for a developer that’s too close to the project. I talk to development teams about bugs from the user’s perspective rather than a developer’s. If a user came across this or that bug, would they think there’s something obviously wrong? How would a user behave to get around the issue? Would a user consider it a workflow blocker and leave the site? Sometimes developers don’t see this because they’re deep in the code; they might not have the distance to consider how the user is navigating the site or getting to that particular bug.

What makes someone a good quality assurance professional?

A good tester is able to pay attention to details and willing to drill down. If I see something wrong in something I’m testing, I have to pursue it, try to find steps to reproduce the issue, check to see if it’s happening across devices and browsers, and then follow up with the developers. It requires a lot of patience.

We have to be able to find various small disparate issues but also larger overarching problems. For example, I’ll often find little bugs like links not highlighting but I also find bigger issues with user flow or ways to navigate the website. Someone that’s doing QA has to be able to find all sorts of issues from large to small to difficult to define problems. It takes a good QA person to not only find that range of problems, but to also be able to articulate them in precise, intelligible bug reports.

There are also different types of testing, and experienced QA people will have their own style. I like to test methodically, ensuring I hit everything I set in the test plan. Some people are better at exploratory testing, like clicking on things randomly; unusual user behaviors might uncover the software behaving in unexpected ways.

What’s the best part of QAing?

I find it really satisfying to pick apart features or entire websites, find little issues, follow up on them, and see them resolved quickly. It’s very rewarding to see a project come together, getting better and better every day.

What’s the most challenging part of QAing?

The most difficult part is that my imperative is to find bugs and get them fixed. But nothing is perfect - there will always be unresolved bugs that make it to release, and coming to terms with that is really hard! At what point is it good enough before you can say “ship it”? It’s important to have open lines of communication with the project manager or product owner so that they are aware of the existing issues and can prioritize them along with the rest of the development work.

What is your favorite QA tool?

The most recent one that I found has been really revolutionary for me: Ghostlab. Ghostlab does synchronized browser testing so I can look at the same application and interactions on as many browser windows and devices as I can see in front of me at the same time. It saves me tons of time and I really love it!

If someone were starting a QA process from scratch and they were part of an agile company, what advice would you give them?

I would say that the best way to approach that situation is to try to integrate the QA process with the existing development process. For example, at Caktus, we use scrum. QA actively participates in scrum as part of the development team. This is absolutely key. Developers would usually already have a verification and review step where they review each other’s code. QA comes in the step afterwards. They are part of the workflow and part of the success criteria for each task. Every task that goes through QA upholds the acceptance criteria or Definition of Done for the team.

A good QA person has to be able to work within the team along with the developers. QA should not work in opposition to developers, but as part of development. We can’t just throw issues over the wall to developers and call it not our problem anymore - it’s essential to remember that our common goal is to build the best product possible.

Caktus GroupWhat’s “User Experience” and Why It Matters

Caktus recently welcomed UX designer Basia Coulter to our team. We sat down with her to discuss her perspective on user experience design. Basia, like many in tech, came to her role through a nonlinear path. She first earned a PhD in neurobiology while in Poland, and then came to the United States for a postdoctoral fellowship. The experience led to soul searching, including seven years in a Tibetan monastery in upstate New York where, along with her spiritual interests, she pursued a passion for design, particularly web design. She subsequently devoted herself to learning more about digital communication. Basia has been in the North Carolina area for 2.5 years and currently is part of the leadership team of the local chapter of Girl Develop It and a member of local organizations such as TriangleUXPA, Code for Durham, and AIGA Raleigh.

Let’s start simple. What is user experience?

The person who coined the term “user experience” or UX is Don Norman, a cognitive psychologist and a co-founder of Nielsen Norman Group, currently Director of Design Lab at the University of California - San Diego, who used to work for Apple. For him, user experience includes all aspects of end user interaction with the company, the company’s services, and the company’s products. It’s a very broad range of interactions and includes areas like marketing, customer service, product, and, really anything.

What we have come to mean by user experience or “UX” colloquially is probably a narrower definition than that of Don Norman’s. We’re usually referencing some specific system such as an application or a piece of software. When we talk about UX, we think about how a user feels when they interact with that system. What experience do they have?

The goal of good user experience design is to design and build products that are easy to use, that are a solution to an existing problem, and not a cause of frustration or source of more problems.

What is the benefit of focusing on user experience to businesses and organizations?

UX professionals help organizations understand their users and guide teams in employing best practices to build products that solve users’ problems. By designing experiences specifically around users’ needs, we improve customer satisfaction and, by extension, increase ROI and drive profit or app adoption.

When businesses include UX research in the process of product design and development, they ensure they build solutions that target their particular user segment and they invest in solutions that address the specific pain points of their users. Having a UX designer on the team that builds your product means there is a dedicated person whose job it is to advocate for best user experience every step of the way. Great user experience means happy users; happy users translate to satisfied customers, and satisfied customers become loyal customers.

You’re a UX professional that’s worked in many contexts and types of organizations. What is the UX professional’s role in application development at Caktus?

One of the most exciting aspects of UX work is that involves a variety of skills, and my role often depends on the project. At Caktus, I can support projects at the onset, even before a product is well understood; while the product is being defined; then while it’s being designed and developed; and finally toward the end, once it has been built and is undergoing testing.

Before you can build a solution that addresses users’ needs, you have to understand those needs, you have to identify users’ pain points, and that’s why user research is so important. I recently attended a UXPin webinar during which the speaker, Satyam Kantamneni of UXReactor, said, “Any time you have a user, you’ve got to do user research.” I strongly agree with that statement.

So my job could be leading a discovery workshop or a meeting where all stakeholders come to the table to brainstorm in the early stages of defining the product-to-be to understand the problem at hand and to uncover possible solutions. It could be doing user research by conducting user surveys or interviews. Or I could be doing a UX review of an existing application to determine whether or not it complies with best practices of usability and user experience design.

Can you give us a brief overview of principles of good user experience design? We’ll have you dive deeper in a subsequent blog post.

I think it is important to understand that user experience arises from many disciplines coming together. Those disciplines include information architecture, product strategy, content strategy, user research, visual design, interaction design, software development, usability, accessibility, cognitive psychology, and probably more.

So when we talk about principles of good user experience, we’d have to talk about best practices within all of those domains. For the purpose of this conversation, we could talk about a few basic principles that help build good experience for an interface. I would say that anything that helps decrease the amount of mental processing, so-called “cognitive load”, that the user needs to do to be successful, and anything that guides the user in accomplishing a task within an application constitutes a principle of good user experience. That would include, among many other principles, consistency of visual design and interactions, solid and consistent content structure, and presence of affordances and feedback.

To put simply, people are more likely to engage with content that follows principles of good user experience than with content that does not. So if you want to retain visitors on your website or if you want to see more people subscribe to your application, you cannot afford to ignore those principles.

You hold a PhD in neurobiology. What's the link you see between how the brain works and UX?

Understanding sensory perception is very handy in UX design. Take a couple of aspects of visual perception. One, perception of color is contextual—the same color may be perceived differently in different contexts, for example against different backgrounds. Two, our brain cannot process all the information it is bombarded with at any given moment, so it weeds out what it renders irrelevant. The brain also fills in the gaps where information is missing and creates images of what we see as a representation, not a replica of the object of perception. In other words, we think we see what’s out there in the world around us, but in fact we see our brain’s constructs.

So when we are designing, for example, an interface for users who primarily rely on vision, we need to keep in mind that they will not be processing every single element of that interface in order to make sense of it. Instead their brain will be constructing a representation of the interface based on the elements that get the most attention or are assessed as relevant. It is yet a different matter if we are designing an interface for users with vision impairments.

Another topic very relevant to UX design is decision making. The lab where I did my postdoctoral studies was involved in investigating human emotions and decision making. It was there that I first encountered ideas around how emotional responses impact our decision making. I got a first glimpse of the notion that our rational mind is not the only and perhaps not even the primary actor in our day-to-day decision making. There are also studies showing, for example, that users make fast, snap judgement decisions about a website’s trustworthiness based on its aesthetics. Those are very important pieces of knowledge to keep in mind when designing an interface for users whose decisions and choices will be guided by that interface.

Basia, you personally have had exposure to a wide set of cultural perspectives having come to the United States as an adult. What role does cultural perspective play in UX?

It is critical. Take text as an example. If it’s written from left to right, our eyes will track it a different way than if it’s written top to bottom. Another example is color. We rely on color to convey meaning, but different colors have different meanings in various cultures. In western cultures, for example, white is associated with innocence and is often used in design for weddings. In other cultures, white signals death and mourning. So these are totally different connotations. If we use color to convey meaning, we will need to be mindful of different cultures to create the right experience for the user.

There are also generational differences within the same culture that have impact on user experience. You could think about those as subcultural differences. For instance, there is a trend in web design that’s called flat design, in which interface elements look flat and no visual techniques are used to give them a three-dimensional appearance. This trend has become controversial in the UX community; some UX professionals feel strongly that stripping interactive elements such as buttons off of their three-dimensionality removes important affordances and compromises usability. And in fact usability tests have shown that a lot people have a hard time recognizing flat buttons as buttons. However, it turns out that Millennials and younger users do not have as much trouble with flat design as older users do. So if you’re designing an application for a younger audience, you might not have to worry so much about compromising the usability of your application by using flat design, but if you’re designing for an older generation, you should consider your flat design choices carefully.

Caktus GroupPostgres Present and Future (PyCon 2016 Must-See Talk: 6/6)

Part six of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

Coming from a heavy database admin background, I found Craig Kerstiens’s “Postgres Present and Future" to be incredibly well organized and engaging. Of particular interest to me, because I am somewhat new to Postgres (while having more background history with MS SQL), was the deep dive into indexes in Postgres.

Check out 5:44-8:39 to find out when to use different types of indexes, outside of the standard B-Tree. For instance, Gin indexes are helpful when searching multiple values for a single column, ie. an array field or a JSONB field.

Click over to 17:22-19:33 to learn about the new Bloom Filter in Postgres 9.6, which is coming out in a few months. This extension seems like it will be incredibly useful to speed up queries on wide tables with a bunch of different options.


More in the annual PyCon Must-See Talks Series.

Joe GregorioInertial Balance

What to do when it's late at night and your high schooler says his table wasn't able to complete their physics lab today because they were missing equipment, and the teacher said, maybe half jokingly, that they could complete the lab at home if they didn't finish it in class? That's right, you build experimental equipment in your garage.

This is the Interial Balance we built from scratch using two hacksaw blades. It took about about 20 minutes to build and then another 10 to actually run the experiment.

I hope we don't have to "junkyard wars" all of his labs, but this was fun and quick to build.

Joe GregorioGOP Climate Change Denial Timeline

Building on The Republican race: Five degrees of climate denial, extended to the full seven stages:

Stage 1: Denial
Pre 2010 - The climate is not changing.
Stage 2: Ignorance
2010 - The climate might be changing, or it might not, we just don't know.
Stage 3: GAIA Bashing
2014 - Climate change is real, but it’s natural.
Stage 4: We so tiny
2016 - Climate change is real, but humans aren't the primary cause.
Stage 5: We so poor
2018 - OK, humans are the primary cause, but we can't afford to do anything about it.
Stage 6: Acceptance
2020 - This is awful, why didn't you tell us it would be this bad!?!?
Stage 7: Revert to Form
2024 - We would have fixed the climate if it wasn't for Obama.

Caktus GroupWhat We’re Clicking - August Link Roundup

Every month we collect the links we’ve shared on social media or amongst ourselves that have captured our interest. Here are our favorites from the past 30 days.

Write an Excellent Programming Blog (TalkPython)

One of the best ways to contribute to open source is by sharing knowledge. A. Jesse Davis, a frequent speaker on this topic, shares his thoughts on writing excellent blog posts in this TalkPython podcast.

Deploying Django + Python 3 + PostgreSQL to AWS Elastic Beanstalk (Real Python)

We’ve been exploring this very same topic at Caktus. Here’s the blog description: “The following is a soup to nuts walkthrough of how to setup and deploy a Django application, powered by Python 3, and PostgreSQL to Amazon Web Services (AWS) all while remaining sane.”

APIs: A Bridge Between Mobile Operators and Startups in Africa (Venture Capital for Africa)

“In emerging markets, where mobile operators are the main enablers of the digital economy, operator APIs are a powerful channel for unlocking creativity and giving the startup ecosystem a boost. Every time an operator opens a new set of APIs, it creates a powerful cycle of innovation as startups can combine several APIs to create new services.”

Breaking out of two loops (nedbatchelder.com)

“A common question is, how do I break out of two nested loops at once? For example, how can I examine pairs of characters in a string, stopping when I find an equal pair?... make the double loop into a single loop, and then just use a simple break.”

Death of a survey (DevEx)

In this article, there’s a discussion on how humanitarian organizations are now inundated with data. The most critical point is knowing what question to ask about the data: “What information could help both my organization and our partners do our work better?”

Caktus GroupPython Nordeste 2016 Recap

Image via @pythonnordeste #pyselfie

I don’t know anyone there. I don’t know the language. What about this Zika virus? What about this political unrest? These were some of the doubts and fears racing through my mind at the start of my trip. I had barely settled back home from my trip from PyCon US when it was time to start making the trip to Python Nordeste. It’s a long set of flights to Teresina in the northeast region of Brazil, and I was alone.

Those doubts vanished in almost an instant as I came off the plane to find many of the conference organizers there to greet me. From that moment I found that the Python community I’ve come to know and love so well, the community which has always been so warm and welcoming, is alive and well in Brazil.

I had been asked months before to come and deliver a keynote at Python Nordeste 2016, which was now in its fourth year. The organizers explained their mission was to have a conference catering to the poorer northeast region of Brazil where many cannot afford to attend the larger PyCon Brazil, much less conferences outside of the country. I care about the diversity of voices in the Python community and reaching out to underrepresented groups so this mission spoke to me. I worked with them and with people at Caktus to make it possible to attend and speak.

I thought about what I wanted to speak about for a long time. I didn’t know what they wanted or expected from me when they had asked me to speak. Should I give a technical talk on Python or Django? Should I try to inspire or motivate? Finally I set out on a single goal for my talk: to give a talk which only I could give. I wanted to share a piece of myself and my experience as a developer. The more I thought about it, the more I kept coming back to this idea of how my love of running has shaped my approach to development.

The conference itself was three days. The first day was a set of tutorials followed by two days of talks. My keynote was set for just after lunch on the first day of talks. Since the tutorials were in Portuguese, I decided to take that first day to explore the city as well as continue to prepare my talk for the next day. I explored a local market full of artisan sellers. I walked up one of the main streets and saw the local shops, restaurants, and people going to work and school. It was hot and humid walk. The feeling was similar to our summers in North Carolina but it was winter there. The buildings were short, only a few stories tall. Many open air spaces likely due to the warm climate.

This was my first experience with a single track conference format, and overall I liked that everyone saw the same content. It also allowed for questions to run a little bit over which they did several times during the course of the two days. While the talks themselves, other than mine, were in Portuguese, I was able to follow slides with code samples or project demonstrations.

When it came time for my talk, I was nervous but ready. I talked about my love of running, sharing pictures from various races. I talked about how getting better as a runner requires a long view. Progress is slow and it comes from consistency over time rather than big efforts all at once. That’s been my approach to improving as a developer as well. It’s based on steady and focused improvement. My running is focused on being better than myself rather than judging my success versus the ability or accomplishments of others and I bring that same mentality when working on my programming skills. Unfortunately my talk could not be translated into Portuguese live due to the cost, and I’m sure that left some excluded. I received some positive reactions to my talk. One of the organizers, Francisco Fernandes, in particular shared how my talk related to his experience in graduate school and how it had touched him. In the end I felt I had met my goal and delivered a talk that was genuinely me and that made the long trip worth it.

As luck would have it, the Olympic torch was being carried through the city that night, and I had the opportunity to go see it and run alongside it for a brief period. I grew up swimming and always loved watching the summer Olympics. I never dreamed that I would come so close to the torch. Francisco had a connection with one of the torch bearers and brought it to the conference for the second day of talks allowing people to have their picture taken with it. It felt like a once in lifetime experience.

Mark with the Olympic torch

The second day featured a number of talks which stirred a large amount of debate from the audience. Questions pushed well outside the original time and the schedule fell behind. Everyone seemed comfortable with adjusting as needed. While the final lightning talks were eventually lost due to the additional time, there was a great effort to have a round-table discussion about the state of women in technology in their community. There were roughly a dozen women in attendance, less than 10%, and no women speakers. The organizers gave those that wanted an opportunity to speak about their experiences and some of the other attendees responded with questions or their own experiences. I was thankful to have people in the audience willing to translate for me so that I could keep up with the conversation. I hope this leads to more inclusion efforts in their community. After the conference ended I had the chance to visit a local hackerspace before continuing to a post-conference celebration.

My first trip to Brazil was an absolutely amazing experience. There were times when my face hurt from so much smiling. The food was as amazing as the people. I'm so thankful to have the opportunity to attend and thankful to the organizers for their invitation and warm welcome. I’ve always enjoyed my experiences at PyCon US to meet people using Python in places and ways different from my daily use and Python Nordeste gave me a glimpse into a world I’d otherwise never seen. I left feeling more excited and passionate about this community, wanting to share more, and reach more people who all love Python as I do.

Caktus GroupBake the Cookies (PyCon 2016 Must-See Talk: 5/6)

Part five of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

One of the talks that had the most profound impact on me at PyCon was Adrienne Lowe’s talk, “Bake The Cookies, Wear the Dress: Connecting with Confident Authenticity”. It was really impressive to see a woman who is relatively new to coding talk about being herself and not being swayed by advice to appear "less feminine." Another really important point was that effective mentors need to model their slogging and struggle as well as their success. It's impossible for learners to emulate someone who appears to just magically "get" things. She used helpful metaphors from another passion of hers, cooking, that illustrated her points very clearly. Adrienne was forthcoming with her own personal challenges, which was brave and will be helpful to anyone listening to her talk who is experiencing similar challenges or who is in a position to mentor someone through those challenges.


More in the annual PyCon Must-See Talks Series.

Caktus GroupTrainspotting: Real-Time Detection (PyCon 2016 Must-See Talk: 4/6)

Part four of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

To see real life use of Raspberry Pi with GoPro, watch Data Scientist Chloe Mawer’s “ Trainspotting: real-time detection of a train’s passing from video”. Mawer focuses on Caltrain in this video. Caltrain is a train for commuters traveling between Palo Alto and San Francisco, used by more than 18 million commuters in California. The train's schedule is unpredictable and there is a lack of trustworthy data on the train's status.

Chloe Mawer, a Stanford PhD, designed an algorithm that uses OpenCV via Python to track the train's timing via video. Mawer talked through each facet of the OpenCV algorithm and how to read a video taken with a camera attached to a Raspberry Pi. It was incredibly interesting, especially because of my interest in public transit and public data. The slides are available on her Github.


More in the annual PyCon Must-See Talks Series.

Caktus GroupHow I Built a Power Debugger (PyCon 2016 Must-See Talk: 3/6)

Part three of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

While at PyCon 2016, I really enjoyed Doug Hellmann’s talk, “How I built a power debugger out of the standard library and things I found on the internet” (video below). It's listed as a novice talk but anyone can learn from this talk. Doug talked about the process of creating this project more than the project itself. He talked about his original idea, his motivations, and how he worked in pieces towards his goal. His approach and attitude were refreshing, including talking about places that he struggled and now how long the process took. A beautiful glimpse into the mind of a very smart, creative, and humble developer.


More in the annual PyCon Must-See Talks Series.

Tim HopperPhotos Featured on Smithsonian Magazine

A few weeks ago, I introduced my wife to backpacking in the beautiful Grayson Highlands State Park in southestern Virginia. Part of my reason for picking this location was to see the herd of wild ponies that life at 5000' on the grassy balds.

I shared some of my best pictures from the trip on Flickr under a Creative Commons license (CC BY-NC-ND 2.0). On Saturday, I stumbled acrosss an article about the Grayson Highlands ponies on the Smithsonian Magazine website. I was pleasantly surprised to see they selected two of my images for the story! I've been spending more time lately exploring my longtime interest in wildlife photography, and I'm thrilled to see others sharing my work.

You can find more of my photography on Flickr or Instagram.

Wild Ponies of Grayson Highlands

Untitled

Caktus GroupShipIt Day Recap - July 2016

We finished up last week with another successful ShipIt Day. ShipIt Days are quarterly events where we put down client work for a little bit and focus on learning, stretching ourselves, and sharing. Everyone chooses to work together or individually on an itch or a project that has been on the back of their mind for the last few months. This time, we stretched ourselves by trying out new frameworks, languages, and pluggable apps. Here are some of the projects we worked on during ShipIt Day:

TinyPNG Image Optimization in Django

Kia and Dmitriy started on django_tinypng. This project creates a new OptimizedImageField in Django which uses the tinify client for the tinypng project to compress PNG files. This means that files uploaded by users can be reduced in size by up to 70% without perceivable differences in image quality. Reducing image sizes can free up disk space on servers and improve page load speeds, significantly improving user experiences.

Maintaining Clean Python Dependencies / Requirements.txts

Rebecca Muraya researched how we, as developers, can consistently manage our requirements files. In particular, she was looking for a way to handle second-level (and below) dependencies -- should these be explicitly pinned, or not? Rebecca did some research and found the pip-tools package as a possible solution and presented it to the group. Rebecca described pip-tools as a requirements file compiler which gives you the flexibility to describe your requirements at the level that makes sense to your development team, but have them consistently managed across development, testing, and production environments. Rebecca presented ideas for integrating pip-tools into our standard development workflows.

Elm

Neil and Dan each independently decided to build projects using Elm, a functional language for frontend programming.They were excited to demonstrate how they rearranged their concept of development temporarily to focus on state and state changes in data structures. And then, how these state changes would be drawn on the screen dynamically. Dan mentioned missing HTML templates, the norm in languages where everything is not a function, but loved that it forced programmers to handle all cases as a result of the strict type system (unlike Python). Neil dug not only into Elm on the frontend, but also a functional backend framework Yesod and the Haskell language. Neil built a chat app using Websockets and Yesod channels.

Firebase + React = Bill Tracking

Hunter built a bill tracking project using Google’s Firebase database and the React frontend framework. Hunter walked us through his change in thought process from writing code as a workflow to writing code that changes state and code that updates the drawing of the state. It was great to see the Firebase development tools and learn a bit more about React.

Open Data Policing Database Planning

Rebecca Conley worked on learning some new things about database routing and some of the statistics that go into the Open Data Policing project. She also engaged Caelan, Calvin’s son who was in the office during the day, to build a demonstration of what she had been working on.

Mozilla’s DXR Python Parser Contributions

DXR is a web-based code indexing and searching tool built by Mozilla. For his project, Jeff Bradberry decided to create a pull request contribution to the project that improves Python code indexing. Specifically, he used Python’s own Abstract Syntax Tree (AST), a way to introspect and consider Python code as structured data to be analyzed. Jeff’s contribution improves the analysis of nested calls like a(b(c())) and chained calls like a().b().c().

Hatrack: We all wear lots of hats, switch contexts easily

Rather than working on something completely new, Calvin decided to package up and share a project he has been working off-and-on in his free time called Hatrack. Hatrack attempts to solve a problem that web developers frequently face: changing projects regularly means switching development environments and running lots of local development web servers. Hatrack notices what projects you try to load up in your browser and starts up the development server automatically. For his project, Calvin put Hatrack up on NPM and shared it with the world. You can also check out the Hatrack source code on Github.

Software Testing Certification

Sometimes ShipIt Day can be a chance to read or review documentation. Charlotte went this route and reviewed the requirements for the International Software Testing Qualifications Board (ISTQB)’s certification programs. Charlotte narrowed down on a relevant certification and began reviewing the study materials. She came back to the group and walked us through some QA best practices including ISTQB’s seven principles of software testing.

Cross Functional Depth & Breadth

Sarah began work to visualize project teams’ cross-functional specialties with an eye towards finding skill gaps. She built out a sample questionnaire for the teams and a method of visualizing the skill ranges in specific areas on a team. This could be used in the future when team members move between teams and for long-term planning.

Demographic Data

Colin and Alex each separately investigated adding demographic data into existing project data sets using SunlightLab’s Python bindings for the Cenus API. While the Census dataset contains tens of thousands of variables in various geographic resolution levels (states, counties, down to block groups), using the Census’ API and Sunlight Lab’s bindings made it relatively quick and painless.

Caktus GroupWhat We’re Clicking - July Link Roundup

Here’s external links our team's been chatting about and sharing on social media since the last roundup.

Web Service Efficiency at Instagram with Python (Instagram)

"Instagram currently features the world’s largest deployment of the Django web framework, which is written entirely in Python. We initially chose to use Python because of its reputation for simplicity and practicality, which aligns well with our philosophy of 'do the simple thing first.'"

Survey: US City Open Data Census

“Since its launch in 2014, the US City Open Data Census has helped push cities to make their data open and easily accessible online. The US City Open Data Census team is now looking at ways to improve the Census and make sure it's up-to-date with the needs of today's open data community. As someone interested in open data, we'd like to hear from you about how you think we can make the Census even better!”

A beginners guide to thinking in SQL (Soham Kamani)

“It’s always easy to remember something which is intuitive, and through this guide, I hope to ease the barrier of entry for SQL newbies, and even for people who have worked with SQL, but want a fresh perspective.”

The state of containers: 5 things you need to know now (TechBeacon)

“Docker adoption is up fivefold in one year, according to analyst reports. That's an amazing feat: One year ago, Docker had almost no market share. Now it's running on 6 percent of all hosts, according to a survey of 7,000 companies by Datadog, and that doesn't include hosts running CoreOS and other competing container technologies. “

Well-Tempered API (K Lars Lohn)

The Caktus team watched this during our public video lunch. Here’s a description: “I can play 400 year old music, but I can't open a Word document from 1990. Centuries ago, a revolution in music enabled compositions to last for centuries with no bit rot. There are innumerable parallels between music and software, why don't our programs last longer? Software Engineering has some things to learn from the parallel world of music.”

Creating Your Code Review Checklist (DZone)

“Learn about the steps of undergoing your rite of passage to review code and how to ask the right questions to make the process easier.”

Extracting Video Metadata using Lambda and Mediainfo (Amazon)

“In this post, I walk you through the process of setting up a workflow to extract technical metadata from multimedia files uploaded to S3 using Lambda.”

Bootstrap 4: A Visual Guide (Bootply)

“Here is a visual guide that will show you what’s new in Bootstrap 4 as compared to Bootstrap 3.”

Caktus GroupCaktus at DjangoCon 2016 - Talks that Have Us Talking

Caktus is headed to Philadelphia for our seventh DjangoCon. We’re Gold Sponsors this year, so definitely come by our booth to chat with our team and grab t-shirts, stickers, and a 40% off coupon for Lightweight Django, co-written by our technical director, Mark Lavin. Also be sure to enter our raffle for an Amazon Echo.

In addition to enjoying cheese steaks and saying hi to everyone, we can hardly wait to check out the lineup of talks. I asked our staff which talks they’ve earmarked and the verdict is in— there’s too many good talks to cover by a single person. Here’s the talks our team are especially looking forward to:

Monday, July 18th

Tuesday, July 19th

Wednesday, July 20th

Sign up to chat with us

If you have a Django project that you'd like to chat with our experts about at DjangoCon, please be sure to sign up for a chat.

Looking forward to seeing everyone at the conference!

Caktus GroupBetter Testing With Less Code (PyCon 2016 Must-See Talk: 2/6)

Part two of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

I really enjoyed Matt Bachman’s talk, “Better Testing With Less Code: Property Based Testing With Python”. I heard about property-based testing from this great talk by Jessica Kerr (you should totally watch that talk too!). I'm not sure why, but I had assumed that this was only really doable in static-typed functional languages, so I was very interested to see Matt's talk to hear how this might be done in Python. He demonstrates using a library called Hypothesis to help specify "properties" of your code, and to then generate tests with random, valid input to assert that those properties hold. While this is an interesting approach to testing, I also think it's a valuable way to think about your code, which could help you write better programs. I really liked how Matt presented his topic. He encouraged listeners to start trying things out and sharing those trials so that we can all learn how to properly use this new approach.


More in the annual PyCon Must-See Talks Series.

Caktus GroupFrom Developer to Manager by Sean O'Connor (PyCon 2016 Must-See Talk: 1/6)

Part one of six in our annual PyCon Must-See Series, a weekly highlight of talks our staff especially loved at PyCon. With so many fantastic talks, it’s hard to know where to start, so here’s our short list.

“From Developer to Manager” is a funny talk about the importance of management and the difficulty of managing people while also programming. The talk works for anyone thinking about transitioning into management. Sean discusses the career path of managers and common challenges with deadpan, hilarious honesty.


More in the annual PyCon Must-See Talks Series.

Caktus GroupCaktus Endorses Principles for Digital Development

Caktus is proud to announce that we endorse the Principles for Digital Development or “Digital Principles”, a set of best practices that support technology creation for international development. We feel strongly that they help technology companies like us improve our contributions to the Information and Communications Tool for Development (ICT4D) community. In offering technical design and software development services, we often work alongside our clients to help shape the tools they need. Because of our influence in this process, having established principles are helpful to use as a guide in our conversations and design of the tools we build.

The Principles

Back in February 2016, Caktus joined over 150 organizations at the conference on Digital Development: From Principle to Practice in Washington, DC. The conference called on the ICT4D community to not only support but also deeply engage in practicing and reflecting on the 9 Principles of Digital Development:

  • Principle 1: Design with the User
  • Principle 2: Understand the Ecosystem
  • Principle 3: Design for Scale
  • Principle 4: Build for Sustainability
  • Principle 5: Be Data Driven
  • Principle 6: Use Open Standards, Open Source, Open Data and Open Innovation
  • Principle 7: Reuse and Improve
  • Principle 8: Address Privacy and Security
  • Principle 9: Be Collaborative

Each principle is meant to be a guideline aimed at improving the way technology is designed, funded, built and implemented in development and humanitarian efforts. These principles exist and will continue to evolve to promote technology and program design that is contextually appropriate and sustainable. Please see the report “From Principle to Practice: Implementing the Principles for Digital Development” for a great in-depth look at each principle.

In coming blog posts, we’ll discuss the principles that stood out to us right away— specifically using open source and design with the user.

Caktus GroupQuery Expressions are Amazing

The Django 1.8 release added support for complex query expressions. The documentation has some nice examples but they don't do justice to how crazy awesome these are. In this post, we will go through some additional examples of how to leverage these expressions.

Django has had one form of a query expression for several years now: the F expression. F() can be used to reference an existing column in a query. This is often used for atomic update operations, such as incrementing a counter. However, F can also be used to compare two columns on a model when filtering. For instance, we may be interested in users who haven't logged in since their first two weeks on the site. That requires comparing the value of the last_login column and the date_joined on the standard User model from contrib.auth:

from datetime import timedelta

from django.contrib.auth.models import User
from django.db.models import F
from django.utils.timezone import now

# Create some fake data: 10 active users and 20 inactive ones
today = now()
active_count = 10
inactive_count = 20
for i in range(1, active_count + inactive_count + 1):
    active = i <= active_count
    prefix = 'in' if active else ''
    domain = 'example.com' if i % 3 == 0 else 'caktusgroup.com'
    attributes = {
        'username': '{}active-{}'.format(prefix, i),
        'email': '{}active-{}@{}'.format(prefix, i, domain),
        'date_joined': today - timedelta(days=30),
        'last_login': today - timedelta(days=0 if active else 21),
    }
    User.objects.create(**attributes)
# Query inactive users
inactive = User.objects.filter(last_login__lte=F('date_joined') + timedelta(days=14))

The F expression supports basic arithmetic operations including some date math, as seen in the example above. However, it is still very limiting in comparison to what is available in SQL.

Relational databases such as Postgresql support a number of built-in functions which you can leverage in the ORM using the Func expression added in 1.8. For example, you may want to examine the email domains of your user base. For that, you might use the split_part function in Postgresql to extract the domain of the email address column. To normalize the domain values you can compose this with the built-in Lower expression:

# Continued from above
from django.contrib.auth.models import User
from django.db.models import F, Func, Value
from django.db.models.functions import Lower

qs = User.objects.annotate(domain=Lower(
    Func(F('email'), Value('@'), Value(2), function='split_part')))

This translates into the SQL call split_part("auth_user"."email", @, 2) and annotates every user with a new domain attribute which is the domain of their email address. The value 2 passed to split_part says to take the second value after splitting the string. Unlike Python this is a 1-based index rather than a 0-based index. With this we can find out what the most popular domains are for the users:

# Continued from above
from django.db.models import Count

popular = qs.values('domain').annotate(count=Count('id')).order_by('-count')
print(popular)
# Result
# [{'count': 20, 'domain': 'caktusgroup.com'},
# {'count': 10, 'domain': 'example.com'}]

As noted in the example, this returns a list of dictionaries of the form {'domain': <domain name>, 'count': #} ordered by the highest counts first. We can take this even further using the conditional expressions.

Two more new expressions Case and When can be used to build conditional aggregates. For instance, we may want to only count users who have logged in recently:

# Continued from above
from django.db.models import Case, When
from django.utils.timezone import now

active = When(
    last_login__isnull=False,
    last_login__gte=now() - timedelta(days=14),
    then=Value(1))

active defines the conditional expression when the last_login is not null and is a date later than 14 days ago. If there is a match then this row will add the value of 1 to the aggregate. This conditional expression can be passed into an aggregate expression such as Count, Sum, or Avg. To get the popular domains, we’ll count the number of active users for a given email domain.

# Continued from above
popular = qs.values('domain').annotate(
    count=Count('id'), active_count=Count(Case(active))).order_by('-active_count')
print(popular)
# Result
# [{'active_count': 7, 'count': 20, 'domain': 'caktusgroup.com'},
#  {'active_count': 3, 'count': 10, 'domain': 'example.com'}]

This adds a new key/value to the resulting dictionaries which include the number of active users for the domain. Here caktusgroup.com has the most active registered users but it also has the most registered users overall. For one last usage, we can look at the percent of users for each domain who are active again using the F expression:

# Continued from above
popular = popular.annotate(
    percent_active=Value(1.0) * F('active_count') / F('count') * Value(100)
).order_by('-percent_active')
print(popular)
# Result
# [{'active_count': 7, 'count': 20, 'domain': 'caktusgroup.com', 'percent_active': 35},
#  {'active_count': 3, 'count': 10, 'domain': 'example.com', 'percent_active': 30}]

Again this adds another data point to the returned list of dictionaries which is the percent of active users. Now we know which email domains are associated with the most users, the most recently logged in users, and the percent of users with that domain who have been recently active.

Query expressions like Func allow you to make more complex queries, leveraging more of your chosen database’s power without having to drop to raw SQL. Combined with the aggregation and conditional expressions you can roll up additional statistics about your data set using the expressive power of the ORM. I hope these examples give a good overview of some of the queries that are now easy to handle in the ORM and which previously required raw SQL.

Caktus GroupThe Journal of Medical Internet Research Features Epic Allies Phase 1 Study Results

The Journal of Medical Internet Research recently published “Epic Allies: Development of a Gaming App to Improve Antiretroviral Therapy Adherence Among Young HIV-Positive Men Who Have Sex With Men”. Epic Allies, initially funded by a federal Small Business Innovation Research (SBIR) grant, represents a partnership between Caktus, UNC’s Institute of Global Health and Infection Diseases, and Duke Global Health Institute.

The article highlights the challenges of medication adherence, emphasizing the concerns of study participants directly:

“Yeah, cause honestly, it was a good few months before I ever took medication. And in that timeframe of diagnosis to taking medication, it was very easy for me to detach. It was very easy for me to say, this is not real, nahhh, whatever. It didn’t become real until I had to take a pill. When you take a pill, it’s real.” - Study participant.

The team used continuous participant feedback to iteratively develop the application. Ultimately, the study found that this iterative approach to application development was what made it “highly acceptable, relevant, and useful by YMSM (young men who have sex with men).”

The study authors are Sara LeGrand, PhD; Kathryn Elizabeth Muessig, PhD; Tobias McNulty, BA (Caktus); Karina Soni, BA; Kelly Knudtson, MPH; Alex Lemann, MS (Caktus); Nkechinyere Nwoko, BA (Caktus); and Lisa B Hightow-Weidman, MPH, MD.

To read the study in full, visit http://games.jmir.org/2016/1/e6/.

Edited to add: Epic Allies was built with Unity and the Python backend is built on top of Django and Django REST Framework.

Caktus GroupPyCon 2016 Recap

PyCon, beyond being the best community event for Python developers, is also an event that we happily began thinking about eleven months ago. Almost as soon as PyCon 2015 ended, we had the good fortune of planning the look and feel of PyCon 2016 with organizer extraordinaires Ewa Jodlowska, Diana Clark, and new this year, Brandon Rhodes. Our team has loved working with the organizers on the PyCon websites for the past three years now. They’re great people who always prioritize the needs of PyCon attendees, whether that’s babysitting services or a smooth PyCon web experience.

Seeing the PyCon 2016 Artwork

The Caktus team arrived in Portland and were almost immediately greeted with large-scale versions of the artwork our team made for PyCon. Seeing it on arrival, throughout the event, and especially during the keynotes was surreal.

PyCon 2016 sponsor banner

Getting ready for the tradeshow

Our team got ready for the booth first, ahead of the PyCon Education Summit and Sponsor Workshops where we had team members speaking. Here’s the booth before everyone came to grab t-shirts and PyCon tattoos and to learn more about us.

The Caktus booth at PyCon before the festivities begin.

Here’s a closeup of our live RapidPro dashboard too.

The RapidPro live dashboard Caktus built for PyCon.

Supporting our team members

This year, at the PyCon Education Summit, Rebecca Conley spoke about expanding diversity in tech by increasing early access to coding education. Erin Mullaney and Rebecca Muraya spoke at a Sponsor Workshop on RapidPro, UNICEF’s SMS application platform. Sadly, we didn’t get a picture of Rebecca C, but Erin shared this picture of herself and Rebecca M. on Twitter.

Erin and Rebecca M. after giving their RapidPro talk at PyCon.

Tradeshow time!

PyCon, for our booth team, is always intense. Here’s a taste of the crowds across three days.

A busy crowd around the Caktus booth.

The excitement, of course, included a giveaway. Here’s the winner of our BB8 Sphero Ball raffle prize, Adam Porad of MetaBrite, with our Sales Director, Julie White:

PyCon attendee wins the Caktus BB8 Sphero giveaway.

So many talks

With our office almost empty and most of our team at PyCon, there were a lot of talks we went to, too many to list here (don’t worry, we’re going to start highlighting the talks in our annual PyCon Must See Series). We do want to highlight one of the best things about the talks— the representation of women, as described by the PyCon Diversity chair:

Across three packed days, here’s some of the topics we got to learn more about: real time train detection, inclusivity in the tech community, and better testing with less code. With the videos now available, we can still catch all the great talks even if we couldn’t be there.

PyLadies auction

One of the highlights of PyCon is definitely the PyLadies auction. Every year, it’s a raucous event that’s just plain fun. This year, we contributed original concept art for the PyCon 2016 logo. It went for $650 to Jacob Kaplan-Moss, the co-creator of Django. Since we’re a Django shop, there definitely was quite a bit of excited fandom for us.

Jacob Kaplan-Moss holds won auction item: Caktus' early concept art for PyCon 2016 logo

And we can’t leave without a cookie selfie

Whoever came up with the cookie selfie idea is brilliant. Here’s Technical Director Mark Lavin with his cookie selfie.

Hope to see you next year!

In the meantime, make sure to return to our blog for our annual PyCon Must See Series.

Caktus GroupMy First Conference Talk: Reflecting on Support and Inclusivity at DjangoCon Europe 2016

The environment at Caktus is, above all, one of encouragement. I experienced that encouragement as an intern and continue to experience it as a full-time developer. In addition to providing workplace mentorship, Caktus encourages all of its employees to submit talks to conferences. My manager Mark Lavin and mentor Karen Tracy encouraged me to get over my concerns about being new to the field and to start submitting talks.

Along with the support from Caktus was the impetus from Djangocon Europe for first-time speakers to submit. Djangocon Europe’s Call For Papers(CFP) includes suggested topics and offers of support beginning with brainstorming a topic and including mentors if your talk is chosen. I took them up on this offer and floated a couple of ideas over email. I got a very quick response with the suggestion that I expand a previous blog post I had written on my mid-career transition into a talk. Baptiste Mispelon and Xavier Dutreilh continued to be helpful and responsive throughout the application process and made me feel like my contribution was valued and that I was being taken seriously, whether my talk would be ultimately selected or not.

A week later, I received the notification that my talk was selected. The support continued from Caktus, the broader local development community, and the Djangocon Europe organizers. Mark helped me refine my talk content, and Caktus coworkers and Pyladies helped me organize public previews of the talk. Djangocon Europe opened a slack mentor channel in which I was able to ask a lot of questions about talks in general and about how to communicate effectively with an international audience. The refinement and confidence gained from these experiences helped send me to Europe excited about giving my first talk.

The organizers made travel easy, opening a Slack channel for attendees to ask general questions about the conference. I arrived in Budapest, got a shuttle to the hotel, and checked into my room. Then I got on the Slack #speakers channel and asked if anyone wanted to join me for dinner. I ended up with two fantastic dinner companions, Andrew Godwin and Anna Schneider. Over dinner I learned about developing for non-profits, London’s economic development, and many other fascinating things.

Budapest is beautiful and extremely friendly and walkable. In general, google maps and google translate worked to help me get around after initially leading me astray on my first walk to the venue. Once I arrived, I was greeted with signs telling me I was welcome, I looked awesome, what the code of conduct was, and what phone numbers and email to use if I had any concerns. The conference was well staffed with friendly folks to direct attendees and to answer questions. The food and snacks were good. There were dedicated quiet spaces and a separate prayer room. Attention to all of these details showed that the conference organizers carefully considered the needs and comfort of all the attendees in their planning and made us feel valued.

Throughout the conference and afterwards, Djangocon Europe showed particular dedication to the Code of Conduct and the principles behind it, which generally amount to “be kind to others.” All conference slides were reviewed by the organizers to make sure they adhered to the Code of Conduct. Light-hearted but direct signs in the bathrooms made it clear that gender non-conforming attendees were welcome and that their safety and comfort were important. During the conference, an announcement was made regarding a slide that had been added after the screening and brought to the attention of the organizers as a violation to the Code of Conduct. This announcement demonstrated that complaints were taken seriously and handled quickly. It served to make us all feel that our safety and comfort was a priority. I even saw an interaction on Slack that enforced these values of inclusion and kindness. The conversation started with er someone giving a lightning talk asking an organizer to screen his slides The organizer pointed out a slide with a photo that could be seen as objectifying the woman in the photo. The speaker agreed and removed the slide. It was a simple interaction from which everyone learned. As a female speaker, I felt that the organizer was absolutely looking after both my interests and the interests of the presenter. Soon after the conference, Djangocon Europe published a Transparency Report detailing protections put in place, issues that arose, and the way those issues were handled. No conference can completely control attendee behavior, but attentiveness and transparency like this should set the standard for how conferences create safe and inclusive environments.

DjangoCon Europe 2016 venue

The venue was very attractive and comfortable, with a small theater for the talks as well as a library and balcony where talks were streamed for those who wanted a smaller, quieter setting. Having those options definitely helped me enjoy the conference as I had speech preparation in mind, along with getting the most I could from the talks.

The first talks emphasized the tone of welcoming and mutual respect. In their talk, “Healthy Minds in a Healthy Community” Erik Romijn & Mikey Ariel spoke frankly and personally about the struggles many of us face in the open source community to maintain physical and mental health while facing the demands of our jobs as well as the added desire to contribute to open source projects. As a new developer, it was really important for me to hear that all the people I perceive as “rockstars” and “ninjas” are just as human as I am and that we all need to take care of each other. It also inspired me to reflect on my gratitude that I work at Caktus, where we are all valued as people and our health and happiness is a priority of the company.

The talks were all fantastic, a nice blend of topics from debugging to migrations to the challenge and necessity of including languages that don’t read left-to-right, given by women and men from all over the world. I felt honored to be among them and pleased that the organizers felt a mid-career transition into programming merited a speaking slot. The whole experience continued to be enjoyable, especially the speaker dinner consisting of traditional Hungarian food. At the dinner I had the chance to learn about the developing tech scene in Eastern Europe and the assumptions we had about each other on either side of the Iron Curtain in the mid 1980’s. Software itself is impressive. However, it is only when we get to understand the people who are making it and the people for whom we are making it that software’s real meaning and value become evident. Djangocon Europe definitely facilitated that kind of understanding. Another highlight of the evening was receiving my speaker gift, some souvenirs from Budapest and a handwritten note thanking me for participating, which made me feel very appreciated.

Before a talk.

My talk was on the last morning, and while I expected everyone to be tired from the party the night before with live music in one of Budapest’s “ruin pubs,” there was a good crowd. The emcee Tomasz Paczkowski did a great job preparing speakers, including me before we spoke enforcing the “no comments, only questions” policy after we finished speaking. Speakers were also given the option to have no questions from the stage. I didn’t choose that, but I see how that option would be valuable to some speakers.

What I didn’t know when I first submitted my talk was that it was a single-track conference. I learned that when I saw the schedule. My audience was the whole conference, as it was for all the speakers. It was daunting at first to know that all eyes would be on me (at least the eyes of everyone who chose to attend a talk). I went into the room the night before and stood on stage, looking at several hundred empty chairs and absorbing the idea that they would be full of people watching me the next day. Fortunately I now knew who at least some of these people were, and I had seen in general how they responded positively to each other and to other speakers. I have performed as a dancer in front of large crowds plenty of times, but had never given a professional talk to an audience of that size. The beauty of the space and the familiarity of being on stage certainly helped ease my apprehension.

By the last day of the conference, I felt so comfortable and appreciated that I enjoyed giving my talk, From Intern to Professional Developer: Advice on a Mid-Career Pivot immensely. I was a little bit nervous, but just enough for it to be motivating. A number of people made a point of encouraging me throughout the week and being present as smiling faces close to the front during my talk. It went by quickly. I tried to remember to breathe and look up (both things I forget to do when I’m nervous). The crowd was polite and responsive.I got some good questions and follow-up from people who had made a similar transition or were thinking about it, as well as questions from some hiring managers. I felt like I was able to make a valuable contribution to the conference and to the community through my talk, and I am grateful to Djangocon Europe and Caktus for making it all possible.

Caktus GroupCode for Durham: National Day of Civic Hacking Recap

Code for Durham recently participated in Code for America’s National Day of Civic Hacking. Hosted in the Caktus Group Tech Space, the event was attended by more than 50 local participants.

Community manager of opensource.com, Jason Hibbetts acted as emcee for the day. He introduced the two planned projects from Code for Durham and fielded project pitches from other attendees.

During the day, participants broke into teams to work on a total of five projects. These included Code for Durham’s School Navigator—a platform for geolocating nearby publicly-funded schools and accessing information on those schools—and CityGram—a notifications platform for local issues like crime data reporting or building permit changes.

School Navigator team

Caktus’ Chief Business Development Officer Alex Lemann helped coordinate the team working on Durham’s School Navigator. The group consisted of thirteen developers, two UX designers, and Councilman Steve Schewel, who came to learn about the project. Code for Durham members were able to onboard all of the day’s participants to the project, add them as collaborators to the GitHub repo, and introduce them to the project’s backlog. Team members then selected their own tasks from the backlog to tackle.

A number of updates were made to the school navigator, including changes to frontend code, revisions to school policy information, deployments to help keep the site running smoothly, and an entire redesign of the school profile pages from a UX perspective.

School Navigator Project Intro

Alex was especially excited by his contribution to the day’s work. Noticing that GitHub only measures collaboration on a project in terms of committed code, Alex created a way of honoring alternative contributions. To accomplish this, he developed contributors.py, which uses the GitHub API to look up every comment added to a particular project. This data is then compiled into a contributors list, making recognition of contributions to a project more transparent and inclusive.

Ultimately, the day was a success. “Participants were enthusiastic and made significant contributions to the projects,” Alex commented. “It is important to contribute to open source projects and give back to the technical community. But it is additionally rewarding to contribute to projects you know are helping people nearby in your very own neighborhood.”

Edit your city

Though the National Day of Civic Hacking is over, work on these projects is ongoing. To get involved, check Code for Durham’s list of upcoming events and be sure to attend one of their civic hack nights.

Not local? According to U.S. Chief Technology Officer Megan Smith, more than 100 civic hacking events were held nationwide. To learn how to get involved in your area, visit codeforamerica.org.

Jeff TrawickA few quick notes on trying out Windows 10 WSL (Windows Subsystem for Linux)


I had a chance to play with the Ubuntu userspace support in Windows 10 recently. I started with Windows Insider Preview build 14295 from MSDN, enabled the "Fast Ring" for Windows Insider updates, and then updated to build 14316 and activated the subsystem. After rebooting, running bash from PowerShell installed the Ubuntu base image.

The first use case was building the APR 1.6.x tree from svn and running the test suite. The build was uneventful and most testcases pass. From looking only at the test suite output, it seems that Sys V semaphores and the FREEBIND socket option aren't implemented, epoll isn't close enough to Linux, sendfile isn't close enough to Linux, and perhaps something about file locking isn't close enough. That's not so bad all in all. I think the next steps here are to identify some particular discrepancy from Linux, report it, and see if they bite. Separately, setting some ac_cv_FOO variables might be close to enough to get the testcases to run completely (e.g., use poll() instead of epoll(), use a different default proc mutex, etc.).

The second use case was trying out a Python+Django+PostgreSQL project, checking if it is usable and test suites for some projects I work on pass. (Such code "should" work fine on native Windows given enough time to mess around with installing different server software, getting native Python extensions to compile, etc. Yuck.) Unfortunately, the lack of SysV IPC breaks PostgreSQL setup. (See this GitHub issue for the PostgreSQL failure and this suggestion for supporting SysV IPC.)

So what is the point of WSL, for me at least?

  • Garbage collect the last 3 or so ways to get Unix-y stuff running on Windows and make the installation and update effort for such tools largely disappear (apt-get). (I.e., greatly improve the theoretical best-of-both-worlds development setup.
  • Improve some use cases for the occasional savvy Windows-bound consumer of <dayjob>, which would require being able to install a Django application using the same Ubuntu and Python packages and the same setup instructions as on Ubuntu, in order to run management commands and maybe occasionally use the dev server for testing very simple changes.
  • Have something new for me to yell at OS X users with broken tools who have been ignoring my pleas to install an Ubuntu VM.

References

Caktus GroupWhat We’re Clicking - May Link Roundup

Below you can find this month’s roundup of articles and posts shared by Cakti that drew the most attention on Twitter. The list covers coding for matrix factorization algorithms in Python, designing apps that optimize for sequential dual screen usage, preventing technical debt, and understanding the complexities and limitations involved in building apps for low-income American families.

Finding Similar Music Using Matrix Factorization

A step-by-step guide to calculating related music artists with matrix factorization algorithms. This tutorial is written in Python using Pandas and SciPy for calculations and D3.js for interactive data visualization.

Windows on the Web

Completing a task across multiple devices is a common habit. And yet, according to the author of this piece Karen McGrane, this practice is rarely considered in user design scenarios. In this article, McGrane contemplates how to design the best user experience for sequential dual screen usage.

Technical Debt 101

This article is a detailed explanation of technical debt and the negative consequences of sacrificing code quality.

What I Learned from Building an App for Low-Income Americans

Ciara Byrne’s thoughtful article on lessons learned from her experience building an app for low-income Americans. Byrne reflects not only on the challenges involved in designing for this particular community of users but also the complex definitions of low-income that must be taken into account when approaching similar projects.

Caktus GroupCode for Durham and a National Day of Civic Hacking

This Saturday, June 4th, Caktus Group will be hosting Code for Durham as they join Code for America’s National Day of Civic Hacking. The day is a chance for everyone from developers, to government employees, to residents who care about their city to come together and use their talents to help the community. Attendees will collaborate on civic tech projects to be used by citizens and government employees. These projects seek to provide data on or improve government processes, addressing issues like health care, affordable housing, criminal record access, police data, and more.

The Code for Durham event will support work on several ongoing projects. These include CityGram and the Durham School Navigator. CityGram is a notifications platform for local issues like crime data reporting or building permit changes. Durham School Navigator enables users to geolocate nearby publicly-funded schools and view information like performance ratings, filter by magnet, charter, or public schools, and demystify school zoning patterns. Aside from these two projects, there will also be a period for attendees to pitch new project ideas.

The day will be filled with opportunities to contribute to projects that make Durham better. See the rest of the day’s schedule below and register for the event here.


National Day of Civic Hacking - Saturday, June 4th

Kickoff & Project Sprint Pitches (10:00 am - 10:45 am)

Hear from the two existing, documented projects (Citygram and Durham School Navigator). Open floor for other project ideas.

Sprints (11:00 am - 3:30 pm)

Break out into work groups for the various projects.

Civic Hacking 101 (11:00 am - 12pm)

Open meeting with Red Hat’s Jason Hibbetts, community manager of opensource.com, and City of Durham’s Laura Beidiger. They will provide background on Code for America and demo some of the local civic apps Code for Durham has built. Finally, they will facilitate signing up to participate in a community app user testing group.

Lunch (12:00 pm)

Catered lunch from The Farmery in Durham ($5.00)

Read Out (3:30 pm - 4:00 pm)

Sprint teams demo their projects and report on progress from the day’s work.

Caktus GroupWhere to Find Cakti at PyCon 2016

As Django developers, we always look forward to PyCon. This year, working with the Python Software Foundation on the design for PyCon 2016’s site kindled our enthusiasm early. Our team is so excited for all the fun to begin. With an array of fantastic events, speakers, and workshops, we thought we would highlight all the events we’ll be participating in. Come find us!

Sunday, May 29th

Python Education Summit: Outside the Pipeline: Expanding Early Access to Coding as a Career Choice (3:10 pm)

Rebecca Conley will be speaking about how to increase diversity in tech by expanding early access to coding education.

Sponsor Workshop: Leveraging Text Messaging in 2016 with RapidPro (3:30 pm)

Attend Erin Mullaney and Rebecca Muraya’s workshop on building SMS surveys. The workshop will include an overview of up-to-date case studies involving the use of SMS for surveys, crises, elections, and data tracking. Erin and Rebecca will also cover RapidPro, UNICEF’s open source SMS mobile messaging platform. In addition to basic functionality, they will demonstrate how to extend RapidPro’s core functionality through use of the API and how to manage SMS security.

Monday, May 30th - Tuesday, May 31st

Trade Show (8:00 am - 5:00 pm)

Don’t forget to stop by our trade show booth where you can take our 5-question project health quiz and chat about results with our experts. You can also sign up for one of our limited time slots to discuss upcoming projects.

Our booth, double in size this year, will also feature a live RapidPro survey about PyCon attendees. We’ll also have some sweet swag, like PyCon 2016 temporary tattoos, Django-themed t-shirts, and more. Plus, you can enter to win an authentic BB8 Sphero!

pycon 2016 temporary tattoos

Mon, May 30th - Wednesday, June 1st

Open Spaces

We’re planning on hosting a few Open Spaces discussions. Times TBD. Be sure to look for our discussion topics on the Open Spaces board!

  • Scrum at Caktus
  • RapidPro Flow Editor overview
  • Using the Django project template
  • Open data policing and getting involved in civic tech
  • RapidPro deployment and usage
  • AWS deployment using Python
  • Python in civic tech and Code for America projects
  • Building APIs
  • Teaching community-centered Python classes
  • Python meetup groups - supporting PyLadies and Girl Develop It

Tuesday, May 31st

Charity Fun Run (6:00 am)

Mark Lavin will be running in the annual 5k charity fun run. This year’s funds will be donated to Big Brothers Big Sisters Columbia Northwest.

PyLadies Auction (6:30 pm)

There are always fun and fantastic items up for bid during the PyLadies benefit auction. This year, Caktus is contributing a framed piece showcasing early concept sketches of the PyCon 2016 website.

early designs for PyCon 2016

Wednesday, June 1st

Job Fair (10:00 am - 1:00 pm)

Find out how you can grow with us! Stop by our booth at the Job Fair for information on our open positions. We’re currently looking for sharp Django web developers to fill full-time and contractor positions.

Caktus GroupMark Lavin to Give Keynote at Python Nordeste

Mark Lavin will be giving the keynote address at Python Nordeste this year. Python Nordeste is the largest gathering of the Northeast Python community, which takes place annually in cities of northeastern Brazil. This year’s conference will be held in Teresina, the capital of the Brazilian state of Piauí.

Mark will be speaking from his love of long-distance running . Using endurance sports training methodologies and applying them to development training, he will provide a roadmap for how to improve as a developer throughout your career.

Mark Lavin is a co-author of Lightweight Django from O'Reilly, The book was recently published in Portuguese and is available in Brazil under the title Django Essencial from publisher Novatec. He has also recorded a video series called "Intermediate Django" which focuses on integration background jobs with Celery and best practices for growing Django developers. Mark is an active member of the Django community and you can often find him contributing to the Django Project or answering questions on StackOverflow.

Tim HopperInstall Apache Storm with Conda

I'm looking into using Apache Storm for a project, and I've been fiddling with several different versions in my local testing environment.

I made this easier for myself by adding binaries for Storm 0.10.1 and Storm 1.0.1 to my Anaconda.org channel. That means you can add the Storm binary to your path with

conda install -c tdhopper apache-storm=1.0.1

or

conda install -c tdhopper apache-storm=0.10.1

Caktus GroupCaktus CTO Colin Copeland Invited to the White House Open Police Data Initiative

We at Caktus were incredibly proud when the White House Police Data Initiative invited CTO Colin Copeland to celebrate their first year accomplishments. While at the White House, Colin also joined private breakout sessions to share ideas with law enforcement officials, city staff, and other civic technologists from across the country. Colin is the co-founder of Code for Durham and served as lead developer for OpenDataPolicingNC.com. OpenDataPolicingNC.com, a site built for the Southern Coalition for Social Justice, displays North Carolina police stop data.

When he returned, we couldn’t wait to learn more about his perspective on the talks given (video available) and the breakout discussions held. Here’s a condensed version of our conversation with him.

Can you tell us what the White House Police Data Initiative is?

It’s an effort by the Obama administration to use open data and partner with technologists to strengthen the relationship with citizens and police. The goal is to increase transparency and, as a result, build trust and accountability. It has grown a lot—53 law enforcement agencies. It’s an incredible initiative.

What was it like to be at the White House Police Data Initiative celebration?

It was super exciting to be at the White House and to see demonstrations of what has been accomplished. Fifty-three new law enforcement agencies had signed on to the initiative with close to 100 datasets. I felt lucky to be part of it since what we do with OpenDataPolicingNC.com is such a small piece of the whole effort.

Seeing other initiatives and what other police departments are doing was invigorating. It really made me feel motivated to keep doing what we’ve been doing, especially after seeing other Code for America projects. I also liked being able to hear the perspectives of people from vastly different backgrounds, whether it was someone in the police department or the city. The event was about learning from people all over the country.

Can you describe a couple perspectives you found interesting?

Ron Davis [Director of the federal Office of Community Oriented Policing Services] had a perspective grounded in making sure all police officers, from the rank and file to leadership, understood why open data benefits them. It’s an important aspect. If they don’t see the benefit and buy-in, it’s much harder to advocate for open data.

Also Chief Brown [Dallas] emphasized that releasing data led to community trust. The value you get out of that outweighs any internal pressure not to do it. He was straightforward about how doable it is to release data, how good it was for Dallas, and encouraged other police departments to do the same.

What do you think is the greatest challenge to open data policing efforts for interested agencies?

Knowing what to share is a hurdle for most new agencies. There was some discussion of building a guide or toolkit to share ways to implement this in your city. Small police agencies do not want to reinvent the wheel, so they need easier onboarding. We need to make it easier for everyone to get involved.

What was something new you learned about open data policing?

I learned a lot. It was a lot of interesting, new perspectives, innovative partnerships. But there was one aspect: there’s not a lot of data standards for how police track and report various metrics, including use-of-force. So you can’t compare one jurisdiction to another always. It can look bad for one department versus another because you used a different set of criteria. There needs to be greater standards in order to better share information.

What’s next for you with open data policing?

There’s going to be an expansion of OpenDataPolicingNC.com and that’s through Code for Durham. We’re going to be using geolocational data provided by Fayetteville and Police Chief Harold Medlock. He asked us to map the data to see what it highlights. We hope other agencies can use it, too, once the Fayetteville one is online. It’s an exciting project and we’re honored Chief Medlock asked us to help out.

Colin Copeland at the White House Police Data Initiative. Also pictured, representatives from other open police data initiatives.

Caktus GroupWhat We’re Clicking - April Link Roundup

It's time for this month’s roundup of articles and posts shared by Cakti that drew the most attention on Twitter. The list highlights new work in civic tech and international development as well as reasons for the increasing popularity of Python and open source development.

Python is an Equal Opportunity Programming Language

An interview with David Stewart, manager in the Intel Data Center Software Technology group, about the unique accessibility of the Python programming language as well as the inclusivity of its community.

Why Every Developer is an Open Source Developer Now

A short article on why the future of IT lies in open source collaboration.

A Debate Where the Voters Pick the Questions

The Atlantic’s profile of the Florida Open Debate platform. Caktus Group helped build the tool on behalf of the Open Debate Coalition. The platform powered the first-ever crowd sourced open Senate debate.

Making it Easy to Bring Cellphone Apps to Africa

A wonderful Fast Company's profile on Africa’s Talking, a startup devoted to making it easier for developers to disseminate SMS-based apps to cell phone users in Africa.

Caktus GroupFlorida Open Debate Platform Receives National Attention (The Atlantic, USA Today, Engadget)

Several national publications have featured the Florida Open Debate platform, including USA Today, Engadget, and The Atlantic. Caktus helped develop the Django-based platform on behalf of the Open Debate Coalition (ODC) in advance of the nation’s first-ever open Senate debate held in Florida on April 25th. The site enabled citizens to submit debate questions as well as vote on which questions mattered most to them. Moderators then used the thirty most popular questions from the site to structure the debate between Florida Senate candidates David Jolly (R) and Alan Grayson (D). According to The Atlantic, more than 400,000 votes were submitted by users on the site, including more than 84,000 from Florida voters.

Florida Open Debate user-submitted questions

“Normally, the press frames important US election debates by choosing the questions and controlling the video broadcast,” wrote Steve Dent. “For the first time, however, the public... decide[d] the agenda.”

In his article for The Atlantic, Russell Berman also applauded the site’s effort “to make bottom-up, user-generated questions the centerpiece of a debate.” But possibly more significant were the results of this crowd-sourced content. “What transpired was, by all accounts, a decent debate,” Berman writes. “For 75 minutes, Grayson and Jolly addressed several weighty policy disputes—money in politics, Wall Street reform, the minimum wage, climate change, the solvency of Social Security—and often in detail.”

The Florida debate was streamed live on Monday to more than 80,000 viewers. The Open Debate platform is receiving attention and interest from various potential debate sponsors as well as the Commission on Presidential Debates for possible use in the in this fall’s presidential elections.

Caktus GroupES6 For Django Lovers

ES6 for Django Lovers!

The Django community is not one to fall to bitrot. Django supports every new release of Python at an impressive pace. Active Django websites are commonly updated to new releases quickly and we take pride in providing stable, predictable upgrade paths.

We should be as adamant about keeping up that pace with our frontends as we are with all the support Django and Python put into the backend. I think I can make the case that ES6 is both a part of that natural forward pace for us, and help you get started upgrading the frontend half of your projects today.

The Case for ES6

As a Django developer and likely someone who prefers command lines, databases, and backends you might not be convinced that ES6 and other Javascript language changes matter much.

If you enjoy the concise expressiveness of Python, then ES6's improvements over Javascript should matter a lot to you. If you appreciate the organization and structure Django's common layouts for projects and applications provides, then ES6's module and import system is something you'll want to take advantage of. If you benefit from the wide variety of third-party packages the Python Package index makes available to you just a pip install away, then you should be reaching out to the rich ecosystem of packages NPM has available for frontend code, as well.

For all the reasons you love Python and Django, you should love ES6, too!

Well Structured Code for Your Whole Project

In any Python project, you take advantage of modules and packages to break up a larger body of code into sensible pieces. It makes your project easier to understand and maintain, both for yourself and other developers trying to find their way around a new codebase.

If you're like many Python web developers, the lack of structure between your clean, organized Python code and your messy, spaghetti Javascript code is something that bothers you. ES6 introduces a native module and import system, with a lot of similarities to Python's own modules.

import React from 'react';

import Dispatcher from './dispatcher.jsx';
import NoteStore from './store.jsx';
import Actions from './actions.jsx';
import {Note, NoteEntry} from './components.jsx';
import AutoComponent from './utils.jsx'

We don't benefit only from organizing our own code, of course. We derive an untold value from a huge and growing collection of third-party libraries available in Python and often specifically for Django. Django itself is distributed in concise releases through PyPI and available to your project thanks to the well-organized structure and the distribution service provided by PyPI.

Now you can take advantage of the same thing on the frontend. If you prefer to trust a stable package distribution for Django and other dependencies of your project, then it is a safe bet to guess that you are frustrated when you have to "install" a Javascript library by just unzipping it and committing the whole thing into your repository. Our Javascript code can feel unmanaged and fragile by comparison to the rest of our projects.

NPM has grown into the de facto home of Javascript libraries and grows at an incredible pace. Consider it a PyPI for your frontend code. With tools like Browserify and Webpack, you can wrap all the NPM installed dependencies for your project, along with your own organized tree of modules, into a single bundle to ship with your pages. These work in combination with ES6 modules to give you the scaffolding of modules and package management to organize your code better.

A Higher Baseline

This new pipeline allows us to take advantage of the language changes in ES6. It exposes the wealth of packages available through NPM. We hope it will raise the standard of quality within our front-end code.

This raised bar puts us in a better position to continue pushing our setup forward.

How Caktus Integrates ES6 With Django

Combining a Gulp-based pipeline for frontend assets with Django's runserver development web server turned out to be straightforward when we inverted the usual setup. Instead of teaching Django to trigger the asset pipeline, we embedded Django into our default gulp task.

Now, we set up livereload, which reloads the page when CSS or JS has been changed. We build our styles and scripts, transforming our Less and ES6 into CSS and Javascript. The task will launch Django's own runserver for you, passing along --port and --host parameters. The rebuild() task delegated to below will continue to monitor all our frontend source files for changes to automatically rebuild them when necessary.

// Starts our development workflow
gulp.task('default', function (cb) {
  livereload.listen();

  rebuild({
    development: true,
  });

  console.log("Starting Django runserver http://"+argv.address+":"+argv.port+"/");
  var args = ["manage.py", "runserver", argv.address+":"+argv.port];
  var runserver = spawn("python", args, {
    stdio: "inherit",
  });
  runserver.on('close', function(code) {
    if (code !== 0) {
      console.error('Django runserver exited with error code: ' + code);
    } else {
      console.log('Django runserver exited normally.');
    }
  });
});

Integration with Django's collectstatic for Deployments

Options like Django Compressor make integration with common Django deployment pipelines a breeze, but you may need to consider how to combine ES6 pipelines more carefully. By running our Gulp build task before collectstatic and including the resulting bundled assets — both Less and ES6 — in the collected assets, we can make our existing Gulp builds and Django work together very seamlessly.

References

Tim HopperBackpacking for the Very Tall

I created a single page website to collect notes on one of my other hobbies: ultralight backpacking. In particular, notes on ultralight gear for the very tall.

Caktus GroupKittens at Caktus: Raising Money and Awareness for Local Charity Alley Cats and Angels

This week we welcomed three kittens into our office as part of a campaign to raise funds and awareness for local non-profit Alley Cats and Angels. This organization is dedicated to improving the lives of stray, abandoned, and feral cats. They also work to reduce the number of homeless cats in the Triangle through adoption, farm cat, and spay/neuter assistance programs.

Kittens at Caktus (affectionately called Catkus Day), was a direct result of Lead Developer and Technical Manager Karen Tracey’s volunteer efforts with Alley Cats and Angels. After seeing some of her adorable fosters, the team suggested that they come to our office for a day.

As an overwhelmingly cat-friendly office and a long-time supporter of Alley Cats and Angels, inviting kittens into our office turned out to be a fun, new way to support a great cause.

Follow the link to donate to Alley Cats and Angels or to find other opportunities for helping stray and abandoned cats and kittens in the Triangle.

NC Nwoko with kitten

Daryl Riethof with kitten

Caktus GroupFlorida Open Debate Site Powers First-Ever Crowd-Sourced Open Senate Debate

Florida Open Debate launched ahead of the upcoming, bi-partisan debate between candidates for the Florida Senate. The site, which crowdsources debate questions from the general public, was met with national acclaim. Citizens can not only submit questions, but also vote on which ones matter most. Caktus helped develop the tool on behalf of the Open Debate Coalition (ODC), a non-partisan organization dedicated to supporting participatory democracy through the use of civic tech.

The coalition formed during the 2008 presidential election to combat a sharp decline in voter participation as well as lack of representation in the debate arena. According to The Los Angeles Times “the job of designing and choosing questions is left to the media host.” The ODC recognized the need to increase participation from and provide access to the debate system by reaching as many American citizens as possible.

"The tool truly is an open forum for US citizens to participate in the political process,” says Ben Riseling, Caktus project manager. “Anyone can submit a question and vote on which questions should or should not be discussed. We’re extremely honored to be asked to participate in making this site available during this election season.”

The debate between Florida Senate candidate David Jolly (R) and Alan Grayson (D) takes place Monday, April 25th at 7:00 pm EDT and will be live streamed on the Florida Open Debate site itself.

Jeff TrawickBrief notes on replacing the HD in an Asus M32CD with SSD


The goal: Clone the Windows 10 OS partition from the original 1GB hard disk to a new 480GB Crucial SSD

The 'net said to use the Clone feature of Todo Backup Free 9.1. I tried. The clone didn't work. Both the sector for sector copy and the SSD optimize options were selected. The problem symptom after switching cables was that the machine didn't boot but instead loaded the Asus BIOS screen. (What may have been different for me: GPT instead of MBR, the particular arrangement of partitions on the 1TB source drive, user error, etc.)

The 'net said to use AOMEI Partition Assistant Standard Edition. I tried. This free version does not support GPT partitions, so it wouldn't do anything. I didn't pay up to get the version that does.

The 'net said to use Paragon Migrate OS to SSD 4.0. PC Magazine had a review that suggested that it would be easy to use for my situation. (That is not different than what I read before about other solutions.) I paid the $19.95. It just worked, no drama. Done.

Caktus GroupShipIt Day Recap: Q2 2016

Last Friday, the Cakti set aside regular client projects for our quarterly ShipIt Day, a chance for personal development and independent projects. People work individually or in groups to flex their creativity, tackle interesting problems, or expand their personal knowledge. This quarter’s ShipIt Day was all about open source contributions, frontend fixes, and learning new (or revisiting old) programming languages. Read about the various ShipIt Day projects for Q2 of 2016 below.


Dan Poirier decided to teach himself a new programming language for ShipIt Day and made his way through Learn You a Haskell for Great Good. Despite its mystifying title, Dan found the guide very helpful for learning this purely functional programming language. He ended the day by writing a “hello, world!” program as well as a program to count words and lines of an input.

Like Dan, Neil Ashton decided to play around with a purely functional programming language, throwing himself into a refresher with Clojure. By the end of the day he felt comfortable working his way around Clojure’s libraries and built a basic web application using Clojure for the backend and ClojureScript for the frontend. He especially enjoyed the way ClojureScript plugin Figwheel enabled him to edit code with instantaneously visible results.

David Ray wrote a Python script for automating parts of the sales process in Google Drive. His work greatly improved internal processes for the sales team. As always, it was APIs to the rescue! He is now looking into integrating with other 3rd party services to automate even more of the process, with the end goal of creating a bot to generate the scaffolding from our HipChat room.

Inspired by a recent workshop hosted by the [Triangle User Experience Professionals Association TriUXPA on designing better meetings, NC Nwoko teamed up with Tania Lee. The two used UX Design principles to generate ways to improve meetings here at Caktus. They focused on ways to prioritize ideas to determine meeting agendas, make meetings more visual, improve meeting experience for those working remotely, ask better questions, and become better meeting facilitators.

Caktus meetings improvement notes

Hunter MacDermut developed a game using Phaser that combines fitness and gaming. Using the FitBit’s API, Hunter created a game requiring in-app purchases in the form of FitBit steps. In other words, the games uses up a bank of total steps the user can build up throughout the day. Once this bank is used up, the user has to take more steps in order to continue game play.

Vinod Kurup took a look at Lambda, Amazon’s serverless computing service, which requires that the user only write one function. Lambda then provides the infrastructure to scale that function up and down as needed. Vinod explored the benefits, functionality, and scalability of the service for potential use in later projects. He also found out that Mark Lavin had created a Pingdom-type project using Lambda.

Erin Mullaney, Mark Lavin, and Dmitriy Chukhin collaborated to build a front end application with map and chart data visualizations using javascript and Python. The data for these visualizations was drawn from a RapidPro SMS survey taken live during their presentation. The project will ultimately be a part of the Caktus booth at this year’s PyCon!

dogs versus cats pie chart

In preparation for her sponsor workshop at PyCon this year with Erin, Leveraging Text Messaging in 2016 with RapidPro, Rebecca Muraya reviewed the history of RapidSMS and RapidPro as well as Caktus’ involvement in and relationship to these platforms.

Kia Lam returned to front-end basics to play around with CSS. She decided to recreate the Caktus logo using only HTML and CSS. She substituted the Caktus logotype with a Google Web font with similar characteristics. Using the power of CSS pseudo-elements she was able to add the necessary styling and elements of the Caktus logo without adding any additional markup to her HTML. She then positioned each element absolutely within a relatively positioned container, therefore ensuring that the styling elements would always be positioned perfectly in relation to the logotype. For the finishing touch, she applied an “eraser” container that would make it appear as if the bottom half of the styling elements were cut off to create the desired effect. You can see her work on codepen

Calvin Spealman made improvements to Caktus’ Django Project Template around our frontend build tooling. He replaced it with a few lines of code that enable the sharing of build tasks between multiple projects and developed a library for the Cakti to use across future projects.

Fascinated by images he had seen altering photographs with convolutional neural networks, Jeff Bradberry familiarized himself with the available Neural Network libraries including Theano, Caffe, and Torch.

Charlotte Fouque evaluated a few cross-browser testing tools and settled on Browsera to find bugs, frontend mistakes, and javascript errors and clean up a few websites. She also took some time to play around with Selenium to automate web functionality testing.

Tobias McNulty took the extra time to contribute back to the open source Django community. He modified and reviewed several tickets and gave a brief talk on why contributing to open source is so important.

Alex Lemann and Victor Rocha helped out Code for Durham by continuing to build additions to the Durham School Navigator. The two continued to build a survey tool to improve the site’s school profiles and worked on the layout of the profiles.

Colin Copeland improved the UX to OpenDataPolicingNC, a project he launched in cooperation with the Southern Coalition for Social Justice as well as a team of volunteer developers. He also built a sample homepage for possible expansion of the open data policing tool to other states.

OpenDataPolicingNC

Caktus GroupFrom Intern to Professional Developer: Advice on a Mid-Career Pivot

A few weeks ago, Rebecca Conley attended DjangoCon Europe 2016 in Budapest, Hungary. The event is a five-day conference that brings together Django lovers from all over the world to learn about and share each other’s experiences with Django.

Rebecca gave an inspiring talk on her transition into web development from other fields, including the non-profit sector. She approached the topic from a unique diversity in tech perspective, arguing that developers making such transitions have a great deal to offer the developer community as a whole. You can watch her talk below or check out the many other great talks here.

Tim HopperEcontalk

Listening to Russ Roberts' Econtalk podcast for the last 5 years has given me a whole new perspective on the world. Roberts has exposed me to a whole new way of economic thinking, refined my scientific skepticism, and introduced me to copious topics and scholars.

Here are episodes from over the years that I've particularly enjoyed.

One of my favorite guests is Duke economist Mike Munger. Here are some great interviews with him:

Incidentally, Priceonomics recently published a great article on Roberts and Econtalk.

Footnotes