A planet of blogs from our members...

Astro Code SchoolAstro Launch Party

RSVP: astro-caktus.eventbrite.com
What: Astro Code School Launch Party
Where: 108 Morris Street, Suite 1B, Durham, NC 27705
When: May 1, 2015, 6pm - 9pm

You are invited to the Astro Code School launch party! We’ll have light refreshments and opportunities to meet the fine folks at Astro and Caktus Consulting Group. Come learn more about the first full-time code school to specialize in Python and Django on the East Coast!

Please RSVP at the URL above. I hope you can make it!

Astro Code SchoolFULL Class Syllabus for Python & Django Web Engineering

A day by day full class syllabus with a lot more information about what you can learn in our Python & Django Web Engineering class is now available. It's now all on it's own page. (BTW, we call this class BE 102. BE stands for Back End. It's a formal name to differentiate it from other classes we plan on providing.)

The deadline to apply for our first class is May 11. If you're interested please head on over to the Apply page and fill out the form.


Caktus GroupEpic Allies Team Members to Speak at Innovate your Cool

The Art of Cool festival is a staple of spring happenings in the Triangle. A three-day festival to present, promote, and preserve jazz and jazz-influenced music, The Art of Cool always promises to be a great time for those interested in music, art, and delicious food from Durham’s many food trucks. But what does music have to do with programming and app development? This year, Caktus Group is helping to sponsor a new portion of the festival called Innovate Your Cool. Innovate Your Cool celebrates the power of cool ideas, advancing innovative thinking by bringing together intelligent people with radically new ideas.

Not only is Caktus helping to sponsor the event, but our very own Digital Health Product Manager NC Nwoko will be giving a lightning talk on “Hacking HIV Stigma with Game Apps” with Assistant Professor at UNC Gillings Scool of Global Public Health Kate Muessig. Both Kate and NC are part of the team of intelligent people working on the Epic Allies gaming app for young men and teens who are HIV positive.

The Epic Allies project, originally begun in 2012 in collaboration with Duke and UNC, is a gaming app that seeks to make taking HIV medication—as well as creating and maintaining healthy habits—fun. The app uses games and social networking to reinforce drug adherence, thereby lowering viral loads and curbing the spread of HIV. It is an innovative mHealth solution for a high-risk population in critical need, an ideal topic for the Innovate Your Cool conference.

Also present will be the keynote speaker, Wayne Sutton, talking about diversity in the fields of tech and innovation. Other topics of the event will include startup culture, innovation on “Black Wall Street,” community and economic development, and a panel discussion on Code the Dream. Dr. Chris Emdin of #hiphoped will also be leading a hackathon combining science and hip hop and geared towards high school aged students.

Innovate your Cool is Saturday, April 25th from 10 am to 4pm and will be hosted at American Underground. Register today. We can’t wait!

Caktus GroupFrom Intern to Professional Developer

Quite often, people undertake internships at the beginning of their career. For me, it was a mid-career direction change and a leap of faith. In order to facilitate this career move, I took a Rails Engineering class at The Iron Yard in the fall of 2014. I had limited experience as a developer and no experience in Django prior to my internship at Caktus. Because of the structure and support Caktus provided and my enthusiasm for becoming a developer, my internship turned out to be the ideal way for me to make the transition from a novice to a professional developer.

What I Expected

When I chose to make this career shift, I read and thought a great deal about the challenges I might face due to my age and gender. I had minimal apprehensions about coding itself. I like math and languages, and I’m a good problem-solver. My concerns were about how I would navigate a new industry at this point in my life.

While I had general concerns about making this leap, I was sure Caktus was the place I wanted to try it. When I was in code school, I met Caktus employees and saw some of the work they do, particularly SMS apps in the developing world. It was clear that Caktus’ values as a company align well with mine. They are principled, creative people whose apps make significant and sustainable positive impact on people’s lives. I was excited to be part of a team whose work I supported so wholeheartedly.

What Caktus did

Caktus did a number of things, both consciously and subconsciously, to create a welcoming and supportive environment in which I could learn and succeed. The sexism and ageism that is allegedly rampant in tech is notably absent at Caktus. My co-workers understood that I was a capable but inexperienced developer. They were all eager to share their knowledge and help without making any assumptions about me or my abilities. Sharing knowledge cooperatively is standard operating procedure throughout Caktus, and I think it’s one of the reasons the company is so successful.

Something Caktus did, very deliberately, to help me was to provide me with a female mentor, Karen Tracey, Lead Developer and Technical Manager at Caktus. While any of the developers at Caktus would make great mentors, pairing me with a woman who has worked as a developer for her entire career was incredibly valuable. Karen provided me with thoughtful guidance and insight gained from her experience. She was able to guide me in career choices, goal setting, and on navigating an industry that can be very unwelcoming to women. She showed me that I can succeed and be happy in this industry and, more importantly, helped me figure out how. She also helped me strategize about how I can open doors for others in this industry, particularly those from groups underrepresented in tech. That’s a personal goal of mine, and I one I know I will find support from Caktus in pursuing.

What Rob did

Caktus provided additional support in the form of another co-worker, Rob Lineberger who worked very closely and patiently with me on coding itself. We worked on a real client project, and Rob was very good at scaling work for me so that I could experience some challenges and some accomplishments each day. When I was stuck on a problem, Rob intuited what conceptual background I needed to move forward. He walked me through problems so that I would be able to use the skills and knowledge I was acquiring in the future when I was working on a problem on my own. Working with Rob on this project ended up being a series of lessons in the fundamentals of web development that, in the end, gave me a broad and useful toolbox to use after the internship.

What I did

Because the project was well managed, I was able to work on a variety of different pieces in order to get a really good sense of how a Django app works. One piece of which I took significant ownership was a routing app that communicated with the google directions API. This app in particular required that I explore Javascript and JQuery in a confined, practical context, a very useful opportunity for me to expand my skills. Having discrete, challenging, yet attainable assignments like this created an ideal learning experience, and I was able to produce code that was demonstrated to the client.

In addition to this app, I worked with tables, database logic, and testing, all essential to understanding how Django apps work. I gained knowledge and confidence, and I had a lot of fun coding and getting to know my co-workers professionally and personally. The experience allowed me to see myself as a developer, more specifically as a developer at Caktus. Happily, Caktus saw me the same way, and I am thrilled to continue as a full-time developer with this passionate, dedicated, and inspiring group of people.

Astro Code SchoolAstro at PyCon 2015

Hello from Montréal, QC! We're here participating in the annual North American 2015 Python Conference.

So far Caleb has helped out at the Django Girls Workshop with three other Caktus Group colleagues.

Caleb teaching at the Django Girls workshop at PyCon2015

I went to the PyCon Education Summit. Great to see folks from around the world, including North Carolina, share cutting edge education ideas. Lots of amazing K-12 and University examples of how Python is teaching programming.

Caleb teaching at Django Girls Workshop at PyCon 2015

We're now hanging out at the Expo telling folks from around the world about Durham and our school. So far I've met people from Poland, Canada, India, Hawaii, and lots of US States. Very fun to represent for North Carolina.

Frank WierzbickiJython 2.7 release candidate 2 available!

On behalf of the Jython development team, I'm pleased to announce that the second release candidate of Jython 2.7 is available! We've now fixed the windows installer issues from rc1. I'd like to thank Amobee for sponsoring my work on Jython. I'd also like to thank the many contributors to Jython.

Please see the NEWS file for detailed release notes. This release of Jython requires JDK 7 or above.

This release is being hosted at maven central. There are three main distributions. In order of popularity:
To see all of the files available including checksums, go here and navigate to the appropriate distribution and version.

Frank WierzbickiJython 2.7 release candidate 1 available!

[Update: on Windows machines the installer shows an error at the end. The installer needs to be closed manually, but then the install should still work. We will fix this for rc2.]

On behalf of the Jython development team, I'm pleased to announce that the first release candidate of Jython 2.7 is available! We're getting very close to a real release finally! I'd like to thank Amobee for sponsoring my work on Jython. I'd also like to thank the many contributors to Jython.

Jython 2.7rc1 brings us up to language level compatibility with the 2.7 version of CPython. We have focused largely on CPython compatibility, and so this release of Jython can run more pure Python apps then any previous release. Please see the NEWS file for detailed release notes. This release of Jython requires JDK 7 or above.

This release is being hosted at maven central. There are three main distributions. In order of popularity:
To see all of the files available including checksums, go here and navigate to the appropriate distribution and version.

Caktus GroupHow to Find Cakti at PyCon 2015

We’re very excited for PyCon 2015 and can’t wait for the fun to begin. Working on the PyCon website stoked our excitement early, so it’s almost surreal that PyCon is finally here. With an overwhelming number of great events, we wanted to highlight ones Caktus and our staff will be taking part in. Below you’ll find a list of where we’ll be each day. Please join us!

Wednesday: Building an SMS App with Django (3:30pm)

Ever wanted to build an SMS app? It’s UNICEF’s tool of choice for reaching the most remote and under-resourced areas in the world. Our team (Mark Lavin, David Ray, Caleb Smith) can walk you through the process.

Thursday: DjangoGirls Workshop (9am)

We’re a proud DjangoGirls sponsor. For this workshop, Mark Lavin and Karen Tracey, the leaders of our development team, David Ray, and Astro Code School lead instructor Caleb Smith, will act as TAs and help participants create their first Django app.

Thursday: O’Reilly Book Signing and Opening Reception (6pm)

Mark Lavin will be signing 25 free(!) copies of his O’Reilly book, Lightweight Django. Stop on by while you can— it’s first come first serve. You’ll also find many of us at the reception itself.

Friday - Saturday: Tradeshow

Stop by our tradeshow booth. You can also visit our latest venture, Astro Code School at their own booth. Bring a friend and have a showdown with our Ultimate Tic Tac Toe game (you’ll get some pretty sweet stickers too). We’ll also have daily giveaways like Mark’s Lightweight Django and some mini-quadcopters.

Saturday: PyLadies Auction (6pm)

There’s some fantastic art and other items being offered during the PyLadies auction. Caktus is contributing a framed piece showing early concept art and sketches for PyCon 2015.

Sunday: Job Fair

Do you want to join our team and become a part of the nation’s largest Django firm? Then please come by our booth at the job fair. We’d love to talk to you about ways you can grow with Caktus.

The Whole Thing: Outings with Duckling (24/7)

We cannot neglect to mention the giant duck. You’ll find our duck, nicknamed Quaktus, standing next to the "Meet here for Outings" sign. We built Duckling.us to help people create impromptu get togethers during PyCon. You can use the app to figure out where everyone is going for dinner, drinks, etc. and join in the fun.

Astro Code SchoolPyCon 2015 : See You in Montreal!

Caleb Smith and I are going to Montréal, Quebec, Canada next week for PyCon 2015! It's a huge conference all about the open-source Python programming language. Python is a big part of what we teach here at Astro Code School.

We’ll be at booth #613 in Exhibit Hall 210 in the Palais des Congres. Please come look for us. We’ll have the usual swag like t-shirts for women and men. PLUS we’ll have the very addictive game Ultimate Tic Tac Toe. Play against one another on our big touch screen. It’s harder than it sounds. Will you be a Ultimate Tic Tac Toe champion? Can we win more games than Caktus Group?

Caleb is co-presenting with our Caktus colleagues on Wednesday April 8 from 3:30 p.m. - 5 p.m on Building SMS Applications with Django. He’s also coaching at the Django Girls Workshop April 9. No programming experience required. Just bring a laptop and some energy to learn. You’ll be going through the awesome Django Girls tutorial.

I’ll be attending the Python Education Summit. I’m really looking forward to learning more from other professional and amateur python educators. The talk schedule looks nice!

Are you going to PyCon 2015? What parts of PyCon 2015 are you looking forward too? Tutorial Days, Lightening Talks, or Dev Sprints? Let us know by tweeting at us @AstroCodeschool.

Caktus GroupDiamondHacks 2015 Recap

Image via Diamond Hacks Facebook Page

This past weekend, Technical Director Mark Lavin came out to support DiamondHacks, NCSU’s first ever hackathon and conference event for women interested in computer science. Not only is NCSU Mark’s alma mater, but he’s also a strong supporter of co-organizer Girl Develop It RDU (GDI), of which Caktus is an official sponsor.

The weekend’s events began Saturday with nervous excitement as Facebook developer Erin Summers took the stage for her keynote address. Most memorable for Mark was a moment towards the end of Summers’ talk, in which she called for collaboration between neighboring audience members. It was at this point Mark first realized he was the only male in the room—a unique experience for a male developer. “I’m sure there’s lots of women who have felt the way I did,” Mark commented. The moment not only flipped the norms of a traditionally male-dominated field, but also filled Mark with a renewed appreciation for the importance of active inclusivity in the tech industry.

Aside from helping fill swag bags for the weekend’s participants and attending several of the talks, Mark gave a lightning talk, “Python and Django: Web Development Batteries Included” on Python and Django. Knowing attendees would be thinking about their upcoming projects and which language to build in, Mark chose to advocate for Django (he’s a little biased as the co-author of Lightweight Django). He highlighted the overall uses of Python as well as the extensiveness of its standard library. According to Mark, “Python comes with a lot of built-in functionality,” so it’s a great coding language for beginning developers. Mark also covered the basic Django view and model in his talk, emphasizing the features that make Django a complete framework—an excellent language for a hackathon.

Since supporting diversity in the tech industry was a key focus of the day, Mark also wanted to emphasize the inclusiveness of the Python and Django communities. From the diversity statement on Python’s website, to Django’s code of conduct, the Python community and Django subcommunity have been at the forefront of advocating for diversity and inclusion in the tech world. For Mark, this element has and continues to be “important for the growth of [the] language,” and has contributed to the vitality of these communities.

All in all the weekend was a great success, with especially memorable talks given by speakers working for Google, Trinket, and Hirease. Mark was impressed with the students’ enthusiasm and focus and lingered after both of iterations his talk to speak with attendees about their careers and interests. The next day he was equally affected by the range and talent behind Sunday’s hackathon projects as he followed the progress of various teams on Twitter. “These are the students [who] are going to help define what’s next,” he remarked.

Can’t get enough of Python, Django, and the talented Mark Lavin? Neither can we. Mark will be leading a workshop at PyCon on Building SMS Applications with Django along with fellow Cakti David Ray and our code school’s lead instructor, Caleb Smith. We’ll hope to see you there!

Tim HopperParsley the Recipe Parser

A few years ago, I created a Github repo with only a readme for a project I was hoping to start. The project was a tool for parsing ingredients from cooking recipes. I never did start this project, and I just decided to delete the Github repository. What follows is the README file I had written.

The parser should take in an unstructured ingredient recipe string and output a structured version of the ingredient.

In particular, we follow the structure described by Rahul Agarwal and Kevin Miller in a Stanford CS 224n class project. They four aspects of an ingredient (bullets quoted directly): AMOUNT: Defines the quantity of some ingredient. Does not refer to lengths of time, sizes of objects, etc. UNIT : Specifies the unit of measure of an ingredient. Examples include "cup", "tablespoons", as well as non-standard measures such as "pinch". INGREDIENT: The main food word of an item that is mentioned in the ingredient list. Groups or transformations of sets of ingredients (such as “dough”) do not fall into this category DESCRIPTION: A word or phrase that modifies the type of food mentioned, such as the word "chopped".

For example, the ingredient string

1 teaspoon finely chopped, peeled fresh ginger

will be parsed as follows:

  • AMOUNT: 1
  • UNIT : tsp
  • INGREDIENT: ginger
  • DESCRIPTION: finely chopped, peeled


2 (11 ounce) can mandarin orange segments, drained

will/might be parsed as:

  • AMOUNT: 22
  • UNIT : oz
  • INGREDIENT: mandarin orange segments
  • DESCRIPTION: drained

Astro Code SchoolMeet Caktus CTO Colin Copeland

This is the third post in a series of interviews about the people at Astro Code School. This one is about Colin Copeland the CTO and Co-Founder of Caktus Consulting Group. He’s one of the people who came up with the idea for Astro Code School and a major contributor to it's creation.

Where were you born?

Oberlin, Ohio

What was your favorite childhood pastime?

Spending time with friends during the Summer.

Where did you go to college and what did you study?

I went to Earlham College and studied Computer Science.

How did you become a CTO of the nation's largest Django firm?

I collaborated with the co-founders on a software engineering project. We moved to North Carolina to start the business. I was lucky to have met them!

How did you and the other Caktus founders come up with the idea to start Astro Code School?

Caktus has always been involved with trainings and trying to contribute back to the Django community where possible, from hosting Django sprints to leading public and private Django trainings on best practices. We're excited to see the Django community grow and saw an opportunity to focus our training services with Astro.

What is one of your favorite things about Python?

Readability. Whether it's reading through some of my old source code or diving into a new open source project, I feel like you can get your bearings quickly and feel comfortable learning or re-learning the code. The larger Django and Python communities are also very welcoming and friendly to new and long time members.

Who are your mentors and how have they influenced you?

So many, but especially my Caktus business partners and colleagues.

Do you have any hobbies?

I'm co-captain of the Code for Durham Brigade.

Which is your favorite Sci-fi or Fantasy fiction? Why?

Sci-fi. I've always loved the books Neuromancer and Snow Crash. Recently I've been enjoying the Silo science fiction series.

Caktus GroupWelcome to Our New Staff Members

We’ve hit one of our greatest growth points yet in 2015, adding nine new team members since January to handle our increasing project load. There are many exciting things on the horizon for Caktus and our clients, so it’s wonderful to have a few more hands on deck.

One of the best things about working at Caktus is the diversity of our staff’s interests and backgrounds. In order of their appearance from left to right in the photos above, here’s a quick look at our new Cakti’s roles and some fun facts:

Neil Ashton

Neil was also a Caktus contractor who has made the move to full-time Django developer. He is a keen student of more than programming languages; he holds two degrees in Classics and another Master’s in Linguistics.

Jeff Bradberry

Though Jeff has been working as a contractor at Caktus, he recently became a full-time developer. In his spare time, he likes to play around with artificial intelligence, sometimes giving his creations a dose of inexplicable, random behavior to better mimic us poor humans.

Ross Pike

Ross is our new Lead Designer and has earned recognition for his work from Print, How Magazine, and the AIGA. He also served in the Peace Corps for a year in Bolivia on a health and water mission.

Lucas Rowe

Lucas joins us for six months as a game designer, courtesy of a federal grant to reduce the spread of HIV. When he’s not working on Epic Allies, our HIV medication app, he can be found playing board games or visiting local breweries.

Erin Mullaney

Erin has more than a decade of development experience behind her, making her the perfect addition to our team of Django developers. She loves cooking healthy, vegan meals and watching television shows laden with 90s nostalgia.

Liza Chabot

Liza is an English major who loves to read, write, and organize, all necessary skills as Caktus’ Administrative and Marketing Assistant. She is also a weaver and sells and exhibits her handwoven wall hangings and textiles in the local craft community.

NC Nwoko

NC’s skills are vast in scope. She graduated from UNC Chapel Hill with a BA in Journalism and Mass Communication with a focus on public relations and business as well as a second major in International Studies with a focus on global economics. She now puts this experience to good use as Caktus’ Digital Health Product Manager, but on the weekends you can find her playing video games and reading comic books.

Edward Rowe

Edward is joining us for six months as a game developer for the Epic Allies project. He loves developing games for social good. Outside of work, Edward continues to express his passion for games as an avid indie game developer, UNC basketball fan, and board and video game player.

Rob Lineberger

Rob is our new Django contractor. Rob is a renaissance man; he’s not only a skilled and respected visual artist, he’s trained in bioinformatics, psychology, information systems and knows his way around the kitchen.

To learn more about our team, visit our About Page. And if you’re wishing you could spend your days with these smart, passionate people, keep in mind that we’re still hiring.

Tim HopperAuto Deploying Stigler Diet with Travis CI

I've been using Travis CI for automated testing at work for the last year. It never occurred to me that it could be used to deploy a static website.

Greg Reda wrote a great post on using Travis to automatically build his site and deploy it to S3 every time he pushes to Github. I borrowed his brilliance to implement the same technique here.

Tim HopperSundry Links for March 23, 2015

Start Using Landsat on AWS: "The Landsat program has been running since 1972 and is the longest ongoing project to collect such imagery. Landsat 8 is the newest Landsat satellite and it gathers data based on visible, infrared, near-infrared, and thermal-infrared light. … You can now access over 85,000 Landsat 8 scenes" on AWS.

Beginner's Guide to Linkers: I’m getting back into doing a little C++ programming. Having spent the last 5 years in scripting languages, this was a helpful refresher on compilation.

How to Auto-Forward your Gmail Messages in Bulk: Use Google App Scripts to autoforward emails by simply adding a label. I use this to add things to my Omnifocus task link.

Which one result in mathematics has surprised you the most?: On Mathematics Stack Exchange. It might have been Huffman Coding for me.

Ruby Midwest 2013 The Most Important Optimization: Happiness: Ernie Miller explains why he doesn’t let his career trump his happiness.

Sake by tonyfischetti: Something of a modern GNU Make: "Sake is a way to easily design, share, build, and visualize workflows with intricate interdependencies. Sake is self-documenting because the instructions for building a project also serve as the documentation of the project's workflow."

n1k0/SublimeHighlight: "An humble SublimeText package for exporting highlighted code as RTF or HTML."

Caktus GroupAstro Code School Now Accepting Applications - Intermediate Django + Python




I'm really happy to officially announce the first Python and Django Web Engineering class at Astro Code School. I’ll outline some details here and you can also find them on our classes page.

This class is twelve weeks long and full time Monday to Friday from 9 AM – 5 PM. It'll be taught here at the Astro Code School at 108 Morris Street, Suite 1b, Durham, NC. We will conduct two Python and Django Web Engineering classes in 2015. The first one in term two starts May 18, 2015 and ends August 10, 2015. The second one in term three starts September 22, 2015 and ends December 15, 2015.

Enrollment for both sections opens today March 20. There is space for twelve students in each class. More information about the enrollment process is on our Apply page. Part of that process is an entrance exam that is designed to ensure you're ready to succeed. The price per person for Python and Django Web Engineering is $12,000.

The Python and Django Web Engineering class is intended for intermediate level students. Its goal is to help you start your career as a backend web engineer. To start down this path we recommend you prepare yourself. A few things you can do are: read some books on Python & Django, complete the Django Girls tutorial, watch videos on Youtube, and take an online class or two in Python.



Python and Django make a powerful team to build maintainable web applications quickly. When you take this course you will build your own web application during lab time with assistance from your teacher and professional Django developers. You’ll also receive help preparing your portfolio and resume to find a job using the skills you’ve learned.

Here's the syllabus:

  1. Python Basics, Git & GitHub, Unit Testing
  2. Object Oriented Programming, Functional Programming, Development Process, Command Line
  3. HTML, HTTP, CSS, LESS, JavaScript, DOM
  4. Portfolio Development, Intro to Django, Routing, Views, Templates
  5. SQL, Models, Migrations, Forms, Debugging
  6. Django Admin, Integrating Apps, Upgrading Django, Advanced Django
  7. Ajax, JavaScript, REST
  8. Linux Admin, AWS, Django Deployment, Fabric
  9. Interviewing Skills, Computer Science Topics, Review
  10. Final Project Labs
  11. Final Project Labs
  12. Final Project Labs


This comprehensive course is taught by experienced developer and trained teacher Caleb Smith. He's been working full time at Caktus Consulting Group, the founders of Astro Code School and the nation’s largest Django firm. He’s worked on many client projects over the years. He’s also applied his experience as a former public school teacher to teach Girl Develop It Python classes and as an adjunct lecturer at the University of North Carolina-Chapel Hill. I think you'll really enjoy working with and learning from Caleb. He's a wonderful person.

For the past six months we've been working very hard to launch the school. A large amount of our time has been spent on a application to receive our license from the State of North Carolina to conduct a proprietary school. As of today Astro is one of two code schools in North Carolina that have received this license. We found it a very important task to undertake. It helped us do our due diligence to run a honest and fair school that will protect the rights of students who will be attending Astro Code School. This long process also explains why we've waited to tell you all the details. We're required to wait till we have a license to open our application process.

Thanks for checking out Astro Code School. If you have any questions please contact me.

Astro Code SchoolAnnouncing the Python & Django Web Engineering Class


I'm really happy to officially announce the first Python and Django Web Engineering class at Astro Code School. I’ll outline some details here and you can also find them on our classes page.

This class is twelve weeks long and full time Monday to Friday from 9 AM – 5 PM. It'll be taught here at the Astro Code School at 108 Morris Street, Suite 1b, Durham, NC. We will conduct two Python and Django Web Engineering classes in 2015. The first one in term two starts May 18, 2015 and ends August 10, 2015. The second one in term three starts September 22, 2015 and ends December 15, 2015.

Enrollment for both sections opens today March 20. There is space for twelve students in each class. More information about the enrollment process is on our Apply page. Part of that process is an entrance exam that is designed to ensure you're ready to succeed. The price per person for Python and Django Web Engineering is $12,000.

The Python and Django Web Engineering class is intended for intermediate level students. Its goal is to help you start your career as a backend web engineer. To start down this path we recommend you prepare yourself. A few things you can do are: read some books on Python & Django, complete the Django Girls tutorial, watch videos on Youtube, and take an online class or two in Python.

Python and Django make a powerful team to build maintainable web applications quickly. When you take this course you will build your own web application during lab time with assistance from your teacher and professional Django developers. You’ll also receive help preparing your portfolio and resume to find a job using the skills you’ve learned.

Here's the syllabus:

  1. Python Basics, Git & GitHub, Unit Testing
  2. Object Oriented Programming, Functional Programming, Development Process, Command Line
  3. HTML, HTTP, CSS, LESS, JavaScript, DOM
  4. Portfolio Development, Intro to Django, Routing, Views, Templates
  5. SQL, Models, Migrations, Forms, Debugging
  6. Django Admin, Integrating Apps, Upgrading Django, Advanced Django
  7. Ajax, JavaScript, REST
  8. Linux Admin, AWS, Django Deployment, Fabric
  9. Interviewing Skills, Computer Science Topics, Review
  10. Final Project Labs
  11. Final Project Labs
  12. Final Project Labs

This comprehensive course is taught by experienced developer and trained teacher Caleb Smith. He's been working full time at Caktus Consulting Group, the founders of Astro Code School and the nation’s largest Django firm. He’s worked on many client projects over the years. He’s also applied his experience as a former public school teacher to teach Girl Develop It Python classes and as an adjunct lecturer at the University of North Carolina-Chapel Hill. I think you'll really enjoy working with and learning from Caleb. He's a wonderful person.

For the past six months we've been working very hard to launch the school. A large amount of our time has been spent on a application to receive our license from the State of North Carolina to conduct a proprietary school. As of today Astro is one of two code schools in North Carolina that have received this license. We found it a very important task to undertake. It helped us do our due diligence to run a honest and fair school that will protect the rights of students who will be attending Astro Code School. This long process also explains why we've waited to tell you all the details. We're required to wait till we have a license to open our application process.

Thanks for checking out Astro Code School. If you have any questions please contact me.

Caktus GroupWhy RapidSMS for SMS Application Development

Caktus has been involved in quite a few projects (Libyan voter registration, UNICEF Project Mwana, and several others) that include text messaging (a.k.a. Short Message Service, or SMS), and we always use RapidSMS as one of our tools. We've also invested our own resources in supporting and extending RapidSMS.

There are other options; why do we consistently choose RapidSMS?

What is RapidSMS

First, what is RapidSMS? It's an open source package of useful tools that extend the Django web development framework to support processing text messages. It includes:

  • A framework for writing code to be invoked when a text message is received and respond to it
  • A set of backends - pluggable code modules that can interface to various ways of connecting your Django program to the phone network to pass text messages back and forth
  • Sample applications
  • Documentation

The backends are required because unlike email, there's no universal standard for sending and receiving text messages over the Internet. Often we get access to the messages via a third party vendor, like Twilio or Tropo, that provides a proprietary interface. RapidSMS isolates us from the differences among vendors.

RapidSMS is open source, under the BSD license, with UNICEF acting as holder of the contributors' agreements (granting a license for RapidSMS to use and distribute their contributions). See the RapidSMS license for more about this.


Here are some of the alternatives we might have chosen:

  • Writing from scratch: starting each project new and building the infrastructure to handle text messages again
  • Writing to a particular vendor's API: writing code that sends and receives text messages using the programming interface provided by one of the online vendors that provide that service, then building applications around that
  • Other frameworks

Why RapidSMS

Why did we choose RapidSMS?

  • RapidSMS builds on Django, our favorite web development framework.
  • RapidSMS is at the right level for us. It provides components that we can use to build our own applications the way we need to, and the flexibility to customize its behavior.
  • RapidSMS is open source, under the BSD license. There are no issues with our use of it, and we are free to extend it when we need to for a particular project. We then have the opportunity to contribute our changes back to the RapidSMS community.
  • RapidSMS is vendor-neutral. We can build our applications without being tied to any particular vendor of text messaging services. That's good for multiple reasons:
  • We don't have to pick a vendor before we can start.
  • We could change vendors in the future without having to rewrite the applications.
  • We can deploy applications to different countries that might not have any common vendor for messaging services.

It's worth noting that using RapidSMS doesn't even require using an Internet text messaging vendor. We can use other open source applications like Vumi or Kannel as a gateway to provide us with even more options:

  • use hardware called a "cellular/GSM modem" (basically a cell phone with a connection to a computer instead of a screen)
  • interface directly to a phone company's own servers over the Internet, using several widely used protocols


RapidSMS is a good fit for us at Caktus, it adds a lot to our projects, and we've been pleased to be able to contribute back to it.

Caktus will be leading a workshop on building RapidSMS applications during PyCon 2015 on Tuesday, April 7th 3:00-5:30.

Tim HopperSundry Links for March 13, 2015

Dynamically Update a Plot in IPython: One thing I miss about Mathematica is Animate and Manipulate. IPython is slowing getting similar functionality. Here’s how to dynamically update a plot.

Jiahao's IPython Notebook customizations: Drop this CSS file on your machine, and suddenly your IPython notebooks look quite beautiful!

Duet Display: I tried Air Display a few years ago, and it wasn’t worth the hassle. But Duet Display is a fantastic way to turn your iPad into an external display.

Creating publication-quality figures with Matplotlib: Plotting in Python frustrates me to no end. But here’s a nice tutorial on creating nice figures in with Matplotlib.

retrying 1.3.3 : Python Package Index: Python decorators "to simplify the task of adding retry behavior to just about anything." These work like a charm!

Astro Code SchoolMeet Brian Russell Our Director

This is the second in a series of interviews about the people at Astro Code School. This one is about Brian Russell the Astro Code School Director. He's the guy who does day to day management and works to tell the world about the school.

Where were you born?
I was born in Richmond, Virginia.

What was your favorite childhood pastime?
Drawing with a pencil on paper.

Where did you go to college and what did you study?
I went to Virginia Commonwealth University and earned a Bachelors of Fine Arts in Sculpture and Painting. The majority of my studio work involved creating video installation art. This was done using early non-linear video editing software to create short movies. Those movies were then displayed in sculpture that involved performance art and dance choreography.

How did you go from being a fine artist to a director of a school?
It's a rather long and winding road. But after college I immediately started working as a graphic designer. This led to web design and development work. Later I started a business called Carrboro Creative Coworking and cut my teeth running a business. During that time I worked for many different corporations and several Universities doing tech support and teaching multimedia software. Technical literacy education is a real thread of interest in my career.

What is one of your favorite things about Python?
I've really like Python's readability and how approachable that makes it. Plus the people I've met in the Python community locally and internationally are really cool.

Who are your mentors and how have they influenced you?
Besides my art professors I've learned a lot from my accountant. Seriously, she's awesome! :)

Do you have any hobbies?
I am an avid film photographer. Right now I'm deep into medium format film.

Which is your favorite Sci-fi or Fantasy fiction? Why?
Science Fiction hands down. I've been a sci-fi geek ever sense I saw the "first" Star Wars in the theater. Plus I love William Gibson's writing and Star Trek. LLAP!

Caktus GroupCaktus is Durham Living Wage Certified

Caktus Group recently became a Durham Living Wage Certified Employer! The Durham Living Wage Certification Programis a project of the Durham People’s Alliance. The group asks that local businesses voluntarily certify themselves as living wage employers in order to identify, acknowledge, and celebrate those businesses. A living wage is the amount of income needed for an individual to meet her or his basic needs without public or private assistance. Caktus is proud to be a part of efforts to build a just and sustainable local economy.

You can read more about the program and the requirements for certification at DurhamLivingWage.org, or you can apply online to get your business certified.

Josh JohnsonDevOps Is Bullshit: Why One Programmer Doesn’t Do It Anymore

I’ve always been handy with hardware. I was one of “those kids” you hear about that keeps taking things apart just to see how they work – and driving their parents nuts in the process. When I was a teenager, I toyed with programming but didn’t get serious with it until I decided I wanted to get into graphic design. I found out that you don’t have to write HTML yourself, you can use programming to do it for you!

But I never stopped tinkering with hardware and systems. I used Linux and BSD on my desktop for years, built my LAMP stacks from source, and simulated the server environment when I couldn’t – when I used windows for work, and when I eventually adopted Apple as my primary platform, I first started with cross-compiled versions of the components, and eventually got into virtualization.

In the early days (maybe 10 years ago) there seemed to be few programmers who were like me, or if they were, they never took “operations” or “sysadmin” jobs, and neither did I. So there was always a natural divide. Aside from being a really nice guy who everyone likes, I had a particular rapport with my cohorts who specialized in systems.

I’m not sure exactly what it was. It may have been that I was always interested in the finer details of how a system works. It may have been my tendency to document things meticulously, or my interest in automation and risk reduction. It could have just been that I was willing to take the time to cross the divide and talk to them, even when I didn’t need something. It may have just boiled down to the fact that when they were busy, I could do things myself, and I wanted to follow their standards, and get their guidance. It’s hard to tell, even today, as my systems skills have developed beyond what they ever were before, but the rapport has continued on.

And then something happened. As my career progressed, I took on more responsibilities and did more and more systems work. This was partly because of the divide widening to some extent at one particular job, but mostly because, I could. Right around this time the “DevOps Revolution” was beginning.

Much like when I was a teenager and everyone needed a web site, suddenly everyone needed DevOps.

I didn’t really know what it was. I was aware of the term, but being a smart person, I tend to ignore radical claims of great cultural shifts, especially in technology. In this stance, I find myself feeling a step or two behind at times, but it helps keep things in perspective. Over time, technology changes, but true radicalism is rare. Most often, a reinvention or revisiting of past ideas forms the basis for such claims. This “DevOps” thing was no different. Honestly, at the time it seemed like a smoke screen; a flashy way to save money for startups.

I got sick of tending systems – when you’re doing it properly, it can be a daunting task. Dealing with storage, access control, backups, networking, high availability, maintenance, security, and all of the domain-specific aspects can easily become overwhelming. But worse, I was doing too much front-line support, which honestly, at the time was more important than the programming it was distracting me from. I love my users, and I see their success as my success. I didn’t mind it, but the bigger problems I wanted to solve were consistently being held above my head, just out of my grasp. I could ignore my users or ignore my passion, and that was a saddening conundrum. I felt like all of the creativity I craved was gone, and I was being paid too much (or too little depending on if you think I was an over paid junior sysadmin or an under paid IT manager with no authority) to work under such tedium. So I changed jobs.

I made the mistake of letting my new employer decide where they wanted me to go in the engineering organization.

What I didn’t know about this new company was that it had been under some cultural transition just prior to bringing me on board. Part of that culture shift was incorporating so-called “DevOps” into the mix. By fiat or force.

Because of my systems experience, I landed on the front line of that fight: the “DevOps Team”. I wasn’t happy.

But as I dug in, I saw some potential. We had the chance to really shore up the development practices, reduce risk in deployments, make the company more agile, and ultimately make more money.

We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it. These things included continuous integration, migrating from subversion to git, building and maintaining code review tools, and maintaining the issue tracking system. We had other, less explicit responsibilities that became central to our work later on, including developer support, release management, and interfacing with the separate, segregated infrastructure department. This interaction was especially important, since we had no systems of our own, and we weren’t allowed to administer any machines. We didn’t have privileged access to any of the systems we needed to maintain for a long time.

With all the hand wringing and flashing of this “DevOps” term, I dug in and read about it, and what all the hubub was about. I then realized something. What we were doing wasn’t DevOps.

Then I realized something else. I was DevOps. I always had been. The culture was baked into the kind of developer I was. Putting me, and other devs with similar culture on a separate team, whether that was the “DevOps” team or the infrastructure team was a fundamental mistake.

The developers didn’t come around. At one point someone told a teammate of mine that they thought we were “IT support”. What needed to happen was the developers had to embrace the concept that they were capable of doing at least some systems things themselves, in safe and secure manner, and the infrastructure team had to let them do it. But my team just sat there in the middle, doing what we could to keep the lights on and get the code out, but ultimately just wasting our time. Some developers starting using AWS, with the promise of it being a temporary solution, but in a vacuum nonetheless. We were not having the impact that management wanted us to have.

My time at this particular company ended in a coup of sorts. This story is worthy of a separate blog post some day when it hurts a little less to think about. But lets just say I was on the wrong side of the revolution and left as quickly as I could when it was over.

In my haste, I took another “DevOps” job. My manager there assured me that it would be a programming job first, and a systems job second. “We need more ‘dev’ in our ‘devops'”, he told me.

What happened was very similar to my previous “DevOps” experience, but more acute. Code, and often requirements, were thrown over the wall at the last minute. As it fell in our laps, we scrambled to make it work, and work properly, as it seemed no one would think of things like fail over or backups or protecting private information when they were making their plans. Plans made long ago, far away, and without our help.

This particular team was more automation focused. We had two people who were more “dev” than “ops”, and the operations people were no slouches when it came to scripting or coding in their own right.

It was a perfect blend, and as a team we got along great and pulled off some miracles.

But ultimately, we were still isolated. We, and our managers tried to bridge the gap to no avail. Developers, frustrated with our sizable backlog, went over our heads to get access to our infrastructure and started doing it for themselves, often with little or no regard for our policies or practice. We would be tasked with cleaning up their mess when it was time for production deployment – typically in a major hurry after the deadline had passed.

The original team eventually evaporated. I was one of the last to leave, as new folks were brought into a remote office. I stuck it out for a lot of reasons: I was promised transfer to NYC, I had good healthcare, I loved my team. But ultimately what made me stick around was the hope, that kept getting rebuilt and dashed as management rotated in and out above me, that we could make it work.

I took the avenue of providing automated tools to let the developers have freedom to do as they pleased, yet we could ensure they were complying with company security guidelines and adhering to sane operations practices.

Sadly, politics and priorities kept my vision from coming to reality. It’s OK, in hindsight, because so much more was broken about so-called “DevOps” at this particular company. I honestly don’t think that it could have made that much of a difference.

Near the end of my tenure there, I tried to help some of the developers help themselves by sitting with them and working out how to deploy their code properly side-by-side. It was a great collaboration, but it fell short. It represented a tiny fraction of the developers we supported. Even with those really great developers finally interfacing with my team, it was too little, too late.

Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow.

I had one final “DevOps” experience before I put my foot down and made the personal declaration that “DevOps is bullshit”, and I wasn’t going to do it anymore.

Due to the titles I had taken, and the experiences of the last couple of years, I found myself in a predicament. I was seen by recruiters as a “DevOps guy” and not as a programmer. It didn’t matter that I had 15 years of programming experience in several languages, or that I had focused on programming even in these so-called “DevOps” jobs. All that mattered was that, as a “DevOps Engineer” I could be easily packaged for a high-demand market.

I went along with the type casting for a couple of reasons. First, as I came to realize, I am DevOps – if anyone was going to come into a company and bridge the gap between operations and engineering, it’d be me. Even if the company had a divide, which every company I interviewed with had, I might be able to come on board and change things.

But there was a problem. At least at the companies I interviewed at, it seemed that “DevOps” really meant “operations and automation” (or more literally “AWS and configuration management”). The effect this had was devastating. The somewhat superficial nature of parts of my systems experience got in the way of landing some jobs I would have been great at. I was asked questions about things that had never been a problem for me in 15 years of building software and systems to support it, and being unable to answer, but happy to talk through the problem, would always end in a net loss.

When I would interview at the few programming jobs I could find or the recruiters would give me, they were never for languages I knew well. And even when they were, my lack of computer science jargon bit me – hard. I am an extremely capable software engineer, someone who learns quickly and hones skills with great agility. My expertise is practical, however, and it seemed that the questions that needed to be asked, that would have illustrated my skill, weren’t. I think to them, I looked like a guy who was sick of systems that was playing up their past dabbling in software to change careers.

So it seemed “DevOps”, this great revolution, and something that was baked into my very identity as a programmer, had left me in the dust.

I took one final “DevOps” job before I gave up. I was optimistic, since the company was growing fast and I liked everyone I met there. Sadly, it had the same separations, and was subject to the same problems. The developers, who I deeply respected, were doing their own thing, in a vacuum. My team was unnecessarily complicating everything and wasting huge amounts of time. Again, it was just “ops with automation” and nothing more.

So now lets get to the point of all of this. We understand why I might think “DevOps is bullshit”, and why I might not want to do it anymore. But what does that really mean? How can my experiences help you, as a developer, as an operations person, or as a company with issues they feel “DevOps” could address?

Don’t do DevOps. It’s that simple. Apply the practices and technology that comprise what DevOps is to your development process, and stop putting up walls between different specialties.

A very wise man once said “If you have a DevOps team, you’re doing it wrong“. If you start doing that, stop it.

There is some nuance here, and my experience can help save you some trouble by identifying some of the common mistakes:

  • DevOps doesn’t make specialists obsolete.
  • Developers can learn systems and operations, but nothing beats experience.
  • Operations people can learn development too, but again, nothing beats experience.
  • Operations and development have historically be separated for a reason – there are compromises you must make if you integrate the two.
  • Tools and automation are not enough.
  • Developers have to want DevOps. Operations have to want DevOps. At the same time.
  • Using “DevOps” to save money by reducing staff will blow up in your face.
  • You can’t have DevOps and still have separate operations and development teams. Period.

Let me stop for one moment and share another lesson I’ve learned: if it ain’t broke, don’t fix it.

If you have a working organization that seems old fashioned, leave it alone. It’s possible to incorporate the tech, and even some of the cultural aspects of DevOps without radically changing how things work – it’s just not DevOps anymore, so don’t call it that. Be critical of your process and practices, kaizen and all that, but don’t sacrifice what works just to join the cargo cult. You will waste money, and you will destroy morale. The pragmatic operations approach is the happiest one.

Beware of geeks bearing gifts.

So lets say you know why you want DevOps, and you’re certain that the cultural shift is what’s right for your organization. Everyone is excited about it. What might a proper “DevOps” team look like?

I can speak to this, because I currently work in one.

First, never call it “DevOps”. It’s just what you do as part of your job. Some days you’re writing code, other days you’re doing a deployment, or maintenance. Everyone shares all of those responsibilities equally.

People still have areas of experience and expertise. This isn’t pushing people into a luke-warm, mediocre dilution of their skills – this is passionate people doing what they love. It’s just that part of that, is launching a server or writing a chef recipe or debugging a production issue.

As such you get a truly cross functional team. Where expertise differs, first, there’s a level of respect and trust. So if someone knows more about a topic than someone else, they will likely be the authority on it. The rest of the team trusts them to steer the group in the right direction.

This means that you can hire operations people to join your team. Just don’t give them exclusive responsibility for what they’re best at – integrate them. The same goes for any “non deveoper” skillset, be that design, project managment or whatever.

Beyond that, everyone on the team has a thirst to develop new skills and look at their work in different ways. This is when the difference in expertise provides an opportunity to teach. Teaching brings us closer together and helps us all gain better understanding of what we’re doing.

So that’s what DevOps really is. You take a bunch of really skilled, passionate, talented people who don’t have their heads shoved so far up their own asses that they can take the time to learn new things. People who see the success of the business as a combined responsibility that is eqully shared. “That’s not my job” is not something they are prone to saying, but they’re happy to delegate or share a task if need be. You give them the infrastructure, and time (and encouragement doesn’t hurt), to build things in a way that makes the most sense for their productivity, and the business, embracing that equal, shared sense of responsibility. Things like continuous integration and zero-downtime deployments just happen as a function of smart, passionate people working toward a shared goal.

It’s an organic, culture-driven process. We may start doing continuous deployment, or utlize “the cloud” or treat our “code as infrastructure” but only if it makes sense. The developers are the operations people and the operations people are the developers. An application system is seen in a holistic manner and developed as a single unit. No one is compromising, we all get better as we all just fucking do it.

DevOps is indeed bullshit. What matters is good people working together without artificial boundaries. Tech is tech. It’s not possible for everyone to share like this, but when it works, it’s amazing – but is it really DevOps? I don’t know, I don’t do that anymore.

Astro Code SchoolMeet Caleb Smith Our Lead Instructor

This is the first in a series of interviews about the people at Astro Code School. This one is about Caleb Smith the Astro Code School Lead Instructor. He’s the guy who writes our curriculum for our Python & Django Web Engineering class that he’s teaching this year.

Where were you born?
I grew up in Hickory, in the piedmont of North Carolina.

What was your favorite childhood pastime?
Programming DOS games in BASIC. I spent far too much time working on making an RPG I called "Water and Stone".

Where did you go to college and what did you study?
I studied Music Education at UNC-Greensboro.

How did you get into Web Development with Python and Django?
After about two years of learning C++ and front-end web development on my own, I moved to the Triangle area hoping to find a role in the tech sector. I applied for the Caktus summer internship and was able to ramp up quickly thanks to some excellent mentorship from the team. I was hired on as a junior developer after that as my first professional job doing web development.

What did you do professionally before becoming a web developer?
I taught elementary music K-5 in Asheville, North Carolina for two years. I found public school teaching really rewarding but difficult. I spent a lot of my free time doing hobby programming until deciding to pursue programming professionally.

What is one of your favorite things about Python? What about Django?
I like the readability of Python the most and I also appreciate that it is well designed but practical considerations are allowed to trump purity. It makes for a really nice language and system to work in. I like that Django makes so many details of web development irrelevant because it abstracts over them well and is also careful about correctness and security concerns.

Who are your teaching mentors and how did they influence how you teach?
I learned the most from Dr. Randy Kohlenburg, my trombone teacher at UNC-Greensboro. Dr. Kohlenburg thinks a lot about pedagogy and taught us a lot about how to apply those ideas in our own teaching. He's the best mentor I've ever had.

Is there a connection between music and computer programming for you?
When I was about 12 I became really interested in music, joined band, and pursued music education in college. While taking music theory courses, especially in post-tonal analysis, I thought of ways to automate the work involved. I wrote some simple BASIC programs to help double check my work. I rewrote this later in C++ and yet again in Python, which I eventually released as the sator library on PyPI. Through this work, I realized that I had a strong interest in programming that went beyond my initial interest of making games as a kid.

Do you have any hobbies?
I still play trombone and guitar when I can find the time. I've recently been trying to pick up khoomei. (Editor: A type of throat singing.) I'd like to eventually do something with programming and electronic music.

Which is your favorite Sci-fi or Fantasy fiction? Why?
Sci-fi. "Dune" and "Neuromancer" are two of my all-time favorites, and some recent works like "Leviathan Wakes" are great reads too. Good science fiction captures my imagination of where society might be leading in ways that fantasy doesn't, but I do like it too.

Joe GregorioSix Places

One of the questions that comes up regularly when talking about zero frameworks is how can you expect to stitch together an application without a framework? The short answer is "the same way you stitch together native elements," but I think it's interesting and instructional to look at those ways of stitching elements together individually.

There are six surfaces, or points of contact, between elements, that you can use when stitching elements together, whether they are native or custom elements.

Before we go further a couple notes on terminology and scope. For scope, realize that we are only talking about DOM, we aren't talking about composing JS modules or strategies for composing CSS. For the terminology clarification, when talking about DOM I'm referring to the DOM Interface for an element, not the element markup. Note that there is a subtle difference between the markup element and the DOM Interface to such an element.

For example, <img data-foo="5" src="https://example.com/image.png"/> may be the markup for an image. The corresponding DOM Interface has an attribute of src with a value of "https://example.com/image.png", but the corresponding DOM Interface doesn't have a "data-foo" attribute, instead all data-* attributes are available via the dataset attribute on the DOM Interface. In the terminology of the WhatWG Living Standard, this is the distinction between content attributes vs IDL attributes, and I'll only be referring to IDL attributes. So with the preliminaries out of the way let's get into the six surfaces that can be used to stitch together an application.

Attributes and Methods

The first two surfaces, and probably the most obvious, are attributes and methods. If you are interacting with an element it's usually either reading and writing attribute values:


or calling element methods:


Technically these are the same thing, as they are both just properties with different types. Native elements have their set of defined attributes and methods, and depending on which element a custom element is derived from it will also have that base element's attributes and methods along with the custom ones it defines.


The next two surface are events. Events are actually two surfaces because an element can listen for events,

ele.addEventListener(‘some-event’, function(e) { /* */ });

and an element can dispatch its own events:

var e = new CustomEvent(‘some-event’, {details: details});

DOM Position

The final two surfaces are position in the DOM tree, and again I'm counting this as two surfaces because each element has a parent and can be a parent to another element. Yeah, an element has siblings too, but that would bring the total count of surfaces to seven and ruin my nice round even six.

  <img src="">

Combinations are powerful

Let's look at a relatively simple but powerful example, the 'sort-stuff' element. This is a custom element that allows the user to sort elements. All children of 'sort-stuff' with an attribute of 'data-key' are used for sorting the children of the element pointed to by the sort-stuff's 'target' attribute. See below for an example usage:

<sort-stuff target="#sortable">
   <button data-key=one>Sort on One</button>
   <button data-key=two>Sort on Two</button>
 <ul id=sortable>
   <li data-one=c data-two=x>Item 3</li>
   <li data-one=a data-two=z>Item 1</li>
   <li data-one=d data-two=w>Item 4</li>
   <li data-one=b data-two=y>Item 2</li>
   <li data-one=e data-two=v>Item 5</li>

If the user presses the "Sort on One" button then the children of #sortable are sorted in alphabetical order of their data-one attributes. If the user presses the "Sort on Two" button then the children of #sortable are sorted in alphabetical order of their data-two attributes.

Here is the definition of the 'sort-stuff' element:

    function Q(query) {
      return Array.prototype.map.call(
          function(e) { return e; });

    var SortStuffProto = Object.create(HTMLElement.prototype);

    SortStuffProto.createdCallback = function() {
      Q('[data-key]').forEach(function(ele) {
        ele.addEventListener('click', this.clickHandler.bind(this));

    SortStuffProto.clickHandler = function(e) {
      var target = Q(this.getAttribute('target'))[0];
      var elements = [];
      var children = target.children;
      for (var i=0; i<children.length; i++) {
        var ele = children[i];
        var value = ele.dataset[e.target.dataset.key];
          value: value,
          node: ele
      elements.sort(function(x, y) {
        return (x.value == y.value ? 0 : (x.value > y.value ? 1 : -1));
      elements.forEach(function(i) {

    document.registerElement('sort-stuff', {prototype: SortStuffProto});

And here is a running example of the code above:

  • Item 3
  • Item 1
  • Item 4
  • Item 2
  • Item 5

Note the surfaces that were used in constructing this functionality:

  1. sort-stuff has an attribute 'target' that selects the element to sort.
  2. The target children have data attributes that elements are sorted on.
  3. sort-stuff registers for 'click' events from its children.
  4. sort-stuff children have data attributes that determine how the target children will be sorted.

In addition you could imagine adding a custom event 'sorted' that 'sort-stuff' could generate each time it sorts.

So there's your six surfaces that you can use when composing elements into your application. And why the insistence on making the number of surfaces equal six? Because while history may not repeat itself, it does rhyme.

Astro Code SchoolSeven Features an Introductory Programming Language Should Have

Python Logo

Python has recently supplanted Java as the most popular introductory teaching language at top U.S. universities. There are many articles covering this fact from the perspectives of computer science faculty at major universities. I wanted to take a moment to add my own thoughts on the subject.

There are several key features of Python that make it more suitable as an introductory language compared to Java:

  1. A more gradual learning curve

  2. Object-oriented programming is not required

  3. Designed for readability

  4. Less verbosity and boilerplate

  5. An interactive shell for exploratory development

I'd like to tease out each of these points. While both languages can be used to write large and complicated programs, the path from an empty directory to a simple and working program is much more straightforward in Python.

Programmers with little experience can use Python to do simple tasks such as web scraping within a few days or weeks of using the language. There are advanced concepts to learn, but the learning curve is more gradual because more can be accomplished in Python with only simpler, more foundational concepts such as variables and control flow.

Courses that use Java as the teaching language focus heavily on object-oriented programming (or "OOP"). While Python is also object-oriented, it is a multi-paradigm language that can also be used with the functional or structured programming paradigms. While it is important to learn OOP eventually, many learners catch on more quickly to the more concrete structured programming paradigm. It is my view that learning about OOP in the level of detail needed to write a Java program, before completing several small programs that work, is a pedagogical mistake that fundamentally puts these steps out of sequence. i.e. A learner should first write small programs before approaching the techniques and concepts used for writing larger ones well.

Furthermore, Python was designed with readability in mind, and is known for mimicing pseudo-code more closely than other programming languages. Learning the keywords and syntax needed for Java programming obfuscates the overall goal of an introductory course; to teach fundamental programming concepts that surpass a given language or problem domain and enable the learner to obtain key insights that will continue to serve them as they learn more computer science and software engineering concepts.

Lastly, like many other languages, Python features an interactive shell that allows the programmer to try small bits of code at a time and to explore the program being developed from within. This shortens the feedback loop of trying out new ideas compared to having separate compile and run steps. The advantages of an iterative approach with a quick feedback loop for a beginner should be obvious.

With all of this in mind, I find it hard to imagine how Java ever became a common introductory language at all. Explanations for this usually center around the importance of object-oriented programming and the ubiquity of Java in the industry. While these are both good reasons to learn Java, possibly even as a second language, they are far from convincing for the purpose of introducing programming.

I would argue that the features of an introductory programming language are:

  1. A shallow learning curve

  2. A clear and consistent language design

  3. Many libraries available for a variety of needs

  4. An interactive environment such as a shell

  5. Light on clutter, boilerplate or superfluous details

  6. An obvious path toward creating small and simple programs

  7. A rapid rate of development

I've outlined how Python meets most of these points already. On the point regarding libraries, in this regard I think Java and Python both feature a rich ecosystem for beginners and experienced programmers alike. However, considering all of these points, I think a number of languages are more appropriate as an introductory teaching language than Java, including at least the following:

  1. JavaScript

  2. Ruby

  3. Scheme

This brings to mind a much more interesting and difficult question. What makes Python a more appropriate first language than each of these? I'll leave this to a future blog post because I think it needs careful and long form comparisons.

I hope to have made clear why I'm glad that major universities are making the shift to Python for introductory courses. In the future, I hope to broaden this argument and describe how Python is the best first language to learn.

Joe GregoriogRPC

Today Google launched gRPC, a new HTTP/2 and Protocol Buffer based system for building APIs. This is Google's third system for web APIs.

The first system was Google Data, which was based on the Atom Publishing Protocol [RFC 5023]. It was an XML protocol over HTTP. The serving system for that grew, but started to hit scalability issues at around 50 APIs. The scaling issues weren't in the realm of serving QPS, but more in the management of that many APIs, such as rolling out new features across all APIs and all clients.

Those growing pains and lessons learned led to the next generation of APIs that launched in 2010. In addition to writing a whole new serving infrastructure to make launching APIs easier, it was also a time to shed XML and build the protocol on JSON. This Google I/O video contains good overview of the system:

Now, five years later, a third generation API system has been developed, and the team took the opportunity to make another leap, moving to HTTP/2 and Protocol Buffers. This is the first web API system from Google that I haven't been involved in, but I'm glad to see them continuing to push the envelope on web APIs.

Caktus GroupPyCon 2015 Ticket Giveaway

Caktus is giving away a PyCon 2015 ticket, valued at $350. We love going to PyCon every year. It’s the largest gathering of developers using Python, the open source programming language that Caktus relies on. This year, it’ll be held April 8th-16th at the beautiful Palais des congrès de Montréal (the inspiration we used to design the website).

To enter, follow @caktusgroup on Twitter and RT this message.

The giveaway will end Tuesday, March 3rd at 12pm EST. Winner will be notified via Twitter DM. A response via DM is required within 24 hours or entrant forfeits their ticket. Caktus employees are not eligible. Winning entrant must be 18 years of age or older. Ticket is non-transferable.

Bonne chance!

Astro Code SchoolPython & Django Web Engineering Class Syllabus

Our first Python & Django Web Engineering class is fast approaching. More information is forthcoming with a big update to the website. Until then here is a sneak peek at the syllabus. Join our email list to find out the latest info first.

Python & Django Web Engineering 2015

  1. Python Basics, Git & GitHub, Unit Testing

  2. Object Oriented Programming, Functional Programming, Development Process, Command Line

  3. HTML, HTTP, CSS, LESS, JavaScript, DOM

  4. Portfolio Development, Intro to Django, Routing, Views, Templates

  5. SQL, Models, Migrations, Forms, Debugging

  6. Django Admin, Integrating Apps, Upgrading Django, Advanced Django

  7. Ajax, JavaScript, REST

  8. Linux Admin, AWS, Django Deployment, Fabric

  9. Interviewing Skills, Computer Science Topics, Review

  10. Final Project Labs

  11. Final Project Labs

  12. Final Project Labs

Frank WierzbickiJython 2.7 beta4 released!

[Update: some of the download links where wrong, they should now be correct. Sorry for the mistake!] On behalf of the Jython development team, I'm pleased to announce that the fourth beta of Jython 2.7 is available. I'd like to thank Amobee for sponsoring my work on Jython. I'd also like to thank the many contributors to Jython.

Jython 2.7b4 brings us up to language level compatibility with the 2.7 version of CPython. We have focused largely on CPython compatibility, and so this release of Jython can run more pure Python apps then any previous release. Please see the NEWS file for detailed release notes. This release of Jython requires JDK 7 or above.

Jim Baker put together a great summary of the recent work for beta4.
As a beta release we are concentrating on bug fixing and stabilization for a
production release.

This release is being hosted at maven central. The traditional installer can be found here. See the installation instructions for using the installer. Two other versions are available:
To see all of the files available including checksums, go here and navigate to the appropriate distribution and version.

Caktus GroupTriangle Open Data Day and Code Across

International Open Data Day is this weekend, February 21st and 22nd. As part of the festivities, Code for America is hosting its 4th annual CodeAcross. The event aims to unite developers across the country for a day of civic coding, creating tools that make government services simple, effective, easy to use. Put simply, “the goal of CodeAcross is to activate the Code for America network and inspire residents everywhere to get actively involved in their community.”

Technology Tank is hosting CodeAcross as part of Triangle Open Data Day, a chance for people living in the Triangle to come together to learn about open data and hacking for civic good. This year will involve a civic hackathon with something for everyone, from novice to expert coders alike.

Not only is Caktus an official bronze sponsor of this year’s Triangle Open Data Day, but CTO Colin Copeland is also the founder and co-captain of Code for Durham, the Durham chapter of Code for America.

Not sure whether you should attend? Don’t worry, Code for America has provided a helpful flowchart to help you decide. We’ll hope to see you there!

Josh JohnsonClojure + Boot Backend For The React.js Tutorial

Last weekend I worked my way through the introductory tutorial for React.js. It’s very well written and easy to follow, I was really happy with it overall.

For the uninitiated, React.js is a framework that provides a means to create reusable javascript components that emit HTML in a very intuitive way. Taking that concept a step further, its possible to use React on the backend, utilizing the same components to build the UI that is served to the user initially. The end result is very interesting. React prescribes an intuitive and scalable approach to building complex, dynamic user interfaces as highly reusable components.

These user interfaces avoid the redundancy of generating and manipulating HTML twice – once on the server, and again in the browser.

The server-side rendering feels like a natural pattern in a Node.js environment, but there are examples in the wild of doing server-side rendering with other platforms, most notably clojure. This is exciting stuff.

React has been around for a while, but this is the first time I’ve taken a close look at it.

The tutorial focuses on building a simple font-end application rendered entirely in the browser. Initially, you work with a standalone HTML page, and near the end, you integrate it with a simple web application.

The source repository for the tutorial provides some example applications written in Python, Ruby and Node.js.

A simple application like this seemed like an ideal use case for a simple boot script, so I decided to write one of my own. Here’s the code inline, but I’ve forked the repository if you’d like to examine the code along-side its cohorts.

#!/usr/bin/env boot
  #(into % '[[org.clojure/data.json "0.2.5"]
             [ring/ring-core "1.3.2"]
             [ring/ring-jetty-adapter "1.3.2"]]))

(require '[ring.adapter.jetty     :as jetty]
         '[clojure.data.json      :as json]
         '[ring.middleware.params :refer [wrap-params]]
         '[ring.util.response     :refer [file-response response]])

(defn static
  "Handle static file delivery"
  (let [uri (:uri request)
        path (str "./public" uri)]
    (if (= uri "/comments.json")
      (file-response "./_comments.json")
      (file-response path))))

(defn save-comments
  "Save the comments to the json file, and return the new data"
  (let [data (json/read-str (slurp "./_comments.json"))
        input (:form-params request)
        out (concat data [input])
        new-json (json/write-str out)]
    (spit "./_comments.json" new-json)
    (response new-json)))

(defn handler
  "Simple handler that delegates based on the request type"
  (case (:request-method request)
    :post (save-comments request)
    :get (static request)))

(def app
  "Add middleware to the main handler"
  (wrap-params handler))

(defn -main
  [& args]
  (jetty/run-jetty app {:port 3000}))

Essentially, it sets up two handlers, and then a dispatcher that proxies between them depending on the type of request. If the request is a GET, a static file is assumed. This serves the html and any local dependencies. If the request is specifically for comments.json, the handler serves the _comments.json file.

If the request is a POST, its assumed that the body of the request contains a JSON-encoded comment to add. It deserializes that data and the _comments.json file, and appends the new comment to the list. The result is then saved to the filesystem.

Obviously, there is little in the way of error checking going on here. This tracks with the scope of the other example applications.

Note: It’s not clear to me exactly why they used _comments.json to store the data – in my initial prototype I named it comments.json and placed it with the other static files.

Interestingly, this boot script also serves as a minimalistic example of a web application using ring – including adding middleware.

This was a fun way to finish up a really informative tutorial – I’m excited to continue exploring what React.js can do, especially with Clojure!

Special thanks to alandipert and ul from #hoplon for code review and some great advice on cleaning up my initial implementation!

Caktus GroupAstro Code School Tapped to Teach App Development at UNC Journalism School

Our own Caleb Smith, Astro Code School lead instructor, is teaching this semester at UNC’s School of Journalism, one of the nation’s leading journalism schools. He’s sharing his enthusiasm for Django application development with undergraduate and graduate media students in a 500-level course, Advanced Interactive Development.

For additional details about the course and why UNC School of Journalism selected Caktus and Astro Code School, please see our press release.

Caktus GroupPyCon Blog Features Caktus Group

Brian Curtis, the director of the Python Software Foundation, recently interviewed and featured Caktus on the PyCon website. PyCon is the premiere event for those of us within the Python and Django open source communities. Brian writes about our work designing the PyCon 2015 website, our efforts in Libya, and what’s on the horizon in 2015. We're excited about this recognition!

Caktus GroupDjango Logging Configuration: How the Default Settings Interfere with Yours

My colleague Vinod recently found the answer on Stack Overflow to something that's been bugging me for a long time - why do my Django logging configurations so often not do what I think they should?

Short answer

If you want your logging configuration to behave sensibly, set LOGGING_CONFIG to None in your Django settings, and do the logging configuration from scratch using the Python APIs:

LOGGING = {...}  # whatever you want

import logging.config


The kernel of the explanation is in this Stack Overflow answer by jcotton; kudos to jcotton for the answer: before processing your settings, Django establishes a default configuration for Python's logging system, but you can't override it the way you would think, because disable_existing_loggers doesn't work quite the way the Django documentation implies.

The Django documentation for disable_existing_loggers in 1.6, 1.7, and dev (as of January 8, 2015) says "If the disable_existing_loggers key in the LOGGING dictConfig is set to True (which is the default) the default configuration is completely overridden." (emphasis added)

That made me think that I could set disable_existing_loggers to True (or leave it out) and Django's previously established default configuration would have no effect.

Unfortunately, that's not what happens. The disable_existing_loggers flag only does literally what it says: it disables the existing loggers, which is different from deleting them. The result is that they stay in place, they don't log any messages, but they also don't propagate any messages to any other loggers that might otherwise have logged them, regardless of whether they're configured to do so.

What if you try the other option, and set disable_existing_loggers to False? Then your configuration is merged with the previous one (the default configuration that Django has already set up), without disabling the existing loggers. If you use Django's LOGGING setting with the default LOGGING_CONFIG, there is no setting that will simply replace Django's default configuration.

Because Django installs several django loggers, the result is that unless you happened to have specified your own configuration for each of them (replacing Django's default loggers), you have some hidden loggers possibly blocking what you expect to happen.

For example - when I wasn't sure what was going on in a Django project, sometimes I'd try just adding a root logger, to the console or to a file, so I could see everything. I didn't know that the default Django loggers were blocking most log messages from Django itself from ever reaching the root logger, and I would get very frustrated trying to see what was wrong with my logging configuration. In fact, my own logging configuration was probably fine; it was just being blocked by a hidden, overriding configuration I didn't know about.

We could work around the problem by carefully providing our own configuration for each logger included in the Django default logging configuration, but that's subject to breaking if the Django default configuration changes.

The most fool-proof solution is to disable Django's own log configuration mechanism by setting LOGGING_CONFIG to None, then setting the log configuration explicitly ourselves using the Python logging APIs. There's an example above.

The nitty-gritty

The Python documentation is more accurate: "disable_existing_loggers – If specified as False, loggers which exist when this call is made are left enabled. The default is True because this enables old behavior in a backward- compatible way. This behavior is to disable any existing loggers unless they or their ancestors are explicitly named in the logging configuration."

In other words, disable_existing_loggers does literally what it says: it leaves existing loggers in place, it just changes them to disabled.

Unfortunately, Python doesn't seem to document exactly what it means for a logger to be disabled, or even how to do it. The code seems to set a disabled attribute on the logger object. The effect is to stop the logger from calling any of its handlers on a log event. An additional effect of not calling any handlers is to also block propagation of the event to any parent loggers.

Status of the problem

There's been some recent discussion on the developers' list about at least improving the documentation, with a core developer offering to review anything submitted. And that's where things stand.

Tim HopperSundry Links for January 19, 2015

Matt Blodgett: But Where Do People Work in This Office?: "After looking through tons of cool office photos of many of the hottest companies in the Valley, I started to play a fun game I made up called 'spot the desks’. I’ll show you what I mean."

Why We (Still) Believe in Private Offices: Joel Spolsky and Fog Creek Software have been relentless defenders of quite, private offices for developers. They continue that here.

Pandoctor: An Alfred GUI for Pandoc: If you use Pandoc and Alfred, this is worth trying.

Alfred Hop: I use a little bash tool called Hop to bookmark frequently used directories. I made this tool to give me quick access to my bookmarks from Alfred.

Discover Flask: Flask, the lightweight Python framework, is a joy to use. Here’s a nice introduction to it.

Introduction to dplyr: I haven’t used R much since leaving my last job, but the ecosystem has been booming with great tools; dplyr is one of them.

Josh JohnsonBoot: Getting Started With Clojure In < 10 Minutes

With the power of boot, it’s possible to go from “never used java before” to budding Clojure-ist cranking out jars like a pickle factory in record time. This post walks you through the process, and provides some post-‘hello world’ examples, with pointers to more information.


You will need the following: A JDK installed. Really, that’s it. Sun’s JDK or OpenJDK will work. Use the newest version. You’ll need a way to download things. Feel free to use your browser. The examples below use wget. If you’re on Linux or Mac OS, you’ll also need root access via sudo – this is not a hard requirement but allows you to install boot for everyone on your machine to use. There’s an expectation that you know basic Clojure, and tries not to be too clever. For a good introduction, check out Clojure For The Brave and True, specifically Do Things: a Clojure Crash Course. If you need help with specific forms used, the Clojure Community Documentation is extremely helpful, especially the Clojure Cheat Sheet. It may be helpful to give the boot readme and wiki documentation a read. If you have questions about boot, IRC is a great way to get boot and clojure rockstars to help you out. Come join us on freenode, in #hoplon.

Dales la Bota (Give ’em The Boot)

Boot is ‘installed‘ by simply downloading an executable file and putting it somewhere where you can execute it:

$ wget https://github.com/boot-clj/boot/releases/download/2.0.0-rc13/boot.sh
$ mv boot.sh boot &amp;amp;amp;&amp;amp;amp; chmod a+x boot &amp;amp;amp;&amp;amp;amp; sudo mv boot /usr/local/bin

The real magic happens when boot is run. Boot sets everything up in a .boot directory in your home folder. Without having any code to execute yet, you can trigger this by simply asking boot for help:

$ boot -h

Let’s Play With Clojure

Clojure utilizes a concept called a REPL (Read, Evaluate, Print, Loop). REPLs allow you to interactively run code and experiment.

$ boot repl

Boot then provides you with a prompt, where you can play around:

boot.user=&amp;amp;gt; (+ 1 2 3 4 5)
boot.user=&amp;amp;gt; (/ 10 0)

java.lang.ArithmeticException: Divide by zero

Boot also works as a scripting platform – you can construct applications, specifying dependencies, and parse command-line arguments.

Here’s a simple Clojure function that prints the fibonacci sequence to a given number of digits:

(defn fib
    (fib [0 1] n))
  ([pair, n]
    (print (first pair) &amp;amp;quot; &amp;amp;quot;)
    (if (&amp;amp;gt; n 0)
      (fib [(second pair) (apply + pair)] (- n 1))

You can paste this into your REPL and try it out:

boot.user=&amp;amp;gt; (defn fib
       #_=&amp;amp;gt;   ([n]
       #_=&amp;amp;gt;   (fib [0 1] n))
       #_=&amp;amp;gt; ([pair, n]
       #_=&amp;amp;gt;   (print (first pair) &amp;amp;quot; &amp;amp;quot;)
       #_=&amp;amp;gt;   (if (&amp;amp;gt; n 0)
       #_=&amp;amp;gt;     (fib [(second pair) (apply + pair)] (- n 1))
       #_=&amp;amp;gt;     (println))))
boot.user=&amp;amp;gt; (fib 10)
0  1  1  2  3  5  8  13  21  34  55
We can transform that function into a command-line tool using the power of boot scripting. Assume this file is called fib.boot:
#!/usr/bin/env boot

(defn fib
    (fib [0 1] n))
  ([pair, n]
    (print (first pair) &amp;amp;quot; &amp;amp;quot;)
    (if (&amp;amp;gt; n 0)
      (fib [(second pair) (apply + pair)] (- n 1))

(defn -main [&amp;amp;amp; args]
  (let [limit (first args)]
    (println &amp;amp;quot;Printing fibonacci sequence up to &amp;amp;quot; limit &amp;amp;quot;numbers&amp;amp;quot;)
    (fib (Integer/parseInt limit))))

Make the script executable:

$ chmod u+x fib.boot

Now you can run the script:

$ ./fib.boot 10
Printing fibonacci sequence up to  10 numbers
0  1  1  2  3  5  8  13  21  34

The script can declare dependencies, which will be downloaded as needed when the script is run. Here, we’ll show the use of an external dependency: we can write a new fibonacci sequence that utilizes the fact that numbers in the sequence are related to each other by approximately the golden ratio (ca 1.62). Rounding makes it all work, but rounding isn’t “baked in” to Clojure, so we’ll use an external library to do it for us, called math.numeric-tower. Ok, actually, it’s there, you just need to use some existing Java libraries to make it work – I admit this is a bit of a strain!

#!/usr/bin/env boot

(set-env! :dependencies '[[org.clojure/math.numeric-tower &amp;amp;quot;0.0.4&amp;amp;quot;]])
(require '[clojure.math.numeric-tower :refer [floor ceil round]])

(defn fib
  (loop [counter 0 x 0]
    (if (= counter 0)
      (do (print 0 &amp;amp;quot; &amp;amp;quot; 1 &amp;amp;quot; &amp;amp;quot; 1 &amp;amp;quot; &amp;amp;quot;)
        (recur 3 1))
    (let [y (round (* x 1.62))]
      (print y &amp;amp;quot; &amp;amp;quot;)
      (if (&amp;amp;lt; counter 9)
        (recur (+ counter 1) y))))))

(defn -main [&amp;amp;amp; args]
  (let [limit (first args)]
    (println &amp;amp;quot;Printing fibonacci sequence up to&amp;amp;quot; limit &amp;amp;quot;numbers&amp;amp;quot;)
    (fib (Integer/parseInt limit))

When you run this code the first time, you’ll notice boot tells you that it’s downloaded some new jars:

$ ./fib.boot
Retrieving clojure-1.4.0.jar from http://clojars.org/repo/
Retrieving math.numeric-tower-0.0.4.jar from http://repo1.maven.org/maven2/
Printing fibonacci sequence up to  10 numbers
0  1  1  2  3  5  8  13  21  34

The syntax to define our -main function and parse our command line options can be a bit tedious. Luckily, we can borrow a macro from boot.core that lets us specify CLI options using a robust syntax. For the full syntax, check out the documentation. Here, we’ll let the user choose which implementation they’d like to use, and utilize the task DSL to do some simple command line options:

#!/usr/bin/env boot

(set-env! :dependencies '[[org.clojure/math.numeric-tower &amp;amp;quot;0.0.4&amp;amp;quot;]])

(require '[clojure.math.numeric-tower :refer [floor ceil round]])
(require '[boot.cli :as cli])

(defn fib
    (fib [0 1] n))
  ([pair, n]
     (print (first pair) &amp;amp;quot; &amp;amp;quot;)
     (if (&amp;amp;gt; n 1)
       (fib [(second pair) (apply + pair)] (- n 1)))))

(defn fibgolden
  (loop [counter 0 x 0]
    (if (= counter 0)
      (do (print (str 0 &amp;amp;quot;  &amp;amp;quot; 1 &amp;amp;quot;  &amp;amp;quot; 1 &amp;amp;quot;  &amp;amp;quot;))
        (recur 3 1))
    (let [y (round (* x 1.62))]
      (print y &amp;amp;quot; &amp;amp;quot;)
      (if (&amp;amp;lt; counter 9)
        (recur (+ counter 1) y))))))

(cli/defclifn -main
  &amp;amp;quot;Print a fibonacci sequence to stdout using one of two algorithms.&amp;amp;quot;
  [g golden bool &amp;amp;quot;Use the golden mean to calculate&amp;amp;quot;
   n number NUMBER int &amp;amp;quot;Quantity of numbers to generate. Defaults to 10&amp;amp;quot;]
  (let [n (get :n *opts* 10)
        note (if golden &amp;amp;quot;[golden]&amp;amp;quot; &amp;amp;quot;[recursive]&amp;amp;quot;)]
    (println note &amp;amp;quot;Printing fibonacci sequence up to&amp;amp;quot; n &amp;amp;quot;numbers:&amp;amp;quot;)
    (if golden
      (fibgolden n)
      (fib n)))

Now you can see what options are available, tell the script what to do:

$ boot fib.boot -h
Print a fibonacci sequence to stdout using one of two algorithms.

  -h, --help           Print this help info.
  -g, --golden         Use the golden mean to calculate
  -n, --number NUMBER  Set quantity of numbers to generate. Defaults to 10 to NUMBER.

$ boot fib.boot
[recursive] Printing fibonacci sequence up to 10 numbers:
0  1  1  2  3  5  8  13  21  34  

$ boot fib.boot -g -n 20
[recursive] Printing fibonacci sequence up to 20 numbers:
0  1  1  2  3  5  8  13  21  34  55  89  144  233  377  610  987  1597  2584  4181

Working At The Pickle Factory (Packing Java Jars and More Complex Projects)

Now that we’ve got a basic feel for Clojure and using boot, we can build a project, that creates a library with an entry point that we can use and distribute as a jar file. This opens the doors to being able to deploy web applications, build libraries to share, and distribute standalone applications. First, we need to create a project structure. This will help us keep things organized, and fit in with the way Clojure handles namespaces and files. We’ll put our source code in src, and create a new namespace, called fib.core:

$ mkdir -p src/fib

In src/fib/core.clj, we’ll declare our new namespace:

(ns fib.core
  (:require [clojure.math.numeric-tower :refer [floor ceil round]]
            [boot.cli :as cli])

(defn fib
    (fib [0 1] n))
  ([pair, n]
    (print (first pair) &amp;amp;quot; &amp;amp;quot;)
    (if (&amp;amp;gt; n 1)
      (fib [(second pair) (apply + pair)] (- n 1)))))

(defn fibgolden
  (loop [counter 0 x 0]
    (if (= counter 0)
      (do (print (str 0 &amp;amp;quot;  &amp;amp;quot; 1 &amp;amp;quot;  &amp;amp;quot; 1 &amp;amp;quot;  &amp;amp;quot;))
          (recur 3 1))
    (let [y (round (* x 1.62))]
      (print y &amp;amp;quot; &amp;amp;quot;)
      (if (&amp;amp;lt; counter 9)
        (recur (+ counter 1) y))))))

(cli/defclifn -main
  &amp;amp;quot;Print a fibonacci sequence to stdout using one of two algorithms.&amp;amp;quot;
  [g golden bool &amp;amp;quot;Use the golden mean to calculate&amp;amp;quot;
   n number NUMBER int &amp;amp;quot;Quantity of numbers to generate. Defaults to 10&amp;amp;quot;]
  (let [n (if number number 10)
        note (if golden &amp;amp;quot;[golden]&amp;amp;quot; &amp;amp;quot;[recursive]&amp;amp;quot;)]
    (println note &amp;amp;quot;Printing fibonacci sequence up to&amp;amp;quot; n &amp;amp;quot;numbers:&amp;amp;quot;)
    (if golden
      (fibgolden n)
      (fib n)))

To build our jar, there are a handful of steps:

  1. Download our dependencies.
  2. Compile our clojure code ahead of time (aka AOT).
  3. Add a POM file describing our project and the version.
  4. Scan all of our dependencies and add them to the fileset to be put into the jar.
  5. Build the jar, specifying a module containing a -main function to run when the jar is invoked.

Helpfully, boot provides built-in functionality to do this for us. Each step is implemented as a boot task. Tasks act as a pipeline: the result of each can influence the next.

boot -d org.clojure/clojure:1.6.0 \
     -d boot/core:2.0.0-rc8 \
     -d org.clojure/math.numeric-tower:0.0.4 \
     -s src/ \
     aot -a \
     pom -p fib -v 1.0.0 \
     uber \
     jar -m fib.core

A brief explanation of each task and command line options:
Line 1-3: the -d option specifies a dependency. Here we list Clojure itself, boot.core, and math.numeric-tower.

Line 4: -s specifies a source directory to look into for .clj files.

Line 5: this is the AOT task, that compiles all of the .clj files for us. The -a flag tells the task to compile everything it finds.

Line 6: the POM task. This task adds project information to the jar. The -p option specifies the project name, -v is the version.

Line 7: the uber task collects the dependencies so they can be baked into the jar file. This makes the jar big (huge really), but it ends up being self-contained.

Line 8: finally, the jar task. This is the task that actually generates the jar file. The -m option specifies which module has the -main function. Running the above command, produces output something like this:

$ boot -d org.clojure/clojure:1.6.0 \
&amp;amp;gt;      -d boot/core:2.0.0-rc8 \
&amp;amp;gt;      -d org.clojure/math.numeric-tower:0.0.4 \
&amp;amp;gt;      -s src/ \
&amp;amp;gt;      aot -a \
&amp;amp;gt;      pom -p fib -v 1.0.0 \
&amp;amp;gt;      uber \
&amp;amp;gt;      jar -m fib.core
Compiling fib.core...
Writing pom.xml and pom.properties...
Adding uberjar entries...
Writing fib-1.0.0.jar...

At this point, there is a file named fib-1.0.0.jar in the target directory. We can use the java command to run it:

$ java -jar target/fib-1.0.0.jar
[recursive] Printing fibonacci sequence up to 10 numbers:
0  1  1  2  3  5  8  13  21  34

You can send this file to a friend, and they can use it too.

Introducing build.boot

At this point we have a project and can build a standalone jar file from it. This is great, but long command lines are prone to error. Boot provides a mechanism for defining your own tasks and setting the command line options in a single file, named build.boot. Here’s a build.boot that configures boot in a manner equivalent to the command line switches above:

(set-env! :dependencies
          '[[org.clojure/math.numeric-tower &amp;amp;quot;0.0.4&amp;amp;quot;]
            [boot/core &amp;amp;quot;2.0.0-rc8&amp;amp;quot;]
            [org.clojure/clojure &amp;amp;quot;1.6.0&amp;amp;quot;]]
          :source-paths #{&amp;amp;quot;src/&amp;amp;quot;})

  pom {:project 'fib
       :version &amp;amp;quot;1.0.0&amp;amp;quot;}
  jar {:main 'fib.core}
  aot {:all true})

With build.boot in the current directory, you can now run the tasks like this:

$ boot aot pom uber jar
Compiling fib.core...
Writing pom.xml and pom.properties...
Adding uberjar entries...
Writing fib-1.0.0.jar...

The convenience of build.boot one step further, we can chain the tasks we want to use into our own task, using the deftask macro:

(set-env! :dependencies
          '[[org.clojure/math.numeric-tower &amp;amp;quot;0.0.4&amp;amp;quot;]
            [boot/core &amp;amp;quot;2.0.0-rc8&amp;amp;quot;]
            [org.clojure/clojure &amp;amp;quot;1.6.0&amp;amp;quot;]]
          :source-paths #{&amp;amp;quot;src/&amp;amp;quot;})

  pom {:project 'fib
       :version &amp;amp;quot;1.0.0&amp;amp;quot;}
  jar {:main 'fib.core}
  aot {:all true})

(deftask build
  &amp;amp;quot;Create a standalone jar file that computes fibonacci sequences.&amp;amp;quot;
  (comp (aot) (pom) (uber) (jar)))

Now, we can just run boot build to make our standalone jar file. You’ll also see your task show up in the help output:

$ boot -h
   build                      Create a standalone jar file that computes fibonacci sequences.
$ boot build
Compiling fib.core...
Writing pom.xml and pom.properties...
Adding uberjar entries...
Writing fib-1.0.0.jar...

Where To Go From Here

At this point we’ve touched most of the awesomeness that boot gives us. With these basic tools, there’s all sorts of interesting things we can do next. Here are some ideas:

Og Maciel2014 in Book Covers

For my last post of 2014 I wanted to show, with pictures, the books I read and spent so much time with this year.

Back in January of 2014 I set out to read 30 books as part of my Reading Challenge. I wanted to focus on reading Brazilian authors early on as I felt that I really needed to learn more about Brazilian literature and this time, read books for fun and not because I was told to back when I was much younger.

books 1

I took advantage that UNC has a vast books collection with a very decent section on Brazilian authors and I was able to read some really awesome books by Erico Verissimo, Joaquim Manuel de Macedo, José de Alencar, Jorge Amado and Machado de Assis! I also fell in love with these authors and the fact that it took me a couple of decades to truly appreciate them doesn’t bother me at all, since I believe that it took me this long to reach the right maturity level… in other words, I was not ready for them yet until this year.

books 2

I also fell in love with John Steinbeck and Ray Bradbury, and I think that Grapes of Wrath and Dandelion Wine are two of the best books I have ever read!

Lastly, 2014 was also the year I started reading short stories (to sample different authors and see what they ‘have to offer’), and I highly recommend the short stories of Flannery O’Connor, John Cheever, Simon Rich, and Gabriel Garcí­a Márquez.

books 3

In 2015 I plan to continue reading more Brazilian authors and exploring different authors and genres! I may even re-read The Lord of the Rings again. :)

Tim HopperSundry Links for December 29, 2014

Sublime: Nice Features & Plugins: A brief talk introducing my favorite editor.

Alfred Workflow for Pinboard: I've started using Pinboard a bit for organizing links. Here's something that has the chance of getting me much deeper into pinboard: a powerful Alfred Workflow for interacting with Pinboard from your Mac's keyboard. HT Aaron Bachya

fullPage.js: I've been using this jquery plugin in a forthcoming project. It makes it really easy to create slide-like single page websites.

Obfuscating "Hello world!": The author attempts to write the worst 'hello world' possibe in Python. He does a good job.

Show Time in Multiple Time Zones with TextExpander: As I spend more time working with people in different time zones, tools like this help remove the cognitive challenge of translating time.

Tim HopperSundry Links for December 22, 2014

Time: Programmers all hate time, timezones, etc. Here are some helpful "notes about time".

Python strftime reference: Speaking of time: "A quick reference for Python's strftime formatting directives." I have to look this stuff each time I need it.

gitup: "A console script that allows you to easily update multiple git repositories at once"

The “How Does a Google Coder Work?” Edition : I enjoyed this interview. My favorite quote? "When you're reading code is it as clear as reading English?" "If I'm reading C++ code, it's clearer."

Sunset Salvo: John Turkey discusses practical data analysis and statistical humility.

10th Conference on Bayesian Nonparametrics: This is coming up in my own back yard. I’m excited!

Machine Learning: The High-Interest Credit Card of Technical Debt: I haven’t read this in detail, but the premise makes tons of sense to me: "It is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning."

Tim HopperShouldIGetAPhD.com

Last year, I published nine interviews with Internet friends about how an academically-minded, 22-year old college senior should work on a Ph.D. Many people have told me the interviews have been helpful for them or that they've emailed them to others.

I decided to make a dedicated website to host the interviews. You can find it at shouldigetaphd.com.

I hope this continues to be a valuable resource. I'd encourage you to share this with anyone you know who is thinking through this question.

Tim HopperSundry Links for December 6, 2014

Sketching as a Tool for Numerical Linear Algebra: A neat paper on sketching algorithms for linear algebra. No, not that kind of sketching. "One first compresses it to a much smaller matrix by multiplying it by a (usually) random matrix with certain properties. Much of the expensive computation can then be performed on the smaller matrix, thereby accelerating the solution for the original problem."

Maps and the Geospatial Revolution: Coursera is teaching a class in the spring on how geospatical technology has changed our world.

Geoprocessing with Python using Open Source GIS: Speaking of geospatial technology, here are some slides and problems from a class on "Geoprocessing with Python".

How to use the bash command line history: Bash's history can do more than I realized!

A geometric interpretation of the covariance matrix: Superb little post explaning covariance matrices with pictures and geometry.

Og MacielThree Years and Counting!

Making a quick pit stop to mark this milestone in my professional career: today is my 3-year anniversary at Red Hat! Time has certainly flown by and I really cannot believe that it has been three years since I joined this company.

I know it is sort of cliche to say “I can not believe that it has been this long…” and so on and so forth, but it is so true. Back then I joined a relatively new project with very high ambitions, and the first few months had me swimming way out in the deepest part of the pool, trying to learn all ‘Red Hat-things’ and Clojure for the existing automation framework (now we are fully using Python).

I did a lot of swimming for sure, and through the next months, through many long days and weekends and hard work, tears and sweat (you know, your typical life for a Quality Engineer worth his/her salt), I succeeded in adding and wearing many types of hats, going from a Senior Quality Engineer, to a Supervisor of the team, to eventually becoming the Manager for a couple of teams, spread over 4 different countries. Am I bragging? Maaaybe a little bit :) but my point is really to highlight a major key factor that made this rapid ascension path possible: Red Hat’s work philosophy and culture of rewarding those who work hard and truly embrace the company! Sure, I worked really hard, but I have worked just as hard before in previous places and gotten nowhere really fast! Being recognized and rewarded for your hard work is something new to me, and I owe a great debt of gratitude to those who took the time to acknowledge my efforts and allowed me room to grow within this company!

The best part of being a Red Hatter for 3 years? Being surrounded by an enormous pool of talented, exciting people who not only enjoy what they do, but are always willing to teach you something new, and/or to drop what they’re working on to lend you a helping hand! There is not a single day that I don’t learn something new, and thankfully I don’t see any sign of this trend stopping :) Have I mentioned that I love my teammates too? What a great bunch of guys!!! Getting up early in the morning and walking to my home office (yeah, they let me work remotely too) day in, day out, is never a drag because I just know that there are new things to learn and new adventures and ‘achievements to unlock’ right around the corner.

I am Red Hat!!!

Tim HopperSundry Links for December 4, 2014

How do I draw a pair of buttocks?: Have you ever wondered how to plot a pair of buttocks in Mathematica? Of course you have.

Frequentism and Bayesianism: A Python-driven Primer: Jake Vanderplas wrote a "brief, semi-technical comparison" of frequentist and Bayesian statistical inference using examples in Python.

skll: Dan Blanchard released version 1.0 of his very cool command line tool for doing experiments with scikit-learn.

Personalized Recommendations at Etsy: A fantastic post from Etsy's engineering blog on building scalable, personalized recommendations using linear algebra and locally sensitive hashing. I like math.

Pythonic Clojure: Andrew Montalenti wrote a post analyzing Clojure from a Python programmer's perspective. It's great.

Joe Gregoriowsgicollection | BitWorking



The idea of RESTful "Collections", i.e. doing CRUD over HTTP correctly, has been percolating for years now. A Collection is nothing more than a list, a container for resources. While the APP defines a Collection in terms of Atom Feed and Entry documents we don't have to be limited to that. It's time to complete a virtuous circle; RESTLog inspired the Atom Publishing Protocol which inspired David Heinemeier Hansson's World of Resources (pdf) and now it's time to come full circle and get that world of resources in Python.

In particular look at page 18 of that slide deck, where dispatching to a collection of people, the following URIs are to be handled:

  GET    /people         
  POST   /people
  GET    /people/1
  PUT    /people/1
  DELETE /people/1
  GET    /people;new
  GET    /people/1;edit

Now the 'new' and 'edit' URIs can be a bit ambiguous, only in the sense that you might not guess right away that they are nouns, and remember, URIs always identify nouns. I prefer to make the noun-ishness of them more apparent.

  GET    /people;create_form
  GET    /people/1;edit_form

In general, using the notation of Selector, we are looking at URIs of the form:


And dispatching requests to URIs of that form to functions with nice names:

  GET    /people               list()
  POST   /people               create()
  GET    /people/1             retrieve()
  PUT    /people/1             update()
  DELETE /people/1             delete()
  GET    /people;create_form   get_create_form()
  GET    /people/1;edit_form   get_edit_form()

Introducing wsgicollection, a Python library that does just that, simplifying implementing such a Collection under WSGI.

Wsgicollection uses Selector indirectly, relying on it to parse the URIs for {id} and {noun}. In theory it will work with any WSGI middleware that sets values for 'id' and 'noun' in environ['selector.vars'] environ['wsgiorg.routing_args']. Here is how you would define a WSGI application that implements a collection:

from wsgicollection import Collection

class RecipeCollection(Collection):

    # GET /cookbook/
def list(environ, start_response):
# POST /cookbook/
def create(environ, start_response):
# GET /cookbook/1
def retrieve(environ, start_response):
# PUT /cookbook/1
def update(environ, start_response):
# DELETE /cookbook/1
def delete(environ, start_response):
# GET /cookbook/;create_form
def get_create_form(environ, start_response):
# POST /cookbook/1;comment_form
def post_comment_form(environ, start_response):

And this class can be easily hooked up to Selector:

import selector

urls = selector.Selector()

urls.add('/cookbook/[{id}][;{noun}]', _ANY_=RecipeCollection())

Now that I have this Collection class it will ease implementing the APP, but as I indicated earlier, the collection (CRUD) model goes beyond that of just Atom, and we'll dig into that next.

You can find the code here.

Update: Fixed a bug where wsgicollection directly imported selector, which it does not need to do. You will, however, need selector installed to run the unit tests.

Update 2: Updated to support routing_args


Joe GregorioConfession of an Infinite Looper | BitWorking


Confession of an Infinite Looper

I admit it. I'm a looper. I will load up a single track on CD or mp3 and put it on infinite loop. The same song. Over and over. For possibly days at a time. I know I'm not the only one, so fess up if you're a looper too.

Some of my favorites for looping:

  • Ozzy Osbourne - Crazy Train
  • The President of the United States of America - Feather Pluckin
  • Iron Maiden - Childhood's End
  • Talking Heads - Once In A Lifetime

Today's post is brought to you by Warrant's "Down Boys", which I've been looping since Monday...

I saw "Lost in Translation" last Friday and have been playing Jesus & Mary Chain's "Just Like Honey" ever since.

Posted by Larry O'Brien on 2003-10-03

I am a looper too although I usually loop for only a few hours playing songs like:

Alphaville - Forever Young
Vivaldi - Spring
Eagles - Hotel California
Emotions - Best of My Love
Isley Brothers - Shout
James Brown - Play that funky music
Lee Soo Young - Final Fantasy X Theme
Madonna - Ray of Light
Norah Jones - Don't Know Why
Savage Garden - I Knew I Love You
Vanessa William - Colors of the Wind

I think the wide variety is due to my being moody. :-)

Posted by Don Park on 2003-10-03

i keep looping Tori Amos and Nirvana UNplugged to death - a winning combination for me - still strong after few years ...

Posted by Alek on 2003-10-04

Songs I've looped for several days:
Red House Painters - Revelation Big Sur
Alanis Morissette - No Pressure Over Cappucino
Everything But The Girl - Driving (cover)
Jeff Buckley - Last Goodbye
Massive Attack - Protection
They Might Be Giants - Where Do They Make Balloons

As for whole CDs I have looped for days or even weeks, the list goes on. I "learn" entire CDs at a time. For the most part, I wake up with a song in my head and have to scramble to put it on so that it doesn't eat up my brain for the rest of the day.

In the past few months I consistently loop through all my Pizzicato Five tracks on iTunes (8 hours' worth) to stay awake. It's pretty hard to feel down when you're listening to J-pop. :)

Posted by steph on 2003-10-06

My wife happen to look over my shoulder when I was reading this.  "Hah! See?!? I'm not the only one."  She's more of a James Tailor and Van Morrison looper (as classics go).  Otherwise, its every new album.  Right now, it's Dido and Ben Harper, with REM likely on the horizon.

This also reminds me of a guy I knew in college who was really into the various RPGs.  In one (Rifts maybe), he had created a character who was some sort of techno-enhanced soldier.  The background story was that in the heat of a battle, a direct hit caused his music device to get fused to the jack in his head, also breaking the device so it could not be turned off.  One song played over and over and over: "rooster" by Alice in Chains.

There was no point to the story.  The post just reminded me of it.

Posted by Seairth on 2003-10-08

I tend to play Meat Beat Manifesto's "It's the music" over and over again when driving my car. Other favs are "Circle Jack (Chase the magic word lego-lego" by Melt Banana; "The Empty Page" by Sonic Youth; come to think of it, I'm a pretty damn looper myself...

Posted by Adriaan on 2003-10-09


Joe GregorioClearCase as a leading indicator of small technology company failure | BitWorking


ClearCase as a leading indicator of small technology company failure

Is it just me or is ClearCase a leading indicator of small technology company death? I've never used the product, never even seen a demo, and yes, I know, they position themselves as software asset management for medium to large teams. The question comes from the fact that the only people I know that have used ClearCase have used it in the past, all at small technology companies, and they all, without exception, are companies that have gone out of business.

So fill up the comments with your experiences with ClearCase, good or bad, hopefully with someone from at least one small technology company that has succeeded inspite of deploying ClearCase.

Update: In case you are wondering what CMS you should use, consult kittenfight, where subversion beats ClearCase, and so does BitKeeper.

I can't speak for ClearCase, but I worked for a small tech company that nearly destroyed its market share using Rational's "Unified Process." We didn't recover until we dumped all the weight and moved to something more XPish. Rational is where ClearCase originated.

Posted by petrilli on 2004-08-29

Yes, seen it happen at a .com.  They tried to get clearcase (and the paradise of clearquest integration) going for years, while the dev team plodded along using visual source safe (horrible, but it worked...  as long as you don't try to branch.)  The company was delisted, almost went bankrupt, then sold itself for pennies a share.  Then again, it's hard to blame clearcase when the place was offering free overnight shipping on every order.

An ever surer sign of impending doom: the email that comes out from some new vice president you've never heard of, announcing the deal just signed with $MAJOR_VENDOR to implement AN ERP SYSTEM.  When you see this, you are already dead.

Posted by steve minutillo on 2004-08-30

The thing that always amazed me about ClearCase was the sheer incompetence of their Web interface.  I have never seen an app break THAT many usability rules.  It was a nightmare.

I thought the software was pretty good at the core, but that Web interface was (and may still be, for all I know), utterly laughable.

Posted by Deane on 2004-08-30

Actually, Clearcase originated at Atria and was designed by defectors from Apollo Computers back in the late '80s.  I was supporting Apollo dev tools back then when HP bought and dissected the company, and I was one of the original tech supporters of Clearcase.  Even caught one of Paul Levine's marketing talks for 1.0 beta.

Last I used it was 5 years ago with the Chandra X-Ray Observatory project, which worked well since that project was fractured among many companies and locations around the country and had a large diverse code base.  But after 5 years in the eCompany space I'm not sure who uses it anymore or why.

Posted by chris burleson on 2004-08-30

Now you've probably gone and ruined my Pagerank for "Clearcase sucks".  I wrote down all my grudges against The Great Satan of Version Control (ooohhh, I like that) here: http://www.trmk.org/~adam/blog/archive/000106.html

Posted by Adam Keys on 2004-08-30

Clearcase is neat in a large, multi-team environment, but yes, I can see why it would be the death of a small company.

The worst thing about Clearcase is Rational's attempts to integrate it with the rest of their product range, ClearQuest in particular. The best thing is the ability of a central SCM-guy to keep many teamsworth of source code under some kind of control.

Posted by Alan Green on 2004-08-30

Whoever said that ClearCase has the worst web interface ever hasn't used ClearQuest. We had a vendor who required us to use their ClearQuest server to handle tickets during QA and it was a painful effort. FogBUGZ has some issues, but it's at least quick and easy.

Posted by Bill Brown on 2004-08-30

I've used Clearcase some years ago when I was working for a very large
IT companies. They could afford to have 2 Clearcase admins.

For smaller companies Clearcase is just too complicated and requires too much resources.

It has lot's of estoric features, and you probably have some problem with your process, if you really need them :]


Posted by anonymous on 2004-08-31

I know for certain that here in Singapore, Motorola uses ClearCase in some of their projects.

Posted by Deepak on 2004-08-31

I used it once when I joined a company for a short term contract. The company is still there but I really do not think it will be there next year... if comments are still opened next year, I will confirm this ;)

They had an actual team of 5-6 people devoted to managing that beast. 5 experts that touched ClearCase all day long. As a new developer there, it took me 2 full weeks to manage to get a build working on my machine. Some will argue that the build has nothing to do with ClearCase and you are probably right. But the complexity of things like managing branches caused developers to be lazy and almost never do full updates to newer code. Result: Daily broken builds.

This was (is) no small project: 50 developers working on all tiers of an amazingly complex J2EE application built with WebSphere. Before joining this company, I thought I had seen all nightmares out there. I had not. ClearCase's setup, complexity plus code organization was a huge part of this nightmare.

We used the Windows client. I also remember that for certain operations, we had to login to a unix box and use X to do other specific things (I do not remember what though) not available on the windows client. There were ClearCase admins available 24/7 to support developers that were working around the clock.

Posted by Claude on 2004-09-01

ClearCase is failure, whether the company is big, medium or small. Projects that use it are incapable of recognising their impending death.

Some managers like to buy things that they don't understand - they think they must be impressive. But sometimes they just don't work, and that's why they don't understand them. Developers suffer.

In theory ClearCase can do some things that CVS (or similar) can't. In practice, you don't need to do those things, and you'll never get that far with ClearCase anyway. Get CVS and enjoy actually being able to change a line of code now and then.

Posted by Murray Cumming on 2004-09-02

Interesting.  My company just paid to have me brought up to speed on ClearCase.  I don't think they will be fading away any time soon considering they have already made it past the century mark.  What blows my mind is that this is only one of four versioning systems they use in different parts of the company.

Disclaimer: I am not a fan of ClearCase.

Posted by Peter on 2004-09-04

Interesting comments on Rational service since big blue has taken over: http://www.gripe2ed.com/scoop/story/2004/9/5/14453/96974

Posted by John Beimler on 2004-09-06

Tality Corporation. A spin off from Cadence corporation, doing design services.

Clearcase was purchased. SW development tanked. Tality folds.

Pretty clearcut I'd say.

Posted by Ex Tality Employee on 2004-09-08

Well, I used to work for a multinational that used clear case on large and small projects. It was VERY expensive, but as far as I can tell having since used CVS it is much more powerful. The branching and merging are done properly, and the tagging is also done properly (ie you can see tagged and when...). It also lets you have distributed versioning databases so that you can work in multiple places without having to depend on a central server in overseas somewhere.

The web interface mat not have been the best, but the solaris and windows interface for merging multiple streams of development hammers CVS into a pulp.

Posted by Ryan on 2004-09-08

The branching and merging are done

Branching is done per-file, only when a file has actually changed. As far as I can tell that's for ClearCase-specific performance reasons.

So, instead of one person at one time saying "this project has branched", and then just letting people work on that branch, this means that individual people need to branch each file before they change it,

Firstly, they forget, secondly it's very difficult to make ClearCase do this automatically, thirdly you have to tell people to make ClearCase do this automatically. But most importantly, someone working on a main branch who has no interest in a second branch, will not branch the file because he doesn't care. And then your build breaks. The CVS solution is simpler and usually what you want.

Posted by Murray Cumming on 2004-09-09

ClearCase is really slow.  Explicit checkouts are painful (but the Vim plugin makes it bearable).

The versioning is file-based, like CVS, instead of changeset-based.  I don't know why people would pay so much money for something so fundamentally flawed.

Posted by Kannan Goundan on 2004-09-14

Having switched to a project at wireless telco in Atlanta, I have to agree with everything I've read that you folks have said.

Clearcase and Clearquest are both unnecessarily heavy and unusable. Views, dynamic views, no support for generic linux distros, bastardized command line support, the list goes on and on.

The one thing keeping us on it is the fact that we can't seem to make it so subversion or CVS require a defect number from Clearquest to commit. That alone would give us an out.

There is nothing about the software I like. And I've used VSS, Starteam, CVS, Subversion (My favorite now. Atomic commits, Whoot!), and sccs.

Never work for a team using this software if you have a choice. If you don't supplement it with a subversion repository so you don't have to check in but every once in a while.

Posted by anonymous on 2004-09-16

I am working on a development project, that has 500+ developers in 6 locations worldwide, with 80,000+ files in the system. We use the UCM option in clearcase. Yes clearcase has alot of annoying bits, and the admin overhead can be a big pain. BUT as an admin/developer i see alot of issues are with developer misuse of the system. they beat at it repeatidly, ignoring alot of the main concepts, and then shout when it isnt smooth anymore. And if one more developer comments about how it was much better with CVS, i will break a keyboard over their head. AND finally it can be slow, and the web or WAN options are next to useless

Posted by Liam on 2004-10-16

Odd comments.

I guess we are a smallish company with 200 developers in our office and 800 developers worldwide working on large enterprise development systems.

We use CC and CQ tied together with UCM.
As well as being a developer I am also the admin for CC/CQ on our project.

I have to say that once you get over the initial shock of the appaling interfaces and terrible way the products interface with each other (why do they have different logins neither of which tie into Windows or Unix?), the products do what they claim and rarely need any intervention in my experience.

If you don't use UCM then I can imagine that any company is going to get overwhelmed pretty quickly but if you can live with the restrictions of UCM then I don't see a problem. Merging and versioning is all handled automatically.

I just wish that IBM would rework the products to all be on a common base so that they dovetail together, with the OS, update the interfaces, and improve the performance. (And bring down the crippling price.)

But basically it works for us.

Posted by Rob Scott on 2004-10-19

Let's just clarify, a small company does not have 800 developers. Nor does it have 200 developers.

A small company has less than 10 developers.

Posted by Joe on 2004-10-20

My group has used CC and other systems (primarily for worldwide multisite hardware projects) since the Atria/Rational days.  Our company's worldwide software groups also use CC.

We do have dedicated CC admins, but ours generally do not dictate or enforce configuration mangement (they prefer to let projects choose their own path to success or ruin.)  They just keep the multisite syncs running.

Over the years our company has developed a robust release and configuration management method on top of CC.  Branches are often lifesavers, and we can reliably and automatically generate coherent releases of our designs.  CC is very fast on adequate servers, and ClearMake's DOs and CRs are unmatched.

I am amused by some of the comments above and in the links, which apparently don't understand what is required by CC and what might be imposed by a clueless CC admin.

It doesn't bother me that CC implements its own filesystem and that the raw data is in some database; our RAID NAS works the same way.  I'm never going to see the raw data on a disk somewhere anyway.

Given the resource (staff, hardware, etc) requirements of CC, I heartily agree it isn't for the small developer, just as a (insert-name-of-exotic-car) isn't for the average commuter.  IMHO, if you can afford the infrastructure for it and you take the time to learn to use it properly, I haven't seen anything that can beat it.

Posted by L on 2004-10-26

I have been using Clearquest for about 2 years now. We also have ClearCase. Both are used very widely in our organization. Even though I know ClearQuest and ClearCase have problems but I didnt feel it bad to the extent described in this page.

We are planning to integrate ClearCase with Clearquest and looks like we will have to take our decision very carefully.

Victor Nadar , Mumbai Udyan

Posted by Victor Nadar - Mumbai Udyan on 2004-11-01

We use both CC and CQ in a company of around 100-150 developers. I don't think it's all that bad - granted, the interfaces are shocking and there's a bit of overhead, but I can't see it being the company-killer people are describing...

That said, after four years of CC, we're investigating other options - I gather one team is now running a CVS repository that they periodically sync with CC, and I'd like to see a move to Subversion in future.

Posted by Simon on 2004-11-09

I work for a company that almost exclusively use CC. There have been problems with it ofcourse but nothing major. The features CC provides outweighs the problems (atleast in my book).

There is an initial hurdle before one understands the config-spec syntax. This ofcourse may be the problem with CC because if one can't write correct CS then stuff will not work as expected.

Someone mentioned problems with branching. This is not something I have noticed. If the config-spec is written correctly all files checked out will automagically be branched out to the desired branch.

Posted by anonymous on 2004-11-09

I've used CC at a number of small and medium sized companies, and it seems to me that most of the negative comments here are based on a misunderstanding of how to use the product.  As with most products, CC has a large number of features available to the user, but that doesn't mean you have to use them.

I have a team of 2 C++ programmers who have just migrated from VSS to Base CC.  They use it in exactly the same way (i.e. single stream, no branching) and have no additional overhead with CC.  What they do have is a far more robust system, better merging etc.

I also have a team of 6 Java programmers who migrated from VSS to CC.  At first, I tried to implement the same process in VSS, as I would have done in CC.  I don't think it was even possible!!  Now we use CC UCM and it's a piece of cake.  We have a release stream (branch), a maint stream and a dev stream.  We can easily merge changes between the streams and take full advantages of activities and baselines.

As for admin, there are two of us (of the 8 programmers above) who take care of things.  This basically involves the creation of new dev and maint streams at the end of a release(about 2 hours work max) - not too much of an overhead.

It seems to me that the companies wishing to use CC should think about engaging someone with CC expertise.  This person should be able to set up procedures relative to their project size.

To answer the interface problems, I believe IBM is planing to build CC into the Eclipse framework.  This should improve things in that area.

Posted by Phil Kelly on 2004-11-11

Phil (or anyone), do you mean a dev stream per user or a shared dev stream?  For the teams that share a dev stream, are you using dynamic views (where checked-in items appear instantly in all other developers' workspaces) or snapshot views (where developers "pull" changes when they're ready to integrate)?

If you use per-developer streams (common in UCM), do you find the process of rebasing, delivering, baselining, and recommending baselines to be cumbersome? Is it clear that per-developer streams is implicitly the "edit-merge-commit" style common to CVS and Subversion, but with all the overhead of branches?  (CC streams are really branches.)

We've recently ditched per-developer streams (unless there is the rare need) and use snapshot views on the mainline stream.  ClearCase's support for snapshot views is a little primitive compared to, say, Subversion but is far simpler than using UCM.

Posted by A CC User on 2004-11-12

I have used Clear Case for the past 2 years.  I was the buildmaster/local Clear Case admin for our development teams.  Clear Case is by far (out of any application I have ever used in general) the most difficult and over engineered tool I have ever used.  I have spent countless hours troubleshooting it and dealing with our CC admins.  It is so tightly coupled with the OS that starting up a Windows pc when the network was down caused each developer a half hour startup delay while CC tried to start itself over and over again.  The merge and Diff tools are complete wastes of time.  Imagine dealing with a crap tool every day you came into work.  I would not recommend Rational Head Case to anyone.

Posted by scranthdaddy on 2004-12-01

why are all comments tagged as spam?
oh well forget it

Posted by Another CC user on 2004-12-07

clearcase blows.

Posted by spunkboy on 2005-02-11

I was just browsing the internet for some ideas on software development when I stumbled across this page. I've had a read through and I don't think it represents a fair argument.

I started using CC back in 97 when it was still Atria or Pure Atria (can't remember which!) and CQ from it's beta days in 98/99. I was also in my early 20's and didn't know much about config management so I was just happy to go with the flow. Basically I've just grown up with it, got used to it's quirks along the way and it's the only source control tool I've used professionally. Yes this makes be a bit biased towards ClearCase but I'm not protective of it!

I'll also admit that it took me a long time to fully understand it's methods, and error messages, but at that time the support from Atria was outstanding. Their help desk staff knew their product inside and out. If I had an issue it was usually resolved there and then on the first phone call. If it wasn't then it really was a problem and needed to be investigated. Then Rational turned up and ever since then the support has just gone from bad to worse. Now when I call I have to provide ccdoctor reports, event logs and a million and other things. This usually means a very long email trail and I've usually solved the problem before Rational bother to call me back. Now that IBM have taken over, it's even worse. First of all I have to navigate their website. Ha! At my last company we actually had an expert on the IBM website. If you had a problem you talked to him and he'd navigate their site and send you the URL! Anyway...

ClearCase on it's own with no UCM and CQ integration is a fantastic product. Companies fail with ClearCase implementations for many reasons and license cost is not usually one of them. For example,

1. The projects themselves are planned badly. I have often come in on a Monday morning to find an email saying that development on a new version started last Friday.... How on earth am I supposed to support a new release when I have no input in to it's development strategy, have no ideas about it's lifecycle and can not spend half an hour setting up the new CC environment and perhaps closing off the old one! I could go on but I think most of you will have experienced this whether you are a developer or CC administrator.

2. The CC administrator is usually some poor sole who has no idea about configuration management principles, probably didn't want the job to start with and has a 1 week IBM/Rational course under his/her belt if their lucky. Then the unreasonable requests start coming in from the management teams who think ClearCase is no more complicated than notepad and that this new CC administrator is now a guru! They want projects migrated within the week with no consideration for processes and policies...back to point 1

3. Branching strategies are the next big failure. Don't mean to offend any UNIX guys but ClearCase is like a UNIX system. If it can do it it will do it. It won't complain if it's the wrong way of doing it or suggest a better way. I have seen so many companies using way too many levels of branches that things become unmanageable. A project typically needs 3 or 4 levels branches - main, development, integration and possibly task based branches. Again, back to point 1. Consultation between admins and managers to understand expectiations and plan a strategy.

ClearCase is just a database like any other. Take Oracle for example. If you just slap in a bit of SQL and hope for the best then you might as well use Access. Instead you have to write triggers, procedures, packages and build tools around it. And, as with any database, you have to plan it.

In terms of overhead, ClearCase maintenance should take no more than 3 hours per week. The rest of the time the CC administrator should be writing tools such as build scripts, reporting queries, triggers, automation tools, CM policies/strategies and just generally improving the environment. i.e. things to help developers and keep the boss happy!

I have never had a project fail because of a ClearCase implementation (even with UCM) and I would also expect that those small companies that went under probably would have anyway regardless of the version control tool. Saving the AU$200,000 spent on licensing would probably not have saved the company anyway.

UCM... hmm! First of all this is targeted at managers especially non-technical ones. They like the fancy pictures and models of their project's development. I was disgusted at IBM/Rational's sales pitch (I heard it last week) because, for a non-technical manager, it's like giving candy to a baby. It just gets gobbled up. I personally think UCM should be wiped from this Earth and that anyone who thinks UCM is the solution to their problems or project should be shot! Having said that my boss wants the full UCM rollout..... I know I've just been very harsh but in any event any failure of UCM comes back to points 1, 2 and 3 above. Whether I like UCM or not is irelevant.

The ClearCase Exploder GUI? Let's just say ?bring back ClearCase details?! Why do I need view shortcuts. I'm an administrator and chances are I created most of the 2000 views on the network. I don't need 2000 shortcuts!

I usually work for corporates, oh and by the way, my last job was to support 1500 developers in 8 sites across 6 countries. I was the only CC administrator until I trained someone else up. Total time spent on CC maintenance was probably 1 day a week usually less (assuming no server crashes or catastrophes). Anyway... most of my other jobs had between 10 and 20 developers which is small for a corporate. None of those projects failed because of ClearCase. Most failed because of changing times or crap management. I now work for a small company of 20 staff of which 9 are developers. They are forking out for both CC and CQ licenses at AU$13000 per CC+CQ license. If this company fails because they implemented ClearCase it won't be ClearCase's fault; it will be mine. It will be my fault for not implementing the solution correctly, for not educating the developers properly and overall for being a crap CC & CM administrator.

Problems with CC on Windows? Chances are it's a Microsoft related issue affecting ClearCase. Buy a UNIX box! Windows explorer has just locked up because my server has just crashed. Most of my other applications are now locked up too as a result. If I kill explorer, the other apps spring back to life! CCWeb, ah yes, it runs on a Windows platform and is designed for IE. Apache web server or not I rest my case.

Finally, ClearCase is not always the right tool for the job. Brings me back to point 1. Good project management means evaluating the available software and making the right choice. Whether thats ClearCase, Perforce, Subversion, Bitkeeper or even SourceSafe. It also means employing someone with the right skills, enthusiasm and passion for CM and not just re-training someone to fill an open position. ClearCase is just a tool to assist the CM process. If the CM process is flawed then the tool will fail. Managers and developers will hate it and projects/companies can fail. Back to point 1.

Well I hope I didn't bore you too much and I hope this goes through in one piece because I spent a long time writing it!

Posted by BB on 2005-02-24

How do I integrate Clear Quest in Visual Source Safe.

Posted by Roland Ehi on 2005-03-09

To briefly echo the sentiments of spunkboy, a fool with a tool is still a fool. 

You need to know what you want to do with the tool.  If you can do all of those things with another tool, more power to you.

I have made a reputation out of intergrating tools thaat aren't supposedly integrated.  Base ClearCase (without UCM) is great for that because it has a lot of hooks.  Granted, you have to know how to take advantage of it. 

On a single OS platform ClearCase doesn't take a dedicated admin, except as you work to set it up with automating your processes and putting in enforcement triggers.  I speak from 10 years experience on this.

When working with cross platform (PC to Unix), I have found it is a bit of a mess.  But the mess is strictly based on  crappy networking support on the PC side, not anything to do with ClearCase.

Posted by vobguy on 2005-04-27

Ouch, poor old ClearCase.
I'm trying to set ClearCase up at the moment, and to be fair, it's a bit of pain in the arse. My main question is why the hell do you need ClearCase for 10 or less developers?!?!
Seems like you're using a sledgehammer to crack an egg.
I think VSS would probably do, if anything. You just need a decent, workable CM process.

Posted by Dave LR on 2005-05-27

Well I can see why people with little clearcase skills would not like the tool. However I can asure you in the correct hands a team using clearcase can manage change far better then a team using CVS or worse yet, no cm tool at all.

ClearCase isnt bad, clearcase in unskilled hands is.

Posted by chad on 2005-06-30

I've used CC on four projects now and they've all suffered as a result.

I do believe that CC is fundamentally powerful (in the right hands) BUT the conceptual complexity it brings is overwhelming and it outweighs it's benefits, IMO. Most developers do not have the time nor the inclination to spend the weeks learning how to use the SCM system.

As developers we have enough complexity to worry about. An SCM tool should make working with code easier, not harder.

To me, CC is a fetid putrid piece of stinking dog poop. I hate it deeply.

Posted by Tony on 2005-07-08

Hi Guys,
I was just reading all the comments made from you guys about CC. Well I've been using CC and CQ since a while already. We've started with an old IBM product CMVC. I know that many had their issues with this one as well, but you would not believe how many comapnies out there still use it. I must say that I've seen many other products (VSS, PVCS,CVS,MKS and more recently Seapines software suite), but in some of the concepts CMVC was ahead of its time.
I've originally started in a company origined in germany. We had been aquired (and later sold ) several times. When we started migrating to CC/CQ we used many concepts from CMVC (for good or for the bad).
Well overall I would not say that CC is that bad. It has many many thinks other tools don't have. But at the same time, I believe that with many tools you can make it usable for the needs it is bought for. You just need to know how.

We've succesfully used CC/CQ in a multinational environment utilizing multisite. We've managed to create (with extra effort) a distributed development environment for development around the globe and around the clock.

Posted by Raymond Masocol on 2005-08-05


Judging from the feedback above, ClearCase is an administrator's dream. All the control, all the features, but at the cost of being heavyweight.  To a developer, it represents another complex tool to be overcome in their workday. Hence the schism evident between the developer's viewpoint and the administrator's viewpoint.

This line of discussion raises the question: who is more important to the software development process? The developer or the adminstration staff?



Posted by Will Waterson on 2005-09-25

We've used Rational at our small (< 10 person) company for a couple of years. I wouldn't say it's been an unmitigated disaster, but it sure as hell is not worth the money. I think a lot of companies make the decision to go to Rational because Rational claims to have a process (RUP) and they convince you that you can't do RUP without their tools. They also try to convince you that every single person needs a full enterprise suite, which is crap. Companies could save a lot of money by figuring out what a good process for them actually is and THEN deciding what tools fit the bill. The most commonly used tools in the suite(CC and CQ) don't do any better IMHO than freely available ones like SubVersion and Bugzilla respectively, and the other ones like Purify/Quantify can be nice but you can probably share one or two licenses across the entire group.

Posted by Ted on 2005-09-27

Just came across this link today as company stated using ClearCase (suckered into it by a Rational Sales team IMHO as  we have succesfully been using CVS for past two year with no problems)!

"ClearCase isnt bad, clearcase in unskilled hands is."
If the tool was any good it would be developer proof! CVS up and running in less than an hour (I know a little slow) CC a week and still it's causing the development team nightmares - We will probably thorw out and go back to CVS and just let managment think using it!!!

.... Don't even get me started on the "RUP'ish" - A great process if throw out the "R" bit of it - UP as put forward in XP world great!

Posted by John on 2005-10-03

I fully agree with many of the comments above

1. UCM is bad, very bad. It is all pretty pictures and management-speak. Anybody with a few minutes to spare can use the trigger feature available in base ClearCase to implement a much better, site-specific version of UCM without loosing many of the better features of base ClearCase. Do not use UCM!

2. Base ClearCase config specs are incredibly powerful and knowing how to at least read them is important for a developer.

3. If I gave a responsible, apparently intelligent, adult a chainsaw and they cut their leg off would it be my fault? I love the commment that a fool with a tool is still a fool. That is so true. OK, ClearCase may need a bit of thought to set up (it certainly isn't plug 'n' play) but it is worth it for the right size team.

4. ClearCase is overkill for small projects

5. CC/CQ integration is easy and in my experience has never failed

6. One company I worked at who used ClearCase very nearly went under. It was not the fault of ClearCase. ClearCase wasn't (allegedly) playing with the books, writing crap contracts and deceiving the stock market.

So in summary, don't use ClearCase if you are small because it's not worth it, don't use UCM no matter what your size, and ClearCase can't bring your project/company down without significant help and assistance.

Posted by Jim on 2005-10-10

I fully agree with many of the comments above

1. UCM is bad, very bad. It is all pretty pictures and management-speak. Anybody with a few minutes to spare can use the trigger feature available in base ClearCase to implement a much better, site-specific version of UCM without loosing many of the better features of base ClearCase. Do not use UCM!

2. Base ClearCase config specs are incredibly powerful and knowing how to at least read them is important for a developer.

3. If I gave a responsible, apparently intelligent, adult a chainsaw and they cut their leg off would it be my fault? I love the commment that a fool with a tool is still a fool. That is so true. OK, ClearCase may need a bit of thought to set up (it certainly isn't plug 'n' play) but it is worth it for the right size team.

4. ClearCase is overkill for small projects

5. CC/CQ integration is easy and in my experience has never failed

6. One company I worked at who used ClearCase very nearly went under. It was not the fault of ClearCase. ClearCase wasn't (allegedly) playing with the books, writing crap contracts and deceiving the stock market.

So in summary, don't use ClearCase if you are small because it's not worth it, don't use UCM no matter what your size, and ClearCase can't bring your project/company down without significant help and assistance.

Posted by Jim on 2005-10-10


Joe GregorioChrysler 300 | BitWorking


Chrysler 300

Chrysler 300

The first time I saw the new Chrylser 300 I hated it.

The next time I saw it I was intrigued.

The third time I saw it the Dick-Tracy-Muscle-Car-Terra-Plane shape had firmly wrapped it's tentacles around my lizard brain.

I don't own one.


I use a car service for some of my business trips (travel to the airport).  The service I like best uses the 300 and it is a spectacular car for this purpose. 

From the perspective of a rider in the back seat, the car if very comfortably and spacious.  It also has enough cup holders and arm rest space to be quite functional as an en-route office.

I haven't driven the car, but the drivers have all commented that they like the car very much, and since they're driving for 6+ hours a day I think their opinion is quite valuable.

Posted by Lou on 2005-06-21


Joe GregorioChristmas 2003 | BitWorking


Christmas 2003

There were cookies.

Picture of decorated sugar cookies


Joe GregorioChina - Day 9 and 10 Chongqing to the White Swan Hotel in Guangzhou | BitWorking


China - Day 9 and 10 Chongqing to the White Swan Hotel in Guangzhou

Day 9 was pretty uneventful, Lynne, Caden and I spent some time in the morning shopping with Mark, Moya and Andie. The afternoon was filled with paperwork and the rest of the day was spent packing for our travelling to Guangzhou the next day.

Part of our shopping was to buy a new large suitcase. We didn't really need it for this leg of the trip but we plan on doing a lot of shopping in Guangzhou, enough that we know we'll need a whole extra suitcase for everything we are going to purchase.

The paperwork took over two hours to complete, not that it bothered me much as Lynne took care of it while Caden and I had together time. One of the things we did together was get more pictures. Here is a shot of the building I was talking about the other day, where a shop has been setup on the ground floor and the building isn't even complete.

Store and noodleshop setup in a building still under construction

I also got some pictures of what our group referred to as "the big dig" next to the Marriott. It is a large excavation for a basement, one of the three new highrises being constructed in the front of the Marriott. The culture here is definitely not as safety conscious as it is in the US. The pictures I took here are from the driveway of the Marriott. The only thing that separated me from a 50 foot fall to the bottom of the pit was a thin landscaped area. No fence. No barriers. Nothing.

Large excavation for the basement of a highrise.

The paperwork that needed to be done is much longer and more complex than anything we've had to fill out so far. That's because it is all for the American government and not for the Chinese government. Over the next 4 days we will complete all the steps we need to adopt Caden in the US.

Day 10

After packing we turned in early, got up early and were on the bus to the Chongqing airport by 9AM. A one and a half hour flight landed us in Guangzhou and a 20 minute bus ride got us to the White Swan Hotel. The White Swan is a 5 star hotel that is a favorite of adoption agencies to put up their clients because it is so nice and it is conveniently located to the services we need. The American consulate is literally a 5 minute walk down the street. Now we were fortunately warned from others that had gone before us that if you stayed at the Chongqing Marriott then not to expect too much when you go to the White Swan. They were right. I am quite honestly not that impressed. They messed up our room assignment, first giving us a room on the 11th floor then retracting it, saying the room wasn't ready, then after waiting they gave us a room on the 15th floor, but it too wasn't cleaned so they gave us the information, but refused to give us the keys until the room was done. Wendy, one of our agency's representatives out of New York was kind enough to let us use her room on the same floor until ours was cleaned. They assured us that they would get us from Wendy's room and give us the keys as soon as it was done. Well, they didn't and 20 minutes later we checked on things to find that the room was done, but our keys were not available, and they'd forgotten to set up a crib in the room. They eventually got the crib set up and the keys to us, but all-in-all it was a pretty shabby introduction to the hotel. Oh, and the rooms are smaller, the beds are smaller, the bathroom is absolutely cramped with no counter or drawer space. In general the staff have been professional, but they verge on snooty and are nowhere near as nice as the staff at the Chongqing Marriott.

The entire feel of the area is very different from Chongqing. The air is much cleaner and the area surrounding the White Swan is almost alien, as it is all European styled buildings, something to do with Britian and France and the Opium Wars. We'll go into that more tomorrow.

We did a little exploring, mostly to get some bottled water and Coca-Cola, and to find the local laundry.


Water has been constantly on our minds during this trip because none of the tap water is potable. As a matter of fact it has been various shades of brown and tan over the last few days. Given that we don't in anyway want to ingest the water we go through a lot of bottled water. It also means that we have been washing all of Caden's bottles and bowls with bottled water too. You never realize how much water you use washing dishes until you're pouring that water out of a bottle and not from a tap. The water restriction means that at every hotel we have been hanging a towel over the tap to remind ourselves not to use it, even for things like rinsing your mouth after brushing your teeth, or for rinsing your tooth brush off afterwards. Bottled water for everything. Luckly we can pick it up at a reasonable price, a two liter bottle runs only about 70 cents.


Special thanks to Ralf for converting the flower street videos into divx, quicktime and mpeg4!


Joe GregorioChina - Day 8 - Another walk around Chongqing | BitWorking


China - Day 8 - Another walk around Chongqing

Another day and another walk around Chongqing. This time Caden and Lynne accompanied me and Lynne took the pictures.

Porter on a crowded street

Whenever we go out with Caden she gets put in the sling, which does several things. The first thing it does is save my back. The second thing the sling does is cover most of Caden, with a hat only her eyes are visible outside of the sling if she's awake, and she is completely hidden if she falls asleep. The reason you want to keep her hidden is to avoid the "clothing police". The "clothing police", as we call them, are well meaning older chinese women that will run up to you and cover any exposed part of a child and admonish you to keep them covered. The first time we went out Caden had the lower part of her legs exposed and three times on the walk we had ladies approach and pull her pant legs down further over her socks. The "clothing police" are only bested by the "thumb police". Caden sucks her thumb which is apparently a big no-no in China and complete strangers have no problem walking up to her and yanking her thumb from her mouth while admonishing us. Now you could try to explain that we just adopted her and that this is a particularly stressful time for her and we'll deal with the thumb sucking at a later date, but when you only speak english and they only speak Mandarin, well, you just keep the thumb hidden.

Caden in the sling.

Being so deep in China anybody who isn't non-Asian is a rare sight and Lynne and I are constantly stared at just for our looks alone, but the attention is much greater when were out with Caden. Lots of pleasant stares and smiles as we walk along, but if we stop, a crowd will gather. This has an interesting dynamic as many people try out their English on us and if one person 'makes contact', that is they say something we understand, they instantly get promoted to 'local translator' and everyone around them starts peppering them with questions to ask us. News that we are Americans and that we just adopted Caden is always greeted with huge smiles and great appreciation. Here I am getting slightly mobbed on the pedestrian mall.

A crowd gathers around me and Caden

These two girls were at first shy, but after asking about Caden and getting a couple pictures of themselves taken they started to ham it up.

Two children.

One of the strangest sights in both Beijing and Chongqing has been the sight of decidedly non-Asian mannequins. And it doesn't just end with the mannequins, a good portion of the advertising is populated with decidedly European looking models.


Did I mention that there's a lot of people here.

A crowd

One of the things I won't be able to get across, no matter how many pictures or videos I take, is the amazing contrasts that you can find in Chongqing. The brand new building in the front is built right up to, and includes a bit of a facade for, the aging apartment building behind it. Everywhere you see these contrasts and amazing changes. There is a high-rise across the street from the hotel that is under construction. In the uncompleted ground floor of the building people have already setup a noodle shop. From our hotel window I can count 17 cranes on 15 high-rises currently under construction.

Two buildings

Even in the midst of the all the construction and modernity there are reminders that this is an ancient land and culture. Less than half a block from our hotel is this drug-store which is six feet wide it is a single aisle twenty feet long.

Drug store

I hope you're all enjoying the photos. Once I get back to the US I need to learn some more about photography in general and digital photograpy in particular. For example, here is my current 'system' for handling digital photos.

  1. Load photo into Paint Shop Pro.
  2. Crop and resize.
  3. Press "One step photo fix..."
  4. If it doesn't look good then choose another photo.

Joe - it's been really interesting reading about your trip. I can totally relate to some of your "Lost in Translation" moments, as my wife is Chinese (I am not) and we occasionally visit Taiwan.

Congratulations on the adoption!

Posted by Craig Andera on 2004-01-08


Joe GregorioChina - Day 7 - A short walk around Chongqing | BitWorking


China - Day 7 - A short walk around Chongqing

This afternoon I went for a short walk walk in the area immediately around the Marriott. I looped through the 'pedestrian mall' which is a shipping area where the streets have been closed to cars, and the local flower market.

A crowded sidewalk

This Zhonghua Road, a busy shop lined road on the way to the pedestrian mall. There are a mass of shops along these roads, some no more than 5 feet wide, selling everything from fine suits, to noodles, to shoes. Some of the clothing and shoe shops have their production setup right on the sidewalk, ancient looking sewing machines running all day long. It might look like a trick of the light but the leaves really are that dull grayish-green color. The coal fired smoke and fog combine to produce a heavy smog that rains down a fine soot over everything in the city.

There aren't many private vehicles on the roads, mostly buses and taxi cabs. This is actually a pretty rare shot with the street completely empty. The careful art of crossing the street as a pedestrian, well, I'll leave the description of that for anther day.

An alley way.

The alley way above is a nice study in contrasts. The smaller buildings nestled back in the alley are much shorter and older than the ones near the front of the building. In the back you can see two high-rises in the mist. Both buildings are incomplete, the one on the left you can see the ever present crane on top.

Busy street with a porter

The man on the right hand side of the above picture with the long bamboo pole in his hands is a member of what the locals call the "Pole Army". Chongqing is a very mountainous region and bikes are not much use here, instead there are droves of men with poles that work as porters that carry items around the city. I've seen them carrying some pretty heavy loads. In the center of the picture is one of the clothing shops with the sewing machines set out on the sidewalk.

This brings out a recurrent theme I've seen across the whole trip into China. Whatever problem we would solve with technology in the US, the Chinese solve with people, and where we would use people to do a job in China they would use more people. Where we would use trucks, they have porters. Where we would use backhoes, they use laborers. Another example, in every grocery store and department store we have been in from Beijing to Chongqing, they are staffed with one person per aisle. Yes, you read that correctly, one person per aisle. We have yet to eat in a sit-down restaurant where the patrons outnumbered the staff.

McDonalds sign on a busy street

What good is a city without a McDonalds? How about 60 of them in Chongqing alone. There are two within walking distance of our hotel. This picture is taken on the pedestrian mall, with the McDonalds and a noodle shop on the left. The large mass of people in the middle of the picture are sitting on the benches eating noodles. Huge steaming mounds of delicious smelling noodles served in thin plastic bowls and eaten with the same disposable wooden chopsticks you get at Chinese restaurants in the US.

entrance to the flower market

Here is the flower market across the street from the hotel. I bought Lynne a half-dozen roses here on my way back. It was my first time haggling and I didn't do a very good job, but I got her the half-dozen (actually 7 roses) for 10 Yuan or $1.25US. One of the nicest things about being in China is that I am getting used to the prices and going back to the US is going to be difficult, and we haven't even gotten to do a lot of shopping outside of department stores where there isn't any haggling.

open air flower market

The market is more like an alley, with vendors selling everything from individual flowers, to huge arrangements, to bonsai trees, to gardening supplies. The alleys winds through the block and turns right and as turns it trasitions from flowers to food staples and cooked food. The above picture is a shot looking back down the alley towards the entrance. Each of the rooves on the left is a different shop selling different tools, produce or plants. For those of you with good connections I shot a short video (440KB) of the pedestrian traffic flowing into the entrance of the flower market . This is a pretty good representation of the pedestrian traffic and noise levels that are continuous throughout the city. It is a WMV file that I've tested in both Windows Media Player and Real Player. I can provide the original AVI file to anyone that wants to convert it into different formats, for which I would be eternally grateful.

Special thanks to Ralf for converting the flower street video into divx, quicktime and mp4!

Flower vendor wrapping up a sale.

This final shot is about half-way up the alley, a vendor is wrapping up a sale to the people standing in front of him. Behind him wends another alley, orthogonal to the current one that contains more plants and trees for sale. There aren't many flowers on the branches he's sold but they have a very strong scent. Speaking of scent, that's one of the things about China that I can't blog. There are wonderful scents like the flower market above, and the great smells of food wafting out of some of the stalls. On the other hand there are some pretty awful stenches, some that are nauseatingly familiar, others that of unknown origin and are completely beyond any previous experience.

Thank you so much for your tour.  I have a friend who lives in Chongqing and it is my desire to visit her soon.  You have permitted a glimpse, I thank you for that.


Posted by Stephen Cotta on 2004-04-18


Joe GregorioChina - Day 6 | BitWorking


China - Day 6

Today was a very low key day.

I wanted to keep a low profile as I am still recovering from the stomach flu, and we also want time with Caden, so Lynne and I skipped out on the scheduled site-seeing and stuck around the hotel. In the morning we walked around the nearby pedestrian mall with Mark and Moya doing some window shopping and stopping by the grocery store on the way back for necessities. I'm still not coherent enough, I forgot to bring along the camera, a sure sign that staying back was a good idea. The rest of the day was spent playing with and taking care of Caden, eating and exploring the hotel (you laugh, but this place is 39 stories tall, has 8 restaurants and about that many shops).

Caden playing with a rattle. Caden playing with a teething ring.

We took a lot of pictures of Caden, but because she is always laughing and playing so hard most of the shots end up looking like this one.

Blurry Caden playing.
I have to keep saying it -- she's beautiful, and looks thoroughly happy!  I couldn't be more happy for you.  This has definitely been a long gestation :)

Posted by Eric Vitiello on 2004-01-07


Joe GregorioChina - Day 5 - Gotcha Day | BitWorking


China - Day 5 - Gotcha Day

This is the day, Gotcha Day, the day we get Caden.

Lynne with Caden

She came right to us, with no crying at all.

Joe and Caden

It was very cute, she sat on Lynne's lap and just stared at me for a good 5 minutes. Then she reached out for me. We sent pictures out to the orphanage earlier and I think the Aunties did a good job of showing her the pictures because she seemed to recognize us.

Officially Adopted in China

After getting the babies we went to complete the official adoption paperwork. We have now officially adopted her in China. Now we wait for some of the paperwork to get back to us and then on to Guangzhou to adopt her in the US.

What you don't see in the pictures is that I am deathly ill. I caught the stomach bug. For the last picture just seconds before it I was sleeping on a chair in the lobby, shivering with the chills. I went back there immediately after the picture was taken. After we returned to the hotel I slept for 15 hours. I'm feeling better now and can hopefully more fully enjoy this time with our daughter.

Congratulations!!  She's beautiful!  I'm glad it's working so well, and wish you a safe trip home.


Posted by Eric Vitiello on 2004-01-05

Congratulations!!  She's beautiful!  I'm glad it's working so well, and wish you a safe trip home.


Posted by Eric Vitiello on 2004-01-05

Congratulations Joe & Lynne!  What a wonderful thing for you.  Best wishes to you all.

Posted by Jason Clark on 2004-01-05

Congradulations!  Just wonderful.

Posted by Don Park on 2004-01-08


Joe GregorioDetecting Benchmark Regression

Subtitle if this were an academic paper, which it’s not: A k-means clustering derived point statistic highly correlated with regressions in commit-series data with applications to automatic anomaly detection in large sets of benchmark results.

TL;DR: To detect regressions in benchmark data over a series of commits use k-means clustering on mean and variance normed commit-series. For each of the clusters find the best fitting step function to each cluster’s centroid. The metric |step/fit| is highly correlated with regressions, where step is the height of the step function, and fit is the mean square error of the fit of the step function to the centroid.

Below is the description of how we detect performance regression for the Skia graphics library. I’m writing this up because after much searching I haven’t found anyone describe the method we came up with for detecting performance regressions and maybe this writeup will be useful to other people.

Problem Statement

Skia is an open source cross platform 2D graphics library. In Skia, like many other software projects, we have large number of performance tests, aka benchmarks, and we run those benchmarks every time we change the code. Just having a large number of benchmarks isn’t a problem, but being cross platform means running those tests across many different platforms; Linux, Mac, Windows, Android, ChromeOS, on different GPUs, etc. which leads to a combinatorial explosion in benchmark results. For every commit to Skia today the testing infrastructure generates approximately 40,000 benchmark measurements. That number of results tends to change frequently as tests, platforms, and configurations are added and removed regularly. The number of results has been over 70,000 per commit in the past several months.


To make the following discussion easier let’s define some terms.

A Trace is single benchmark and configuration tracked over a series of commits. Note that this isn’t exactly a time series since the measurements aren’t taken at equidistant times, but are spaced by commits to the codebase. Also note that for each benchmark there may be multiple traces, for example, one for Windows 8, one for Linux, and one for Android.

Fig 1 - Trace

A “performance regression” is a significant change in either direction of a metric. Now a metric that drops may actually be a good performance increase, but could also be an indication of a test that is broken, or has stopped working entirely. So regardless of the benchmark, we are looking for step-like changes in either direction.

The issue with tens of thousands of traces is that you just can’t look at the raw numbers, or even plot all the data, and figure out when you’ve had a regression. At first we tried setting performance ranges for each trace, i.e. an upper and lower bound for each trace. If a later commit caused the benchmark results to move outside those bounds then that would trigger an error. There are many drawbacks to monitoring benchmarks by manually placing bounds on each trace:

  1. The most important drawback is that in such a system a single test result can trigger an alert. You know old phrase, “the plural of anecdote isn't data”, a single benchmark measurement is virtually meaningless as any number of anomalies could actually be responsible for that benchmark result changing. For example, a machine could overheat forcing a move to frequency scaling, or other processes on the machine may starve the test of CPU cycles. You can work to mitigate these eventualities, but they never completely go away. SPC systems such as the Western Electric rules might be applicable in this case, but we’ve never tested them.
  2. Constant manual editing of trace bounds is time consuming and error prone.
  3. Constantly adding manual trace bounds for new benchmarks is time consuming. Add one test and you may have to add many trace bounds, one for each member of that combinatorial explosion.
  4. Forgetting to add new ranges for new benchmarks another source of error.

Even if you automate the placing of trace bounds, you still have the issue of transient behavior that looks like a regression, and you also have to take pains that the automatic calculation of trace bounds doesn’t mask a true regression.

Fig 2- Is this a regression or an anomaly?

So we needed a better system than setting trace bounds. The next section explains the system we implemented and have successfully run for several months now.

Before we go further let’s define a few more terms.

Normalized Traces
Normalization is the process of modifying each Trace so that it has a mean of zero and a standard deviation of 1.0. Note that if the standard deviation of a trace is too small, then blowing that up to a standard deviation of 1.0 would introduce nothing but noise, so there’s a lower limit for the standard deviation of a trace, and below that we don’t normalize the standard deviation of the trace. The idea is to extract just the shape of the trace, so that all the normalized traces are comparable using a sum of squares distance metric. The smaller the sum of squares error is between two normalized trace, the more similar their shapes.
k-means clustering
I’m not going to explain k-means clustering in one paragraph, you should go look it up on Wikipedia or any of the fine explanations available on the web. The important point is that we are using k-means clustering to group normalized traces together based on their shape. The idea is that many traces will move together in the same direction from commit to commit. For example, if I speed up rectangle drawing then all tests that use rectangles should get faster, if not in the same proportion.
The centroid is the center point at the center of a cluster. In this case the mean of the normalized traces in a cluster, which acts as a prototype shape for the members of the cluster.
Regression Factor

For each cluster of normalized traces we find the best fitting step function to the centroid. From that best fitting step function we calculate Fit and Step, where Fit is the sum of squares error between the step function and the centroid, and Step is the height of the step function.

From there we calculate the Regression Factor:

R = Step / Fit

A smaller Fit values gives you a larger R, which means that the more a centroid looks like step function the larger R gets. Similarly the larger Step gets the larger R gets, which is a measure of how big of a change the centroid represents.

Putting it all together.

So finally, with all the preliminaries set up, we can get to the core of the system.

  • Collect all Traces over a specific range of commits. We usually use around the last 100-250 commits worth of data.
  • Normalize all the Traces.
  • Perform k-means clustering on all the Normalized Traces.
  • For each cluster calculate the Regression Factor of the centroid.
  • Any cluster with a Regression Factor whose absolute value is over 150 is considered interesting enough to need triaging. Note that 150 was chosen after observing the system in action for a while, the cutoff may be different for your data.

Here’s a view of the system at work, finding regressions in performance. Note that out of 40,000 benchmarks the first cluster contains 1336 traces and has a Regression Factor of -4.08e+3.

Screenshot 2014-11-23 at 1.14.19 PM.png

Continuous Analysis

The above system works for finding regressions once. But what happens if you want to check for regressions continuously as new commits land and new benchmark results arrive? One last definition:

A cluster is considered interesting if it’s Regression Factor is over 150. This is only a rule of thumb based on observing the system and may be relevant only to the Skia benchmarks, while a different cutoff may be appropriate for other datasets. The important point in that as |R| grows so does the likelihood of that cluster being a regression.

To continuously monitor for Interesting clusters, start by running the above analysis once and find interesting clusters. If there are any then triage them as either really needing attention, such as a CL needs to be rolled back, or ignorable, say in the case where a true performance increase was seen.  Then on a periodic basis run the analysis again when new data arrives. What should be done with the new set of interesting clusters produced from the analysis? The existing interesting clusters have already been triaged, and those same clusters may appear in the output of the analysis, and new interesting clusters may appear. The process of only raising up new interesting clusters for triaging while folding existing clusters with similar clusters that appear in the analysis results is called cluster coalescing.

Cluster coalescing currently works by looking at all the new interesting clusters and if they have the same traces as the 20 best traces in an existing cluster then they are considered the same cluster. Note that ‘best’ means the 20 traces that are closest to the centroid of a cluster. Note that this is an area of active work and we are still experimenting regularly with new cluster coalescing schemes.

Wrap Up

I hope that was useful. Please shoot me any questions on Twitter @bitworking. The code for the software that does the above analysis, and much more, is open sourced here.

Tim HopperSundry Links for November 24, 2014

brushfire: Avi Bryant has been building a 'Brushfire is a framework for distributed supervised learning of decision tree ensemble models in Scala.' Fun stuff!

What are the lesser known but useful data structures?: I always enjoy StackOverflow questions like this, but it is not considered a good, on-topic question for this site, of course.

Free Programming Books: A huge, crowd-sourced list of free programming books by language and topic.

PhD Dissertations-Machine Learning Department: Seven years of ML PhD dissertations from Carnegie Mellon University. I wish I had time to read Tools for Large Graph Mining.

Tim HopperSundry Links for November 17, 2014

There's no magic: virtualenv edition: I didn't really get virtualenvs until long after I started programming Python, though they're now an essential part of my toolkit. This is a great post explaining how they work.

Traps for the Unwary in Python’s Import System: "Python’s import system is powerful, but also quite complicated."

pyfmt: I recently learned about gofmt for auto-formatting Go code. Here's a similar tool for Python.

Q: Setting User-Agent Field?: A 1996 question in comp.lang.java on how to set the user agent field for a Java crawler. The signature on the question? Thanks, Larry Page

alecthomas/importmagic: Python tool and Sublime extension for automatically adding imports.

Tim HopperSundry Links for November 12, 2014

Amazon Picking Challenge: Kiva Systems (where I interned in 2011) is setting up a robotics challenging for picking items off warehouse shelves.

contexttimer 0.3.1: A handy Python context manger and decorator for timing things.

How-to: Translate from MapReduce to Apache Spark: This is a helpful bit from Cloudera on moving algorithms from Mapreduce to Spark.

combinatorics 1.4.3: Here's a Python module adding some combinatorial functions to the language.

Special methods and interface-based type system: Guido van Rossum explains (in 2006) why Python uses len(x) instead of x.len().

Tim HopperSundry Links for November 3, 2014

Public Data Sets : Amazon Web Services: Amazon hosts a number of publicly datasets on AWS (including the common crawl corpus and the "Marvel Universe Social Graph").

Rapid Web Prototyping with Lightweight Tools: I've shared this before, but my boss Andrew did a fantastic tutorial last year on Flask, Jinja2, MongoDB, and Twitter Bootstrap. Combined with Heroku, it's surprisingly easy to get a website running these days.

rest_toolkit: REST has been my obsession of late. Here's a little Python package for quickly writing RESTful APIs.

The Joys of the Craft: A quote from Fred Brooks' The Mythical Man-Month on why programming is fun.

How do I use pushd and popd commands?: I recently learned bash has push and popd commands for temporarily changing directories. This is very handy for scripting.

Tim HopperSundry Links for November 1, 2014

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!): I guess the title says it all. By Joel Spolsky.

Unix Shells - Hyperpolyglot: Very cool comparison of basic command syntax in Bash, Fish, Ksh, Tcsh, and Zsh.

Better Bash history: I'm pretty stuck on Bash at the moment. Here's a way to get a better history in Bash. (Other shells often improve on Bash's history.)

usaddress 0.1: I always love seeing a Python library for something I've tried to do poorly on my own: "usaddress is a python library for parsing unstructured address strings into address components, using advanced NLP methods."

more-itertools: A great extension to the helpful itertools module in Python. Some particularly helpful functions: chunked, first, peekaboo, and take. Unfortunately, it doesn't have Python 3 support at the moment.

Tim HopperPyspark's AggregateByKey Method

I can't find a (py)Spark aggregateByKey example anywhere, so I made a quick one.