Write Other Programs

04 March 2013

I made a pretty stupid mistake while working on a personal project last Friday. I was working on Canary, an (unreleased) project that alerts users when macroeconomic trends suggest a recession will hit the US in the next 6-18 months. Canary has two parts: a tiny web application for storing email addresses and displaying charts, and an offline component that pulls data from economic data APIs, crunches some numbers, and produces JSON for the web app to consume. This second component is what I was working on yesterday.

When I sat down, I had most of the actual number crunching logic already done. However, the code that fetched the data from the external API, fed it to the model and serialized the result was a mess, so I sat down with the goal of rewriting those components from scratch.

Without really thinking about it, I did the same thing I would do on a green-field application at work - I wrote a controller-level test asserting I could get JSON out of the program and proceeded to iterate, in TDD style, until it passed. It all went according to plan, until I remembered that:

• I didn’t have a web framework in front of the controller.
• This part of Canary was going to be run as as cron job, not a web application.

Without thinking about it, I applied the MVC style of program architecture even though I wasn’t writing a web app. Yea. Whoops?

After thinking about it, though, it’s not that surprising that I made this mistake. Yes, I should have thought about my program’s architecture before I started, but I had never written a production-quality program that wasn’t a web application before. In other words, I really had no idea how else I was supposed to lay out my application, despite reading plenty of books and articles on e.g. modular design, object orientation, and so on. Because I didn’t think about the architecture, I defaulted to what I already knew.

This is what’s referred to as an abstraction problem in psychology. The idea is that it’s extremely hard to extract abstract patterns from a single example; you’ve hit an abstraction problem when you fail to extract the right abstraction from your experiences.

People understand this problem in some contexts. When I studied math in school, I didn’t solve a single problem and assume I understood the theorem behind it. Instead, I solved many problems, because I knew that was the only way to force my brain to identify the underlying pattern from the specific examples. You probably did the same thing. The strange thing is that people usually do not think about why they behave this way, even though it’s really important.

Thinking carefully about patterns is a fundamental skill for software developers. For the most part, we’re pretty good at identifying repeatable patterns, extracting them, and remembering them for later. But we aren’t perfect. We take the wrong lessons away from projects and fail to recognize opportunities to apply old patterns. We also often fail to realize when we can’t apply an old pattern, which is basically what I did on Friday.

Once I realized what I had done, I thought more carefully about the architecture of the Canary cron job and ended up making some modifications to my approach. It wasn’t a lot of work, and I was better off having gone through the exercise. That’s the good thing about the abstraction problem: all you need to solve it is experience and some time to think.

As developers, we have to keep pushing beyond the designs and patterns we’re familiar with. If you’re a web developer, write programs that aren’t like what you write at work. Write programs that aren’t web applications.

comments powered by Disqus