Friday, August 28, 2009

Divide and conquer your project, part 1

James, who is a regular commenter, is a programmer looking for more experience. James has studied a few languages and probably has a decent grasp on the basic logic that it takes to write a program, but he wants to work on a personal project because he doesn't get enough variety of practical applications at work.

This is an admirable goal, first because developing your programming skills requires constant practice, and second because personal projects can make good resume padding. Potential employers like to hear that you program for fun, because it will make you more likely to throw yourself into your work. Also, if they ask "Have you worked with technology X?" you can say "Not professionally, but let me show you this project that I posted on my web page." It gives you the opportunity to talk intelligently about the technology.

However, starting a new project can be overwhelming, and it's hard to know where to start. You have all these things in your head that you would like the program to do, but you can't see how to get from what you have now (a blank screen) to what you would like to have (an awesome program with flawless logic and a clean interface). What do you do first?

I spent a few posts discussing the technique of "divide and conquer." As it turns out, that is also an excellent plan for approaching your programming projects. Here's a little secret of programming: you won't get it right the first time. Fuhgeddaboutit. Don't try. If you sit down for a couple of hours and try to write an enormous, complex program in one shot, and only after you think it is done do you compile it for the first time, you will be confronted by a screen full of errors and another few hours of headache, tracing back through what you did to figure out where each of numerous bugs exists in your thousand line program.

Actually, my proudest moment as an undergrad was writing a single assembly language module that executed perfectly with no debugging on my first try. But that's not something you should count on, and in any case, this was one line in a much larger program.

When you are building a castle of air, you do it brick by brick. Except they're, you know, wind bricks. You don't wave your hands and get a fully formed castle. You break down the project in your mind into easily manageable pieces, and at each step you (1) Look at that step to determine whether you should break it down even further before proceeding, (2) Write complete code that does something visible, and (3) test that component thoroughly.

That third step is crucial, and it's a hallmark of a solid programmer that they develop testable and tested code at every step. When you test a piece of your code until you know it's solid, you will save yourself a lot of time later that would have been spent backtracking to figure out where your code is going wrong.

How do you break it down? Let's consider the project that James set for himself.

One idea I've had is a program that accepts inputs on your credit card balances, minimum payments, etc, and calculates the most efficient payment plan to snowball. I've got a spreadsheet I downloaded from somewhere that I'd model, but the spreadsheet can't jigger your payments to determine the best way to pay off the balances the soonest, or with the least amount of interest. I envision it as an Iphone App, although I learned that without a Mac I can't write it that way.

Actually a spreadsheet probably can do that, but writing a full featured program would be cooler. So let's consider what components this program has.

A popular way of dividing up pieces of a program is to start by splitting it into three layers: a presentation layer (the way your program handles input and output); a data layer (the information that your program will be manipulating); and a business layer (the program logic that figures out what to do). If you split your program up into pieces this way, then you can hopefully change up the pieces at any stage without risking ruining your entire program.

How you start your problem will largely depending on which component is most interesting to you personally. In many projects, the graphics are the most interesting thing. In that case, your top priority would be to get something, anything to display on the screen.

For example: My first real programming job was modeling equations for scientists. There was an eccentric old physicist who wanted me to graph some of the equations he wrote for me, using Fortran on a green and black terminal. As I progressed on getting the equations to display the right answer, he seemed distinctly bored with hearing about the programming details. But then I figured out how to generate a line diagram. It excited the hell out of him, and suddenly he couldn't stop talking about his ideas for what we could do with this.

The diagram, of course, bore almost no relation to the actual equations he had described. But it didn't matter. It was a launching point for his concept of the program. Similarly, as you write your own programs, hopefully you will keep on continually inspiring yourself by producing code that does something that will tickle your interest in the next phase of your project.

Now, in James' case, the interesting part of the project is probably not the graphics. It's the business logic. After all, the ultimate objective is not to create something that looks cool, but something that provides useful information. Since this is a program that you will be using yourself, the inspiration will come from your feeling that "Wow! This is really telling me some valuable stuff! But I bet it would be even more useful if..." And then whatever comes after "if" will determine which piece of the program you conquer next.

So the business layer is the most important part of your program. Therefore, I would recommend that you pick a language, and start with an extremely simple presentation layer and data layer. So for presentation, let's say you'll use plain text input and output. For data, let's say that you'll write a module or an object that hard codes the value of your credit card balances, and then you can tweak the code to change the values.

These aren't good solutions, but they are stepping stones for the program you really want to build. Write good business logic first, using crude presentation and data models, and then make it better later. How will you make it better? Well, you might decide to turn your text input into a web form interface, and perhaps add graphs. You might change your data from hard-coded values, to a separate text file that you will edit by hand, to a database table. As long as you keep these parts of the program reasonably well separated, you can build them up a bit at a time, focusing on the parts that strike your fancy.

Whew, that was a mouthful. Pausing for feedback now. James, do you think this is enough to get you started? Do you have an idea which language you'll start in? Any thoughts or questions?


  1. Maybe I'm misunderstanding you, but it sounds as though you're advocating writing a good, solid business layer before really starting to work on the data and presentation layers.

    Personally, I'm a fan of what Fred Brooks called growing software: start with bare-bones data structure, bare-bones business logic, and plain text output to stdout. The fact that it does something, anything, as opposed to code that does nothing right now, but will be beautiful two weeks from now, is a strong motivator to continue.

    I also tend to learn a lot from personal projects, which means that most of the time I don't know what I'm doing. In these situations it's important to remain flexible. If you write a thousand lines of business code that assume one set of data structures, only to find out later that you need to scrap those data structures and start from scratch, can be a major annoyance.

    Yes, you're right about "you won't get it right the first time". That's why it's valuable to start small: you won't have to throw away as much code.

    Oh, and you forgot another major layer, probably because like everyone else, you hate doing it: documentation.

  2. Personally, I'm a fan of what Fred Brooks called growing software: start with bare-bones data structure, bare-bones business logic, and plain text output to stdout. The fact that it does something, anything, as opposed to code that does nothing right now, but will be beautiful two weeks from now, is a strong motivator to continue.

    That's exactly what I meant to do first. What I was trying to convey was that once you have a basic program that prints "something," the business layer is the thing to be working on first. James' idea is focused on computation and delivery of new information, so that's the part that will motivate him to improve the program.

    Regarding the data structures: That's one reason why it's important to separate interface from implementation. You want your program to be able to rely on functions that consistently tell you something, so that if the way the data is actually structured changes, the business layer shouldn't have to change along with it.

  3. Wow. All this attention on little 'ol me. Well, the gauntlet has been thrown down, so now I'll take the challenge. Since JavaScript is the language I've been working with most recently, theoretically it's the freshest, so I'll start with that.

    Breaking it into bite-size pieces makes sense, and one of the first lessons in my JavaScript book is "accepting input from the user," so I'll start with on-screen prompts for my Debt Snowballer program to accept credit card amounts, interest rates, and minimum payments and store them in variables.

    Thanks for the nudge, and I'll let you know when I've got something working.

  4. When I was writing the post, I considered recommending that you do it in JavaScript, but I did not. The main reason is that eventually you will want to come up with a good method for saving and loading data, and JS was deliberately written with the intention that you will NOT mess with files on the client's computer.

    There are ways around this, such as running the file read component in an ActiveX control, but as I noted on the last post, this greatly reduces the number of people who will run your program. ActiveX needs to run on IE or a simulation of IE, and you have to convince people to click a confirmation that you have permission to mess with their computer. People are understandably paranoid about this.

    A better way to handle persistent data is to store it on your server. Then you can either access it through CGI when you load the page, or pull it up dynamically with AJAX. Either way, you'll need to learn a little about server programming. That's the route I recommend.

    However, until you're ready to cross that bridge, there's nothing especially wrong with making a JavaScript program to handle the logic with hard coded data.

  5. Forgot to add, another way JavaScript can save and load information is through a cookie. It's not ideal, but you could, for example, save the information about your credit cards in one long string when the user adds or removes cards. Then when the page loads, you could look for the cookie and parse it, or if it's not there, just use a set of default sample data.