Coding for fun: Roman Numeral Converter with unit tests

I was listening to an audiobook this morning about the evolution of communication technology. It's The Information by James Gleick. Good book. I should probably write a longer review later. For now though, I happened to hear a section where Claude Shannon amused himself by writing an early program that translates Roman numerals. I figured if it's good enough for Shannon, it's good enough for me, although I have better tools that make it easier. So anyway, just for fun:

Conversion class:

package com.russellglasser.romannumeral;

import java.util.HashMap;
import java.util.Map;

class RomanNumeralConverter {

   private final String[] romanDigits;
   private final Long[] arabicDigits;
   private final Map toArabicTable;
   private final Map toRomanTable;

   RomanNumeralConverter() {
      romanDigits = new String[]{"M", "CM",
            "D", "CD", "C", "XC",
            "L", "XL", "X", "IX",
            "V", "IV", "I"};
      arabicDigits = new Long[]{1000L, 900L,
            500L, 400L, 100L, 90L,
            50L, 40L, 10L, 9L,
            5L, 4L, 1L};
      toArabicTable = new HashMap<>();
      toRomanTable = new HashMap<>();
      for (int i = 0; i < romanDigits.length; i++) {
         toArabicTable.put(romanDigits[i], arabicDigits[i]);
         toRomanTable.put(arabicDigits[i], romanDigits[i]);
      }
   }

   String toRoman(Long arabic) {
      StringBuilder builder = new StringBuilder();
      Long remaining = arabic;
      for (Long digit : arabicDigits) {
         while (remaining >= digit) {
            builder.append(toRomanTable.get(digit));
            remaining -= digit;
         }
      }
      return builder.toString();
   }

   Long toArabic(String roman) {
      String remaining = roman;
      Long result = 0L;
      for (String digit : romanDigits) {
         while (remaining.startsWith(digit)) {
            result += toArabicTable.get(digit);
            remaining = remaining.replaceFirst(digit, "");
         }
      }
      return result;
   }
}

Unit test class:

package com.russellglasser.romannumeral;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;

class RomanNumeralConverterTest {

   private final RomanNumeralConverter converter = new RomanNumeralConverter();

   @Test   void toRoman1() {
      check(1L, "I");
   }

   @Test   void toRoman3() {
      check(3L, "III");
   }

   @Test   void toRoman4() {
      check(4L, "IV");
   }

   @Test   void toRoman8() {
      check(8L, "VIII");
   }

   @Test   void toRoman14() {
      check(14L, "XIV");
   }

   @Test   void toRoman18() {
      check(18L, "XVIII");
   }

   @Test   void toRoman1979() {
      check(1979L, "MCMLXXIX");
   }

   @Test   void toRoman2018() {
      check(2018L, "MMXVIII");
   }

   @Test   void toRoman9999() {
      check(9999L, "MMMMMMMMMCMXCIX");
   }

   private void check(Long arabic, String roman) {
      assertEquals(roman, converter.toRoman(arabic));
      assertEquals(arabic, converter.toArabic(roman));
   }
}

Tech talk about Pokemon Go

Quick background for people who don't already know: Pokemon Go is a mobile game that launched last month on Android, and more recently on iPhones. It is billed as an "augmented reality game," in that the gameplay incorporates a requirement to walk around in the real world and visit physical locations, and some virtual objects are treated within the game as if they physically exist at those locations. It was immediately popular, and has been plagued with ongoing problems with server response and performance. In this post I will discuss these issues on a level that laymen should be able to understand.

Techie Rant: Mobile apps shouldn't override web browsers

Dear Facebook:

I would like to politely offer feedback on this "instant article" feature that you are so proud to have updated on my mobile device.

I don't want to have to read long articles in the Facebook mobile app.

I don't EVER want to have to read long articles in the Facebook mobile app.

Thanks for the help, cloud AI!

This is just a quick silly story that illustrates how the "intelligent" programs that monitor our lives can be both funny and creepy.

At my work, we have an intranet application -- something that runs in a web browser, but it's only visible from inside the office -- which tracks change requests and bugs we are working on. I am on the Client Account team, so most of my projects are identified with "CA-" and then a number, such as "CA-1234."

Usually after I have been working on one project for a while, I have memorized the number, and so I just type "CA-1234" into my browser. Since I have visited the feature page often, it automatically fills in the URL to take me there. But sometimes I mistype, or it doesn't work for some other reason, so my browser helpfully pulls up a Google search for CA-1234 instead.

That does not contain any information that is useful to me. However, it does often match a flight number on China Airlines.

So one morning I got an alert on my Android. It was Google's calendar program. It told me "It's time to leave for the airport if you want to catch that flight on China Airlines that you looked up!"

Advice for aspiring software engineers

I went for my Master's Degree after living in Austin for several years and working at a low level programming job at a large company. I felt really stuck. The nature of my work hadn't changed significantly in many years, and my team wasn't even trying to keep up with new technology. I was bored and didn't see things changing.

When I completed my degree in 2008, I was now overqualified for my job, but I also lacked the experience to get the kinds of jobs I was aiming for. The economy was also a disaster. I bounced around a number of contract jobs for several years, not managing to get a reliable permanent job until 2011. Three years later, I got a call from a recruiter at Blizzard Entertainment, a company I've been a big fan of since I played Warcraft II in college. I went through two lengthy interviews and joined the web development team. I'm pretty happy with the track my career has taken.

Last December, my cousin David wrote to me on behalf of a friend in college who wanted to know how he should proceed in order to make himself attractive to employers and land a decent job. I sent him the advice below. Due to the nature of my online media activities, a lot of young people write to me with similar questions, so I thought I'd recreate the advice for everyone to read.

Explain like I'm five: DDoS attack

A friend of mine asked me to literally "explain like I'm five" how a DDoS (Distributed Denial of Service) attack works, because World of Warcraft was having troubles yesterday after the launch of Warlords Of Draenor, and he was trying to tell his kid why. My answer:

When you play Warcraft, the computer that represents you is always "talking" through the internet to the computer that has everything in the WoW world. Computers have a limited attention span; they can talk to a lot of people at the same time, but not everybody in the world.

Usually they have enough attention to talk to all the people who are trying to play WoW at the same time. But sometimes bad people don't want the game to work right, so they pretend to be a million people and all try to talk to the computer at once. The world computer doesn't know who's a real person and who's not, so it gets confused and talks to all the million fake people. Then it doesn't have time to talk to you, so the game goes slow because it can't send you information about what's happening fast enough.

Sometimes people do this because they're mad at the company, and sometimes just because they think it's funny. It's against the law, but it's hard for the police to understand, so usually the company that is getting attacked has to deal with it on their own.

Adventures in junior programming, episode 4

As I've mentioned in previous posts (1, 2, 3), I've been teaching my son Ben a little Python here and there over the last few years. We're not very consistent, and in fact we haven't really done it much at all in a year or so. But that's not too terrible; it sort of reflects my own experience at Ben's age. I tinkered a little bit with BASIC once in a great while, copied some programs out of books and magazines, "hacked" some programs, forgot many things, and didn't really get deeply into program structure at all until I reached high school. So this is not an urgent task for me.

Ben and I did some work over the weekend. I have taught him a few things over the last couple of years, but we do it so sporadically that it's hard to keep the information fresh in his mind. So we started with this exercise: opened several of the programs that he has worked on in the past, then read through each one and predicted what it was going to do, and why. Doing this helped him get back into his headspace from earlier sessions, and also drilled the syntax back into his mind pretty quickly. Speaking for myself, I also find that reading my own code can help to refamiliarize myself with a language much more quickly than reading somebody else's examples.

Up until now it was clear that we'd done a lot of programs that were really brief -- single problem, single solution. Print all the powers of two up to a million. Print out a times table. Create a simple password verification program (which insults you if you fail; that's a ten year old mindset for you). These problems are interesting in their own right, but the real fun of programming is making a project with no immediate end goal, something you can tinker with over a long period.

Ben is very into gaming, just like his dad. We've been playing World of Warcraft together since he was seven. This seemed like a promising starting point, so I had him create a simple system of rules which could represent a fight between a hero and a monster. So we started out with a few basic hard coded numbers: Hero hit points, monster hit points, hero damage value, monster damage value. Then we wrote a loop so that the fight would keep going until one of the hit point values dropped below zero.

The program is pretty predictable at this point; the beauty of the concept is that we can build on the basic concept over time. So the first thing Ben wanted to do was tweak the numbers: The hero should always win, but just barely. So we did some math to figure out how to make that happen, and we saw the outcome change as the starting values got closer to numbers that "felt right."

This was a good time to introduce functions, which I don't think I've explained in detail before. We had a function for printing the current status ("The hero has X hit points! The monster has Y hit points!"); a function called "hithero" and another called "hitmonster". Initially the hit functions removed a fixed amount of damage; then we got a little more sophisticated, passed in the damage as a parameter, and also printed the action to the screen. After that we also added in weapon descriptions, so it then said "The hero hit the monster for 20 damage with his sword of epicness!"

It's a fun exercise and it has Ben's enthusiasm up, so he now thinks of it as "The first program I wrote mostly by myself!" (I did give him a whole lot of guidance, actually, but I've let him steer the creative direction of the program, so he does have plenty to be proud about.)

We now have a bug and feature wish list which is surely going to grow:

• Make it so hit points can't go negative.
• Make a "death" message.
• Have multiple kinds of weapons; enable weapon switching.
• Randomize damage within a range, like 16-24.
• Allow the hero to pick a type of action, a la Pokemon, such as a direct damage attack or a debuff attack.
• At some point we should probably start discussing object-oriented programming, before we get to the point where we bring in multiple monsters.
• Item drops?
• Visiting locations?
• Ben really wants to make graphics, but that is a very advanced topic, and I'd rather he get solid with the fundamentals before touching that one.
• I'm a web programmer, and putting stuff in web pages is a great way to showcase your work to friends. So maybe that's a better direction to go than graphics.

Obviously the best part of programming, as per the title of this blog, is building castles in the air -- constructing projects which seems to take on their own reality. So I'm looking forward to some more sessions where we make this thing interesting, while keeping it accessible to a kid who will be twelve soon.

Affordable Care Act Blues 2

Following up on my post from yesterday, a friend linked me to an article on Salon that attempted to diagnose the problems with the healthcare.gov website. The article is full of bad logic and half-baked assumptions that I couldn't resist correcting.

First of all, the article refers to a discussion on Reddit that was just ridiculous.

“The Redditors picking apart the client code have found some genuine issues with it, but healthcare.gov’s biggest problems are most likely not in the front-end code of the site’s Web pages, but in the back-end, server-side code that handles—or doesn’t handle—the registration process, which no one can see. Consequently, I would be skeptical of any outside claim to have identified the problem with the site.”

This is WAY too charitable. I’m not “skeptical.” I think the Redditors are dumb as rocks for wasting any time on this at all.

Server side (back end) code handles pretty much everything that is important about a web application's internal logic. The client side (front end) stuff -- which you can see printed out if you right-click on any page in your browser and select "View Source" -- handles cosmetic stuff. How the website looks, what kind of warning messages you get if you enter bad data, etc. It's possible for a site to break because of bad front end code, but that were true then it would almost certain be broken for everybody, not up and down sporadically.

I passed along the article to a friend who is the architect of our software at my job. His explanation is more thorough than mine, so I'm reprinting it with his permission.

Here’s how I would have written the same article but with my speculation….

I think the biggest and most important challenge is the overall architecture. From an architectural standpoint you need to ensure that your system is

1. Secure
2. Can scale appropriately
3. Handles the scenario where it is overloaded

Secondly, for the end user experience, from a UI perspective you need to be

1. User Friendly
2. Ensure proper feedback
3. Have as much client side validation as possible

Once you go live you are in a situation where you have to deal with all types of scenarios quickly. One issue that I see referenced a lot is the fact that the security question dropdowns aren’t populating. Maybe they load tested this but didn’t verify the contents of the actual html rendered. If you rely on older back ends then that needs to be part of your load testing and you need to reduce talking to those systems the least amount possible.

As far as server load there are 3 types of congestion you need to deal with. Memory, CPU and information I/O (Network and Drive). The dropdown thing might be related to either one of those three.

This, of course, is informed speculation about the nature of the problem itself, drawing on real experience about how websites are designed, and how to go about debugging the problem. The Salon author, David Auerbach, doesn't do this sort of thing. Instead, he casts about wildly to find a place to assign blame. He claims that he can identify it as an Oracle problem based on a single error message:

“Error from: https%3A//www.healthcare.gov/oberr.cgi%3Fstatus%253D500%2520errmsg%253DErrEngineDown%23signUpStepOne.” 

To translate, that’s an Oracle database complaining that it can’t do a signup because its “engine” server is down.

What? It is? How did he know that? I looked up “ErrEngineDown” to see if it might be a standard Oracle message. It is not. So my reading is that it’s simply the name that the developers themselves chose to assign to this particular error. There is literally nothing you can determine from this one status result, as far as identifying what kind of database they used, or why the database failed.

After that, Auerbach goes on to state that

That is, the front-end static website and the back-end servers (and possibly some dynamic components of the Web pages) were developed by two different contractors. Coordination between them appears to have been nonexistent, or else front-end architect Development Seed never would have given this interview to the Atlantic a few months back, in which they embrace open-source and envision a new world of government agencies sharing code with one another.

It's true, apparently there are at least two different developers who've had their hands on the system: Development Seed and CGI Federal. This is not a legitimate criticism of the process. The federal government is big. The ACA is big. It is routine and normal to have multiple companies working with one set of data. After all, each individual state seemingly has their own website which has to connect to the ACA. In that case, you typically have a web service host on the back end, which processes results, and the results come from many different clients -- i.e., the state's web site, which was probably developed at least partially by someone in the state.

And that's all we know. There is no evidence I can find in any of those links, that "coordination between them appears to have been nonexistent." Auerbach simply made that up. He also makes up a cute little fictional dialogue that has no grounding in reality whatsoever. That conclusion also does not follow from the fact that they use open source code and are enthusiastic about promoting open source principles, unless I’m missing some other piece of information cited. It may be the case, but it’s not demonstrated at all. Open source is just a model of developing something with transparency. It doesn't say anything at all about the process which the two developers followed.

To reiterate: We're talking about a website that is not working, for some people, some of the time. A friend posted just today that he successfully created his account and is done with the process. This is not a site that is fundamentally broken or flawed; it is a server that is sometimes overloaded by a very high volume of traffic. That's it. That's the whole problem.

Affordable Care Act rollout blues

I keep hearing journalists speculate about what the explanation might be for the Affordable Care Act website being so unreliable in its first week. While it sounds like a good talking point, the conversations have generally been pretty dumb.

You want an explanation? It's a massive online system that got hit with nine million users during launch week. That is more than World of Warcraft has subscribers right now.

As an IT engineer and an online gamer, I would be shocked NOT to see an online system on that scale experiencing technical problems. I played WoW since launch day. It had sporadic connections for the first week or two. The same thing happened when the first couple of expansions came out. Most other MMO's and numerous other games experienced similar glitches early in their life cycles also. Who remembers the PR nightmare of the latest rollout in the SimCity series? People who pre-ordered the game, mostly couldn't play at all for a week.

Non-functioning websites during launch week are the rule, not the exception. The reason is, an individual server can handle a certain number of simultaneous connections. For a large project, you use multiple servers running in parallel, so that they can spread out the load. There are also multiple redirections to other computers on the network; for instance, the database is housed on another server, and there might also be multiple parallel databases running in some cases. In the case of a big government system accessing social security numbers, it's likely to interface with some really old legacy systems somewhere along the line.

So in the first place, the throughput of the whole system can only be as reliable as its weakest connection. In the second place, systems are designed to handle an expected average load each day. One reason WoW was so slow at first, was because an especially high number of people wanted to be the first to play. After some time, usage settles down a bit.

You can't always predict how high the early load will be on the system. And even if you do guess correctly, it's not necessarily a good idea to buy double the number of computers that you'll need on an average day; that's a lot of expensive computing power that you'll just wind up having to sell off fairly quickly.

So, don't be surprised that there are problems. Be surprised that it's working at all most of the time.

Shotgun recruiting sucks

Dear recruiter at hire-knights.com, who has spammed me three times in the last two days:

Subject line: "HIGH PRIORITY !!! C++ Developer - Atlanta,GA - 6 months Contract"

Let me explain a few things to you. I am employed full time, with benefits. My primary expertise is not C++. I live in Austin. And the amount that you are offering is pitifully low -- considering that you want me to quit my job, uproot my life here, and be unemployed in six months.

Look, I sympathize, it's hard work being a recruiter, and maybe I'd be tempted to take shortcuts too. But in all honesty, the damage that you are doing to your reputation and your company by sending email about this "HIGH PRIORITY" position to every resume you can find on the internet, probably outweighs the likelihood that you will be successful filling the position this way.

You see, instead of helping me to take you seriously as a person worth networking with, all you've done is convince me that hire-knights.com is an organization that traffics in spam, that is bad at finding people jobs they want, and that is wasting the time and money of companies that might consider letting them fill their positions. Is that really what you want?

So now you're in my filter list. If I receive another message from hire-knights.com which is not a direct acknowledgment of this problem, all emails from that domain will go straight into a filter so that the emails will never be seen by me again. And either way, it's going to be forever a matter of record in all the search engines, that

hire-knights.com is a terrible company that is bad at hiring people

So now you've succeeded in attracting my attention. I hope you're happy.

A few more thoughts about Bitcoin and numeric values

Sorry to keep harping on this one topic, but I was reading the "Myths" page on the Bitcoin wiki and something struck me as kind of funny and strange.

Bitcoin has been compared to the gold standard, because the coins "exist" (so to speak) in limited supply. Indeed, believers in Bitcoin and believers in precious metal currencies seem to be the same people in a lot of cases.

But here's the interesting part: some of the arguments for Bitcoin openly undermine and negate the arguments for using gold and silver as currency. Check this out.

Further discussion about Bitcoin

My previous post about Bitcoin invited discussion, but much of that discussion took place on Facebook and Google+. A lot of good insights and links were offered, but brief messages on social media are hard to search and learn from in the future. So I'm writing a second post to acknowledge these responses, clear up some of the misconceptions I had in the first, and offer useful links for anyone who wants more information in the future.

Open discussion on Bitcoin

I've been hearing a lot of stories lately about Bitcoin, and while I don't fully get it, I'm currently on the side of people who think it's ultimately going to wind up being a very technically innovative Ponzi scheme. I'm not making this post to argue about that. Realizing there are a lot of cheerleaders for Bitcoin out there, this post will be heavily moderated. Sales pitches for Bitcoin will not be approved, nor will posts calling me names, although pro-Bitcoin posters are welcome as long as they can contribute to explaining the either the technological details or the details of economic distribution. A lot of the purpose of writing this post is to foster some technical discussion and assemble my thoughts about how it works.

. 

Bitcoin is an "alternative currency," which is something political Libertarians are always saying we need. It is meant to be decentralized, not controlled by any one government or individual. At the present time, one Bitcoin is worth around $100, down from a high last week of around $200. I am personally not particularly interested in getting involved with Bitcoin, either by trading my dollars for bitcoins and spending the bitcoins, or by speculating on buying low/selling high, or by mining for them. However, with my computer science degrees I'm mildly interested in working out exactly how the system works. I found a FAQ page, and I found the original technical paper by Satoshi Nakamoto, but I still don't have a full handle on it yet. Eventually I'm probably going to have to download the open source code and look at it, but I'm still trying to decide if that's worth my while.

Thoughts on piracy

According to Forbes, Game of Thrones will likely be the most pirated show of all time. There are many reasons for this, one being HBO's business model, as summed up nicely in this comic by The Oatmeal.

I myself am watching Game of Thrones legally, yet not paying for it. My sister records the episodes and I watch them at her place. In effect, I'm piggybacking on her account and we're getting two views for the price of one subscription, since I don't have cable at all.

Personally, I've had mixed feelings about piracy for a long time, and I'm still not sure what my position will eventually evolve into. I've got friends in two camps on this. My artistic friends (game designers, people involved with film and music, and those who can draw well) generally think that piracy is one of the greatest sins of the modern world. Meanwhile, my techie friends seem to have not even the slightest trace of guilt about it.

A great case in point: I watched Breaking Bad for the first time last year, on the advice of a coworker. First he said "You should check it out." I said I'd see if it was on Netflix, it was, so I got hooked. I feel good about watching things on Netflix, because in effect I've already paid for it, and the money that I pay indirectly gets back to the studio via whatever contract they've negotiated.

When I'd finished the third season, I realized that the fourth and final season (so far) was not available yet. So I told my coworker, "Well, I'm gonna wait a while until Netflix uploads the next season." I got funny looks. He said "Why don't you just torrent it?" -- as if there is no reason in the world not to do that, and I must be some kind of Amish hippy or something to not have thought of it.

I'm not saying I took the moral high ground here. I held out for a few more days before I talked myself into torrenting it. But at least I'm aware that there is a moral issue. I discussed it a few times with said coworkers over lunch, and they at least acknowledge that it might be a problem for the studios. I have a strong suspicion that most people under the age of 20 would not even go as far as recognizing that it's illegal.

To be clear: Most of the movies I watch are either in theaters, purchased or rented DVDs, or legally endorsed streaming sources. Most of my music is from CDs I own or MP3s I purchased online. Most games I've played and enjoyed are either free or paid for. Most, but definitely not all.

Let me play "angel's advocate" and try to fairly represent the side of my art friends.

• Being an artist, of one sort or another, is hard work, usually for low pay.
• Making money as an artist depends on some sort of reliable revenue.
• Without a business model that produces reliable revenue, big budget art will not be viable. Movies like The Avengers have to show in theaters and sell DVDs, or there's no economic incentive (hence no ability) to make them. Games like Diablo III and Skyrim need to pay their designers, actors, modelers, and developers, and that means they can't afford to give it away. Musicians need to make enough money to live on. And so on.
• When you consume art for free that you have been asked to pay for -- watch a movie, play a game, put music in your collection -- you are stealing it. The artist deserves to make money for producing things that you enjoy, and you are taking advantage of them by not paying for it. (This is one of the points I am a little ambivalent about. Just bear in mind that I'm trying to accurately represent the artist side of the equation in making these points.)
• The more art people steal, the more difficult it becomes to make money as an artist. It's a tragedy of the commons situation. Eventually we may get to the point where the quality of art declines dramatically, because the really talented people will not be able to produce art full time, nor will the budgets be there for big projects.
• Therefore, by pirating, you're hurting everyone in the long run.

I don't really want to post the pirate's justification for pirating in much detail. I've heard them presented in many conversations; I've even tried using a few myself. But even to me, they ring a little bit hollow. They strike me as the rationalizations of someone who knows they're doing something wrong but wants to keep doing it.

For the sake of putting them out there, here are some briefer hits on the pro-piracy arguments:

• It's not really stealing if you copy something without destroying the original.
• Information should be free anyway.
• I wouldn't pay for it even if I couldn't pirate it, I'm too poor or it's not that good.
• I tried to give HBO my money but they made it too hard. (See the cartoon above.)

Many of the arguments come up in this Reddit conversation about "Thrones," which the Forbes article also links.

I don't want to pretend that these are really strong arguments from an ethical point of view, but I do want to point out a few things about managing incentives properly.

There is a saying among economists, that you put a lock on your bike to keep honest people from stealing it. In other words, leaving your bicycle unlocked is just too tempting, and some people who wouldn't normally steal a bicycle may succumb if it's just sitting there unlocked. Meanwhile, a really determined criminal will still steal your bike with or without the lock. It's just that if you have the lock, the probability that your bike will be stolen on any given day goes way down.

In other other words, we all have a certain moral threshold, some lower than others. I'm pretty sure I wouldn't steal a bike, with or without a lock; and yet I stole season 4 of Breaking Bad. Where does the moral calculus lie?

It seems to me that people decide on a course of action based on a variety of factors, of which the primary motivators are

1. How great the benefit is for doing something unethical (If there's no benefit, the choice would be easy) versus how great is the fear of being punished for your actions (taking into account both the likelihood of being caught and severity of punishment).
2. How difficult the action is to perform. (In the bike lock example, moral objections plus the difficulty of breaking a lock will be enough to deter some people from stealing a bike, whereas without the difficulty factor, they succumb.)
3. What magnitude of harm they think their actions might cause. (Robin Hood is a prime example. In this case, the principle that "Stealing is wrong" butts up against the observation that it will do more good than harm. Most people would place "Stealing $1,000 from a billionaire" as a lesser evil than "Stealing $1,000 from a person who needs that money to eat.")

There may be more factors there, but let's just take these three to start with. On these axes, media piracy falls in an area for most people that makes it really easy to rationalize.

How great is the benefit of piracy? Well, not really that great. You can watch a movie that you would otherwise miss. It might not even be a very good movie, or else you'd be more likely to pay for it. There is benefit, though, as The Oatmeal points out. Sometimes you're saving the cost of a ticket or DVD, and sometimes you're seeing something now that won't even be available for a year or more.

But what about punishment? Despite a few well publicized cases in the last decade that turned out to be a PR disaster for the RIAA, generally people know that the chances of being caught and making charges stick for something that millions of people do, is minuscule at best.

How difficult is it? The first time you try it, it takes a little research. On subsequent tries, it's trivially easy, requiring only some nearly free bandwidth, and a few hours of slightly slower internet access.

What magnitude of harm? This is the major point of dispute. Even if we completely grant that it's wrong to pirate, and even if we accept the fact that artists need that money, the individual harm that I cause by pirating a movie is still very small. Depending on when they say it, the MPAA claims that piracy costs their industry $250 billion, $58 billion, or $6 billion per year. A piece in Ars Technica suggests that it's not nearly as bad as any of those.

Certainly, if someone pirates a movie that they would have otherwise have paid $20 for, then the studio loses $20. But part of the effect of piracy is that people wind up seeing a lot more movies than they would actually buy, and most people see their "theft" of any individual movie as being worth a buck or two at most... and one TV episode being worth far less. Again: I'm not saying any of this to argue that it's actually okay to do this, just pointing out how the calculation shakes out for people who pirate regularly. When a pirate steals an episode of a show, they probably think of that individual action as costing a few cents.

In HBO's case, on one hand I think they're being a little bit foolish by making customers jump through so many hoops to get a copy of the episodes. Yes, if they force somebody to subscribe to their full service then they wind up with a lot of money. But if somebody doesn't subscribed even though they would have been willing to buy some episodes on Netflix or Hulu for, say, $20, then that's $20 that HBO simply doesn't get which they could have. I'm not the CEO of HBO, of course, and they might have calculated the difference. But my feeling is that if their business model is to force customers to stay subscribed to traditional cable service forever, I don't think that model will last very far into the future. I know I'm not the only one who's just abandoned cable entirely in favor of paid online entertainment, and I imagine this will become more common going forward.

On the other hand... the fact that HBO doesn't charge a reasonable price for their services doesn't automatically entitle people to steal them. As with all of capitalistic offers, your options within the law are to either accept the services, find a legitimate bargain, or just don't use them.

But despite all that rationalizing, we still have those issues in the background that make piracy an easy thing to do. The act of pirating is ridiculously easy, and will only get easier. There are steps that many game companies are taking to prohibit piracy, such as the growing trend to do what Blizzard has just done with Diablo III, which is to require that even mostly single player games connect to a server at all times. However, in the case of music, media and text... I'm fairly well convinced that it will never get more difficult to pirate them from a technical perspective.

The reason is simple. No matter how many safeguards they put on DVDs and ebooks, eventually you have to let paying customers see and hear it. That means that you have to allow every customer to decode it and play it back visually and audibly, and that means that you can capture the output to a file as well as the screen. Look at it this way: in the worst case scenario for pirates, even if the software was completely flawless, it wouldn't be able to prevent external recording devices from just taking a video of the video.

When you look at it that way, legal wrangling like SOPA and PIPA are all that media companies really have to turn back this tide, and they're not good tools. They'll never make media harder to copy, and they won't convince people that the pirated video is costing them more than a couple of bucks per "theft". So the cost/benefit calculation of pirating is their only weapon -- trying to impose draconian punishments on people who get caught, so that they won't do it.

Yet SOPA and PIPA had all kinds of problems because they overreached, causing companies which do not encourage piracy to protest that this would hurt their business model. In effect, the cost to the society for implementing those measures was worse than the cost to the media companies. In my example above, I said you could copy movies with a video camera. Suppose the MPAA decided to push Congress for a law that made owning a video camera a federal violation subject to a hefty fine. That might solve some of their problems, but it wouldn't be a solution that citizens would stand for, for many legitimate reasons.

So as I've felt for years, I don't know what the solution to piracy is. Since it is a tragedy of the commons problem, everyone who pirates contributes a small amount to the problem, but the overall consequences are large. And I don't want HBO to go away. From what I've read about them, it seems to me like it would be extremely difficult for any other company to undertake such large scale projects with such a generous lack of censorship. As popular as it is to talk about "crowdsourcing" these days, I think people underestimate the magnitude of shared resources that has to go into a huge entertainment project.

There is, of course, a lot of fat that can be trimmed out of the publishing industry in general. Greta Christina recently proved that you don't have to go through a giant publishing corporation to make money on book sales, and Joss Whedon showed that you can make money on a silly one-off independent film project. But that's probably not a fair standard, since he's Joss Whedon. Already an established Hollywood presence (more so now that he's directed one of the biggest box office hits of all time) and with fairly big name actors willing to work with him pro bono.

It could be that some art forms will simply be unavailable, or will decline sharply in quality, because it's no longer feasible to produce expensive things and make money with them. That'll be sad. But with all that said... I may be a hypocrite, but torrenting is still fun.

Troubleshooting my own user idiocy

Random dumb tech story: For months my computer microphone has had a lot of loud static on it. Every time I say something in a game like Left 4 Dead 2, people complain and tell me not to talk again.

I finally got around to investigating the issue, which involved unplugging things, looking at the sound controls, etc.  I was bewildered to find that the computer was recording noise even when the mic was unplugged.  Then I realize: I have a USB webcam that I rarely use.  It's plugged into the back, it has its own built in mic... and it's hanging behind my desk RIGHT NEXT TO THE FAN.

Everything's crystal clear now.

Operation HackMaster Crit Tables, episode 2

Now that I've explained what data structures are for, I can finally explain how I approached the problem of deciphering those ponderous HackMaster tables.

First of all, I discovered to my dismay that the tables were only available in the form of PDF files -- images, not text.  Just as I was about to tell my friend that I didn't want to waste time converting six pages of tiny print to very usable data, my lovely assistant Lynnea (the aforementioned fiancee who actually plays the game) stepped in and volunteered to do it.  This is one thing about her personality that I've never been able to understand, but she loves doing data entry, filling out forms, etc.  I think it's some kind of OCD thing.  But for whatever reason, she's extremely enthusiastic, diligent, and thorough about this kind of work.  And by the way, if you need this kind of work done, she's available to hire!  Ask me for a resume.  :)

With this powerful slave human resource at my disposal, I wrote up a few sample lines of text in a spreadsheet to show how I wanted them, wound her up and let her go at it.  She cranked the rest out in a surprisingly short time.  I then converted the results to standard comma-separated value format, some of which you can download from here: Hacking weapon table part 1; List of effects.  In case you're not familiar with them, .csv files are a generic text-only format which can be read in a spreadsheet program like Excel, or any standard text editor.

Working with just my sample rows, I set out to work out what the abstract properties of the data were.  The first thing to consider is the way a body part is selected.  In the hacking weapon table, you can see that if you roll a 1-100, you get hit in the "Foot, Top"; if you roll 101-104, you get hit in the heel, 105-136 is Toe, and so on.

This is like a hash table, almost, but it's not one.  If it was a hash table, you'd usually have one body part per number: 1 -> Foot Top, 2 -> Heel, and so on.  Here, we're working with a range of numbers corresponding to each lookup value.

I decided to start with a generic lookup table, where you start with objects which contain a "low" value, a "high" value, and a generic object which can get returned from the lookup. The declaration looks like this:

public class RangeLookup<T>
{
   private List ranges;

   private class Entry
   {
      protected T item;
      protected int low, high;

      public Entry( T i, int l, int h )
      {
         item = i;
         low = l;
         high = h;
      }
   }
   ...
}

In Java using the "<T>" notation means that "T" could be anything.  Even though I wasn't going to be using this lookup table more than once, I like to keep structures as all-purpose as possible.  That's partly because I might want to reuse them in the future, and partly because I want to be able to test how the component works without making it dependent on the equally complex item which will be retrieved by the lookup.

Every structure needs an interface -- a means of communicating with it that only does what you want and hides the guts of it from the rest of the program.  I created an "addEntry" function to the RangeLookup class, so that you could insert a new entry with a high, a low, and a retrieved object of type T.  Then I added a "lookup" function where you send in a number, it gives you an object.  In my implementation, the lookup function simply walks through all of the possible results and checks whether the requested number is between the high and the low.  This would be inefficient if there were going to be a lot of entries, so I might have come up with some kind of hashing structure or tree search; but since there are only about 20 or so body parts, it wasn't worth the extra effort and runs fine as is.

After verifying that this was working right, I created the following additional structures:

• Looking at the Effects table, it is a basic mapping (in my case, placed in a HashMap) from one string to another.  You put in the code "f", and the resulting effect is "fall prone and drop items".  So, I created a simple object called an "effect," containing "key" and "description."
• It's a bit more complicated than that, though.  Often the table will contain numbers, but the effects will contain only the symbol "X".  For instance, if the table says "d4" then the relevant effect is "dX", which means "reduce Dexterity by X".  Therefore I made another class called an "Outcome," which contains an Effect AND a number (which may be zero if it's not necessary).
• I made an EffectTable, which implements the HashMap of Effects.
• Almost ready to create an actual table object, I first made a class called "CritTableEntry."  This represents a cell in the table.  It contains: a low roll, a high roll, the name of a body part, and a List of effects (because each cell may result in several outcomes, not just one).
• A CritTable class to put them all together.  This class has an addEntry method and another method for retrieving the entries.

As a final step, I created a "Reader" class which did the heavy lifting of reading and interpreting the CSV files and adding one row at a time into a generated table.  I don't like to reinvent the wheel, so I googled class libraries which would read CSV files and interpret them as lists.  I settled on using OpenCSV.  I could have written my own parser, but when the task is as common as reading a CSV, I tend to assume that somebody has already done all the work before me and has already been through the process of making all the mistakes and catching the bugs which come up.

Notice that none of these objects deals with input and output directly.  It's preferable to test each component of your program separately as much as possible BEFORE trying to decide what kind of user interface to make.  Your interface should be tailored to the problem space.  As it turns out, I wound up creating several different interfaces before I settled on created a web application.  I'll discuss these concerns in a later post.

When testing your data structures it's a good idea to create unit tests.  A unit test is a small, self contained application which is designed to test one thing at a time.  You need to think about every possible way that your program might break, create a unit test for each one, and make sure that it works right at the boundary conditions.

Off the top of my head, here are some boundaries of the crit tables that needed to be tested:

• Spot check several "body part" rolls with random(ish) numbers and see that the returned information matches the table.
• Spot check several "outcome" rolls in one row and see that the returned effects match the table.
• Test the boundaries of some rolls.  For instance, on the table I linked, "4301-4492" corresponds to "Arm, upper inner", and "4493-4588" corresponds to "Elbow".  Therefore I have to make sure that a roll of 4492 returns a different part from 4493.
• Test when happens when the body part roll is 0 (invalid), 1, 10000, and 10001 (invalid).
• Test what happens when the effect roll is 0, 1, 24, and 25.

Keep all your unit tests around forever.  If something breaks, that's a quick way of figuring out which part is not working.  If it's a problem with your data model rather than your user interface, the unit tests will catch it.

Next time I'll be talking about all the different ways of making an interface on the same models.

A bit about data structures

I wanted to write another HackMaster post, but what I wanted to write about was the way I approached deciphering the data in the tables and converting them into data structures.  Then I skimmed through some older posts looking for reference points about data structures, and it occurred to me that I've never written any. In order to provide a foundation for the rest of the HackMaster breakdown, I'll have to digress and talk about structures in the abstract.

Whenever you are presented with a problem of modeling some numbers in conceptual space, the first thing you have to figure out before you write a single line of behavioral code is what kind of data structures you are going to use.  Going all the way back to the beginning of this blog, I've emphasized the importance of considering the efficiency of your design and the effect that it has on the Big-O performance of your program.  Thinking about proper data structures can buy you a lot of speed, and it can also make it really easy to visualize your program in small chunks as the complexity increases.

So what's a data structure?  The first thing programmers learn is how to use variables for individual chunks of information, like this:

int x = 3;
String str = "Hello world.";

 (Technically, of course, a String object in Java is a whole bunch of characters, which makes it a data structure in itself.  But the nice thing about object-oriented programming is that you don't have to think about it if you want to.)

To understand data structures, consider an array.  An array is one of the first slightly more advanced concepts that a beginning programmer will run into.  Instead of storing just one integer, it can store several.  For example, here's a simple representation of part of the fibonacci sequence:

int[] fib = new int[10];
fib[0] = 1;
fib[1] = 1;
fib[2] = 2;
fib[3] = 3;
fib[4] = 5;
fib[5] = 8;
fib[6] = 13;
fib[7] = 21;
fib[8] = 34;
fib[9] = 55;

When you create a single "int," you're asking the program to set aside a chunk of space in memory, large enough to hold one number.  When you create an array like this, you're asking the program instead of set aside a bigger chunk of memory ten times that size, plus (for some languages) a little bit of extra information about size constraints and such.

But arrays can be wasteful.  What if you want to set aside space that sometimes houses a hundred numbers, and sometimes houses just a few?  You could create an array of size 100, but most of the time that space would be wasted.  That's when you want to use a linked list, where you ask for new memory only at the moment that you actually need it.

I'm not dedicating this whole post to the implementation fundamentals of lists, but interested beginners should go check out the Wikipedia article to find out how this works.  (Sidebar: While relying on Wikipedia for information about controversial topics is often unwise, most of the technical topics that are covered are really good.)

Besides linked lists, there are lots of other data structures that you can use depending on your situation:

• A tree (which may or may not be binary) will hierarchically organize information for you, much like the folder structure on your computer does, shortening the search time as long as you know where you are going.
• A hash table or map is a structure which will find a value associated with a key, usually very quickly.  An example would be a dictionary search: you supply a word, and the program would retrieve a definition.

You can write your own versions of these structures, or if your language supports it, use predefined classes that create common structures.

Understanding what purpose the various structures serve, and when to use each one, is a very key skill in programming interviews.  Often when you are asked "How would you solve this problem?" the best answer is not to blurt the first notion that comes into your head, but to start applying data structures to model the problem space: lists (or specifically, stacks or queues), trees (binary or otherwise), tables (sometimes you can just assume the existence of a database, which is centered around associative tables).

When I hear a problem that lends itself to this, I usually make a beeline to the whiteboard and start thinking out loud: "You're asking about a list of items, so let's describe what's in an item first... then build a linked list out of items..."  Then I'll be either writing code to illustrate what I'm thinking, or (if the interview is shorter) just sketch out diagrams so that the interviewer understands the description and will probably accept that I know how to implement it.

Software is built a piece at a time.  If you start explaining how you visualize the problem in your head, you can give a much better insight into how you think than if you just start solving the problem directly.  In fact, if you start off strong with this approach but then go off on the wrong track, often the interviewer will be eager to guide you towards his concept of the solution because he's being carried along with your thought process.  This often changes the dynamic of the interview entirely.  Instead of being a room with an interrogator and a suspect, the interviewer may start thinking of himself as your ally and not your judge.  And that's exactly where you want to be when you're looking for work.

Digression's over.  Next time I'll illustrate this when I get back to decoding HackMaster tables.

Operation HackMaster Crit Tables, episode 1

This post is about a project I recently did for fun.  Although the project itself has extremely limited application, it inspired me to do something I've been meaning to do for a long time, namely upgrade the web server for my apollowebworks.com domain to something which supports Java applications with Tomcat.  (I chose a company called Arvixe based on scanning the features they offer and reading a bunch of reviews.  Shout out to my boyz at Arvixe!)

Because I'll be touching on a wide range of topics, I'll split them up into multiple posts, and that should keep me from neglecting this blog for a little while.  So, partly to tell you what's coming and partly just so I can keep track for myself, here's a road map of the topics I plan to hit with this discussion.

• General geekery about using programming to automate complex tasks.
• Building a project from the ground up, starting by visualizing the data structures instead of just diving in blindly.
• How separating interface from implementation makes it easier to translate your program to multiple platforms.
• Good riddance to the bad old days: How the web has made program content delivery easier.
• A primer for noobs on what web servers do, and why I bought another one.

First let me explain the problem.  My fiancee, Lynnea, has gotten into paper-and-pencil roleplaying games.  She has been running a light Dungeons & Dragons campaign occasionally for me and my son Ben over the last few months. She also plays with a group of friends one night a week.  For my part, I've played RPGs a few times before but never been a strong part of that scene, so I just listen to the stories of her sessions.  Currently they're playing with a rules system called "HackMaster."

As I understand it from skimming a few chapters of the rulebook, HackMaster was written by someone who hates doing things the easy way.  When D&D was first published in 1974, it was at first an ever-expanding set of complicated rules involving tons of die-rolling for all kinds of special situations.  Or as a German exchange student my family hosted in high school put it, "D&D isn't a role-playing game, it's a roll-playing game."  Har.  Har.  Very droll, those gamers.  (Oh hey, how are things going, Max? ;)

Still, as the years went by, D&D started to reverse the trend and become more streamlined, or so I've heard.  The latest release, fourth edition rules, was clearly heavily influenced by World of Warcraft, and reduces a lot of the unnecessary choices in favor of more interesting combinations of focused options.

The HackMaster author just hates that.  Not that I would recognize the difference, but I have the impression that if anything, he's aggressively chosen to make every little action even more complicated than it would have been under old-school D&D, to the point where a party can spend thirty minutes meticulously checking a single room for traps before seeing any combat.  And the critical hit tables are an utter monstrosity.

Critical hits (or "crits") for you non-gamers, means that for every attack there is a chance (1 in 20 under D&D rules) that it will be a super-strong attack which does extra damage.  In WoW, this is handled automatically by the game; you can build up gear that will increase your crit chance, but usually there's a flat formula that says "If you crit, your strike does twice as much damage."  Rules in 4th edition D&D are pretty close to the same.

Not HackMaster.

In HackMaster, you roll a number between 1 and 10,000 for the body part where your blow lands, and there are all kinds of "realistic" outcomes that result if you land a powerful blow on that body part.  Furthermore, rolling a 20 is just your cue to pull out a ginormous, six page tables full of teeny little numbers and two accompanying rules pages.  You then roll another number that might range from 1-24 to see just how badly your crit hurt the monster (or the monster's crit hurt you).

For instance, if you get hit on the top of the head, the crit damage can range from "8 extra damage" to "twice normal damage AND you lose some hit accuracy AND you lose some dexterity AND you fall down and drop everything you're carrying"... to "brain goo."  Do not pass go.

Each one of those individual clauses, by the way, is represented in the table by a little symbol like "dX" which you have to look up on a smaller table of effects.  I think you have to look at a sample to see what I'm talking about.

(Click for a larger image.)

Not to put too fine a point on it, this sounds like a game that I would personally hate playing.  But no accounting for taste, you know?  Even so, problems of managing huge sets of data are piles of fun to solve with a computer.  NO, that's NOT sarcasm, you non-programmers, that's fun.  Shut up.

So when the DM tentatively asked if it would be possible to write a program that could handle the die rolls and just spit out some nice, clean output, I said "Piece of cake!"  I wouldn't even accept his offer to pay for it.  Just like when I wrote a Sudoku solver, sometimes figuring out how to automate gameplay is more fun than actually playing.  (Disclaimer: I used to claim that Sudoku is a fun math problem and a crappy game.  Now I like playing it.  It's my Android Evo's fault, damn it... it's the perfect game for a handheld, and it has built in strategy advice.  Once I started improving at harder puzzles, there was no escape.)

And so it began.

To be continued...

Recruitment spam

A new category of spam has been getting worse and worse. It's recruiters. They are tapping me with impersonal job requests (i.e. "Dear Technology Professional..."). Some of them are even a bit close to my skill set, but the jobs are nothing I would ever accept. Most recent was a six month contract in Kentucky. Honestly, even if I did have any desire to move to Kentucky, I wouldn't pick up my life and move for a six month contract..

This kind of spam is particularly insidious, because I can't just label it as spam and then let the GMail filter handle it... because it kind of resembles message that I would actually like to read in the event that I need to find work again. Honestly though, it's despicably lazy of the recruiters. They're just blasting everyone they can find, probably using a spider to automatically scan resumes and send out the messages without regard for location or a very good match. Argh.

On learning to love your users, who are idiots

In the process of discussing error checking in the comments section of a recent post, I flippantly remarked that "all users are idiots." This is a sentiment that I expect all veteran programmers will have encountered or stated themselves on some occasion.

From a new programmer struggling with this problem, I hear this: "This would be a heck of a lot easier if users just weren't idiots." And of course, that is something we all wish. But then again, if we didn't have to assume user idiocy, we wouldn't be writing programs.

There's a fundamental difference between constructing a program and writing a novel or painting a picture. For static forms of media, you only have to create one thing from beginning to end. Once you've finished writing your book, it's over. For better or for worse, your characters have finished interacting with each other. They've said what they have to say. People will either like it or not like it; they may debate endlessly about what your words or images "really mean," but they can't affect its behavior.

Unfortunately, programs aren't like that. Once your release your program into the wild, users get to do whatever they want with it. And if they do something utterly crazy and your program breaks, they'll blame you.

Murphy's Law ("Whatever can go wrong, will") was written by an engineer in the 19th century, but it is used most by software engineers. It's not that everything possible will go wrong every single time your program is run. It's just that if you have millions of users (which you will if you are successful) then even a very small chance that one person will do something unexpected, must necessarily magnify into a virtual certainty that somebody, somewhere will find a way to blow up your program.

With that in mind, writing a program is just as much about covering every possible angle of what some idiot user might do to you, as it is about creating a pleasing presentation when the program is Used As Intended.

As a gamer, I have heard several interviews with voice actors who are veterans of film or television, but new to performing voices for video games. The universal sentiment seems to be "This is the craziest thing I've ever had to do. You have to perform a dozen different lines for every single scene, and they're not just different takes for the editor to select from. They're ALL USED in the game. I'll have to perform a scene featuring my dramatic death, and in the very next scene I'm alive again. I'll have to answer the same question five different ways, just in case the player decides to harass me by asking my character over and over again." And so on.

So, because we must cover every possible use of the program, we create elaborate error scenarios and use all kinds of tricks to keep the user on track. One way to handle user error is to simply give the user very strict instructions, like this: "You MUST TYPE A NUMBER from 1-100, and if you do ANYTHING ELSE, then the program WILL CRASH and that is YOUR RESPONSIBILITY." That's not very satisfying, though. People make mistakes, even people who are not idiots. It's much nicer to recover gracefully from an error. At every step, you're asking yourself: "What can go wrong here?" And then you add a clause to your program: "If bad thing xyz occurs, say something polite to guide the user back on track, and try it again."

Often, an even better solution is to tie the user's hands so he can't actually make the mistake in the first place. For example, slider bars and drop boxes exist exactly so that the user can pick an integer from 1-100 without the capability to do something stupid.

Obviously, it takes a lot of work to write a bulletproof program, and the more thoroughly you prepare for bad user behavior, the harder the work is going to be. Often you have to strike a happy medium. The smaller your audience is, the less effort you have to put into mistrusting your users. If you are just writing a program for yourself, it's probably less work to try and give good input than to write error handling routines.

Also, the more general the application, the more you have to just let the errors happen, and your only responsibility becomes to make sure the errors don't cause a crash. For instance, if you're writing a calculator program, you can't just stop all operations that divide by zero. The user might just DECIDE to divide a number by zero. You just have to tell him he made an illegal operation, and move on to the next step.

When you write a program, you're not designing a static thing for people to look at. You're designing a universe of possibilities to cover all uses. The more time you spend thinking about what an idiot might do, the better you can guarantee that you will make it a pleasant experience for those who are not idiots.