Affordable Care Act Blues 2

Following up on my post from yesterday, a friend linked me to an article on Salon that attempted to diagnose the problems with the healthcare.gov website. The article is full of bad logic and half-baked assumptions that I couldn't resist correcting.

First of all, the article refers to a discussion on Reddit that was just ridiculous.

“The Redditors picking apart the client code have found some genuine issues with it, but healthcare.gov’s biggest problems are most likely not in the front-end code of the site’s Web pages, but in the back-end, server-side code that handles—or doesn’t handle—the registration process, which no one can see. Consequently, I would be skeptical of any outside claim to have identified the problem with the site.”

This is WAY too charitable. I’m not “skeptical.” I think the Redditors are dumb as rocks for wasting any time on this at all.

Server side (back end) code handles pretty much everything that is important about a web application's internal logic. The client side (front end) stuff -- which you can see printed out if you right-click on any page in your browser and select "View Source" -- handles cosmetic stuff. How the website looks, what kind of warning messages you get if you enter bad data, etc. It's possible for a site to break because of bad front end code, but that were true then it would almost certain be broken for everybody, not up and down sporadically.

I passed along the article to a friend who is the architect of our software at my job. His explanation is more thorough than mine, so I'm reprinting it with his permission.

Here’s how I would have written the same article but with my speculation….

I think the biggest and most important challenge is the overall architecture. From an architectural standpoint you need to ensure that your system is

1. Secure
2. Can scale appropriately
3. Handles the scenario where it is overloaded

Secondly, for the end user experience, from a UI perspective you need to be

1. User Friendly
2. Ensure proper feedback
3. Have as much client side validation as possible

Once you go live you are in a situation where you have to deal with all types of scenarios quickly. One issue that I see referenced a lot is the fact that the security question dropdowns aren’t populating. Maybe they load tested this but didn’t verify the contents of the actual html rendered. If you rely on older back ends then that needs to be part of your load testing and you need to reduce talking to those systems the least amount possible.

As far as server load there are 3 types of congestion you need to deal with. Memory, CPU and information I/O (Network and Drive). The dropdown thing might be related to either one of those three.

This, of course, is informed speculation about the nature of the problem itself, drawing on real experience about how websites are designed, and how to go about debugging the problem. The Salon author, David Auerbach, doesn't do this sort of thing. Instead, he casts about wildly to find a place to assign blame. He claims that he can identify it as an Oracle problem based on a single error message:

“Error from: https%3A//www.healthcare.gov/oberr.cgi%3Fstatus%253D500%2520errmsg%253DErrEngineDown%23signUpStepOne.” 

To translate, that’s an Oracle database complaining that it can’t do a signup because its “engine” server is down.

What? It is? How did he know that? I looked up “ErrEngineDown” to see if it might be a standard Oracle message. It is not. So my reading is that it’s simply the name that the developers themselves chose to assign to this particular error. There is literally nothing you can determine from this one status result, as far as identifying what kind of database they used, or why the database failed.

After that, Auerbach goes on to state that

That is, the front-end static website and the back-end servers (and possibly some dynamic components of the Web pages) were developed by two different contractors. Coordination between them appears to have been nonexistent, or else front-end architect Development Seed never would have given this interview to the Atlantic a few months back, in which they embrace open-source and envision a new world of government agencies sharing code with one another.

It's true, apparently there are at least two different developers who've had their hands on the system: Development Seed and CGI Federal. This is not a legitimate criticism of the process. The federal government is big. The ACA is big. It is routine and normal to have multiple companies working with one set of data. After all, each individual state seemingly has their own website which has to connect to the ACA. In that case, you typically have a web service host on the back end, which processes results, and the results come from many different clients -- i.e., the state's web site, which was probably developed at least partially by someone in the state.

And that's all we know. There is no evidence I can find in any of those links, that "coordination between them appears to have been nonexistent." Auerbach simply made that up. He also makes up a cute little fictional dialogue that has no grounding in reality whatsoever. That conclusion also does not follow from the fact that they use open source code and are enthusiastic about promoting open source principles, unless I’m missing some other piece of information cited. It may be the case, but it’s not demonstrated at all. Open source is just a model of developing something with transparency. It doesn't say anything at all about the process which the two developers followed.

To reiterate: We're talking about a website that is not working, for some people, some of the time. A friend posted just today that he successfully created his account and is done with the process. This is not a site that is fundamentally broken or flawed; it is a server that is sometimes overloaded by a very high volume of traffic. That's it. That's the whole problem.

Affordable Care Act rollout blues

I keep hearing journalists speculate about what the explanation might be for the Affordable Care Act website being so unreliable in its first week. While it sounds like a good talking point, the conversations have generally been pretty dumb.

You want an explanation? It's a massive online system that got hit with nine million users during launch week. That is more than World of Warcraft has subscribers right now.

As an IT engineer and an online gamer, I would be shocked NOT to see an online system on that scale experiencing technical problems. I played WoW since launch day. It had sporadic connections for the first week or two. The same thing happened when the first couple of expansions came out. Most other MMO's and numerous other games experienced similar glitches early in their life cycles also. Who remembers the PR nightmare of the latest rollout in the SimCity series? People who pre-ordered the game, mostly couldn't play at all for a week.

Non-functioning websites during launch week are the rule, not the exception. The reason is, an individual server can handle a certain number of simultaneous connections. For a large project, you use multiple servers running in parallel, so that they can spread out the load. There are also multiple redirections to other computers on the network; for instance, the database is housed on another server, and there might also be multiple parallel databases running in some cases. In the case of a big government system accessing social security numbers, it's likely to interface with some really old legacy systems somewhere along the line.

So in the first place, the throughput of the whole system can only be as reliable as its weakest connection. In the second place, systems are designed to handle an expected average load each day. One reason WoW was so slow at first, was because an especially high number of people wanted to be the first to play. After some time, usage settles down a bit.

You can't always predict how high the early load will be on the system. And even if you do guess correctly, it's not necessarily a good idea to buy double the number of computers that you'll need on an average day; that's a lot of expensive computing power that you'll just wind up having to sell off fairly quickly.

So, don't be surprised that there are problems. Be surprised that it's working at all most of the time.

Shotgun recruiting sucks

Dear recruiter at hire-knights.com, who has spammed me three times in the last two days:

Subject line: "HIGH PRIORITY !!! C++ Developer - Atlanta,GA - 6 months Contract"

Let me explain a few things to you. I am employed full time, with benefits. My primary expertise is not C++. I live in Austin. And the amount that you are offering is pitifully low -- considering that you want me to quit my job, uproot my life here, and be unemployed in six months.

Look, I sympathize, it's hard work being a recruiter, and maybe I'd be tempted to take shortcuts too. But in all honesty, the damage that you are doing to your reputation and your company by sending email about this "HIGH PRIORITY" position to every resume you can find on the internet, probably outweighs the likelihood that you will be successful filling the position this way.

You see, instead of helping me to take you seriously as a person worth networking with, all you've done is convince me that hire-knights.com is an organization that traffics in spam, that is bad at finding people jobs they want, and that is wasting the time and money of companies that might consider letting them fill their positions. Is that really what you want?

So now you're in my filter list. If I receive another message from hire-knights.com which is not a direct acknowledgment of this problem, all emails from that domain will go straight into a filter so that the emails will never be seen by me again. And either way, it's going to be forever a matter of record in all the search engines, that

hire-knights.com is a terrible company that is bad at hiring people

So now you've succeeded in attracting my attention. I hope you're happy.

A few more thoughts about Bitcoin and numeric values

Sorry to keep harping on this one topic, but I was reading the "Myths" page on the Bitcoin wiki and something struck me as kind of funny and strange.

Bitcoin has been compared to the gold standard, because the coins "exist" (so to speak) in limited supply. Indeed, believers in Bitcoin and believers in precious metal currencies seem to be the same people in a lot of cases.

But here's the interesting part: some of the arguments for Bitcoin openly undermine and negate the arguments for using gold and silver as currency. Check this out.

Further discussion about Bitcoin

My previous post about Bitcoin invited discussion, but much of that discussion took place on Facebook and Google+. A lot of good insights and links were offered, but brief messages on social media are hard to search and learn from in the future. So I'm writing a second post to acknowledge these responses, clear up some of the misconceptions I had in the first, and offer useful links for anyone who wants more information in the future.

Open discussion on Bitcoin

I've been hearing a lot of stories lately about Bitcoin, and while I don't fully get it, I'm currently on the side of people who think it's ultimately going to wind up being a very technically innovative Ponzi scheme. I'm not making this post to argue about that. Realizing there are a lot of cheerleaders for Bitcoin out there, this post will be heavily moderated. Sales pitches for Bitcoin will not be approved, nor will posts calling me names, although pro-Bitcoin posters are welcome as long as they can contribute to explaining the either the technological details or the details of economic distribution. A lot of the purpose of writing this post is to foster some technical discussion and assemble my thoughts about how it works.

. 

Bitcoin is an "alternative currency," which is something political Libertarians are always saying we need. It is meant to be decentralized, not controlled by any one government or individual. At the present time, one Bitcoin is worth around $100, down from a high last week of around $200. I am personally not particularly interested in getting involved with Bitcoin, either by trading my dollars for bitcoins and spending the bitcoins, or by speculating on buying low/selling high, or by mining for them. However, with my computer science degrees I'm mildly interested in working out exactly how the system works. I found a FAQ page, and I found the original technical paper by Satoshi Nakamoto, but I still don't have a full handle on it yet. Eventually I'm probably going to have to download the open source code and look at it, but I'm still trying to decide if that's worth my while.