Error Code 406: Not Acceptable

A pro-Obama, pro-Affordable Health Care technologist is horrified by the incompetence and waste in the exchange website project:

http://www.digitaltrends.com/opinion/obamacare-healthcare-gov-website-cost

It’s been one full week since the flagship technology portion of the Affordable Care Act (Obamacare) went live. And since that time, the befuddled beast that is Healthcare.gov has shutdown, crapped out, stalled, and mis-loaded so consistently that its track record for failure is challenged only by Congress.

The exact cost to build Healthcare.gov, according to U.S. government records, appears to have been $634,320,919, which we paid to a company you probably never heard of: CGI Federal. The company originally won the contract back in 2011, but at that time, the cost was expected to run “up to” $93.7 million – still a chunk of change, but nothing near where it ended up.

You really should read the whole thing. It's short. Go! Read!

Subject line joke / reference explained here.

Update: the cost figure above has been called into question here. I am not an expert in government procurement, but 'ordinary gentleman' seems to be competent at it, and suggests that what appears to be a gross increase in cost is actually because we're comparing apples and oranges – the bigger number is actually the price tag for an umbrella program. Go read the post; it's interesting stuff.

Last 5 posts by Clark

Comments

  1. says

    Was anyone expecting any chunk of software this complex to work out of the gate? OK, believing that government can change reality by passing laws is pretty much the definition of "leftist", but expecting massively complex systems designed against an unchanging deadline to work correctly first time out goes above and beyond the call of delusion. (I do not even want to think how many security holes must be in it, not even counting the ones deliberately put there by the NSA. I expect a monthly "Obamacare database hacked" article for the next 18-24 months, minimum.)

  2. xtmar says

    The big question to me is if they can get it fixed by January 1 or so. If not, then it will be difficult for them to enforce the individual mandate, because how can you punish people for not buying insurance when there is not market? Then it all goes down the tubes.

    I actually think this would be an interesting, if unlikely, solution to the debt limit/government shutdown problem. The GOP goes double or nothing on ACA failing, and essentially bets Obama that if the exchanges don't work by January 1, ACA implementation stops there (leaving in place only those parts that were in force before October 1, 2013), and if the exchanges do work, he wins.

    Obama would like cowardly if he didn't have confidence in the ACA exchanges, and the GOP would likely be correct, as the probability of either fixing the existing code, or starting from scratch, in the next three months to achieve operationality is relatively small, though I'm sure some of the more technologically informed out there can make a better judgement on that.

  3. A. Nagy says

    From what I heard the problem was they started too late, and the government kept running in and changing things after the ball got rolling. The 93million number was probably the lowball estimate on the original requirements, and adding new requirements late in the game costs ALOT more then putting them in originally.

    I'm not really sure why anyone thought it would be working day one just the media bum rush alone and everyone checking it out would overload the system. But by now it should really be working.

  4. xtmar says

    @Lizard

    Why would they need to put security holes in it? It's controlled by the government. They could just ask.

  5. Graham says

    Having been part of large projects in the past, i suspect part of the issue is someone high up confusing a "Project manager" with a "manager", which are two different things with two mostly differing skillsets. Its really very difficult to find good project managers for big projects, and their expertese is expensive.

    However, someone decides it needs a manager, appoints a senior administrator, the administrator doesnt really know how to organise a bunch of It people, so delegates, but demands regular reports, Bureaucracy gets out of control, The administrator doesnt know what to do, so throws people at the problem, fails to control expectations and looses control of what the people under them are doing, and it all explodes into a shower of wasted

  6. says

    The big question to me is if they can get it fixed by January 1 or so.

    This depends on how you might define "fixed".

    The problems that are purely a matter of throwing more hardware at it to deal with the load — if the software was written scalably — are minorish. But anything as huge as this application must be is going to have hundreds, if not thousands, of bugs. Some will be very minor and only occur under odd circumstances. Some will be major. The odds are good at least a few of the major ones will require large-scale refactoring of code to fix.

    If the standard of "fixed" is 100% bug free, it won't happen by January, or ever. Too big, too many moving parts, too much of the knowledge base split up among hundreds of developer's minds, many of whom will move on after performing minimal or no knowledge transfer.

    If the standard of "fixed" is "most people who want to use it can, reliably"… eh, I'm still betting on "no". Combine all the inefficiencies of complex system development with all the inefficiencies of government, and this thing is going to be a complex patchwork of hacks from now until the Singularity. (Bear in mind, also, that every change to the rules is going to necessitate an implementation of that change. If someone decides to add "How many cups of coffee do you drink?" to the formula to decide plan cost and options, you need to add in that field to the DB, and then work this in to all the parts of the code that might touch it. If they're smart, all the rules are external to the code, and you can edit/change them without altering the code itself… but even in that case, the design of the rules data structure will have been shaped by the needs of the existing rules, and Congress can and will come up with rules the old structure did not account for.)

    Assuming the ACA isn't defunded completely, whoever has the maintenance contract for this is set for life.

  7. Shane says

    @Lizard

    OK, believing that government can change reality by passing laws is pretty much the definition of "leftist",

    Bah, Lizard this is a "right" phenomenon too.

  8. Shane says

    @A. Nagy

    … government kept running in and changing things after the ball got rolling.

    This is so typical of anyone that wants a software product built. Not just the government here, but the software firm should have known better.

  9. says

    @Shane: Upon reflection, I must agree. Scratch that bit. Point to you.

    On changing software requirements — always happens, always will happen. At the risk of "argument from authority", I've been a corporate code monkey for 25 years, and the same patterns repeat over and over no matter how much the technology changes. (Even worse, there's a refusal to change — one place I worked found that due to how the tool they used work, the interface mockup they'd shown to the bosses would be unusably slow. A slightly different interface would be hundreds of times faster. However, the bosses signed off on one design, and that was the design that would be implemented, even though it meant the program could not be used, at all, with more than 5-10 data elements (it was supposed to handle thousands).)

    The topic here is MMORPGS, but it applies to any complex software, except the bit about the hats: http://mrlizard.com/rants/why-havent-they-fixed-this-bug/ (A main distinction, of course, is that you can stop paying for an MMORPG whenever you wish. The same is not true of government programs, unless you like being arrested.)

  10. says

    Was anyone expecting any chunk of software this complex to work out of the gate?

    Yes. Our team typically delivers production-worthy prototypes from the ground up in 3-6 months, and we've been delivering on highly complex enterprise applications approximately monthly for years. I see no reason that a team with multiple years of lead time, a well-defined objective, and a ridiculous budget shouldn't be held to a high standard.

    And let's face it: "works" is not that high a standard, and "works at least as far as core functionality is concerned" is even lower.

    expecting massively complex systems designed against an unchanging deadline to work correctly first time out goes above and beyond the call of delusion.

    You're not a professional software developer, right?

    I've been a corporate code monkey for 25 years

    Oh. So you're jaded! :D

  11. Hamilton says

    @Lizard:

    Was anyone expecting any chunk of software this complex to work out of the gate?

    Respectfully, if this were a private company then, well, yes. Or at least mostly. And for way less money.

    I wonder, if the government had instead said "we'll offer up $100Meg for the first person/company to drop a fully-functioning web site against this spec on our doorstep" what you might have gotten. I'm willing to bet something a damn sight better than what is out there, for a hell of a lot less money.

  12. says

    Note: everything said so far in the thread about ineffectual management, evolving requirements, unforeseen problems, unrealistic expectations of bug incidence, the need for refactoring, the overbearing complexity of large web applications and hydra-like datasource integrations, and the difficulty of making government bureaucracies engage efficiently is true.

  13. says

    You're not a professional software developer, right?

    For the definition of "professional" that means "I get paid for it, and stay employed in one place long enough that if I was utterly and completely inept, someone might notice", then, yes. As to meeting some standard of professionalism you might hold to, I have no idea.

    At the moment, I'm untangling code written over the span of 10 years, by multiple unknown and long-vanished developers, at different companies (each swallowed by a bigger fish), with barely any documentation as to either design goals or implementations. I have found that the code was halfway refactored, then refactored again (also halfway) leaving three versions of most of the classes. The purpose and design goals of each redesign are lost to time. I am currently tasked with verbing nouns, and also with evaluating the state of the code and estimating if we should try to bring the pre-refactored code into compliance with the rest of the code base, try to continue the twice-abandoned redesign, or smeg it all and start over.

    GTA V cost 250 million and took 5 years, and still has bugs (google "GTA V bugs"), for example. You could do it bug-free in 3-6 months? I am well and duly impressed. (I'd say GTA V is closer to the ACA app than to most business software, for reasons both technological and metaphorical.) Expecting the ACA application to get done in half the time and a bit more than twice the cost (I'd love to see the cost breakdown, as I'm pretty sure the real development costs were a fraction of the amount shoveled into that bottomless pit) strikes me as overly optimistic. I suppose we'll see how well it's working over time.

    The ACA app has to be designed to handle hundreds of thousands of concurrent requests, be usable by not just computer-illiterate, but actually-illiterate, people, run on a public internet, not a private intranet, and is subject to conflicting design demands from people whose motivations are politics, not profit. I can't claim you *couldn't* have built it in 3-6 months. I don't know who you are or what you're capable of. I will claim I find it dubious.

    {Edited by request to merge the contents of two posts. –D }

  14. A. Nagy says

    @Shane

    This is so typical of anyone that wants a software product built.

    This is why design processes exist but I have a feeling the government was disinclined to listen to the contractors protests. That and I have a bad feeling the planning stage at the start was rushed which is a really bad idea.

  15. Zack says

    I maintain that they should have just called Blizzard or EA. Blizzard has dealt with the launch of several always-online games and features (WoW, D3, the bulk of StarCraft II), maintains a robust website that processes payments 24/7 and secures data, has mastered the art of data security (as much as is possible in an online world) by offering physical and phone-based authentication programs to make data more secure, and deals with server loads of millions at a time, and understands how to conduct non-interruptive or minimally interruptive maintenance. EA is newer to the always online game (SimCity, ME3, Kingdoms of Amalur, etc.) but has similar experience.

    It's kind of ironic that the government is having to re-solve a set of problems that a videogame company solved pretty well and (arguably) decisively a few years ago.

  16. says

    GTA V cost 250 million and took 5 years, and still has bugs (google "GTA V bugs"), for example. You could do it bug-free in 3-6 months?

    I've neither said nor implied such a thing. As Pontius Pilate pondered, "What is bug-free software?"

    It's misleading to suggest that a massively multiuser business application is equivalent to a massively multiuser RPG, though. Wholly apart from the issues surrounding near realtime graphical rendering, just consider the difference between concurrent, siloed users in the biz app (who might, at worst, content for db writes) and concurrent interacting users in an RPG.

    Suggesting that the ACA-failure is understandable because MMORPGs are complex is, in truth, sort of lame.

  17. Shane says

    @Hamilton

    I wonder, if the government had instead said "we'll offer up $100Meg for the first person/company to drop a fully-functioning web site

    That would have required requirements that weren't shifting like the sands of the Sahara :)

    I obviously don't know if this is the case, but I strongly suspect it.

  18. Shane says

    At the moment, I'm untangling code written over the span of 10 years

    This is softwarespeak for cleaning the toilet.

  19. klover says

    All one needs to do is look at the crappy code linked in the article to see why this is a LOLZFAIL. Nothing like outsourced coding and Windows 2003 infrastructure to stand up a new govt IT process.

    Good thing to see, I guess, that the gov't bureaucracy/procurement process is just as bad as big corporate.

    Oh wait….

  20. Xenocles says

    "…how can you punish people for not buying insurance when there is not market?"

    Punishment? But it's a tax, not a penalty!

  21. Shane says

    @Zack

    I maintain that they should have just called Blizzard

    OMG NOOOOOOOOOO!!! Then it would have crashed the internet. Blizzards code is beyond buggy, and I have often wondered if they haven't started hiring government game developers and software engineers for their products.

  22. says

    @David: Apparently, a version of my post posted while I was still editing it, probably due to me hitting the wrong key. The later version represents what I intended to post. Please ignore the too-early posting. If an admin type could delete it, I'd appreciate it. If not, well, that's that.

    If you don't like MMOs, look at the bug lists and patches of office apps, single-player "A List" games, or any other equally complex product. "Late and buggy" is the default state for damn near anything more complex than "Hello, World". If you and your company avoid this — you're pretty much in the top tier of developers. People in the top tier of developers don't get hired by the government, for obvious reasons.

  23. Jonathan says

    @Lizard,

    Speaking as a total newb/amateur coder (so I Really Don't Know), would you agree that "late and buggy" is an inadequate description of the ACA website launch?

  24. David C says

    "Late and buggy" is the default state for damn near anything more complex than "Hello, World".

    True, but we're also not talking about edge cases here where the site breaks if your birthday is February 29 or if you are running version 2 of Opera on Windows ME. The site was pretty much unusable for everyone, from what I hear, and not just on the first day when everything was overloaded (and the volume is barely an excuse, since they should have known it would be that high.)

    Normally I'd demand that the government get its money back. However, they most likely did change the specs on them several times, so that won't happen.

  25. says

    @Jonathan: Well, it's not late — this was the scheduled launch day (probably part of the problem — "Ship whatever's checked in on the server!" never ends well). Having not used it myself, I also can't comment on *how* buggy it is; I've just heard a lot of complaints and comments that matched my expectations.

  26. says

    @Lizard. As you requested, I've combined the contents of your two posts . I did this without loss and in the spirit in which I think you intended them. Please let me know if you'd like anything changed.

    My bottom line is that a Massively Multiplayer Online Game is vastly more complex than a business application addressing a large user base because (a) the former is graphically intensive and (b) the former requires interaction among users (events, stims, callbacks, etc., all flowing in near realtime) and not just interaction with a workflow.

    Scalability as an architectural problem is neither new nor inadequately understood in the enterprise application space. Thread pooling for concurrent use has been available in every relevant framework for a good long time, transaction isolation levels are a normal part of database design, load balancing is well understood, and horizontal scaling is by no means uncommon.

    Yes, a competent team should have been able to provide usable and sufficient software, even in the face of evolving business requirements, in that time and with that budget. It wouldn't be bug free, and perhaps it would lack desirable enhancements, but core business functionality is not too much to expect under the stated conditions.

    False analogies from business logic to realtime graphical rendering of concurrent mutual interactions and contingent events in a graph of many thousands of users don't really support your claim to the contrary.

  27. Shane says

    @TomB

    Matt Drudge linked to a page of script from the HealthCare website.

    I certainly hope that this is a generated .js file, because I would have blown off .js validation if I was against a deadline, and left it on the server level. Slow, to be sure but less chance of bugs, and it can be added later when things stabilize.

  28. xtmar says

    @Lizard
    @David

    Since you two seem to have the most software related expertise, what do you think the end game is for the ACA exchanges? Do you think they scrap it and start over, try to jury rig it into an acceptable state, or do they just proclaim that it works, user experiences be damned? For that matter, what can they do in a realistic timeframe?

  29. Shane says

    @Sinij

    Dear conservatives, please get real.

    Why is it that you believe the only people that don't like the ACA are conservative?

  30. mcinsand says

    @Shane

    OK, believing that government can change reality by passing laws is pretty much the definition of "leftist",

    Bah, Lizard this is a "right" phenomenon too.

    'Partisan' is the word that I was thinking, although that isn't totally fair… close, but I have to believe that there might be a handful of people still in the country that register republican or democrat while still holding principles above party. Call me an incurable optimist, the shoe fits. Anyway…

    There does seem to be a tipping point at which people drink enough of the partisan Kool-Aid to believe that their adopted party can actually rewrite reality. Don't want immigrants? No problem. Make a fence and suddenly shovels and pickaxes will magically disappear. Don't want criminals to have guns (while having thousands of miles of land border). Also no problem. Just make them illegal. Don't want people to have marijuana, alcohol, video games, … Same deal.

    @whoever_is_responsible_for_the_new_preview_window

    You are absolutely, totally awesome! Well done, very well done!!!

  31. says

    @Sinij: Just wondering, who here has said "Bad coding=repeal ACA"? My stance, pretty clearly is, "Huge software program is buggy and had cost overruns. GIFs at 11." David thinks I'm too burned out from working in the dark, satanic, mills of software maintenance and patching to believe writing good software on time and on budget is possible. He may have a point. Xtmar is the only one really talking abut the political implications.

    Also, @David: I've got to work on my communication skills. ("Shutting up" would be a good start, says the peanut gallery.) My intended post was the complete second post, as I decided the deleted paragraphs weren't especially useful or productive. IAE, it's done, and since I've argued for the value of preserving text posted rather than dropping it down the memory hole, I think that's that. I will be more careful in the future. (This whole "thinking before I post" and "trying not to insult people unless I'm damn sure I *want* to insult them" is a new concept to me, and I'm still working it out.)

  32. says

    @xtmar: My bet is "proclaim it works, while juryrigging it fanatically".

    (And, to the joy of all, I will probably go very silent for a while. I have a lot of freedom to schedule my work time, but an hour spent on PH is an hour I have to somehow make up later on.)

  33. says

    @xtmar, I haven't examined the source code, configuration files, deployment artifacts, and production environment. I also haven't reviewed the capabilities and practices of the teams responsible for the current state of affairs.

    For this reason, I can't offer a professional opinion on what's likely to happen. Clearly there's a problem with risk mitigation, and quality assurance is wanting. But who knows what horrors lurk beneath? To answer would be sheer speculation based on no direct analogies.

  34. Shawn says

    CGI Federal is actually a pretty well run company in my personal experience. They also were featured on an episode of 'Undercover Boss', and it's not one of the best episodes, but it's still worth watching, if you want a good idea of what the company does.

  35. says

    "To answer would be sheer speculation based on no direct analogies." Also known as politics. And Lizard provides the perfect (and likely) political answer: "proclaim it works, while juryrigging it fanatically"!

  36. melK says

    At this point, I think they kinda have three choices:
    1) add servers and IP addresses to handle the load
    2) call for gradated enrollment ("Okay, California is done. Ohio, your turn.")
    3) Let the problem solve itself as frustated consumers give up, lowering the load on the server for now, and raising the odds of a class action suit against the federal government for enacting a law and putting unfair hurdles for those complying.

  37. says

    @Lizard – On this one, I'm going to have to disagree with you. It's not b/c of being burned out or anything else – I just think expecting failure up front is a little off. Good/Fast/Cheap pick two. Just think about the budget – it's not 10% over it's several times more. Time, plenty of time to get login and registraton pages working. Yes, big load, many moving parts – but there's not a single aspect of it that I know of that worked well. Big user load. Well, Amazon , Google, YouTube all do it regularly – now you don't even have to figure it out (it being super high volume) any more, it's a commodity and you can just copy what's already been done. $600.00 million + you can certainly get the first 25% of the experience handled and tested thoroughly. I guess it depends on where we draw lines on what's acceptable, but I think having 30% of the app 95% right is certainly doable (and should be expected). Sure there will be bugs, sure I'd expect problems, but not rampant throughout the app and not on the up front portions of it.

  38. says

    Oh god. I have heard of them. CGI bought the consulting company I used to work for out of grad school, AMS. They're a Canadian company…that might explain it. I think they spend too much time watching Curling, Hockey, and saying "eh" to know how to write a massive eCommerce website.

  39. stillnotking says

    The comparison with GTAV is instructive: GTAV cost less than half as much, gainfully employed more people, filled an actual human demand (i.e. one measurable in willingness to allocate finite personal resources, rather than willingness to pull a lever every 2 or 4 years), resulted in millions of productive, mutually beneficial economic transactions, and even — in whatever small way — increased the sum of human happiness on this benighted planet. Obamacare, as of this writing, has functioned as nothing more or less than a massive rent payment to a politically connected industry.

    The fact that conservatives are as quick to excoriate the former as the latter is the main reason I'd never call myself a conservative.

  40. Malc. says

    One could argue that the load problems categorically disprove the Republican thesis that no-one wants ACA aka Obamacare!

    It's worth noting that this system "only" provides marketing of 3rd party commercial health care planes, and the actual enrollment is handled by the usual Blue Cross/United/Kaiser/Whoever suspects. So it seems to me that there are two parts of the problem: the customer facing stuff (that is the subject of the "nothing works" claims), and the backroom side where they load data from the 3rd party providers.

    Since we know that the third party carriers are, in general, not thrilled by the upset to their private money mill that the ACA created, so it doesn't take a genius to conclude that they may have been less-than-helpful in working with CGI Federal.

    That leads to the situation where, due to a lack of cooperation, the backroom module development might consume vastly more time than public facing stuff, squeezing the latter's development schedule…

  41. Jonathan says

    @Malc,

    Your conspiracy theory doesn't strike me as very compelling. If the private insurance companies had been deliberately impeding the development of the ACA website, it doesn't take a genius to conclude that the administration would be using that as a (Very reasonable) defense for the current shortcomings of the product.

    But let me know how that knee-jerk defense of the government at the expense of private industry goes for you.

  42. David W says

    Since we know that the third party carriers are, in general, not thrilled by the upset to their private money mill that the ACA created

    You mean that 'upset' that says everyone must buy their product, and the government will also be giving them money? Why wouldn't they be thrilled about that?

    Obamacare is excellent for the insurers. It's the rest of us I'm worried about.

  43. Asher says

    @ shane

    Can you think of any right of center body of thought that even approaches feminism in terms of thinking it's possible to legislate away reality? Forget similar, I'd be willing to consider one that is even one-tenth as delusional as feminism.

    Hell, feminists make young earth creationists look like sober paragons of scientific reason.

  44. Shane says

    @Malc

    One could argue that the load problems categorically disprove the Republican thesis that no-one wants ACA aka Obamacare!

    Or that lots of people slow down to see horribly mangled bodies on the side of the road … news at 11.

  45. Asher says

    No, Shane, I don't think it's at all a ubiquitous human phenomenon, certainly not among the sorts with whom I willing converse. Aside from a couple of glaring blind spots, I see little of the "legislate reaity" mindset from the right. On the other hand, it's rampant on the left.

    Hell, anyone who utters the phrase "human rights" is already mired in that mindset.

  46. Asher says

    @ shane

    This reminds me of a story about a single mother with three children who got, iirc, close to a 1000 per month because they were diagnosed with "dyslexia". Some sort of caregiver stipend for caregivers of the "disabled".

    Dyslexia is a notoriously subjective diagnosis and what's more likely is that the kids just had low IQ, i.e. they were stupid. Basically, the government is paying stupid people to pop out stupid kids. What could go wrong?

    Paying stupid people to pop out stupid kids is just another example of leftists attempting to change reality via legislation.

  47. Zack says

    @Asher: Not necessarily. Human rights=inalienable rights, at least for me. When I use the phrase 'human rights', I mean rights that are inherent to sentience- rights inherent in sentience and sapience. Meaning, rights that are independent of law, government, biology, time, society, morality, and beliefs- rights that ought to be universal to all sentient and sapient beings, across the whole of time and space. Rights that, if someone violates them, it should make them an enemy of all thinking being.

    In short, if you find a caveman on a verdant planet five thousand light years away, or a brainy bug species on a cave planet fifty light years away, or a lizardman species underneath our feet, "human rights" are those rights that all these things have or ought to have in common.

  48. Asher says

    @ Zack

    I cannot make heads nor tails of your comment. It seemed a coherent description of your perceptions until you introduced the term "ought" into the last sentence. Is your theory of human rights descriptive or prescriptive? If something is inherent then I don't see how "ought" has anything to do with it.

    For example, I was born with two legs and, unless I lose them in an accident, two-leggedness is an inherent property of my entity as identity. It makes no sense to attach the term "ought" to a purely descriptive statement involving my two legs.

  49. Asher says

    Let me offer a proposition that is more concise:

    I have a right to 23 pairs of chromosomes

    Does that make any sense? I mean I already have 23 pairs of chromosomes and they are inherent to me as long I am alive. The point is that if something is truly inherent to something then it is superfluous to call it a right because in no possible world could that inherency be removed. Using the term "human rights" to describe something that is inherent is unnecessary and redundant.

  50. Kevin says

    @grouch: I'm not sure how you mean for that article to be an example of the "curvature of history" idea. Can you elaborate?

  51. Asher says

    @ Zack

    Basically, "human rights" ideologies are an attempt to legislate inherency where, clearly, none exists. I have never encountered human rights claims that do anything besides declaring something inherent that is very obviously not inherent.

  52. Eric says

    I'll read through the comments in a minute, but wanted to add my two cents……$634 MILLION for that hunk of crap? Hell, a kid fresh out of high school could code a site better than this monstrosity. I've been one of the ones trying to enroll, just to see what it would cost my family. I didn't make it past entering my family info the first night before it stopped saving.

    Then, when I return, I have to go back through information I have ALREADY entered, and either verify or re-enter it entirely. Ummmm…anyone heard of a database? It can SAVE information for later retrieval? I tried again earlier today, only to be met with more issues. I'm almost to the point to say forget it, and take my chances. The worst they could do is take it out of my tax refund. While that would be a hit, it wouldn't hurt me too bad…at least not right now.

    Still, just like the bill, the site is a POS that could have been done in a better way….by a high school kid probably.

  53. says

    Lizard, I'm with David – yes I think expecting this to work was reasonable, and I do not think an MMO is several orders of magnitude more complex. This software is not complex. The grant/aid/etc process on the back door is, and I have no doubt the UI for setting all of that up is an unmitigated disaster, because apparently nobody understands how all of it works.

    Mind you, working is a pretty broad concept. One of the problems I see in this, one I think you understand, is that the normals all see "nobody can log in" and just assume that means "bugz" and the ones that know what games are assume "and, just like a mahmorpigah" (proper pronunciation inserted for fun). It could work but have scalability issues, but I think we're way beyond that here.

    If the original ars-technica article is to be believed(2200 lines of HTML, 56 js files loaded, 11 css files loaded), this thing was destined for trouble regardless of load.

    And yes, I am a software developer by trade.

  54. says

    I'm not even really thinking of the log-in/load issues. If the code is even remotely half-way designed well, scaling should be straightforward.

    I'm thinking of all the rules logic, state, and interface, and how many combinations there are, and how well the system recovers from bad data, or a timeout, and what the user sees.

    The following is speculation based on articles and such. If someone with direct experience with the system wants to tell me I'm wrong, then, I'm wrong. Won't be the first time. Third. Fourth at the most. Anyway… I'm assuming this whole thing is filled with "If you're over 65 and make more than 56,451 annually and you're Native American but you're not living in North Dakota and you have three or more cats and only one of them is named 'Mr. Cuddlewhiskers' than the maximum subsidy is 45.12 percent of the square root of the number of paintings of Elvis on black velvet you own, but you need to select from options 1, 3, or 5, and if you select the wrong one you need to wait 10 minutes for the system to process it and then we send you back 15 pages, erase everything you've entered, and tell you there's an 'Invalid Form Data Block Helpful Code Alpha Zebra Romeo' and you start over."

    And every single one of those conditions, tests, and rules can change any time, and the interrelationships/dependencies are never as clear, intuitive, and automatic as they're supposed to be. Suddenly, "0" becomes valid when it wasn't before, and vice-versa, and code that didn't check for 0 because "higher level code will not allow 0 to be in this field" fails, but the actual error occurs a few thousand lines of code later and then comes days of tracing through the code to figure out exactly where it went wrong, because nothing in the routine that's actually causing the error says, or hints, "Oh, BTW, this should never be 0, 'cause Bad Stuff will happen in what looks like a completely unrelated section of code, but only if the following 14 other things happen first, which they don't in our test suite of data, but do in the real world."

    That's the complexity I'm talking about. One null pointer or invalid value or the like deep in any of a hundred different check/rule/validation routines propagates out error after error, and even with automated testing, odds are, you don't catch them all. In a perfect world of perfect code design you don't see:

    catch(someError e)
    {
    //I'll do something with this error — Joe.
    return randomPlaceholderVar;//temp code, Joe will fix when he's back from vacation — Fred
    }

    In the real world? You see it all the time. (Joe never gets back. He's got a better job. Fred, who works with Joe, dimly remembers this, but it's not one of his tickets. Fred's manager doesn't read code. Etc.) Sure, code review is supposed to catch it — but code reviews get missed or fail. QA is supposed to test it, but QA just saw "no error" and "randomPlaceholderVar" might work for 99% of the data. Unit Tests aren't always unit tested, and the test suite isn't always updated, and schedules are short and someone is going to just comment out a test or a condition so they can move on to "higher priority" items and "get back to it".

    Even if these sorts of things occur in one in a hundred routines, or one in a thousand, how much code is sitting on the back end of this monstrosity? What's the aggregate effect? When the deadline is fixed, corners get cut, testing gets rushed, "good enough is good enough". (Without a fixed deadline, of course, the contractor just keeps billing and billing. If the contractor's budget is fixed, they just declare bankruptcy and leave you with a few million lines of undocumented, buggy, code.)

    Turning on/off options dynamically based on data entry. Checking rules, rules, and rules. Making sure the code that runs the UI and the code that interprets the rules is completely synced up. (Gods help you if the rules are in the code, not in a rules database. Gods condemn you to every hell if you've actually put the rules into the interface elements directly.) Dealing with multiple browsers, with multiple plug-ins. (Salon refuses to render properly for me due to, I think, either AdBlock or Ghostery. This is a feature, not a bug.) Remembering that, to use another one of those hated MMO references, as Raph Koster said, "The client is in the hands of the enemy."

    "It shouldn't be that way!" is a nice sentiment. Maybe for some of you, it's not that way. I'm suitably envious. Need a tech writer? But I've seen it be "that way" at too many companies, and shared horror stories of it being "that way" with too many colleagues, to believe I'm just unlucky or that my experiences are extremely atypical.

    The last time I did a web app, it was 1996. I will assume the state of the art has advanced since then, and stuff I had to partially hand-code is now all done by well-tested libraries and it's a lot more plug-and-play. I admit to being biased by what a mess I remember it being, so that's something I should adjust for.

    The work I've done since involves exactly the same kind of rules-based stuff, and I am way too experienced with simultaneous changes in the rules and in the design goals to believe hitting the moving target that is the ACA would ever be straightforward or simple. Should the rules not change, or the design goals? Sure. But as Crow T. Robot wisely noted, "Wish in one hand, crap in the other, see which fills up first."

  55. George William Herbert says

    Way back at comments beginning, Lizard committed:

    Was anyone expecting any chunk of software this complex to work out of the gate?

    I was part of the team that implemented the blockbuster.com video sales website for Blockbuster in the late 1990s. It was designed as all-new, from the ground up. My particular responsibility was architecting the performance analysis and testing, hosting environment (computers and network it ran on, and the management of those systems), and leading the team that built those systems and networks and operated them. The same web company also did the web code, business back end programming, databases, etc.; my team was a very small part of the total package of programmers, web UI developers, database people, etc.

    It was committed to a launch date six plus months ahead of time by ads in the New York Times and other newspapers and magazines of record. We launched on time, on the specified day, at 95% of target performance (far more than actual customer demand load), and had no serious outages until we tried updating content and discovered a process bug.

    That site was, in total, connected to less back end entities than healthcare.gov, but had about the same total complexity of code and data, and demonstrated performance (tested loads) that would have supported at least the 5 million visitors that healthcare.gov saw its first day.

    The timeline commitment being so firm is unusual in the industry. Being able to launch a functional site is not. It requires that you have people designing it who know how website performance, stability, and operations work, and that you both test the heck out of things and that you build operations updates and process in from the beginning. Not all clients are willing to pay for it, but for those that are, they have a right to expect that it will work.

    (I have worked on other websites and large web-delivered services nearly continuously since this one, but I can't talk about those due to confidentiality agreements still in force. I am at one right now that everyone here will know by name, but I can't describe what I've been doing here.)

  56. Ken Hamer says

    To me it's rather sad that everyone here appears to have taken the $634 million at face value, when in fact it appears that is incorrect to a Prenda-like degree.

    Did nobody think to look further? Is everyone so down on government (I mean the worker bees, not the porcine elected types) that any non-fact-checked claim is accepted as is, so long as it corresponds to your preconceived notions?

  57. says

    A blockbuster video site being the same complexity of a heath care site that has an insane number of privacy laws pointed at it. Yeah. Sure. And if you believe that I have this lovely bridge to sell you.

    This post by a guy who works at Google delivering actually complex software in a challenging environment weighs in with some thoughts people might consider before spouting off on tech they clearly know little about:

    https://plus.google.com/u/0/116222833568410151476/posts/3N3yoaqeXHQ

    As an aside, 418 is a far better error code.

  58. George William Herbert says

    Kevin Lyda:

    A blockbuster video site being the same complexity of a heath care site that has an insane number of privacy laws pointed at it. Yeah. Sure. And if you believe that I have this lovely bridge to sell you.

    That would start at HIPPA and extend some distance from there, regulations which I regrettably have also worked under and had months of training in. Second only to NASA human spaceflight paperwork…

    What, did you think I was blowing smoke? There are commenters here who were born after I started building these sorts of things.

    The bridge is not for sale, it's for competitive bid to companies with existing qualified indefinite term indefinite delivery contract vessels, and it's a very shiny bridge. Any resemblance to the paperwork required for that other filthy commercial bridge you built over a deeper wider canyon is purely coincidental.

    The bridge is not required to be able to hold up its own design and contracting paperwork without collapse…

  59. Alkad Mzu says

    I work for CGI Federal. We have over two-hundred people working on this system. Predictably, they are freaking the fuck out. Unfortunately I know enough that I can't say much, besides "this is exactly what I would expect from dealing with websites CGI has produced on a day-to-day basis."

    Welcome to my world, guys!

  60. says

    If the code is even remotely half-way designed well, scaling should be straightforward.

    But it isn't straightforward. It's very much doable. But this being government work makes it a lot more complicated. And that's to say nothing of the fact that the idea that 200 people can work coherently on a system like this is quite dubious (if that number is accurate; Aklad Mzu that's not an accusation). This isn't like an MMORPG where we know we're going to need dozens of people on art, a dozen or two on rendering, actually coders, designers, guys dedicated to working on the physics engine, etc.

    Also, David is dramatically underselling his response to your 1996 comment.

    One null pointer

    I'm going to pick on you just a little bit by pulling this out of the larger context, before addressing the larger context, but if this is a possible issue in the server side code this project is an even bigger boon doggle than I thought. It's not, of course, that this couldn't be done in C or C++. It's that there are higher order languages that are better suited to this kind of work, and they take this sort of thing out of the equation unless someone is allowing so called "rock-star" developers to do whatever they want (in which case the failure is different but just as great). That's how different web development is (this is true of non web development as well, of course). Java had just appeared, C# wasn't even a wink in Hejlsberg's eye. We're just under a decade before Django would really start to change the way we think about web development (and a decade before DHH would beget Rails. . . and so on and so forth).

    As to the broader point: yes, yes. These things happen. The MMO bug rant is well stated. But this is why we have any number of other tools, technologies, conventions (well, I'm guessing there wasn't a lot in the way of convention on this one). Pick anything. Characterization tests, user stories, state diagrams. Yes, sometimes individual things get missed. The thing is, we aren't talking about a project where the problem appears to be a series of edge cases. The failure is deeper and more central.

    I applaud your effort to point out how complicated this stuff can get. But don't oversell it.

    Also, my point about the login stuff is that we're seeing all these comparisons to MMORPGs, but the problems here are fundamentally very different. It looks the same to normals (you guys are precious! hugs). But it's not.

  61. says

    "Null pointer" was on my mind because I'm chasing them in Java now. But it could be "improperly initialized object" or "invalid cast exception" or a bazillion other things. Any situation where the code for one function fails to check that what it got back from another function is valid for its purposes, and passes it on, can lead to the cascade-o-errors I discussed. But putting in all that checking code slows the code, and is redundant, because "Look, every single call sequence starts HERE. We put all the check code here, and we know all the data is clean from then on." Then another team, trying to solve another problem, but without the internalized knowledge of the structure of the other code, says, "Hey, why reinvent the wheel? We can use this bunch of code here, we just call it from here, and change this one line!" Maybe they even contact the other developers, and say, "Hey, this line, can we change it?" and the other developer is new, or doesn't have the whole code base in his head, or worked on that bit two months ago and no longer remembers it 100% precisely, and says "Sure, whatever." And so it goes…

    Listening to other posters here talk about their clean, perfectly designed, brilliantly engineered, on-schedule, bug-free, kludge-free, patch-free, OMGTheClientIsPissedDoSomethingQuick-free projects, I have to conclude I'm living and working on some kind of parallel hell world.

  62. says

    One quick addendum before I get back to my job on Earth-Kludge: A lot of my comments and experience come from working on projects started long before I got there and continuing, I presume, long after I leave. So, consider what this code will look like in 5 or 10 years, if it's a mess *now*. Especially consider that the rush to get it to be NOT a mess, driven by politics, will mean even the best, most careful, and cleanest of programmers will be forced to cut corners, leave out tests for edge cases, implement the C&P model of object inheritance, etc. (And then there's those who, confronted with this task, will simply walk away, taking their knowledge with them. How many cells can leave the brain, before it stops being a brain?)

  63. says

    Yes, Lizard. Unhappily, it sounds as if you're working in a place where the future has left you behind and where you don't know that your everyday struggles are, in many cases, long-solved problems of practice or implementation.

    Your sense of humor is great, btw, but you should know better than to opine on technologies you haven't touched in nearly 20 years. Why do you tolerate the mess you describe?

    Believe it or not, there are indeed boutique shops out there– even contracting to the government– that manage to do things cleanly, more or less correctly, and with some sort of broad transparency that leads to continuous improvement rather than continuous decay.

    For the sake of your heart, maybe you should seek one out!

    Also, what Grandy said.

  64. Klover says

    George – One of my pet-peeves, but it is not HIPPA. It is HIPAA (Health Insurance Portability and Accountability Act)

    I work in the medical device arena and whenever I hear folks in the industry use 2 P's for that acronym they immediately fall into the "do not listen, has no clue" category.

    Not saying you fall into that, unless you work in the healthcare arena at all?

    Cheers!

  65. HandOfGod137 says

    @Lizard

    Listening to other posters here talk about their clean, perfectly designed, brilliantly engineered, on-schedule, bug-free, kludge-free, patch-free, OMGTheClientIsPissedDoSomethingQuick-free projects, I have to conclude I'm living and working on some kind of parallel hell world.

    No, I'd actually say it means you are in the real world and a lot of other posters are basing their opinions of software development on a half-remembered reading of Neuromancer from 1993 and an episode of CSI Miami featuring the lab nerd rewriting the Unix kernel in real-time. Real software has bugs: Gödel's Completeness Theorem essentially guarantees it for systems above the most trivial level of complexity, and distributed heterogeneous systems makes matters worse. Object oriented technology mitigates the issue a bit, but doesn't make it go away, and you've still got to work with some middleware like CORBA to tie the whole thing together. This is not a system a "kid fresh out of high school" could do better, and saying so just illustrates a huge ignorance of the issues involved.

  66. says

    No, I'd actually say it means you are in the real world and a lot of other posters are basing their opinions of software development on a half-remembered reading of Neuromancer from 1993 and an episode of CSI Miami featuring the lab nerd rewriting the Unix kernel in real-time.

    Sorry Cochise. David and I do software development for a living, and we do web development for a living. You have failed to crack the case.

    Real software has bugs: Gödel's Completeness Theorem essentially guarantees it for systems above the most trivial level of complexity, and distributed heterogeneous systems makes matters worse. Object oriented technology mitigates the issue a bit, but doesn't make it go away, and you've still got to work with some middleware like CORBA to tie the whole thing together. This is not a system a "kid fresh out of high school" could do better, and saying so just illustrates a huge ignorance of the issues involved.

    My mistake, I see you were doing a sort of meta thing that blended commentaries from Lizard's post and magic-bullets in software development (a doubling down given the latter topic's datedness!). Clever.

  67. Khaim says

    @Shane

    Why is it that you believe the only people that don't like the ACA are conservative?

    That's not quite what he said – rather, he was implying that only conservatives want to repeal the ACA. I think that's entirely accurate. Sure, the ACA may be full of new bugs and hilariously inefficient, but it does fix most of the old bugs that it said it would, and overall is no more expensive than the previous system. Only people with a tribal interest in seeing the current administration fail want to rollback to the previous version of US Healthcare. Or the healthy, greedy, middle-class folks actually hit with the individual mandate, I suppose.

  68. George William Herbert says

    Klover wrote:

    George – One of my pet-peeves, but it is not HIPPA. It is HIPAA (Health Insurance Portability and Accountability Act)

    I work in the medical device arena and whenever I hear folks in the industry use 2 P's for that acronym they immediately fall into the "do not listen, has no clue" category.

    Not saying you fall into that, unless you work in the healthcare arena at all?

    Cheers!

    I rotate in and out, most recently in the last five years extended stints (six months to year) at a drug company and at a EHR company, most recently ending a bit over a year ago. It's probably been 18 months plus since my last round of mandatory training.

    Hmm. One newer client built devices, but I was far enough away from the product side that the FDA rules didn't apply to what I worked on, so no retraining.

    Since I don't work anything like full time in it, I don't presume to have memorized the terms or definitions. What sticks is the overall shape of things like mandatory reporting, the HIPAA privacy requirements and what and when you might have breached them (and what to do reporting if you do), etc.

    I would not, right now, attempt to comprehend the particular regulatory details without retraining. The details don't stick well enough. But I do "get it", have trained and worked under the current rules. I comprehend and can work with the required scope. I've done it before, repeatedly.

    Having varied clients helps compare project complexity across fields, but is hell on regulatory retraining every time you switch…

  69. Asher says

    CORBA! OMG….

    Back in the day I was in management at a healthcare company. If this is a reference to COBRA then this brings back some hilarious memories.

  70. HandOfGod137 says

    @Asher

    No, CORBA was/is a standard object request broker service. It allows heterogeneous systems to talk to one another by translating native method calls and arguments into an intermediate interface description language, so your C# application can talk to my Java EE business tier across a network. It's one of the technologies that glues big systems together (SOAP and REST may be a little better known). It's also one of the technologies with known defects (like every OS and language on the planet), which is why "defects per thousand lines of code" is a standard metric in development (and a developer who manages 4 defects/KLOC is considered very good indeed).

    David may be making a joke above (CORBA falls under the oversight of the Object Management Group, AKA OMG…I laughed until I stopped), but he and Grandy clearly work in some magical place where defects don't exist, and where the halting problem is "dated". And presumably where unicorns bring you your lunch at mid-day. In the real world no test environment matches what a system will really see and even formally proven methods can fail.

  71. says

    @HandOfGod137 You got the OMG joke, but failed to grok the fact that it's possible to produce good-enough, complex software in a timely way without achieving perfection or similar impossibilities. As a consequence, you fail to recognize what I've been saying, with which Grandy concurs: being a realist about professional software production (as we are) does not require you to deny that the rollout of the federal ACA exchange system is an unmitigated disaster.

  72. says

    I don't think anyone's denying it's a disaster. My initial comment of "How could it not be?" was not to minimize or excuse it. Given the likely requirements and the timeframe, I do not think anyone *should* have expected to complete it on time, and in a more honest culture, someone would have said that, and the deadline should have been extended. (Time, not money, is really the limiting factor. What's that old joke? "A project manager is someone who thinks 9 women can produce a baby in 1 month."?)

    You've pointed out my knowledge of web application design is woefully out of date, and I can't dispute that, esp. since I provided the underlying facts. I have to, therefore, give your opinion on this more weight, as you have more data at hand than I do.

    You believe (I think, correct me if I'm wrong) that both the time frame and budget allotted were reasonable, and this could have worked, given a suitably competent development team. You have more experience with this kind of work and the current technologies than I do. So, I'll cede all my pessimism, and try to turn this in a more positive direction. Throw out my "It's impossible, given the starting conditions and the shifting design goals." attitude. What do you think was the point of failure? How could it have been made to work, given when it started, when it had to launch, and the features it had to implement? Was the failure on CGI's end? Too little staff, not enough money, too much staff, too much money? Was it too much government meddling in the design process? (And could there be a government project without such meddling?)

  73. Shane says

    @Khaim

    From @Sinij:

    Oh no, website is down. ACA is doomed! Repeal!

    Dear conservatives, please get real.

    Why is it that you believe the only people that don't like the ACA are conservative?

    That's not quite what he said – rather, he was implying that only conservatives want to repeal the ACA.

    And my point is that conservatives aren't the only ones. Plus the whole tribal warfarce thing is pointless.

    … overall is no more expensive than the previous system.

    Asserted without evidence.

    Only people with a tribal interest in seeing the current administration fail want to rollback to the previous version of US Healthcare.

    And what about the Pepsi tribe that wants to ram a system down everyone's throat that creates an enormous incentive for government to regulate every decision that a person can make.

    Or the healthy, greedy, middle-class folks actually hit with the individual mandate, I suppose.

    Because it is greedy to try to stay healthy so that you don't have to pay a shit ton of doctor bills. But it is not greedy to have those that don't care about their health subsidized by those who do.

  74. says

    @Lizard Can't do an autopsy without examining the body in a lab. ;) You know the usual suspects, though: failure to buy into the methodology; failure to measure; failure in transparency; lack of accountability; violation of Brooks's law; religious commitment to particular technologies irrespective of compatibility, capability, extensibility, or resilience; mismanagement of clients' expectations, teams' deadlines, the feature set, the budget, or the personalities; failure to continually prioritize and mitigate risks; failure to perform unit, functionality, usability, reliabililty, regression, and acceptance testing often, in keeping with a plan, and with authority; inadequate issue lifecycle management, artifact management, allocation of labor, documentation of time usage, or talent; and who knows what all. This list of possible failure points could go on and on.

  75. AlphaCentauri says

    The code does look like it was written by committee. I have had the misfortune to deal with multiple healthcare-related software programs over the years, and that's a typical problem. Due to the rapid changes to specifications and relatively small market, the stuff being released is often not quite ready. If the average electronic health record were a video game, it would be considered a beta release (and would cost about $99,950 less for the same degree of programming complexity).

    The Ars Technica article describes the problem well. A purchasing system for acquiring toilet paper or inkpens isn't going to be adequate for ordering software — not at the rate that technology is changing. They can hire a top-rated healthcare software company, but that company may have a culture of accepting sloppy programming because they're used to constantly patching the programs they already produce every time the diagnosis codes or documentation requirements get changed. And government agencies are ordering something so far in advance that no one really knows enough about what the final product should look like to create a detailed enough RFP to separate the competent vendors from the lowball bidders. It might be more similar to ordering a weapons system that hasn't been invented yet, and we know how bad those cost overruns get.

    The bright spot is that half the legislators are ideologically opposed to Obamacare, so that means only half as many are pushing to get the associated jobs located in their own districts. Otherwise we'd probably have a website that kills people when it crashes, like the Osprey aircraft.

  76. says

    Here is one example from a carrier–and I have received numerous reports from many other carriers with exactly the same problem. One carrier exec told me that yesterday they got 7 transactions for 1 person – 4 enrollments and 3 cancellations.

    For some reason the system is enrolling, unenrolling, enrolling again, and so forth the same person. This has been going on for a few days for many of the enrollments being sent to the health plans. It has got on to the point that the health plans worry some of these very few enrollments really don’t exist.

    The reconciliation system, that reconciles enrollment between the feds and the health plans, is not working and hasn’t even been tested yet.

    Yikes indeed!

  77. says

    @HandOfGod137

    but he and Grandy clearly work in some magical place where defects don't exist, and where the halting problem is "dated". And presumably where unicorns bring you your lunch at mid-day. In the real world no test environment matches what a system will really see and even formally proven methods can fail.

    My people have a saying. "Flesh speaks, wood listens". This can be reasonably taken to mean "a child speaks, an adult listens". You appear to have (basic) theoretical but very little practical knowledge of how software development works. If you want me to treat you like a child, I will treat you like a child. If you want to actually discuss this, you need to learn to listen.

  78. says

    @Lizard

    You believe (I think, correct me if I'm wrong) that both the time frame and budget allotted were reasonable, and this could have worked, given a suitably competent development team.

    It's more than this. David will agree, but a suitably competent development team can absolutely still fail due to a variety of factors (many listed in his previous post), factors which are not directly related to the day to day specifics of implementing this or that feature. I believe that part of the failure with healthcare.gov is due to this. I believe other failures are "code related", but I suspect many of them were driven by the work being done in a very non ideal environment (too many teams working on the project, inadequate communication, inadequate management, not enough time for testing, etc etc etc).

  79. says

    I don't see this as an example of government waste. What we have with the ACA is the natural result of special interests writing the bills in Washington. Healthcare.gov has to go through private credit agencies to verify identity and this interface needs to add in an alphabet soup of government agencies and private contractors to sign somebody up. Healthcare.gov would have been a mess with the best software team working on it, let alone the incompetents who got the call.

Trackbacks

  1. […] However, there's a story making the rounds claiming that not only is the website a complete failure thus far, but it has suffered from cost overruns on the order of 1000 percent, with an initial award of $57 million (and additional option years bringing the potential value to a little less than $100 million) but payments to date of over $634 million.  This story has taken in people who should know better. […]