Traffic Didn’t Crash the Obamacare Site Alone. Bad Coding Did Too.

  • Share
  • Read Later
Brooks Kraft / Corbis

President Barack Obama speaks about the Affordable Care Act from the Rose Garden of the White House in Washington on Oct. 21, 2013.

Nearly 20 million Americans have now experienced the broken Obamacare website first hand. But Ben Simo, a past president of the Association for Software Testing, found something more than a cumbersome login or a blank screen—clear evidence of subpar coding on the site.

In mid-October, he went to Healthcare.gov to help a family member get insurance, only to find his progress blocked. When he investigated the cause, he discovered that one part of the website had created so much “cookie” tracking data that it appeared to exceed the site’s capacity to accept his login information.  That’s  the mark of a fractured development team.

Even more alarming were the security flaws. An error message from the site relayed personal information over the internet without encryption, while the email verification system could be bypassed without access to the email account. Both security vulnerabilities could be exploited to hijack an account. “Because this is a huge system that people are mandated by law to use, the standard should be higher,” says Simo. “People are going to see it as a high value target.”

At the time, President Obama was still arguing that the main culprit for the breakdowns was the popularity of the site. “The website got overwhelmed by the volume,” he said on Oct. 4. The reality, of course, was far more dire.

The basic architecture of the site, built by federal contractors overseen by the Department of Health and Human Services, was flawed in design, poorly tested and ultimately not functional. “You need there to be good people on the inside to make good contracting decisions and good people on the outside to do the work,” explained Clay Johnson, a Democratic technology consultant who recently worked as a White House fellow. “Right now, it’s the blind leading the blind.”

Even on the back end of the site, data was garbled and, in some cases, unusable. The nightly reports  that insurance companies receive from the federal government on new enrollees in the health plans have been riddled with errors, including syntax mistakes, and transposed or duplicate data, according to industry veterans.  In other cases, insurers received multiple enrollments and cancelations from the same person, but since the documents lacked timestamps, it has been impossible to know which form is the most recent. Companies have resorted to contacting enrollees directly to get answers, a solution possible only because so few have been able to sign up.  “We are seeing and hearing that enrollment files going to carriers are incomplete, there are errors,” said Dan Schuyler, a director of exchange technology at Leavitt Partners, a firm that consulted with several states in setting up their websites. “In three weeks or so when they start receiving these in mass volume, tens of thousands per day, it doesn’t matter if there’s a 1 percent error rate. Insurers don’t have resources to go through them and clean them up.”

After three weeks of breakdowns, Obama decided that he could no longer stand by his own spin. “Nobody is madder than me about the fact that the website isn’t working,” he said Oct. 21 in a Rose Garden speech that instructed others to stop “sugarcoating” the problems. In fact, the warning signs have been clear for months inside government, even if the White House failed to sniff them out. Federal auditors raised alarms in June, warning of missed deadlines and unfinished work. Administration officials have since put out the call for new contractors, and Silicon Valley talent, to fix the work. Jeffrey Zients, a top White House aide and former management consultant, has been tasked with leading the effort. But the pivot has not come with any new transparency about the problems. Obama’s aides refuse to confirm any particular bugs, or describe just what is wrong, as part of an effort at damage control aimed at keeping the public enthusiastic about the insurance marketplace. More clarity could come Thursday when information technology contractors plan to testify before Congress on their work.

Senior White House aides, including Chief of Staff Denis McDonough, met with insurance executives on Wednesday to chart a plan for solving the problems. “We have worked with the insures and the ‘alpha teams’ we jointly established made up of insurers’ technology experts and CMS technology experts, to iron out the kinks,” said Press Secretary Jay Carney in an emailed statement after the meeting.

Experts say the White House only has only weeks to fix the problems before they start to directly effect the success of health reform. Government officials hope to enroll 7 million people in Obamacare by the end of 2014. If they get substantially less, costs could rise for others in the system. Brett Graham, a managing director of Leavitt Partners, who has also consulted on the exchanges, says “insurers really need that process to be reliable by about Nov. 1.”

For his part, Simo tried to report the security vulnerabilities he found by contacting an online operator at the Department of Health and Human Services. But he has little hope that his message will get to the right people. The operator seemed confused about what to do with the information. After a half hour of delay, Simo was told his complaints would be forwarded the Federal Trade Commission, an agency that typically investigates consumer complaints, who would contact law enforcement as necessary.

With reporting by Zeke Miller, Massimo Calabresi/Washington and Kate Pickert/Los Angeles

252 comments
watsv
watsv

In reading this article, I am not surprised at the problems experienced with implementing Obama Care which could only occur because of the same fundamental reasons why several software projects fail, especially in the public sector; lack of clearly defined and agreed scope of work, lack of proper and thorough testing, lack of efficient and delivery focused project management etc..........Read On



HealthscareGov
HealthscareGov

we fixed it! apply for your free health scare... www.healthscaregov.com

Mitch_
Mitch_

Obamacare stinks, and apparently so does the webiste.

Honestly, I could write a better website in less than a month, less glitches, better design, and more security.

How come those developers cant, I mean, where did they learn?

Web development is EASY.

I wrote my website in C, compiled it as a Linux x64 executable I mean, come one, its not that hard!

Using PHP, for a website is even easier, whats wrong with these people.

And I am unemployed??

LAHs
LAHs

That is a great article Consumerist, esp. para 5, a controlled rollout. I myself, with limited knowledge of this topic was surprised that the all or nothing approach was taken. I think it would have been better to have beta tested it state by state ironing out bugs as they came in. I also love the conclusion: "Taxpayer dollars are helping write a great software project management case study. "


KiernanHolland
KiernanHolland

Custom web development is more complex than utilizing a content management system which is more geared toward blogs and forum utilization. Unless you've custom coded your own web software, you won't won't be able to easily go froma CMS to a custom solution. Its probably just a result of assignin the task to the wrong kind of web developers. You need real web software programmers, not web designers with fundamental programming skills.

SavvyJon
SavvyJon

I am very concerned about the current problems with the ACA (Affordable Care Act).  

I have been involved with very large system implementations for a top-10 US bank and they pale by comparison to the amount of code in the ACA.

The pressure is on system developers to "fix it fast", which is a recipe for disaster.  In the haste to introduce production-fixes, security considerations are frequently put to the side.  There is an old IT axiom:  “It is much better to build security in than to try and bolt it onafter it is in production”.

Initial review of the ACA web site by independent cyber security sources identified numerous vulnerabilities that could easily be exploited for exfiltration of private data.

The ACA database containing data on all Americans will make a very tempting target for organized cyber-criminals.  In this case it is more important to get it right than to get it fast.

I suspect that there are other problems, beyond those identified with the enrollment process that are yet to be discovered.

I have reached out to my Congressman and requested that the implementation be delayed until thorough review and testing of the code can be completed.  This should also include solid security policy and practices including employee access controls and monitoring.  

In banking, we employed field masking of sensitive (PII) data, and controlled access to this data through the use of role profiles.  The goal was to make sure that folks only had access to the information they need to do their job.  We even monitored account “touch” activity for signs of internal fraud or information abuse by employees.  In the haste to get the system up for enrollment, I believe we are prioritizing the urgent over the important.

I recommend delaying the ACA implementation until they can address the security issues as well as the code fixes

daridekas
daridekas

it is really hard to believe what kind of irresponsible peaple are dealing with so serious matters.admin2:invetrics

drudown
drudown

(enter exasperated Ron Burgundy talking to Congress from a telephone booth): "wait, hold on, let me say something…why isn't the twofold 'solution' for the ACA website problem to simply offer walk-in registration and registration via US Mail?" 

Spare us the campaign contributor sob song and use your brains for a change.

"To know and to act are one and the same." - Samurai Maxim

GlobDesign
GlobDesign

Infosys senior level meetings:

"We will dump 6 million Indians in US and capture their entire IT market and no American will ever come to know about this. We will throw these Americans out of their own country. They don't know what we are doing over here."

GlobDesign
GlobDesign

Wow all those Indian IT geniuses who we've imported to work on this stuff are a joke. 15 years ago when Americans ran IT everything worked flawlessly.

Aeropage135
Aeropage135

If it's from India, it's wrong.


(Massively Politically Incorrect, yet remarkably accurate, assessment of a developer of 20+ years)

americanLatina
americanLatina

How about the multi-hundred-million dollar NO BID contracts given to Obama donors? 
I know this won't surprise anyone, but that is where the half a billion dollars in contracts went.

Does anyone still think the unaffordable care act is about giving you free healthcare? It isn't. It is about power and control--and giving hundreds of millions of dollars in contracts to friends of the president.

BruceStrong
BruceStrong

What you kidding me, I figured it was Bush's fault, we know Obama didn't know anything, because well he really doesn't know anything. “[N]o matter how we reform health care, we will keep this promise to the American people: If you like your doctor, you will be able to keep your doctor, period,” President Obama said to an audience at the annual conference of the American Medical Association. “If you like your health care plan, you’ll be able to keep your health care plan, period. No one will take it away, no matter what. Again, [the Affordable Care Act] is for people who aren’t happy with their current plan. If you like what you’re getting, keep it. Nobody is forcing you to shift,” he later added.

CanePazzo
CanePazzo

Interesting that two Obama presidential campaigns were largely won through the use of technology and yet once inside the government confines of contracting policies these vendors were selected with the abysmal results.

mothergoosemc4
mothergoosemc4

no pun here but one "smart" cookie. KISS use paper and people it do not smart so much. If only the Cookie Monster still had a job.

youjustcannottrustanyone
youjustcannottrustanyone

Now we know what the republican party intended when they said Obamacare fight is not over.  The extortionist clowns have done it this time.

mothergoosemc4
mothergoosemc4

How many contractors on this project are not from this country. The people here are not real happy over the surveillance thing. Just wondering if we should have thought about fall out on that. We the people are a little more forgiving for being a nosy A hole. Others though might have tied into your systems. Seeing as this was publicized long before and the direct link to the IRS just maybe they bugged you?  A good indicator is time. If it is a laps bug cyber infection would not occur until after the FIX. SO who knows.  

CanuckinGermany
CanuckinGermany

Anyone who has been downsized or outsourced by companies like CGI is probably smiling right now.

mothergoosemc4
mothergoosemc4

Perhaps this is not a failed BUG in the system. The system did not roll out until after Snowden. Has anyone of your experts looked at weather data from the log on could be routing IP adress to another country? Kinda like a drag net reversed? Is any contractor connected to epytec corp tied to the build out of this system? All the way around I personally do not love this whole idea. None the less the US has made so may friends and are so well like in the world it is a realist question to ask.  Either way the delay could have resolved these problems. Not delaying may have created a whole different beast. I really just want to say that the whole idea is crap. But I am willing to give benefit of doubt. And maybe look at it from a different perspective.

CharlesEdwardBrown
CharlesEdwardBrown

Obamacare is a train wreck. The problems at the website are only the tip of the iceberg. Democrats keep defending it, but they should remember that the road to Hell is paved with good intentions.

Seppie
Seppie

It failed before the site, when it was claimed as Affordable Care Act.

yakasha
yakasha

Meanwhile, on time.com (this story):

1.  69 cookies  (really, 69 cookies when I haven't even logged in?)

2.  new comments push the comment you're currently reading down, making it difficult to read.


I'm already bored "investigating".


Throwing stones in a glass house?  tsk tsk.

jonokie
jonokie

9th paragraph, first sentence - the word "effect" is misused. Should be "affect." Bad code everywhere!

endloser
endloser

It sounds to me like, blah, blah, bs, bs, you should trust me because I've been a professional website coder network admin server architect backend buzzword for umpteen years.


Seriously? All these people who speculate so much can not be in "the biz". And if so, they are likely not very successful since they tend to speak from the posterior. We do know there are backend issues. And yes, there was likely some additional stress placed from load. We know this because we have been told this. It is a closed system. Anyone pretending to know more about this is doing just that, pretending. The only people who know what is wrong can't tell you because of NDAs. Don't believe any of these comments where mister/misses topcoder knows all the woes.

GlobDesign
GlobDesign

@americanLatina How about giving hundreds of millions of dollars in contracts to foreign firms like Tata and Cognizant whose workers are even allowed to be here working under US law as long as Americans are unemployed.

All of these contracting cons are really about redistributing America's wealth via the conduits of our jobs and their remittances back to India. Gorbachev called it "International Socialism".


All we're getting in return is a broken country where nothing works.


Thanks India.

spookiewriter
spookiewriter

@americanLatina Be very careful when criticizing anyone about no-bid work.

The last administration did loads of that as well. Or, do you really think Halliburton getting all that work was just happenstance? I'm sure Cheney being the previous CEO meant nothing?

barneydidit
barneydidit

@BruceStrong Yeah, we should impeach the guy for only being 96% correct with his statement instead of the 100% correct we're so used to with Republican "I'm going to shrink the size of the federal debt" Presidents. 

mothergoosemc4
mothergoosemc4

@CanePazzo  Scary to think we trust them with nuclear, bio, and chemical weapons. Can not wait for the ball drop on that. 

GlobDesign
GlobDesign

@mothergoosemc4 Most of it went to Indian RICO conjob shops which siphon huge amounts of $ out of America via their IQ81 workers who ship all the $ back home. They get all our $, wreck our jobs and companies, and we get broken junk. Welcome to globalization.

GlobDesign
GlobDesign

@CharlesEdwardBrown Companies ruined or almost ruined by imported Indian labor

Adaptec - Indian CEO Subramanian Sundaresh fired.
AIG (signed outsourcing deal in 2007 in Europe with Accenture Indian frauds, collapsed in 2009)
AirBus (Qantas plane plunged 650 feet injuring passengers when its computer system written by India disengaged the auto-pilot).
Apple - R&D CLOSED in India in 2006.
Australia's National Australia Bank (Outsourced jobs to India in 2007, nationwide ATM and account failure in late 2010).
Bell Labs (Arun Netravalli took over, closed, turned into a shopping mall)
Boeing Dreamliner ES software (written by HCL, banned by FAA)
Bristol-Myers-Squibb (Trade Secrets and documents stolen in U.S. by Indian national guest worker)
Caymas - Startup run by Indian CEO, French director of dev, Chinese tech lead. Closed after 5 years of sucking VC out of America.
Caterpillar misses earnings a mere 4 months after outsourcing to India, Inc.
Circuit City - Outsourced all IT to Indian-run IBM and went bankrupt shortly thereafter.
ComAir crew system run by 100% Indian IT workers caused the 12/25/05 U.S. airport shutdown when they used a short int instead of a long int
Computer Associates - Former CEO Sanjay Kumar, an Indian national, sentenced to 12 years in federal prison for accounting fraud.
Deloitte - 2010 - this Indian-packed consulting company is being sued under RICO fraud charges by Marin Country, California for a failed solution.
Dell - call center (closed in India)
Delta call centers (closed in India)
Fannie Mae - Hired large numbers of Indians, had to be bailed out. Indian logic bomb creator found guilty and sent to prison.
GM - Was booming in 2006, signed $300 million outsourcing deal with Wipro that same year, went bankrupt 3 years later
HP - Got out of the PC hardware business in 2011 and can't compete with Apple's tablets. HP was taken over by Indians and Chinese in 2001. So much for 'Asian' talent!
HSBC ATMs (software taken over by Indians, failed in 2006)
Intel Whitefield processor project (cancelled, Indian staff canned)
JetStar Airways computer failure brings down Christchurch airport on 9/17/11. JetStar is owned by Quantas - which is know to have outsourced to India, Inc.
Lehman (Spectramind software bought by Wipro, ruined, trashed by Indian programmers)
Medicare - Defrauded by Indian national doctor Arun Sharma & wife in the U.S.
Microsoft - Employs over 35,000 H-1Bs. Stock used to be $100. Today it's lucky to be over $25. Not to mention that Vista thing.
MIT Media Lab Asia (canceled)
MyNines - A startup founded and run by Indian national Apar Kothari went belly up after throwing millions of America's VC $ down the drain.
PeopleSoft (Taken over by Indians in 2000, collapsed).
PepsiCo - Slides from #1 to #3 during Indian CEO Indra Nooyi' watch.
Polycom - Former senior executive Sunil Bhalla charged with insider trading.
Qantas - See AirBus above
Quark (Alukah Kamar CEO, fired, lost 60% of its customers to Adobe because Indian-written QuarkExpress 6 was a failure)
Rolls Royce (Sent aircraft engine work to India in 2006, engines delayed for Boeing 787, and failed on at least 2 Quantas planes in 2010, cost Rolls $500m).
SAP - Same as Deloitte above in 2010.
Singapore airlines (IT functions taken over in 2009 by TCS, website trashed in August, 2011)
Skype (Madhu Yarlagadda fired)
State of Indiana $867 million FAILED IBM project, IBM being sued
State of Texas failed IBM project.
Sun Micro (Taken over by Indian and Chinese workers in 2001, collapsed, had to be sold off to Oracle).
UK's NHS outsourced numerous jobs including health records to India in mid-2000 resulting in $26 billion over budget.
Union Bank of California - Cancelled Finacle project run by India's InfoSys in 2011.
United - call center (closed in India)
Victorian Order of Nurses, Canada (Payroll system screwed up by SAP/IBM in mid-2011)
Virgin Atlantic (software written in India caused cloud IT failure)
World Bank (Indian fraudsters BANNED for 3 years because they stole data).

I could post the whole list here but I don't want to crash any servers.

spookiewriter
spookiewriter

@CharlesEdwardBrown I guess you didn't get the Fox memo about "our Jesus" Jon Stewart as he tore into the roll-out.

Yes Virginia, we do criticize ourselves.

TigerFlower
TigerFlower

@CharlesEdwardBrown There are plenty of Democrats who are not defending it. Silly blanket assumptions really don't contribute to the dialogue.

shepherdwong
shepherdwong

@yakasha Good catch! TIME's blog knocking an IT roll-out. They've been at this for years and it's still a "trainwreck."

(To be fair, do like the new edit feature, tho).

GlobDesign
GlobDesign

@endloser Google "Obamacare Indian contracts". India's job-hungry RICO shops were salivating over this like your dog does for scraps when you sit down to eat a steak.

yakasha
yakasha

@endloser "All these people who speculate so much can not be in "the biz"."

So, are you saying you ARE "in the biz", and are therefore qualified to explain why nobody else "in the biz" could come up with legitimate comments concerning the bad design?

As somebody very qualified to be "in the biz", I can tell YOU that you have no clue what you're talking about.  So, either you're NOT "in the biz", or you are completely incompetent and should not be "in the biz".


kthx.

QualityFrog
QualityFrog

@endloser I have not claimed to know all their woes. I do do know what I can see. There is enough exposed that gives me reason to question the design. Please see my comment at http://ti.me/1c0OEAu for info on what I do know. -- Ben Simo

yakasha
yakasha

@shepherdwong @yakasha You don't notice the cookies if they're not abused.  But I could do without the comment movement, and the ads that make the entire page move as they splash across and javascript that hijacks common keyboard commands like "control^u" when I'm trying to view their source.

protip for Time's developers:

http://stackoverflow.Com/questions/2903991/how-to-detect-ctrlv-ctrlc-using-javascript

This is how you detect the control key combined with any other key you're not actually using in your site to avoid breaking user's browsers.

aztecian
aztecian

@QualityFrog you don't work there, you can't see the code.  there could be a number of issues disrupting the system including hackers and people paid by the right wing to take it out.  there could be many angles to this.  this will come out in time.

drudown
drudown

@notofthisworld @aztecian @QualityFrog 

Says the person against an independent investigation? Spare us the notion that the People should take a SINGLE word of the GOP at face value. 

"Opportunity makes a thief." - Francis Bacon

GlobDesign
GlobDesign

@aztecian @QualityFrog Or it could be IQ81 coders from India who paste localized test from several different languages directly into JavaScript code. That's just one of the examples uncovered in this mess. Face it: redistribution of America's tech jobs to people who cannot even build enough toilets back home is the cause. No one else's.

notofthisworld
notofthisworld

@aztecian @QualityFrog You're an idiot. If it was hackers, then it just tells you how vulnerable your personal info really is. That means hackers are helping expose this disaster! Wake up and start thinking for yourself.

yakasha
yakasha

Aztecian, read my comment below.  You don't know what you're talking about.  You're embarrassing yourself.

aztecian
aztecian

@QualityFrog @aztecian i think you're extrapolating too much without seeing the engine of the system.  in time we'll see there was more going on than just bugs, glitches and software structure.  there is definitely many in the right wing working all angles to take out this system and that cannot be denied after witnessing the Shutdown, orchestrated by a minority to foil the rule by the majority.

QualityFrog
QualityFrog

@aztecian I can see the client-side code sent to my browser. I can see the service requests (and the responses) made by my browser. My assessment is based on what I can see; and what I see is concerning.

People trying to take out the site do not cause bad design and security vulnerabilities (which I can see from outside) in the design and implementation.