Blog

Google glitch causes confusion

This morning, an apparent glitch at Google caused nearly every [update 11:44 am] search listing to carry the "Warning! This site may harm your computer" message. Users who attempted to click through the results saw the "interstitial" warning page that mentions the possibility of badware and refers people to StopBadware.org for more information. This led to a denial of service of our website, as millions of Google users attempted to visit our site for more information. We are working now to bring the site back up. We are also awaiting word from Google about what happened to cause the false warnings.
[Update 12:31] Google has posted an update on their official blog that erroneously states that Google gets its list of URLs from us. This is not accurate. Google generates its own list of badware URLs, and no data that we generate is supposed to affect the warnings in Google's search listings. We are attempting to work with Google to clarify their statement.
[Update 12:41] Google is working on an updated statement. Meanwhile, to clarify some false press reports, it does not appear to be the case that Google has taken down the warnings for legitimately bad sites. We have spot checked a couple known bad sites, and Google is still flagging those sites as bad. i.e., the problem appears to be corrected on their end.
For more information about how the process works and the relative role that Google and StopBadware.org play, please see our Clearinghouse page or this question in our FAQ.
[Update 1:36] Google updated its statement to reflect that StopBadware does not provide Google's badware data.
[Update 2:35] Hopefully this will be the last update, as Google has acknowledged the error, apologized to its customers, and fixed the problem. As many know, we have a strong relationship with Google, which is a sponsor and partner of StopBadware.org. The mistake in Google's initial statement, indicating that we supply them with badware data, is a common misperception. We appreciate their follow up efforts in clarifying the relationship on their blog and with the media. Despite today's glitch, we continue to support Google's effort to proactively warn users of badware sites, and our experience is that they are committed to doing so as accurately and as fairly as possible.

109 responses to

Google glitch causes confusion

Moses says:

I wish there is a option to turn off this function.

arun says:

Just shows that even the mighty google is prone to error ! However, what is a bit scary is that such a simple error can have such devastating effect on millions of websites and users. Important lesson for all of us as we rely increasingly on technology for everything,

Ivo says:

You wrote:

...[Update 1:36] Google updated its statement to reflect that StopBadware does not provide Google’s badware data.

How comes, that Marissa Mayer, VP, Search Products & User Experience still pronounces you as the cause of the error?!

She writes:

"...We work with a non-profit called StopBadware.org to come up with criteria for maintaining this list, and to provide simple processes for webmasters to remove their site from the list.
We periodically update that list and released one such update to the site this morning. Unfortunately (and here's the human error), the URL of '/' was mistakenly checked in as a value to the file and '/' expands to all URLs."

Get the full statement at http://googleblog.blogspot.com/2009/01/this-site-may-harm-your-computer-...

One plays with monkeys, one gets bitten!

Kenneth says:

Good the glitch is over. Thought my computer was infected or something like that.
Just hope it was a one-time issue.

@William: I see the same thing, guess it will be fixed soon.

Google Error says:

I'm glad to see that google admits it was an error and tell us what really went wrong. that blogpost was far for reassuring than just a plain 'apology' statement.

Keep the good work guys, after all nobody is perfect. :)

Maxim Weinstein says:

@Ivo: She doesn't.

Saquib says:

I really wish to read more and more on that topic, as how it happened, and what was the affected stats?
like when stopbadware site goes down, how many hits it receive before being down?
and several other statistical information is what I would love to have on this incident:)

Michael says:

Various websites are still listed with malware-alerts and Bitdefender which is a wellknown and wellreputed AV-company is suffering from this.

I noticed a while ago that a discussionforum for adult webmasters AWEmpire also were facing this issue.

The question that remains is whether this will happen again.
Unfortunately Im afraid it will.

@ver says:

Hm... Small bug has broken one of the biggest websystem in the web - google...
:-)

Emilio Vicari says:

To me, as a software developer, it's simply unbelievable that whatever manages those badware entry lists doesn't check for the infamous "/" entry...in a company such as Google someone should have thought about this.

DotA AI Maps says:

Yeah. Human error. Its ok. :)

ryan says:

Are you people seriously so handicapped that if Google disappears or malfunctions you are no longer able to browse the web? This is a pathetic display. "Web 2.0" has made you all like internet babies who need to be spoon-fed their digital content.

Michael says:

Bitdefender is still damaged by this and I wonder when Google will correct their position.
I also observed that one of the nets leading sites within the adult industry AWEmpire
was going on and off with the remark of "This site may harm your computer"

Guess some webmasters out there lost a lot of revenue today and I can't help thinking if this might happen again.

Themes says:

All our PCs were effected. For a moment there, we thought some new form of Internet browser worm had spread across the network! Good to see everything is back to normal. phew!

http://kipram.com/ says:

Pure human error

Maxim Weinstein says:

@Michael: Probably unrelated to this morning's incident, BitDefender.com was flagged due to an extremely rare false positive by Google's scanning. The Google team has corrected this, and the warning will disappear shortly.

Webdesign Kapeljosie says:

What 1 / can do..

Michael says:

Various websites are still listed with malware-alerts and Bitdefender which is a wellknown and wellreputed AV-company is suffering from this.

I noticed a while ago that a discussionforum for adult webmasters AWEmpire also were facing this issue.

The question that remains is whether this will happen again.
Unfortunately Im afraid it will.

NameGuy says:

The wording of a statement must be made with great care in situations like this... tensions run high when people don't get what they normally take for granted.

Smartone says:

You guys screwed up. Probably you will have to pay 2-3 million bucks to Google!

makine says:

i love google =)

Very Concerned ... says:

WHAT IF THIS GOOGLE SITUATION HAD BEEN PART OF A COORDINATED INTERNET ATTACK?

I am personally aware of these critical *technology* risks:
1. Seven of 13 Internet backbone servers down in 2002
2. Additional Denial of Service attacks on the Internet backbone
3. Major accounting firms have been sending IRS tax return processing offshore for several years
4. Recent major Internet slowdowns in Asia
5. Iran totally offline in early 2008 just before its oil bourse was scheduled to go live
6. Chinese hackers gained access to Department of Defense computers/trojan uploaded into DOD computers
7. Recent CheckFree attack/Identity Theft due to redirection of their domain to a criminal site
8. The intended Fannie Mae sabotage from last October that is just being reported now, possibly with a date of activation of *today*
("Rosenstein says that on that day, Makwana programmed a computer with a malicious code that was set to spread throughout the Fannie Mae network and destroy all data this Saturday. ") What if Fannie Mae is not the only company that was sabotaged??

I have spent a full week trying to escalate a situation that could have been the same as the CheckFree attack by fraudulent redirection in relation to IRS "free file" forwarding to a private organization with a GoDaddy domain created in 12/08 with proxy privacy.

I had phone conversations with High Tech Crimes and the IRS free filing manager, but there was no ability to talk with anyone live at Intuit or Quicken, the hosting providers of the only weeks-old/private GoDaddy domain.

When the Fannie Mae sabotage was made public, I escalated my concerns about the incredible risk if all of IRS information was forwarded or hacked.

Just imagine if everyone's full SSN, exact birthdate, full bank account information, valid e-mail address, UserID/password that could be the same as used for bank access, all W-2 EIN numbers of the employer companies, ... was accessed/stolen!

QUESTION: If someone simply attempted to access all of those bank accounts to transfer funds out of the country, WOULD THAT QUALIFY AS A RUN ON ALL BANKS? Would that qualify as a "high tech 9/11"?

Then, a couple of days ago, GoDaddy suspended my draft website because "Afilias, the .info registry, recently advised the Abuse Department that the domain zzz.info was associated with malware" and I was given a link to a Google page explaining that another domain was involved and I believe the malware was on its server.

I have been unable to get an answer as to whether my shared server had been hacked with malware or "just" that my domain was being forwarded to another site, just as CheckFree had been attacked.

Then, on top of my IRS concerns, GoDaddy's having suspended my own domain with the Malware search result, and the Fannie Mae grand jury results about a planned attack for *today*, 1/31/09, Google suddenly is very visibly sabotaged, by accident or by intention (which Google would never/ever admit to).

Upon further research, I found that Intuit, where all of the critically personal IRS data is stored, is now owned by Google and they are both located in Mountain View, CA.

Both Google and Intuit use www.markmonitor.com as their registrar. That international registrar states that it works with over 50 of the Fortune 100 companies.

QUESTION: Could MarkMonitor be accessed (legally or illegally) as it appeared Network Solutions was for the CheckFree redirection attack? Are those Fortune 100 companies at risk, like CheckFree proved to be?

THE MOST IMPORTANT THING I LEARNED IS THAT, JUST LIKE 9/11, THERE IS NO "RED ALERT" PROCESS TO REPORT AN INTERNET ATTACK!

Even the Chicago commuter trains now have emergency contact information to call or text, if the train is hijacked.

I tried every single one of these avenues:

Online form:
Internet Fraud report via FBI.gov site

E-mails:
- cert@cert.org
- abuse@GoDaddy registrar for possible spoof site
- abuse@IANU registrar for possible spoof site (due to a Read Receipt received via GoDaddy proxy with a different domain)
- zzz@IRS.gov

Phone calls:
- Counterterrorism Section (part of National Security Divison below), receptionist transferred me to on duty attorney, a MESSAGE MACHINE WITH NO CALLBACK
- National Security Division, receptionist thought I should call DHS, put me on hold for 6-7 minutes (clocked), still couldn't give me DHS phone number
- DHS Citizen Line, receptionist transferred me to an executive assistant who indicated he receives quite a few phone calls that he doesn't know how to handle. He gave me the name and phone number for the Director of Community Preparedness (which would be for *after* a disaster). I called - yet another MESSAGE MACHINE WITH NO CALLBACK.

NOTE: It is very obvious that those in charge of protecting our country before and after emergencies failed with 9/11, failed with Katrina, and are totally incapable of dealing with an Internet attack of any kind, even though there have already been such attacks.

WHY IS THERE NO "RED ALERT" PROCESS TO REPORT AN INTERNET ATTACK?? WHAT IF GOOGLE WERE STILL FAILING AS I WRITE? WHAT IF ALL BANK ACCOUNTS OF IRS E-FILERS WERE BEING CLEANED OUT RIGHT NOW?

John Howard says:

Why do so many people think this is a big deal? Why, just today I saw a person write today that linux is the most virus-free OS in the world and he didn't have to worry about malware and free internet and free software is the natural order of things. Google is one of those free things.

JAB_au says:

Human error, it's a wonderful thing.

Brandon Palmen says:

@John Howard -- As you might imagine, our servers are under a heavy load, and have been for much of the day. We're doing our best to keep everything running smoothly.

Ulco says:

An incredible but unfortunate error from Google. I am happy to see that they have taken responsibility for it and cleared the name of stopbadware.org.

Human error is everywhere, kudo's to Google (and you guys) for their quick and public correction.

goalpost says:

I did wonder why we got a hit on our Premier league site stats. This isnt good overall for Google!

John Howard says:

What are these Apache errors? Looks like this is the day for false error messages.

Nik@guru.com says:

Google will not last for long...

Facebook will become unreliable source for cash-flow on advertisement.

Mateo says:

I am going to sue somebody!

Cogmios says:

From a software developer point of view... this will become the biggest example of what can go wrong when putting stuff in production... marking ALL websites in the world as unreliable.... LOL! (well... do you remember some years ago when google marked all websites in thw world as unreliable.... *proest*).

I & Y says:

What was very strange was the fact that during the crisis period, the search term "youtube" turned up results that were fine.

sedat says:

this was really interesting happening, real chaotic and complicated for all web around the world, google is the best search engine..

Remember Google... says:

Anyone who remember this here, when guugle got hacked in germany last time?!

www.google.com/... says:

Wrongly labeling all legitimate sites in the world as dangerous to user computers is penny-ante stuff.

Another error that Google has sat on was a snaffu in Google Analytics (GA) that corrupted all data for any sites using segments.

GA started reporting laughable figures like 0.04 pages per visit (yes, not a typo... and when Google Support got a ticket they were AFRONTED we could doubt that GA is always right) meaning people visited and saw less than one page per visit.

It is alleged that GA fixed this with a change to _setVar as of Jan 27, but considering they compleltely messed up average bounce rates, time on page, visits, and basically everything until now, you be the judge.

------------------------------------
Dear Google Analytics User,
As an Administrative user on an account that is currently using custom visitor segments in Google Analytics, we would like to inform you of some changes to the _setVar method and its impact on bounce rate and time on page metrics. Starting Wednesday, January 27, 2009 a call to the _setVar method will no longer be counted as an interaction hit with the result that you may see higher bounce rates and more accurate time on page metrics in your reports.
[...]
Sincerely,
The Google Analytics Team

Sam J says:

It's interesting that BitDefender is still flagged as malware... I haven't found a bad word anywhere about them (quite the contrary in fact) and you can be sure that something like this would have ended up in their Wikipedia article at least. It's not just www.bitdefender.com either - it's everything under their domain (eg myaccount.bitdefender.com.

They're probably the last company that wants to be accused of being malware - you can be sure that everyone who sees this today won't bother them with their business again. I wonder what the liability situation with false positives is?

Sam

Krag says:

I am glad Google found the fault so quick!
It is funny that billions of web pages were blocked after only one wrong statement...

Lance says:

Google's been running how long now..

The claim is that this was human error. Yet it would seem illogical to leave such a mission critical piece of the puzzle for Google's search results at a wetware 'eyes-only' level for so long..

That and the fact the story keeps on contradicting itself by shifting blame (first Badware, now some anonymous Google employee) seems a greater tell than what is being said.

Wait and see when the anon Google employee decides to fess up..

Mathurin says:

In french : et bien ça a foutu un sacré boxon tout ça !
Dire qu'on pensé que cela provenait de stopbadware

Very Concerned ... says:

From http://googleblog.blogspot.com/:

"We periodically update that list and released one such update to the site this morning. Unfortunately (and here's the human error), the URL of '/' was mistakenly checked in as a value to the file and '/' expands to all URLs. Fortunately, our on-call site reliability team found the problem quickly and reverted the file."

@Cogmios/Lance - isn't "reverted the file" the offshore equivalent phrase for "restored the prior backup"?

Exactly who has access to Google production code? Are any of those people from China?

Obviously, there was no testing or even a manual code review by a second person of what was checked in to be moved into production.

What if in the "rollout", the problem had, instead, spread a trojan everywhere?

What if it *was* intentional, just like Fannie Mae and CheckFree? Is the anonymous employee long gone?

What if Google was still failing this many hours later?

It is my understanding that the power usage leveling of our entire electrical grid requires the Internet, as do ATMs and any debit/credit purchases-

Isn't the plotline of this season's TV show, 24, about holding the US hostage via a technology hijacking?

If Hollywood can imagine such things, so can others-

TheAphid! says:

That person should *NOT* be sacked. He or she is now the BEST person for the job because they will not make that mistake again!

Kostenlose Klei... says:

Hopefully this will not happens again ;-(

Are we human, or are we ...

google.com says:

In 8th grade English I learned that we use the passive voice when we've done something wrong and want to avoid taking responsibility, e.g. "the URL of '/' was mistakenly checked in as a value to the file."

Tomislav says:

At the end of the day me happy. It's not me picking up some annoying worm but Google messed up the internet :)

Roseli A. Bakar says:

People does make mistake. It happens.

mgmiller57@yahoo.com says:

Yeah, this created some real problems for me this morning.

I trusted you guys.

hr@wuce.net says:

I didn't use google at that time,so I did not miss the problem.I just know this news from yahoo.

Hok says:

I got Samsung official page as a bad site warning. And I think something isn't right. I am right :). But it is ok. I hope this mistake will not happen again.

wuce.net says:

I did not use google at that time,so I did not miss the problem.I just know this news from yahoo.

qwezz says:

I can't believe how ridiculous some people are about this.
BFD! This was an error, and certainly no reason to freak out and threaten to stop using Google. Torrence, mgmiller57, etc... you guys need to take a pill.

P.S. mgmiller -- enjoy the gay porn subscriptions.

Pages