Malware reporting study: more infomation leads to higher cleanup rate

Posted on March 21, 2012 - 10:22 by mvasek

I’m Marie Vasek, a computer science and mathematics student at Wellesley College and the resident testing intern at StopBadware. When a website is on one of our data providers’ malware blacklists and a person responsible for the site asks StopBadware for an independent review, I test the website to see if it is actively delivering badware. This past fall, I completed a study in conjunction with StopBadware and Tyler Moore of Wellesley College. We found that following StopBadware’s Best Practices for Reporting Badware URLs helped get badware sites cleaned up or taken down.

At StopBadware, we have a list of URLs that community members have reported to us as containing badware. We manually test all URLs from this feed to see if they contain badware, and when badware is present, we report the URLs to appropriate parties. In July, I started reporting URLs from the community feed in accordance with StopBadware’s Best Practices for Reporting Badware URLs; I tracked responses and regularly checked back to see if the sites had been cleaned up or taken down.

In October 2011 I began an academic study based on StopBadware’s pilot reporting project. My methodology was as follows: On day 0, I manually tested a URL taken from StopBadware’s community feed. If it was actively delivering badware, I randomly assigned the URL to one of three groups: control, minimal, and full. For the control group of URLs, no reports were sent out. For the URLs assigned to the minimal group, I sent out badware reports to the appropriate parties, but the reports contained only a minimal amount of information*. For the URLs assigned to the full group, I sent out minimal reports with additional detailed information* at the end. After the reports were sent out, I followed up on each of the URLs 1, 2, 4, 8, and 16 days after the day that I first found badware (day 0) to see if that badware had been removed.

The table below shows the probability that a URL will be “permanently” cleaned up after so many days. For the purposes of this study, I considered a URL "permanently" cleaned up on a day if on this day and every future follow-up day the URL was clean.

  1 day 2 days 4 days 8 days 16 days
Full report 32.1% 43.4% 45.3% 49.1% 62.3%
Minimal report 23.6% 25.5% 27.3% 36.4% 49.1%
No report 13.5% 17.3% 32.7% 38.4% 46.2%

*percentages represent the probability that a URL is “permanently” clean after x days with the specified level of reporting.

As you can see, sending a full report substantially improved the likelihood that an infected URL would be cleaned up. Full reports were also observed to be significantly more effective than minimal and no reports on every single day that I followed up on a URL.

But what does this all mean? It means that sending a detailed badware report appears to be an effective measure for getting a badware URL cleaned up. Furthermore, providing more details seemed to be helpful to the site owners and abuse teams who had the ability to clean up the badware.

We’re currently working on ascertaining whether other forms of notification sent in the same time frame (e.g., malware notifications from Google Webmaster Tools) could have prompted some of the badware URL clean-up we observed. Tyler Moore and I are in the process of writing an academic paper with the complete methodology and full results of this study; the paper will be published later this year.

*For examples of minimal reports and additional information, please see pages B-2 to B-4 of StopBadware’s reporting best practices.

StopBadware's 2011 Checklist

Posted on December 29, 2011 - 10:16 by ccondon

Last year, we posted a checklist of key accomplishments in our first year as a standalone organization. Our 2010 checklist included a lot of numbers—like the millions of users and webmasters who learned about badware via our educational pages or read our Tips for Cleaning & Securing Your Website—and while those numbers are still important to us, 2011 has been much more about engaging collaboratively with the security ecosystem to define new ways of thinking about the badware problem—and its solutions.

StopBadware's 2011 Checklist

  • By the numbers: Nearly 5 million people searching for information on preventing, identifying, and getting rid of badware found that information on our website. Those millions of people came from 211 countries and territories and spoke 204 different languages. Over 900 webmasters on our community forum,, asked for and received help getting rid of bad code that had compromised their websites. Our blog flourished, and our social media following grew by an average of 55%. And if that weren't enough, we also processed over 16,000 independent review requests from webmasters whose sites ended up on our data providers' blacklists.
  • We gained eight new partner companies this year, and all of them are fantastic, responsible, forward-thinking organizations dedicated to making the Web more open and secure: thanks for the great year, Verizon, Qualys, SoftLayer, Sophos, and Tucows! The other three we can't yet tell you about yet (though you're welcome to guess!), but look for announcements very soon. We also completely revamped our Partner Program so as to better engage and recognize our Partners. Have a look.
  • We published our inaugural State of Badware report, which analyzed badware trends, identified systemic weaknesses in the security ecosystem, and discussed key ways industry and policymakers could evolve to make the Internet more resilient to badware. It also leapt tall buildings in a single bound.
  • With advice from our cross-industry working groups, we developed and released two sets of industry best practices. Yep, count 'em. Two: Best Practices for Web Hosting Providers: Responding to Badware Reports, and Best Practices for Reporting Badware URLs. These best practices were a big first step for us in creating a collaborative, realistic industry standard that helps both reporters and report recipients streamline the badware reporting process, from detection to cleanup.
  • We commissioned a legal white paper on web hosting provider liability for malicious content from Harvard's Berkman Center for Internet & Society; this helps allay hosting provider concerns about taking good faith steps to address badware on their networks.
  • We launched the We Stop Badware™ Web Host program to recognize web hosting providers who are committed to security and to drive adoption of our web hosting best practices among the responsible hosts of the world. The program now has 28 participating providers from 13 countries across five continents. It's a big step, both for us and for the hosting industry.
  • We started a pilot reporting project, in which we reported URLs from our community feed in accordance with our Best Practices for Reporting Badware URLs. A research publication on the statistical results of this project will be forthcoming in 2012, but even preliminary results indicated that our initial foray into reporting was yielding a positive outcome.
  • We made appearances! Our executive director graced multiple panels and conferences with his badware-busting wisdom, a few of us rocked out and raised badware-awareness (badwareness?!) at HostingCon in San Diego, and we hosted our first-ever dinner in the Bay Area to get an in-depth discussion going on the badware threat and what industry players can do to combat it.
  • We got an award! Thanks to the ever-obliging Online Trust Alliance for bestowing us with the Online Trust Leadership Award for excellence in collaboration. We're digitally blushing.

We also physically moved this year: we left our beloved shared office in Harvard Square and hustled on over to the Cambridge Innovation Center, where espresso flows freely and start-ups of all stages huddle in iPad-controlled conference rooms. Staff Technologist Isaac regularly abuses snack privileges and our raconteur Caitlin still can't figure out how to use the office phones, but we have an office of our very own and two white boards on which we've already reinvented the Follow Friday Twitter hashtag. It's from here that we'll continue to build StopBadware and expand our badware karate chopping capabilities; with our amazing StopBadware Partners, hard working staff and intern, and lofty Board of Directors, the future is looking bright! 2011 has clearly been a big year for us (yeah yeah, we know—we said that last year, too). We're feeling like 2012 will be even better.

We're entering the New Year with our strongest group of StopBadware Partners yet. There's still much to be done; if you're interested in joining the discussion and the action in our partner community, let us know. We also welcome individual donations to help us continue and expand our existing programs.

A fond farewell to CastleCops

Posted on January 5, 2009 - 11:50 by egeorge

The anti-malware world lost an important resource recently, as the venerable volunteer community at CastleCops shut down. CastleCops was a leader in focusing the generosity of many technically savvy volunteers on the malware problem, and members of its community have an impressive record of impact over the years of its operation. We at StopBadware are grateful to all the members of CastleCops whose efforts helped make the web safer, and our jobs a little easier.
The close of CastleCops leaves a hole in the anti-malware community. While we know the uniqueness of CastleCops cannot be replicated, we would like to extend a warm invitation to the CastleCops community from our own Since its beta launch in the fall, has grown as a resource for ordinary internet users dealing with the fallout from badware, and we always welcome new members to our community.