Malware reporting study: more infomation leads to higher cleanup rate

Posted on March 21, 2012 - 10:22 by mvasek

I’m Marie Vasek, a computer science and mathematics student at Wellesley College and the resident testing intern at StopBadware. When a website is on one of our data providers’ malware blacklists and a person responsible for the site asks StopBadware for an independent review, I test the website to see if it is actively delivering badware. This past fall, I completed a study in conjunction with StopBadware and Tyler Moore of Wellesley College. We found that following StopBadware’s Best Practices for Reporting Badware URLs helped get badware sites cleaned up or taken down.

At StopBadware, we have a list of URLs that community members have reported to us as containing badware. We manually test all URLs from this feed to see if they contain badware, and when badware is present, we report the URLs to appropriate parties. In July, I started reporting URLs from the community feed in accordance with StopBadware’s Best Practices for Reporting Badware URLs; I tracked responses and regularly checked back to see if the sites had been cleaned up or taken down.

In October 2011 I began an academic study based on StopBadware’s pilot reporting project. My methodology was as follows: On day 0, I manually tested a URL taken from StopBadware’s community feed. If it was actively delivering badware, I randomly assigned the URL to one of three groups: control, minimal, and full. For the control group of URLs, no reports were sent out. For the URLs assigned to the minimal group, I sent out badware reports to the appropriate parties, but the reports contained only a minimal amount of information*. For the URLs assigned to the full group, I sent out minimal reports with additional detailed information* at the end. After the reports were sent out, I followed up on each of the URLs 1, 2, 4, 8, and 16 days after the day that I first found badware (day 0) to see if that badware had been removed.

The table below shows the probability that a URL will be “permanently” cleaned up after so many days. For the purposes of this study, I considered a URL "permanently" cleaned up on a day if on this day and every future follow-up day the URL was clean.

  1 day 2 days 4 days 8 days 16 days
Full report 32.1% 43.4% 45.3% 49.1% 62.3%
Minimal report 23.6% 25.5% 27.3% 36.4% 49.1%
No report 13.5% 17.3% 32.7% 38.4% 46.2%

*percentages represent the probability that a URL is “permanently” clean after x days with the specified level of reporting.

As you can see, sending a full report substantially improved the likelihood that an infected URL would be cleaned up. Full reports were also observed to be significantly more effective than minimal and no reports on every single day that I followed up on a URL.

But what does this all mean? It means that sending a detailed badware report appears to be an effective measure for getting a badware URL cleaned up. Furthermore, providing more details seemed to be helpful to the site owners and abuse teams who had the ability to clean up the badware.

We’re currently working on ascertaining whether other forms of notification sent in the same time frame (e.g., malware notifications from Google Webmaster Tools) could have prompted some of the badware URL clean-up we observed. Tyler Moore and I are in the process of writing an academic paper with the complete methodology and full results of this study; the paper will be published later this year.

*For examples of minimal reports and additional information, please see pages B-2 to B-4 of StopBadware’s reporting best practices.

New best practices for reporting badware URLs

Posted on October 7, 2011 - 10:25 by ccondon

We’re happy to announce today the public release of our second best practices document: Best Practices for Reporting Badware URLs. These best practices lay out steps that individuals and organizations can follow to effectively report badware URLs to the parties best able to address them. 

The seeds for our new set of best practices were sown during the development of our Best Practices for Web Hosting Providers earlier this year. A common question during our Web Hosting Working Group’s tenure was, “What about best practices for reporting?” The reasoning behind the question was simple: after spending several months determining the most responsible and effective ways for web hosting providers to respond to badware reports, it seemed eminently sensible to develop a complementary set of best practices for reporters so as to shape a clear path all the way from badware detection to resolution. 

After a summer full of discussion with another brilliant and distinguished cross-industry working group, we have a new set of Practices that we feel we can confidently say are “best” when it comes to reporting badware URLs. Our best practices divide the reporting process into four main stages: determining report targets, identifying contact information, assembling contents, and delivering the report. Best practices are laid out for each stage, along with specific steps for report escalation should initial reports fail to receive a satisfactory response. It’s our intention and our hope that the final document will promote reporting in a way that’s useful to hosting providers and other report recipients while offering reporters both clear instruction and flexibility.

Last month, we wrote that “a full 67% of the URLs we reported [in accordance with the reporting best practices] were cleaned up, many within a short time.” Happily, that statistic is holding steady as our sample size (and our reporting experience) increases; moreover, when the report recipient acknowledged receipt of the report, the badware URL cleanup rate jumped to 75%. Our Best Practices for Reporting Badware URLs take a different approach than the Best Practices for Web Hosting Providers, but the goal, of course, is the same: to get badware URLs noticed, acknowledged, and either cleaned up or taken down—quickly and responsibly.

You can download a copy of StopBadware’s Best Practices for Reporting Badware URLs here. Today’s press release is available here.

Last chance to submit feedback on new best practices for reporting badware URLs

Posted on September 12, 2011 - 10:08 by ccondon

Several weeks ago, we put out a request for comments to anyone who might have feedback on our newly developed best practices for reporting badware URLs. As a reminder, we’ll be accepting comments on the new best practices until  this Friday, September 16.

As we mentioned previously, one of the catalysts for our developing these best practices for reporting was feedback we received while creating the first installment of our Best Practices for Web Hosting Providers. We’re extremely interested in any comments web hosting providers may have on these new best practices. Please see our previous blog post on this topic for a list of specific questions about which we’d like feedback.

You can see for yourself the results of our first shot at reporting badware URLs according to these best practices. Given the success of these initial attempts, we’re excited to hear comments on the reporting best practices and integrate them so as to make the final document as complete and effective as possible. Comments can be submitted here or sent to contact/at/stopbadware/dot/org.

Download the draft as a Word doc (.docx) or as a PDF.