International group pushing for anti-malware testing standard

The first-of-its-kind standards are being planned in response to fast-changing malware that cannot necessarily be stopped with traditional signatures, Mark Kennedy, a Symantec distinguished engineer and member of the new organization, The Anti-Malware Testing Standards Organization (AMTSO), told SCMagazineUS.com today.

As a result, many anti-malware products with proactive capabilities have emerged, but almost all of the world's approximately 80 testing firms cannot evaluate them based on their new functionality, such as behavior- or heuristics-based functionality, Kennedy said.

Instead the tests only conduct static file scanning of malware samples, he said.

“Internally, we had been concerned that not all the technologies that make up our products were being adequately represented in testing,” Kennedy said. “If you're just going to put a file on the machine and scan it [for malware], it's no longer really indicative of whether a customer running a security suite who is exposed to a threat will be infected.”

The 21 companies making up the AMTSO – which includes such well-known testing firms as AV-Comparatives and Virus Bulletin – convened in Spain on Jan. 21 and 22. This week, officials announced that the organization established a charter.

The charter includes goals such as offering a forum for discussion, promoting education and awareness of anti-malware testing issues, providing tools and resources to help standards-based testing methodologies and creating standards.

The standards, which are expected to be approved during the next scheduled meeting in April, will require dynamic testing, Kennedy said.

“That means you have a machine running a piece of security software, and you go out and actively attempt to infect it and then determine if that infection has occurred,” he said.

The organisation will be charged with raising public awareness on three fronts: Convincing the testing firms the new standards are worth it, persuading customers – such as magazines – to pay higher fees for the new, more comprehensive evaluations, and assuring consumers that they are making a more informed decision by choosing products that underwent the tests, Kennedy said.

To this end, AMTSO plans to invite academic leaders and consumer advocates to support the new standard.

Matt Williamson, principal scientist at Sana Security, told SCMagazineUS.com today that his company's unique approach to blocking malware prevents its products from being analysed in tests.

“I don't believe the current tests are a good measure of overall efficacy," Williamson said. “Testing on a large, historical sample doesn't make a lot of sense. The current tests aren't a good measure of how the products will work in the wild.”

End-users may be the biggest winners, he added.

“I think it will be good for the consumer because the test will be standardised and will be more meaningful, which will allow them to make better comparisons between products and technologies,” Williamson said.