Testing bodies AV‐Comparatives, AV‐TEST and Virus Bulletin comment on
allegations of inappropriate behavior
Today, three of the world’s most renowned and trusted security testing bodies, AV‐Comparatives, AV‐
TEST and Virus Bulletin, stand united to censure security vendor Qihoo 360 after finding the firm
submitted products for comparative and certification testing which behaved significantly differently
from those made available to its users and customers. The three testing bodies will revoke all
certifications and rankings awarded to the company’s products so far this year, and going forward will
insist on more open and fair dealings to ensure users are provided with the most accurate information
possible.
Investigations by the three labs found that all products submitted for testing by Qihoo had one of the
product’s four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s
own QVM engine, was never enabled. This included versions posted to ostensibly public sections of the
company’s websites.
By contrast, as far as can be determined, all versions made generally available to users in Qihoo’s main
market regions had the Bitdefender engine disabled and the QVM engine active. According to all test
data this would provide a considerably lower level of protection and a higher likelihood of false
positives. Options are provided in the product to adjust these settings, but as the majority of users leave
settings unchanged, most tests insist on using the default product settings to best represent real‐world
usage.
As part of the investigation into Qihoo 360, counter‐accusations were levelled by the company against
two fellow Chinese security firms, Baidu and Tencent. Analysis of products submitted for testing by
these companies turned up some unexpected flags within their products, marked with the names of
several test labs and implying some difference in product behavior depending on the environment they
were run in – similar flags were also found in Qihoo products. However, no evidence could be found that
this gave any significant advantage to either product, and in some cases it even seemed to put them at a
disadvantage. Both firms were able to provide good reasons for including these flags in their products.
On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings
had been adjusted for testing, including enabling detection of types of files such as keygens and cracked
software, and directing cloud lookups to servers located closer to the test labs. After several requests for
specific information on the use of third‐party engines, it was eventually confirmed that the engine
configuration submitted for testing differed from that available by default to users.
Qihoo’s awards and certifications attained since the start of 2015 will thus be stricken from the records
by all three testing bodies, and all three will be imposing stricter demands on test participants to avoid
any further gaming of results by vendors.
“This sort of thing doesn’t really help anyone,” said John Hawes, Chief of Operations at Virus Bulletin
[editorial@virusbtn.com]. “Independent tests serve both users and developers, showing which productsare performing best and highlighting areas where developers need to work harder. If the products being
tested aren’t those being used in the real world, nobody’s getting any useful information.”
Andreas Clementi, CEO of AV‐Comparatives [media@av‐comparatives.org], said: “Independent anti‐
malware testing plays a key role in raising the standard of protection on users’ devices, which in turn
makes the Internet a safer place for everybody. Misuse of such tests for marketing purposes will, in the
long run, result in more successful malware attacks, making Internet users less secure.”
Maik Morgenstern, CEO of AV‐TEST [presse@av‐test.de], said: “Comparative testing and certification
plays an important role in the anti‐malware industry, both for the users and the vendors. Users rely on
independent results to make an educated decision regarding their protection software. If vendors start
to manipulate the testing process, they are hurting everyone involved.”
About AV‐Comparatives
AV‐Comparatives [av‐comparatives.org] is an independent organization offering systematic testing that
checks whether security software, such as PC/Mac‐based anti‐virus products and mobile security
solutions, lives up to its promises. Using one of the largest sample collections worldwide, it creates a
real‐world environment for truly accurate testing. AV‐Comparatives offers freely accessible results to
individuals, news organizations and scientific institutions. Certification by AV‐Comparatives provides an
official seal of approval for software performance which is globally recognized. Currently, AV‐
Comparatives' Real‐World Protection Test is the most comprehensive and complex test available when it
comes to evaluating the real‐life protection capabilities of anti‐virus software. Put simply, the test
framework replicates the scenario of an everyday user in an everyday online environment – the typical
situation that most of us experience when using a computer with an Internet connection. AV‐
Comparatives works closely with several academic institutions, especially the University of Innsbruck’s
Department of Computer Science, to provide innovative scientific testing methods.
About AV‐TEST
AV‐TEST [av‐test.org] GmbH is an independent supplier of services in the fields of IT security and anti‐
virus research, focusing on the detection and analysis of the latest malicious software and its use in
comprehensive comparative testing of security products. Due to the timeliness of the testing data,
malware can instantly be analyzed and categorized, trends within virus development can be detected
early, and IT‐security solutions can be tested and certified. The AV‐TEST Institute’s results provide an
exclusive basis of information, helping vendors to optimize their products, special interest magazines to
publish research data, and end‐users to make good product choices. AV‐TEST has operated out of
Magdeburg (Germany) since 2004 and employs more than 30 team members, professionals with
extensive practical experience. The AV‐TEST laboratories include 300 client and server systems, wheremore than 1,000 terabytes of independently collected test data, containing both malicious and harmless
sample information, are stored and processed.
About Virus Bulletin
Virus Bulletin [virusbtn.com] is an online security information portal and certification body providing
users with independent intelligence about the latest developments in the threat landscape, as well as
conducting bimonthly certifications of anti‐malware and anti‐spam products. Both its VB100 anti‐
malware tests and VBSpam spam filter tests are well recognised and highly respected in their fields.
Virus Bulletin also organises the VB Conference, an annual event at which the brains of IT security from
around the world gather to learn, debate, pass on their knowledge and move the industry forward – the
25th Virus Bulletin Conference will take place in Prague 30 September to 2 October 2015. Virus Bulletin is
supported by an Advisory Board comprising some of the world's leading anti‐threat experts.
然后再高调退出测评，果然。。。。