The August report from AV-comparatives was released. But be careful when reading or using these statistics:
We are rather tired of repeating that VirusTotal was not designed as a tool to perform AV comparative analyses, but as a tool that checks suspicious samples with several AV programs and helps AV labs by forwarding them the malware they failed to detect. Those who use VirusTotal to perform AV comparative analyses should know that they are making many implicit errors in the methodology, the most obvious being:
- VirusTotal AV engines are commandline versions, so depending on the product, they will not behave quite like the desktop versions: for instance, in such cases when desktop solutions use techniques based on behavioral analysis and count on personal firewalls that may decrease entry points and mitigate propagation, etc.
- In VirusTotal desktop-oriented solutions coexist with perimeter-oriented solutions; heuristics in this latter group may be more aggressive and paranoid, since impact of false positives is less visible in the perimeter. It is simply not fair to compare both groups.
There is also the independant
AV-Test.org Testlab with their august reviews:
Detection AV database size
1. AVK 2007 99,88% 22,4 MB
2. WebWasher 99,86% 22,0 MB
3. BitDefender 99,51% 12,3 MB
4. AntiVir 99,29% 19,4 MB
5. Kaspersky 98,86% 14,3 MB
6. F-Secure 97,93% 15,3 MB
7. Avast! 96,99% 11,0 MB
8. AVG 96,81% 32,7 MB
9. Symantec 96,75% 40,2 MB
10. Microsoft 96,42% 22,5 MB
11. Ikarus 95,92% 11,9 MB
12. Sophos 94,63% 13,7 MB
13. Nod32 94,26% 8,7 MB
14. Fortinet 94,20% 51,9 MB
15. McAfee 93,71% 18,2 MB
16. Dr Web 92,48% 8,9 MB
17. Rising 90,43% 29,7 MB
18. Panda 90,15% 34,6 MB
19. TrendMicro 88,85% 67,2 MB
20. VBA32 88,59% 33,8 MB
21. F-Prot 87,03% 20,9 MB
22. Norman 86,05% 24,7 MB
23. Command 82,57% 15,2 MB
24. VirusBuster80,49% 19,2 MB
25. QuickHeal 79,02% 22,9 MB
26. ClamAV 78,66% 11,8 MB
27. eTrust-VET 78,25% 13,8 MB
28. Ewido 74,91% 11,9 MB
29. eSafe 73,61% 91,4 MB
Please be advised that AVK uses the engines of Kaspersky AND Avast, and Webwasher is also a gatewayproduct with strong (and strict) heuristics. It's always advised to use a product with strict heuristics on the gateway level (and possibly using two engines) and use another vendor for the desktop anti-virus scanning. It is also to be noted that Microsoft has come from 80% compared to
previous results.
Still, you have to be careful when
Interpreting anti-malware reviews. Depending on the methodology and sample set, you can get very different results. To make your head spin even more, here are some
statistics from samples captured by honeypots (Shadowserver). If you want to do your own comparative tests, the
AVIEN Malware Defense Guide discusses some do's and don'ts.