Copyright (C) 1994 Luca Sambucci. All rights reserved.
Since some years a program called "Virus Simulator" is being distributed as
shareware software from Rosenthal Engeneering. This program claims to generate
"simulated viruses" to allow the users to do their own tests on antivirus
products without any fear to have their computer infected.
A lot of antivirus researchers find the program absolutely useless (and some think it could be harmful, too), because the "simulation files" created by the program aren't viruses at all, and many AV products will not mark them as infected.
In my opinion, the results of an AV-test done with the "simulated-viruses" collection is simply misleading and, in some cases, even harmful.
Why misleading?
The files created by the shareware version of the program (the latest one is the 2.c version, the .DOC file is dated 4 Apr 93, the executable file is dated 6 Aug 91) aren't viruses at all: they cannot "infect" anything, they're only "parts" of viruses. Not all AV programs will detect them as viruses, so the detection rates can be very different compared with the rates of a test on real viruses.
Why harmful?
A user, after doing such test, could think that the "X" AV product, that failed to detected all simulations, isn't good enough. This will cause the user to rely on another AV product ("Y") for the security of her/his computer; this second product performed better on the "simulated-viruses" test, but perhaps its detection rate on real viruses is worse than the first one. In the future the "Y" product could fail to detect a real virus (virus that the "X" product would have detected), and the user's computer will become infected only beacause she/he relied to a worse AV product (but apparently good when tested on the simulations).
Ok, but now, as usual, I'll bring evidences to what I say.
I've created some simulations with the Virus Simulator v.2c program. Then I took from my library (containing only real viruses) the same viruses that the program claimed to have created. The total number of viruses is 70. There are some reasons why I took only 70 viruses:
Name Version Date Producer ---- ------- ---- -------- AntiVir IV (AVScan) 1.64 08/03/94 H+BEDV GmbH AV Toolkit Pro (-V) 2.00e 07/13/94 KAMI Ltd. AVTK (Findviru) 6.64 05/11/94 S&S Int. Plc. F-Prot 2.13a 07/27/94 Frisk Soft. Int. IBM Antivirus/DOS 1.06 07/11/94 IBM Corp. Integrity Master 2.22a 05/25/94 Stiller Research Sweep 2.64 08/01/94 Sophos Plc. TBAV (TbScan) 6.22 07/11/94 ESaSS B.V. Virex PC (VPCScan) 2.94 07/05/94 Datawatch Corp. VirusScan 2.1.0 07/18/94 McAfee Inc.You can find more information in the TESTINFO.ZIP archive, available at all our distribution sites.
And here the results of the two tests.
First of all, I tested the products like the users that use Mr. Rosenthal's program would do: with the "simulated viruses."
Antivirus percent of simulations product detected as infected --------- ---------------------- AVScan 1.64 98 % AVP 2.00e 0 % Findviru 6.6 0 % F-Prot 2.13a 71 % IBMAV 1.06 55 % I-Master 2.22a 100 % Sweep 2.64 60 % TbScan 6.22 42 % VPCScan 2.94 100 % VirusScan 2.1.0 45 %Note: although the final report of F-Prot stated that there were infected files, the message I received was always "Destroyed by the VCL virus."
So, which one performed better? Which one seems to be an "unreliable" antivirus? Here's the final list (1 = better, 8 = worse):
Antivirus percent of infected files product correctly detected as infected --------- ------------------------------ AVScan 1.64 100 % AVP 2.00e 100 % Findviru 6.6 100 % F-Prot 2.13a 100 % IBMAV 1.06 100 % I-Master 2.22a 99 % Sweep 2.64 100 % TbScan 6.22 100 % VPCScan 2.94 99 % VirusScan 2.1.0 96 %Yes, all AV products had an excellent score. This because all the viruses, as I already stated before, are very old, and (almost) all AV programs should now detect them.
I don't waste time typing the list: you all have seen that the results are "a little bit" different than the ones showed above. Almost all AV products with a bad performance on the "simulated-viruses" test, had a very good performance in the real test.
Important:
The aim of this test isn't showing which AV is better, but how the files created by Mr. Rosenthal's "Virus Simulator" aren't suitable for AV-testing. Seventy viruses aren't enough for a real AV test. If you wish to see which AV performed better in our tests, please refer to the General Antivirus Test.
Anti-flame:
I showed with facts that Mr. Rosenthal's "Virus Simulator" isn't very good at all to test AV software. If someone likes to reply to this test, I ask she/he to bring facts, not only words.
Last thing. Let's think for a moment that the "Virus Simulator" creates real viruses that can be used for testing purposes, let's think that Mr. Rosenthal is right and that his program is suitable for all users who wish to test their AV programs. There will be another false step: all AV products will have a 99-100% score! The viruses (simulations) generated by the program are too old to be used for testing purposes! Almost all AV products will detect them. How will the user be able to choose between a program that detects all viruses and another one that detects the whole collection too??
Right. Here's the test-set of the viruses used for this test (to allow other researchers to check my results) according to the CARO naming standard:
Alabama.A; Ambulance.A; Amoeba.A; Armagedon.1079.A; Black_Monday.1055.A; Carioca.A; Cascade.1704.C; Cascade.1704.Formiche; Dark_Avenger.1800.A; Darth_Vader.344.A; DataLock.920.A; Devil's_Dance.A; Diamond.1024.A; Diamond.Greemlin; Diamond.Lucifer; Dir.691; Flip.2153.A; Friday_13.416.A; Frodo.Frodo.A; Frodo.Fish6.A; Guppy.A; Halloechen.A; Hymn.Hymn.A; ItaVir; Jerusalem.1808.A; Jerusalem.AntiCAD.2900.Plastique.A; Jerusalem.AntiCAD.4096.A; Jerusalem.Fu_Manchu.B; Jerusalem.Solano.Subliminal.A; Jerusalem.Sunday.A; Jerusalem.Sunday_II.A; July13th.1201; June16th; Keypress.1232.A; Kukac.Turbo; Lehigh; Leprosy.Plague; Liberty.2857.A; Little_Pieces; Murphy.1480.A; Nomenklatura.A; Ontario.512.A; Paris; Pixel.847.Pixel; Raubkopie; Russian_Mirror.A; Staf; Star_Dot.600; Star_Dot.801; Suomi; Suriv.1_01.April_1st.A; SVC.3103.A; Sylvia.1332.A; SysLock.Macho.A; Tenbytes.1554.A; Tequila; Tiny_family.133; Traceback.2930; Vacsina.TP-06; Vacsina.TP-23; Vacsina.TP-24; Vcomm.637.A; Vienna.Stone-90; VirDem.1336.German.A; Voronezh.1600; Whale.00; Wolfman.A; XA1; Yankee_Doodle.TP-44.A; Zero_Bug.A
Note: I haven't tested the other two features of the "Virus Simulator" (boot viruses simulations and TSR signatures), although I don't think there are considerable differences. If there's enough request I'll do it.