(Not sure exactly what WinRAR does, but generally speaking for a compressor..) Backwards compatibility.
Let's say you're putting together WinRAR 4.5 and have a fancy new compression method you want to implement. .RAR files generated with Version 4.5 that use it, won't be compatible with earlier versions.
The way around this is to embed, in the .RAR file itself, a program (written in the WinRAR VM) to decompress the data using the new method. WinRAR 4.0 can just run this embedded program without knowing the precise details of the Version 4.5 algorithm.
So you get some of the advantages of a self-executable archive without the portability and security issues.
I think that WinRAR in particular implements a lot of pre-filters, which are things you run over the data before your main compression algorithm in order to make the data more compressible. This seems like a good use for such a VM - they're simple, less-speed-critical than your core compressor, and you can always write more of them.
Why WinRAR team didn't add digital signing to the code for that VMs? Old WinRAR versions would contain VM interpreter, public RSA key and signature verification code; new WinRAR versions would embed already signed blobs of VM code into archive files; private RSA key would stay on WinRAR developers' computers only. And no one would be able to execute arbitrary VM code on end user PC.
What exactly is the drawback of executing arbitrary VM code on an end user PC? If there aren't security flaws in the VM, then the code is sandboxed, and worst case you extract a really big file or hang WinRAR.
Now, if there's a security flaw, that's a different story. But it looks like the VM just gets to play around with memory and registers and doesn't get any libraries or IO, and doesn't rely on type safety for correctness -- which eliminates the typical sources of security holes in more complex VMs such as JVM. And if you don't need great performance, then you can put bounds checks everywhere.
Let's say you're putting together WinRAR 4.5 and have a fancy new compression method you want to implement. .RAR files generated with Version 4.5 that use it, won't be compatible with earlier versions.
The way around this is to embed, in the .RAR file itself, a program (written in the WinRAR VM) to decompress the data using the new method. WinRAR 4.0 can just run this embedded program without knowing the precise details of the Version 4.5 algorithm.
So you get some of the advantages of a self-executable archive without the portability and security issues.
I think that WinRAR in particular implements a lot of pre-filters, which are things you run over the data before your main compression algorithm in order to make the data more compressible. This seems like a good use for such a VM - they're simple, less-speed-critical than your core compressor, and you can always write more of them.
Examples of prefilters are BCJ2 (http://en.wikipedia.org/wiki/7z#Pre-processing_filters) and the PNG image filters (http://www.w3.org/TR/PNG-Filters.html).