The Ur-Quan Masters Discussion Forum

The Ur-Quan Masters Re-Release => Technical Issues => Topic started by: kmunoz on July 04, 2006, 05:09:34 pm



Title: Unzipping .uqm files question
Post by: kmunoz on July 04, 2006, 05:09:34 pm
I'm on a Mac and when I unzip the .uqm files they come out as folders, rather than .uqm files. Am I doing something wrong? Should I be able to put the folders into the package or do I need to somehow get the unzip process to avoid unzipping the entire contents?


Title: Re: Unzipping .uqm files question
Post by: Anthony on July 04, 2006, 05:55:58 pm
I was under the impression that .uqm was simply the .zip format renamed.  Try changing the entension to .zip and try to open it with Windows Explorer or Winzip.


Title: Re: Unzipping .uqm files question
Post by: meep-eep on July 04, 2006, 08:11:46 pm
Don't unzip the .uqm files. Yes, they are really .zip files, but the game can read from inside .zip files.
These files actually used to be called .zip at one time, but we renamed them to .uqm because people kept unpacking them for some reason.


Title: Re: Unzipping .uqm files question
Post by: Novus on July 05, 2006, 01:00:59 am
These files actually used to be called .zip at one time, but we renamed them to .uqm because people kept unpacking them for some reason.
... And some stupid software keeps renaming them back, confusing people and installers alike even more. Would it be possible to change the first three bytes of the files from "50 4b 03" to something less easily recognised by over-curious magic byte recognisers (patching src/sc2code/libs/uio/zip/zip.c correspondingly)? If the file doesn't look like a ZIP archive at all, the chances of something deciding to change the extension are much smaller.


Title: Re: Unzipping .uqm files question
Post by: meep-eep on July 05, 2006, 07:18:11 pm
As an open project, the ability for people to access and modify the content files with standard tools is important. We chose a standard format for a reason.
Also, making modifications to the unzip code would damage the general purpose nature of the code.


Title: Re: Unzipping .uqm files question
Post by: Novus on July 05, 2006, 11:09:43 pm
On the one hand, we want to prevent people from unzipping the content files from ignorance (thinking they have to be unpacked), on the other, we want to allow easy access to the insides of the files.

The problem here is that the first requires the packages to not be ZIP files (or any other well-known archive), the second requires them to be ZIP files (or something like that). I'm not quite sure why the release version's content has to be so easily accessible; isn't it enough that a simple tool can convert the archive into a ZIP file and vice versa by overwriting a magic byte or two (there is no particular reason to require UQM to require the magic bytes to be one way or another, so removing the check doesn't remove generality, just an (unnecessary) failsafe)?

As long as the .uqm files are valid ZIP files, some systems will detect them to be ZIP files and some users will see a .zip extension and try to unpack them.

An alternative approach is to try to eliminate the extension changing back to .zip instead. A quick sampling of SourceForge mirrors (using Mozilla 1.7) suggests that .uqm files are simply identified as "application/octet-stream"; i.e. any old file. This means that any identification of the file as ZIP has to take place on the client machine (or that some server under some circumstances transmits another MIME type). However, we can't really affect the software used by the player to download UQM.


Title: Re: Unzipping .uqm files question
Post by: meep-eep on July 06, 2006, 02:15:17 am
Instead of violating the zip specification by changing the first few bytes and patching uqm to accept these files, a better way would be to prepend the file with something. This is how self-extracting archives work -- an extractor program with a zip file attached. (The central index of .a zip file is at the end of the file.)
Zip extracting programs would still be able to handle the files, and also uqm should be able to handle them without modification. And the prepended bytes could even be text, warning the user against unzipping them, and letting the user know where the file should be put.
A much more elegant solution imho.


Title: Re: Unzipping .uqm files question
Post by: Novus on July 06, 2006, 01:16:19 pm
Adding something to the start of the file sounds like a good idea (unzip generates a warning but works fine otherwise, and most other software should behave similarly), but I'm not sure if this is sufficient to avoid the problems we've been discussing. At least on my SuSE 10.1 setup, magic byte-based detection fails after adding a single zero byte to the start of the file (file(1) reports it as "data" instead of "Zip archive data" and a version number). Amusingly, using a 1 byte instead causes file to report a "mc68k COFF executable". With the magic bytes moved away and the extension changed, processing the file as a ZIP works fine but you have to explicitly tell every unpacker with multiple format support that it's a ZIP file (or change the extension), just like with an SFX archive.

It would be nice if we could figure out what browser/server combination is causing all this trouble to begin with; we don't know what detection mechanism is being used, really; but concatenating one byte to the release version of the file shouldn't be much of a problem so it might be worth a shot anyway.


Title: Re: Unzipping .uqm files question
Post by: fossil on August 08, 2006, 12:37:37 am
Adding something to the start of the file sounds like a good idea (unzip generates a warning but works fine otherwise, and most other software should behave similarly), but I'm not sure if this is sufficient to avoid the problems we've been discussing.

I think it is sufficient. See more below. I was going to suggest the exact same thing as I was reading this thread, warning text included, but meep-eep already got the same idea  ;)

It would be nice if we could figure out what browser/server combination is causing all this trouble to begin with; we don't know what detection mechanism is being used, really;

We do, mostly. The major foe is Internet Explorer and perhaps browsers that copy its behavior. The trouble comes from IE's data-sniffing (yeah, it's a technical term ;) and you can even look it up on http://msdn.microsoft.com), which tries detecting the type of file being transfered by looking at the first N data bytes. The Content-Type sent with the HTTP response body normally overrides the results of data-sniffing, except when Content-Type is application/octet-stream, so that is what happens. Of course, the end result is prompting the user to save the file with a .zip extension, which is actually a bug on MS's part -- they simply do not check the actual file extension before invoking the Save As dialog, and the Save As dialog gets the default extension as .zip since the file is detected as a zip file. Uhh... nice chain of events, eh?
Do not blame MS, however, for trying to combat lame web site admins who do not supply the correct MIME types in the first place.