|
Pages: [1]
|
 |
|
Author
|
Topic: Can't we glob the content? (Read 3116 times)
|
Electric Monk
Guest
|
One thing that I see as being a slight problem is the number of files in the content directory - both of the source and the game. Nearly 14000 files is just a touch excessive, isn't it?
One solution, which is fairly simple to implement, is to store the files in a zip file (or collection of zip files) and use something like the PhysFS library to access them in-game. This not only saves time when moving things around, etc. but also takes up considerably less space on most file systems. On my NTFS partition a zip with all of the content in it uncompressed takes up just over 143MB, while the files themselves take up over 182MB of disk space. That's almost 40MB of unnecessary slack space.
PhysFS is free, open and LGPL'd. It's reasonably simple to drop into an app, and supports files in ZIP, Build Engine GRP, Quake 1/2 PAK and native (un-globbed) filesystem. Adding another format, should you need it, seems to be relatively simple.
Find it at: http://www.icculus.org/physfs/
NB: Yes, I know, this is probably something that'll have to wait until later. Still, it''d be nice to see it in Alpha 0.3 
|
|
|
|
|
Logged
|
|
|
|
Novus
Enlightened
 Offline
Gender: 
Posts: 1938
Fot or not?
|
At this stage of development, keeping the content in lots of small, standard files is a great help to developers. First of all, you can edit most of the files just by loading them directly into a suitable editor. If you put the files in a big archive, you can't edit them without unpacking them first.
Another more severe problem pertains to CVS. If you put everything in one big ZIP archive or similar, CVS will have to download the entire archive even if you've just changed one image. With everything in separate files, CVS only has to download a one small file for every changed picture.
|
|
|
|
|
Logged
|
|
|
|
Nic.
Guest
|
This is actively being worked on, and has been on Meep-Eep's plate for quite a while. I'm guessing that the bulk of the holdup surrounds the fact that the dialogue, music, etc. all demand that the files be seekable, which pipes generally aren't. Getting around that limitation is likely very non-trivial, but I'll try not to stuff too many words in the dev team's mouths..
Rest assured, by the 1.0 release, it will be much leaner and meaner. Maybe even before that.. 
|
|
|
|
|
Logged
|
|
|
|
Novus
Enlightened
Offline
Gender:
Posts: 1938
Fot or not?
|
I'm guessing that the bulk of the holdup surrounds the fact that the dialogue, music, etc. all demand that the files be seekable, which pipes generally aren't. Getting around that limitation is likely very non-trivial, but I'll try not to stuff too many words in the dev team's mouths..
One quick (but crude) way of implementing seeking is to buffer the entire file in memory. As the only really large files being handled here are the music and a few long speech files, I don't think this should increase RAM use by more than 4 or 5 MB (a drop in the ocean, considering the amount of RAM most people have). Alternatively, you could split the files into suitably sized chunks and compress the chunks separately. Hacking this into an existing library could be nasty, though. PhysFS appears to have a seek function, but I have no idea how it's implemented on e.g. ZIP archives.
A possible compromise would be to put all the small files (speech, images, sound effects) in a big archive or three and leave the music in separate files.
|
|
|
|
|
Logged
|
|
|
|
|
|
Electric Monk
Guest
|
At this stage of development, keeping the content in lots of small, standard files is a great help to developers. First of all, you can edit most of the files just by loading them directly into a suitable editor. If you put the files in a big archive, you can't edit them without unpacking them first.
For the developers, this is fine. For patching, likewise. For those who, for the most part, just want to play the game... it's an inconvenience. Using PhysFS though, you don't need to worry about whether the content is stored in a zip or not. In fact depending on which order you initialize the PhysFS path, you can override the files in the zip with stuff in the filesystem, then update the zip at your leisure.
Works for updating too. Add PhysFS (and maybe InfoZip) to the updater, have it check for and download changed content, then optionally recreate the zip and remove the downloads.
I've got the source compiling now (updating from CVS as we speak). I'll have to have a browse and see how big a job it is to convert it over to PhysFS. Since the PhysFS stuff is similar to the stdio functions, it shouldn't be a major problem... just time consuming.
|
|
|
|
|
Logged
|
|
|
|
|
|
|
|
Shawn Walker
Guest
|
" PhysFS leaves a lot to be desired."
PhysFS from what I've seen does everything quite well. If it doesn't, sending an email message to the physfs mailing list outlining specifically what is "wrong" or "missing" would be a lot more helpful then ignoring it.
|
|
|
|
|
Logged
|
|
|
|
|
|
|
Pages: [1]
|
|
|
|
|
