The Ur-Quan Masters Home Page Welcome, Guest. Please login or register.
Did you miss your activation email?
November 03, 2024, 09:52:17 am
Home Help Search Login Register
News: Celebrating 30 years of Star Control 2 - The Ur-Quan Masters

+  The Ur-Quan Masters Discussion Forum
|-+  The Ur-Quan Masters Re-Release
| |-+  Technical Issues (Moderator: Death 999)
| | |-+  Can't we glob the content?
« previous next »
Pages: [1] Print
Author Topic: Can't we glob the content?  (Read 3183 times)
Electric Monk
Guest


Email
Can't we glob the content?
« on: March 02, 2003, 12:03:23 pm »

One thing that I see as being a slight problem is the number of files in the content directory - both of the source and the game.  Nearly 14000 files is just a touch excessive, isn't it?

One solution, which is fairly simple to implement, is to store the files in a zip file (or collection of zip files) and use something like the PhysFS library to access them in-game.  This not only saves time when moving things around, etc. but also takes up considerably less space on most file systems.  On my NTFS partition a zip with all of the content in it uncompressed takes up just over 143MB, while the files themselves take up over 182MB of disk space.  That's almost 40MB of unnecessary slack space.

PhysFS is free, open and LGPL'd.  It's reasonably simple to drop into an app, and supports files in ZIP, Build Engine GRP, Quake 1/2 PAK and native (un-globbed) filesystem.  Adding another format, should you need it, seems to be relatively simple.

Find it at: http://www.icculus.org/physfs/

NB: Yes, I know, this is probably something that'll have to wait until later.  Still, it''d be nice to see it in Alpha 0.3  Cool
Logged
Novus
Enlightened
*****
Offline Offline

Gender: Male
Posts: 1938


Fot or not?


View Profile
Re: Can't we glob the content?
« Reply #1 on: March 02, 2003, 07:50:59 pm »

At this stage of development, keeping the content in lots of small, standard files is a great help to developers. First of all, you can edit most of the files just by loading them directly into a suitable editor. If you put the files in a big archive, you can't edit them without unpacking them first.

Another more severe problem pertains to CVS. If you put everything in one big ZIP archive or similar, CVS will have to download the entire archive even if you've just changed one image. With everything in separate files, CVS only has to download a one small file for every changed picture.
Logged

RTFM = Read the fine manual.
RTTFAQ = Read the Ur-Quan Masters Technical FAQ.
Nic.
Guest


Email
Re: Can't we glob the content?
« Reply #2 on: March 02, 2003, 11:54:39 pm »

This is actively being worked on, and has been on Meep-Eep's plate for quite a while.  I'm guessing that the bulk of the holdup surrounds the fact that the dialogue, music, etc. all demand that the files be seekable, which pipes generally aren't.  Getting around that limitation is likely very non-trivial, but I'll try not to stuff too many words in the dev team's mouths..

Rest assured, by the 1.0 release, it will be much leaner and meaner.  Maybe even before that..  Smiley
Logged
Novus
Enlightened
*****
Offline Offline

Gender: Male
Posts: 1938


Fot or not?


View Profile
Re: Can't we glob the content?
« Reply #3 on: March 03, 2003, 04:02:31 pm »

Quote
I'm guessing that the bulk of the holdup surrounds the fact that the dialogue, music, etc. all demand that the files be seekable, which pipes generally aren't.  Getting around that limitation is likely very non-trivial, but I'll try not to stuff too many words in the dev team's mouths..

One quick (but crude) way of implementing seeking is to buffer the entire file in memory. As the only really large files being handled here are the music and a few long speech files, I don't think this should increase RAM use by more than 4 or 5 MB (a drop in the ocean, considering the amount of RAM most people have). Alternatively, you could split the files into suitably sized chunks and compress the chunks separately. Hacking this into an existing library could be nasty, though. PhysFS appears to have a seek function, but I have no idea how it's implemented on e.g. ZIP archives.

A possible compromise would be to put all the small files (speech, images, sound effects) in a big archive or three and leave the music in separate files.
Logged

RTFM = Read the fine manual.
RTTFAQ = Read the Ur-Quan Masters Technical FAQ.
meep-eep
Forum Admin
Enlightened
*****
Offline Offline

Posts: 2847



View Profile
Re: Can't we glob the content?
« Reply #4 on: March 03, 2003, 08:03:38 pm »

Seeking has never been an issue. The issue has been time management for the implementation.
It will likely be there in 0.3
Logged

“When Juffo-Wup is complete
when at last there is no Void, no Non
when the Creators return
then we can finally rest.”
Electric Monk
Guest


Email
Re: Can't we glob the content?
« Reply #5 on: March 05, 2003, 01:10:31 am »

Quote
At this stage of development, keeping the content in lots of small, standard files is a great help to developers. First of all, you can edit most of the files just by loading them directly into a suitable editor. If you put the files in a big archive, you can't edit them without unpacking them first.


For the developers, this is fine.  For patching, likewise.  For those who, for the most part, just want to play the game... it's an inconvenience.  Using PhysFS though, you don't need to worry about whether the content is stored in a zip or not.  In fact depending on which order you initialize the PhysFS path, you can override the files in the zip with stuff in the filesystem, then update the zip at your leisure.

Works for updating too.  Add PhysFS (and maybe InfoZip) to the updater, have it check for and download changed content, then optionally recreate the zip and remove the downloads.

I've got the source compiling now (updating from CVS as we speak).  I'll have to have a browse and see how big a job it is to convert it over to PhysFS.  Since the PhysFS stuff is similar to the stdio functions, it shouldn't be a major problem... just time consuming.
Logged
PhracturedBlue
Frungy champion
**
Offline Offline

Posts: 88


Me developer. Here me roar!


View Profile
Re: Can't we glob the content?
« Reply #6 on: March 07, 2003, 08:20:35 pm »

Quote


I've got the source compiling now (updating from CVS as we speak).  I'll have to have a browse and see how big a job it is to convert it over to PhysFS.  Since the PhysFS stuff is similar to the stdio functions, it shouldn't be a major problem... just time consuming.


Feel free to do this.  Feel free to submit teh patch in bugzilla.  Please DO NOT release those binaries to others.  This has all been done by others before.  We (well Meep-Eep) have a specific goal for the resource management, and know how to do it.  It will eventually happen, but please do not rush things.
Logged
meep-eep
Forum Admin
Enlightened
*****
Offline Offline

Posts: 2847



View Profile
Re: Can't we glob the content?
« Reply #7 on: March 09, 2003, 04:59:38 am »

Quote
Feel free to do this.  Feel free to submit teh patch in bugzilla.

I'd like to say the opposite. Please don't waste your time on this.
PhysFS leaves a lot to be desired. I'm already working on the packaging problem and it should be
ready for the 0.3 version. A patch submitted to bugzilla for this will likelly be ignored.

Logged

“When Juffo-Wup is complete
when at last there is no Void, no Non
when the Creators return
then we can finally rest.”
Shawn Walker
Guest


Email
Re: Can't we glob the content?
« Reply #8 on: April 14, 2003, 06:25:02 am »

" PhysFS leaves a lot to be desired."

PhysFS from what I've seen does everything quite well. If it doesn't, sending an email message to the physfs mailing list outlining specifically what is "wrong" or "missing" would be a lot more helpful then ignoring it.
Logged
meep-eep
Forum Admin
Enlightened
*****
Offline Offline

Posts: 2847



View Profile
Re: Can't we glob the content?
« Reply #9 on: April 14, 2003, 05:16:34 pm »

Quote
" PhysFS leaves a lot to be desired."

PhysFS from what I've seen does everything quite well. If it doesn't, sending an email message to the physfs mailing list outlining specifically what is "wrong" or "missing" would be a lot more helpful then ignoring it.

That would help for a missing feature, but not for a bad design. In particulary, every setting is global! Zziplib is a lot better in that respect, but it misses something like the PhysFS 'search path', and is very slow. These are pretty fundamental things and a mail to their mailing list won't help there.

Logged

“When Juffo-Wup is complete
when at last there is no Void, no Non
when the Creators return
then we can finally rest.”
Pages: [1] Print 
« previous next »
Jump to:  


Login with username, password and session length

Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!