PyPack problem

Announcements and discussion regarding any projects related to Cyan Worlds' Plasma Engine including (but not limited to) CyanWorlds.com Engine, Drizzle, OfflineKI, PyPRP, and libHSPlasma.

PyPack problem

Postby diafero » Fri Feb 13, 2009 12:51 pm

I've got a problem with the PyPack application: While writing a script which unpacks and decompyiles pak files, everything worked fine with the pak files I used for testing it, but then I startet using others and the decompyiler threw errors. I elaborated a bit more, and obviously files which were generated by others and distributed in ages work, while those I packed myself using UruPython don't. I then unpacked the same pak file with UruPython and with PyPack and compared the checksums - they are different. The UruPython output can be decompyled, so it obviously is correct, while the PyPack output is faulty. Example files can be found in my Offline KI. I use the latest libPlasma SVN and did a complete "make clean && make -j3".

EDIT Using a HexEditor, I looked at the differences. The PyPack version of the file has 0x00 for Bytes 5 to 9 (the 2nd 32 bit value), while UruPython writes 0x36 0x8f 0x96 0x49. In addition, the UruPython version of the unpacked file is 4 Byte longer than the PyPack version (the 4 additional Bytes differ from file to file). After manually fixing these, the PyPack version can be successfully decompyled.
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: PyPack problem

Postby diafero » Sat Feb 14, 2009 3:11 am

I managed to write a patch which makes PyPack correctly decompyle the files created by UruPython:
Code: Select all
diff --git a/Tools/src/PyPack.cpp b/Tools/src/PyPack.cpp
index 148a235..67324e7 100644
--- a/Tools/src/PyPack.cpp
+++ b/Tools/src/PyPack.cpp
@@ -276,7 +276,8 @@ int main(int argc, char** argv) {
         }
         for (size_t i=0; i<pakObjects.getSize(); i++) {
             IS->seek(pakObjects[i].fOffset);
-            pakObjects[i].fSize = IS->readInt() - 4;
+            pakObjects[i].fSize = IS->readInt();
+            if (IS->pos() + pakObjects[i].fSize > IS->size()) pakObjects[i].fSize -= 4;
             pakObjects[i].fData = new hsUbyte[pakObjects[i].fSize];
             IS->read(pakObjects[i].fSize, pakObjects[i].fData);


That ugly if clause is necessary for files which are not created by UruPython to also be correctly read... I could not find a better way around that, even though UruPython can unpack both "types" of files and does not have any special treatment.
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: PyPack problem

Postby Zrax » Fri Feb 20, 2009 11:40 am

Sorry for not responding sooner; I've been really busy these past couple of weeks. I think the solution is even simpler than that, since I believe I was mis-copying my code from PrpShop for creating pak files. The ones created before were probably invalid, so go ahead and try the new version and see if it fixes things. Thanks
User avatar
Zrax
 
Posts: 206
Joined: Fri Sep 28, 2007 5:19 pm
Location: Waist-deep in a conecano

Re: PyPack problem

Postby diafero » Fri Feb 20, 2009 11:55 am

It now reads the correct pak files fine, but the invalid ones (and there seems to be a bunch of them) now give that error
Code: Select all
terminate called after throwing an instance of 'hsFileReadException'
  what():  Error reading file: Read past end of buffer
which is precisely what I wanted to prevent with my (ugly) hack. I don't expect all of these ages to be updated, so that there will be many pak files Uru can read, but PyPack can not (like PaahkwehNew.pak, Pahts.pak or Bahvahnin.pak, just to mention a few).
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: PyPack problem

Postby Lontahv » Fri Feb 20, 2009 9:38 pm

You could just add a "try" and "catch" to the read section and then just give a warning rather than raising an exception.

EDIT: I haven't been following this thread so this fix could be way off. :P
Currently getting some ink on my hands over at the Guild Of Ink-Makers (PyPRP2).
User avatar
Lontahv
Councilor of Artistic Direction
 
Posts: 1331
Joined: Wed Oct 03, 2007 2:09 pm

Re: PyPack problem

Postby diafero » Sat Feb 21, 2009 1:32 am

It will of course prevent the application from crash, but it will also result in the last of the packed python files to to be unpacked, so that's not what we want (I really suggest reading the thread, it's not that long and I don't want to repeat everything). I really see no better solution than what I posted above, but I would be glad if someone did :)

EDIT: Btw, the last commit breaks libPlasma for me on Linux (it doesn't compile any more, not even with a "make clean && make"), I have to revert the changes to the PyOccluder file to fix it
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: PyPack problem

Postby Zrax » Thu Mar 05, 2009 11:57 am

Again, sorry it took a while to get to this -- I've been pretty busy at work and school. I just decided to use your patch, except I modified it a bit to be more adaptive, and to print out a warning when it has to truncate a file.
User avatar
Zrax
 
Posts: 206
Joined: Fri Sep 28, 2007 5:19 pm
Location: Waist-deep in a conecano

Re: PyPack problem

Postby diafero » Thu Mar 05, 2009 1:08 pm

Ok, so I can finally use trunk again without custom patches, thanks :)
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany


Return to Plasma Development

Who is online

Users browsing this forum: No registered users and 0 guests

cron