CWE Patching System

[nathan2055]

I do have a test UU Shard in a virtual machine, not sure if that qualifies me as UU admin
Admittedly, the dataserver of an Alcugs/TPOTS Shard works exactly the same as a UU one. However, as D'Lanor pointed out, I do not use the patching functionality. On the DI server, there is a huge directory containing all the prp, age, sdl, ... files for all available ages in their appropriate folder (to make organizing easier, they are not all in one folder, but rather separated in one Uru folder structure for each age). A python script is reading all those files, and doing some magic (involving Java and native C++ apps... yeah, it's quite messy^^), to create a consistent dataserver out of it (so the files end up on the server twice: uncompressed in the dataserver source folder, and compressed available for download). If the client sees that a file is outdated, it just re-downloads it entirely.
I wrote together most of what I know about dataservers at http://alcugs.almlys.org/AlcugsDataserver, but I could never find any documentation or tools for the patching functionality.

Well, there goes my last idea .
To be honest, we need someone who not only uses the UU patching system, but someone who can activate it in MOUL.

Oh, I'm going to change the name of this topic from "Building a MOSS Shard" to "CWE Patching System".

[D'Lanor]

cjkelly1 (from the MOSS dev team) has successfully created UU/Alcugs patches in the past. He doesn't seem to have an account here but you can get hold of him at OpenUru.

[Paradox]

I do have a test UU Shard in a virtual machine, not sure if that qualifies me as UU admin
Admittedly, the dataserver of an Alcugs/TPOTS Shard works exactly the same as a UU one. However, as D'Lanor pointed out, I do not use the patching functionality. On the DI server, there is a huge directory containing all the prp, age, sdl, ... files for all available ages in their appropriate folder (to make organizing easier, they are not all in one folder, but rather separated in one Uru folder structure for each age). A python script is reading all those files, and doing some magic (involving Java and native C++ apps... yeah, it's quite messy^^), to create a consistent dataserver out of it (so the files end up on the server twice: uncompressed in the dataserver source folder, and compressed available for download). If the client sees that a file is outdated, it just re-downloads it entirely.
I wrote together most of what I know about dataservers at http://alcugs.almlys.org/AlcugsDataserver, but I could never find any documentation or tools for the patching functionality.

Well, there goes my last idea .
To be honest, we need someone who not only uses the UU patching system, but someone who can activate it in MOUL.

Oh, I'm going to change the name of this topic from "Building a MOSS Shard" to "CWE Patching System".

Okay, I've asked some people, done a bunch of investigating, and figured out exactly how this worked...

The .pat files were essentially PRP files that stored a BSDiffBuffer of each object that had changed between versions. They included the full plPageInfo header and keyring/index, with special flags indicating that it was a patch and which objects needed updating. Objects that did not change had a DataPosition/DataLength of -1. plPagePatcher would read the .pat file, and for each key in the keyring that needed updating, it would read the plBSDiffBuffer and apply it in memory to the object data. The updated objects (along with a new keyring and new plPageInfo header) would be written out to disk.

Good news: We understand how most of that worked.

Bad news: Cyan removed all the code to read the .pat files, read the patch keys, apply the diffs, and rewrite the page to disk. Also, with the new (version 6) PRP format, plPageInfo no longer has a flags field, so there's no way to indicate that a file is a patch instead of a page. Unless we reimplement all the code that Cyan removed (and break compatibility with MOUL's file format), this is impossible for MOUL.

Cyan changed the PRP format when they moved to version 70 (also when they switched to episodes), where they also removed all the plRegistry, plRegistrySource (plRegistryPackedSource), plRegistryDataSource (plRegistryPackedDataSource), plPagePatcher, and plKeyImpPublic classes/files. I find the old way of doing things much more well designed than the current way in a lot of regards, but that seems true for a lot of things added during the MOUL era

[diafero]

Just out of interest, what did all these plRegistry etc. things do?

[Stucuk]

Bad news: Cyan removed all the code to read the .pat files, read the patch keys, apply the diffs, and rewrite the page to disk. Also, with the new (version 6) PRP format, plPageInfo no longer has a flags field, so there's no way to indicate that a file is a patch instead of a page.

Can't you work out if its a patch based on its filename's extension and store a boolean in the plPageInfo to indicate if its a patch or not?

[Chacal]

This was Cyan's way to do it.
It doesn't strike me as the only possible way, or even the best possible way.
All this processing can, and probably should, be done by a separate process before the client is started.
Also they did it at the application level, but it can be done at the file level like any other diff.

[Paradox]

Just out of interest, what did all these plRegistry etc. things do?

While the plResManager handles all the plKeys, the plRegistry handled all of the PRP files (plRegistryPageNodes). There were functions for verifying that a page was valid, adding/removing pages from the registry, and opening streams to the index or data sections of a page (some of this is a result of when the index and data were separate files, rather than a single "packed" file).

Can't you work out if its a patch based on its filename's extension and store a boolean in the plPageInfo to indicate if its a patch or not?

That might be possible, but the flags also indicated whether the page header was updated and whether it was a partial patch.

The advantage (and elegance) to Cyan's method of patching is that it reused all of the existing registry code, and all of the object patching was done with plRegistryKeyIterators.

I've got almost all the old code (missing one important function in plPagePatcher), and I can get plKeyImpPublic and the iterators implemented easily... I'm not sure what to do about the rest of plPagePatcher. It can probably be hacked to work with whatever Cyan is doing now, but part of me still wants to revert Cyan's removal of the plRegistry stuff. >.>

[Stucuk]

That might be possible, but the flags also indicated whether the page header was updated and whether it was a partial patch.

A solution around that would be to create your own class and put it on the end of the classlist. You could have it as a class that stores the flags and anything else that may be needed. If that class exists in the PRP then its a patch. That way you don't need to change any of the existing PRP "format" but you still can store the same information.

[JWPlatt]

It might be possible to eventually (spare time and all that - not even "soon") get these plRegistry* sources to reimplement it if there is interest and you don't just go ahead and do it youselves first anyway.

[Paradox]

It might be possible to eventually (spare time and all that - not even "soon") get these plRegistry* sources to reimplement it if there is interest and you don't just go ahead and do it youselves first anyway.

Well, I'd definitely love to see the original sources with all their comments... but that's mostly just for interest's sake.

Re-implementing the necessary portions of the classes is fairly easy (and I've already got plKeyImpPublic and the iterators done). My worry about re-implementing plRegistry is that there might have been a good technical reason for the change, that we aren't aware of. I've emailed Mark regarding that, but given the low priority I'm not expecting a speedy response:

Email to Mark Show Spoiler

Hi Mark,

I realise you're probably busy with Android work and other projects (and hopefully a patch contribution system for MOUL); so I'll mention right away that this is a low priority email.

One of the discussions that had come up recently on the Guild of Writers forum was about hosting shards and customizing existing content. The core of the issue is that there's no licence to redistribute Cyan's content, and no easy way to provide patches that could upgrade existing files on the player's computer. That got me thinking about how patches were handled in UU, and I remembered some sort of binary diff patches. A bit of investigating in old UU executables, and I found what I was looking for: plPagePatcher.

However, almost all of that code was removed part way through MOUL. I'm referring to the old plRegistry, plRegistryPackedSource, plPagePatcher, et al. classes that had been used to handle all the PRP files. Most of the functionality looks like it was merged into plResManager and plRegistryPageNode, but I'm curious to know the reasons behind removing it in the first place.

I know some of it was a hold-over from the days of separate PRX/PRM files, which eventually merged into PRP files. There were also some efficiency problems back in the original Path of the Shell codebase, where object lookups were done using string comparisons. Colin fixed that for the Myst V codebase by introducing unique object IDs, which he backported to Plasma 2.0. That seemed to work until the PRP/plPageInfo format was changed in May 2007 (which means the implementation was probably done in January/February, and it seems likely that Bob was responsible for coding it). Do you know what the reasons for this change were? It definitely brings the 2.0 codebase closer to the 2.1 (and 3.0) codebase, but I fear we may have lost some of the flexibility that the original plRegistry system afforded.

Part of me would like to reimplement the plRegistry system and lay the groundwork for binary diff patching for shards, but I worry that (being an outsider to the actual development) I could be missing technical reasons for the change.

Hoping you might be able to provide some enlightenment,
~Darryl (Paradox)