Shell Memory Limit

[Robert The Rebuilder]

In addition to the limits of the dimensions of a shell, we should consider the limits to memory. There are 150 shells, I believe, and if all of them are occupied, will our PCs be able to handle the load? Probably not, but we should at least make an effort to conserve memory. So, I'll start by making some assumptions, then I'll recommend a limit. Feel free to respond with your thoughts.

First, what should be the overall maximum memory footprint of Ahra Pahts? That depends on the target platform, which we'll arbitrarily define. I'll suggest that we don't focus on the amount of video memory instead of main memory, since the Plasma engine should be capable enough to swap out geometry/textures/transforms from main memory at the appropriate times. So, let me suggest that the target platform should have 1 GB of RAM. Granted, the operating system and other necessary processes (anti-virus software, firewall) will occupy a couple hundred megabytes, but we'll assume that we can use virtual memory for that portion.

Next, how should this be divided among the shells? There should be a significant amount devoted to the city itself: the streets, courtyards, district walls, Prin Pahts, and the Hub. So, let's give 40 MB for the city, and 960 MB for the shells. That works out to 6.4 MB per shell.

Finally, how does this translate to number of polygons and texture size? Well, I don't think that matters as much as the resulting file size of the shell's PRP. That's because the shell consists of many Plasma classes that include the geometry and textures, and all those structures are in the file.

Recommended limit: keep our PRP file size under 6.4 MB

Some notes:

1. The structures contained in the file are actually less than the size they occupy in memory. That's because the classes that use these structures contain run-time variables that don't exist in the file structures. So, even though our file size is only 6.4 MB, the size in memory will probably be at least 10 MB. This exceeds our target limit of 1 GB, but I think it will be hard enough to meet this restriction. Also, see the next note.

2. Although the shells are being snatched up quickly, the fact is that not everyone will complete their shell. Many shells were obtained over a year ago, and we haven't heard from some of the shell owners. So, these uncompleted shells will help us keep the overall memory footprint of Ahra Pahts under a gigabyte.

3. Each shell uses textures that are common to all shells: the walls, the ground, the steps, etc. These total up to nearly 3 MB. I have a planned modification to the plugin that will allow us to tag those textures so that, upon a per-page texture export, it will skip these textures. When the shell is loaded into Uru, it will use the copy of these textures in the Pahts_District_Textures.prp page instead. So, you can count on your shell PRP file decreasing by 3 MB in the near future.

Comments?

[Aloys]

Yes plenty of comments:
1) First: argh! this is what I forgot to mention in the FAQ.. I'll write a blurb right away.

2) I remember this was discussed in the past and we were always hesitant to impose hard numbers so as not to limit creativity (and not to scare away people). I still think we should avoid it but maybe we could try to estimate a rough maximum number of polygons and textures memory usage (although there would have to be a trade-off between those on a per-case basis..)

3) "Each shell uses textures that are common to all shells: the walls, the ground, the steps, etc. These total up to nearly 3 MB. I have a planned modification to the plugin that will allow us to tag those textures so that, upon a per-page texture export, it will skip these textures This is great! I thought this was not possible! This is something that was also discussed long ago when the support for per-pages textures was first added, and at the time it was infered that it was not doable. But I might remember wrong. Anyhow this is great news..

4) Memory footprint of the city alone at around 40 megs? hmm while I'd love it I wouldn't count too much on it. The PRPs right now are around 40 megs, but they will increase quite a bit in the upcoming weeks. With new content coming (especially new courtyards) there will be quite a lot more to load in memory. Again this is for the city as a whole, and hopefully it is not always fully loaded in memory, but that will still have an impact. (although we lack tools to measure this precisely; or even vaguely)

5) Right now we have no way of doing any manul optimisation to the Age with PyPRP. But I am confident that in the future, things like LODs and PlRelevance regions will become available and will help us. (I know I am assuming a lot as this is not a high priority in the current development list -- but "just in case this feature becomes available" I have already started working on a CSV file )

[Robert The Rebuilder]

You're right, aloys, in that limits tend to hamper people's creativity. And yet, there needs to be something in place to prevent someone from hogging all the available memory on their 1 million polygon statue of themselves (or whatever may be the case). The shell developer can keep the limit in mind while creating the shell so they don't go crazy on the textures/polygons, and then when they're nearly finished, they can make an optimization pass over their shell.

If you like, we can lower the recommended per-shell size limit to 6 MB. That will allow the city to have 100 MB.

One thing that may help is if we make use of the Plasma functions for swapping in/out pages: PtPageInNode/PtPageOutNode. These are used in the various *EmgrPhase0.py scripts to make pages disappear/reappear according to SDL variables, among other places. The downside is that all pages must initially be loaded into memory; then you can decide what to page out. [*] By capping the whole age at 1 GB, we can keep the virtual memory swapping to a minimum.

Here's an example of how we could use the PtPage*Node functions: upon startup, Prin Pahts is immediately paged out. [We start on the surface and cannot see it.] We can also page out all districts (and their shells) that are not currently visible. Then, when someone uses a Prin Pahts linking book, it is paged in. The person links down there, and all the rest of the city is paged out. When they use a city linking book, it will page in the destination district (and its shells), plus any visible districts. Once they arrive in the city, Prin Pahts is paged out again.

Does anyone have any experience in using these functions?

[*] EDIT: I stand corrected. Take a look at Cleft.py. In its constructor, it decides which items to page in, depending on the vault chronicle values. So, somehow those PRP files must have been flagged as to not be loaded. By default, all pages are loaded; otherwise, we would have to explicitly call PtPageInNode in every age's ptResponder subclass constructors.

Anyone know how to flag a page to not be loaded initially?

[boblishman]

OK...now I really need some guidance over this subject ...

Obviously, I understand the need to keep the prp files as small as possible.... the thing is ... how?

I have noticed several things whilst reading the forums and working with the ULM:

1) Most of the ages say "....(comp.) " when they are downloading in the ULM. I have Paah'kweh uploaded to the ULM ... but the files are not "compressed" : How is this done?

2) What file format is the most "economic" for textures?
(I am currently using .png files .... and a 512 x512 texture usually comes in at about 1 MB (as per the Texture cache folder files reports) .

3) Aloys, I read that you "forgot to compress the files" in the origianl AP upload. How do you set Blender (or the prp) to "compress" the files???

4) Does this mean that (if I keep my present settings) I would be limited to 4 texture files for my Shell in AP ?????

I really would appreciate it if someone could give me a set of "guidelines" that would help me to keep my final prp files to a minimum... (i.e. Texture file formats, compression, total vertices, total polygons... etc. I'm sure this information is out there somewhere....but not in a single place... and (almost) impossible to find

Any help would be appreciated...

Bob

[Aloys]

Do not panic.
We are just discussing and doing some research, and until we can reach an agreement you can do whatever you want with your shell; just try and stay 'reasonnable'. (yes that word is intentionnaly vague).
Even when we will have established guidelines for all this IMO they should remain guidelines, and not strict requirements.. Ahra Pahts is first of all meant to be a place of freedom. And shell(s) should reflect that, as long as you don't crash the whole Age.

One of the problem, as Robert mentionned, is that the size of the PRPs and their actual size in memory are not always related, so it's not easy to figure a precise limit. That's what we are working on.

I read you "forgot to compress the files" in the origianl AP upload

Yes, the textures were not compressed. When you export your Age you should export it with the "Generate release settings"; which will compress the textures. This is what I forgot to do. When textures are compressed they use less memory, the Age loads faster, and the PRPs are smaller.

[Robert The Rebuilder]

Do not panic.
We are just discussing and doing some research, and until we can reach an agreement you can do whatever you want with your shell; just try and stay 'reasonnable'. (yes that word is intentionnaly vague).

Yes - what Aloys said.

I rephrased my initial post to be more of a suggestion than a hard-and-fast ruling, since this should be an open discussion.

[Lontahv]

(I am currently using .png files .... and a 512 x512 texture usually comes in at about 1 MB (as per the Texture cache folder files reports)

I remember when you put your age on the ULM: it was huge. I think that that is not a problem of texture but texture formats, I use 100% quality jpg for every non transparent texture and my texture files seem kind of ok. 1 megabyte is really huge even for png--when you use png try to have it very highly compressed(all the way up) I can't even see the difference of a 2meg png and a 500kb png.

[Aloys]

Actually the file format you use before exporting the Age doesn't matter. When you export the Age all the textures used are converted to a different format (a specific DirectX image format). And that format can be compressed or not, depending on what exporting settings you use ("Generate Release" or not-- see my previous post).
The visual difference between compressed and non-compressed textures is minimal most of the time, and compressed textures have many advantages because of their smaller size.
Trylon recently did a change in PyPRP so that it compresses textures by default, which is a good idea I think. It will avoid ending up with very large PRPs by mistake.

[boblishman]

WOW!!....thanks for this info....

This makes a HUGE difference (exporting the Age as a 'General Release') ... on my latest version of Paah'kweh, this has brought the texture.prp down from 64MB to .... 9.2Mb!

The only drawback is that the (alpha layered) pictures in my journal have dissappeared. I'm guessing that this is due to either the compressing of the alpha texture....or the resizing of it (the .log file reports that it was resized to the usual 256 x 128 type size... when before, it was exported as it's true size of 400 x 204).

(Aloys, is this why the picture has dissappeared from the linking books in AP?)

It seems that transparancy images in journals need to be uncompressed... is there a way to stop the plugin from compressing specific files during the 'General Release' export? (I'm sure I read somewhere about a naming convention ... something like 'noCompressfilename.png' . Does that still work with the 'stable' plugin?

Accordingly I have ammended my post (and subseqently the tutorial) about Creating Journals with 'sketches' - Help for new creators until (and if) there is a solution to this problem ....

[Robert The Rebuilder]

(Aloys, is this why the picture has dissappeared from the linking books in AP?)

It seems that transparancy images in journals need to be uncompressed... is there a way to stop the plugin from compressing specific files during the 'General Release' export? (I'm sure I read somewhere about a naming convention ... something like 'noCompressfilename.png' . Does that still work with the 'stable' plugin?

You've hit the nail on the head, boblishman! That is indeed why the linking panel picture has disappeared. When exporting a release, the image is compressed with the DXT5 format. For some reason, Plasma requires the A8B8G8R8 format for linking panels.

And yes, if you prefix your texture name with "noCompress" (use a capital 'C' - the case is important) the plugin will keep the texture in the A8B8G8R8 format. So, all linking panel texture files need to have this prefix.