Iceworld issues on DI + General advice for Age writers

If you feel like you're up to the challenge of building your own Ages in Blender or 3ds Max, this is the place for you!

Iceworld issues on DI + General advice for Age writers

Postby diafero » Mon Jul 06, 2015 11:16 am

Recently, it has been brought to my attention that some people are having issues accessing Iceworld on Deep Island. The issue that I noticed is, I believe, not at all restricted to Iceworld, so I'd like to use this opportunity to bring it to all you age writers' attention. Unfortunately, we are currently in a situation where it's already hard enough to build an age for Uru, that it's a pure wonder that people are still doing it. (Thanks a lot for that!) Naturally, most of these ages are only tested by their authors in the offline game, so they are built with little consideration for Shards. I certainly can't blame their authors for that! I'd just like to bring one particular property of how Shards work to your attention, so that you can make your ages work better on them. I'm sorry this gets a bit lengthy, but the issue needs some introduction to be understandable.

When the age is running online, Uru is uses a system called "SDL" to synchronize the age state between all the people visiting it. Some of you may have encountered SDL when you wrote SDL files for your age, and manipulated the SDL through Python, or by re-using existing Python mods. If you change the SDL through Python, your client tells the server, and the server tells all the other clients, and then everybody knows that some door just opened. Even better, the server remembers this, so someone new joining in will immediately be told that the door is open. So far, so good.
However, SDL is also used by Uru to synchronize animations, making sure that everybody sees the animated objects in the same position. When people join an age, the SDL is synchronized between client and server, and Uru does this using a single message for every SDL record. The SDL file you write is one SDL record, but every single animation also gets an SDL record. That means if you have 100 animations, your client will exchange 100 packages with the server! And to make things worse, the Uru client is pretty dump in doing that, and it neglects all its other duties during the sending. For hundreds of animations, this can quickly take several seconds. The result is that the client "forgets" telling the server in time that it received some data, so the server thinks the client is gone, and closes the connection. Ouch.

This is what happens in Iceworld. In the case of this particular player, the client sends 275 (!) SDL messages to the server. That takes 5 seconds. That's the maximum delay the server accepts between the server telling the client "here's your data", and the client saying "okay, got it" - so the client is kicked. Now, the client is not even done yet sending SDL messages, so it may well be that much more than 275 messages have to be sent.

So, what can be done? Do we have to remove all these nice animations? No! Many of these animations will be just decoration. If you have some flame, shown as animated texture, it doesn't matter whether the animation of the texture is synchronized between everybody looking at it. If somebody is, like, 5 seconds "ahead" in the animation - how are they even going to notice? For such animations, it is possible to tell Uru not to synchronize the animation, and then no SDL object will be created, and less messages have to be sent.
Now, unfortunately, I have no idea how to build an age. So I don't know which buttons you have to press in Blender or 3ds Max to set this flag. It's probably something about synchronization. PyPRP developers, which Blender button did you wire that flag to? (I hope it is wired to something^^). And 3ds Max users, is there anywhere in the properties of animations and/or objects some kind of synchronization checkbox you can tick/untick? Please tell us, so we can collect this here. even better, we should collect it in the Wiki. I just wonder what would be the best place for a collection of "things to consider to make your age working better online, that you'll not notice when you test offline".
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: Iceworld issues on DI + General advice for Age writers

Postby Sirius » Mon Jul 06, 2015 1:38 pm

Ah, now that's an interesting issue ! The flag that must be set is called ExcludeAllPersistentStates.

I had a quick look at PyPRP, it seems the flag is mostly on by default:
Last-minute pass over the SDL flags to make sure everything is sane for online environments
Here's the rub:
- If the flags have already been set, do nothing.
- If there is a Dynamic HKPhysical, see PFM (looks like a Cyanic hack).
- If there is a Responder attached, we exclude the Responder state
- If there is a PythonFileMod attached, we exclude everything a PFM might touch:
-- AGMaster
-- Layer
-- Responder
-- Sound
-- XRegion
- Otherwise, exclude everything!
If I understand this correctly, the animation's state will be saved if the object has physics, a python script or a responder attached to it (shouldn't this filter unnecessary stuff already ?...).
However it's possible to force it on. Alcscript:
Code: Select all
ObjectName:
    synchflags:
     - excludeallpersistentstate


From what I can see in the Max plugin documentation (page 71), this option is in the NetSync tab. I would say builders should tick the "Local Only" checkbox.
OR, maybe tick the "Override High Level SDL" and set all other buttons to "Deny". One of these should do the trick.

[EDIT: According to Hoikas, it seems there is no need for 3DsMax users to do this. Just leave everything at their default values and it should be fine.]

As for existing Ages which don't have this flag, if we can figure out what PyPRP misses, maybe we could write a utility to edit PRPs to automatically add it when required ?

As an alternative to syncing animations, it's possible to sync what triggers them (for instance, since all clients receive the "I clicked a button" message, every client could play the animation on its own without having to sync it with the server) - that's what mostly happens in Cyan's Ages. I have no idea how we could do that in PyPRP though.

(Don't Myst V Ages suffer from this issue too, by the way ? I recall the flag is not on the same bit on Myst V and Uru, which means Uru reads it as something like HasConstantNetGroup...)
Last edited by Sirius on Wed Jul 08, 2015 5:17 am, edited 1 time in total.
User avatar
Sirius
 
Posts: 1506
Joined: Mon Jul 26, 2010 4:46 am
Location: France

Re: Iceworld issues on DI + General advice for Age writers

Postby Tsar Hoikas » Mon Jul 06, 2015 4:14 pm

In the git master PyPRP, the animation state is only saved if the author touches the SDL flags with AlcScript. This was done for in consideration of ages produced in PyPRP for Gehn or other shards. In 1.6.1, however, all SDL states are sent to the server and need to be disabled by using the AlcScript as provided by Sirius:
Code: Select all
    synchflags:
     - excludeallpersistentstate


No change should be needed to ages with 3ds Max.
Image
Tsar Hoikas
Councilor of Technical Direction
 
Posts: 2180
Joined: Fri Nov 16, 2007 9:45 pm
Location: South Georgia

Re: Iceworld issues on DI + General advice for Age writers

Postby denost2 » Tue Jul 07, 2015 12:30 am

Diafero,

I read your presentation with great attention.
Thank you for the effort to explain some aspects I'm not familiar with at all.

I had just a first lecture. I should have to read it again.

For the moment:
- I can say Iceworld SDL file concerns 41 objects
- the positions/states of these objects are included inside the global logic of the game
- The last Age I released; Elsewhere SDL file concerns about 80 objects
- I have noticed the advice to use the simple script:
Sirius wrote:ObjectName:
    synchflags:
     - excludeallpersistentstate
denost2
 
Posts: 83
Joined: Mon Jan 10, 2011 11:41 am

Re: Iceworld issues on DI + General advice for Age writers

Postby diafero » Tue Jul 07, 2015 2:07 am

For the moment:
- I can say Iceworld SDL file concerns 41 objects
- the positions/states of these objects are included inside the global logic of the game
- The last Age I released; Elsewhere SDL file concerns about 80 objects


The SDL file you are writing yourself is perfectly fine. Even way more than 100 entries are not an issue here, because Uru groups all of them together in a single SDL record. So, all these 40 or 80 objects become one record, and that's just one message between Uru and the server. However, the SDL you write is not the only SDL record that applies to your age. In particular, every single animation becomes yet another record, and it's the large number of records, not the number of entries per record, that causes trouble.
(Did you ever wonder how Uru remember where kickables are positioned in the age? Well, that's also done using SDL. Every kickable stores an SDL record on the server, remembering its position and orientation. But people typically don't have hundreds of kickables, so we're good here.)

However, if you could add that AlcScript snippet to your age, that would be great. Sirius, Hoikas - where would this have to be added? To all animated objects?

(Don't Myst V Ages suffer from this issue too, by the way ? I recall the flag is not on the same bit on Myst V and Uru, which means Uru reads it as something like HasConstantNetGroup...)
Drizzle translates flags. However, now that you mention it, I actually remember that we did have this issue with Myst V ages - with Noloben, to be more specific. It was exactly the same problem, people ended up disconnecting on link-in. Now, if I only remember what we did to solve this. Maybe we changed Drizzle to no longer emit this flag? *scratches his head*

In the git master PyPRP, the animation state is only saved if the author touches the SDL flags with AlcScript. This was done for in consideration of ages produced in PyPRP for Gehn or other shards.
Is there any chance of a 1.6.2 release soonish so that age writers and Shards can actually benefit from this?
If I understand you correctly, the defaults are already appropriate for Shards in 3ds Max.
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: Iceworld issues on DI + General advice for Age writers

Postby Sirius » Tue Jul 07, 2015 6:25 am

Diafero wrote:However, if you could add that AlcScript snippet to your age, that would be great. Sirius, Hoikas - where would this have to be added? To all animated objects?
Hmmm... Any non-essential animated object should. For instance, the Age has flags (waving ones ;) ) and falling snow, these don't need to be synced and could use the AlcScript. Animations that are part of puzzles can usually be left as-is (I /think/ you could also de-sync puzzle animations, as they are often triggered by a LogicModifier which is itself synced anyway. But that's just a wild guess, it's safer to leave them synchronized).

After a deeper look at Iceworld's PRP, I can notice a lot of ATCAnims. Some are for the puzzles, some for the flags/snow, but the bottleneck are the snowstorms moving around the Age. It's all the objects named "RV1...." and "V1...." (totaling roughly 500 objects, from what I can see. Ah, I wish we could use particle effects to avoid such annoyance...). These represent roughly half the animations in the Age. De-syncing only these would solve most problems with the Age, I believe. We can leave synchronization for the "master" objects (the objects named "GRV...") which parent all the particles, this would ensure the snowstorms are always seen at the same location for every player.
However, this means applying the AlcScript to 500 objects, which is simply madness, so that's not a solution.

Actually, it seems absolutely none of the objects have the ExcludeAllPersistentState set - while PyPRP's default behavior is to always turn it on for most objects (except a few with responders or physics). I guess that's because the Age was not exported with the latest version of PyPRP (which should also fix the bouncy collisions bug present in the Age).
If I understand what Hoikas said, if Denost re-exported the Age using the latest PyPRP, the issue would be solved without the need of additional scripting ? It might completely de-sync the snowstorms, though...

As an alternative to re-exporting the Age (which can take a lot of time), I can simply use a script to set the flag on all "RV1" / "V1" object in the existing Age. This would only take a minute to apply, it would fix the buggy synchronization, while ensuring every player can see the snowstorm at the same location.


Diafero wrote:Drizzle translates flags. However, now that you mention it, I actually remember that we did have this issue with Myst V ages - with Noloben, to be more specific. It was exactly the same problem, people ended up disconnecting on link-in. Now, if I only remember what we did to solve this. Maybe we changed Drizzle to no longer emit this flag? *scratches his head*
Nope, it looks like the current version of Drizzle still doesn't correct the flag, so maybe it was fixed on the server-side.
Noloben has something like a thousand different animations, mainly because it has lots of birds and snakes, which all require a dozen bones each. Add to that animations for the ferns, the ladder (which has hundred of bones too), the windows in the lab, and all Slate-related animations, and you end up with a lot of things to synchronize...
User avatar
Sirius
 
Posts: 1506
Joined: Mon Jul 26, 2010 4:46 am
Location: France

Re: Iceworld issues on DI + General advice for Age writers

Postby denost2 » Tue Jul 07, 2015 12:10 pm

Sirius wrote:As an alternative to re-exporting the Age (which can take a lot of time), I can simply use a script to set the flag on all "RV1" / "V1" object in the existing Age. This would only take a minute to apply, it would fix the buggy synchronization, while ensuring every player can see the snowstorm at the same location


Interesting.

Thank you Sirius,
You have carte blanche to test the solution you propose on Iceworld.
denost2
 
Posts: 83
Joined: Mon Jan 10, 2011 11:41 am

Re: Iceworld issues on DI + General advice for Age writers

Postby Sirius » Wed Jul 08, 2015 3:10 am

Okay, done. The things I de-synchronized:
  • the snowstorms' particles
  • the flags
  • the candles' flames
These were objects with most of the animations, it should help avoid the bug.

While I was at it, I also disabled shadow casting (since it was eating performance for nothing), and fixed the physics bug (which should make exploring a bit more natural).

https://www.dropbox.com/s/8szqpnof6iu9oqn/Iceworld_District_mainRoom.zip?dl=1
User avatar
Sirius
 
Posts: 1506
Joined: Mon Jul 26, 2010 4:46 am
Location: France

Re: Iceworld issues on DI + General advice for Age writers

Postby diafero » Wed Jul 08, 2015 3:30 am

Sirius wrote:As an alternative to re-exporting the Age (which can take a lot of time), I can simply use a script to set the flag on all "RV1" / "V1" object in the existing Age. This would only take a minute to apply, it would fix the buggy synchronization, while ensuring every player can see the snowstorm at the same location.

That sounds certainly like it's worth a try. However, of course, the next version denost produces would bring the issue back.

Sirius wrote:Nope, it looks like the current version of Drizzle still doesn't correct the flag, so maybe it was fixed on the server-side.
Noloben has something like a thousand different animations, mainly because it has lots of birds and snakes, which all require a dozen bones each. Add to that animations for the ferns, the ladder (which has hundred of bones too), the windows in the lab, and all Slate-related animations, and you end up with a lot of things to synchronize...

That's strange, I could have sworn... but you're right, I just went to Noloben and there are ~500 messages sent from client to server on log-in. That worked fine here, though. But then I have a really fast network where I am right now.
I now think that there are two things we did to help with this issue:
* We increased some timeouts on the server. I found this quote in the source code: "choose higher timeout for client connections - the clients netcore sometimes seems to be blocked long beyond any reasonable time (even when sending a whole lot of packets, e.g. when linking to Noloben, it does not reply to acks)."
* Set the "connection speed" (that you can configure in the launcher) to the fastest possible one per default.

I just added a small patch to the server to give Uru clients more time for re-sending the packets. I'll wait for a reply of the explorer who emailed me, to see whether that helps.
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Re: Iceworld issues on DI + General advice for Age writers

Postby diafero » Wed Jul 08, 2015 3:33 am

Sirius wrote:Okay, done. The things I de-synchronized:
  • the snowstorms' particles
  • the flags
  • the candles' flames
These were objects with most of the animations, it should help avoid the bug.

While I was at it, I also disabled shadow casting (since it was eating performance for nothing), and fixed the physics bug (which should make exploring a bit more natural).

https://www.dropbox.com/s/8szqpnof6iu9oqn/Iceworld_District_mainRoom.zip?dl=1


Thanks! I'll give this file a shot later, when we know whether the timeout change on its own is good enough for Iceworld.
I prefer e-mails to "diafero arcor de" (after adding the at and the dot) over PMs.

"Many people's horizon is a circle with a radius of zero. They call it their point of view."

Deep Island Shard | Offline KI
diafero
Deep Island Admin
 
Posts: 2966
Joined: Mon May 05, 2008 5:50 am
Location: Germany

Next

Return to Building

Who is online

Users browsing this forum: No registered users and 2 guests

cron