In the Myst online forum there is a long discussion about Loading Myst 5 End ofAges on a vista system.
http://mystonline.com/forums/viewtopic.php?p=269599
Erik posts the link to a zip file that contains a patch. http://cho.cyan.com/patches/Myst5InitFiles.zip
I tried this using the directory from the readme in the zip and it didn't work.
Before you follow this instructions, you should have installed Myst V with the dvd-rom.
Copy the folder in the zip file (Myst V End of Ages) to:
C:\Users\<your windows username>\AppData\Local
That's it! Myst V should now start up properly.
(If you can't find the folder, remember that AppData is a hidden directory.
Turn on hidden files, or enter the folder path directly into the address bar of Windows Explorer.)
I poked around and found the following directory.
C:Users\<my windows username>\AppData\Local\Myst V End Of Ages\init
When I dropped the zip contents into the init directory the game started working.