Xanadu II Translation Development Blog

[elmer]

Just in case anyone is interested ... I took a break and had a look at the original Legend of Xanadu 1 again.

The scripting language is almost identical.

The code "markers" that identify where the scripts are located are all different, but it doesn't take too long to find them and to fix up the search algorithm.

I've found and extracted 143 script chunks from it so far.

[SamIAm]

I didn't want to say anything prematurely, but it looks like a translation of Xanadu I is really going to happen.

I've already got a script for the cutscenes far along, and if all goes well, I'll be working on the in game stuff within the next week or two!

We're going to need a LOT of people for a dub, though.

[jtucci31]

Oh man this is amazing news! The cutscenes in Xanadu I are so damn cool, especially after playing some long winded area where the fetch quests drag because you have no idea what's going on (*cough*area 8*cough*).

Currently on area 10, another one that feels long. I can't wait to replay this game in all its English glory!

[Necromancer]

Awesome news!    Both LoXes for your bagels!

[shawnji]

I'll be involved in the dub if you want.  I used to work on stage professionally, but the only problem is the lack of a good mic setup.  I have wanted to get a boom mic for a long time, though...

[LentFilms]

We're going to need a LOT of people for a dub, though.

If you need me to reprise my role for "Sailor/Thug" I'd be happy to do so.

[johnnykonami]

I would love to try, but I might be a horrible actor.  If you need a small part or something, it would be cool to do though.

[Gredler]

Hahaha man I'd send a few voice samples if needed, that'd be fun

[spenoza]

I would also be willing to audition, and I have access to a couple nice microphones of different types at work. I may need help from someone with a clue to know which one is most appropriate, however...

[SamIAm]

I appreciate everyone's interest, as I will surely need a lot of help. However, for the time being, you'll just have to sit tight. I'm a ways away from having a recording-ready script, and I'd rather deal with one thing at a time.

I'll be sure to post on this forum before recruiting anywhere else!

[elmer]

Just in case anyone is interested ... I took a break and had a look at the original Legend of Xanadu 1 again.

The scripting language is almost identical.

The code "markers" that identify where the scripts are located are all different, but it doesn't take too long to find them and to fix up the search algorithm.

I've found and extracted 143 script chunks from it so far.

Hmmm ... well Xanadu 1 is simultaneously both an "easier" and a "harder" game to hack than Xanadu 2.

The nice thing is that the architecture is basically the same as Xanadu 2, with a "permanent" set of boot/utility code from $2000-$3fff in memory, and then "overlay" code from $4000-$9fff, and "script" chunks that get decompressed as needed into $a000-$bfff.

The game "overlay" code is the same for every top-down level, and the game just loads in a different 176KB META-BLOCK compressed data file that contains the level's graphics and scripts.

So far, so good.

Common scripts, such as item-names are located in the game overlay to save memory.

But then they were still running out of memory ... a couple of the levels have only a few bytes free in the 176KB allowed for each level.

So they also mapped another block semi-permanently into $c000-$dfff and started putting some scripts into that area.

And they were still running out, so that last level splits the script area into 2 4KB chunks and hacks the loading system to decompress 2 different scripts into $a000-$afff and $b000-$bfff. This let them get some more reuse out of the code in that level.

Yuk!

So finding all the scripts has been a bit nasty ... particularly the side-view Weapon Shop scripts which are done in a very different method to everything else. (BTW ... up to 181 scripts with text, now.)

Anyway ... the conclusion from all of this is that Xanadu 1 is pretty short on free memory for the translation.

***********************

Falcom already compresses the original SJIS text by encoding the 192 most-common katankana/kanji into a single byte, and this really works out well for them.

Xanadu 1 has 235,523 SJIS glyphs stored using 271,679 bytes.
Xanadu 2 has  96,139 SJIS glyphs stored using 116,346 bytes.

That's an approx 1.2 multiplier, much better than the 2.0 multiplier of pure SJIS.

Apart from showing that Xanadu 2 really is a much shorter story than Xanadu 1, it shows that we've got a problem.

In order to get a good English translation, SamIAm estimates that we're going to need approx 1.5 to 2.0 times the amount of English characters as Kanji glyphs.

This gives me 2 problems ... how to fit all this English text into a level's compressed 176KB META-BLOCK that gets loaded into memory ... and then how to actually free up enough memory so that a large English script-chunk can be decompressed and accessed in the game.

***********************

Luckily, the first part is easy(ish) ... Xanadu 1 stores all it's data compressed in what I've called the FALCOM1 data format.

If I recompress ALL the data in the game in with SWD, it'll shrink each level's compressed META-BLOCK so that there will be enough memory to store all the extra English text.

The expectation that we were going to hit this sort of problem is one of the reasons for spending so much time messing around with compression earlier.

So here are the results of recompressing each of the 12 levels in different formats (the numbers in braces are the Falcom1 compressed and decompressed sizes).

Blk $00d9800  71 Chk (161,912 / 313,390), Fal2 135,530, Swd4 131,437, Swd5 130,601
Blk $0105800  69 Chk (150,455 / 304,021), Fal2 126,807, Swd4 123,600, Swd5 122,706
Blk $0131800  85 Chk (179,364 / 346,665), Fal2 150,903, Swd4 146,514, Swd5 145,484
Blk $015d800  83 Chk (177,218 / 344,515), Fal2 148,163, Swd4 143,686, Swd5 142,635
Blk $0189800  76 Chk (168,666 / 334,124), Fal2 141,733, Swd4 137,780, Swd5 136,851
Blk $01b5800  78 Chk (175,358 / 333,670), Fal2 146,457, Swd4 142,661, Swd5 141,742
Blk $01e1800  80 Chk (169,941 / 329,813), Fal2 142,395, Swd4 138,714, Swd5 137,754
Blk $020d800  79 Chk (179,178 / 334,334), Fal2 147,208, Swd4 143,309, Swd5 142,467
Blk $0239800  67 Chk (160,874 / 302,719), Fal2 136,316, Swd4 132,410, Swd5 131,443
Blk $0265800  84 Chk (178,110 / 338,782), Fal2 146,571, Swd4 142,236, Swd5 141,287
Blk $0291800  60 Chk (133,598 / 266,361), Fal2 113,042, Swd4 109,641, Swd5 108,871
Blk $02bd800 102 Chk (177,269 / 361,138), Fal2 144,124, Swd4 137,984, Swd5 137,007

The game allows for 180,224 bytes (176KB) for the compressed block.

Taking a look at the largest level ...

FAL1 compressed  179,364
SWD5 compressed  145,484

Which means we should be able to afford not only to have SamIAm do the best translation, but I should also be able to afford to leave 8KB of that space free to use for decompressing the text.

It's definitely going to be a pain to completely replace Xanadu 1's original compression code/data, but I really don't think that there's much of an alternative.

***********************

The second part, allowing decompressed scripts to be larger, is going to be tricky.

If the Xanadu games had just used a nice-and-simple text-printing routine, then it would have been easy to hack the code to switch in a new bank of text at the start of the routine, and then switch it back again at the end.

Unfortunately, since the text is contained within the game's scripting language, and those scripts are located in nearly every possible banked-region in memory, and the script itself is read from multiple different pieces of code ... I don't think that I can get away with anything that simple.

The only solution that I can come up with at the moment is going to mean switching out the CD BIOS code that's mapped into $e000-$ffff.

If I map the 8KB bank of RAM that I've freed up from the 176KB of compressed level data into that area, then I'm going to have a lot of extra space for the translation and for the English font code.

It'll mean hacking the loading code to decompresses scripts into both $a000-$bfff and $f000-$ffef, but that's not too horrible.

The idea would be to map the CD BIOS vectors and interrupt vectors to new code that switches to the original BIOS and executes the original BIOS functions, and then switches back to my RAM bank afterwards.

IIRC, Bonknuts suggested doing something like this on his blog, but I'm not sure if anyone has done this yet in practice.

If they have, then I'd love to hear about it, and about what the potential problems are!

[gekioh]

Dude hell yes, this is awesome. I've always wanted to play this game and actually KNOW what was going on. woo hoo!!!

[Bonknuts]

So if I'm reading this right, the problem (mentioned at the end) is not enough logical address space? Do you have a free 8k bank of ram to swap into page #7? If you do map another bank there, I would definitely disable interrupts the alt bank is mapped there. Another approach, to logical address issues hacking, is to swap out page #0. Though that means re-routing interrupts from the CD bios routine to your hook code, so it can map it back in for that interrupt call, and then pull the TAM value off the stack and put the page #0 correct on exit (your bank).

 But, if you need to swap out banks to have access to more free space for decompression - aren't there other pages you could swap out first? If original library mapping is a concern, maybe disable interrupts for that extended decompression part (writing into an extended buffer)? Same for when the game routine need to read a character from the extended buffer?

[elmer]

So if I'm reading this right, the problem (mentioned at the end) is not enough logical address space? Do you have a free 8k bank of ram to swap into page #7? If you do map another bank there, I would definitely disable interrupts the alt bank is mapped there. Another approach, to logical address issues hacking, is to swap out page #0. Though that means re-routing interrupts from the CD bios routine to your hook code, so it can map it back in for that interrupt call, and then pull the TAM value off the stack and put the page #0 correct on exit (your bank).

It's a logical-address-space issue after decompression, and during game execution.

I can deal with the amount of space that each level's data takes up before-decompression by using a better compressor.

That will also free up an 8KB physical RAM bank that I can then use however I wish.

The problem is that the new English script-language data that's been decompressed is going to bigger than original Japanese script-language data.

It's got to be stored somewhere that's accessible to the script-interpreter.

AFAIK there's 2 possible solutions to that ...

1) Put the overflow somewhere in physical memory and only map it into logical memory just for the instant that the interpreter reads a byte from the script.

2) Put the overflow somewhere in physical memory and have it permanently mapped into logical memory.

***********************

If I choose option 1, then I've got to be 100% certain that I know every location in code where the interpreter reads a byte from the current script-pc. I've also got to either find somewhere "safe" in logical address space to temporarily map the bank, or else I've got to disable interrupts while I map in the bank and read the byte.

If I disable interrupts, then anything that relies on the interrupt timing will glitch.

I don't know that anything does ... but it's not the kind of thing that I like to do, especially when it's not my code in the first place, so I don't know exactly how it was designed.

If I were the original developer ... this is probably the method that I'd choose to implement. It's the cleanest, and the most standards-compliant.

***********************

If I choose option 2, then I've got to decide whether to use bank 0 or bank 7, and bank 7 is the obvious choice.

If I do that, then I can write new interrupt vectors and new BIOS function vectors, and just map the CD BIOS bank into $e000-$ffff when it actually needs to be there to handle an interrupt or a function call.

The new vectors will waste a bit of space, but the flexibility gained is pretty tremendous.

Now, if I were the original developer, I could do all of this without any knowledge of the BIOS internals, but since I don't have the luxury of having the source, I can make things easier for myself by taking advantage of some inner knowledge of how the CD BIOS is written.

This would totally break any console manufacturer's standards ... but that's not a problem these days.

At this point, we know that the game requires SuperCD, and there are only the Japanese and English CD BIOS 3.0 cards to worry about.

That'll let me take advantage of knowing how the BIOS code operates in order to create the RAM stubs that I have to write.

Naughty ... but this is a hack, after all.

[NightWolve]

Oh wow, looks like the game was lucky enough to finally come into the hands of someone knowledgeable enough to do this! Ys IV was tight only as far as wanting to implement subtitles instead of dubbing according to Neill Corlett, I remember that. But uh, I hope it doesn't come down to using the idea that's now available with the Turbo Everdrive's writable RAM and limiting the fan translation to that...