Author Topic: Xanadu II Translation Development Blog  (Read 39425 times)

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #135 on: December 05, 2015, 05:21:51 PM »
If you get a chance, see if you can rip the custom cd read part. I have one from Seiya Monogatari that I'd like to compare.

I doubt that I'll have the time to seriously investigate the custom CD routines for a few months, but I can tell you where they are.

Here is the function that loads a code overlay from both Xanadu 1 and Xanadu 2.

You can see that they're almost identical, and that new custom routines are set up to use the same input parameters as the original System Card routines.

All you have to do is to run Xanadu 2, and by the time that you get to the Main Menu, you'll be able to find the new CD read routine at $a807 in bank $84.

Good hunting!


*********************
XANADU 1 LOAD_OVERLAY
*********************

$3308   jsr   $334a     ; _find_file_index

$330b   jsr   $3377     ; _find_file_pos_len

        lda   $02       ; local addr
        sta   $fa
        lda   $03
        sta   $fb
        lda   $01
        sta   $ff

        jsr   $e009     ; ex_cd_read

        sta   $45
        cmp   #$00      ; $00 = OK
        beq   $3328

$3323   jsr   $33a1
        bra   $330b

$3328   rts


*********************
XANADU 2 LOAD_OVERLAY
*********************

$332b   jsr   $3371     ; _find_file_index

$332e   jsr   $339e     ; _find_file_pos_len

        lda   $02       ; local addr
        sta   $fa
        lda   $03
        sta   $fb
        lda   $01
        sta   $ff

        tma   #$20      ; Remap bank $84 into $a000
        pha
        lda   #$1c
        clc
        adc   $203f
        tam   #$20

$3348   jsr   $a807     ; !!! CUSTOM CD LOADING CODE !!!

        pla
        tam   #$20
        rts


elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #136 on: December 08, 2015, 02:02:48 PM »
I gotta say that Xanadu I has absolutely fantastic, awesome music.... but the instruments aren't that complex and they don't change all that much.  It's quite typical of a JP computer composition/translation, I think.  Jinmu Denshou (Yaksa) on the PCE has a similar thing going on.  The focus is on polyphony with a lot of sawtooth sounds.

Compare that to Ankoku Densetsu or Dungeon Explorer, where in a lot of situations simple square waves are used, but 2 channels are paired, playing almost the same note.  The phasing between the square waves does all of the heavy lifting to make a powerful sound.

Thanks for the analysis, I find that very, very interesting.  :D

A pair of slightly de-tuned channels is a great trick for making a wonderful sound.

**************

Just a quick update on the current progress, since I don't have any new pretty pictures to show.

The game seems to be happy to run with SWD5-compressed data instead of the old FALCOM1/FALCOM2 compressed data.

I'm testing the first scripts, and the overflow functionality (that lets the English translations be much larger than the Japanese originals) seems to be working fine.

My changes to the script compiler to allow for that introduced a couple of bugs that's taken a while to fix.

The current annoyance is that the game is stomping all over my English font ... so that place where I've put it isn't as "safe" as I thought.  :roll:

I think that I can free up some memory for the font by optimizing how the CD directory information is stored ... and that's the next task.

So, two steps forward, half-a-step backwards.  ](*,)

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #137 on: December 18, 2015, 08:02:18 AM »
Time for another couple of screenshots ...







It's nice to see some progress, but they don't look too impressive when compared to the previous screenshots, do they?

It's just from the first building that you go into when you start the game. Wasn't that already done?

The answer is "No" ... this is actually a major step forward, and one that's taken a lot of invisible time-and-effort.

Let me explain ...


******************

As SamIAm has said in the main thread, in the past couple of weeks I've actually got all of his current translations inserted into the game (the first 5 levels).

My goal is to get to the point that I can just give him a set of insertion tools, and then he'll be able to make small edits and quickly see how they look in the game.

We're still quite a way away from that goal, but now we're one big step closer.

That's because the game has been built like an onion ... with many layers.

The 12 levels (which contain a total of 168 script files) that SamIAm has been translating are at the outer layer.

They're the most important part of the game ... but they often refer to little snippets of script data that are actually stored in the inside the middle layer, the Game Code overlay itself.

The Game Code overlay uses functions that are built into the Boot Code ... the inner layer.

This is a very structured way of putting a game together, but it can be a real PITA to hack.

In contrast, Zeroigar was constructed like a book ... totally separate pages that really didn't refer to each other. It was also a PITA to hack, but in a totally different way.

In Zeroigar, with its 35+ different programs, I'd often have to find/implement similar hacks in all 35+ different programs. But since each one was separate, each hack was independant of the others, and nothing relied on the previous hacks. That is time-consuming and terribly boring to do ... but it's not particularly difficult.

In the Xanadu games, changes can affect each other. That's a more efficient way of doing things, but it's a bit more complicated to manage.

In particular ... the method that I'm using for recompiling the translated scripts, and for handling the overflow that happens because there's not enough room to the English text, and then repackaging everything back into each level's META_BLOCK so that it gets loaded correctly ... well, that just doesn't work for those inner layers.

Modifying the whole process to be able to handle those inner layers is what has taken a lot of time over the last couple of weeks.

In Xanadu 1, nearly all of the item description script data is stored in the Game Code overlay. The name "Arios" is also in there, as are various common scripts such as the generic things that the "Sister" says when you're in a temple.

Interestingly enough ... each level's META_BLOCK also contains a special DATA_CHUNK with up to 8 level-specific custom items that overwrite the default item descriptions in the Game Code overlay when that level is loaded.

It was only a couple of days ago that I found out that those existed, and then tracked down how they were done. We weren't extracting those scripts before ... now we are.

The problems with having all those scripts in the Game Code overlay is that ...
[uldecimal][li] there isn't enough memory in the Game Code overlay for the English versions.[/li][li]the scripts in the levels directly refer to the locations of the scripts in the code.[/li][/ul]That's meant that I've had to figure out both how to load up the extra data for the translations, and also how to tell all the level scripts that the Game Code scripts have moved to a different location.


******************

Solving the 1st part has meant splitting the scripts that are in the Game Code into 3 different parts.

The script table that is used to locate each item's description has been moved into some of my extremely-precious free space in the CD Boot Code section. That means that it is always available for reference/modification, which is needed for when the Game Code loads the level-specific item descriptions and overwrites some of the table entries.

The item descriptions themselves have been moved into the bank of extra memory that I've freed by recompressing all of the game's data (the META_BLOCKs) with the more-efficient SWD5 compression.

I've written some new code that's been added to the Boot Code itself to actually load those item descriptions and then copy them into that bank.

All the "names" and "common" scripts are still in the Game Code overlay, and now that the item descriptions have been removed from it, there's plenty of room in the overlay for the larger translations.

FYI ... the current English translations that have been inserted are sometimes taking up to 50% more space than the original Japanese scripts.


******************

Solving the 2nd part has been easier, but needs to be improved.

When I first started to disassemble the scripts and found that Falcom were structuring the game like this, I knew that I'd have to deal with it at some point, and so added the information into the extraction process.

Each script file that SamIAm is translating has a section at the top that contains a list of the "external" scripts that that particular script can refer to that aren't actually located in that particular script file.

It looks like ...

  @extscript( $997e, .table997E  )
  @extscript( $9ea8, .script9EA8 )
  @extscript( $9ead, .script9EAD )
  @extscript( $9ecd, .script9ECD )
  @extscript( $9f1b, .script9F1B )
  @extscript( $9f8b, .script9F8B )


This tells the script compiler that when it sees a script referring to something with one of those labels (the 2nd value), then it should actually compile in a different address (the 1st value).

Now that I've actually got those scripts compiling, that table is changed to the following, which corresponds the the new locations of each of the scripts ...

  @extscript( $3f7e, .table997E  )
  @extscript( $9614, .script9EA8 )
  @extscript( $998a, .script9EAD )
  @extscript( $99bb, .script9ECD )
  @extscript( $9a2a, .script9F1B )
  @extscript( $9ad1, .script9F8B )


This is fine, and works great (you can see the results in the screenshots above), but it's a real PITA to manage.

It means that whenever SamIAm or I edits one of those scripts in the Game Code, then I've got to change that table in all 168 script files.

I need to implement a "better" solution for the script compiler (like "include" files) ... that's the next task on my list.  :-k


******************

So, the conclusion from all of this rambling, is that what you are seeing in those screenshots at the start of this message is actually the result of scripts that are located in a number of different places and that are loaded into memory at totally different times.

The 1st screenshot comes from a script that's located in the 1st script file for the 1st level (the Prologue).

Then, that same script calls some code that gives the player the "Clothes" item, and then calls another script in the Game Code overlay to display the "item-received" script (which has been re-compiled into a new location).

The "item-received" script then looks up the item number in the item table (which has been re-compiled into the Boot Code), and then prints out the item description for "Clothes" (which has been re-compiled into the extra script memory that I've freed-up), and finally prints out " received."

So, even though it just looks like a simple pair of messages on screen ... getting all those things working and properly-coordinated is actually a pretty major step forward in getting the translation done.  :wink:

SamIAm

  • Hero Member
  • *****
  • Posts: 1835
Re: Xanadu II Translation Development Blog
« Reply #138 on: December 18, 2015, 01:14:42 PM »
Awesome post, elmer. :)

So awesome, it inspired me to drop what I was doing and get back to work on Chapter 5. ;)

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #139 on: December 27, 2015, 03:59:05 AM »
This "blog" has definitely been getting quieter in the last few weeks as there's not too been much "interesting" stuff to post, since Xanadu 1 is pretty-much hacked-to-death at this point.

The only interesting problems that have had to be solved were in what I keep on calling the "Weapon Shops", i.e. the "Buy" shop and the "Sell" shop.

First of all, they contain "custom" code for printing the text messages, and that code wasn't extracted-and-patchable in the way that the other code overlays were.

The solution for that was to add some new patching commands into the script compiler to apply the patches at the same time that the translations are applied.

That's a pretty ugly hack ... but it works, and it keeps the ugliness restricted to only those few scripts that use the new script commands.

***********

More interesting was that those Weapon Shop scripts are the only time that I've seen so far that Falcom is decompressing graphic data while the game/music is still running ... well, it's the only one that has caused a problem anyway.

The Weapon Shops use a custom set of tiles for the dialog boxes, and it decompresses that graphic data whenever the dialog box is cleared.

My new SWD5 decompressor is using the HuC6280's TII instruction for data copies ... which, as any PCE programmer knows, blocks interrupts while it is executing.

That doesn't cause any problem if you're only disabling the interrupts for a short time, but in this particular case, the graphics for the dialog box compress really well, and the SWD5 decompression executes TII copies with large chunks of repeated data.

That was blocking interrupts too long, and was causing a skip in the music and a nasty glitch in the graphics on the screen.

The obvious solution is to replace the TII instruction with an assembly routine that does the data copy without blocking the interrupts.

The problem is that that routine is both larger and slower than using the TII instruction.

Sometimes that's the best solution to use, particularly if you're doing a lot of complex stuff with interrupts ... but in this case, there's another solution available.

That solution is just to change the compression code to set a lower limit on the maximum size of any repeated chunk of data that is found.

In this case ... changing that limit from 8KB, all the way down to 128 bytes, was enough to fix the glitching.

In order to avoid hurting the compression too much, the "fix" is only applied to that 1 specific chunk of graphics data ... which was another problem ... but it's all solved now.

***********

As Dave so very correctly pointed out a while back ...

The data that you will be recompressing later will NOT be the same as the data in those blocks today, so a test on SWD3/SWD4 compressibility of existing data is not very relevant.

For one thing, text is usually encoded in these things as 2-byte SJIS for kanji, and 1-byte JIS for kana.  It doesn't compress anywhere near as well as English (though it is generally slightly more dense to begin with).

So, now that we've got a couple of levels of English translations running in Xanadu 1, I thought that I'd take a look at how the compression is doing.

META_BLOCK "Z10" $000d9800, originally 160KB,  SWD5 Jpn 134KB,  SWD5 Eng 138KB.
META_BLOCK "Z11" $00105800, originally 148KB,  SWD5 Jpn 126KB,  SWD5 Eng 130KB.
META_BLOCK "Z12" $00131800, originally 176KB,  SWD5 Jpn 148KB,  SWD5 Eng 154KB.
META_BLOCK "Z13" $0015d800, originally 174KB,  SWD5 Jpn 146KB,  SWD5 Eng 152KB.
META_BLOCK "Z14" $00189800, originally 166KB,  SWD5 Jpn 140KB,  SWD5 Eng 148KB.


The game allocates 176KB for loading a META_BLOCK into memory, and I need to reduce that to 168KB maximum in order to free up the 8KB block that I'm using for the English translations.

Those results show that we're doing really well so far, and that we should have no trouble getting Xanadu 1 to run with a Super System Card 3, and that we're not going to need the extra memory in a Turbo Everdrive 2.  :D

Just to state the obvious ... it's still way too early to tell if Xanadu 2 is going to fit.

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #140 on: December 27, 2015, 11:40:15 AM »
Sometimes things work out ...




Sometimes things don't ...




So, once again, with Xanadu 2 it's the Weapon Shops that are being a royal PITA.  :roll:

Falcom are not only doing a completely new trick with them, but they're using custom display code for the 2 English strings that bypasses my regular text-display fixes, and causes character spacing problems.

The new trick is actually pretty neat ... in Xanadu 2, all of the item descriptions are stored in the script chunk for the "Pause Menu", and not in the Game Code like in Xanadu 1.

Each Weapon Shop is contained in it's own script chunk, just as you'd expect, and this is decompressed and copied into $A000-$BFFF for execution just like all the other script chunks.

But in order to print out the item descriptions, the Weapon Shop itself decompresses the Pause Menu's script chunk into the temporary decompression buffer and maps that buffer into $C000-$DFFF, just so that it can access the table of item descriptions (which is located $2000 higher than normal).

I think that's quite a neat trick.  :)

However ... now we've got a couple of problems.

The easier one is going to be finding/hacking the new custom display code so that the 8x12/4x12 English font spacing works properly.

The harder one is that even when that is done, the English item descriptions are just too long to fit into the small box that's been provided.  #-o

That will need some thinking ... and possibly a new 6x12 font.

There should be space for a new font where Xanadu 2 is currently loading it's custom 8x12 font ... thankfully.

But it's going to be a real PITA to switch the entire game from working in 4-pixel text-coordinate steps into 2-pixel (or 1-pixel) steps in order to cope with both 4, 8, and now 6-pixel wide English glyphs.  ](*,)

dshadoff

  • Full Member
  • ***
  • Posts: 175
Re: Xanadu II Translation Development Blog
« Reply #141 on: December 27, 2015, 01:01:50 PM »
I appreciate all the thought that's going into this, but...

All other RPG game where this has happened, have resorted to short-form names - at least within the context of the weapons shop.

Is this not an option ?  Or just an option you'd prefer not to entertain ?

-Dave

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #142 on: December 28, 2015, 03:04:21 AM »
All other RPG game where this has happened, have resorted to short-form names - at least within the context of the weapons shop.

Is this not an option ?  Or just an option you'd prefer not to entertain ?

It's definitely an option, and I can certainly understand why translation teams do it, but take a look at this example of what's going on ...




Yes, I could just ask SamIAm to abbreviate both examples to "Lethr Armor" and "Dragn Slayr" ... but it's not a nice work-around, and it doesn't fit with the care-and-attention that Falcom put into these games.

If I can find a reasonable solution to avoid it then, personally, I'd find it pretty embarrassing (as a professional programmer) to have the game display the name of the game's legendary weapon, the weapon that gives it's name to the entire series of games, as "Dragn Slayr" instead of "Dragon Slayer".

Unlike Xanadu 1, Xanadu 2 already contains an 8x12 font that it uses for some character's names, for some special graphical characters, and for that item description in the Weapon Shops (which is why the spacing is almost correct).

It only loads up that font during the game, so I couldn't use that memory for the main English font (because it has to work in the Main Menu), but it does mean that I've got some space that I can use for a condensed font.

If I can change the game to use 1-pixel text coordinates, then it opens-up the possibility to switch the current bi-width English font into a fully variable-width font ... which will look nicer.

IMHO, the current font looks pretty good, and I've tweaked the glyphs to give them a consistent spacing ... but, to-be-honest, that has made the outlined versions in Xanadu 1 overlap so much that they look terrible!

I've spent over 600 hours, so far, hacking these games and recreating an important chunk of Falcom's toolchain, and it would be a shame to take the "easy" solution now just to save a few more hours of trying to do it "right".
« Last Edit: December 28, 2015, 03:05:04 PM by elmer »

dshadoff

  • Full Member
  • ***
  • Posts: 175
Re: Xanadu II Translation Development Blog
« Reply #143 on: December 28, 2015, 06:42:51 AM »
Fair enough.
Well, there are also other possible solutions too...

One could have the grid layout of the bottom line adjusted to make a bit more room for the weapon name...

But another one I just thought of, which may look even better than the original....
If possible, switch the location of the weapon name (3rd line) with the power up/down report ("Attack +12" on top line).  That layout would afford more room for the weapon name, and would probably make more sense to the user.

I also have a pretty good feeling that the box on the third line wouldn't be overflowed by "Attack +12" or whatever other possible text could show there.  And even if it couldn't handle all lengths, these phrases would be better candidates for shortening than the weapon names.

Not sure how much work this would be, but given what you've already achieved, it may not be so difficult.

-Dave
« Last Edit: December 28, 2015, 06:44:43 AM by dshadoff »

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #144 on: December 28, 2015, 12:42:31 PM »
One could have the grid layout of the bottom line adjusted to make a bit more room for the weapon name...

I thought about this, but I'm a bit worried to rely on it because I'm not sure what the maximum width of the numbers in the other 2 boxes is.


Quote
But another one I just thought of, which may look even better than the original....
If possible, switch the location of the weapon name (3rd line) with the power up/down report ("Attack +12" on top line).  That layout would afford more room for the weapon name, and would probably make more sense to the user.

I also have a pretty good feeling that the box on the third line wouldn't be overflowed by "Attack +12" or whatever other possible text could show there.  And even if it couldn't handle all lengths, these phrases would be better candidates for shortening than the weapon names.

Now this is a great idea, I wish that I'd thought of it myself!  #-o

I'm currently trying to get the 1-pixel coordinates working, but even if I do, I'd favor trying to shift around which box is used for which string anyway ... it's a much more logical layout for a Western game player.

Thanks!  :D
« Last Edit: December 28, 2015, 03:05:25 PM by elmer »

Black Tiger

  • Hero Member
  • *****
  • Posts: 11242
Re: Xanadu II Translation Development Blog
« Reply #145 on: December 28, 2015, 01:58:13 PM »
Yeah, purely from a game player perspective, I'd be much more comfortable with the item names up top.
http://www.superpcenginegrafx.net/forum

Active and drama free PC Engine forum

IvanBeavkov

  • Full Member
  • ***
  • Posts: 154
Re: Xanadu II Translation Development Blog
« Reply #146 on: December 30, 2015, 06:02:18 AM »
elmer, I just want to say I find this thread immensely interesting. I just love reading the details about how the game was built. It makes me wish I had the time to devote to learn how to do this.

Keep up the good work!

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #147 on: December 31, 2015, 06:47:04 AM »
elmer, I just want to say I find this thread immensely interesting. I just love reading the details about how the game was built. It makes me wish I had the time to devote to learn how to do this.

I'm glad that I'm not just boring everyone!

IMHO Falcom really put these games together well, I think that they're a great example of the best-practices in Game Engine design for the time.

It's been interesting to get a look behind-the-scenes as I've been hacking into the game, and to get an appreciation of just how good Falcom were at putting this kind of game together after they'd spent nearly 10 years of putting together the "Dragon Slayer" and "Ys" series on 8-bit and 16-bit home computers.


Yeah, purely from a game player perspective, I'd be much more comfortable with the item names up top.

It took a while to figure out how Falcom were setting up the positions of the text sprites, but here you go ...




There are a lot of different text messages that go into that "Attack +12" string area ... and the number itself can be up to 6 digits long.

I think that I'm going to need that 6x12 font in order to keep any abbreviated text messages understandable.  :-k

dshadoff

  • Full Member
  • ***
  • Posts: 175
Re: Xanadu II Translation Development Blog
« Reply #148 on: December 31, 2015, 07:11:05 AM »


It took a while to figure out how Falcom were setting up the positions of the text sprites, but here you go ...




There are a lot of different text messages that go into that "Attack +12" string area ... and the number itself can be up to 6 digits long.

I think that I'm going to need that 6x12 font in order to keep any abbreviated text messages understandable.  :-k


Wait... did you say "text sprites" ?
If they're using sprites, you need to know that beyond 16 sprites on the same scan line, there will be hardware flickering.  One more difficulty with longer text phrases.

elmer

  • Hero Member
  • *****
  • Posts: 2153
Re: Xanadu II Translation Development Blog
« Reply #149 on: December 31, 2015, 07:45:44 AM »
If they're using sprites, you need to know that beyond 16 sprites on the same scan line, there will be hardware flickering.  One more difficulty with longer text phrases.

Whoops, I guess that I wasn't clear enough.  :oops:

It is just text rendered into a large VRAM bitmap, which is then displayed as sprites.

Falcom are allowing for 128x16 for the weapon name (4 sprites), and 192x16 for the description/benefit message (6 sprites).

The box at the bottom only has room for approx 88 pixels of the description/benefit message, so I've left the extra 2 sprites up at the top of the screen where they won't get in the way.

Falcom are actually using the same low-level text rendering code for those strings as all the other text in the game, but they're setting a flag to bypass all the normal text cursor positioning code, and they're just using lower-level 4-pixel-sprite-strip position coordinates.

That makes it easier for them to control the exact pixel positioning for single-line messages rather than having to abuse the higher-level teletype-style console text code that handles the normal game text.

It wasn't that hard to find once I saw that the text was broken ... but it means putting in a bunch of small fixes for every string that's displayed in that way, rather than the clean fix-it-all-in-one-place that worked for the other game text.