Author Topic: Remember that Tengai Makyo Ziria translation project?  (Read 3177 times)

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #60 on: February 19, 2015, 03:22:51 PM »
OK, I changed about ~9 strings to using this idea temporarily. E.g.

Quote
[07]<YELLOW>Townswoman<brk><\r>
<WHITE>IHearThe<GREEN>Mayor<WHITE>IsHaving<\r>
someTrouble.YouOughtTo<\r>
goLendHimAHand.<\p>His<\r>
houseIsAtTheNorthwest<\r>
cornerOfTown.<\p><\0>

When I recompressed before the change, I had 462 bytes free, afterwards, free space kicked up to 482 - I freed up another 20 bytes. So far it seems to incrementally free up space. There are 101 strings in full, am not gonna test anymore, but yeah, it seems to work at first glance. I am hoping NOT to have to do this, I wanna see how things look after I eliminate some "fat" (as Sam refers to it), repetitious restating of the character's name/description, etc. and eventually even packing large contiguous text blocks with pointer re-computation (I've done it before with Xak III).

EDIT: Did some more (maybe another 10 or so), and kicked up free space to 514 bytes. One sentence at times caused no change, but then the next one freed up 12 bytes as a result, etc. So, overall, it continues to free up space despite cases of no benefit, etc. Ultimately you need a big enough block to compress for it to be worth it and be able to hack the print routine to respond properly to strings encoded like this... I would have to do more tests to determine how worthwhile this idea would be to seriously consider implementing it. It will suck having to edit the script looking like that, but yeah... It's an option on the table that's somewhat feasible.

There's definitely some fat in my translation, and I think we could make some significant reductions without damaging any meaning. However, if the task is to cut the length in half, then losing some major detail will be inevitable.

Let's see what this 2.0 Everdrive can do.

Since Ys IV was accomplished and I think this can work with some trimming, I would not want to rely on this idea down the road. I think it's for impossible font hacks that Bonknuts has this in mind for. But yeah, we'll be alright as is for Emerald Dragon I suspect. I don't want to break compatibility.

Well, here's what he actually said:

Quote from: Bonknuts
as for modifying the print routine, you're heading back into an issue I hate...
finding free room for your replacement code
and making sure that room is always available
which is why I'm moving toward a custom system card
either one with more ram, or one with a modified rom
a modified rom one would only work for duo/scd units though. not the originals
The space encoding thing sounds kinda cool though. I mean, no dictionary needed or such

I think if there was no English font support whatsoever, and to add things like subtitles when the game is packed pretty tight like Ys IV, as a total last resort, then yeah, I guess if Bonknuts tells you for this or that game there is no way for him to do it, then a custom system card might be the only solution. Anyhow, we're not gonna need that for ED, can say that with confidence! I only need minor changes and I should be able to free up code space thanks to all the S-JIS processing stuff.
« Last Edit: February 19, 2015, 04:58:11 PM by NightWolve »

dshadoff

  • Full Member
  • ***
  • Posts: 175
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #61 on: February 19, 2015, 05:06:19 PM »
Maybe we should move the Emerald Dragon talk over to the Emerald Dragon thread...

But according to my extraction notes, most of the superblocks have relatively large empty spaces at the end (ie. data padded with 0xff until the end of the sector and/or memory bank - in some cases, well in excess of 1 KB).  A couple of them were tight, but most were not.

So if you're comfortable adjusting the pointers to use the entire available space - not just the original superblock size, that should probably be sufficient in most cases.

English should be more compressible than Japanese for this, but I recall that this Japanese was fairly dense, since kana were single-byte.

I'd really rather keep to the original hardware platform - or AT LEAST a version of the original platform that was native (ie. promote CDROM to SuperCDROM would be acceptable).  Targetting specialized hardware is like targetting emulators only.

I believe that the Arcade Card is actually already used by Emerald Dragon (if attached), so even if it were a good place to store English text, it likely wouldn't work in this case (unless the original code were turned off).  Easier to just try to work with what we have, until we reach an insurmountable roadblock.
« Last Edit: February 19, 2015, 05:16:17 PM by dshadoff »

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #62 on: February 19, 2015, 05:16:04 PM »
I'd really rather keep to the original hardware platform - or AT LEAST a version of the original platform that was native (ie. promote CDROM to SuperCDROM would be acceptable).  Targetting specialized hardware is like targetting emulators only.

Right, and I believe Bonknuts has done CDROM to SuperCDROM hacks before.

SamIAm

  • Hero Member
  • *****
  • Posts: 1835
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #63 on: February 19, 2015, 07:19:21 PM »
I'd really rather keep to the original hardware platform - or AT LEAST a version of the original platform that was native (ie. promote CDROM to SuperCDROM would be acceptable).  Targetting specialized hardware is like targetting emulators only.

Believe me, I totally get where you're coming from. I want as many interested people to play a translation as possible. At this point, though, I just have to support the idea of using a new system card. Here's why:

1. Some games may fit an English script and new code just fine. Others might fit with some expertise and elbow grease. But others, unfortunately, look pretty hopeless. Tengai Makyo 2, for one. Legend of Xanadu for another. It's not that you can't get them to show any English text at all, don't get me wrong. But if the new English script and print routine have to fit in the framework of the original Japanese script, then lines like this:

//...An orphan.\n

//Now don't get any bright ideas.
//We've got enough mouths to feed as it is!

...will have to become something like this:

//A n    O r p h a n !

//F o r g e t   i t .
//W e ' r e   t o o   p o o r !

2. If you have to have a special card for one translation, you might as well have it for others, too, even if they don't necessitate it quite as much. If it speeds up the hacking in any significant way, I'm all for it. How many PCE-CD games have been hacked to completion in the last 10 years?

3. Even if you have to stick in a special new system card, the game will run on your real hardware exactly as it would otherwise. The only difference will be that you'll see a more fleshed out script and/or a better print routine and/or a prettier font. I know you know this, but I just want to stress that the only barrier here is the price of the card.
« Last Edit: February 19, 2015, 10:00:51 PM by SamIAm »

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #64 on: February 19, 2015, 10:02:05 PM »
Fair points, Sam. I think what made David nervous was forwarding the idea that we needed it for Emerald Dragon. We should be able to manage OK, plus there's no timetable for when something like that would ever be designed/released either, so...

Now, when it comes to other games that you mentioned, sure, I guess. If Bonknuts ever produces something like this, or if you guys try to lobby krik while you got the chance (the window is gonna close fast since he's 2-3 months from his TurboEverdriveV2.0 release), then great, I guess it would be useful down the road. Maybe it's just a matter of hacking the game to go from SCD to ArcadeCD if krik declines anything specific to what Bonknuts has in mind, but adds ACD support to his flashcart as stated. I do wonder though if there might be a temptation to go the easy route with it first when a traditional hack job might be sufficient enough and thereby break compatibility when you didn't have to... But yeah, Bonknuts would be the best person to tell you if there's no other way out of the situation but to go with a custom system card.

A big reason why you haven't seen PCE CD games hacked to completion in the last 10 years is because unlike SNES, the system just didn't gain the much larger user fanbase that it did. I managed 2 RPG translation projects (XakIII, Ys IV) and Ys IV was with a big team (Neill Corllett and David Shadoff as programmers besides me, 2 translators, and other people, 5-6 for varying lighter stuff, etc. NOT counting voice actors for the English dub). That was a tough project and I took a break from the system because of text codec usage, lack of font hacks and not being able to handle that myself; I didn't think I'd ever return, preferring to focus on Falcom Windows PC games (of course, I got screwed big time given the people I made the mistake of working with...). Neill was done with the system, few like him were around, and I was proud to have gotten at least another a feather in my cap with Ys IV.

But yeah, overall, there aren't enough people in the fanbase to come forward and start hacking some of these remaining CD RPG games. The lack of folks translated to lack of tools [debugging or otherwise] for many years, lack of documentation, lack of more computer wizards on our "team" like Bonknuts, or EsperKnight, etc. Compare that to the plethora of tools built around SNES because of the attention that it got which fostered a movement trying to translate EVERY LAST Japanese game ever made! We just don't have that in our little corner and that's part of why I wanted to work on PC Engine games when I got started, I was enough of a fan, saw that there was a vacuum I could fill and be the "first" at something, a CD RPG project (Xak III), etc.

Anyway, blah blah, you know what I'm trying to say here. If the system was as popular as SNES, lots of shit would've gotten done already and likely without having to go as far as a custom system card to boot is my guess!

Question: Is upgrading the game to Arcade Card not enough ?
« Last Edit: February 20, 2015, 12:48:49 AM by NightWolve »

SamIAm

  • Hero Member
  • *****
  • Posts: 1835
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #65 on: February 20, 2015, 02:31:21 AM »
Fair points, Sam. I think what made David nervous was forwarding the idea that we needed it for Emerald Dragon. We should be able to manage OK, plus there's no timetable for when something like that would ever be designed/released either, so...

I did get in touch with Krikzz just a little, and while there hasn't been anything conclusive, he seemed optimistic about this expanded memory idea working with the Turbo Everdrive v2.

If you guys can pull Emerald Dragon off nicely as-is, that's great. Like I said, I would be very happy to see this available for as many people as possible. If that idea motivates you, then go for it.

Quote
I do wonder if there might be a temptation to go the easy route with it first when a traditional hack job might be sufficient enough and thereby break compatibility when you didn't have to...

I understand your concern, but if a new expanded-memory card comes out and someone is at all inclined to use it for translation purposes, I think that they should do so regardless. The PC Engine CD hasn't gotten a damn thing in years. I'm not saying that there haven't been some flaky translators, myself included, but these projects begin and end with hacking work. Esperknight, Bonknuts and dshadoff have the skills to do the deep surgery on these games, but they don't have the time to put to make everything fit back together again inside the 256k of RAM currently available and have it work throughout an entire RPG.

Meanwhile, aging nostalgic gamers are dropping huge money on even the loosest of hueys. I think they can justify setting aside a little for a Turbo Everdrive v2, or possibly even a cheaper alternative manufactured by this community, in exchange for access to games they had no hope of playing before. Just trust me, if parking the translated script and all the new code in some expanded RAM cuts the hacking time in half, it's what you want the hackers to do.

Motivation is the biggest factor. If making Emerald Dragon work on the normal hardware drives you, then go for it. If using expanded memory to simplify the hacking gives others motivation, then they should do that. I'm just frustrated of seeing so little happening around here.
« Last Edit: February 20, 2015, 02:35:56 AM by SamIAm »

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #66 on: February 21, 2015, 09:42:17 PM »
Hey Bonknuts, when/if you see this. What about getting by with upgrading the game to Arcade Card ? I'm trying to see, is that not enough for these real difficult cases, if not, why not, etc. and so it must be something custom (like another Games Express) ?

SamIAm

  • Hero Member
  • *****
  • Posts: 1835
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #67 on: February 22, 2015, 01:44:05 AM »
Hey Bonknuts, when/if you see this. What about getting by with upgrading the game to Arcade Card ? I'm trying to see, is that not enough for these real difficult cases, if not, why not, etc. and so it must be something custom (like another Games Express) ?

Here's something he posted when I asked the same question:

Quote
The ACD has a the drawback that it lacks memory that you can execute code from. So you still need to find free areas in the original CD ram layout, etc. It would have been nice if they had added something like 8k of direct accessible ram, if only for us translators/hackers. I've been wanting a new system card with more direct code accessible ram for a while now, for translations. Tail Chao's hucard can do this. Its specific mapper can allow 512k of ram easily as well as the original system card 3.0 ram. It's a real card and mednafen started adding support for it. That would be extremely ideal for translation hacking. Though a card made from the ground up would work as well (no mapper needed, just a few discrete chips to handle memory layout. Also mirror the first 1k of the ram to open bus space of bank $ff. That would allow the hooks to be static/fixed in memory layout and code to map in new banks of hook code from there).

elmer

  • Hero Member
  • *****
  • Posts: 2154
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #68 on: February 22, 2015, 04:13:54 AM »
[07]<YELLOW>Townswoman<brk><\r>
<WHITE>IHearThe<GREEN>Mayor<WHITE>IsHaving<\r>
someTrouble.YouOughtTo<\r>
goLendHimAHand.<\p>His<\r>
houseIsAtTheNorthwest<\r>
cornerOfTown.<\p><\0>

It will suck having to edit the script looking like that, but yeah... It's an option on the table that's somewhat feasible.
Errr ... isn't that when you ask your programmer to write a pre-processor that takes the human editable text and spits out the compression-ready text? It would save a lot of human-error that would otherwise occur.

It looks like it's saving you a few bytes of RAM and a little compression space ... but then you need a customized printing routine.

Or a customized compressor, or a customized whatever-other-neat-gizmo.

Which gets back to code space and the Arcade Card ...

Here's something he posted when I asked the same question:

Quote
The ACD has a the drawback that it lacks memory that you can execute code from. So you still need to find free areas in the original CD ram layout, etc.
And that's where your task as post-mortem translators really sucks, and why I have such respect for what you guys do.

You've got to analyze the game, completely without source, and figure out where there is some spare space ... and how to actually get code/data to load into that spare space.

I can think of some strategies ... but they're all difficult and nasty work ... like the old C64 and Amiga hackers that would rip games off tape/floppy and actually rewrite the games' loading code so that they could bypass copy protection and release compilation disks.

So when David said ...

But according to my extraction notes, most of the superblocks have relatively large empty spaces at the end (ie. data padded with 0xff until the end of the sector and/or memory bank - in some cases, well in excess of 1 KB).  A couple of them were tight, but most were not.

So if you're comfortable adjusting the pointers to use the entire available space - not just the original superblock size, that should probably be sufficient in most cases.
That sounds much more like the kind of CD/loading system that I'm used to ... and that you may actually have some free space to play with ... but, and it's a huge "but" ... then you are starting to enter the realm of remastering the CD image.

That would give you a lot of added flexibility, but comes at the cost of needing someone to write the tools.

And good tools programmers have always been hard to find ... it's vital but completely un-glamarous work.

Anyway, I'm rambling rather than actually contributing anything useful, so I'll stop bombing this thread with my distractions.
« Last Edit: February 22, 2015, 06:18:27 AM by elmer »

dshadoff

  • Full Member
  • ***
  • Posts: 175
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #69 on: February 22, 2015, 06:08:37 AM »
Hey Bonknuts, when/if you see this. What about getting by with upgrading the game to Arcade Card ? I'm trying to see, is that not enough for these real difficult cases, if not, why not, etc. and so it must be something custom (like another Games Express) ?

If we are talking about Tengai Makyo (the game meniotned in the thread), there are two versions of it:

1) CD, which is what I have always counselled to base extractions from, because you can easily extend RAM usage into the SCD area, which will be untouched.

2) SCD, which was a re-release a few years later, which tried to address load times.  I would advise to try not to use this one, as it's not clear what its memory footprint is.


If on the other hand, we are talking about Emerald Dragon again/still (even though it has its own thread), then there a few problems with Arcade Card, which stem from the complexity of CDROM games in the first place:

1) According to my information, it already uses Arcade Card memory if it detects it.  Disabling such use would be a LOT of trouble, and then would come the problems of the repurposing

2) If a game did not use ACD memory, then great, the ACD can be used.  But ACD storage would really only be viable as text storage, because it can't be used to execute code from.  So it wouldn't solve problems like print functions.  Also, you would still need ACD access routines which must reside in main memory, and that memory may not be easy to find.

3) Using ACD memory for text storage could be theoretically OK, but still has some issues which need to be surmounted:

- First, it needs to be stored on the disc in the first place.  You need to find that storage.  And the way that code and data are stored on the disc currently is like a gigantic ROM file, which gets paged into memory a few blocks at a time.  You can't just shift something over without huge consequences.  You could theoretically extend the end of the CDROM data track, but you'd better hope that existing CDROM access code in the game doesn't reference MM:SS:FF or LBA addresses on disc - rather that they would address tracks (which would be fine, as they would shift).

- While the text could be stored in ACD as either compressed or uncompressed, it would need to be copied to main memory prior to print.

-- If compressed, you would need to copy to the same compress buffer in main memory as would have previously been used (ie. a text page loaded from disc), and decompress as per usual.  This is probably the more difficult choice.

-- If stored in ACD as uncompressed text, then the string locate (or 'pre-print') function could be replaced with ACD accesses.  In such a case, instead of addressing a text page to find compressed data, and then uncompressing into a RAM buffer, and sending the start of the string to the print function, there could instead be a lookup (based on superblock, sub-block, and string #) to a lookup table to find out ACD location instead.  Then all that would be needed would be a block-move into the uncompress buffer (to the start of the buffer !), and send that address to the print function.  Of course, this becomes complicated if the superblock-subblocks are expected to already be decoded into the uncompress buffer at the time of print (a very real possibility).  Then you wouldn't have the hook to the uncompress to support you, and you would need to worry about how big the uncompressed strings are in memory - and whether your uncompress buffer was big enough to hold the block in the first place.


So, it's not so easy.... because even imagining all of these problems, and devising theoretical techniques to deal with them is really only about 30% of the work.  The techniques would then need to be implemented, debugged, and as is always the case, unforeseen events happen along the way.

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #70 on: February 22, 2015, 04:23:37 PM »
Errr ... isn't that when you ask your programmer to write a pre-processor that takes the human editable text and spits out the compression-ready text? It would save a lot of human-error that would otherwise occur.

Yeah, I can do that eventually if I needed to really try to this idea, bit more grunt work, but yeah. Really, I'd have to cause I have auto-wrapping code which relies on spaces to wrap strings depending on the dimensions of message boxes. Wrapping at the click of a button would break if I edited every string like that, so it would have to be in the pre-processing, detokenizing phase prior to compression.

Quote
It looks like it's saving you a few bytes of RAM and a little compression space ... but then you need a customized printing routine.

Compression space is saved, right, and you have to hack/mod the print routine a bit. The changes seem feasible enough so I think it's a pretty good idea if you've got a hacker good enough to mod the print routine and space is tight for whatever game. So, the idea is out there for others to try. Bonknuts found it interesting when I mentioned it to him on Facebook and from my quick test it looks like it works.

Here's something he posted when I asked the same question:
Quote from: Bonknuts
The ACD has a the drawback that it lacks memory that you can execute code from. So you still need to find free areas in the original CD ram layout, etc. It would have been nice if they had added something like 8k of direct accessible ram, if only for us translators/hackers. I've been wanting a new system card with more direct code accessible ram for a while now, for translations. Tail Chao's hucard can do this. Its specific mapper can allow 512k of ram easily as well as the original system card 3.0 ram. It's a real card and mednafen started adding support for it. That would be extremely ideal for translation hacking. Though a card made from the ground up would work as well (no mapper needed, just a few discrete chips to handle memory layout. Also mirror the first 1k of the ram to open bus space of bank $ff. That would allow the hooks to be static/fixed in memory layout and code to map in new banks of hook code from there).

Ah, OK, that answers that question. I see.

If on the other hand, we are talking about Emerald Dragon again/still (even though it has its own thread), then there a few problems with Arcade Card, which stem from the complexity of CDROM games in the first place:

Relax. ;) We'll be OK with ED with trimming and contiguous text block packing, I think. The interest I took in the discussion here is about future games and this idea I heard from Bonknuts, his desire to produce a custom system card and willingness to break compatibility to get difficult projects done, etc. I guess if there's no other way, why not. I just would hope that the easy route isn't taken first if such a card was produced. I also wondered about why the Arcade Card couldn't be enough if you *really* had to go this route. Again, for me that was about other/future projects, a general debate. Anyway, I read the rest of your post, so that answers the question as well.

Dicer

  • Hero Member
  • *****
  • Posts: 1905
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #71 on: February 22, 2015, 05:28:05 PM »
Pardon my ignorance on all things translation hacking based, but how would they have gone about doing this say they had actually ported the game back in the day?

I guess I'm not understanding the reason that there is such a limitation, someone maybe lay it out in simpleton terms, as I don't get why replacing one wall of text with another would be such a stumbling block.

Thanks

NightWolve

  • Hero Member
  • *****
  • Posts: 5277
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #72 on: February 22, 2015, 08:24:23 PM »
By ported, you mean if a US publisher bought the Rights to localize and release it ? The Japanese company/IP holder will give you the full source code if a deal is made, if they decide to give you permission to do it. Or, you let their own programmers do it, so they give you the Japanese script, you translate it, give an English script back to them, and they will rebuild the game with it - that's how it's done with that criminal outfit X.X.XSEED Games and Falcom actually. XSEED doesn't have much of a staff for reprogramming the game, so Falcom does it for them based on the deal in place. They do have a contract programmer to port the game on Steam after Falcom inserts their translated results, though.

So yeah, by being able to recompile the videogame having all the source code, you can grow/change it far more easier instead of how we as fans are forced to do it by hacking the final binary output (hacking = what's known as reverse-engineering, since we don't have access to the original source code).

Dicer

  • Hero Member
  • *****
  • Posts: 1905
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #73 on: February 23, 2015, 02:31:35 AM »
By ported, you mean if a US publisher bought the Rights to localize and release it ? The Japanese company/IP holder will give you the full source code if a deal is made, if they decide to give you permission to do it. Or, you let their own programmers do it, so they give you the Japanese script, you translate it, give an English script back to them, and they will rebuild the game with it - that's how it's done with that criminal outfit X.X.XSEED Games and Falcom actually. XSEED doesn't have much of a staff for reprogramming the game, so Falcom does it for them based on the deal in place. They do have a contract programmer to port the game on Steam after Falcom inserts their translated results, though.

So yeah, by being able to recompile the videogame having all the source code, you can grow/change it far more easier instead of how we as fans are forced to do it by hacking the final binary output (hacking = what's known as reverse-engineering, since we don't have access to the original source code).


There it is...thanks for the explanation

Bonknuts

  • Hero Member
  • *****
  • Posts: 3292
Re: Remember that Tengai Makyo Ziria translation project?
« Reply #74 on: February 23, 2015, 03:35:22 AM »
Dshadoff pretty much hit on everything. AC memory can help with scripts (doing a dictionary compression scheme or just tokens to larger strings) and font storage. Finding free space in the ISO is probably the easiest thing as most data tracks are extremely fragmented (rarely is this not the case). And I could write a boot handler to load your AC stuff on the games initial boot (easy to do). But like Dave said, it's a bi-compatible AC game. And that means making absolutely sure you hunt down all instances of AC detection routines (some games, like Gulliver Boy, periodically do an AC check and change appropriately). That's too bad.

 Dave said there is sometimes free space near the of compressed blocks? That would be a great place to start; figure out that pointer system and try to maximize that area for your new compressed blocks. Maybe in conjunction with this compression scheme you already have in place (the no-space run?).