PCEngineFans.com - The PC Engine and TurboGrafx-16 Community Forum
Tech and Homebrew => Turbo/PCE Game/Tool Development => Topic started by: Bonknuts on June 02, 2016, 09:18:25 AM
-
I know some people have duplicated or written their own CD access code for the units, outside of using the system card, but most of them are probably lost over the ages.
Notably, Charles MacDonald had a ADPCM source lib that could be used for hucard projects (enchanced hucards).
I'm really interested in something like this. A game that's on the hucard, but is meant for the CD system (you supply the CD to for the audio, etc).
Anyone else experiment in do this? Or interested in this?
-
Notably, Charles MacDonald had a ADPCM source lib that could be used for hucard projects (enchanced hucards).
Having an ADPCM library could be useful. :-k
Then you could just run a huge game off of a Turbo Everdrive v2 and still have some ADPCM sound ... but I don't think that you could make it stream the ADPCM, and so that would limit its usefulness.
As for running a CD game off of a custom HuCard ... I can't see the advantage myself, but there's obviously something that you're seeing in the idea that I'm missing.
It would just seem to bump up the costs in duplicating a homebrew game for little real advantage.
Now ... having Hudson's fast-access CD code disassembled so that we could all use it in CD games ... that is something that I can see being a huge benefit to homebrew. :wink:
-
Now ... having Hudson's fast-access CD code disassembled so that we could all use it in CD games ... that is something that I can see being a huge benefit to homebrew. :wink:
That I already have and use it from time to time, if you want it. It's 8k because it has other new functions. I didn't strip them out.
I can't see the advantage myself
Hacking hucard games for ADPCM sound FX and to use an audio CD if present for music. Soo much easier than trying to convert a hucard to fit to CDRAM form factor, bank layout, etc. Just through in the hucard and audio CD, and be done with it ;)
Or make bi-compatible hucards. Function as normal hucards, or enhanced if used on a CD system (the base 64k is available to hucards, which is nice too).
-
That I already have and use it from time to time, if you want it.
Yes, please.
Or make bi-compatible hucards. Function as normal hucards, or enhanced if used on a CD system
Or add extra levels on the cd...
(I'd love to see tuokos chuck no-rice game with loadable characters to use)
-
Here's the lib that I ripped from Seiya Monogatari: http://pcedev.net/CD_read_lib/CD_read_lib.zip
Just to note: For some reason the CD lib routine zeroes out the third upper most byte for the LBA offset. So you're limited to 128megabyte address range within a data track. Can be fixed/altered, but I didn't bother changing it.
The CD read lib is fast; much faster than the system card one. I listed what I figured out what the other library entries do, but I didn't document their arguments, since I was only really interested in the CD read routines (CD to VRAM one is documented).
For the curious, here's the list of lib functions:
4000 initialize system environment
4003 play audio track (???)
4006 internal (reinitialize ???)
4009 wait_vblank (ver1)
400c LZ decompress to ram
400f mem->mem DMA
4012 setup for 404b
4015 NULL (jumps to itself)
4018 LZ decompress to vram
401b header/prep setup (???)
401e LZ decomp ADPCM to vram
4021 bram wait loop (???)
4024 LZ decomp ADPCM to mem
4027 MEM->VRAM DMA
402a 402b + wait_vsync(ver2)
402d VCE pal arg update
4030 VCE pal write
4033 wait_vsync(ver2)
4036 update satb
4039 CD_READ to vram
403c CD_READ to ram
403f another CD_READ
4042 CD_audio/ADPCM related
4045 NULL
4048 NULL
404b (long) time delay loop
404e internal reinitialize
4051 internal update (???)
4054 interal check (???)
4057 get status (???)
405a internal reinitialize
405d update INT jmp addr(hsync branch)
4060 PSG reinitialize
-
Thank you tom. And I don't see 128M per data track as a problem. Yet.
-
Cool. The data limit is easy to fix. Not sure why they did that, but long seek ranges probably present a problem (stall more?).
Also, the LZ routines are decent. Especially the ring buffer stuff (port to port, or local to port). I could probably adapt LZSS compressor for it. Though I've been using PuCrunch (aplib, and one other one), since they compress better - but decompress slower. That, and I saw some other things that elmer and touko posted about fast decompress routines (LZ4 ?).
-
There is a PCe game which don't use a sat_b transfert routine and write sprite attributes directly in VRAM ??
-
Also, the LZ routines are decent. Especially the ring buffer stuff (port to port, or local to port). I could probably adapt LZSS compressor for it. Though I've been using PuCrunch (aplib, and one other one), since they compress better - but decompress slower. That, and I saw some other things that elmer and touko posted about fast decompress routines (LZ4 ?).
Hmmm ... I'll have to take a look to see what they're doing.
It's always fun to steal good ideas from other people's code! :wink:
IMHO, you can avoid looking too hard at LZ4 ... but then, I'm somewhat biased. O:)
The current version of SWD5 that's in the Xanadu games is compromised in terms of performance by having to fit into the minimum space (approx 256 bytes), and to avoid zero-page optimizations because I didn't know what Falcom was using or relying on.
Allowing it to use $F8-$FF for a temporary block-transfer instruction call makes a number of other optimizations possible.
Anyway ... getting back to the System Card ... I finally got pissed-enough with the PDF of the Hu7 CD documentation that's on Tom's PCEDEV site, that I used Acrobat Pro to disassemble the thing back into individual pages, then edited every single page in Photoshop to remove the most nasty of the artifacts from the original scan to leave almost-clean pages, and finally reloaded it all back into Acrobat to be deskewed and OCR'd.
So, there's now a copy of the CD documentation that looks clean and reasonably-horizontal, and that contains "searchable text" so that you can quickly find particular function call documentation with a simple search.
It's still a fairly crappy scan, there's nothing that I can do about that, particularly in comparison to the rest of the system docs that are "out-there", but it's a million times better than not having the documentation at all! :)
Anyone that's interested can get it from here (for a while) ...
https://www.dropbox.com/s/yqxxee0893378nv/Hu7%20CD%20System%20Development%20Manual.pdf?dl=0
-
I'm not 100% positive (I could be wrong), but I Think there's already a block transfer instruction area at $26cc in the user area. It might be something HuC does, though.
The other 'neat' option I saw was setting it up in an empty area in the card memory. The code tom posted has a tia (?) instruction that gets modifed in place, iirc :)
-
I'm not 100% positive (I could be wrong), but I Think there's already a block transfer instruction area at $26cc in the user area. It might be something HuC does, though.
The other 'neat' option I saw was setting it up in an empty area in the card memory. The code tom posted has a tia (?) instruction that gets modifed in place, iirc :)
That's a pretty common practice with PCE devs (from what I've seen). Turns the Txx instructions into DMA instructions instead of fixed instructions. Txx, src,dest,len,rts.
I was just mentioning the compression stuff, because sometimes you want the slower, better compression ratio, and sometimes you need fast decompression with an OK ratio.
elmer: Is it ok if I host this at my site?
-
I'm not 100% positive (I could be wrong), but I Think there's already a block transfer instruction area at $26cc in the user area. It might be something HuC does, though.
Yep, that's just HuC, I'm afraid, and so not something that I want to count on. Although any compiler library for HuCard usage is going to have the same thing somewhere in memory.
In my case, I need the TII in zero-page memory while a file is decompressing in order to get a speed improvement.
That would allow the decompressor to use the TII's src and dst addresses as indirect pointers for short 1 byte (and maybe 2 byte) copies in order to avoid the subroutine call and TII overheads.
That's a pretty common practice with PCE devs (from what I've seen). Turns the Txx instructions into DMA instructions instead of fixed instructions. Txx, src,dest,len,rts.
Yep, Hudson really added some very useful stuff to the PCE's CPU.
I was just mentioning the compression stuff, because sometimes you want the slower, better compression ratio, and sometimes you need fast decompression with an OK ratio.
Yep, I agree. :wink:
I'll have to take a look at that code more deeply, but I'll be a bit surprised if it's a lot faster than a speed-optimized version of SWD.
We're just talking about variants of the same LZSS concept. They both end up doing lots of memory-to-memory copies.
I'm curious if they're really keeping a separate ring-buffer ... that's pretty slow!
But what I'm more curious about, is what compromises they made in order to decompress directly into VRAM or ADPCM memory.
If they're keeping a 4KB ring-buffer in RAM, then that may negate a lot of their potential performance improvements over just decompressing into RAM and then copying to VRAM.
OTOH ... it is probably the most sensible way to do it if you're dealing with the limited 8KB RAM in the base PCE.
elmer: Is it ok if I host this at my site?
Sure, that would be great. I have no idea how long it will last on my dropbox. :)
I should really write out a 2nd PDF file with just the section about the BIOS calls ... that's what I want to refer to all the time, not the stuff at the start or the end of the document.
-
If they're keeping a 4KB ring-buffer in RAM, then that may negate a lot of their potential performance improvements over just decompressing into RAM and then copying to VRAM.
I'd rather go for a smaller ring buffer like 4k, even for a CD project, if decompressing to a port. 256k is already tight as it is, and keeping free work mem for other things is important (speeds things up).
While I only tested Pucrunch, which IS more than just LZSS, the LZ window size didn't impact much from the tests that I did. Even windows as small as 512bytes were very efficient. While I realize plain LZSS wouldn't have the same results for a window that small, 4k is decent for LZss. Note: I have no idea how big the LZ window is in that CD lib.
I wonder how much performance increase one could get with specialized code using the arcade card registers, for C.
-
I'd rather go for a smaller ring buffer like 4k, even for a CD project, if decompressing to a port.
Sometimes I feel like Morhpeus in "The Matrix".
I can only say ...
https://www.youtube.com/watch?v=uAXtO5dMqEI
To paraphrase ... "There Is No Ring Buffer".
The ring-buffer concept is an artifact of Haruhiko Okumura's need to find a way for his LZSS.EXE program to LZSS compress floppy-disk sized files within the confines of a 64KB-or-less computer.
There is no need for the concept within the actual LZSS algorithm, or any of the derivative compressors/decompressors that don't share the same limitations.
It is just a way to access the last 4KB of decompressed data ... the "window".
If you're decompressing directly into RAM, then you automatically have that "window" available to you without the need for any ring-buffer processing.
****************
My SWD code isn't either unique, or particularly smart.
I've followed the same kind of logic paths that lead to PuCrunch ... how to more-efficiently store the basic LZSS repeat-count & offset pairs.
You can just decompress directly into the destination buffer ... no extra memory is needed, and there's particularly no need for that 4KB ring-buffer.
The only exception that I can see is when decompressing to VRAM/ADPCM RAM, and then, IMHO, it's trivially easy (and much faster) to decompress directly into RAM and then copy the data to VRAM afterwards.
Since the LZSS "window" is usually only 4KB anyway, it doesn't really hurt you much to just split your data-to-be-compressed into 8KB-maximum chunks, and to process them separately. That also means that the output will fit within a single bank if started at the beginning, or 2 banks if you want a "stream" of decompressed data.
That's the approach that Falcom take in Xanadu 1 and Xanadu 2 ... and I agree with them.
The LZSS window size has only a limited effect upon the compression (statistically), which is why longer offset codes are allocated to matches that are further away, and shorter offset codes are allocated to closer matches.
The question is whether you even bother processing RLE sequences at all.
Since an RLE sequence is both rather rare, and easily compressed within the regular LZ77 scheme, I decided "no".
-
To paraphrase ... "There Is No Ring Buffer".
The ring-buffer concept is an artifact of Haruhiko Okumura's need to find a way for his LZSS.EXE program to LZSS compress floppy-disk sized files within the confines of a 64KB-or-less computer.
The ring buffer very much exists, but only if you write the decode to use a ring buffer instead of a sliding window. The two work slightly differently. I modified Pucrunch's sliding LZ window to a ring buffer function so that I could decompress large chunks of data to vram without needing to decompress the whole thing into ram first (which I could for hucard projects because of the ram limitations). The ring buffer concept didn't exist inside the original decompressor.
If you're decompressing directly into RAM, then you automatically have that "window" available to you without the need for any ring-buffer processing.
Yes, to ram. Which is why I added "if decompressing to a port".
I rarely ever need to decompress anything LZ level compression to ram. Almost all assets are vram bound (tiles, sprites, but not tilemaps). Even if I did use it for text blocks, they'd be pretty small (easily 4k-8k). I wouldn't want waste sys card ram for more than that (have a larger buffer just to avoid using a ring buffer, for vram stuffs). One size doesn't fit all, but if you're decompressing to the ring buffer and vram at the same time, it's going to be faster since you're writing directly to vram - the ring buffer is just for history access. Faster than decompressing to the whole thing to local ram, and then re-copying it to vram. Of course if its a background process, and the objects are small enough, local ram decompression would be the better choice for obvious reasons (to avoid vram write pointer corruption).
-
The ring buffer very much exists, but only if you write the decode to use a ring buffer instead of a sliding window. The two work slightly differently. I modified Pucrunch's sliding LZ window to a ring buffer function so that I could decompress large chunks of data to vram without needing to decompress the whole thing into ram first (which I could for hucard projects because of the ram limitations). The ring buffer concept didn't exist inside the original decompressor.
I wasn't sure if you were one of those folks that still believed that the ring-buffer was a integral part of the LZSS algorithm ... which, it seems we actually both agree, it isn't. :)
It is an "implementation-detail", a method of keeping the history window in a fixed 4KB location as a work-around for when you don't have easy random access to the window through normal pointer manipulation.
As such, I totally agree that the method may have its place, when used by someone that understands the trade offs.
I just have some difficulty imagining how you'd write a good inner-loop that can deal with the window wrapping.
Are you using self-modifying "abs,X" or "abs,Y" to address the window in order to avoid "[zp],y"? :-k
I rarely ever need to decompress anything LZ level compression to ram. Almost all assets are vram bound (tiles, sprites, but not tilemaps).
Yes, that makes sense. It all depends upon your needs and your code design.
I suspect that Falcom's usage of so many compressed code segments may not be typical.
I did take a quick look at Anearth to see what it was doing, and found Hudson using a lot of compressed text and graphics segments, with multiple decompression runs for a single text-box.
Yuk! #-o
They were also using the preloaded ring-buffer technique that you have mentioned before, so I finally got to see that in action.
Given the values that they were preloading it with, and the time taken to do the preloading, I'm absolutely amazed that they thought/found that the benefits outweighed the costs.
-
Time to resurrect this old thread!
I'm pretty close to having the fast CD code from Xanadu 2 completely disassembled-and-commented so that it can be included as source code in homebrew games.
It's got replacements for the BIOS functions AD_RESET, AD_TRANS, AD_CPLAY, AD_STOP and CD_READ.
It's approx 2KB of code, and I can see some wasted space in there that could be optimized away to make it smaller ... for instance, there's still code in there to read from Hudson's Development Kit SCSI.
This opens up the possibility completely disabling the CDROM BIOS, taking over the whole system, and mapping RAM into bank 7 instead of ROM.
Does that idea interest the other programmers here?
What other BIOS functions would you really need (in practice) during a game's execution?
Certain things like CD_DINFO and CD_BASE come to mind ... but they could easily be done at boot time and the results could be cached before switching to the "replacement" BIOS.
Does anyone ever use things like CD_SEEK, CD_SEARCH, CD_SUBRD, or CD_PCMRD?
Wasn't there a mention of someone having source code for some BM_* backup memory replacements?
<EDIT>
I guess that there's also he question of AD_READ and AD_WRITE.
Has anyone found a practical use for those since the Super System Card came out and cured the worst of the PCE's RAM problems?
<EDIT2>
D'oh! #-o Of course ... AD_READ does still have a potential benefit, in that it can be used for background streaming of data off the CD.
It *might* just be faster to read a sector or two at a time from the CD directly ... or it might not.
Perhaps someone else has investigated the HuVideo player and knows how it handles its streaming?
-
The disassembly/analysis of the Xanadu2 CD fast code is now done, and it has been tested to confirm that it actually assembles correctly and matches the original code in the game.
So now it's on to reverse-engineering some of the missing functions from the CD BIOS, and then doing some optimization on the results.
I just took a look at the bank of auto-disassembled code that Bonknuts posted from Seiya Monogatari earlier in the thread.
Just to make sure that everyone understands ... the work that I've done is the next (and very laborious) step beyond that kind of automated disassembly.
The Xanadu 2 code has been investigated and commented, and is fully-ready to be modified and changed.
A quick look the the Seiya Monogatari code shows that it's based on the same Hudson source-code.
For instance this routine, from my Xanadu 2 disassembly, is clearly identifiable in the Seiya Monogatari code ...
;
; Send SCSI "Request Sense" command (used to clear drive's error status).
;
_retry_recovery:jsr _stop_scsi_bus ; Release the SCSI bus.
_error_recovery:jsr _init_scsi_bus ; Acquire the SCSI bus.
bcs _retry_recovery
.test_scsi_bus: ldx #$14
cly
.wait_scsi_bus: jsr _get_cd_status
and #$80
bne .proc_scsi_loop
dex
bne .wait_scsi_bus
dey
bne .wait_scsi_bus
; Timeout, retry.
bra _error_recovery
; Process SCSI states.
.proc_scsi_loop:jsr _get_cd_status
cmp #$d0
beq .send_command
cmp #$c8
beq .read_reply
cmp #$d8
beq .command_done
bra .proc_scsi_loop
; Send SCSI command.
.send_command: lda #< _request_sense
sta <_bl
lda #> _request_sense
sta <_bh
jsr _send_scsi_cmd
bra .proc_scsi_loop
; Read SCSI reply.
.read_reply: lda #< scsi_recv_buf ; Buffer for result.
sta <_bl
lda #> scsi_recv_buf
sta <_bh
lda #$0a ; # of bytes to read.
sta <_al
stz <_ah
jsr _get_scsi_reply
bra .proc_scsi_loop
; SCSI command done.
.command_done: jsr _get_exit_code
cmp #$00
bne _error_recovery
lda scsi_recv_buf + 9 ; Last byte of SENSE reply is either
rts ; an error code or a retry count.
;
; Use return code from _error_recovery to decide what to do.
;
_pick_next_base:cmp #$00
beq .try_swap_track
cmp #$04
bne .skip
bra .try_swap_track
.skip: cmp #$0b
beq .try_same_track
cmp #$0d
beq .try_same_track
cmp #$11
beq .try_swap_track
cmp #$15
beq .try_swap_track
cmp #$16
beq .try_swap_track
cmp #$1c
beq .try_same_track
cmp #$1d
beq .try_same_track
cmp #$20
beq .try_same_track
cmp #$21
beq .try_same_track
cmp #$22
beq .try_same_track
cmp #$25
beq .try_same_track
cmp #$2a
beq .try_same_track
cmp #$2c
beq .try_same_track
; Swap to alternate data track.
.try_swap_track:lda #$01
sec
rts
; Stay on current data track.
.try_same_track:cla
clc
rts
-
I'd love to see tuokos chuck no-rice game with loadable characters to use
It's already here for the two players mode :wink:
It works fine,but i don't know if it's slow or not, only tested on mednafen,you'll see it soon with the demo version .
-
It *might* just be faster to read a sector or two at a time from the CD directly ... or it might not.
Perhaps someone else has investigated the HuVideo player and knows how it handles its streaming?
From what I remember, and it's been years since I looked at it, it's reading directly from the CD data port. It writes updates to ADPCM ram, and after it reads out data from the CD port, it does a few things. And returns back to polls the CD status.
The strange thing I found, was the very tail end of the data block (2k I think?) is read out much slower than reading out the rest. I suspect because the MCU interface must be triggered at some point (by reading the CD port X number of times/bytes) and starts filling in the buffer from the front. It's only speculation, but I can't figure out why the last bit of data is read out slower. Some tests should be done, but I'm suspecting this is how the next sector read from the CD is triggered.
elmer: Do you have the disassembly of the system card?
-
elmer: Do you have the disassembly of the system card?
I've got the one that Dave Shadoff and Oliver Jolly (NeoGrad) did ... do you have anything more complete than that?
It's a great starting-point, but putting *meaning* to the code is still a laborious process.
-
It's a great starting-point, but putting *meaning* to the code is still a laborious process.
What version of bios is that?
Would it help to have an over-commented, re-assembleable version using names?
If so, I'll zip the files I have for banks 0-3, along with matching variable definitions, and mail them to you.
There's still stuff in them I don't completely understand, but most of it I got. I just wish I understoof the irq stuff better, like why there are 2 irqs for the adpcm stuff. (Top half/Bottom Half?)
PM me with an email address if you want it.
-
What version of bios is that?
It's version 3.00 ... which is the Japanese SuperCD BIOS IIRC.
Would it help to have an over-commented, re-assembleable version using names?
Absolutely!
I'm about 75% of the way through labeling/commenting the BIOS CD/ADPCM routines that aren't included in the fast CD code that I've already done, but it would be wonderful to have someone else's version of the same stuff to compare notes with.
As annoying as it's been ... I really did need to do it myself in order to get a good handle on the differences between the BIOS code and the fast-read code. That's the only way that I'll be able to merge the two methods together in order to save space in a new ROM-replacement library.
Having all the extra functions available that you've already done, and that I've not even looked at, may well open up the potential for that BIOS-compatible replacement that I seem to remember either you or Bonknuts talking about.
Personally, I'd just like to get to where I can have a fully-functional fast CD/ADPCM library that lives in RAM in $F000-$FFFF, and that leaves the rest of that bank usable by a game.
I just wish I understood the irq stuff better, like why there are 2 irqs for the adpcm stuff. (Top half/Bottom Half?)
Hudson's fast-cd code only uses a single IRQ hook for the AD_CPLAY function. You'll get a better idea when I'm finished and it's all tested, and I can put it up on github or somewhere else.
-
It's version 3.00 ... which is the Japanese SuperCD BIOS IIRC.
Okay, I have the backup files removed, and it's all zipped up. What's the best way to send it to you? (Please don't post e-mail here. PM it, if needed)
It may take a day or two; I have to figure out how to do it in gmail. (no, I seldom use e-mail so I don't remember all the gotchas; I vaguely remember I can't send a .zip, though)
This is Bios 3.01 - region desn't matter, they are just bit-flipped. Only difference is 3.01 has a bunch of nop's at boot to allow reset to settle, i presume. Other pages beyond the boot page should match. You can see where I left off, somewhere in bank 3 with the grp functions...
I really did need to do it myself in order to get a good handle on the differences between the BIOS code and the fast-read code.
Understood. It's one thing to read the code and say "It does this". It's an enirely different matter to disassemble the code and say "It does this because....".
My goal is to split bios apart; I'm 99% sure a lot of it is 'library' code, used across multiple games with little change. I'd like to bring that across to Huc, so we can build cd/Hucard versions against the same library code. Hopefully, we can then get rid of the HuC library stuff when we need the space in a cd game, and still test it as a HuCard.
And before I forget: What setup are you using to re-build Huc? I need to know what os/tools I need just in case...
-
And before I forget: What setup are you using to re-build Huc? I need to know what os/tools I need just in case...
I use msys2/mingw-w64 to build it. It's a modern 32bit and 64bit replacement for the old (and out-of-date) mingw project.
You can find it here ...
https://sourceforge.net/projects/msys2/
I've written some batch files that make the initial setup and updating easier, and they're being used by the guys that are using my V810 GCC compiler patches for the VirtualBoy.
I really need to get them put up on github, but there always seems to be another "priority".
I can send you those if it would help.