Archiving?

[akamichi]

I've been studying the layout of PCE CDs in an attempt to make the "perfect" copy.  Before I say anymore please read the below disclaimer.  I don't want this to be about "warez".

Disclaimer:  This is not about copyright infringement.  Copying directly from the original CD to a blank CD isn't the goal, but ultimately the end result is the same.

Let me explain what I'm trying to do.  I want to store the contents of a CD on my HD (and subsequently to a backup device) so that I can:
A)  use the contents of the CD in an emulator that has that kind of support.
B)  listen to the music tracks (compressed or not) from the CD contents via some music player.
C)  recompose the original CD (aka copy) from the data on the HD which would work on the real machine w/o any issues (or an emu that supports actual CDs.)
D)  never lose the capability to do A, B, or C due to changing technologies and/or obsolescence.

Is anybody interested in this?  Anybody else thought of doing or is doing something similar?  Should this topic just be deleted without a trace?  Should I be banned for bringing this up?

Before I go any further into detail, I thought I'd ask around first.

[zborgerd]

Since the CDs don't have a usable filesystem to most normal operating systems, and they also contain CD audio (in most cases) that doesn't read as a filesystem, the only real way that you can do this is by making bin/cue images of the disks.  I do this on Linux with a program called CDRDAO.  There is a version for Windows that works very well called "Burnatonce".  It used these opensource tools and combines them with a realatively easy to use GUI.

http://www.burnatonce.com/index.htm?news

I'm not certain, but I think that some emulators do support bin/cue.  I know that HU6280 or Hugo were at one time able to open images of CDs.  They may have been ISO/MP3 though, but Bin/CUE may be supported.

Another method is to rip all of the audio into a lossy format like MP3, and create an ISO image of the data portion of the disk.  I wouldn't suggest archiving anything in a "lossy" MP3 (or similar) format in the long term.  I think that Hugo was able to play games with this method.  Not sure about Magic Engine though.  I seem to recall that there were people in the Magic Engine forums that were playing images of games though.

I've heard that Windows users can use programs like Daemon Tools to mount Bin/Cue images so that the Windows OS sees them as regular CDROMs.  You may be able to use this method to play backups of games.

http://www.daemon-tools.cc/dtcc/portal/portal.php

If that's the case, the CD Audio tracks of the bin/cue image would always be accessable through Daemon Tools.  It appears that Daemon Tools supports a dozen or so archival image formats; Nero, DiskJuggler, etc.  I prefer Bin/Cue because it's supported across all popular OS/platforms and doesn't require proprietary software to be used.

[vtnwesley]

I have had good luck with Daemon Tools, as Magic Engine doesn't support booting CD Image files (rather it intellegently uses a REAL CD if it sees one is there). By using Daemon Tools, this problem is avoided, plus this isn't the first time I have had a good use for the program.

[twor2005]

Daemon tools is the best thing ever. I always thought that would work with the Magic Engine, but never got around to trying it.

[Keranu]

Daemon Tools works great for me.

[akamichi]

Thanks to everyone for the replies.  I guess I'll elaborate on what I said above and hopefully it'll be more clear on what I'm trying to accomplish and the reasoning behind I've made certain decisions so far.

First off, I've already written a program and a number of helper scripts to make an image of a CD.  It's not perfect, but it's a start nonetheless.  In the most basic form, I can insert a PCE game, run my script and it'll rip into iso/wav.

When I listed the goals (points A-D) in my original post, I left some details out.  I'll try to explain the best I can.  One big thing I left out was the process has to be cross-platform.  

I figured I'd have to store the CD into some intermediate format (IFM).  The flow would simply be original CD->IFM<->CD-R or Emu ISO Format.  Some IFMs would be the commonly known ones such as iso/mp3, bin/cue, etc.  These of course, would be saved to HD.  Because my primary goal is archiving and restoring, any kind of lossy compression on the audio files isn't acceptable.  Lossless is fine and I've been thinking of using FLAC to compress the audio.

zborgerd, thank you for your detailed reply.  I did consider using cdrdao and cdrecord and I came to realize that having one big file just isn't practical.  I'd like to queue up the GoT music in XMMS, Winamp, shoutcast it, or whatever and having one big image file just makes this cumbersome.   That's why I'm making iso/wav "images" right now.  

Also, although Daemon Tools is a kick ass program, it doesn't really help people who don't use Windows.  However programs like Daemon Tools will facilitate the ability I mentioned in point A.  Since Daemon Tools supports a large number of CD formats, one can write a script that makes, for example, a cuesheet from the IFM that Daemon Tools can use.  In fact, I have scripts already that do this.

Because most people use Windows but yet I primarily use Linux and OSX, having everything cross-platform is very important to me.  Unfortunately, my initial data track reading program only works in Linux.  I'm trying to learn how to accomplish the same thing in Python so that it works in the major environments.

Finally, in order to protect the archive from becoming unusable due to tools no longer being available, I'm trying to do everything with open source programs.  

In short, the intermediate format (for now) for the files is simply iso/wav.  From there, I have scripts that will create a cuesheet that should work with any program that uses cue/iso/wav or toc file (cdrdao).  I've been able to use my program to extract the Ys IV data track for use with the English patch and burn back to CD-R all in Linux so far.  However, I haven't thoroughly tested the copy.

Hopefully I've explained enough so everyone knows what I'm attempting.  If anybody has anything to contribute, please post.  I still need to clean up the programs and add some more functionality.  It's still alpha quality stuff.  Once my new server is setup, I can post my programs if anyone wants to check them out.  Just keep in mind, as of right now, it's Linux only.

[zborgerd]

Thanks to everyone for the replies.  I guess I'll elaborate on what I said above and hopefully it'll be more clear on what I'm trying to accomplish and the reasoning behind I've made certain decisions so far.

First off, I've already written a program and a number of helper scripts to make an image of a CD.  It's not perfect, but it's a start nonetheless.  In the most basic form, I can insert a PCE game, run my script and it'll rip into iso/wav.

When I listed the goals (points A-D) in my original post, I left some details out.  I'll try to explain the best I can.  One big thing I left out was the process has to be cross-platform.  

I figured I'd have to store the CD into some intermediate format (IFM).  The flow would simply be original CD->IFM<->CD-R or Emu ISO Format.  Some IFMs would be the commonly known ones such as iso/mp3, bin/cue, etc.  These of course, would be saved to HD.  Because my primary goal is archiving and restoring, any kind of lossy compression on the audio files isn't acceptable.  Lossless is fine and I've been thinking of using FLAC to compress the audio.

zborgerd, thank you for your detailed reply.  I did consider using cdrdao and cdrecord and I came to realize that having one big file just isn't practical.  I'd like to queue up the GoT music in XMMS, Winamp, shoutcast it, or whatever and having one big image file just makes this cumbersome.   That's why I'm making iso/wav "images" right now.  

Also, although Daemon Tools is a kick ass program, it doesn't really help people who don't use Windows.  However programs like Daemon Tools will facilitate the ability I mentioned in point A.  Since Daemon Tools supports a large number of CD formats, one can write a script that makes, for example, a cuesheet from the IFM that Daemon Tools can use.  In fact, I have scripts already that do this.

Because most people use Windows but yet I primarily use Linux and OSX, having everything cross-platform is very important to me.  Unfortunately, my initial data track reading program only works in Linux.  I'm trying to learn how to accomplish the same thing in Python so that it works in the major environments.

Finally, in order to protect the archive from becoming unusable due to tools no longer being available, I'm trying to do everything with open source programs.  

In short, the intermediate format (for now) for the files is simply iso/wav.  From there, I have scripts that will create a cuesheet that should work with any program that uses cue/iso/wav or toc file (cdrdao).  I've been able to use my program to extract the Ys IV data track for use with the English patch and burn back to CD-R all in Linux so far.  However, I haven't thoroughly tested the copy.

Hopefully I've explained enough so everyone knows what I'm attempting.  If anybody has anything to contribute, please post.  I still need to clean up the programs and add some more functionality.  It's still alpha quality stuff.  Once my new server is setup, I can post my programs if anyone wants to check them out.  Just keep in mind, as of right now, it's Linux only.

Heh.  I didn't know you were using Linux and OSX.  That makes things easier for me to understand because I *also* use Linux, as my only operating system at home.  I also am part of a GNOME desktop development project.  It's my platform of choice, so I can certainly undestand how important it is to use tools and formats that will always be usable in the future.

I *love* the FLAC format.  I archive all of my music to my system in FLAC format.  Maybe we could try to patch it for Zeograd to add support for ISO/FLAC support of games in HuGo?  It sounds like the only suitable method for the moment would be to use ISO/WAV, unless an emulator could support ISO/FLAC.

You may want to test that patched version of Ys IV.  I also attempted to patch Ys IV and Xak III on Linux.  I found that CDRDAO could not rip an image of Xak III that completely matched the image that CDRWIN would make.  There was just enough of a difference in the size that the patch wouldn't apply to the right places.  This is not to say that the reburn of the original image wouldn't be just the same as the original copy...  It's just that CDRWIN's BIN files are slightly different than CDRDAO's, and the patcher program isn't smart enough to patch the proper locations.  I ended up having to rip the image on someone else's Windows machine (since CDRWIN only works on Windows) and then burnt it back to the disk on the same Windows machine with Burnatonce since CDRWIN has such crippled writing support for unregistered versions.  That's the unfortunate side-affect of relying on propritary software to make backups of games.  CDRWIN is a complete requirement for the RIGG patches, even though they (strangely enough) include patcher binaries for Linux and OSX.  I mentioned it on the RIGG board, but it didn't seem too imprtant to NightWolve because "nobody uses Linux or Mac OS X", even though I offered to do the work to make a patch for users of UNIX/UNIX-like operating systems.  Ah well.  

What are these programs that you are working on?  Is it a front-end for archiving games?  I'm interested in this.  Is it a script, or is it a GUI frontend of some sorts?

[nodtveidt]

I was going to mention FLAC but it looks like you beat me to it. FLAC is truly the best choice for audio these days...completely lossless and awesome compression too. We're going to use it for all of our releases at Frozen Utopia.

[akamichi]

Heh.  I didn't know you were using Linux and OSX.  That makes things easier for me to understand because I *also* use Linux, as my only operating system at home.  I also am part of a GNOME desktop development project.  It's my platform of choice, so I can certainly undestand how important it is to use tools and formats that will always be usable in the future.

I *love* the FLAC format.  I archive all of my music to my system in FLAC format.  Maybe we could try to patch it for Zeograd to add support for ISO/FLAC support of games in HuGo?  It sounds like the only suitable method for the moment would be to use ISO/WAV, unless an emulator could support ISO/FLAC.

Hey, that's cool!  Finally another Linux/unix guy!  Yup.  Having that ability in hugo would be useful.  If I'm not mistaken, Hugo uses .hcd files to describe the CD right?  If so, somebody could just write a hcd making program similiar to the one I have for cuesheets and toc files.

You may want to test that patched version of Ys IV.  I also attempted to patch Ys IV and Xak III on Linux.  I found that CDRDAO could not rip an image of Xak III that completely matched the image that CDRWIN would make.  There was just enough of a difference in the size that the patch wouldn't apply to the right places.  This is not to say that the reburn of the original image wouldn't be just the same as the original copy...  It's just that CDRWIN's BIN files are slightly different than CDRDAO's, and the patcher program isn't smart enough to patch the proper locations.  I ended up having to rip the image on someone else's Windows machine (since CDRWIN only works on Windows) and then burnt it back to the disk on the same Windows machine with Burnatonce since CDRWIN has such crippled writing support for unregistered versions.  That's the unfortunate side-affect of relying on propritary software to make backups of games.  CDRWIN is a complete requirement for the RIGG patches, even though they (strangely enough) include patcher binaries for Linux and OSX.

I tested my copy far enough to get to the town so to see the English text.  That's about it.  My program rips the data track to the exact size that the Ys 4 patcher expects.  I plan on playing through the game of course, but just haven't had much time to just sit down and play.  However, a PCE emu for PSP (pcep) was just released that added CD support.  I might just use that to play thru Ys 4.  

 I mentioned it on the RIGG board, but it didn't seem too imprtant to NightWolve because "nobody uses Linux or Mac OS X", even though I offered to do the work to make a patch for users of UNIX/UNIX-like operating systems.  Ah well.  

Yeah.  I tried to convince them to use FLAC but NightWolve went with APE.  That was when I decided to branch off and do it the way I've described so far.  No big deal.  RIGG has a different goal than I do, so I respect their decision.

What are these programs that you are working on?  Is it a front-end for archiving games?  I'm interested in this.  Is it a script, or is it a GUI frontend of some sorts?

Right now, I have a C program (command line) that dumps all the data tracks into separate files.  For ripping the audio tracks, the program is hardcoded to use cdparanoia.  One of the changes to the prog is to be able to specify whatever audio ripping prog you want.  Then I have a simple bash script that just makes the operation a little easier.  Basically it accepts choosing which cdrom you want to use and a name for the game.  It creates the directory and calls the C program which rips the CD.  The output is like:
01 - GateOfThunder.wav
02 - GateOfThunder.iso
03 - GateOfThunder.wav
etc.
so the "wrapper" script would be called like

Code: [Select]


./go.sh /dev/cdrom GateOfThunder


Finally, I have another bash script that will take a list of files and make a cue and/or toc file.  Then I can feed that to cdrdao or cdrwin for writing.  That basically was the process I used to make the Ys 4 patched copy.

Some things that I want to work on next are:
1.  Dump the CD's toc to a file.  That'll be easy since my program already reads it in and parses it.  
2.  Add support for normalizing the audio tracks and encoding to FLAC.  The audio rips seem to be very quiet, almost every audio track I've done needed normalization.  Without it they were just too quiet.
3.  Add enhancements here and there to improve the usability of the program... like an options file or something.  Gotta get that hardcoded cdparanoia thing out and into some kind of command line option.

That's all I can think of right now.  I'd like to have a full-blown GUI driven one-click rip thing, but I'm in no hurry.

[zborgerd]

Awesome.  I'd like to try the programs some time, if you wouldn't mind.  

[akamichi]

Sure.  I'd love to get some people to try it out.

I just got my new server, so as soon as I get that up and running, I'll organize the stuff and let you know where to get it.

I'll need to clean up the source too.  It was originally a basic command line CD player that never quite worked. hehe.