I've done it in Win7 x64, so the operating system shouldn't be an issue.
Sounds like you just clicked on the hdtopce.exe file on your windows explorer. As this is an old-school command-prompt software, you want to run the entire process while you're in the command prompt mode. (If you're not a commmand prompt guy at all, send me a pm, and I'll walk you through).
You essentially have to run pcetohd while you're in command prompt mode, and run the executable with passing an output filename. For example, if you're trying to save an Ys save file, go to the directory where the nanami files are, and you type something like "pcetohd ys.sav", then it'll get into a mode where you can start punching in the hex data that you took shots of/wrote down.
BTW, if you haven't done so already, you also need to download HuC and put all the HuC files in the nanami directory as well. This will be needed for later steps when you want to restore the save files and burn the save data onto CD.