Yeah, N64 emulation isn't very good. Still it's good enough for me to get my Zelda:OoT fix in from time to time! The most recent RetroPie update seemed to make N64 games play a little smoother, but Zelda is almost perfect now. My Pi3 is also overclocked a bit, so that might help too.
Since I've gotten my Pi3 all set up I've definitely slimmed down my physical collection. Many of the games, though, I've had for decades and it's hard to let them go. There's just something about holding the game in your hands, especially if you have fond memories of purchasing it and playing it for the first time bitd.
From what I've seen, Dreamcast games are for the most part playable, but few run at 100% speed. I need to create a config file for my controller because I can't get out of the date/time screen, and it's pretty low on my priority list because I have a Dreamcast that works just fine.
I haven't noticed any issues with PS1 games on the Pi3, but have only played a handful of games. Castlevania:SotN seems to work fine and I've put a few hours into it. Other than that, I've stuck mainly to RPGs and they all work fine too. Hooray for savestates!