I took apart the cable and sure enough, pin 3 was not connected to Ground. So I cut the cord back 6 inches and redid the wiring. It seems better now; wiggling the connector does not cause interference anymore. The humming/buzzing (whatever you want to call it) is still there though.
Is it my imagination or is buzzing inherent in some games more than others? I noticed that if the system is idle (no music playing) it buzzes less. When I am in just the CD menu, the buzzing is much less pronounced than say, at the pause screen for Bonk's Revenge. If I boot out of a CD game with Select+Run, the menu sometimes has that buzz I'm talking about. Almost like EMI noise.
For full disclosure, I am using an Everdrive V2 to play the games and I haven't region modded my PCE to play my US Hucards (I have no Japanese Hucard games).
EDIT: I also noticed that when I press Start to load a CD game, I hear 3 pops out of the right speaker (pop... pop-pop).
EDIT 2: I just also noticed that Hucard games are significantly quieter than CD games Redbook Audio.