Are you (or anyone experiencing this problem, Necromancer, I'm looking at you) using an aftermarket A/V cable? As in, one other than the one that CAME with the Duo when new?
If so, that is your problem, not the port.
It seems any A/V save for the official NEC ones will have this problem. It always appears like the port is at fault because wiggling it around even when plugged fully in feels "loose." When I got my Duo (used) it came with a cable like that, and it appeared the port was to blame. However I swapped it with the A/V cable that came with my SuperGrafx (an NEC original) and the problem disappeared 100%. It doesn't even feel loose anymore.
The funniest part is, I swapped the crappy aftermarket cable over to use on the SuperGrafx (since it gets used less than the Duo) and it actually doesn't exhibit those symptoms on the SGX! Imagine that. Anyway, it was a win-win situation in A/V land that day.