My recent experiences with a PSP have shown me that Sony's MD-era format firewall is still firmly in place, only now with video making this sort of thing extra stupid harder than it every was with audio, and I fear for you, really, in trying to get a similar thing going on something 1/1000th as popular as a PSP. I'm convinced Sony just doesn't want us watching mobile video, even on a mobile video viewing device we just bought from them.
Handbrake can made DVDs into PSP-usable movies no problem, but trying to convert anything else is just hell. Recent PSP firmware updates have actually made the PSP less compatible with some video formats. Yes, that's right, stuff that would run on 2.5 is now broken in 4.x. How's that for progress?
For me mobile video will be iPod all the way for the foreseeable future since Quicktime does trouble-free, iPod-compatible, perfect conversions every time from almost anything with the greatest of ease.