In general, is JP impossibly hard to learn with no professional instruction?
Well, I only took 2 semesters of it in school, (1 regular, 1 summer session), so all that did was just get my feet wet.
Then about 3 years after that, I ended up working in Japan for a year, and so while I was there, I studied the hell out of the thing on my own time. So I can hold a conversation now, but I'm still no where near fluent. So in all, even though I did most of the learning on my own, bear in mind that I had the huge advantage of complete immersion. Plus I'm a freelance Korean->English translator (I do some Tokyopop stuff, if anyone's into that manga stuff), and since Korean and Japanese language has similar grammar points and words, that also helped me out tons learning Japanese.
To make the long story short, I guess it's possible, but you definitely need interaction with native speakers. Do some language exchanges. That's what I'm doing now to keep my Japanese from deteriorating too much.