Buying a peke is like buying a Ferrari. It's a very expensive quest of love. If you are one of those types that will be whining about money, stop now.
However if you are prepared to dish out, the best way to go is to start simple and build up. I live in Japan and it took me two years to complete my dream machine.
1. Pick your model. XVI is the best bang for the buck, if you want cheaper go Super. I do not recommend the older SASI machines, and the compacts tend to fall apart sooner or later.
2. Monitor. The Peke is designed to be connected to a proper tri-sync (15/24/31khz) monitor. This is something most people do not have. A lot of people use an old VGA CRT monitor (31khz) which will allow you to play maybe 80% of the library. Sooner or later you will want to get a real tri-sync monitor, or you can get the Micomsoft XPC-4 scan converter which will give 100% compatibility at 1080p if you wish. However the XPC-4 costs about $400...
3. Keyboard and Mouse. You need these plain and simple. Keyboard for certain, mouse can be had when you need it, but keyboard is mandatory. Getting a machine that comes with a keyboard will save you in the long run.
4. Compact Flash board. This will run you another $100. But the benefit is that you can play 95% of the games off it directly. Meaning you do not need to buy original software outside of collection purposes and a few special games.
5. RAM. If you get a Super or XVI, you most likely will have it come stock with 2mb of RAM, this is enough to get started, but you will want to upgrade as soon as you can afford to. More RAM = more games that can be played off Compact Flash. If you max out the machine at 12mb of RAM, you will be able get the full benefits of the CF route.
6. Controllers. Most games use the simple d-pad and 2 button MSX standard. You can easily make an adapter for MD or NeoGeo controllers. Some games need the mouse, and many games offer analog control using the cyberstick, which is highly recommended and makes a lot of games way more fun to play.
That's it for the basics. MIDI setups are optional (Angel Dive is the only game that requires MIDI afaik). Going MIDI is a long discussion by itself. I highly recommend you wait on this until you are ready, you can easily spend hundreds on MIDI equipment alone, start with simple to use (and totally badass) internal FM. Don't go MIDI until you are ready to.