I say the best bang for the buck is get yourself a refurbished PCE Duo. Then grab yourself an everdrive and buy a stack of high quality blank CD-R's. If you start with a card only system you'll soon realize you want the ability to play CD games too and then you'll be making a second hardware purchase. A region mod is great if you decide to start buying actual copies of hucards. The CD games are region free. If you're using a CRT the stock composite out is fine unless you're a videophile.
Use the everdrive and burned CD-R's to learn about the system's library and enjoy the games. If you find games that are so awesome you feel the need to buy legit copies at least you know what you're getting into. The stuff is getting expensive so informed decisions are the best decisions when working with a set budget.