As far as I'm concerned, the word "flop" strictly relates to the financial level of things, and as such, yeah, the PC-FX was definitely a flop. Not the worst of all, but a flop nonetheless. Just look at the list of games... too many published by Nec, not enough diversity in genre, not enough titles. And didn't Hudson Soft have to cancel Tengai Makyo III in an advanced stage of development because the FX didn't have a remotely large enough user base for them to get a return on their investment by releasing it, let alone make a profit? What does that tell you? That's a flop. Now the system was very interesting, and had it been marketed the whole way as a true successor to the PCE, it could have gone on to have a number of awesome games, but as it is, it didn't live up to its full potential. There's a few cool games, but damn, just looking at them you can get a glimpse of what the console could have been, and what it didn't turn out to be.
Yeah, I think most people use the term "flop" in reference to a financial failure, first and foremost. Often, but not always, the financial failure is attributed to the "quality" of the item in question (this may or may not be the case, but folks often see a causal relationship). So we have at least three issues to deal with: 1) the quality of PC-FX itself (and it's software), 2) the profit netted from hardware / software / etc., 3) the user base.
The example of Tengai Makyo III certainly seems like it supports your conclusion (i.e. that PC-FX was a flop). However, it would be nearsighted to think that the cancellation of TM III sufficiently proves your point.
We need to look at the real data: sales figures of hardware & software (with the profit / loss). Raw sales figures are only useful for determining the user-base, but they tell us nothing about whether NEC was making a profit.
For example, we all know that Sony and Microsoft have sold tons of cosoles, BUT they were losing massive amounts of money. Sony eventually turned a profit (due to strong sales, plus they made cashloads due to software revenues), but I don't think MS ever turned a profit with Xbox (they were losing $100 a console, then the prices dropped, so they're lost a lot). Overall, MS lost millions? on Xbox -- but they are gambling that in the long-term (I don't know, maybe they have a 10 year outlook?) things will be lucrative. They can afford to do this, since MS makes billions in other divisions.
Guess what? Nintendo was making a profit on the GameCube hardware on launch day. That's friggin' amazing. When the price dropped to $99, then Nintendo was losing $10 per console.
Most folks would place Nintendo's Gamecube in distant third place in the recent console wars. But if you look at profit margins, the GameCube outperformed Xbox, and then some. The only reason Xbox is still around is because MS has zillions of dollars and can afford to subsidize the losses.
So, is Xbox a flop? Is GameCube a flop?
If PC-FX was a niche system and profitable for NEC, then was it a flop?
How should we define "flop"?
I don't have the answers, of course, but this is food for thought.
All of this leads to a question that I would love to find the answer to: Was TG-16 profitable for NEC? For how long and to what extent?