Considering how powerful the 6809 is (was), it doesn't surprise me that Motorola didn't offer it in higher clock speed versions compared to the 68k. 8, 10, and 12mhz 6809's could have been competitive to the 68k at the time.
I doubt that they had the capability of doing so ... or, as you say, any commerical incentive.
AFAIK, the 6809 was designed in the "traditional" at-the-time way and it actually did quite a bit of work each clock-cycle.
The can lead to a whole bunch of "race" conditions in the internal architecture as you try to speed up the clock-cycle.
The 68000 was designed to do only a very small amount of work each clock-cycle, and so it was "easier" to ramp up the speed in a controlled manor. The downside is that the original 68000 takes a
lot of clock-cycles to do anything.
IIRC, the 6809 was designed by a small (low-cost) team, that wasn't necessarily expected to succeed. The 68000 was a much larger team, and Motorola basically bet-the-farm on it. The 6809 never really had a chance.
You probably already know the story that it was the specific example of the 6809 design-team's success that lead to the creation of the ARM architecture that we all know (and love) today.
It was the computer I had when I was a kid (CoCo 2 and CoCo 3). Though I couldn't afford OS-9.
I could never understand why Tandy released the CoCo3 ... but then I wasn't in America at the time.
When it came out in 1986, was it marketed as an "affordable" alternative to the Commodore Amiga and Atari ST?