So if we are going to be handing out experience for monsters, and the 100 xp per hit die rule is generally good in my experience, there is one situation where it appears to fall down: overkill, where a higher-level party gets an overly large amount of experience for an encounter with six giant rats that is trivially dealt with.
To put it formally, a group in control of tactics is going to have a much easier time dealing with 8 x 1 HD orcs than 1 x 8 HD giant, for any number of game mechanical reasons. The danger ... the bad player behavior that you don't want the rule to encourage ... is that players will seek out weak rather than challenging combats in order to advance. And even if they don't (because that is a rather dim view of what they want in the game), they will have a weak encounter and realize it gave them quite a chunk of experience and wonder why the system is not pushing them toward the more exciting kind of play.
AD&D, as I've mentioned before, solved this just by giving very low xp awards for low-level monsters, altogether. Combine this with exponentially increasing the amounts of xp needed to level, and you have a situation where a high-level character effectively gets peanuts even from single-handedly wiping out a company of 100 orcs. Allow me to demonstrate the special technique of statistics:
(As I recall from playing AD&D, we were able to level at a reasonable pace not from monsters or even monetary treasure, but thanks to the generous experience awards for magic items. And you wonder, why the obsession with taking treasure and magic items away from the party...)
Of course, AD&D being AD&D, Gygax also put in a completely unnecessary rule toward the same goal, further complicating experience awards by instructing DM's to add up the levels and HD equivalent on each side after a combat, and dock the party proportional experience if they outpowered the foe. This rule, besides being cumbersome to apply and hedged round in even vaguer clouds of subjectivity, seems to be more appropriate to the original 100 xp/hit die rule.
My concession to this logic, within the 100 xp/HD system, was to have characters gain only 10% of the usual experience from monsters they outclassed by 2 or more level equivalents, and 50% if outclassing by 1 (although I didn't really apply this last one). In practice, however, even the 10% rule created an awkward splitting of points between higher and lower level party members.
Here's my latest try, and we'll see how it does in actual play.
Rather than using division, it uses subtraction: higher level characters simply discount one or two creatures of a sufficiently lower level. Although this appears in the "basic" 52 Pages rules that only go up to third level, the intent is to increase geometrically, so that 4 monsters per character are discounted at 3 levels up, 8 at 4 levels,, and so on.
If this means that an 8th level party of 5 can plow through a regiment of 320 orcs without getting any experience ... well, after the first 80 orcs or so fall without any casualties in return, the DM is better off just calling a rout than gaming through the whole tedious sequence, hoping to overwhelm the party tanks with a slew of lucky hit and damage rolls... lest we forget the lesson of the 100 linear attacking kobolds.