Ever since D&D, many RPGs have handed out experience points without a huge amount of thought as to why they're doing it. The standard model of XP, which D&D pioneered, is achievement based. OD&D made it a mix of XP for killing monsters and XP for getting treasure. Many games mindlessly copied this approach, often ditching XP for treasure as unrealistic. But over the years many more methods have emerged.
XP for achievements: mechanistic version – the classic D&D formula, XP awarded in a pre-defined way based on the power of the foe slain or the value of treasure acquired. Often derided as unrealistic or creating perverse incentives, this approach actually gives a pretty strong incentive to keep doing exactly what the game is about, i.e. killing monsters and taking their stuff. It also has the distinct advantage that players get rewarded for acting effectively. If your plan enables you to somehow kill the monsters and get the treasure in a low-risk way, why shouldn't you be rewarded for that? It's a pretty strong system for an adversarial, GM vs players dungeon-crawling game.
XP for achievements: nuanced version – a less mechanistic approach. The GM awards points for clever plans, defeating baddies, achieving story goals, and so on. This has the advantage of enabling a broader spectrum of play – you can get XP for solving a puzzle, succeeding at a non-combat task, or even for personal goals like becoming head of the watch, or whatever. The downside is that the whole reward system is subject to the arbitrary judgement of the GM. Favouritism and bias can become a problem. The GM may reward what they see as a clever plan rather than one that actually is clever. Worse, it can be used to railroad the players towards the GM's preferred outcomes.
XP for turning up – what it says on the tin. Here, XP is used primarily as a means to allow the character's abilities to evolve and improve over time. There isn't much of an incentive mechanism here, beyond the obvious one of actually coming to sessions, and many GMs even drop that requirement (everyone gets XP for a session even if they didn't turn up), effectively removing any incentives.
XP for using abilities: simple version – Call of Cthulhu is the earliest example I'm aware of (but no doubt not the first) of a system that simply gives XP whenever an ability is used, that can be spent on improving that ability alone. (CoC actually went further and asked that you succeed in using the ability at the time, but then roll again and gain XP only if you fail.) This has the dubious advantage of realistic progression, as the more you use something the better it gets (and in the case of CoC creates a pleasing bell curve of progression). It also incentivises players to get stuck in and use their abilities as much as possible, which could be considered an advantage, though equally it encourages the use of abilities when they aren't really needed or interesting. Other games have awarded XP for using stats without requiring it to be spent on the stat that was used, a more flexible approach with similar pros and cons.
XP for using abilities: advanced version – Apocalypse World is an example of a game that gives XP for using abilities but doesn't require you to spend it on the ability that got you the XP. AW develops this method further by asking other players to "highlight" the stats they want to see you use, only rewarding the player for rolling highlighted stats. The result is a strong incentive for players to push themselves into specific situation types that other players have chosen (presumably because they think it will be interesting to watch). It also forces players to mix it up a bit rather than always sticking to the same old turf.
XP as fallout – Dogs in the Vineyard introduced (again, possibly not the first to do so) the interesting method of giving players stat improvements or even new stats as a "reward" for being beaten up in play. Each time you get verbally lashed, physically beaten or shot, you may gain a fallout stat like "a healthy respect for bullets" which can then be woven into future stories. What's nice about this is that it makes the more painful and risky elements of roleplaying pay, and turns each new stat into a reminder of a previous encounter, so that the character sheet actually has character. It creates an slightly odd incentive to up the stakes in conflicts, which is well tailored to the genre of Dogs but perhaps not great for just any old game.
Surveying the above list (which is hardly exhaustive, but I suspect is a reasonably representative sample of common methods), what's striking is how many of them vary the thing that gets you XP, but how few vary what the XP can be used for. They are virtually exclusively about stat improvement. The pace of improvement varies – D&D did levels, while most games now break upgrades down into individual stats and advantages – but it's basically the same thing every time.
There must be a lot more that can be done in the space of Dogs, giving you character changes that are directly based on in-game events. Or maybe even advances that don't change your abilities at all, but interact with some other aspect of your character, like beliefs or relationships.
If you know of any interesting XP systems out there, shout out – I'd love to know about them.