1. Experience and money
Both depend on the level of the Pokémon you defeat. Challenge Mode gives more experience and money, and Easy Mode gives less.
2. The level term in the damage formula
Damage uses a term based on the attacker's level. A trainer's Pokémon at level 37 would use a multiplier of 16 in Normal Mode. In Challenge Mode, it counts as level 40 and the multiplier is 18. In Easy Mode, it counts as level 34 and the multiplier is 15. The stats stay the same, but the attacks hit a bit harder or softer.
Early on, small shifts may not change that multiplier at all. For example, the first rival battle with a level 5 starter (Normal) and level 6 (Challenge) both give the same term. Easy Mode's level 4 starter does give a lower term. Later in the game, the shifts are large enough that the term always changes.
3. Custom teams
In Challenge Mode, Gym Leaders and Elite Four members go further than a simple level change. They get an extra Pokémon, and their other Pokémon have different moves and held items. Easy Mode only lowers levels and AI.
The multipliers above come from the damage formula and apply to those example levels.