Prioritize defeating bosses to gain abilities like the Double Jump (essential for many gems) and the Speed Shoe (crucial for Relics).
The power-ups you earn from bosses — especially the sprint ability — make time trials significantly more manageable. You can also play as Coco in certain vehicle levels, which breaks up the platforming monotony.
To see the specific locations for every gem and secret exit across the trilogy: