I think a lot of it depends on the thickness of the silicone. If it is a huge, fat bead and is still squishy to the touch, wait longer. If you pressed it into the crack and it is a thinner layer, it will probably be cured faster. Push on it and see if it still feels liquidy inside.
Also, if it is just on your baffles, and it ends up with a pinhole leak, no big deal. Baffles don't need to be 100% watertight, just held in place so most of the water goes around them.