Did you use a diamond hole saw, or one of those carbide grit ones? I wouldn't have expected it to take hours.
I used to have a 1/4" diamond bit that was great for starting holes, but can't find it now. And can't find a replacement either. Those spear-point drill bits are much slower as starters.