I don't think the issue is the substrate being toxic.
Rather, has the substrate accumulated more detritus than your system can process.
The larger the particles in your substrate, the easier it is for detritus to "infiltrate" the substrate. And if there is very much detritus in your substrate, you will have a constant source of nutrients for nuisance algae.
Of course, there are beneficial things in your substrate as well. Worms and other critters.
The previously-posted advice to remove and replace your substrate in parts (replacing with either new or cleaned material) is relying on the assumption that the worms and other critters will migrate into the "new" substrate faster than detritus accumulates in it.
I would not put any substrate back in that had particles larger than 1mm. Big particles leave big spaces between them for detritus to fill, and you will be constantly siphoning the substrate to try to keep ahead of the detritus accumulation.