A Better Salt Study Thread

[sihaya]

I asked the mods here to let me replace the old ranting salt study thread with this new, more refined post. I think, especially due to the heated nature of the argument, that the debate should try to be as "intellectual" as possible. Like I said before, I'm not at all trying to attack Mr. Borneman or Ms. Lowe. I'm ONLY questioning the merits of this study.

Ok so here it goes again...

Basics first:

First, let's talk a little about experiment design so you all can understand what's going on here:

The most basic kind of experiment design is where you look at one dependent variable responding/correlating to one independent variable. For example, let's pretend you're doing a study that looks at muscle growth (dependent variable) and steroid use (independent variable). You could take a 100 mice, inject half with steroids and half with saline solution, give them the same exercise routine and diet for 3 months, then measure their muscle mass at the end. You have to use a lot of mice for both the experimental group getting the steroids and the control group getting saline solution in order to minimize error due to differences among individual mice. To understand this more clearly... suppose you had only used two mice, one control and one getting steroids. At the end of the 3 months, you wouldn't be able to "trust" the results because you can't be sure that the mouse who got the steroids wasn't at a genetic advantage for muscle growth. I think everyone gets this basic idea, right?

Moving on...

So, what do you do when you don't have 100 mice? What if you only have two mice? Can you still do the study? Perhaps. You might be able to do a repeated measures study. What the heck is a repeated measures study? Glad you asked...

One of the most popular and well known repeated measures design is the pretest, posttest experimental design. For example, you can take the two mice, measure their muscle mass at the very start of the study, then weekly for 4 weeks. Then you inject both with saline solution and continue your measurements for another 4 weeks. Next you inject both with steroids and repeat weekly measurements for yet another 4 weeks. Because you're not comparing results of two different mice, but results over time at intervals on the same mice, you gain statistical power. Get it? Think about it for a sec, you will.

This of course, is not the only example of a repeated measures design. There are all kinds of these study designs. But the basic idea is the same... to test the individuals with different "treatments" over time. You tend to do this when you don't have enough subjects to separate into study groups as you would in a "normal" experiment.

Now, finally, about the salt study:

We have 10g tanks, one for each salt plus a control of natural sea water. We have one independent variable and multiple dependent variables measured over time. Now, first off, what kind of study does this look like? Does it look more like the first example I gave of having 100 mice or the second of having 2 mice? It kinda looks like a mix of both, right? Let's take a deeper look...

Statistically and conceptually, it looks very much like a classic experimental design flawed by having only one subject per variance of the independent variable (i.e. one tank per salt).
Note: "We show that there can be extreme variation among identical tanks, even without any live animals" - Toonen and Wee (http://www.advancedaquarist.com/2005/7/aafeature)

Mr. Borneman, however, would like us to think of this as being more like a kind of repeated measures study to be analyzed with ANOVA (a mathematical concept/model used to analyze this kind of data). Even being most generous with the boundaries of logic and reason, I could only accept this claim if the salt brands were consistent. But they are not. Again, as Mr. Borneman himself concedes, the salt brands are often inconsistent even between batches. So, even with all the power and forgiveness one can gain from a repeated measures study, it doesn't apply here because the batches of the sand brands weren't consistent and experimenters only made this inconsistency more pronounced by doing 100% water changes with each new batch of salt.

Now for how this study could have been done (in light of the statistical power afforded to some repeated measure study designs):

Instead of studying one salt in one tank, they should have studied all the salts in all the tanks... over time. For example, the experimenters could have started with natural sea water until the tanks were "cycled." Then every 3-4 months, changed the salt brand in all the tanks until all the tanks had seen all the salts for a period of 3-4 months (taking measurements of dependent variables at time intervals all along the way and with each change of salt brand). Granted, there are a lot of salt brands to test, so this could take a long time. However, they could have also split the tanks into groups of 5 and tested half the salts on 5 tanks and the other half of the salts on the other 5. Then they could have halved the time to do this kind of study.

The downfall of this proposed idea, and the problem with many repeated measures studies, is that the subjects can "fatigue" or "learn" over time. In the example given with the mice, the mice may have "bulked" up by the time they got the steroids, therefore perhaps limiting the additional effect the steroids might have. In this case, the tanks would be experiencing the salts at different ages... and that would be a problem. However, that would be a statistically manageable problem since all the tanks would be aging at the same time.

Ok, I could have more I could say, but now I'm getting tired. And I think I've made my point. I'm not being "close-minded" and my objections are not "non-sensical"... nor am I trying to embarrass/offend the experimenters. I'm simply looking at this study with a critical eye and right now it looks worthless.

 

[Scooterman]

I will move this thread to the advanced forum, as not to pressure Boomer any this way

 

[Esper]

Excellent points! Your ideas for a better exp. design are good, with one potential problem: the order you change the salts. Going from brand A to B to C may have completely different effects than C to B to A. or not. you'd have to test it to know for sure...

 

[Ashlar]

I agree that the sample was too small.. would it be statistically significant if this study of 10 tanks were repeated ten times? I guess I don't know enough about the proper way to run an experiment, but I do know this one sets off all my warning bells.

 

[sihaya]

Excellent points! Your ideas for a better exp. design are good, with one potential problem: the order you change the salts. Going from brand A to B to C may have completely different effects than C to B to A. or not. you'd have to test it to know for sure...

Right, right, of course... I think that's called "recall bias" maybe? I'm not sure of the most appropriate term for the kind of bias you're talking about. But I do know what you're talking about. I thought of that while I wrote this, but I was too lazy/tired to go on about it. But, yeah, theoretically, you'd have to do ABC, ACB, BAC, BCA, CAB and CBA... and that could get to be a headache real fast.

But actually, after a little thought, I'd think the best route would be to do every combination of pairs of salts (two for each, AB and BA for example). To see why I say this... think about this, the number of possible sequences for X salts is X!, while the number of possible nonrepetitive pairs for X salts is (X^2) - X or X(X-1), even multiplying that by two, 2X(X-1) is still considerably less than X!. See what I mean?


Oh, I meant to say "thanks" too.

 

[Esper]

Uh-oh, math...I'm not good at math. Fundamentally, though, you are right, every possible combination should be tested. In my science we call it the history effect or learner bias, I'm sure there are other names for it. If each tank is broken down, measured, cleaned, and restocked between salts, however, then it shouldn't matter. Didn't Borneman et al weigh some of the organisms? This requires removal anyway. That's really a second experiment, however. The first study can (and should) just get lots of repeated measures on each salt in multiple tanks, one or two salts at a time, and then do whatever post tests deemed appropriate (growth measured by weight or whatever...if that's even really possible. I'm a behavior analyst, not a biologist, so I'm not sure what the best DV's of interest are).

I don't like the 100% water changes, but I have a hard time pinning down exactly why. If nothing else, it would be interesting to do 25% water changes, and then test the makeup of the water at the beginning, middle, and end to see how the elemental composition of the water changes over time. This also more closely mimics typical aquarium husbandry practices, and would provide more useful data. for example, trace element X [like iodine] could get depleted faster than water changes replenish it, and so should be dosed seperately, or something like that.

And these topics are just about the experimental design...there are tons of potential extraneous variables that could have altered the results (unequal lighting, immaculate copepod conception, etc). Some of those were covered in the last thread, but they might be worth discussing here as well.

 

[sihaya]

Maybe you don't like the 100% changes practice because the rationale given for it seems to contradict itself. That's why I don't like it. Borneman et al (love the fanciness of that btw ), reason that the 100% changes are done BECAUSE the salts are inconsistent batch to batch. But that seems like funny reasoning to me. If the salts are inconsistant from batch to batch, then why would you want to ensure that the full effect, (even beyond that felt by "normal" home aquariums which almost never do 100% changes), is felt by the tanks?

Besides that, as a practical matter, you can't even do a truly complete 100% water change without completely drying everything in the tank, including the sandbed.

 

[sihaya]

And you see, this is where the weakness of the design becomes almost humorous to me. They have a set of treatments that vary (i.e. different brands of salt). But then within each "treatment," each application (each batch) is different. So it's almost like this matrix of varying treatments such that you don't really just have 9 treatments applied 10 times, but actually 90 treatments each applied once. And yet, this is somehow supposed to ultimately make for a "repeated measures" study? Come on...seriously? lol ::sigh::

 

[kylem]

salt changes

I see where you are going with this. I have done my own experiments and found that realistic measurement can't be done over a short time but longer periods of 4 to 6months would be more reasonable. This would provide ample time for Ca, Mg and other trace elements;which could be measureable, to be used up and could be compensated for by noticeable changes in growth volume of the tank. The growth rate of each coral would have to be measured so that a total volume could be accounted for. This could give you a close measurement but would only be 80 to 90% accurate because would could not measure the growth of coraline algae unless we weight each rock and the tank without water.
There is also another problem with this....your sandbeds disolve and disolve at different rates depending on how much sand you have and what grain size it is.

The changes I have noticed is that Oceanic Salt does well and is much more balanced per trace elements But does not have enough Ca to support a reef tank. The Reef Crystals salt batchs vary alot and does not have enough Mg to support the extra Ca that it provides. So my conclusion is that Salt batches can give you a good start and can suppliment some trace element but needs to be supported by a Calcium reactor or suppliment additives.

 

[Reefdaddy1]

I would think it would take much longer then Months to an accurate test. There are way to many variable in his test make one accurate. Lets look for minute at the tanks he used.

Sand: Every batch from everywhere has a different effect. It is proven time and time again. You can ge headache alone by the Variables allowed in sand Quality.

Lighting: Bad move. To absolutely cut down any Varaibale within the Model you have to have as many tanks under the same light. To minimize variables this way is essential in the model. What I mean is, 3 ten gallon tanks under one pendent. This removes the lighting variable by what 30%?

Then time: I would think with testing over a more lengthy time reducing Variables, his test would had some worth, But for me? I knew before it was over IO would be on the bottom Again.

This things is what bothered me me about the whole thing.

 

[sihaya]

Yeah, it's a shame too when you think of the resources wasted on this study (including the mournable waste of creativity, experience, time and intelligence that could have been applied to a more worthwhile cause).

 

[sihaya]

Oh, btw, this is Mr. Borneman's response to this critique. It's no longer viewable to the general public. But fortunately, I was able to save a copy before it became accessable only to MARSH members:

Rob's experiment was not the same, nor was his design. His comments are valid, but not directly applicable. Whether or not what he found in terms of variability holds true for our entirely different experimental design remains to be seen.

That really wasn't my point in quoting Dr. Toonen's study. My only point with that was that even "identically" set up experimental tanks can show VERY different performance.

As mentioned, there is little understanding of the design or analyses of the study by the previous post by sihaya. No results have been presented, so no critique is possible, and when such results are presented they will be judged by scientific peers, not 24 year old law students with a year and half of aquarium keeping experience and no scientific training.

I do have scientific training (at the very least, I have written/published a research paper while interning at the NIH).

I would link some of the hundreds, if not thousands, of peer reviewed works using aquariums, repeated measure ANOVA's, mesososms and nearly identical experimental designs, but why? This "internet reviewer" even used a non-peer reviewed hobby paper to quote and justify their position. Toonen did publish his work in a peer reviewed journal of which sihaya is apparently unaware, meaning that Toonen and Wee's data was usable, despite the variability.

Right, but Dr. Toonen used THREE tanks per treatment, not just one.

We do not even know if there are any significant results ourselves as we have not even finished entering data, and we do not know ourselves what data are amendable or usable to hypothesis testing or post hoc testing. We have not claimed to have found anything, and have presented nothing but qualitative data to date. Any critique would be a critique of....nothing.

Again, I don't know why he says this. It's quite common for research designs to be critiqued before the research is done. For instance, you have to do it for most grant applications

I don't think anything more needs be said.

I respectfully disagree.

 

[Esper]

And you see, this is where the weakness of the design becomes almost humorous to me. They have a set of treatments that vary (i.e. different brands of salt). But then within each "treatment," each application (each batch) is different. So it's almost like this matrix of varying treatments such that you don't really just have 11 treatments applied 10 times, but actually 110 treatments each applied once. And yet, this is somehow supposed to ultimately make for a "repeated measures" study? Come on...seriously? lol ::sigh::

HAHAHAHAHA!!! you nailed it!!! this is exactly what was bugging me!

 

[Ashlar]

As mentioned, there is little understanding of the design or analyses of the study by the previous post by sihaya. No results have been presented, so no critique is possible, and when such results are presented they will be judged by scientific peers, not 24 year old law students with a year and half of aquarium keeping experience and no scientific training.

Ad Hominem and Appeal to Authority..

I would link some of the hundreds, if not thousands, of peer reviewed works using aquariums, repeated measure ANOVA's, mesososms and nearly identical experimental designs, but why? This "internet reviewer" even used a non-peer reviewed hobby paper to quote and justify their position. Toonen did publish his work in a peer reviewed journal of which sihaya is apparently unaware, meaning that Toonen and Wee's data was usable, despite the variability.

Appeal to authority again..

We do not even know if there are any significant results ourselves as we have not even finished entering data, and we do not know ourselves what data are amendable or usable to hypothesis testing or post hoc testing. We have not claimed to have found anything, and have presented nothing but qualitative data to date. Any critique would be a critique of....nothing.

Appeal to ignorance..

Note that the core issue is never addressed.. is 10 tanks with one salt each a large enough sample size to come up with _any_ valid results. No matter how many reams of data collect, or how nice the graphs and charts you produce, is the data (and the conclusions that will inevitably be drawn) valid?

 

[Scooterman]

I don't see why testing is done in this manner, seem to me you would do like in the SR15 test & just make up batches of new mix & see how close they come to NSW & let the winner go home Who care what it can do or what they claim, how can anything ever be better than NSW for corals & fish?

 

[Esper]

I would link some of the hundreds, if not thousands, of peer reviewed works using aquariums, repeated measure ANOVA's, mesososms and nearly identical experimental designs, but why? This "internet reviewer" even used a non-peer reviewed hobby paper to quote and justify their position. Toonen did publish his work in a peer reviewed journal of which sihaya is apparently unaware, meaning that Toonen and Wee's data was usable, despite the variability.

its funny that Borneman keeps bringing up peer review, because his salt study is thus far not peer reviewed. (advanced aquarist doesn't count--they agreed to publish before the study was completed) By his standards, this study is but one of many with his name on them that fall under the category of "hobby papers" traded around by anonymous "internet reviewers." of course, his salt study may actually give him a point.

I'll wait patiently and see if this study gets published in a peer-reviewed journal. if it does, then we can all just co-author a rebuttal to be published in the same journal.

 

[sihaya]

Scooty- I agree. Chemically testing the salts is the way to go.

Esper - I think Advanced Aquarist is a great *hobby* publication and I agree this salt study is not worthy to be published in it. But I'm not going to rag on them too much for blindly trusting Mr. Borneman. Certainly many of us have been guilty of this at some point or another. I'm hoping my posts (and the posts/threads of other astute hobbyists) will be enough to help everyone see the weakness of the study.

And hey, if nothing else, this study is a nice conversation piece for how NOT to do aquarium research.

 

[Esper]

I don't see why testing is done in this manner, seem to me you would do like in the SR15 test & just make up batches of new mix & see how close they come to NSW & let the winner go home Who care what it can do or what they claim, how can anything ever be better than NSW for corals & fish?

This could be debated.

The organisms in the reef (or indeed anywhere) are adapted to their environment; the environment is not adapted to them.

For a species to be present, its organisms need to suit their environment just enough to be able to reproduce and create more offspring, which start the struggle anew. This does not mean that there are not more favorable conditions for the health of the species, or even the health of inividuals of that species, just that they have eked out a living there.

For example, plants are exposed to ~12 hours of sunlight per day in nature, but many greenhouses leave their lights on 24/7 to get continuous growth. "unnatural" conditions turned out to better suit the physiology of the organism.

Similarly, there may be concentrations of elements in saltwater that are more beneficial to certain organisms than those found in NSW. To discover which ones and how, we can take a bottom-up approach by examining the organism's biology, and using this knowledge to optimize specific known parameters. Or, we can can take the top-down approach by simply testing different concentrations and noting their effects. Both are valid approaches.

the first approach is useful because it provides explanations which lead to prediction and control, although it may take forever to get to that point. the second apprach is useful because we can establish prediction and control without needing explanations of why. The downside to the second approach is that if we don't get significant results, then we don't know if this is because our methods were bad or because there are no results to be found in the first place (or where to look if there are). Borneman understandably takes the latter approach, but I'll be surprised if any usable data is generated.

 

[Esper]

Esper - I think Advanced Aquarist is a great *hobby* publication and I agree this salt study is not worthy to be published in it. But I'm not going to rag on them too much for blindly trusting Mr. Borneman. Certainly many of us have been guilty of this at some point or another. I'm hoping my posts (and the posts/threads of other astute hobbyists) will be enough to help everyone see the weakness of the study.

And hey, if nothing else, this study is a nice conversation piece for how NOT to do aquarium research.

You're right, Mr. Borneman has done this hobby a LOT more good than harm, as has AA, and I doubt either of them have bad intentions. And I should keep my own color commentary to myself! That said, there is more than one way to distribute this information, and finding a way to get it published (in advanced aquarist or other journals) will have a larger reach and impact than just forum threads (which seem to have a history of being removed, anyway... )

 

[morgan]

i just want to stay that i read an article in the same online mag about coral spawning which listed a whole bunch of stuff they were gonna do and then he never discussed it
it was just about collecting the spawn and how they did it but end with ow well that didnt work,
i think it good to read alot of or as much information on corals as i can but that article was a waste of time