A Better Salt Study Thread

[ecobalance]

best salt

This is easy and doesn't require a clinical study. the best salt is the salt that comes from a reptable company and is ONSALE that week study complete.

 

[Scooterman]

We see sucess with most higher end salts, I don't like switching from one to another, so far that has protected me from the new & improved salts now avaliable

 

[sihaya]

agreed...

these studies have their limits for sure...but I think the researchers should admit them...and not make conclusions based on their statistics...if they were APA members and submitted studies like the substrate one and made conclusions based on thier experiments like that...they would have been tossed out of the APA so fast it would have made thier heads spin for mis-representing thier statistical findings...

MikeS

I'm sorry, I'm not sure what you mean by "making conclusions based on their statistics." Are you talking about how they're analyzing their data or how they're determining stastical power? In the case of the salt study, I don't know if the data could even be called "statistics." How much statistical power does something actually have to have to be called a "statistic?" I honestly don't know.

But here's another question I have... how are Borneman and Lowe even going to be able to decide what's statistically relevent or not? Does anyone even have any information on "normal" variance to compare their data to? For example, the frags grew more in some tanks than others. EVEN IF we assume the tanks were all "identical", how do we know how much frag growth varies "normally?" I honestly don't know if there's much data/information on that.

About Toonen's substrate study... I actually thought that was pretty well done. It was also published in a peer reviewed journal. Though, granted, I'm not sure how much practical knowledge was gained from it, as a study, I think it was pretty good. And even if I didn't, I'm not quite brave enough to publically critique the peer reviewed work of a scientist published in Nature.

 

[MikeS]

Here you go Sara...

http://www.reeffrontiers.com/forums/showthread.php?t=9524

Ok, this study was a follow up to a previous one, and here is what I would consider as the point to the study...this quote taken from the first study...


Quote:
However, there has never been a comparative experiment to determine the relative effects of sandbed depth, particle size and whether or not the presence of a void space beneath the sediments confers any advantage relative to the presence of the sediments themselves

I could not find a statement anywere in the presentation where they initially claim that the plenum offers no advantage over substrate alone as an actual hypothesis for the experiment, although that does seem to be a theme later when discussing the results and thier conclusions...


Quote:
Put simply, our experiment shows that the presence of a plenum has no measurable benefits over simply depositing the same sediments directly on the bottom of the aquarium (at least for nano-tanks over the time scales that we tested).

Ok...on the statistics...as I read it, they first compared their test results to the AutoAnalyzer results to determine mathmatical correlation. These test results (or dependant variables.... ie the salinity, pH, ammonia, nitrite, nitrate, oxygen, phosphate, alkalinity, and calcium) were then compared to the independant variable in each tank (ie deep, shallow, fine, coarse, plenum, no plenum) and were analyized with statistical models to determine whether the differences between the observed dependant variables were statistically significant enough to infer correlation with the independant variables.

Here's what they found...


Quote:
Analyses of variance for each water parameter revealed no significant differences among the final salinity, ammonia, nitrite, oxygen, or organic concentrations, nor were there any significant interactions among experimental treatments for any of these water parameters

So the independant variable did not correlate with the dependant varible in these cases.


Quote:
There were significant differences among treatments for the remaining water parameters

In the cases of pH, nitrate, phosphate, alkalinity, and calcium, there was a statistically significant difference between the dependant and independant varibles to correlate between them.

I find it worthy to note that livestock mortality was not listed as a dependant variable in the initial statistic summary, but it is inferred to later in the study, they did do some analysis of it...


Quote:
For the among treatment comparison, the overall analysis of variance was not significant (df = 7, F = 0.88, p > 0.5). However, there were nearly twice as many animal deaths overall in shallow as in deep sediment tanks (Fig. 9). On average 2.91 ± 0.46 animals had to be replaced in the shallow sediment treatments, whereas only 1.47 ± 0.46 animals had to be replaced in the deep sediment trials (df = 1, F = 5.23, p < 0.05). No other treatment or interaction term significantly affected the death rate in our experiment.

Basically, the mortality rate between tanks could not be correlated to the independant variable in a statistically significant manner.


On the whole, based on what can be inferred from the results of their experiment, these statements in the Overall Summary are the ones I have a problem with...(and lets face it, many people are going to breeze through the bulk of the article and focus on the overall summary, particularly newcommers to the hobby... )


Quote:
Overall death rates were roughly twice as high in aquaria with shallow sediments as in deep sediment treatments. The highest overall death rates were seen in aquaria with shallow coarse sediments over a plenum, and the lowest death rates occurred in aquaria with a sandbed composed of deep coarse sediments.

While this is their actual observation, it was NOT supported statistically that the reason the mortality rates in these tanks were due to substrate type. I think they should have made that point VERY clear in the summary! To read that summary, one might infer that the substrates correlated directly with mortality, which they simply didn't to a statistically significant degree.


Quote:
We did not test bare bottom tanks, but the data clearly suggest that the shallower the sediment, the higher the mortality rate

Whoa! Sure, they observed higher mortality in shallow systems, but as stated there was no statistical significance there, so one cannot infer with any degree of certainty that substrate depth and mortality correlate...which leads to this...


Quote:
and you can't get much shallower than a bare bottom tank!

Whoa! Now this is a big stretch! First, it wasn't in the test, so why even make a reference to it? Second, since no significat relationship was demonstrated on mortality, how can one possibly infer that?

MikeS

MikeS

 

[sihaya]

Ok, I understand what you're getting at. I still think the Toonen study was well done. But again, I think some of the things that have been read from it (and into it) are a bit extreme (to say the least). Misinterpreting a good study is really not a whole lot worse than putting faith in a bad study. Maybe that's what you're saying? Anyway.. I think we should try to do neither.

 

[MikeS]

Well, actually, I think that study had a ton of flaws to start with myself... But yeah, what bugged me the most was how they summarized their findings...while they did state that they found no statistical support that substrate depth had anything to do with livestock mortality, they however in their summary stated that it did per say, they even made a reference to a BB setup, which they didn't even test in their study...

MikeS

 

[sihaya]

The bottom line for me is that they did the study as rigorously as could reasonably be done (as far as I can tell) and that they were honest in noting the limits of the statistical strength of the study.

I do understand why you're irked though. They skirt dangerously on implying that BB tanks are "deadly." And that's a bit extreme (obviously). However, I try to pick my battles. I don't have quite the energy to attack real scientists for their bold opinions (expressly stated as such) when there are so many annoying bits of total myth and misinformation being said all over the place.

 

[sihaya]

But please... I honestly don't think I'm in much a position to thoroughly critique all the research and all the studies and trials ever done for the hobby. It's simply that THIS particular salt study has struck me as exceptionally and completely worthless. Yes, there are other imperfect studies. And yes, Borneman is not the first to make bold interpretations of studies (In fact, I remember once he tried telling everyone to try "giving their corals a squeeze" based on this study that showed that being picked up with tweezers produced an immune response in coral frags.) But this salt study is just soooo far out of the range of anything I would consider "good science" that I just felt compelled to explain to people why it's so meaningless.

 

[aquariumdebacle]

These two studies remain the "BEST" studies in their respctive catagories simply because they are the first. You may criticize and point out flaws all you want but when you start attacking a person's personality your credibility drops to zero and you are merely expressing your opinion. We have to start somewhere and as flawed as you may find these studies I commend them for at least attempting them no matter how flawed they appear to you and the others that have posted here. They have been done without external financing or grants. Instead of slamming these people let's work to make a better study by coming up with a funding source to get a truly good study done by learning from what these studies lack. Stating that the studiy is "wortheless" is an unfair accusation because the effort alone has merit and worth. Opinions are like butts erveryone has one and they all stink!

 

[sihaya]

Dan- with all due respect, I just don't think you've been paying attention.

1) I've been doing nothing but *supporting* the Toonen study. Like I keep saying, I DO respect his study and think it was well done (even if his commentary could be considered my some to be a bit extreme).
2) Borneman/Lowe's salt study is NOT the first salt study! Shimek did one years ago. (And there have been others too.) Borneman/Lowe's may be the first of this particular design of salt sudy, but I fail to see how novelty of design adds any merit to a study (especially when it's a poor design).
3) Again, I have not commented on Borneman or Lowe's personalities. I have made comments about Borneman's style of presentation of his work and writing in the hobby. That is not the same as commenting on his personality.

 

[little fishes]

Wow- I would like to add some food for thought. It would seem that the biggest complaint about the salt study is the N of one with multiple batches of each of the 10 brands tested by 100% water change to a new batch at certain times. In my opinion the critiques appear to come from educated people who are NOT experts in the area. I too fit into such a category. I think that we should all wait for the result to appear in a peer reviewed journal so that it has been critiqued by experts IN the field before such strong opposition is adamantly defended. It can take six months or more to get a peer reviewed paper in print after all data is gathered and results are in. I personally will be waiting for this sort of article (which I understand is in the works) before I make a final decision on how I will use the information. It seem that those who did this study were asked repeatedly for information before they were finished with the analysis and were kind enough to share some of the information prematurely- A “mistake” I am sure they will not make again. Although I agree this study is not without flaws, most research has to deal with hurdles of one kind or another and to say the results are unusable I believe is more than rash. If such efforts are met with this sort of ”venomous” opposition then the motivation for individuals to make such efforts in the absence of funding or support will wither and advancement in the hobby will be the injury caused by such efforts. Critiques are fine- make your voice heard, but frankly when someone spends so much personal effort to discredit the work of people who seem well intended it IS a personal attack even if that is not how the “critiquer” sees it. I also think that those who are so against this study have also overlook potential flaws in there critiques. The critiques here are somewhat simplistic. Allow me to through out one example of why I think there is more to consider. Using 9 tanks per salt brand each with a different batch of that salt for each of the 10 brands would have allowed for accountability of variation within and between brands- but obviously 90 tank setups was not in the budget (and lets face it, with basic research funding on the decline it will be difficult for anyone to say that top facilities would be employed in a study to benefit a hobby only a small % of people in the world richest countries are involved in when there are so many bigger issues not even funded). OK so working with a more manageable # of tanks- you could test the various brands in the same tank over time replicating this in 10 tanks using the same batch for each brand in each tank, but this risks Bias based on the order tested during the aging of the tanks and it would be difficult to account for variability within a brand as trying different batches in this set up would add another variable that would be difficult to control for. You could also test several batches of the same brand in separate highly similar systems over time. This will allow for within brand variability to be tested and also between brand variability. Also the 100% water changes to a new batch of the same brand could be treated as a new sample to measure and thus increase the N to 9 samples each brand, independent in that the salt was of a different batch. I like the second option, the one used in the recent study. As for the measures of variability you will never account for all variability in a biological systems, instead you must find ways to account for the fact that there is variability. For example if the effects of drugs on all genetic backgrounds was required prior to release we would have no medication. It is sometimes necessary to make decisions based on observation of a subset of a population. Furthermore qualitative observation is admittedly not as good as quantitative measure, however often without it, progress would never be made. A small observation in one study may lead to someone else having an idea and doing a larger study based on that observation. I truly hope that if Mr. Borneman ever reads this thread (which I hope he doesn’t) that he will shake it off, not be disscouraged, and realize there are also many who do not view the study as “worthless”, to the contrary, I commend his efforts and look forward to his peer reviewed published results. I would also encourage him to continue with any contributions he is willing to make to the hobby and to coral reef health and conservation. Just had to get my two cents out there as reading this whole thread commanded a response even if it will be attacked or go unheard.

 

[Boomer]

In my opinion the critiques appear to come from educated people who are NOT experts in the area.

Many who are "experts" in this area disagree with the salt study, me included.

It can take six months or more to get a peer reviewed paper in print after all data is gathered and results are in.

You will NEVER see it in a peer-reviewed article, I'll bet you anything on it.

If such efforts are met with this sort of ”venomous” opposition

It is venomous because of the way it was done. Ron's study was also a flop. They both interpret data based on assumptions, without looking at the cause. That is poor science. Neither study tells you anything about the salt in question, it is all assumptions.

when you start attacking a person's personality your credibility drops to zero

You must not be aware of all the personal attacks Eric has done on Julian Sprung, Dr. Randy Holmes-Farley and Bomber. Calling them all fake chemists or biologists. Yet Eric has NO degrees or peer-viewed articles in marine biology or chemistry but attacks others that do. Why because they have challenged him and shown him wrong, therefore try to discredit them. Ron tried to do the same thing to Randy a couple of years ago.

There are more salt studies than the two mentioned, like 3 of them. These deal with assays of the salt mix.

There have long thread on RC on all of this and even some in FAMA.

I do not condemn Ron or Eric for their studies, it took time and money but if you are going to do a study try to do it right, where the study has some credit. Ron's study would have been a really great study if he did not base his conclusions on assumptions of bad data from mixing salt assay, years apart and the way he presented that data. All Ron had to do in his study is to find out what killed the urchin larvae. Pretty simple but it was never undertook. You have to find out WHY.

 

[sihaya]

I guess part of my "venom" comes from the fact that Borneman keeps insisting on describing the study as "rigorous" science. If he were instead admitting the great limits to the study and if he were a bit less certain that this would pass peer review (which I can't imagine it ever could), then I might be a little more "gentle" in my discussion of the study. But Borneman keeps talking about this study like it's going to determine with some kind of certainty, once and for all, which salt is the best. And that irks me.

 

[sihaya]

Well, it's been about 7 months now since MACNA last year... anyone want to wager when the "final" results for this thing will come out? lol

 

[Esper]

Data? we don't need data! Eric's opinion has already been published:

“I will say that, at least from being familiar with the data and the appearance of the tanks and the survival and appearance of the species that there are certainly some salts I would prefer to use in my tank and some that I probably wouldn't use again, even though I have used them for many years without any obvious negative effects. I have a feeling that the reason these differences are not observed by those using these salts, myself included, is that the complexity of the reef tank community is able to mitigate the good and bad aspects of the salts that became apparent in a more controlled environment.”

http://reefkeeping.com/issues/2007-03/rhf/index.php

Of course, what he is really saying is that the results of the study don't matter...whatever they are. Nevertheless, Eric has managed to form an opinion anyway.

This is what he sounds like:
"I will say that, based on being a Scientist and a Coral Guru, that I am going to completely change my opinion and behavior based on information that I can't tell anyone, even though it contradicts all of my experiences in the past. I believe it contradicts these experiences, and those of others, because the data has nothing whatsoever to do with real life."


:lol:

 

[mojoreef]

Well I wrote this big long post and the deleted it because it was just to long winded and didnt mean anything..just like most posts/studies and conversations on thus subject.
But here is a brief outline for you. All ASW mixes suck...they are crudely manufactured/collected mixed and are made for different clients, they have to be... its money. Reef keeping is minute compared to fish keeping, and in fish keeping most of what we are talking about doesnt pertain. If they didnt make ASW so crudely we wouldnt have a hobby..we couldnt afford it. Go out and by analytical grade components and the equipment to measure and mix it correctly and you will then join those that dont do WC's any more It would cost hundreds and hundreds of dollars to put a bucket of ASW mix together.
If oine looks at ICP tests done on ASW mixes no matter who has done them it always comes out that ASW has more contaminates the natural SW studies, they are either high in some elements that we want or low, some worse then others. SOme have high contents of some kind of metals, some others and so on... do we know if this pertains?? NOPE no study has ever shown that and how these metals react in our tanks is always different and also for the most part unkown.
So what do we do?? Well if the contents of ASW scares you...well you have a few options. One is to buy the components and make your own...good luck though they are going to be expensive. The second would be to not rely on ASW mixes and not use it after your initial start up. The third is the most popular choice, Ignore all the Bunk and use a ASW mix that other sucessful reefkeepers have used. The G-pig method, lol


ANyway more coffie is now needed.


Mike

 

[Elmo18]

I'm such a guinea pig ....

Mike how are you doing?

Best,
Ilham

 

[Esper]

Excellent post, Mike! I think you described the bigger picture really, really well.

Researching the effects of different ASW mixes in our tanks is an EXCELLENT idea. As with all things, the first attempts may be slightly flawed, but we learn and improve over time. Science works by natural selection too!

You're right, though, in the end we will all just use whatever salt everyone else uses, lol.

 

[mojoreef]

DOing good ELmo how about you??

MIke

 

[Damsel13]

I'm understanding from this thread that if it isn't broke don't fix it. My Dad taught me that when I was 16 or whenever it was I started using a car.

Great reading.
Thank you,