A Better Salt Study Thread

[Aquarium Scienc]

Testing multiple samples of as many brands of marine salts is a worth while project. This would indicate the consistency of each brand.

Part of the test should include written documentation regarding which brands mix completely and into a clear solution. This would be a basic indication of the quality of the ingredients employed to formulate the mixture.

These are some of the parameters and observations that can be performed by a competant independent lab.

After that, qualified marine system manager(s) could evaluate each brands ability to produce a solution that maintained the proper parameters for a given period of time. This would also include a written record of which brands produced unwanted algae, etc.

At no time should any salt maker be notified of this test. At no time should any salt samples be obtained directly from any salt manufacturer.

The lab. chosen should have no connection to any salt manufacturer. However great care must be taken to secure the services of an independent lab. that is qualified to test complex liquid chemistry such as NSW or synthetic sea salts.

The difficulty for such a test is when funding is realized. As witnessed by another effort that solicited funds (via internet) from interested hobbyists, they fell far short of the mark to achieve the required funds to procure a qualified lab.

The S-15 was funded by an international group of independents that created the funding orginazation that commissioned the report for that specific project. When those test results were offered, many internet experts made negative comments regarding the funding orginazation (although clearly in print), vs. the actual results.

If a future legitmate salt study was performed, it should be printed in book form and not have bits and pieces put on the internet. The final report (in book or booklet form) should be protected by copyright laws. If it was offered on the internet, it should be offered as a pay for download form.

This would greatly reduce the number of arm chair experts that had nothing better to do than make unqualified comments. Thus the legitmate effort would gain respect and be appreaciated for what it is.

 

[MikeS]

To say that Eric's studies were "flawed initially" is an insult.

Dan...yes, this salt study, just like the substrate depth study, is highly flawed...if that's an "insult" to them, well then...maybe they need to qaulify what they are doing a bit better, rather than passing it off as gospel with a bunch of statistical nonsense behind it...

I applaud anybody who is willing to take the time to do some real research and experiment...where I have a problem is when a half baked study is pawned off on the rest of the hobby as legitimate, with a bit of statistical mumbo-jumbo to make it sound good, that most people don't even understand...I'd have a ton more respect for these guys if they would at least own up to the flaws in the study in the first place....

And let's keep it civil...we love to debate, and this is a good subject to do so in...but keep it nice...we don't want to close the thread...

MikeS

 

[Steven Pro]

I am just curious. Can someone explain to me why an N of 1 in Eric salt study makes it complete crap and why the same N of 1 in Sanjay lamp and reflector testing does not illicit the same venom?

 

[sihaya]

Steven- do you have a link to where Sanjay describes what he did? I'm not familiar with his methods. Maybe his testing does deserve the same "venom."

However, let me repeat this:
"We show that there can be extreme variation among identical tanks, even without any live animals" - Toonen and Wee (http://www.advancedaquarist.com/2005/7/aafeature)


I would think that lighting units would have FAR fewer potential variables than whole tank systems. But right now, I don't know enough about the lighting systems or what Sanjay did to say if those potential variables create the same fatalness in the n=1 as it does for Borneman's study.

I do believe that most aquarium studies/trials done by hobbyists are far from rigorous enough to be considered true science. We could probably attack them all to some extent or another. It's simply that this salt study is very recent and just sticks out in my mind as a "classic" bad hobby experiment. Also, I'm more inclined to tearing apart the work of graduate students looking to earn their place among real scientists than I am tearing apart the work of an average hobbyist merely pursuing curiousity. They're just two different playing fields (though, yes, it would be nice if they were closer than they are).

 

[Steven Pro]

See anything Sanjay has written in Advanced Aquarist on lamps or reflectors.

 

[sihaya]

You mean like in this? http://www.advancedaquarist.com/2005/12/review

In those trials the n=1 is still a problem, it's just not a fatal problem like it is for the salt study. The key difference is that Dr. Joshi knows how the different ballasts differ. He knows specifically how the ballasts differ in performance and he reports their specific differences in the data presentation. Borneman and Lowe don't know how the brands differ chemically or otherwise. And even if they did, they wouldn't likely know what those differences might mean for the the results of the study. In Dr. Joshi's case, since he's working with electronics with much more predictable "behavior," he can possibly understand how the ballast differences effect his data.

But I'll have to read more about ballasts and Joshi's work before I get any more specific. Thanks for drawing the comparison! I think it's a good one and I look forward to discussing it more later.

 

[Boomer]

I would think that lighting units would have FAR fewer potential variables than whole tank systems

I would have to agree with Sara, Steve. Reflectors do not change their dynamics anywhere near the same rate as a seamix and do not contain living breathing life, complicating matters even more.

 

[morgan]

u should just use sea water u get biodiversity and no hassels

 

[sihaya]

I would have to agree with Sara, Steve. Reflectors do not change their dynamics anywhere near the same rate as a seamix and do not contain living breathing life, complicating matters even more.

Not only that, but Dr. Joshi has been diligent enough to report the differences in power watts, imput volts, imput amps, ppfd and cct, and efficiency for each ballast. He's dealing with the variables by identifying and measuring them.

 

[Steven Pro]

But, you are making the assumption that quality control makes the lamps and reflectors consistent. I have seen enough larger displays with three lamps or more using the same brand of ballast and lamps and seeing lamps that appear completely different.

 

[sihaya]

But, you are making the assumption that quality control makes the lamps and reflectors consistent.

No, no, not at all... what I'm saying is that Joshi allows us to look at his data in full light of the variables between the ballasts (no pun intended lol). He gives us all the info we need to consider how the differences between the ballasts may or may not be effecting the data.

I'm not saying that his study wouldn't be better with a n>1, I'm only saying that his n=1 is not fatal... i.e. it doesn't render his study and data meaningless.

I have seen enough larger displays with three lamps or more using the same brand of ballast and lamps and seeing lamps that appear completely different.

Sure, I don't doubt that. But again, Dr. Joshi seems to understand that too... which is why he's reported the ballast differences so that we can consider them ourselves.

 

[Steven Pro]

I think you are missing something Sara. Sanjay tests one lamp on multiple ballasts, so we have an accurate view of how that one lamp will respond to various ballasts, but we are assuming that the one lamp is representative of that brand/color temperature. I don't believe that is the case.

 

[sihaya]

I think you are missing something Sara. Sanjay tests one lamp on multiple ballasts, so we have an accurate view of how that one lamp will respond to various ballasts, but we are assuming that the one lamp is representative of that brand/color temperature. I don't believe that is the case.

OOOoohh... I see what you are saying. You're not as worried about the differences in the ballasts as in the lamps. I should have realized that. Sorry. In the case of the lamps, yes, that does seem like a serious concern.

But again, I don't know enough about lighting equipment at the moment to given a meaningful opinion just yet. Give me the rest of the day to do some more in depth reading...

 

[Steven Pro]

Differences in lamps, ballasts, reflectors, the wiring in the end users home, power draw in the circuit, etc. All of these could concievably impact the final PAR and appearance for the consumer. But, I don't see people freaking about about this like I do when the subject of salt or methodolgy comes up. It just strikes me as strange that certain topics or individuals (I don't know which, perhaps both) create this much heated discussion.

By the way, I like Sanjay and his testing, but I also accept it for what it is, a N=1 study that has some limitiations. But, I don't get all fired up about it either.

 

[sihaya]

To be honest, Joshi's trials always confused me a little. He talks about comparing the lamps, but whenever I'd look at his results, his data always seemed more worthwhile in comparing the ballasts.

I'm going to look up what makes lamps different (from one lamp to another even within a brand). For salts, I have a better understanding of how they can differ in chemical consistency within a brand. But with lamps, I have hardly any idea how specifically one lamp differs from another within a brand... or if those differences can be measured and understood. And that's what I'd have to know before I could say too much about how damaging the n=1 factor is.

 

[sihaya]

But, I don't see people freaking about about this like I do when the subject of salt or methodolgy comes up. It just strikes me as strange that certain topics or individuals (I don't know which, perhaps both) create this much heated discussion.

Well, I can only speak for myself, but I never "freaked out" about Joshi's trials because they came out before I started in the hobby. This salt study has been all the rage only since I started reef keeping. If Joshi's trials had made their grand entrance into the hobby in just the past few months, I would have probably "freaked out" about them too.

By the way, I like Sanjay and his testing, but I also accept it for what it is, a N=1 study that has some limitiations. But, I don't get all fired up about it either.

Well, that's good then. I'm a law student, so naturally, I get fired up pretty easily I guess. And you have sparked new interest for me in these lighting trials now. I do plan to look into it in more depth.

And I've always liked Dr. Joshi too. But I'm afraid that wouldn't spare his trials the same brutality I show any other study (if I knew enough about lighting equipment, which I don't yet).

Steven, you're a smart guy and so you know how to take these n=1 studies with a grain of salt... but you'd be surprised how many average and newbie hobbyists aren't so sharp. I've heard of people actually waiting to set up their tanks because they want to start off with the "right" salt (as they believe will be determined definitively by the Borneman/Lowe salt study). And I'm sorry, but that does concern me. I'd rather hobbyists be more critical of these things.

 

[Esper]

Differences in lamps, ballasts, reflectors, the wiring in the end users home, power draw in the circuit, etc. All of these could concievably impact the final PAR and appearance for the consumer. But, I don't see people freaking about about this like I do when the subject of salt or methodolgy comes up. It just strikes me as strange that certain topics or individuals (I don't know which, perhaps both) create this much heated discussion.

By the way, I like Sanjay and his testing, but I also accept it for what it is, a N=1 study that has some limitiations. But, I don't get all fired up about it either.

Sanjay's website (the ballast/lamp calculator) used to have results from 2 different bulbs of certain brands. The ones I looked at varied in PAR quite a bit...I'm not sure if the website is still set up this way. (if so, one could reconstruct some of sanjay's tables using n=2 of some bulbs). I remember wondering at the time if that was because of the bulb's age, firing frequency, and all the stuff you mentioned above, or if it was differences in manufacture. So many factors are involved in bulb output that I looked at the results like sihaya did, as a comparison across ballasts for certain bulbs, not a comparison across bulbs of certain ballasts. (i'm not sure if this is the way sanjay presents the articles). this is like a multiple baseline across bulbs, where each bulb is held constant and the ballasts are the "treatments." this does not detract from the study's internal validity. at the same time, including more than one of each brand bulb would certainly yield useful information regarding the study's external validity.

I keep expecting sanjay to publish experiements examining bulb output over time (using typical aquarium photoperiods), perhaps comparing different bulb brands, ballast types, or multiple samples of the same brands. i guess there's no shortage of experiments to be run, especially as new products come out all the time. you pointed out very well that for the end user, with these studies, as in Borneman et. al's study, YOUR MILEAGE MAY VARY.

 

[Boomer]

You can not be comparing light bulb output, etc. to salt or salt water chemistry, it is shear nonsesne. They are not even remotely close. It is like comparing apples to a dump truck.

 

[sihaya]

I agree, they're just two totally different things to be studying. Even as lamps vary from lamp to lamp, they're still less complicated than all the chemistry of a salt mix in water. Sanjay himself says that the lamps vary widely and that this is a big problem (not just in studying them, but in actually using them for the hobby.) In fact, if I understand him, part of the point he tries to make is that if you really want to know the PAR output of your light, you need a meter. And his work isn't trying to tell you which lamps performs "better" overall he's just reporting collected data and letting you figure it out (if there's anything to figure out).

Also, there's a leap from reporting what a thing is doing my itself to trying to report on how what it's doing is effecting something else (or a whole variety of things). Sanjay is reporting what the lamps are actually doing (what they're in/outputting). Borneman/Lowe skip that step. They report not on what the salt mixes do (i.e. the varying chemical compositions when mixed with water), he's reporting their effects on these complex reef-tank-like units that are inevitably going to be very inconsistant.

Look, any study were n=1 is going to be hard to bank on. However, I have a lot more respect for Sanjay's work because it doesn't try to be anything more than what it is. The Borneman salt study promises to enlighten us all about how the salts compare to each other in performance in a "reef tank"... and that's just not something it can do.

 

[MikeS]

agreed...

these studies have their limits for sure...but I think the researchers should admit them...and not make conclusions based on their statistics...if they were APA members and submitted studies like the substrate one and made conclusions based on thier experiments like that...they would have been tossed out of the APA so fast it would have made thier heads spin for mis-representing thier statistical findings...

MikeS