How much should marine aquarists rely on anecdotal evidence?
An easy question to ask; hard to answer. There are so many opinions and many of those opinions are based upon anecdotal information. Those new to marine aquariums always seem to find some kind of ‘revelation’ that those of us in the hobby for 35+ years failed to find. Someday, a person may find something new in marine fish husbandry, but how will we sort out opinion from fact, or anecdotal evidence from scientific evidence? How formal is formal enough in our hobby? I hope to shed some light on the answers to these questions in this post.
Besides people posting their opinion and sharing anecdotal information in forums, there is a flood of products which claim to cure, maintain, or prevent marine fish conditions. With those products are a group of people with anecdotal evidence regarding how well those products perform. The anecdotal evidence sells more product than formal data collected on the product performance. Sometimes the real answers are found in the marketing language and package inserts or labeling. It pays to read labels not only for what they say, but for what they don’t say.
-------------
A definition is about due:
In science, anecdotal evidence has been defined as:
--------------------
I think it’s important to note that the above definition/description of anecdotal evidence does not include any judgment of being right or wrong, good or bad. Observations are worth noting and discussing. If you look through the list you may pick up on a common thread – this form of evidence is weak. At some point in time this sort of evidence needs to move on to become formal scientific evidence. There are several different ways of moving from anecdotal evidence to more formal evidence. Let me take as an example a human case.
Alcohol is Good for You! - An example of anecdotal evidence moving towards scientific evidence
The Stages
1) It was observed that some people who drink alcohol lived longer. Someone made this observation and it was totally anecdotal at the time. This comes in the form of, ‘Uncle Ned drank all the time and lived to be 95.†(The anti-observation: ‘Uncle Sam drank all the time and died at 49 with a liver condition.’) This observation begs the question, 'Is drinking alcohol good for you?'
2) With the question raised (and money from the alcohol producers), others moved on to either prove or disprove the observation. More observations were made. It seemed to have some truth to it because it was confirmed by different people in different parts of the globe. There seemed to be a relative agreement from several perspectives.
3) The next step was to test the evidence. Control groups and different groups were evaluated and lo and behold, some who consume liquor did in fact live longer. It was not understood why, but the evidence supported the generality. Mostly scientists are involved in this effort.
4) The evidence required a more formal approach. How much alcohol and how often did its consumption improve longevity? Data by experiment and survey was gathered that showed a small quantity, daily had the desired effect. More volume and the negative effects would set in. Less volume and there seemed to be little gain. Not all people had the same response, but those that had no increase in longevity seemed at least not so show signs of a decrease in longevity. Now, consuming a shot of hard liquor, a glass of wine, or a bottle of beer each day (and no more) might be a good thing to do. Scientists are moving forward.
5) The next step was to determine what ‘living longer’ meant in health terms. Just what is happening to cause a longer life? After additional tests, studies, and data gathering, it was proven that the people who drank small quantities of alcohol daily had an improved cardiovascular health.
6) The next step in the scientific realm is to determine why or how cardiovascular health is improved by alcohol. It wasn’t until just a couple of months ago that scientific proof was presented as to the mechanism by which alcohol does in fact improve cardiovascular health. I won’t elaborate further, but it is connected to blood sugar management within the body and the effect alcohol has on that management.
7) The final step was now to advise how much and of what kinds of alcoholic beverages, drank how often would be of benefit. This now was something that could be recommended, backed by scientific evidence.
Discussion of the Stages
From completely weak observation (anecdotal evidence) came formal proof (scientific evidence). Of course, the liquor industry was thrilled!
At the first stage, 1) the observation wasn’t worthless, but it shouldn’t be the cause of everyone running out to become alcoholics. A lifestyle change is not made with that kind of casual observation. Note that it was an individual’s observation and at this stage, could be accurate or not.
At the next stage, 2) the observation had to be confirmed by others. Even though this observation was further confirmed, it is still not a reason for everyone to go out and become alcoholics. It is and was known that negative effects of alcohol consumption was health damaging. Can you hear your parents saying, ‘Would you jump off a cliff just because your friends did it?’
Stage 3) began the formal gathering of data from groups of people and societies. Groups of people were put together and fed alcohol at some given quantity and frequency to see what happened. NOTE: This is the first point in the example that actual data is being gathered.
At stage 4) it was determined that the quantity of alcohol and frequency was important. The original observation that suggested uncontrolled quantities of alcohol was the cause of extending the life, was not correct. Moderation seemed to reduce the negative affects, yet still extended life in some people. This stage is very important. It was the first effort to formalize the evidence into a form that might form the basis for a recommendation or guidance.
At stage 5) defining the effects of the observation in health terms was accomplished. It was found that the underlying effect of small quantities of alcohol consumed daily improved the cardiovascular system. This usually led to a longer life. Now scientists has sometime more consistent to measure rather than longevity. So the long lives was not the real target – it was cardiovascular health that was effected. The original observation was not totally correct -- that alcohol extended life. What the alcohol was really doing was improving the cardiovascular health of the individual which could lead to extended life. Now the evidence is taking on strong scientific validation.
Stage 6) took a long time. Science had to catch up with the evidence at hand. Thousands of people were working on the question, ‘What did the alcohol consumption mean to cardiovascular health?’ and, 'Why?' 'What was the alcohol doing to the system?' It took a long time, but just a few months ago it was discovered that the alcohol affected how the body manages blood sugar which translated into increased cardiovascular health. Now this can all be formally tested to verify the findings.
Stage 7) is now the final recommendation based upon scientific evidence. It can be cleaned up and what is said has scientific backing. (BTW, only certain alcoholic beverages are recommended).
Compare stages 1) and 7). Without at least getting to stage 4) there is no guidance on what to do. Just guessing or. . .anecdotal evidence.
Back to Marine Aquariums
A new ‘miracle cure’ product enters into the marine fish market. Is it good or not? Does it work, or not? Should you buy it?
The aquarist gets a stage 1) opinion. Supported by others, refuted by some, there is hope for a stage 2) situation. Shouldn’t the aquarist expect to have stage 3) and stage 4) evidence before they spend their money and risk their marine fish? By this I mean, in plain English, where is the data and hard evidence that the product works as advertised? We don't need to go beyond simple proof and data. We don't require the big bucks to get the details. It is nice to have, but not that much money is put into this kind of scientific study. But what does it really take to get some data?
Let’s say the product is a new cure for Marine Ich (Cryptocaryon irritans). What does it take to get to a stage 3) is relatively easy and very inexpensive. The product is targeted to cure Marine Ich, so why not have it independently tested?
It’s not that difficult or costly. A few different aquarists around the world or a single independent study is conducted using the product. Fish in tanks are infected with and verified to have Marine Ich and the cure applied as directed. Are they cured? Were they in fact cured by the product or by some other means like immunity or resistance? The last two possibilities can be reduced or accounted for in the test matrix.
(I spent two years of my college life doing just the above – testing possible cures for Marine Ich).
The cost is today about $500 per test aquarist or about $2,000 for an independent study. Some institutions might be willing to conduct tests for free. Maybe you disagree with the cost. But if the product was always successful (100% like the known cures for this parasitic disease) wouldn’t the $ come back to the manufacturer/inventor several fold?
Consider how much $ a company will spend in advertising, putting up and manning booths at conventions, gathering testimonials, etc. If they can spend this $ in marketing, why not have scientific evidence to back them up and save in advertising. Wouldn’t the evidence be a persuasive sales point? This would be a miracle product bought not just by hobbyists/aquarists but also the marine aquaculture industry, professionals, veterinarians, public and private aquariums, etc. probably making the manufacturer/inventor wealthy.
But why don’t they do this independent testing? Isn’t it obvious? They don't give us stage 3) and 4) data because it doesn't exist and they can't generate it.
Sifting through the Hype
Get the facts.
If the information has come to you on the Internet or forum sites, check it out with others. Seek professionals and experienced commenters, not just others with anecdotal information. That is reinforcement for nothing.
If it's a product staring you in the face, then getting the facts begins with reading between the lines. What exactly is being promised on the label. Does it just say it ‘helps’ or does it say for sure that it ‘cures?’ There is an important difference. Does the product ‘inhibit’ or ‘kill?’ If the product kills, will it completely ‘cure’ the fish. There are differences between these words and how they are used in sentences.
Did you read on the label that a product does in fact do something? Then politely and respectfully communicate with the manufacturer and ask what studies have been done to substantiate the label claims?
Even a company that wants to maintain its proprietary or patented information can have independent tests performed. Those test results can be shared. No data from them? Don't buy their product.
One of the greatest voids in the marine aquarist industry is there are no regulatory controls in the industry. Some new aquarists think that the manufacturer couldn’t possibly lie in their advertising or label information. Think again. There is no government agency which regulates what they say. An individual would have to personally begin litigation against those making the product claims. The manufacturers know this isn’t worth it to the aquarist. Besides that, there are those tricky words like ‘May’ or ‘Could’ or ‘Help’ woven into the marketing text that remove or lessen their liability.
One of my favorite statements on a label is that the product, ‘Contains ingredients known to kill XXX.’ So? That doesn’t mean the product kills all the XXX, does it? It doesn’t mean the fish will be totally cured. Many products have been shown to kill xxx in the laboratory. What test data has been collected on its success in an aquarium at the recommended dosage?
Aquarists more than most hobbyists, need lessons in how to read labels. Knowing what the label really says and what it doesn’t say is almost an art.
Regarding anecdotal evidence. – it is often used to sell products; it is often used by posters to encourage others to try things. The buyer and reader has to be aware at what point that ‘evidence’ is weak or solid. It is often just a testimonial by a person who doesn’t thoroughly understand the role the product played – what it did or didn’t do.
Too often the anecdotal evidence has been refuted by scientific evidence or failed to be supported by scientific evidence. The person making the anecdotal claim is oblivious to the facts already gathered. The use of garlic is one such product – it doesn’t do most of what the hobbyists think it does, let alone what we all would hope it did. Why is it available? Because people are wanting to buy it. It improves appetite: no scientific evidence has supported this claim. Tests that have been conducted show it does not increase appetite with one lone exception. What it does do is irritate the fish. This is an excellent read on the garlic matter: Garlic Article
There’s an old claim that garlic improves immunity: no scientific evidence has supported this claim. Never proven, but it is not hard to do. But no one has publicly provided data evidence, so there is only stage 1) or stage 2) evidence. The problem with stage 1) and 2) evidence is that immunity is not something that can be 'seen' in a fish. The blood of the fish has to be tested to verify immunity. Immunity can’t be observed or separated from a resistance. But it can be tested for and even quantified. So how could it even get a stage 1) or stage 2) level of evidence? Rumors.
A trap exists when a product is listed as “reef safe.†There are no standards, regulations, or guidance as to what reef safe means. Steven Pro did some experiments with pulsating Xenia and reef safe products and found some of the products harm, retard, or slowly kill Xenia. Are they reef safe?
A product manufacturer is not likely to test thousands of marine life forms to claim it is reef safe. This isn’t reasonable. However, the aquarist has the right to inquire why the manufacturer listed it as reef safe. What marine life was it tested against? Aquarists have to ask this to protect the marine life in their care.
The sorting out of the truth of anecdotal evidence is up to the aquarist. You can go online and get information, or ask questions of others for their opinion, but if you’re just going to gather anecdotal evidence, you need to determine the real value of that information and take into account what is known vs. what is thought to be going on. Keep an open mind even on things you think you’ve gather some anecdotal evidence about. Seek out data and the more formal evidence. Be sure contra-data doesn’t exist before taking the plunge into following testimonials and group advice. You and the hobby will greatly benefit.
An easy question to ask; hard to answer. There are so many opinions and many of those opinions are based upon anecdotal information. Those new to marine aquariums always seem to find some kind of ‘revelation’ that those of us in the hobby for 35+ years failed to find. Someday, a person may find something new in marine fish husbandry, but how will we sort out opinion from fact, or anecdotal evidence from scientific evidence? How formal is formal enough in our hobby? I hope to shed some light on the answers to these questions in this post.
Besides people posting their opinion and sharing anecdotal information in forums, there is a flood of products which claim to cure, maintain, or prevent marine fish conditions. With those products are a group of people with anecdotal evidence regarding how well those products perform. The anecdotal evidence sells more product than formal data collected on the product performance. Sometimes the real answers are found in the marketing language and package inserts or labeling. It pays to read labels not only for what they say, but for what they don’t say.
-------------
A definition is about due:
In science, anecdotal evidence has been defined as:
- "information that is not based on facts or careful study"
- "non-scientific observations or studies, which do not provide proof but may assist research efforts"
- "reports or observations of usually unscientific observers"
- "casual observations or indications rather than rigorous or scientific analysis"
- "information passed along by word-of-mouth but not documented scientifically"
--------------------
I think it’s important to note that the above definition/description of anecdotal evidence does not include any judgment of being right or wrong, good or bad. Observations are worth noting and discussing. If you look through the list you may pick up on a common thread – this form of evidence is weak. At some point in time this sort of evidence needs to move on to become formal scientific evidence. There are several different ways of moving from anecdotal evidence to more formal evidence. Let me take as an example a human case.
Alcohol is Good for You! - An example of anecdotal evidence moving towards scientific evidence
The Stages
1) It was observed that some people who drink alcohol lived longer. Someone made this observation and it was totally anecdotal at the time. This comes in the form of, ‘Uncle Ned drank all the time and lived to be 95.†(The anti-observation: ‘Uncle Sam drank all the time and died at 49 with a liver condition.’) This observation begs the question, 'Is drinking alcohol good for you?'
2) With the question raised (and money from the alcohol producers), others moved on to either prove or disprove the observation. More observations were made. It seemed to have some truth to it because it was confirmed by different people in different parts of the globe. There seemed to be a relative agreement from several perspectives.
3) The next step was to test the evidence. Control groups and different groups were evaluated and lo and behold, some who consume liquor did in fact live longer. It was not understood why, but the evidence supported the generality. Mostly scientists are involved in this effort.
4) The evidence required a more formal approach. How much alcohol and how often did its consumption improve longevity? Data by experiment and survey was gathered that showed a small quantity, daily had the desired effect. More volume and the negative effects would set in. Less volume and there seemed to be little gain. Not all people had the same response, but those that had no increase in longevity seemed at least not so show signs of a decrease in longevity. Now, consuming a shot of hard liquor, a glass of wine, or a bottle of beer each day (and no more) might be a good thing to do. Scientists are moving forward.
5) The next step was to determine what ‘living longer’ meant in health terms. Just what is happening to cause a longer life? After additional tests, studies, and data gathering, it was proven that the people who drank small quantities of alcohol daily had an improved cardiovascular health.
6) The next step in the scientific realm is to determine why or how cardiovascular health is improved by alcohol. It wasn’t until just a couple of months ago that scientific proof was presented as to the mechanism by which alcohol does in fact improve cardiovascular health. I won’t elaborate further, but it is connected to blood sugar management within the body and the effect alcohol has on that management.
7) The final step was now to advise how much and of what kinds of alcoholic beverages, drank how often would be of benefit. This now was something that could be recommended, backed by scientific evidence.
Discussion of the Stages
From completely weak observation (anecdotal evidence) came formal proof (scientific evidence). Of course, the liquor industry was thrilled!
At the first stage, 1) the observation wasn’t worthless, but it shouldn’t be the cause of everyone running out to become alcoholics. A lifestyle change is not made with that kind of casual observation. Note that it was an individual’s observation and at this stage, could be accurate or not.
At the next stage, 2) the observation had to be confirmed by others. Even though this observation was further confirmed, it is still not a reason for everyone to go out and become alcoholics. It is and was known that negative effects of alcohol consumption was health damaging. Can you hear your parents saying, ‘Would you jump off a cliff just because your friends did it?’
Stage 3) began the formal gathering of data from groups of people and societies. Groups of people were put together and fed alcohol at some given quantity and frequency to see what happened. NOTE: This is the first point in the example that actual data is being gathered.
At stage 4) it was determined that the quantity of alcohol and frequency was important. The original observation that suggested uncontrolled quantities of alcohol was the cause of extending the life, was not correct. Moderation seemed to reduce the negative affects, yet still extended life in some people. This stage is very important. It was the first effort to formalize the evidence into a form that might form the basis for a recommendation or guidance.
At stage 5) defining the effects of the observation in health terms was accomplished. It was found that the underlying effect of small quantities of alcohol consumed daily improved the cardiovascular system. This usually led to a longer life. Now scientists has sometime more consistent to measure rather than longevity. So the long lives was not the real target – it was cardiovascular health that was effected. The original observation was not totally correct -- that alcohol extended life. What the alcohol was really doing was improving the cardiovascular health of the individual which could lead to extended life. Now the evidence is taking on strong scientific validation.
Stage 6) took a long time. Science had to catch up with the evidence at hand. Thousands of people were working on the question, ‘What did the alcohol consumption mean to cardiovascular health?’ and, 'Why?' 'What was the alcohol doing to the system?' It took a long time, but just a few months ago it was discovered that the alcohol affected how the body manages blood sugar which translated into increased cardiovascular health. Now this can all be formally tested to verify the findings.
Stage 7) is now the final recommendation based upon scientific evidence. It can be cleaned up and what is said has scientific backing. (BTW, only certain alcoholic beverages are recommended).
Compare stages 1) and 7). Without at least getting to stage 4) there is no guidance on what to do. Just guessing or. . .anecdotal evidence.
Back to Marine Aquariums
A new ‘miracle cure’ product enters into the marine fish market. Is it good or not? Does it work, or not? Should you buy it?
The aquarist gets a stage 1) opinion. Supported by others, refuted by some, there is hope for a stage 2) situation. Shouldn’t the aquarist expect to have stage 3) and stage 4) evidence before they spend their money and risk their marine fish? By this I mean, in plain English, where is the data and hard evidence that the product works as advertised? We don't need to go beyond simple proof and data. We don't require the big bucks to get the details. It is nice to have, but not that much money is put into this kind of scientific study. But what does it really take to get some data?
Let’s say the product is a new cure for Marine Ich (Cryptocaryon irritans). What does it take to get to a stage 3) is relatively easy and very inexpensive. The product is targeted to cure Marine Ich, so why not have it independently tested?
It’s not that difficult or costly. A few different aquarists around the world or a single independent study is conducted using the product. Fish in tanks are infected with and verified to have Marine Ich and the cure applied as directed. Are they cured? Were they in fact cured by the product or by some other means like immunity or resistance? The last two possibilities can be reduced or accounted for in the test matrix.
(I spent two years of my college life doing just the above – testing possible cures for Marine Ich).
The cost is today about $500 per test aquarist or about $2,000 for an independent study. Some institutions might be willing to conduct tests for free. Maybe you disagree with the cost. But if the product was always successful (100% like the known cures for this parasitic disease) wouldn’t the $ come back to the manufacturer/inventor several fold?
Consider how much $ a company will spend in advertising, putting up and manning booths at conventions, gathering testimonials, etc. If they can spend this $ in marketing, why not have scientific evidence to back them up and save in advertising. Wouldn’t the evidence be a persuasive sales point? This would be a miracle product bought not just by hobbyists/aquarists but also the marine aquaculture industry, professionals, veterinarians, public and private aquariums, etc. probably making the manufacturer/inventor wealthy.
But why don’t they do this independent testing? Isn’t it obvious? They don't give us stage 3) and 4) data because it doesn't exist and they can't generate it.
Sifting through the Hype
Get the facts.
If the information has come to you on the Internet or forum sites, check it out with others. Seek professionals and experienced commenters, not just others with anecdotal information. That is reinforcement for nothing.
If it's a product staring you in the face, then getting the facts begins with reading between the lines. What exactly is being promised on the label. Does it just say it ‘helps’ or does it say for sure that it ‘cures?’ There is an important difference. Does the product ‘inhibit’ or ‘kill?’ If the product kills, will it completely ‘cure’ the fish. There are differences between these words and how they are used in sentences.
Did you read on the label that a product does in fact do something? Then politely and respectfully communicate with the manufacturer and ask what studies have been done to substantiate the label claims?
Even a company that wants to maintain its proprietary or patented information can have independent tests performed. Those test results can be shared. No data from them? Don't buy their product.
One of the greatest voids in the marine aquarist industry is there are no regulatory controls in the industry. Some new aquarists think that the manufacturer couldn’t possibly lie in their advertising or label information. Think again. There is no government agency which regulates what they say. An individual would have to personally begin litigation against those making the product claims. The manufacturers know this isn’t worth it to the aquarist. Besides that, there are those tricky words like ‘May’ or ‘Could’ or ‘Help’ woven into the marketing text that remove or lessen their liability.
One of my favorite statements on a label is that the product, ‘Contains ingredients known to kill XXX.’ So? That doesn’t mean the product kills all the XXX, does it? It doesn’t mean the fish will be totally cured. Many products have been shown to kill xxx in the laboratory. What test data has been collected on its success in an aquarium at the recommended dosage?
Aquarists more than most hobbyists, need lessons in how to read labels. Knowing what the label really says and what it doesn’t say is almost an art.
Regarding anecdotal evidence. – it is often used to sell products; it is often used by posters to encourage others to try things. The buyer and reader has to be aware at what point that ‘evidence’ is weak or solid. It is often just a testimonial by a person who doesn’t thoroughly understand the role the product played – what it did or didn’t do.
Too often the anecdotal evidence has been refuted by scientific evidence or failed to be supported by scientific evidence. The person making the anecdotal claim is oblivious to the facts already gathered. The use of garlic is one such product – it doesn’t do most of what the hobbyists think it does, let alone what we all would hope it did. Why is it available? Because people are wanting to buy it. It improves appetite: no scientific evidence has supported this claim. Tests that have been conducted show it does not increase appetite with one lone exception. What it does do is irritate the fish. This is an excellent read on the garlic matter: Garlic Article
There’s an old claim that garlic improves immunity: no scientific evidence has supported this claim. Never proven, but it is not hard to do. But no one has publicly provided data evidence, so there is only stage 1) or stage 2) evidence. The problem with stage 1) and 2) evidence is that immunity is not something that can be 'seen' in a fish. The blood of the fish has to be tested to verify immunity. Immunity can’t be observed or separated from a resistance. But it can be tested for and even quantified. So how could it even get a stage 1) or stage 2) level of evidence? Rumors.
A trap exists when a product is listed as “reef safe.†There are no standards, regulations, or guidance as to what reef safe means. Steven Pro did some experiments with pulsating Xenia and reef safe products and found some of the products harm, retard, or slowly kill Xenia. Are they reef safe?
A product manufacturer is not likely to test thousands of marine life forms to claim it is reef safe. This isn’t reasonable. However, the aquarist has the right to inquire why the manufacturer listed it as reef safe. What marine life was it tested against? Aquarists have to ask this to protect the marine life in their care.
The sorting out of the truth of anecdotal evidence is up to the aquarist. You can go online and get information, or ask questions of others for their opinion, but if you’re just going to gather anecdotal evidence, you need to determine the real value of that information and take into account what is known vs. what is thought to be going on. Keep an open mind even on things you think you’ve gather some anecdotal evidence about. Seek out data and the more formal evidence. Be sure contra-data doesn’t exist before taking the plunge into following testimonials and group advice. You and the hobby will greatly benefit.
