Wednesday, August 29, 2012

3 out of 4 doctors confirm: 5-hour energy contains atoms

"We asked over 3,000 doctors to review 5-hour energy.  And what they said is amazing.  Over 73% who reviewed 5-hour energy said they would recommend a low-calorie energy supplement to their healthy patients who use energy supplements.  73%.  5-hour energy has four calories, and is used over 9 million times a week.  Is 5-hour energy right for you?  Ask you doctor.  We already asked 3,000."

Remember that line about lies, damn lies, and statistics?  Well, it's people like the folks over at 5-hour energy marketing department that keep people believing in it.  Because what they say in their most recent commercial is actually quite brilliant, and probably fools many more people than any of us are willing to admit.

I have no gripe about 5-hour energy, mind you.  I don't use it, but I have tried it in the past (got a free sample somewhere long ago).  The taste wasn't good or bad enough to be memorable, and I guess I had some extra energy for a little while?  Hard to remember, it certainly wasn't anything magical or more than you'd get from a small cup of coffee or a quick jog around the block.

Back to the commercial.  What does this commercial tell us, and what does this commercial *try* to tell us?

The second part is easy.  The commercial is trying to get one thing to stick in your head as you're standing at the store in front of a 5-hour energy display.  That one thing is 73%, and what they're hoping gets stuck is that 73% of doctors (out of 3,000; so 2,190) recommend 5-hour energy.  With the commercial fresh in your memory, that's obviously not true.

The first big misdirection here is a old-fashioned bait and switch.  On TV, time is (literally) money.  There is a reason that 'low-calorie energy supplement' keeps getting tossed around despite the time it takes to say.  It's because the question to which doctors responded at 73% isn't *about* 5-hour energy.  The question is about 'a low-calorie energy supplement'.  The people setting up this survey know that they're more likely to get agreement to a vague idea rather than to a specific product.

It's simple:
A) 73% of doctors would recommend low-calorie energy supplements
B) 5-hour energy is a low-calorie energy supplement
Hence: 73% of doctors would recommend 5-hour energy

Unfortunately, there's a pretty decent leap there.  Let's treat this as a math problem with fairness going to 5-hour energy whenever possible.  What we can gain from this that 73% of doctors would recommend *some* low-calorie energy supplement, of which 5-hour energy is.  At absolute best, 73% are thinking about 5-hour energy, but some are likely thinking of something else, let's say at least 1%.  We can take from this, then, that less than 73% of doctors would recommend 5-hour energy.

That's not it, though, because we're actually also being bait and switched *with the first bait and switch*.  There's another bait and switch in there that you've also probably noticed by now: "healthy patients who use energy supplements".

The question is worded in such as way that they are not only asking about healthy patients, but also about those healthy patients who already use energy supplements.  Let's set up a contingency table, shall we?

Takes energy supplements  Does not take energy supplements
Healthy                O                         X
Non-healthy                X                         X

Before we continue, place yourself on this table.  Got it?  Good.  If you've landed on an X, this study and commercial is not about you, and you should not pay attention to any of the information in it.  You should treat this the same way as if you heard that tobacco smoking was linked to longevity in individuals over 105 years old who are Australian.  You shouldn't start smoking because of that, unless you're over 105 (and Australian).

What portion of the population is healthy?  I don't think I'd get too many arguments if I said that at most, 70% of the population is (because it is almost certainly less than that, and we're trying to be overly fair to 5-hour energy).  Of healthy people, how many of them already take energy supplements?  Again, let's aim overly high and say 70% of healthy people take energy supplements.  That means that in the above table 70% of 70% (49%, which we can just round to 50% for the sake of simplicity) of patients are the patients that are actually being talked about.

So, from the talking part of the commercial, less than 73% of doctors would recommend 5-hour energy for less than half of their patients (and probably not you). 

There are also some smaller points, like the fact that 5-hour energy being taken 9 million times a week has no bearing on any of the other information, and is actually working somewhat as a diversionary tactic, but let's move on to another interesting part: the small print.  There's...a lot of it.

Small print slide 1:
Two self-reporting surveys were conducted to determine the opinions of primary care physicians regarding energy supplements and 5-hour energy: 1) an online survey of 503 participants (who were paid $15) for their participation); and 2) an in-person survey by 5-hour energy representatives of 2,500 participants (50% of those approached).  In both, participants agreed to review materials regarding 5-hour energy consisting of label and basic description of its ingredients.  Of the 503 online and 2,500 in-person, 73% said they would recommend a low calorie energy supplement to their healthy patients who use energy supplements.

First off, thank you, DVR.  There's a lot of information here.  The first thing that jumps out at me is that they didn't ask 3,000 doctors like they said in the commercial, they *surveyed* 3,000 doctors.  That may seem like a small semantic thing, but it's pretty important.  They asked 5,500 doctors, and 3,000 decided not to blow them off immediately.  Now, that's not too awful, as response rates vary dramatically in these types of surveys, and I guess there's not a whole lot of ways they could put this out there other than tossing it in one of their small print blocks.  Still, something to watch for.  Along with a number of other blocks of small print.

Small print slide 2:
Of the 73% of primary care physicians who would recommend a low calorie energy supplement to their healthy patients who use energy supplements, 56% would specifically recommend 5-hour energy for their healthy patients who use energy supplements.

Ah, so there's the actual number.  Oh wait, they're still pulling the wool over on you.  It's not that 56% of doctors would specifically recommend 5-hour energy, it's that 56% *of* 73% would.  That number is just slightly over 40%; less than half of doctors would specifically recommend 5-hour energy to less than half of the population (healthy people who already take energy supplements).  Much less impressive than 73%, isn't it?  Makes sense they'd want to spruce things up a bit.

Small print slide 3: Of all primary care physicians surveyed, 47% would specifically recommend 5-hour energy for their healthy patients who use energy supplements.

Hmmmmmm.  That's not the number I came up with based on the information they gave in the *immediately prior slide*, and it's unclear how this number should differ.  Oh well, let's just say less than half of doctors.

As I went back to make sure I didn't miss anything, it turns out that I did.  I'm going to call it small print block 0, as it came before the block I called 1 before.  This one goes by *fast*

Small print slide 0:
All doctors surveyed identified themselves as primary care physicians. 

Primary care physicians weren't even verified to be primary care physicians.  They were simply doctors looking for $15.  The wording doesn't even imply that they're medical doctors.  Oh well.  Close enough.

Wednesday, August 22, 2012

2012 Olympics Winner: The American Women?

I've read a lot of talk recently about how this is the first Olympics where women have really stolen the show.  There are some pretty clear indicators (this is the first Olympics where every competing country had at least one female athlete), but some are a little more tricky to draw conclusions from.  For example, in terms of USA medal count, more women won medals than men (58 to 45)*.

I want to keep this first post fairly simple, so my first thought was simply "given the number of medals won by US athletes, is this difference in medals significant?"

The actual proportions for those medal counts are .437 for men and .563 for women (close to the actual counts due to the fact that the US has, for the past few Olympics, tended to win right around 100 medals).  If there was no difference between men and women athletes, we would expect that those proportions would be .50 and .50.  With just over 100 medals, are these proportions different enough from a 50/50 split?

Another way to think about this question would be to rephrase it.  If you tossed a coin 103 times, how unlikely would it be for it to come up heads 58 times and tails 45 times?  Well, it turns out that there's slightly more than a 20% chance of that outcome occurring by...well, chance.  That is to say that we *can't* say that American women won statistically more medals than their male counterparts at the London Olympics, just that they won more. 

That's hardly the end of the story, though.  One nice thing about the Olympics is the longitudinal aspect of the data.  Medal data is readily available for over 100 years.  Looking back too far would certainly produce a large gender gap, and looking at certain years (e.g. 1980, 1984) would likely bias US data.  With that in mind I looked at the past 6 Olympic competitions, from 1992-2012. 

I mentioned that I wanted to keep this post simple, so with that in mind let's look at the following chart in a purely descriptive way:

From the graph, then, there does appear to be a period of relative stability around a 60/40 split from 1992 up to 2004, with a shift starting in 2008 and a reversal in 2012.  We can examine if the proportions are different across different years - keeping it as simple as possible a Fisher's exact test fails to show an association of year and proportion between 2008 and 2012, but does reveal a difference in proportions between 2004 and 2012.  Thus, something does seem to be changing in terms of proportions between those years, and it does seem that American women have at least negated male dominance at the Olympic podium, and at best surpassed it.  Should be interesting to see what happens to these proportions in 2016. 

There are a few other factors I'd like to examine in this data in future posts, but if you think of any other things that could be examined as a follow-up to this data do feel free to let me know.  I figure it's something I'll come back to every once in a while, as Olympic data is just pretty fun.

*Note: the careful reader will notice that this is one medal shy of the 104 medals won by the US this year.  For the purposes of this analysis the missing medal is a bronze won in mixed doubles tennis.  This medal which must be excluded or double counted in terms of a gender breakdown.  I chose to exclude it rather than double count it.  Similar events occurred in each of the Olympics examined.

Wednesday, August 15, 2012

The Skeptical Statistician

I am a statistician.  Let's face it, most people are not.  What's worse than admitting ignorance is thinking you can use them to make a point without understanding.  There's a famous quote of confusing origin that 'There are three kinds of lies: lies, damned lies, and statistics.',  often used to dismiss the idea that numbers can tell you anything important.  Sadly, these types of people are also the most anxious to use numbers and statistics when they support their own cause.

I have no intentions of this blog taking on any sort of issues.  I do not mean this blog to take one side or another, but rather to simply see what numbers might actually be saying.  I see this taking two forms - the first situations where I find people misusing (or misrepresenting) numbers to, well, tell a damned lie, and the second to answer questions I think of that I find interesting that can be backed with numbers.

By all means, if you come up with a question that you believe can be answered with easily found data, let me know.  If you have an issue with the way I've looked at something, let me know as well.  We all make mistakes, so after I make a few perhaps we can run the numbers on them.   

I'll be trying to come up with things to update on every Wednesday.