Clinical Significance

The Sandbox General
1

So we’re all statisticians to some extent, correlation doesn’t equal causation blah blah blah.
A lot of YOU also dismiss acupuncture or chiropractic or anything that is not proven clinically significant.

But isn’t that kind of a weird line to draw?

If acupuncture didn’t show results for 99 people, but it worked for 1 person, it’ll fail in a clinical study. And you, my dear actuaries, might claim that acupuncture is pseudoscience, but it literally says the opposite? It WORKED for 1 person! Like, it actually did something. It didn’t work for everyone, but for the 1 person, it actually worked!

For you to claim that it’s pseudoscience, you need it to work for no one.

My 2 cents.
bye.

2

Let’s all guess the drug JSM is on right now…

2 Likes
3

The question is usually going to be whether acupuncture did better on average than placebo.

There is no way to know whether it worked better than placebo for a single person, since they cannot be both treated and not treated.

Then statistical significance is about whether acupuncture does better than placebo because it just got lucky for that collection of people.

I think all this is subtly different from correlation vs causation.

4

this is my point.

people be claiming stuff something is pseudoscience left and right because of statistical significance seems a little dismissive.

and you can’t reverse engineer it either. telling someone because chiropractor has not been proven to work, what they experienced must be due to placebo or just luck seems a little insulting too

oh I was just using it as a catchy phrase for statisticians, not that it has anything to do with this thread

5

actually sober.
about to open a bottle of wine though

6

I’d claim acupuncture is pseudoscience not just because there is no evidence that it works better than placebo, but also because it has no mechanism for working (that i know of) that is consistent with our understanding of the body.

It’s impossible to measure zero. So we can never measure that it performs exactly as well as placebo.

7

that’s a fair statement.

8

We know the placebo does work, why ruin that, just let people think it might be working and they’ll be happier…

9

Where are these placebos and where can I get my hands on one

10

Is this when the snake oil infomercials start?

11
2 Likes
12

Let’s define “worked”

1 Like
13

like it actually did something, maybe even beyond placebo, significantly for that matter. The significance of course cannot be detected because you cannot do a test on the same person at the same time multiple times.

14

My MIL won’t eat anything with sugar in it after 4:00 PM because she claims that it will keep her up all night. As evidence, she said that she had a chocolate chip cookie the other week at 5:00 and she couldn’t fall asleep until 1:00 AM. I suppose it is possible that she is the only person in the world who is affected for 8 hours by a small amount of sugar and chocolate. Or, more likely, her insomnia had nothing to do with the cookie but some other thing. Coincidences happen and that is why we look to statistical significance to identify correlation and causation.

15

The thing is people are really different. And a lot of “proven science” is just dancing around the thin line between significance and insignificance.

The only person that can experience your body is you. There’s a Chinese saying: chronic illness makes good doctor (for yourself).

16

Just because something is labeled “pseudoscience” doesn’t mean it does not work. It’s just that the explanations are sketchy.

1 Like
17

I unfortunately have some experience with the reality that certain medical conditions (some kinds of brain damage, some forms of chronic pain conditions) are difficult to fully diagnose and determine effective treatments for.

There are plenty of potential treatments that work for a few patients with such conditions, but not others. Finding the right one / the right combination is a bit of a hit-or-miss experience.

Someone looking at the headlines/abstracts for such treatments would think that at least some of them are bunk, because they may work for only a few, and then in some cases only in conjunction with other treatments…and therefore the successes may not rise to the level of statistical significance when viewed from a high level.

The implications of this (in the US, at least) when seeking insurance coverage for such treatments is left as an exercise for the reader.

1 Like
18

or don’t work broadly. but that doesn’t mean that individual gets no benefit.

19

I don’t think this is what people mean when they say pseudoscience.

There are two ways significance can fail.

  1. There are no benefits beyond noise across the board.
  2. There is a real benefit for an extreme minority, but significance fails on the average.

I think pseudoscience is usually meant to apply to 1 only.

1 Like
20

I’m sure that’s what some people mean, but it’s not what i mean.

Pseudoscience masquerades as modern science. Unlike science, conjectures of pseudoscience do not conform with facts as much as possible. In practice this usually means not working within the mainstream scientific community.

As I understand things, sometimes pseudoscience is done by a lone individual who fantasizes that he is a persecuted genius. This is a “quack”.

Other times it’s supported by a profit motive.

My understanding is that traditional chinese medicine was effectively invented by mau because china did not have enough western style physicians. It’s claim to effectiveness is grounded in an appeal to “tradition” rather than evidence. This even though there is really no single tradition of chinese medicine. Instead different parts of china have historically had very different traditions.

Slate had a very interesting article on that way back:

1 Like