cc >> The title of the talk is "The
End of Evidence-Based Medicine." And it's not a rhetorical figure
I actually mean it. It's difficult for me to give
this talk every time, because I'm an advocate of
evidence-based medicine. I've been teaching
evidence-based medicine. And to have an end to the
evidence-based medicine conversation is problematic
for me. But it turns out that it also is
something that's interesting to people.
If I had said evidence-based medicine reaches its middle
years, or has reached maturity, or anything like that, we'd have
half the crowd today. People like to go to funerals.
So that explains why you're all here.
There's no way in any of this research to have any sort of
financial disclosure, it would be nice but there is nothing
to disclose. *The fundamental thing here is
that there are two principles of evidence-based medicine.
The first one is that the better the research, the more confident
the decision. So that's one fundamental
principle of evidence-based medicine.
It turns out that if you ask people around what they think
of evidence-based medicine, they may say this, and then
they will stop. They will think that this is the
only principle of evidence-based medicine, but there are two,
and this is one of them. And I presume nobody has
a problem with this one. *But there's another one I said.
And the second principle of evidence-based medicine is that
the evidence alone is never sufficient to completely inform
a clinical decision. But one has to incorporate other
aspects of the complexity of the human experience, like
the context of the patient's experience, and the values
and preferences of the informed patient in coming up
with a decision that actually makes sense.
In other words, if you were to train monkeys to analyze
and critically appraise the literature and apply every
randomized trial in a strict way, you will actually cause
major harm. And the monkeys will be
evidence-based. *So there's something to
expertise that needs to be incorporated as well, this is
a graph here, in order to make good high quality clinical
decisions, there's got to be some expertise required
to incorporate what we know from the research, what's going on
with the patient and what the patient's goals for life and
health are. This expertise will vary across
clinicians. And hopefully in our teaching
programs, as we're training clinicians to become
more effective, we're paying attention to those issues.
*Now, I'm going to tell you a little bit of history
about evidence-based medicine since we are in a funeral,
it is appropriate to talk about the life of the dead individual.
And we tend to do in funerals, is we tend to say the positives
and the beautiful things. So I'm going to do that.
Anybody know who this guy is? Feinstein, very good.
Feinstein is a very controversial figure in the
history of research, but the major contribution,
and we can argue about whether his major contribution was
for the tobacco industry, but I think his major
contribution was that he developed the idea of clinical
epidemiology, and the idea that you can use quantitative
methodology and apply it to the work of physicians.
That you can quantify inferences, that you can
quantify diagnostic criteria. And the idea of Clinimetrics.
He was very good with words and he always created words
to describe what he was saying. *Now, I want to stop at this
point, because it's really funny for me, from here, to listen to
myself bounce on the back end, so I'm like having a long
distance conversation with my parents in Peru.
It's exactly the same sound. Are you bothered by my
microphone in any way, or is it working okay for you?
Okay, so I just have to cope with this.
*So this is Feinstein, and arguably, he's one of the people
that starts laying out the foundation for what became
evidence-based medicine. Who's this guy?
Guess? David Sackett.
And David Sackett is a guy that went from being a basic
scientist, nephrologist, a public health officer,
to then being one of the youngest chairs of a clinical
epidemiology department at McMaster University in Canada,
and a big advocate of the use of evidence-based medicine in
clinical practice, particularly starting from randomized trials,
which ended up being a source of discrepancy between these
two. Feinstein quickly went to think
that randomized trials could actually be quite useful
for clinical practice. Anybody?
Tom Chalmers. >> Yes, Tom Chalmers.
Tom Chalmers, from Harvard, a big advocate, again, of using
randomized trials. He's the one that I think
coined the expression that if you have a new technology,
you want to randomize the first patient exposed to it,
and then move on until you learn whether the intervention
works or not. And also a big proponent
of systemic reviews and meta analysis.
( hums "Jeopardy" theme ) So, Ian Chalmers, no relation
to Tom Chalmers, is an obstetrician from the UK who
heard the call from Archie Cochrane, traditional
epidemiologist from the UK, to summarize what we know
about research and created the UK Cochrane center,
which then evolved into the Cochrane Collaboration,
which is a large, multinational volunteer organization
that summarizes what we know in terms of randomized trials,
and more recently about observations, tests, and
diagnostic studies. *So you can see the evolution
of knowledge being summarized and synthesized.
Yes, Brian Haynes. You guys get out, that's good.
You know people. Brian Haynes also from
McMaster University. It's interesting, because what
he did is he harnessed the value of the Internet and
the evolution of Medline and other databases to say, okay
can we now bring this information that's being
produced at dizzying speeds, and put it into the fingertips
of clinicians. So the ACP Journal Club,
the journal of evidence-based medicine, services that are now
available, that are now services by email, that keep clinicians
up-to-date, are things that he has developed, piloted,
or has had his hands on at some point in their development.
*I'm going to show you one last picture.
Yes, Gordon Guyatt. And so, Gordon Guyatt
is a guy that in 1990 took over the residency program
at McMaster University, and sent out a letter to the residents
saying, look, enough of this expert-based teaching here
that I tell you what to do sort of thing, now we're going
to do scientific medicine. It got everybody really upset.
And so in everybody being very upset, he decided he needed a
better label for this, and then went for evidence-based medicine
and published it for the first time in 1991 in
the ACP Journal Club editorial. *There's debate about whether
this account is true, but I'm sticking to it.
There's a very distinguished epidemiologist, an American
epidemiologist, who claims that he came up with the term.
And the way I think it has been resolved, although not to this
latter individual's satisfaction is through a UK--
an NHS official, who remembers being in a taxi cab
in Washington, D.C., when Gordon Guyatt
told David Sackett, "You know, I just came up with this
great term for this thing that we've been struggling.
I'm going to call this evidence-based medicine."
*It is interesting that perhaps the fundamental thing, the only
proof that Gordon came up with it and told Dave
about it is from somebody recalling it from a taxi cab
in Washington. That's how history is written.
And this is 17 years ago. So this funeral is for a
teenager. So it's a 17-year-old thing,
so it's been evolving since its inception.
*But where are we now? I would put forward that this is
the end of evidence-based medicine, because the
evidence base, the research evidence that we can use
to guide our clinical decisions is corrupt.
And I'm going to show some examples of such corruption.
And corruption is of a magnitude such that it cannot support
evidence-based practice. But that's not enough.
I think another reason why it's the end
of evidence-based medicine is because systemically,
patient preferences are being ignored.
We were having a discussion coming here as to why that
might be the case, but the fundamental thing is,
they're being ignored. If you put those two together,
it's the end of this thing. *Now I could stop at this point
and take questions, but I think that would be a little
disappointing, so let's go through to understand the issue
of the corruption of the evidence.
*Most of the effort in evidence-based practice has been
looking at, can we understand where bias is introduced
in clinical research. So of those of you who have
gone through courses to understand clinical research,
the attention has been, how do you ascertain bias?
How do you look for bias and the different signs.
So for instance in randomized trials, how is a randomized
sequence generated, where there was allocation concealment,
who was blinded, and to what extent, is there any major loss
of follow-up. And we recently contributed
to this evidence-base by pointing out clinical trials
stopped early for benefit are likely to produce
an overestimation of the treatment effect.
*So individuals that know critical appraisal can sit down,
look at the paper and look for these things and say yes,
the extent of bias, or the potential of bias in this study
is significant, or is moderate, or is small, or is absent.
This idea of clinical trial stopped early for benefit
introducing bias has become completely illustrated
more recently with the example of perioperative beta blockers.
*Here's a trial by published in the New England Journal of
Medicine in 1999, in which there was an enormous difference
between those patients that were going to undergo
cardiac surgery, had a stress echocardiogram
and were found to be at risk of ischemia, were given
a beta blocker. The alternative arm just
received standard of care. There was a major difference
in outcomes with 100% reduction in the risk of nonfatal MI,
and 80% reduction in the risk of cardiac death.
In other words, perioperative beta blockers cured MI.
( light laughter ) Interestingly enough,
nobody questioned the fact that these estimates appeared
to be a little big. And decided that this will be
a great target for quality improvement.
And many institutions, including the Mayo Clinic, there have been
major efforts put forward to make sure that
every patient at risk that is undergoing cardiac surgery
receive beta blockers very promptly.
*Now we conducted a systematic review of meta analysis of the
evidence, and then we produced a graph that is a meta analysis.
The red line represents the community of evidence in support
of the use of beta blockers, and you can see it's fairly flat
until the trial, and then it goes way up.
And then it stays essentially fairly flat, or coming down, as
the clinical trials accumulate. Since we conducted that review,
two additional trials were published.
*One called POBBLE and the other one called DIPOM,
which were very large trials in and of themselves.
And the whole estimate for the reduction in cardiovascular
influence approached null. *But the confidences were
wide enough that still further research was needed.
Pretty far from being so sure that you have to implement this,
that you would do quality improvement on this measure.
The one trial has since been completed.
We had a chance to participant in that trial.
8,000 patients randomized to perioperative beta blockers
were the control. And the trial in fact,
confirmed a reduction in the risk of coronary events,
in the order of 30%. Not zero, but not 80 or 100%.
*But also, where it is powered enough to be able to show
an increase in the risk of death and severe stroke.
So you get peri-operative beta blockers.
Yes, you prevent the chance of peri-operative heart attacks.
But you will be more likely to die or have a stroke.
( light laughter ) So make sure that every patient
at risk of coronary event undergoing surgery
gets their beta blockers, because that would be
a good quality improvement. *So here's an example where we
were misled by a clinical trial, stopped early for apparent
benefit, and not only overestimated the treatment
effect, but also hid the fact that this intervention
could potentially produce harm. But that's bias.
*And bias is easy because it's something you can read
on the paper. It's there for you to see it.
With the right training, you can get to it.
It's not too difficult. If that was our only problem,
we would be okay. *So what is our next problem?
This is our next problem. Anybody have a problem
with this? So why would science
not be objective? People do it, right?
It's because science is the hands of scientists.
Scientists are not objective. Now why would scientists
not be objective? I mean, maybe you are
not objective but my science is pretty objective.
Right? I love my ideas.
( laughter ) Now I need to love my ideas.
In fact, I have to make you love my ideas,
because you may, in fact, be in charge of my future, right?
So I need to have successful ideas, because if I have
successful ideas, I will have academic success.
I have to not only believe that, I have to make sure
that you believe that. I also have to make sure that
you understand that my findings are critically important.
My research is the only research that matters.
It's the only research that should be highlighted
in the New York Times. My work is of extreme
high quality. I'm rigorous as rigorous comes,
and you should believe all my results.
Because I need my peer review funding, and you're my peers.
And so I need to make sure that money put on my career
and my research is money well spent.
*So this is enough to push me in certain direction regarding
the way I present myself, the way I present my results
to you, the way I present my results in papers
and publications, because I need you to believe this.
This is what we sometimes call putting your best foot forward,
right? In fact, we may have mentored
people to do exactly that. *Now, issues and problems.
This is a survey of almost 200 diabetes trials published
in top journals in 2003, or 2003 looking back.
And we found that when you look at things that are predictors
of bias, they're infrequently reported, 11% report
allocation concealment, half of them or less reported
blinding of different groups, 71% reported
on loss of follow-up. And then we looked at
which of the trials, when you read them, have
the apparent highest methodological quality.
In other words, all these things are present.
And what we found as major predictors were
for-profit patient-oriented parallel trials,
so those trials funded by for-profit interest
have the better quality when you assess them
on critical appraisal tools when they're published.
*So the ones that are putting their best food forward
are the ones that are funded by for-profit interests.
Okay? Here's a systematic review.
I participated in this with some colleagues.
We looked at both drugs and devices, and we looked to see
association between funding source and the mapping
to the treatment effect. And we found that on average
for-profit funding sources had more positive trials
favoring their product. *Now, there's a number
of explanations for this. Maybe for-profit funding sources
are more careful with their money, and they're able to
actually fund trials that they think are going to be positive.
They've done their homework. There's one explanation.
But as you know from the bottom is that this is adjusted
for the quality of studies and adjusted to sample size.
So they're really, really good at selecting the winners
in some way. Well, that's one explanation.
There may be alternative ones. *Let me show you another
point of evidence. This is about the enthusiasm
for the product, or device, or drug that's being evaluated
in randomized trial. Als-Neilson did this in JAMA.
They looked for a sentence like this in the discussion
and conclusion of section of the studies,
"The experimental intervention should be the treatment
of choice." So one of these very exuberant,
positive things that you will say in a study, and then look at
the predictors of ending up with a conclusion like that.
And this was from 370 trials from 25 Cochrane reviews
over a 30-year period of time, 18% in high impact journals.
And what he found was that for-profit funding
was associated, I think fairly strongly,
you will agree I hope, with having that sort of sentence.
But I just show you, they got bigger treatment effects.
Why would they not conclude this sort of thing?
Well, except that this illustration is after adjustment
for treatment effect, and the magnitude of the adverse effect.
So this is independent of the pros and cons, or the
benefits comparative benefits of the control group.
*And then, this is now very much outdated, but this is the best
I can find. There's a major flow,
as you well know. There's a major flow of academic
research from academia to industry.
So remember my ideas and how much I love them?
That was because of my own personal career
and my ability to survive. What if it is 10,000 employees
of a large company whose future is riding on the results of this
particular research project? You may all recall that when
the HDL, the drug that was supposed to elevate HDL
was not found to be associated with major benefit,
a whole Pfizer plant closed and people lost their jobs.
So it's just not about my career and my ideas
and how much I love them. This is a real problem
when research results modify the livelihood
of individuals. *So, then, am I saying that
for-profit research is fraudulent?
No. Am I saying that I'm introducing
a mechanism to cause bias? No, because I told you,
that's easy to pick up. So what other mechanisms
are present there to actually make things look better?
And I'm going to call that spin. *Now spin is a difficult concept
for some groups, so I'm going to give you an example of spin.
I'm from Peru! I'm from South America.
And in Peru, soccer is big. Soccer is huge.
Now it's actually a problem for Peru that soccer is huge,
because we haven't had a team in the World Cup since 1982.
( laughter ) So when your biggest sport
doesn't deliver, a country can go into massive depression.
That's how I would characterize our sports situation in Peru.
But this is spin. So we play against, say, Brazil.
And say, Brazil puts in five goals against us.
And the coach goes on TV and says, "Well, I'm very proud
of my team. My team delivered.
It was outstanding work." You know, "I'm particularly
proud of my goal keeper. He only let five goals in."
That's spin, all right? So when you know the truth
is over here, and you try to create this alternative
explanation or alternative view of the truth.
*So let's see how that plays out in research.
Here's a trial. It's a high quality RCT.
If do all the assessment for bias, it's a high quality
randomized trial, 1700 patients with hypertensive
diabetic nephropathy. Using-- pain, versus-- blocker.
Almost three years of follow-up. This is lifted directly
from the conclusion. Treatment with irbesartan,
which is a receptor blocker, it was associated with a risk
of the primary composite endpoint that was 23% lower
than that of the group. I've given you, the P value
has a bunch of zeros, and I'm giving you what that
composite endpoint was. Doubling of the creatinine
level, onset of end stage renal disease,
or death from any cause. *Now, how many people
are clinicians here, that see patients?
All right, so they tend to be distributed in the back
of the room. Okay.
I wonder if any of you would like to give me an example
of how they would explain this to Mrs. Jones?
( laughter ) ( no responses )
And I may actually go and have a cookie if you don't
say anything quickly. ( laughter )
Anybody? >> Only one-quarter of her
is likely to suffer... ( laughter )
I suspect, if you actually see patients, that's the thing,
you don't talk to them like that.
Seriously, what would you say? How would you explain this?
Say you talk about research with your patients.
How would you say? Give her a PhD in statistics,
you know, and she happens to have diabetes and hypertension,
and you're going to explain this to her.
What would you say? At this point, she's feeling
very uncomfortable, because you're being silent for awhile.
( laughter ) Try it out.
( inaudible ) >> Four people are going to have
the problem, if I give you this medicine only three will.
Thank you doctor. Now I understand, it's better.
So what is it I'm going to avoid having?
Getting bad kidney disease, or dying.
Wow, so this is a drug that will help me live longer.
That's wonderful. And so, that's exactly
the intent, right? Is that you believe there's
a 23% reduction of the risk of dying, or having bad kidneys,
or having a distortion in your creatinine level
in the laboratory. *But if you look at the actual
results, the doubling of the creatinine and
the end stage renal disease goes with that 23% reduction.
But what about the mortality? It's kind of not going with it.
Right? So all of a sudden--
You told me I was going to live forever, right?
And this drug may or may not actually have an impact
on mortality. The point estimate is on
the wrong side of the, right? It's actually going
in the wrong direction. So composite endpoints can
actually hide this important difference in treatment effect,
give you an average that doesn't in fact apply
to each of the components. *And this is particularly
critical if the components do not have the same degree
of patient importance. So I'm pretty sure I'm going
to feel when I die. I don't know if I feel
the elevation of creatinine, maybe I will, but certainly it's
not going to feel like dying, at least not to my family.
So that's kind of weird, isn't it?
But this is one mechanism by which you get a sense
that this is going to cause a bigger impact
than you think you will. *Now this is actually a fun one.
Of the clinicians in the room, how many of you,
if somebody comes in with an exacerbation of emphysema
or bronchitis to the emergency room,
would you give them steroids? Pretty much everybody.
Anybody who would not give steroids to somebody
with COPD exacerbation? Okay, so pretty much everybody.
The fundamental study-- Yes?
( inaudible ) >> Current recommendations are
to use steroids only if you're below a certain level.
They recommend that if you're above a certain level,
that steroids have no benefit. >> Great, fantastic.
So, ughh, I'm in bad shape! You're going to give me
some steroids. >> You give them
bronchial dilaters. >> All right, so we have
a nuanced answer. Other people said do it.
And it's interesting, because the fundamental trial that
started that tendency was published in the New England
Journal of Medicine in 1999. These are the results,
so they're all on the side of benefit.
And the composite endpoint was death, then the need
for mechanical ventilation, a readmission to the hospital
or requiring steroids. And you can see that,
as I'm going to show you, the number of endpoints,
keep in mind that the steroid arm was double the size,
the number of endpoints is actually very small
to drive policy. You had two deaths in one arm
and three deaths in the other. That's not a whole lot
of evidence. Three people require ventilation
in one arm. Three people in the other.
Again, one of the arms was bigger.
Readmission, six and five. So the majority of the evidence
in this trial is about requiring steroids.
In other words, give steroids to the patient,
because if you don't, they are going to need it.
( laughter ) Right?
That's what this trial showed. So we all started giving
steroids, just in case they will need it later.
So these composite endpoints kind of hide this stuff.
*This is my favorite, because I'm a diabetes doctor.
So everybody knows that the UKPDS is the largest of each
trial, about 4,000 patients followed for ten years.
And it showed that intensive glycemic control reduces
the risk of any diabetes-related complication.
This is diabetes-related complications, and 35%
in the intensive therapy arm had these events;
38% of the conservative arm had these events.
The difference was 3.2%, P value .029.
So what does that actually mean? Well, that composite endpoint
had 14 endpoints in them, that included death,
and cataract extraction. ( laughter )
So it's a little bit different, right, the experiments there?
Amputation, stroke. That's kind of relevant,
Cataract extraction? And it's interesting because
as the trial progressed, they were finding nothing,
no difference, no difference, no difference.
So they kept adding components to the composite endpoint.
Right? And so, you notice there are 13,
I said there are 14. There are only 13, but there's
a blank spot, because the last component they added was
photocoagulation of the eye, which is a procedure of the eye
where you laser the back of the eye
to prevent blindness. And of that difference of 3.2%
between the two arms after ten years of therapy,
photocoagulation reflected 2.7%.
So the last component added to this composite endpoint
captured the majority of the impact of this treatment.
*Now does anybody here treat diabetics so this extent
to prevent photocoagulation? No, we all think that
we're preventing all these diabetes complications.
*So we've tried to contribute to this spin.
In order to take the spin out of these things,
these are the questions that we ask you to ask.
Is there a big gradient of patient importance,
death in cataracts? Is there a big range of
treatment effects, you know, in terms of the magnitude
of treatment effect like we saw with the nephropathy example?
Is there a gradient of events, like we saw with
the steroid example? *And when you see big gradients
your confidence in the composite should be very weak,
and you should be focusing your attention on the effective
on the components. The problem with that, is
that all the components will be so underpowered and imprecise
that you may not be able to draw any conclusions
from this study, which is a big problem.
Which begs the question why did we go to composite endpoints
in the first place? And the reason is we needed
a P value less than .05. There is something
fundamentally wrong around how we're doing this.
*Now, the cardiologists are in deep trouble, because
the cardiologists have been very good at adopting
evidence-based medicine, to almost to the point
of evidence tyranny, where if the trial has not occurred
and the trial has not showed it, I'm not doing it.
If the trial shows it, I'm doing it.
Right? There's a bit of that going on.
In their trials, we looked at 242 cardiologist trials,
112 used composite endpoints, and 86%
failed the first two criteria. So in other words, they're
incredibly hard to interpret. And I know cardiologists
are very smart, but I suspect that they also have difficulty
interpreting. And they either have chosen
to ignore the difficulty and just go with the composite,
or should be in major psychotherapy.
( laughter ) I don't know if that's happened.
*This is a large prospective blinded international clinical
trial called the DREAM trial, of diabetes prevention.
Eight milligrams of Rosiglitazone daily, together
with lifestyle recommendations substantially reduces the risk
of diabetes or death by 60% in individuals at high risk
for diabetes. This is lifted
from the conclusion. This medication reduces the risk
of diabetes. The doctors are saying your
sugars went from 124 to 128, and I repeated it
and it was 132. You have diabetes.
Okay? That experience,
versus you're dead. ( laughter )
Big difference, right? There's a 60% reduction
in that difference. And so, when you look at
the actual data, it's really all effect on diabetes.
In other words, the rosiglitazone lowered
the glucose enough to actually make enough people go through
the trial, so there will be fewer diagnoses.
There was no effect on death. And as we know from the recent
discussion, maybe the opposite is exactly true.
*So this composite endpoint is a problem.
But it's certainly not the only way to spin results.
Inadequate patients and comparators is another way.
Haliparadol, at 40mg versus a new antipsych--
anticraziness, you know, antipsychotic drug.
Right? And so, the antipsychotic drug
comes reported as having fewer extra-parametal side effects,
yeah, right? Or, I'm going to use a new
antidepressant with amitriptyline twice a day.
And difference? Daytime somnolence.
We kind of knew that, right? And so when you have
the comparative, you may actually have the wrong answer.
Composite endpoints, we have large gradient.
We've talked about surrogate endpoints of poor validity,
hemoglobin, A1C is a good example of that.
Post hoc changes in outcome definitions, and there's major
literature on that now. Inadequate and abundant subgroup
analysis. It works in this group
and it doesn't work in that group.
We have no idea how many subgroup analyses were done
before that conclusion was reached.
And then, the spinning of the discussion and conclusion
section that we discussed. *And the key risk factor
is small treatment effects. If the treatment effects are
small, you have to compensate. This is the joke, "Daddy,
mom says you have this big heart to compensate for something.
What is it that you're compensating for?"
Right, you know that joke. Same thing here, if you have
small treatment effects, you may need to start spinning
your conclusions, you need to start looking at
composted endpoints to get a bigger P value.
You start looking at surrogate endpoints, because you're
looking at small treatment effects.
*Now, where in Madison are we seeing small treatment effects?
Pretty much everywhere. Right?
We're looking for incremental benefits.
Very small treatment effects. We're not looking for a cure
for rabies, right? So very small treatment effects.
So this is a major problem. *So we talk about bias and spin.
If that was a problem, well we can sort of deal with it because
bias and spin are both present, they're both in your face,
you can look for it, you can potentially see,
you can try to infer it. *So we have another enemy
and the other enemy is bias reporting.
So this is a third mechanism that we can see corruption
of the evidence. This is a trial, a study
by Chan and Altman looking at Danish protocols, followed them
for ten years, so 102 protocols became 122 reports
ten years later. They compared what they said
they were going to measure with what they actually reported
measuring in the trial. This was randomized trial.
So 50% of the outcomes were incompletely reported.
That is they were not reported at all, or they were reported
just by saying "not significant."
The odds of fully reporting an outcome, were 2:1
if the outcome was significant. So what it means in general,
when you're reading the literature, you're keeping up
with the literature, you're basically getting an
over-optimistic view of treatments, because the majority
of what's getting published are positive results
at the full study level. But within the study
the outcomes are interesting. Yes?
*So we actually looked at the literature ourselves to see
if we take all these systematic reviews and we look at
the proportion of trials that contribute to important
outcomes, plus a lot of outcome reporting bias,
we'll have bigger treatment effects in summary
than even if most trials published all the outcomes.
This is what that graph looks like.
*If they had 100 trials in the systematic review, but
only 20 reported on mortality, you can see that there's
an overestimation of the treatment effect on average
than if the majority reported on mortality.
And the only way this could be explained is if
the decision to report is related to the magnitude
of the treatment effect. So in general, the view of the
literature that we have, because of outcome reporting bias
is an overestimation of the treatment effect.
And this again, is not fraud. It's just that
that's what's interesting. That's what people want to read.
*And then there's the problem of the whole study.
So, perioperative beta blockers positive trial, most people
knew about this study. Most people didn't know
about the DIPOM that was published in the BMJ.
It had 900, nine times more patients, and people didn't know
about it. The POBBLE trial, the same
number of patients, but had a relative risk increase.
People didn't know about it. The MAVS trial, five times the
number of patients relative risk reduction of 15%.
People didn't know about it. *In my area of interest, that is
for diabetes, the CARDS trial very positive trial,
30 percent benefit, relative reduction 37%.
A trial with essentially the same design, the ASPEN trial,
essentially started and stopped about the same time.
Two additional years to get published, and was published
in the subspecialty journal the Lancet,
relative risk reduction of 10%. *So, we have the problem
of bias, spin, bias reporting and then fraud.
I'm not going to talk about fraud, because fraud
is incredibly difficult. Somebody discovers it,
there's a whistle blower, you hear about it.
But beyond that, there's nothing we can do about it.
*But if you then say this is corrupting the evidence base.
When people say this is an evidence-based "blank,"
you should be running in the exact opposite direction, right?
Because it is quite possible that what is being
introduced here is the product of inferences drawn
from research that is corrupt. *If you have guidelines,
if you have now guidelines that are evidence-based, that
are telling you what to do, that's becoming a little more
problematic, isn't it? And if now we're talking about,
your quality is going to be measured on the basis of these
parameters, this is becoming a little more uncomfortable,
isn't it? And if I tell you now that your
pocketbook is going to be hurt by this.
If you're going to have pay for performance, that's
going to be based on evidence-based guidelines,
again, you should be running in the opposite direction.
You should all be practicing in Brazil.
They have a better soccer team. ( laughter )
Notice that I chose to say soccer, even though
they call it football. I do not want to hurt anybody
by saying they have a better football team.
( laughter ) And it's warmer.
*So this is a major problem. This is a major problem.
And so, this is the point about the evidence being corrupted.
What about the second point? What about this idea that
patient preferences are being ignored?
*I'm actually going to skip through this fairly quickly
to highlight one thing. This is, I think,
a little bit of a video. This is a patient and a
clinician at the Mayo Clinic. The clinician is telling
the patient, "I think you should be on statins.
You have diabetes. Your cholesterol is high.
I think you should be on statins."
And what is the first thing on that patient's mind?
That guy. ( laughter )
Yeah, I've seen that guy. He's like the artificial heart
guy, and he's in the canoe. He seems to be taking statins.
He's good. I want to look like that.
I want to be like that guy. So what could be
on the physician's mind? Maybe the evidence,
but from an advertisement. So we have the people making
decisions heavily influenced by direct-to-consumer,
direct-to-physician advertising and detail.
*But that's not it. Another push for further
prescribing are these quality measures.
You know, this LDL has to be less than 70, less than 50,
undetectable cholesterol! ( laughter )
We need to use highest dose of statins that we can
in these people. We just had Scott Grundy
present Grand Rounds at Mayo Clinic on Wednesday.
And literally, everybody over the age of 50 should have
their LDL at less than 100. So how do patients respond
to this challenge? By not being adherent.
So after two years of therapy, half of the patients
are no longer taking the stuff. And if you look at the place
where the curve is separating, the large clinical trials,
two years. So that sounds like waste
of resources, doesn't it? *When people have been asked
what do you do with prescription?
15,000 will say, I haven't taken them.
Why? "I don't think I need them," or
"I don't like the side effects." These decisions are happening
outside the visit. They're happening at home.
The clinicians may or may not know about them.
And this is consistent poll after poll, patients actually
second guessing why they're getting all these prescriptions,
and deciding to take them only occasionally,
or not taking them at all, for reasons that may be
completely unfounded. So we have a problem
with communication. And so if this is the case,
this is really the end of evidence-based medicine.
*Now, if I were to stop here, this would feel like
one of those European movies. ( laughter )
The boy meets girl. They seem to be
made for each other. They love each other dearly.
And it's just getting to the end of it, and suddenly they
turn around, and they walk away from each other.
And then the credits roll, and you go, "What the hell?"
( laughter ) Right?
I mean, this is just how you feel.
I won't leave you like that. I think we should have a little
bit of a sunrise in this. Is there hope?
So, this is a Hollywood ending. ( laughter )
Is there hope? And I think the hope
lies right here. I think we need to be advocating
for a completely different funding structure for research.
The funding structure we have is creating this problem.
Is contributing to this state. We need to advocate
for completely different funding structure.
We need to find a way of funding research, for particularly
important research in ways that are not dependent
that don't make the funder care about the results.
*We need to be looking for large evidence.
If we're going to be looking at small treatment effects,
we need large evidence, either by having meta analysis
of small trials that follow the protocol,
or having large trials. But this idea that my trial
of 15 patients will give you a P value less than .05
is ridiculous. That every trial
has to be definitive is ridiculous.
That the IRB says that your trial is unethical, because
you don't have enough patients is ridiculous.
And what happens is, you start creating this enormous
effect sizes to justify your trial of 15 patients,
because you only have money for 15 patients.
And so you're creating some alternative realities to be
able to get your trial through. That is ridiculous.
We need to just basically realize that.
If we have money for 15 patients,
we do it on 15 patients. We report the results.
And we hope that somebody else will get money for another 15,
and we will contribute to an evidence base, and we will get
confidence around the answer. Not thinking that all the trials
that we're going to do are going to be definitive.
*We need to be asking patients important question
and getting patient important outcomes.
I do not know how it feels to have a doubling
of my creatinine. I need to have patient
important outcomes to be able to know, and to be able
to educate Mrs. Jones. Right?
Do you feel funny about that Mrs. Jones encounter?
So there must be ways of doing that better.
We need to have methods to enhance validity.
*We need to have routine registration of clinical trials
and protocols so that we can actually go back
and make sure nobody's cheating. I did a study
of clinical trial registries. I'm going in there,
it's just amazing how many stuff is not there, and how many times
the protocol says, "the rest of our detail
is on our Web site," and they give an institutional
Web site. And you go there and you go to
"Welcome to MERCK..." And then where do you go
from there? You know, where is the data?
And of course, if it lives in my Web site,
I can change it any day. *We need to have rigorous
conduct of research, rigorous complete and open
reporting of all the outcomes and all the data,
so we can actually look for these things.
We need to develop more spin detectors.
We need to get better at this. *The other thing we need to do
is we need to have methods to incorporate patient
preferences and decision-making. This is a decision made that
we need to talk about statins with patients in ways
that would not, hopefully, introduce bias.
And what we found with this tool when we used it in a clinical
trial was that patients really liked it.
It improved their knowledge, and improved their
risk estimation. It decreased their
decisional conflict. It improved their trust.
It improved the action if you think that 70% fewer
prescriptions in the lower risk category.
There were some short-term adherence improvements
of three-fold. And this was just giving
patients and physicians a script they can go through
to understand the patient's risk and the risk reduction
associated with the medication. And this was published
in the Archives of Internal Medicine.
So you can do these simple interventions to do it.
*I'm going to show you one potential mechanism for this
decision aid that has nothing to do with the content
of the decision aid. One physician with a patient,
no decision aid. Same physician
with a decision aid. Now we're having a conversation.
Is that the mechanism of action, or is it the fact that
we're provided information. I don't know.
*We're now doing this for diabetes medication choices.
I think it produces a major change.
Watch this. >> Yes, I agree, we need to
make sure your sugars are better controlled.
That means we need a new medication.
So which aspect of your next diabetes medication
do you want to discuss first? >> Weight change.
I'll let you review that. >> This is the one I'm on now,
right? >> You're already on medformin.
This one looks good to me. >> That one is going to help
you lose some weight. What aspect of that medication
would you want to learn about next?
I'd say daily routine. >> Sure.
Oh, so this is injectable. >> Yes, it's actually very easy.
There are a couple of pens that have needles,
and you inject into your belly. You do that twice a day
before your meals. >> I don't think I really want.
You don't want to go with needles?
I'd much rather just pills. So, the sulfonylureas?
That's the next best one for you in terms of weight?
That's a daily pill. You can start with one pill
a day. >> And just 30 minutes before?
Yes, usually 30 minutes before breakfast.
Would that be okay? The other thing to consider
is that sometimes they can give you low blood sugar,
so we're going to ask you to monitor a little more often
than you're doing right now. But otherwise that will be
okay. If you were to change your mind
after you go home, I'll give you this pamphlet.
It has the same thing as the cards.
So as you're using the tablet, if you find that it's not what
you expect, you can review this again and give me a call, and
we can discuss it again. >> Okay, all right.
Thank you. >> I think, at the end of the
day, we can get better at detecting bias and spin,
and bias reporting if we pay attention to
the revelation of sophisticated forms of reporting bias
and misconduct. If we are able to promote
patient participation and decision-making,
so that we get the whole idea of evidence-based medicine,
not only the evidence piece, but also the patient
participation piece. In fact, are able to advocate
for evidence-based medicine because the way we got to know
that we had the problem we have now, is because we were
following those principles of evidence-based medicine.
So if we were to stop, if we were to declare defeat,
if we don't train people to be alert of these issues,
what will happen is when we all become patients,
our clinicians will not have these tools to help us
be healthy, and to get the medication and treatment
that we would like to get. So maybe it's the end
of evidence-based medicine. But maybe there's
a little bit of hope. And I welcome your questions.
Thank you. ( applause )