Dr. Dennis Bier
I firmly believe that the principles of the scientific method established over the last several hundred years are the same in physics as they are in nutrition to determine proof, there are certain fundamental principles of science that I believe are no different in nutrition or in the age of the universe. Okay. The problem is not that the principles are different, and many people try to claim that the science of nutrition is different, somehow. So I think that’s a slippery slope. The principles of proof are the same.
Matt Miller
Hello, everyone, and welcome to another episode of Methodology Matters. Brad, how are you doing today?
Dr. Bradley Johnston
I’m good, Matt. Good to see you.
Matt Miller
Good to see you. I’m very excited about our discussion today. We’re talking about our, the part two of our interview with Dr. Dennis Bier. And today we’re digging a little deeper into, I think, more of the science behind what the two of you talked about rather than, you know, previously we got a nice view of what writing a scientific journal is like and kind of where studies come from and how you sort through them. Right. Here we’re hearing from Dr. Bier about kind of the standards of evidence in nutrition science and what he’s seen and we’re getting a little more into the nuts and bolts of the scientific research.
Dr. Bradley Johnston
Yeah, exactly. The standards of evidence. He talks a lot about that. We talk a bit about causal language, which we’ve done before, the importance of protocols in terms of driving your research and analysis.
Matt Miller
Yeah.
Dr. Bradley Johnston
Yeah, So looking forward to sharing this interview with our audience.
Matt Miller
Yeah, I think it’s great. He’s a wealth of knowledge. He really is. So you mentioned causal versus associative language. We’ve talked about this a little bit before, but like just as a just as a brief review, you know, what would be considered causal language versus associative language or how do we determine when things are causal or when they just have associative properties?
Dr. Bradley Johnston
Oh, that’s a big question. You know, Dennis talks about how difficult it is to actually differentiate causal language. Sometimes he uses an example from the epidemiology world of attributable risk or attributable. Yeah, I think it’s attributable risk. And he basically says, well, we use this term often in the space of observational data, but just by the term alone, it almost sounds like it’s causal.
Matt Miller
Yeah.
Dr. Bradley Johnston
So, you know, I’ve been involved in research papers before when the editors have said, you know, remove the causal language or tone it down a bit. And, you know, I go through my own work and it is, it’s actually sometimes challenging to understand what’s causal in the way things are worded versus what’s not. And it’s obviously there’s no clear answer sometimes. Right.
Matt Miller
Yeah.
Dr. Bradley Johnston
Like, the use of the term impact, you know, low carbohydrate diets impact the risk of outcome X. That’s almost causal in nature.
Matt Miller
Yeah
Dr. Bradley Johnston
But, you know, some might argue that it’s not.
Matt Miller
Yeah. Interesting. Yeah. It seems like rather than a very distinct line between causal language and associative language, there’s sort of a wide gray area, shall we say.
Dr. Bradley Johnston
Yeah. And then he says in his experience at the AJCN, you know, it’s very interesting how often this causal language still seems to, you know, even after you spend a lot of time trying to get it out of manuscripts, it still seems to kind of creep in or stay in certain parts of the manuscript often, which can be problematic, especially when it’s someone that’s more novice, that may be reading the paper or journalist that’s reading the paper. The distinction is really important that we hear a lot about this idea of correlation.
Matt Miller
Yeah
Dr. Bradley Johnston
Right. You know, there’s many things that are correlated, but that doesn’t mean that they’re causal in nature.
Matt Miller
Yeah, yeah. Well, and you can find a quick search on the Internet will find some very extreme examples of this, which I would encourage all of our listeners to do for a good laugh. But it does sound like, you know, the root of this is really that we that when scientific papers are being published, both the editors and the authors have to be very specific about what they’re sure of and what they’re not sure of and crafting language that is in line with that.
Dr. Bradley Johnston
Yeah. You know, and Denny comments about, we’ll have to get a citation for thisut, b he says about 90 percent of published papers have at least one positive finding. And he says, well, you know, we’re not that good. Now, how is it. You know, what’s really going on? Is it you know, if we do one hundred experiments, are 90 of them really have a positive finding? Is that true?
Matt Miller
Yeah
Dr. Bradley Johnston
That feels like it’s probably not true in the ultimate reality. But for some reason, the scientific literature seems to suggest that we often do find these positive findings, which is slightly odd.
Matt Miller
Yeah. Statistically, that seems. It sounds a little bit like a line from Anchorman. It’s like 70 percent of the time, it works all the time. It’s like, well, what does that mean? So I wanted to also bring up Dr. Bier talks about the difference between observational data and interventional data. And I feel like I have a very rudimentary understanding of that. I’d love for you to kind of expound on that a little bit so that those of us that are listening to this with not, shall we say, a storied of a scientific background, as Dr. Bier or yourself might start with some clarification about those terms.
Dr. Bradley Johnston
Yeah, sure. Great question. So observational studies, there’s many different study designs in nutrition or nutritional epidemiology. Probably the most popular one is a cohort study, typically prospective in nature, where scientists essentially observe a population over time while documenting certain risk factors or exposures such as smoking weight, alcohol use. And they also document health outcomes. And they might follow these people for 10, 20, 30, 40 years. And they do statistical analysis and they often find associations or correlations which are not, you know, one has to be very careful about whether it’s an association or causation and it’s typically associative in nature.
Matt Miller
Gotcha.
Dr. Bradley Johnston
Versus an experimental study. You know, observational studies are in free living human beings.
Matt Miller
Yeah.
Dr. Bradley Johnston
Whereas experimental studies, they’re often allowed to continue to live freely. But we manipulate something. We give an intervention. So you get, for example, randomized to peanut butter or no peanut butter or usual peanut butter intake versus lower peanut butter intake or to a drug versus a placebo.
Matt Miller
Mm hmm. Mm hmm. You mentioned peanut butter, based on my understanding of what you just said. You know, if I was trying to construct a study that was like, does eating peanut butter cause cancer, then like an observational study would be, you know, I’ve got 100,000 people and I send them a questionnaire that says, have you ever eaten peanut butter and do you have cancer? And they give those answers back to me. And I analyze that data and see if there’s any association between eating peanut butter and having cancer. But in an interventional sense, it’s more like I would take a smaller group of people and randomly assign them to the control group where they eat no peanut butter or continue to eat peanut butter undirected. And then the interventional group where they eat a certain amount of peanut butter and then we see if anybody gets cancer. Would you say that’s accurate?
Dr. Bradley Johnston
Yeah, exactly. That’s a great analogy. I think the main thing is you’d have to have a decrement in difference between peanut butter in the experimental arm versus the control arm.
Matt Miller
Gotcha. Gotcha.
Dr. Bradley Johnston
Yes, so maybe the control group continues. Peanut butter, as normal as you suggested, and the experimental group, is asked to abstain from peanut butter, if that’s ethically okay. I’m assuming it is. But, you never know. It’s like, this is quite easy in drug studies because people get a placebo that looks exactly the same, hopefully as the drug and the drug is standardized to a certain amount of ASA, aspirin, for example, or a certain amount of some active component, like 50 milligrams of X and the placebo has nothing. So there’s there’s a real difference between the two groups. That’s much harder to achieve in nutrition science. Typically, if the intervention is is diet oriented. Now, of course, you can do nutrition studies where you give people a probiotic versus a placebo or vitamin D versus a placebo.
Matt Miller
Yeah
Dr. Bradley Johnston
Those are easier studies to do and kind of reflect a drug paradigm, if you will.
Matt Miller
Yeah, that makes sense. I hope that if somebody does that peanut butter study, they don’t find anything because I think peanut butter is delicious. You mentioned, and I’m glad that you brought it up, you mentioned kind of the difference between us being able to do a nutritional intervention versus a drug intervention and how easy it is as far as like drug placebo, you know, taking an actual pill or let’s say a nutritional supplement and taking a placebo, that looks exactly the same. It kind of leads us into the importance of observational data. Right, I mean, it sounds like interventional data can produce science that we’re more sure about, shall we say, I’ll use in layman’s terms. Right. But also, it feels like observational data is more like kind of a breeding ground for hypotheses and ideas. And while that data might not, we might not be as sure about that data as we could be. It sounds like it’s a really important step in the scientific process. And one of the things that Dr. Bier mentioned this, too, in the interview, like whether you’re physically able to do an intervention or B: whether it’s like whether you’re allowed to vis-à-vis ethics, you know.
Dr. Bradley Johnston
Yeah, it’s well known that in nutrition science, a lot of the dietary guidelines, for example, are a lot of the things that maybe a nutritionist might recommend are based on observational data.
Matt Miller
Yeah.
Dr. Bradley Johnston
And the question is, if we’re interested in standards of evidence and knowing what the certainty of that evidence is, the question is how do we best determine that? And there’s been lots of debate in the field of nutrition about whether we should or shouldn’t use the grade methodology, for example.
Matt Miller
Yeah.
Dr. Bradley Johnston
And then Denny, he gets into talking about and I have to agree with him on this. He says that many people in nutrition claim that the science of nutrition is somehow different. And he goes on to say that his belief is that, you know, the principles of proof are the same regardless of what field you’re in. And I believe that to be true. And I think a lot of people in nutrition do as well. The standards of evidence really should be the same in nutrition, medicine, pharmacology, exercise science, physiotherapy, you name it. Like we should ideally be using the same standards of evidence to look at the certainty of evidence, regardless of what our research question is, so that we can compare apples to apples.
Matt Miller
Yeah, that’s interesting. It makes me think of, you know, and somewhere like a retired PTA member is silently clapping for this. My high school science teacher, freshman year of high school, Mr. Livingston, he was the first person that ever sort of taught us the scientific method. Right. And he was a lovely teacher and just a wonderful guy, but also like very strict as far as the methodology he was teaching us. And what we learned from him was essentially that like observational data leads to interventional data. Right? Like, you can’t just say, let’s do this really strict experiment to try and find outcome X without having sort of, I guess, observations or documented observations or observational data to support. You know, doing a more serious study, right, so it seems to me like observational data has a really important role in nutrition, but not the role that interventional data has. Would you say that’s accurate?
Dr. Bradley Johnston
Yeah, it’s got a huge role. And at the end of the day, there’s a lot of nutrition questions that we can’t feasibly randomize people to or follow people long enough for or feed people long enough for in order to really get the answer that we would like to have.
Matt Miller
Sure.
Dr. Bradley Johnston
But it doesn’t mean that we should be flimsical, if you will, about what the standards are. You know, the standards of proof should be the same in whatever field of science you’re in, you know.
Matt Miller
Yeah.
Dr. Bradley Johnston
As you can tell, I do believe this.
Matt Miller
Sure.
Dr. Bradley Johnston
But it’s an area of debate, so. And that’s what makes science fun.
Matt Miller
Yeah. Yeah, I think it’s great. It’s great to sort of like lay all these different points of view out there. I love that Dr. Bier talks about it as well, you know because I think it’s easy for somebody like yourself who is evidence-based in nature to get sort of labeled with a like observational bad, interventional good. And, you know, all we want is randomized control trials and nothing in between. You know, it’s not quite as accurate maybe is as one might think. But I love that Dr. Bier just essentially says, like, you know, good science is good science.
Dr. Bradley Johnston
Yeah. Well, it’s important to keep your mind open and avoid our almost natural habit of ideology.
Matt Miller
Yeah.
Dr. Bradley Johnston
And motivated reasoning. What was that quote that I think we talked about in an earlier episode from Richard Feynman? The job of a scientist is to not let the data trick you. And you’re the easiest person to trick something like that.
Matt Miller
Yeah, yeah, yeah.
Dr. Bradley Johnston
Which is very true. And maybe I’ll just say too, really interesting. There’s been a recent study in the AJCN.
Matt Miller
Oh yeah.
Dr. Bradley Johnston
That looked at how often authors of systematic reviews of cohort studies assess the certainty of evidence. So it was a study by Dena Zeraatkar and Russell de Souza and Team at McMaster University in Canada. And they did a random sample of 150 systematic reviews of nutritional epidemiology studies, and they found that less than 11 percent of studies actually rated the certainty of the evidence. So in other words, they find an estimate of effect for peanut butter and its association with, let’s say, diabetes.
Matt Miller
Sure.
Dr. Bradley Johnston
And they tend to not talk about the certainty of that estimate of effect. Right. You have, let’s say something, a two percent risk reduction. But how certain are we in that two percent?
Matt Miller
Right.
Dr. Bradley Johnston
For whatever reason, I think nutrition science is behind, let’s say medicine, when it comes to people embracing this concept of certainty, of evidence. And Denny talks about this as being one of the biggest advances in medicine and in the health sciences over the last 20 years. Kind of, it’s been led by the great working group, I think. But there’s other groups as well that work within this space, you know, this idea of certainty of evidence and using that in addition to our best estimate of effect to help people make better decisions.
Matt Miller
Gotcha. I love that. I feel so smart when we talk about standards of evidence and certainty of evidence and best estimate of effect. It’s great. You know, you’re a science guy. It’s all in your daily routine. But it makes me feel fantastic about myself.
Dr. Bradley Johnston
Alright. Well, I’m glad we’re having that effect, Matt.
Matt Miller
Anyway, so we’ll link to the AJCN article in the show notes, because I think it’s a really great read, especially when compared to this interview that you had with Dr. Bier. The two of you talk about some really great stuff. Without further ado, here is part two of our interview with Dr. Dennis Bier, former editor of the AJCN.
Dr. Bradley Johnston
With respect to the American Journal of Clinical Nutrition and the field of nutrition, what is your sense of the quality of the methods that are used with respect to applied nutrition research?
Dr. Dennis Bier
Well, this is, you know, I think this is, in a sense, one of the most crucial issues in any field, but certainly in nutrition. I firmly believe that the principles of the scientific method established over the last several hundred years are the same in physics as they are in nutrition to determine proof there are certain fundamental principles of science that I believe are no different in nutrition or in the age of the universe. The problem is not that the principles are different and many people try to claim that the science of nutrition is different, somehow. I think that’s a slippery slope. The principles of proof are the same. What’s different is your ability to actually carry out the principles. So in many ways, for example, certain experiments on the age of the sun are easier than certain human experiments because the ethical issues of experimenting in humans don’t apply. Human behavior doesn’t come into place. Longitudinal behaviors following a diet for 20 or 30 years doesn’t come into place.
Dr. Bradley Johnston
Mm hmm.
Dr. Dennis Bier
So it’s the fact that we can’t always achieve the level of proof that we would like to have. And there are several, you know, kind of approaches to this. One is well, first, the continued search for ethically acceptable ways to experiment in humans to try to determine causality, because you can determine causality with an intervention. And when you’re doing something to a human being, you have to have a way of doing it in a totally ethically acceptable fashion. And even the ethics can depend differently on. the country involved, the people involved, but that’s a principle methodological in human studies is to find acceptable methods that get us closer to causality.
Dr. Bradley Johnston
And is there, can you think of a is there an example in nutrition where there may be ethical implications that don’t allow us to be as experimental as we’d like to be?
Dr. Dennis Bier
Things like breastfeeding. OK, at least among human populations, it’s not generally considered ethically acceptable to randomize women into whether they should breastfeed or not. So all breastfeeding studies at the beginning depend on women who want to breastfeed and accept that in women who don’t and accept an alternative. So right away, you don’t have a truly randomized blinded study.
Dr. Bradley Johnston
Right.
Dr. Dennis Bier
There are lots of other interventions that are fundamentally impossible to blind or almost impossible, and therefore they’re flawed to begin with or not only impossible to blind, but impossible to provide allocation concealment. Someone has to give the treatment so when they know it’s given.
Dr. Bradley Johnston
Mm hmm. Mm hmm.
Dr. Dennis Bier
Now, we have come a long way in that because, as you know, I think, you know, we now have mock surgeries being done, which patients, adults accept as part of a randomized trial and many of those have shown that surgical procedures that everyone thought were really wonderfully magic occurred just as well in the people who got the box surgery. It wasn’t the procedure itself. OK, so these things happen. But the principal ones are the ones you identify the differences between observational and interventional research diets for long-term benefit. That is the issue that’s on the major plate of nutritional science today. So in the first half of the 20th century, we determine the essential nutrients and we determine those because people who are deficient in the essential nutrients when those are replaced, their signs and symptoms of whatever the consequent diseases go away. So we were able to do important interventional experiments that benefited the patient with a hard endpoint. They got better.
Dr. Bradley Johnston
Mm hmm.
We’ve now moved into an era where people want to make predictions of the change in risk over a long period of time based on diet changes, which not everybody benefits by. It’s not like everybody is deficient, then suddenly gets better.
Dr. Bradley Johnston
Mm hmm.
Dr. Dennis Bier
So we have to deal with a very different paradigm and we also have to deal with the practicalities of you can’t blind most diets. You can change them, but you can’t necessarily blind them. You have to deal with the fact that people may need to keep this interventional diet for 20 or 30 years to show a long term benefit. That’s almost impossible.
Dr. Bradley Johnston
Mm hmm.
Dr. Dennis Bier
And in addition, what I think one of the biggest things we know in human interventions is that the human being lives in this environmental exosome, which historically includes a handful of things we know about, like socioeconomic class and education and things of this nature, and that has persisted in observational epidemiology and observational nutrition because we know about them and some of them we can measure. But what we now know through all of the activities and big data over the last 20 years is that those variables are not two or three or five confounding covariates. There are hundred and no one yet knows how to deal with those 60, 80, 100, 200 covariants. When observational studies are done, you show an association between X, whatever you’ve chosen and and Y, but you haven’t been able to deal with all the covariates. It may not be X at all, it may be Q or A, B, plus C and we’ve just not been measuring those.
Dr. Bradley Johnston
Yeah.
Dr. Dennis Bier
To me, the paradigm that you can assign implied causality from knowing the Association of X and Y no longer applies. We no longer can say that with any degree of confidence. The epidemiologists, the observational researchers all know that an association is not causal. It’s an association. But they wouldn’t be doing the studies unless they thought it was causal, who would care whether eating pepperoni pizza was associated with the outcome of pregnancy unless there was a causal link? Otherwise, who would care? So they all think there’s a causal link and they want to imply a causal link. And even though they know better, the observational literature is filled with implied causality. This is one of the major jobs of editors is to take out causal words from studies in which they only showed associations. It’s astonishing how many causal words can creep into a manuscript, particularly into the conclusion of the abstract when they’ve taken it out of all the text in the paper it stays in the conclusion of the abstract.
Dr. Bradley Johnston
Well, and even the Journal of the American Medical Association doesn’t even allow one to use causal language for systematic reviews of randomized trials, for example.
Dr. Dennis Bier
But that’s very subtle. You have to be really alert and educated to what is a causal word. And they appear everywhere in very subtle ways. And in some cases, I believe the observational community in the past has gotten them accepted. So, for example, we have things like attributable risk. Yeah, you know, I mean, that’s a causal association to me.
Dr. Bradley Johnston
Yeah.
Dr. Dennis Bier
And it’s the accepted term in certain kinds of observational research. I mean that I believe that the observational research people got that as the accepted term because then they have to worry about implying causality. It’s right there.
Dr. Bradley Johnston
So, Dr. Bier, even if the editors do a great job of getting the causal language out of the manuscript and the manuscript is penned in a way that’s very cautionary, then one has to deal with what journalists do. They then may start to use causal language.
Dr. Dennis Bier
Well, by the way, I agree with that, but I think I’d like to step back one step before the journalist. Right. So at the end of every paper, most papers or any papers. There’s a section about limitations, the limitations are almost always boilerplate language that start out with the assumption that the association I showed you was in some way correct. They never start out with saying this may have nothing to do with it at all. And you see, I mean, you know, the limitations are always in here. We’ve shown you the nucleus of the right answer, and we’re not entirely sure because we haven’t dotted all the I’s and cross the T’s. They never start out with the fact that this may not be in any way the variable responsible for the association. So it starts out with the authors. And as you well know, 90 percent of the literature, 90 plus percent of the papers that are published include at least one positive finding. That’s just impossible. We’re not that good, you know it’s impossible. And the reason is that people have the human trait of wanting to be right, wanting to show something positive. So all the wording in the papers, how they couched the limitations of things are we’ve actually got it. They may not be as good as we want, but we’ve got something here. And then they pass that on to the journalists. I think if the scientists themselves were frank with the journalist from the start, we’d have less of that. And there are a few journalists who always preface their articles with things about causality and stuff. Although now I’ve noticed that this is my opinion by the way, I have no data for this. I’ve noticed that maybe in the last 10 years the words, remember association is not causality appear more than they used to mean. The frankness of how good your evidence is. People don’t want to admit they don’t have the highest quality evidence. I have no objection to coming to that point. In fact, there were places we can’t get by today’s ethical constraints and human behavior to the highest quality evidence. But that doesn’t mean you promote the evidence you got to the next level. Right. You have to stop where you’ve got it. And that’s the issue. There’s a pervasive promotion of the levels of evidence. We also need to start having more emphasis placed on the confidence we have around our estimates, not just what our estimates are. And I’ve said this about medical education for all the audience and anyone who may want to go to medical school, medical schools spend all their time on what we know and not so much time on what we don’t know well, and we don’t know far more than we know. Right. And so, I mean, I think we need to understand how good our opinion is. And this is a critical issue in all scientific publications. There has to be a way of conveying how confident you are in what the outcomes are.
Dr. Bradley Johnston
And and, of course, that there is a methodology developed for that.
Dr. Dennis Bier
Yes. That’s one of the biggest advances in the medical literature in the last 20 years. Far more emphasis on the confidence intervals and the certainty with which you can say things. And that’s been now codified in various reporting requirements, which I think have improved the quality of the literature immensely. Same time, we haven’t really got the same level of rigor, let me say it that way, in the inflow, and that is the registration and the complete availability of data sets, free description of data, analytical methods, pre-description of terms that are used willy nilly in papers such as validation. I don’t think I’ve ever read a paper which told me in the registration document how close the association had to be valid. It’s always after the fact. It’s good enough. Well, we thought we’d get eighty percent. We got sixty five. That’s good enough. So we’ve made a lot of progress on the reporting end and on the grading end. People like Gordon Guyatt, those that we’ve made great progress, even though it’s still a small fraction of the scientists who embrace this as making their work better. We have far more to go at the front end to make people put their cards on the table before they make a conclusion. And, you know, when I was a young person, all the scientists who taught about publications would say the first thing you do is read an abstract, read the abstract and see what they’re claiming. Well, I no longer believe that. I believe the first thing you do is read the registration document.
Dr. Bradley Johnston
Excellent, yeah.
Dr. Dennis Bier
Registration and document lays out what they say they’re going to do. And it’s extraordinarily remarkable to me how many papers don’t do it.
Dr. Bradley Johnston
Yeah, I totally agree.
Dr. Dennis Bier
The registration document has X as the primary outcome variable. You read the manuscript, it talks about Y, Z, A, B, C. X never shows up or it shows up as line 12 in table six. Never mentions. Right.
Dr. Bradley Johnston
So a priority study protocols and in particular detailed study protocols that layout the statistical analysis that will be done, that lays out the outcomes of interest, all of the outcomes of interest, what’s primary, what’s secondary, that lays out your your subgroups of interest, including ideally the anticipated direction of effect, that you think you might see in males versus females, if that’s a subgroup of interest.
Dr. Dennis Bier
Or how are you going to analyze the data. So you perfectly well, we’re going to analyze the data by Method X and then you look at the paper. Method X doesn’t show up at all. Now, you know, they did Method X first, right? This used to be a problem to say, well, I’ve analyzed it by three methods, but now this is easy. You have supplementary material on the Web. All of these things can be put up or very simple things like you have an observational study and the groups are divided into quintiles. Well, you know, your last paper was quartiles and the next paper was sixtiles. Well, why are they different? Right. I mean, never told about the number of observations that you actually made so that you can determine what a realistic assessment of false discovery rate is.
Dr. Bradley Johnston
Yeah. Yeah. Great. Great points. Thank you.
Matt Miller
Thanks for listening, if you’d like to hear more episodes of Methodology Matters, please head over to methodologymatters.podbean.com, or you can find us on Spotify and on Google podcasts.
Dr. Bradley Johnston
And if you’d like to learn more about Dr. Dennis Bier, you can find his faculty profile at Baylor College of Medicine’s website linked in the show notes for this episode below.
Matt Miller
Thanks for tuning in. We’ll see you on the next episode of Methodology Matters, a podcast on evidence based nutrition.