Ay, yai, yai. It pains me to see faulty news reporting in my alma mater, The Denver Post — so much so that I desperately hope this link, yes, to this story right here, will be broken by the time you click on it because the news organization has been responsible enough to remove it.
The headline trumpets, “Legal pot shops do not boost teen drug use, Colorado study says.” It proceeds to lay out the findings of economists at the University of Colorado Denver, Montana State University and the University of Oregon. As The Post reports, these academics found “no correlation between an uptick in recent teen drug use and the increased availability of pot.” As the story specifically notes, the economists:
… combed wide surveys from 1993 to 2009. In those years, 13 states including Colorado legalized marijuana for medicinal uses, with varying access to dispensaries.
But get this: these economists reportedly examined data sets ending in 2009. Why is that significant — and why does that make their conclusions significantly wrong? And why does that make The Denver Post look foolish at best and indifferent and/or incredibly biased at worst?
Because of the U.S. Department of Justice’s release on Oct. 19, 2009 of what is now commonly known in knowledgeable drug-policy-debate circles as “The Ogden Memo.”
If you’re going to draw hard-and-fast conclusions much like those pronounced by these economists in Colorado, Montana and Oregon, or cover this issue for a news organization, you better make it your business to know the Ogden Memo, authored by former Deputy Attorney General David Ogden. The memo essentially advised federal law-enforcement agencies that the DOJ’s “rational use of its limited investigative and prosecutorial resources” would mean that marijuana-using patients and “caregivers” acting in “clear and unambiguous compliance with state law” weren’t a top priority for law enforcement.
That missive practically changed the landscape of so-called “medical marijuana” in the United States overnight. It is almost singularly the cause of a huge proliferation of dispensaries in states that have legalized medical marijuana — including Colorado, Montana and Oregon. Thanks largely to that 2009 memo, there are more marijuana dispensaries in metro Denver today than there are Starbucks in the entire state.
Before the memo? As in from 1993 to 2009 — the years the economists reportedly “combed wide surveys?” There were a few hundred people who used medical marijuana provided by their individual, registered “caregivers” in Colorado. The marijuana dispensaries we know today largely didn’t exist. The advertising they place in a host of publications — and on billboards, radio stations and at bus stops frequented by children didn’t, either.
In other words: to get an accurate picture of how “legal pot shops” truly have affected rates of adolescent drug use, we must examine data sets post-2009.
Rather than provide any of this insight — which isn’t really all that hard to find — The Denver Post ran with a half-baked story that quoted one of the study’s lead researchers, economist Daniel Rees of the University of Colorado Denver. “There’s just not a whiff of an increase,” he said. “If anything, it (adolescent pot-smoking rates) goes down.”
Well, duh, duuuuuuuude. Adolescent pot-smoking rates were, indeed, on the decline between 1993 and 2009. But then along came the Ogden Memo and hundreds of marijuana dispensaries in Colorado. After that, use rates among youth increased, and associated problems did, too. Studies in Colorado have confirmed this (never mind the 45 percent increase in drug-related out-of-school-expulsions from Colorado public schools in the last four years). National studies have confirmed this, too. Patient referrals to the youth substance-abuse-clinic Dr. Thurstone (yeah, this is his website) directs in downtown Denver have tripled since 2009.
And none of this was taken into consideration when Mr. Reese the economist — paraphrased and unchallenged by The Denver Post — conveyed that “critics pointing to an increase in teen drug use will have to look harder for valid causes.”
That’s only true in the worlds of economists and journalists who aren’t looking at valid data.
Christine Tatum is a former staff writer for the Chicago Tribune and The Denver Post. She served as 2006-07 national president of the Society of Professional Journalists. And yes, she’s married to Dr. T. You can learn more about her here.Â
This post deserves an update: today, we received a message from Ed Wood of Colorado, who ardently champions sensible drugged-driving laws and law enforcement.
He, too, makes valid points about the highly questionable methodology of Daniel Rees’ research. You can read much more here: https://drthurstone.com/base-duid-laws-on-responsible-research
Here’s to hoping more journalists find this information, ask far tougher questions of Mr. Rees’ research and report much more smartly.
Since this post was published, and Michael Booth of The Denver Post first raised the subject here of the Center for Disease Control’s latest numbers, plenty of people have clung like white on rice to the notion that adolescent marijuana smoking rates have dropped. They cite the CDC’s Youth Risk Behavior Surveillance System, of course. (You can see the numbers for yourself here. )
Folks who refer to these numbers as a “drop,” or a “decrease,” or a “fall,” or a “decline” demonstrate they know very little about how to analyze and interpret scientific data. Then again, these people may know exactly what they’re doing — and are just hoping the general public will fall for their deceptive “analyses.”
Tonight, we received this follow-up note from Kevin Sabet, who is referenced above:
“I just ran the raw data on Colorado’s CDC survey, which drug legalization proponents have been touting as an example of lowered marijuana use since dispensaries started in 2009.
“It looks as if they may not have any scientists on their side (surprise, surprise) because, when running the raw numbers, I found two bombshells:
“1) NONE of the differences are statistically significant. That means that there * is * no * difference.
“2) ALMOST ALL of the indicators for ALL drugs show an INSIGNIFICANT decline, meaning that there are OTHER reasons for the “no difference” between 2009 and 2011 than marijuana dispensaries — unless legalization proponents claim dispensaries also make kids not drink, not smoke, not do cocaine, not do prescription drugs — and not do other drugs.
“So the bottom line is this: there is NO difference, AND the NO difference can be found in ALL drugs. The data then do not tell us anything.”
Mr. Sabet’s findings are supported by researchers like Dr. Thurstone, who also has examined the CDC data.
And here’s an interesting side note: the CDC’s latest numbers — based on 2011 data — could be a big, fat flatline because 85 Colorado communities voted to ban dispensaries the previous year, 2010. We could be looking at steady numbers precisely because Colorado voters shut down pot shops, making marijuana less accessible and socially acceptable.
I do like this letter to the editor, addressing the problems with this bogus “research” and the subsequent reporting on it.
The author is Kevin Sabet. He has served three presidential administrations (most recently, the Obama administration) as an advisor on national drug-control policy. Kevin is thoroughly vilified by drug-legalization champions, but I always have found him to be knowledgeable, reasonable, measured and highly respectful of trustworthy science, regardless of its findings. You can find him online at kevinsabet.com.
Dear Editor,
Michael Booth’s piece, “Legal pot shops do not boost teen drug use, Colorado study says” needs some corrections.
First, the “study” in question is anything but — it is a paper that has not been peer-reviewed. And that is no wonder. The paper suffers from serious methodological errors, including the fact that the researchers did not take into account the actual implementation of medical marijuana laws.
For example, California did not have “dispensaries” until 2003, seven years after the law was enacted. Another state, Rhode Island, had about 1,500 people in the entire program, so it’s not a revelation that the state would not see any significant effect on teens. Furthermore, the study ignores the explosion of dispensaries – in Colorado and elsewhere – after 2009.
Finally, the study contradicts peer-reviewed research, published in journals like Drug and Alcohol Dependence, which have shown increases in use resulting from medical marijuana.
This is not so surprising given that the movement behind medical marijuana derives from advocates of full legalization. It’s time to get the legalization lobby out of the business of medical marijuana and instead focus our attention on the science examining the development of non-smoked, marijuana-based medications for the truly ill. That would make this issue no longer the sick joke that it is today.
Sincerely,
Kevin A. Sabet, Ph.D.
Chief, Drug Policy Institute, University of Florida
Kevin A. Sabet, Ph.D.
http://www.kevinsabet.com
@kevinsabet
Christine, if you’re going to do things like accuse my colleagues (and your former colleagues) of recreational pot use, then you should use my e-mail address to continue the conversation so you can try to maintain some semblance of professionalism.
Michael, I am not accusing you of recreational pot use. I am not naming specific names — and I absolutely wouldn’t do so in this forum or any other public forum.
However, do I know from my own correspondence marijuana users are on staff? Yes.
Do I know other journalists at other news organizations who are recreational marijuana users or possess “medical marijuana licenses?” Yes.
Should news organizations stop to consider the ethics of allowing journalists who use marijuana to have anything to do with the production of news coverage regarding issues concerning its legalization? Yes.
C’mon. Any respectable news organization would clamp down on me — and for very good reason — in a heartbeat. Why not someone who uses marijuana? Why scrutinize someone like me and not bother to look this hard truth right in the eye — behind closed doors — and ask other journalists to admit their conflicts? It’s incredibly naive to think those conflicts don’t exist.
And why did I know you would hone in on that one point and fail to address any of the others I have raised? 🙂
Ah, yes. Today, on the eve of legalization of recreational marijuana use in the State of Colorado, Denver Post reporters and staff were reminded by Linda Shapley, director of newsroom operations, to keep their recreational marijuana use in check — to help them keep up public appearances of objectivity, of course.
I can assure everyone these journalists didn’t start using marijuana after the publication of this blog post in June 2012.
The message from Ms. Shapley:
Hey all —
As we enter the hazy new realm of cannabis, we’re looking for you to use your wisest judgment on what you broadcast socially in relation to your personal use. It is not our intention to control your private lives, but your actions could compromise your (and The Post’s) credibility, so we expect you to follow our social media guidelines. Posts, tweets, images, Tumblrs, etc of your cannabis use could raise questions about your objectivity (this is similar to our stance on airing your political views). If you need a refresher, check out the Social Media Guidelines in our ethics policy, found on DP web in the left rail.
In addition, yesterday’s email on “liking” The Cannabist FB page was a suggestion. If you are uncomfortable or against pot usage, of course you do not have to like the page.
One last thing, which goes without saying: If you’re covering the events tomorrow, you’re working and not allowed to partake.
Thanks, and let’s have fun covering this historic moment.
I’m pretty sure you’re familiar with modern media, and that The Denver Post has blogs as well as a printed newspaper. As I had said, we discussed this further in today’s blog at blogs.denverpost.com/health, including the U.S. Attorney’s main points involving the Adams County survey and other claims. I also included the objection that the study in question does not go past 2009 data. I also mentioned, as an interesting point if not as part of a peer-reviewed study, that the 2011 CDC information is now available for everyone to look at, and has interesting statistics on Colorado. The Ogden Memo that you cite as a touchstone is certainly an interesting law enforcement document. Is it a study of actual marijuana use by teenagers in recent years? Not that I could tell, perhaps you can point me to the paragraphs I missed. It’s clear from the information and links provided on your web site where you are coming from on all this, but let’s also be clear that advocacy is different from journalism. If the CDC says marijuana use among Colorado teens dropped in 2011 from the previous sample, is that the “right” data or the “wrong” data? If their Adams County number shows an increase, but another county shows a decrease, which is the “right” data? In your husband’s study, do we know how many of the 80 people interviewed would have found marijuana anyway regardless of proximity to a medical marijuana license? All good questions. Would a respected researcher accept the premise that if more students are being disciplined by schools for marijuana, then it’s a given that a higher percentage of students are using marijuana, when studies show otherwise? I think they’d probably have more questions. If your husband would like to point to other studies he finds respectable and relevant, I’ve been telling everyone today that I’m happy to read any of them, as is John Ingold. We are not defending or attacking a point of view, unlike everyone else who has commented on this so far. Your husband did not choose to share any recent, relevant studies in his e-mail to me this morning. Please keep in mind I gave him the opportunity to comment for the original paper story; you did not choose to give me the chance to comment before writing your personally-compromised critique. If you would like to continue the conversation, I am available as always at mbooth@denverpost.com
You raise very good questions, Michael. I hope people who conduct trustworthy research will continue working to answer them.
However, I’m choosing to remain focused on the story that first appeared on DenverPost.com at 4:56 p.m., June 18 — and continues to remain there (and be distributed) in its original state. I’m also remaining focused on the story that appeared in The Denver Post’s June 19 print edition. They lack fairness and balanced perspective. They allow a very questionable source to trumpet a definitive conclusion unchecked and unquestioned. They make assertions and claims that were blown out of the water by drug-addiction/prevention/public policy/law enforcement/education/prevention-medicine experts in no time flat (and across time zones worldwide, according to my e-mail inbox).
By the time people started reacting to your work (and Mr. Rees’ goofball “findings”), you had already left the newsroom for the evening, and The Post’s print edition was ready to roll without any of these experts’ perspectives.
I couldn’t care less whether you contact my husband, Dr. Thurstone, to inform your reporting. There are plenty of other people locally and across the nation who could — and should — be commenting on Mr. Rees’ handiwork. Again, I’ll send you a short list with hopes you’ll get to know them because it’s a bad idea to slap headlines on “research” that hasn’t been peer-reviewed and is “privately funded” without almost breaking your neck to represent multiple perspectives that either corroborate it or debunk it. Especially when the story is about one of the most important public-health-and-safety issues facing Colorado.
And yes, I do understand “modern media,” including print editions, online news production and blogging. The tools may change, but the responsibilities to present information accurately and fairly do not.
There’s another concept I also know very well in journalism. I call it “Clean-up on Aisle 5.” It is the chief reason you spent a good chunk of your day June 19 “continuing the conversation” on a health-related blog. The pushback you received from your story was strong and well-deserved, so you acknowledged information that puts Mr. Rees and his cohorts’ findings into a more appropriately skeptical and critical light. On a blog. A blog post that I can assure you is not being shared to the degree of your original story in the June 19 print edition and, again, sitting online, untouched, since it first was posted.
(By the way, there’s nothing stopping you from updating your original story and/or linking it prominently to your “continued conversation” over on that health blog. You should.)
You continue to ask about what’s “right” data and “wrong” data. Those are the wrong questions. The question should be, “Do the data examined/analyzed/reported support the conclusion being made?” What Mr. Rees and his colleagues have done is entirely overreach — and they’re overreaching for a reason, Michael. Part of your job should be to look into that, too.
Your comments indicate that you’re looking at the CDC 2011 data in the isolated, annual, up-or-down, year-over-year percentages that are popular in journalism and with people trying to manipulate data to advance specific agendas. This tells me you could stand to develop sources in prevention-medicine research to learn how to view data from their very important perspective.
Why? Because there are very well known lag times between watershed moments (such as, oh, say, the release of the Ogden Memo) and widespread public impact. So, for example, it’s foolish to interpret from CDC 2011 data that dispensaries have had no discernible impact on adolescent marijuana use. That’s overreaching as badly as Mr. Rees and his colleagues. Why? Because the data you’re looking at were collected only a year from the start of the boom of marijuana dispensaries in Colorado. Given the “prediction windows” that are fairly common and standard in prevention medicine, this state won’t begin to have a good idea of how dispensaries have affected adolescent pot smoking rates until at least 2013 or 2014.
I can tell you that there are medical scientists (better folks to talk substance use and addiction rates with than three economists who generate slicked up PDFs for shadowy “private funders”) looking at the CDC data. They are drilling specifically into the sub-sets regarding “social acceptance” and “perceived risk of harm.” Why? Because that data help explain what we should look forward to in the next couple of years.
By then, a lot of public policy — unfortunately shaped by some pretty darned lopsided and ill-informed journalism — will have been decided, of course.
I realize that I would never be allowed to cover this subject in a newsroom because of what my husband does for a living. I wouldn’t even ask because of the journalism ethics I revere. At the same time, I find it royally amusing that you characterize my critique as “personally compromised.” Look around, Michael. You have some colleagues who smoke pot recreationally — and I’m willing to bet not one person at The Post has ever taken note of who they may be and asked everyone who may fall in that category to steer clear of news coverage addressing this issue — and the larger Colorado ballot initiative concerning legalization for recreational use later this year. But me? I’m married to a physician and clinical researcher — a guy who is actually speaking outside his economic interests to inform people about the degree to which marijuana harms youth. And I’m “personally compromised.” Um, OK, all righty then.
I also know this subject very well. I know responsible journalism very well. I don’t have to hide my opinions from the world.
And if I were still working for The Post, I would have put aaaaall of this into one heck of an internal memo.
One other thing I’ll note — at 9:31 p.m. — because of an e-mail Chris just received from a pediatrician in Louisville, Ky. She reported that a daily newsletter she receives trumpeted “Legalizing Medical Pot Doesn’t Increase Teen Use.” Same regurgitation of this sorry PDF posted by this “private, nonprofit research institute.”
What’s really telling and very sad to me is that nonsense such as this gets immediate attention and better play from news organizations than does carefully conducted, heavily scrutinized science resulting from years of hard work.
Christine — Whatever happened to the revered tradition of contacting the journalist before you write a rant? I contacted Dr. Thurstone in plenty of time to comment on the study, he did not respond until today. His only comment was that he wanted to see later data. I have no stake in this study either way, though others certainly appear to take a side. But if you want more data, look at our Daily Dose blog for links to the 2011 CDC behavioral surveys released last week. They show the percentage of high school students smoking marijuana in Colorado at 22.7 percent in 2005, and at 22 percent in 2011. The three economists are now working their way through the new CDC data. Which is the “right” data? I would be interested in a list that you might annotate of what is the “right” data on this issue and what is the “wrong” data. In this case, the “new” data does not appear to be to the satisfaction of people who are against dispensaries.
Hi, Michael. I consider you a friend. It is not at all my intention to criticize you as a person or to criticize your overarching body of work (which I profoundly respect), and I hope to make that emphatically clear from the outset. I read your story, knew the report on which it was entirely based was deeply flawed and worked very late last night to dispute what I knew would be spun into big headlines this morning.
Just so you know: Chris was in meetings all day yesterday. He would’ve welcomed your call had he been able to take it. There are other people in Denver who are very knowledgeable about this subject, and I’ll send you a short list of names and contact information with hopes it will inform and bring greater balance to your work on this topic in the future.
And no, I don’t believe you have a “stake in this study either way.” I really don’t.
The three economists may be working their ways through the new CDC data — but that’s not the story you’ve reported. And that’s not the data they were examining before making their sweeping remarks. They analyzed pre-2009 data, which is glaringly problematic for the reasons cited above. No need to rehash that.
Their work was not peer-reviewed. That should have been mentioned in your story. Their work was not published in a reputable, scholarly journal. That, too, deserved a mention. Their work isn’t supported by locally conducted research, some of which has been peer-reviewed and published and can be found on pubmed.gov. As you might suspect, I think that should have been mentioned, too. I also believe one data source — even if it’s the Centers for Disease Control — should be weighed against other data sets, such as the Adams County Youth Initiative (which doesn’t at all support these economists’ findings).
The economists also do not state the source of funding that supports their work — which should also make their findings suspect. These funding sources also should have been included in The Post.
Then there’s the very place where this “discussion paper” was submitted. Look into The Institute for the Study of Labor in Bonn, Germany, and you’ll see that several of its members and senior researchers have ties to, or have received funding from, the Soros Foundation or billionaire George Soros. Mr. Soros has funded many, many studies and organizations whose soul mission is to legalize drugs. I simply find it curious that this just happens to be the same institute where University of Colorado Denver economist Daniel I. Rees has submitted his findings more than once. Those other findings have also made sweeping pronouncements about marijuana based on old data sets — always conveniently pre-Ogden Memo — and some of his “discussion papers” (particularly the ditty he released in November 2011, about how medical marijuana legalization contributed to decreases in traffic fatalities) have been so thoroughly debunked that I must confess feeling some amusement that he continues to speak with such arrogance. (“There’s just not a whiff of an increase,” he told you.)
I am familiar with the CDC behavioral surveys released last week. Again, those weren’t the subject of today’s story, and they weren’t the data these economists examined. I wholeheartedly agree those numbers need to be studied carefully — and I also know the “overall state use rate,” whether it goes up or down, doesn’t necessarily explain the impact medical marijuana dispensaries have had, or are having, on teen pot use. For example, some counties have dispensaries. Others do not. It will be important to look at individual zip codes. The CDC data also need to be compared to other states that don’t have medical marijuana and do share key characteristics with Colorado.
About your question concerning “right” and “wrong” data: as Chris told Colorado Public Radio today, the problem is not with data itself. It is with making conclusions your data do not support.