Perhaps the single most consequential and controversial issue at the center of the onshore natural gas drilling boom is the question of methane leaks. Natural gas is primarily made of methane, a powerful greenhouse gas, and if enough escapes into the atmosphere, these leaks could potentially make natural gas a worse fuel for the climate than coal.

In mid-September, researchers from the University of Texas published a study that was hailed by a triumphant oil and gas industry, which claimed it definitively showed that methane leaks from fracking are minimal. Major news outlets largely fed this excitement, proclaiming that the study showed EPA had dramatically overestimated methane leaks from the drilling boom.

But as the celebrations died down and more sober and rigorous analysis of the study has begun, scientists are finding that the University of Texas study is riddled with flaws.

The backers of the report cherry-picked the oil and gas wells included in the study, selecting smaller wells that had less capacity to leak and ones that used leak controls that are not currently used at many of the nation’s wells. The authors systematically ignored more recent federal research indicating that as much as 17 percent of natural gas – more than 10 times the estimate indicated by the UT study – leaks from gas fields, and overlooked serious methodological flaws that were pointed out in similar studies dating back as far as 1996.

As scientists have raised these concerns, the Environmental Defense Fund, one backer of the study which was 90 percent funded by the oil and gas industry, have tried to tamp down some of the media excitement surrounding the result and said that their research was misrepresented.

For example, the Wall Street Journal headlined its coverage of the report: “U.S. Overstates Leaks by Gas-Drillers, Says Study.” But some at EDF have said The Journal got it flat wrong.

“No. The WSJ headline is not an accurate reflection of what the study found,” wrote EDF’s Sam Parry in an online discussion. “What the study found was that when you use green completion technologies, you can mitigate the problem of methane leakage. If you don’t use these technologies, the problem is real and very serious.”

Despite Mr. Parry’s allegation, no retractions or corrections have yet been made to the mainstream press coverage of the study.

EDF’s complaints that the study was misreported are finding limited sympathy from some in the research community, who say that EDF officials have allowed their study to be misrepresented in the public eye by failing to mount an effective response to counter the inaccurate press coverage. EDF, they say, actually made things worse by promising that their 16-part research series, of which the UT study is the first, would provide the exclusive and definitive answer on methane leaks.

“It is very bad form to allege that the contributions of others are mere ‘snapshots’, and then offer up a very few snapshots of your own as somehow ‘definitive,’” said Prof. Anthony Ingraffea, one of the foremost experts on methane and climate change, and co-author of a seminal paper that first called into question the claimed climate benefits of natural gas. “It is for other stakeholders to judge the value of this research project, not the scientists and engineers performing it.”

The University of Texas study’s many flaws have cast further doubt on the claim that EDF’s series will provide a definitive – or even reliable – assessment of natural gas’s climate impacts.

The stakes are enormous. Over the next two decades, the greenhouse effect from methane that leaks today will be up to 86 times as powerful as today’s carbon dioxide emissions. The single largest source of methane emissions in America is the natural gas industry – and right now that industry is experiencing a gold rush that has created a vast new network of wells, pipelines, and storage facilities across the U.S. – all holding methane, the main ingredient in natural gas.

If natural gas leaks at more than 3.2 percent overall, any climate change benefit gained by switching away from coal for electricity generation and burning natural gas instead will be lost, although that critical threshold may soon be lowered, as the Intergovernmental Panel on Climate Change recently concluded that methane is even more potent than previously believed.

The UT study had a single narrow aim: to measure the amount of methane that leaked during certain stages of drilling and fracking a natural gas well.

A team of researchers descended upon 190 oil and gas wells owned by 9 major oil and gas companies, studying 27 hydraulic fracturing events and several other parts of the completion process. They used this data to estimate that on average, 0.42 percent of natural gas from a gas well could be expected to leak out at the well pad during a stage called “completing” the well. That’s consistent with EPA’s current official but controversial estimate that 1.5 percent of natural gas leaks overall, after accounting for leaks from places like pipelines (the agency’s own Inspector General called for more data and a closer review of that figure).

Some of the earliest criticism of the UT study focused on a major internal contradiction within the study itself. The well sites were selected with substantial input from the oil and gas industry, which volunteered specific sites, and the vast majority of the wells studied used leak-control technology that has yet to be adopted at many, if not most, oil and gas wells, while others were wells that produced very little gas and consequently even serious leaks would produce relatively small emissions – specifically, the authors noted, those wells had the potential to emit only 0.55% as much as an average well.

Although the study’s authors acknowledged that their measurements were by no means representative of the average gas well nationwide, they nonetheless chose to use that skewed data to estimate gas leaks nationwide.

The methodology that UT chose for making that estimate also has drawn heavy fire from others in the research community. The study, and its principal investigator David Allen, chose to use the same method that EPA used in its 1996 national inventory of methane emissions – and came to a nearly identical conclusion.

That 1996 methodology has been widely criticized, in part because it leaves a large margin of error. And with so much on the line, a large margin of error can obscure vital information. At its high end, the UT estimate of 1157 Gigagrams is roughly 1.5 times its low-end estimate, 757 Gigagrams.

The 1996 methodology uses an unreliably small percentage of wells to make sweeping predictions, researchers have complained. Like the 1996 study, which tested 0.14 percent of wells nationwide, the UT study relied on a tiny sample set of 0.11% of wells nationwide – in other words, less than 500 out of a half million wells.

“This is a grossly inadequate sampling, given that the United States is now home to more than half a million active natural gas wells — and more and more every day; 25,000 new wells were drilled last year alone,” wrote Dr. Alison Stine, a research fellow at Ohio University.

As in the 1996 study, key uncertainties were left unaccounted for. The UT authors wrote that they simply ignored uncertainties surrounding “national counts of wells or equipment and the issue of whether the companies that provided sampling sites are representative of the national population.”

“Unfortunately, these are exactly the most important parameters on which to base a truly representative, nation-wide assessment,” Physicians Scientists & Engineers for Healthy Energy wrote in their assessment of the UT research.

The wells tested by UT belonged to major oil and gas companies like Shell and Chevron – and those same companies have long argued that they have a better environmental track record than the little guys.

In addition, the UT study leaves out key information, researchers charge. The paper itself does not even indicate what type of formation was being drilled at each site – meaning that it’s not clear whether these wells are shale gas wells, which have been at the heart of the onshore drilling boom, or other types of fracked wells, like coal bed methane or tight sand – and industry data shows that there can be a 10-fold difference between the varying types of wells.

Most importantly, the UT research completely ignores conflicting estimates in peer-reviewed literature. A string of studies have found vastly higher leaks than UT’s estimated 0.42 percent leak rate would support. Field testing by the National Oceanic and Atmospheric Administration reported cumulative leaks as high as 12 percent of production in Colorado and 17 percent in the Los Angeles basin.

No effort to address these findings or to explain the discrepancy between the differing estimates was made by the UT researchers.

“It is disappointing that Allen and colleagues seem to have failed to employ basic scientific rules including transparent criteria for the selection of study sites to measure, sufficient sample sizes, and the attempt to place their results in the context of other scientific studies to date,” said Seth B. Shonkoff, Executive Director of Physicians Scientists & Engineers for Healthy Energy.

“This study falls short in its attempt to help answer questions about methane emissions from modern gas development beyond the small number of gas industry-selected wells where measures were taken.”

For its part, EDF has argued that the study does not prove that only small amounts of natural gas are currently leaking, especially because the wells studied nearly all used the leak control technology, called “green completions,” saying that the study merely shows the industry can control 99 percent of methane emissions during the completion stage – not that it does control them.

They argue that many wells will be required to adopt the technique in 2015 when new regulations go into effect, and so understanding its effectiveness is important.

Others have said that something doesn’t add up about the fact that the 1996 study and the UT study happen to reach nearly identical conclusions – despite intervening research that shows the true number is likely much higher and the fact that there was no large-scale shale gas development in 1996. They charge that the coincidence could be because both suffer from the same flaws.

“To allege that emissions have not changed since 1996 is absurd for at least two reasons,” said Prof. Ingraffea.

“EDF and industry cannot say, as they are now boasting, that new technologies have led to reduced emissions and then say they are the same now as in 1996; and there was no large scale commercial shale gas development in 1996, and the present issue and their study is all about shale gas development.”

Photo credit: Detail of an oil-refinery plant, via Shutterstock.