The real problem of irreproducible data

               

You have to know what is real before you know what is relevant. In science we trust that published means “real”. We may debate the impacts of different conditions and ooze skepticism over significance measurements and choose, sometimes less than tactfully, to object to an author’s conclusions, but overall we trust the data are real for the experimental conditions and that repetition of the experiment under the same conditions would yield the same outcome.

However, in a Catch 22 the data on reproducibility disagree.

In a collection of articles in the open source Nature special issue on data reproducibility, the most optimistic studies show 25 percent reproducibility of academic research. One article from Glenn Begley and Lee Ellis led to the development of Begley’s Six Rule for Reproducibility, which include blinded studies, full disclosure of all results, and reagent validation. Based on these rules, the optimistic 25 percent of reproducible studies are probably coming from far less than 25 percent of total academic labs. I know very few researchers that can honestly read all six rules without a little shame. 

On July 9th, Oxbridge Biotech Roundtable (OBR) hosted a panel discussion on this issue of data reproducibility. The panel included Dr Liz Silva, the MIND Program Manager at UCSF and the former Senior Editor at PLoS ONE; Dr Tim Gardner, founder of Riffyn; Dr William Gunn, Head of Academic Outreach at Mendeley; and Dr Corey Goodman, a partner with venBio.

Goodman opened the conversation with the first question he asks of founders pitching their science to him: “who has reproduced this?” The rest of the discussion focused on the problems, like an increasing rate of article retraction, and potential causes, such as intense pressure to publish, funding shortages, and increased oversight. Somewhat frustratingly, very few solutions were offered.  Gardner made a compelling argument for an incentivized approach to solving the problem. He suggested that two factors hold back scientists from solving the reproducibility issue: our culture and our lack of tools. We operate in “a culture of noise” in biology and accept that irreproducibility is unavoidable. However, if we employed adequate documentation tools similar to other industries it could greatly diminish the issue. Through incentives, like lower costs and timesavings by eliminating unnecessary, faulty repetitions, we can develop and adopt new tools and change the culture. (Gardner discusses this more in the Podcast referenced later.) When asked about automation as another tool in this solution, the panel felt it might be in the future, but currently there are other more pressing issues to address. (This could also be a reflection of the fear of the outdated bench scientist, but that is a topic for another day.)

Another potential, but more dogmatic approach, is a mandated NIH requirement for reproducibility standards. However, this reeks of additional strain on an already stretched system and more delays in publication, which will only compound the stresses that contribute to irreproducibility in the first place.

More adaptable and agile solutions are being developed through the Reproducibility Initiative. This partnership between Science Exchange, Mendeley, PLoS, and Figshare was briefly touched upon, but has the potential to revolutionize the way research is conducted as we incorporate tools, such as those being developed by Riffyn, and commercial replication services become cheaper and more convenient.

Overall this is a discussion that needs to continue and it requires total engagement from the entire scientific community. There is no clear solution but isn’t this the kind of problem we live for?

(To solicit more thoughts on solutions and the topic in general from two of the panelists, Silva and Gardner, I was asked to conduct an impromptu interview for a Podcast. By impromptu, I mean this interview was conducted on about five minutes notice.  I even forget Liz’s last name!  Anyway, it is clear I am not ready for the evening news.  As soon as the Podcast is up I will add the link here, so stay tuned!)

Citizen Science brings humanity and the microbiome to the world

                                               

                                              Badge from Beware Comics

Outsourcing is the reality of modern science. I’ve written posts on contract research organizations (CROs) and the pharmaceutical industry’s move to innovation hubs, but these are dry, corporate initiatives. They are important to our understanding of the business of science today, but where’s the humanity in a CRO? Where’s the joie de vivre in an innovation hub? 

Citizen science is the beating heart of science today. It’s inclusive outsourcing.  

No field of biology has embraced citizen science as enthusiastically as microbiome research, and the reception from the world community has been reciprocally enthusiastic. The success of projects like uBiome and American Gut illustrate the power of community involvement. Both the commercial uBiome and academic American Gut took universal engagement another step to additionally address the research-killing paucity of funding by crowdsourcing.

Give us your fecal samples, your skin swabs, your mouth swabs, yearning to be sequenced

uBiome raised over $300,000 on Indiegogo offering microbiome sequencing services similar to 23andMe in exchange for donations starting at $79. In addition to the monetary donation, they gave participants the option to relinquish ownership of their anonymous sample and results for development of population statistics and future research. After Indiegogo, uBiome continues to offer sequencing commercially with the same option to transfer ownership. They now have control over thousands of samples and a treasure of data that make scientists like me salivate. Unfortunately, we need to swallow and wipe our chins since uBiome is currently keeping their data close.

American Gut is the largest citizen science campaign to date. They have raised over $500,000 in one year on FundRazr and have over 3,000 fully sequenced samples. Working off the same principles as uBiome, they offer contribution levels starting at $99 and going up to thousands. Currently the highest donation has been $3,750 for a Functional Feces characterization.

                

Examples of American Gut donors, part of their “All the cool kids are doing it” approach (from left: author Michael Pollan, ultramarathoner Dean Karnazes, and Mrs. USA 2011 Shannon Ford)

Unlike uBiome, American Gut is making their data publically available through the open-source Earth Microbiome Project and through the European Bioinformatics Institute (EBI). They plan to publish their findings in several papers and hope other researchers apply these data ad libitum

Biased data or no data?

Crowdsourced microbiome initiatives are facing the same criticism of biased data as other crowdsourcing campaigns. Recently Steven Novella reviewed CureCrowd for Science-based Medicine (ultimately concluding it is too biased) and the Institute of Electrical and Electronics Engineers (IEEE) published “From Conservation to Crowdsourcing” in 2011, which touches on the bias issue. It is true, citizen science data are biased to individuals who can afford to contribute financially, are aware of these initiatives, and are motivated to join in citizen science. However, we now have data where we had none before; data that did not rely on tax dollars or single-source private benefactors to generate; both routes that empirically take significantly longer and are lower yield than commercial efforts (e.g. NIH Human Genome Project vs. Celera) or now crowdsourcing (e.g. NIH Human Microbiome Project vs. American Gut). As a side note, according to American Gut’s website, it appears they are closely linked to the Human Microbiome Project.

As long as researchers acknowledge the bias, are committed to addressing the issue, and accept that their findings may require reassessment as less biased cohorts become available, then this is not an end game issue.

The larger issue is the proper application and analysis of all these heaps of data being shared. Elizabeth Beam’s Harvard Science Review article, “DATA: The bigger the better? A survey of analytical traps and tricks,” concisely summarizes this issue.  

Engagement beyond money and samples

The appeal of the microbiome to the general population goes beyond crowdsourcing. Manipulating your microbiota is all the rage and DIY projects abound on the internet. There are recipes for probiotic yogurts and other foods, and plenty of diets to alter and optimize your gut bacteria. However, the most impressive is the DIY fecal transplant on The Power of Poop. I hope never to need to self-administer this procedure, but I’m glad to know there is a step-by-step guide openly available should the need arise.

               

               Fecal transplant shopping list from The Power of Poop

 This DIY prevalence circles back to the humanity of citizen science and the positive implications of investment by the greater community in research. Crowdsourcing is a successful, if not perfect, route to doing valuable science while educating people and removing the image of scientists behind locked doors, hunched secretively over arcane experiments.

Citizen science is the proactive science of today and tomorrow. 

BayBio’s F.A.S.T. pitch showcase puts five companies on base

(My apologies, I tried not to carry on the baseball pun but it was a soft pitch.)

               

On Thursday June 12th, UCSF hosted the BayBio Fellows All-Star Team (FAST) Advisory Program final pitch showcase. The FAST Advisory Program provides eight weeks of intensive coaching on business model development, product development planning, and creating a viable commercialization strategy. The five teams of the program were polished and energetic as they presented to a room of biotech investors, advisors, potential future team members, and program alumni, like XCell Biosciences, who were finalists in the recent Oxbridge Biotech Roundtable OneStart Americas competition.

The first pitch from Colleen Cutcliffe of Whole Biome described their Complete Biome TestTM as a step to answering, “What does it mean to bring the microbiome into healthy balance?” This team has already established impressive collaborations with the Mayo Clinic to identify indicators of preterm labor and AOBiome to advance their ammonia oxidizing bacterial products. Cutcliffe kept the audience engaged by sharing Whole Biome’s “naïve entrepreneur” story (every start up has at least one) where the first check was in the mail but they had no where to put it. Their realization, “We need a bank account,” seems so obvious but it exemplifies how the basics get overlooked when consumed by the rush of innovation, partnering, and fund raising.

Cutcliffe envisions a different approach to microbiome modulation than my favorite OneStart finalists, Symbiotic Health (blog on the finals here). Symbiotic Health’s BactoCaps combat Clostridium difficile infections by dosing the patient with alternative, beneficial bacterial strains. Whole Biome will forgo this growing market of bacteria-filled capsules or “poop pills” in favor of more target therapeutics in the future. (For more on the “poop pill” phenomemon see Sarah Zhang’s Nature article here.)

               

Though not related to bacteria, artist Tobias Wong created gold poop pills for the INDULGENCE art collection to add a little shimmer to your bowel movements for $425.

Kelly Gardner pitched Zephyrus Biosciences’ single-cell western blotting technology. Single-cell methods are rapidly growing and improving, as reflected in single-cell sequencing being named Nature’s Method of the Year in 2013. Fluidigm is dominating the single-cell market and will probably show interest in Zephyrus soon if they have not already. This technology is an end-of-the-line assay since the cells have to die to be blotted, but I would assume stem cell cancer therapeutics companies, like Stem CentRx, that require single-cell resolution tumor typing will provide an eager market for this technology. 

               

               Zephyrus Biosciences team (image from Sky Deck, Berkeley)

Applied Molecular Transport, pitched by Tahir Mahmood, is adapting an alternative cholera-causing toxin identified in Peru to transport drugs into the cells of the gut. This allows for the conversion of injectable drugs into an oral format. Currently they are focused on inflammatory bowel disease, but could this be applicable to antibiotics, especially those to treat serious Gram-negative infections, like Achaogen’s plazomicin, thus shortening expensive hospital stays? Mahmood admits that ensuring “stability from mouth to gut cell wall” is a challenge but one they are actively addressing.

Kate Garrett and Ciel Medical are combating ventilator-associated pneumonia (VAP) with a suction catheter device and ventilator placement tool. When introducing Kate, Stephanie Diaz, President and CEO of Vida Strategic Partners, praised her clarity and ingenuity by reflecting, “Why can’t I think of things like that?” This ingenuity comes across in their MacGyver-style testing of these devices at the largest nursing conference in the US where they proved that 99 percent of nurses could place the suction catheter properly, but none could properly place the current alternative. VAP is  a leading cause of hospital related death and these devices will start saving lives immediately. 

The pitches closed with Adam Mendelsohn of Nano Precision Medical. The team members, described by Jo Whitehouse, CEO of JumpStart BioDevelopment, as rare, entrepreneurial “good listeners”, have developed the NanoPortalTM. This device is an implantable, all-titanium capsule for long-term, constant-rate delivery of therapeutic molecules. Currently the focus is on delivery of exenatide for Type 2 diabetes mellitus. Mendelsohn shared some eye candy graphics of the delivery dynamics that are now on their website, which went live two days before the event. Nano Precision is leaving FAST with more than extensive training: one of their program advisors, Wouter Roorda, introduced himself by telling the team all the reasons why they were doomed and is now righting all those problems as Vice President of Pharmaceutical Product Development.  

               

Static image of videos of the Goldilocks Effect. Visit the website to watch the video and trace the curve. It’s worth it! 

The night was a positive one, except for what has become the frustrating norm at life science and biotech events: the repetitive comments about angst-riddled postdocs and the constant internal struggle whether to stick it out or abandon science.  When introducing Zephyrus Biosciences, Jenny Rooke of Fidelity Biosciences remarked that every day she sees disenchanted and lost graduate students and postdocs. Every event continues to reinforce the obvious deterioration of the traditional system.

However, entrepreneurship is offering alternative options to challenge and leverage PhD scientists as exemplified by FAST and similar programs. There is still hope for science.

In addition to BayBio, the FAST Advisory Program is supported by Abbott, Pfizer, Bayer HealthCare, Gilead, and VWR

The opaque problem of medical pricing transparency

It’s easy to see the benefits of our free-market, transparent economy:  it keeps our iPhones affordable and WebMD always at our fingertips to tell us that our headache could be from stress, or a brain tumor. As you race for an MRI, you probably won’t consider the cost; and it doesn’t matter because that information is careful concealed until you get a bill for more than an iPhone…or four. Medical pricing in the US is an opaque and secretive system that leads to Americans spending far more on health care with fewer returns on investment than other Western nations. San Francisco-based Stroll Health is developing a disruptive app to make finding the price of procedures as easy as, and far more accurate than, self-diagnosis.

               International_Healthcare_Costs_Per_Capita_2010.gif

               Per capita spending according to OECD Health Statistics 2013 

According to the Centers for Disease Control and Prevention, in 2010 the US spent $8,402 per capita on healthcare, which totals 17.9% of GDP. Compared to other developed nations, we spend almost twice as much per capita with no improvement in outcomes. A Washington Post blog on March 26, 2013 by Ezra Klein highlights national differences through 21 graphs of outrageous pricing and spending on medical procedures in the US.

              International_Comparison_GDP_2010_21graphs.jpg

              From 21 graphs that show America’s health-care prices are ludicrous” 

              OECD.png

              Life expectancy in the US (blue) compared to the rest of the world according to OECD Health Statistics 2013

American health care is a sluggish and entangled behemoth. We are currently facing the daunting task of not only identifying the causes of the dysfunction, but also implementing effective solutions that directly address the underlying issues. A November 2013 article in The Journal of the American Medical Association (JAMA), found that price increases account for 91% of the increased cost of health care, which far outpaces the costs associated with an aging population. The root cause of this dysfunction is a fragmented healthcare system in which parties act as lone wolves in segmented areas of the system. The lack of coordinated efforts to manage costs or fight diseases, coupled with lagging information technologies, minimizes the role of patient as consumer and further exacerbates the dysfunction. Author David Matheson stated for Russ Britt’s Market Watch article, “The extreme fragmentation makes it hard for anyone to be held accountable for health care.”

In the same issue of JAMA, Dr Uwe Reinhardt expanded on the culpability of fragmentation for sky-high health care costs. Fragmentation results from a private health insurance system that does not allow individual insurers to influence market pricing as in all other capitalist markets. With prices secretly negotiated in the private sector, health care providers do not have to provide the costs of their services to patients and doctors.

Several states have enacted policies in response to the federal government’s resistance to address the issue of health care pricing. While these policies clearly identify a pressing issue, most are too weak and have proven ineffective. Case in point, New Hampshire is one of two states, along with Massachusetts, cited in the Health Care Incentives Improvement Institute’s (HCI3) Report Card on State Price Transparency Laws as having adequate price transparency laws. However, if you visit New Hampshire’s HealthCost website, you find, “a failure in the data reporting process has made updating price estimates at the moment impossible.” The delay could last months.

              

               New Hampshire’s HealthCost website has yet to provide costs

What is the solution to reining in runaway health care costs in America? Federal intervention represents a coordinated effort, but it risks bureaucratic creep. The University of California, San Francisco held the first “The Future of Healthcare Transparency in California: A Multi-stakeholder Summit” last October to discuss the issue with key opinion leaders in the field. The summit emphasized encouraging voluntary action on the part of health care providers to provide pricing data and increase access to claims data. This appeal, predicated on acting together for the greater good, naively neglects the motivations of individual players within the health care complex.

Alternatively, disruptive information technology in the form of publicly accessible websites and mobile apps provide a more flexible and pragmatic solution. Nimble, innovative startups can increase price transparency by putting the consumer back into the market. To asses the growing number of available tools, HCI3 published their list of comprehensive specifications for price transparency tools based on the capabilities and drawbacks of existing applications.

Dr Reinhardt included Healthcare Blue Book and Castlight Health in his article on disruptive technologies. While Healthcare Blue book is an excellent resource, it provides “fair prices” for services based on your zip code rather than the actual prices of specific providers.  Castlight Health is more specific, but it is only available through employers and for their health care plan. Additionally, neither of these tools are available as a mobile app. A quick search of the App Store for “health care prices/health care costs” and related permutations returns less than seven potential iPhone apps including FAIR Health’s Healthcare Cost Estimator and the Health Cost Estimator, which are both based on “fair pricing”. Welter by DynaHIT LLC is the only one that approaches price transparency. In its beta version, it provides the cash price for basic primary care appointments in Chicago. However, in reality their “data was generated through a cost survey and should be considered cost estimates”.

To effect change someone needs to fill this need and fill it fast.  Enter Stroll Health, an energetic  group ready to solve this gigantic problem. The San Francisco-based startup founded in 2013 by Jordan Epstein, Matt Maurer, and Drew Moxon, has created an iPad app that begins to put real prices for health services in the hands of doctors. They aim to enrich the doctor-patient relationship by allowing doctors to instantly identify the best service for the patient based on actual cost, quality, and location. “In our experience, we’ve seen that tools marketed directly to the patient are always underutilized,” said Epstein, “we believe that bringing the cost transparency discussion to the physician’s office is a big win for doctors and patients.” Doctors are being confronted more frequently with concerns about personal out-of-pocket costs for the services they order and are frustrated by their inability to answer. Unfortunately, costs outside of their offices are as much a mystery to doctors as patients.

              

              Stroll cofounders, Maurer and Epstein, with OBR volunteer, Laura Sasportas, at OneStart Bootcamp

In tandem with these disruptive technologies, high-deductible plans are growing. These plans make patients responsible for a higher percentage of their health care costs. By increasing the patient’s financial incentive to seek out lower priced and higher quality services the market pressures can correct the lopsided system. Employers are indirectly driving this as they opt to reduce their health care burden by moving to these high-deductable consumer-directed health plans (CDHPs). This 2007 debate provides a good primer on CDHPs. If the consumer is expected to contribute more, they should have access to pricing. Dr Nathan Handley, physician at UCSF, “While it certainly makes sense that the patients who are most interested in discussing cost would be the ones who clamor most for Stroll, so far in my experience nearly every non-HMO patient I’ve talked to wishes they were informed of these choices with their physicians and had the ability to be engaged meaningfully with their physician—they just don’t know this opportunity is available.”

This recent trend further shows the relevance of Steven Brill’s article for TIME magazine, Bitter Pill: Why Medical Bills are Killing Us, which conveys the impact of opaque pricing on the individual. Everyone gets sick and worrying about the potential and unpredictable financial burden of an illness is not conducive to getting well fast.

Stroll recently won the first Prebacked hackathon and competed as finalists in the OneStart Americas competition. The data in the beta version is drawn from samples from insurance companies and Medicare. Currently they are seeking additional partnerships with insurance companies and physicians willing to conduct pilot tests.   

(Thank you to Charlie Carbery and Jordan Epstein for their comments and insights.)

The right people in the room for World Malaria Day

                    image

UC Berkeley hosted Zagaya’s third Bay Area World Malaria Day Symposium on April 25th.  All the west coast powerhouses of malaria research were in attendance along with key contributors from across the US. Attendees and presentations spanned discovery research, industrial therapeutics production, public health, and more. The efficiency and effectiveness of the meeting was unbelievable and it was absolutely a day of putting the right people in the room.

The three talk sessions, broken down into research, technology, and implementation, consisted of a brief keynote and a few five-minute talks followed by a panel Q&A with the speakers. A Nerf dart gun enforced the five-minute limit for everyone: Joe DeRisiAdam Renslo, postdocs, and students. This egalitarian approach created a casual atmosphere of approachability. Plus, watching the poor student moderators grapple with the possibility of career suicide at enforcing the rules added a little human drama to the day.

In the area of drug development, Andrew Horitz’s talk on Amyris’s ability to produce artemisinin in yeast, an important advance for more affordable drug production, was met with high interest from the audience. In a related talk, Agnes Nabasirye spoke on JLM, International’s work with PATH to manage the overuse of artemisinin-based combination therapies (ACTs) and, “Improve malaria delivery systems,” through rapid diagnostic tests in high transmission areas.

In a novel twist on diagnostics, the malaria world is turning to mobile apps and gaming to crowdsource sample screening. Aydogan Ozcan’s group at UCLA have developed both an Android app and a web-based game in which users screen images of blood smears to determine if they are positive or negative for malaria. He offered a prize to any attendee who could beat the top score by the end of the conference; if anyone succeeded is unclear but many were trying. A word of caution, it can be addictive!

                   image

                    Gaming diagnostics: identify malaria infected cells

Another tech-focused presentation came from Johnny Gannon on M@RS, the Mosquito Awareness Reckoning System, and the winner of the first Diagnostics by Design: a Hack Day for Global Health. The core of M@RS is its mPod, which attracts and kills mosquitos. Gannon, a Berkeley MBA student, sought advice on improving the research utility of M@RS as well as on commercialization possibilities. He showed impressive composure and humor when inundated with questions and comments from a room full of scientists, all excited by this new tool. 

                              image

                              The mPod for M@RS

A research symposium is never complete without someone sharing their story of self-experimentation. The favorite of the day came from Genevieve Tauxe of UC Riverside who collected and analyzed her own “trail funk” to refine the methods necessary to research controlling mosquito behavior with odorants.

The day ended with a sobering reminder of what brought everyone together: the goal to alleviate suffering across the globe for the benefit of all. We were reminded of the UN’s eight Millennium Development Goals, include malaria eradication, and the international movement to achieve these goals. Perhaps before the end of the millennium there will no longer be a need for a World Malaria Day Symposium.

(Visit the new Malaria.com for more on research and trends in diagnostics.)

OBR OneStart Finals Gala

                           image

                           Resilience takes home the signed check

The Oxbridge Biotech Roundtable (OBR) and SR One hosted the finals gala for their OneStart Americas competition on May 22. The ten finalists, narrowed down from an impressive pool of 35  semifinalists, each gave their two-minute elevator pitch throughout the course of dinner at The City Club of San FranciscoResilience Pharmaceuticals, founded by Retsina Meyer and Veronica Weiner, took home the grand prize with their posttraumatic stress disorder (PTSD) preventative and will now be relocating from Boston to San Francisco as the newest members of QB3@953

                           image

                          Lively pre-dinner networking

Resilience’s story has that American Dream air about it, well a PhD biologist’s American  dream, a dream to see academic research change lives in years not decades. Started as a MIT thesis project by Meyer, the cofounders took their academic research out of the lab and are commercializing a preventative for PTSD based on manipulating levels of the “hunger hormone”, ghrelin. Boston Magazine published a synopsis of their work last year. As research funding shrinks and scientists are forced to evaluate the efficiency, efficacy, and applicability of their research more critically, Resilience represents the new paradigm and not an outlier.

My personal favorite teams of the night were Symbiotic Health of New York and MetaMixis of Vancouver. Gerard Honig presented on their BactoCap technology that exploits other commensal bacterial strains to cure Clostridium difficile infections. Cameron Strachan of MetaMixis shared their plans to replace chemical manufacturing processes with brilliantly engineered bacteria. Jens Echstein, President of SR One, praised MetaMixis beginning with, “Microbes are the best chemists.” Sitting together at dinner, Cameron enthusiastically shared some of MetaMixis’ current projects and since they will be relocating to Stanford this summer, hopefully I can keep up with their progress firsthand. 

              image

              Strachan pitches MetaMixis to the judges

Jordan Epstein of Stroll, the only IT team in the finals, made a strong case for their medical pricing transparency app. The problem they are tackling is an overwhelming and important one and their success could be hugely disruptive to healthcare in this country (blog post on medical pricing transparency coming soon). 

Many of the semifinalist teams came out to the event too and it was exciting to hear about the recent progress of EpiBiome and ViVita, two of the teams I met at the OneStart Bootcamp back in February. A OneStart win would have just been the icing on the cake for these young entrepreneurs.

It was an impressive night of inspiring innovation from biotech pioneers under 36 and these intrepid innovators are the ones to watch as they create the future.

                              image

         Amazing OBR volunteers like Henok Eyob and Jeroen Blokhuis made OneStart possible