Replication is not validation

Sometimes you run an experiment right but the result is still wrong

A few months ago we came across a 2019 paper that claimed a magnetic field combined with an acoustic field hyperpolarized cells. This was exciting for us because one of our goals is to control the voltage of cells - and here was a publicly reported claim of exactly that having been achieved, in the hard direction, with field properties that would supposedly translate directly to working in deep human tissue.

So we adapted their setup to real-time imaging. Static magnetic fields, acoustic fields, human epithelial kidney cells measured with DiBAC4(3). DiBAC4(3) is a dye that when added to cells allows you to track changes in membrane potential by calibrating the fluorescent intensity directly to membrane potential. Dimmer signal = more hyperpolarized. Brighter signal = more depolarized.

We ran it. Signal dimmed. Hyperpolarization - yay!

Except this is not where we set the bar for confidence in an in-vitro result in order to translate it.

One of the reasons we wanted to reproduce this experiment at all is because of the so-called “reproducibility crisis” in science, where a large number of reported results don’t replicate in other labs. Private companies across industries report that less than 25% of public studies validate well enough to justify continuing research and systematic efforts to understand the biology landscape specifically pin this number closer to <15%.

Some would say this may be caused in part by other labs not using the right controls relative to the original set-up. I say in return that if the results are overly reliant on a specific set-up, then this is lame science because it probably won’t translate anyways.

I talked to a peer once running a biotech who was flabbergasted that an exciting experimental result originating in the southeastern US didn’t replicate in Boston. After months of head scratching and testing for answers, the cause was apparently atmospheric pressure differences between the two cities.

If an experiment doesn’t replicate and you write the original scientist asking why and after some back-and-forth they say OOOOHHHHH, now I see the problem: you bought this chemical from stock #121 when my chemical was from stock #89, you’re going to need to go find some of that chemical from stock #89 for this to be successful...this is lame science.

Furthermore, if the results are not robust enough to reproduce across different batches of the same base molecule, or even less stringently, across different brands of the same cell media, this is not a robust enough result to be hanging your hat on to make external claims about the value its findings may create when translated. I’d go so far as to say it shouldn’t have even been published if this was known to be the case at the time of writing the manuscript.

So anyways, when we saw that we got what we expected on the first try, we were excited because that’s what you’d hope to be the case and usually isn’t. Despite this we were not confident enough in this one reported success to take the paper’s claims of cellular hyperpolarization at face value. The immediate next thing we did was translate the results from human epithelial kidney cells to primary human dermal fibroblasts.

And... we again saw the same results! Yay x2. At least across two cell types, not to mention in a primary vs. an immortalized cell line¹, the reported findings replicated.

This is where the wheat gets separated from the chaff.

Replication across cell types wasn’t enough to make us confident. The next step was to pause and think about what was physically happening inside the cell.

DiBAC4(3) is a Nernstian dye. It partitions across the membrane according to voltage. If the cell truly hyperpolarizes, the dye redistributes out and the signal dims. But that’s not the only reason dye leaves a cell. If the membrane gets damaged even slightly, the dye leaks out too. And we were hitting these cells with an acoustic field. Sonoporation was an alternative explanation for everything we were seeing. Sonoporation is the formation of transient pores in the membrane of a cell from acoustic cavitation.

We needed a voltage readout that was robust against membrane disruption. We moved to a cell line we have on hand expressing a genetically encoded voltage indicator. Unlike a free-floating dye, a genetically encoded indicator is a protein expressed directly in the membrane. It doesn’t redistribute or leak. It reports what’s actually happening at the voltage level regardless of whether the membrane is intact.

So, we ran it with a more accurate sensor, and... nope. Not seeing hyperpolarization.

In fact we were seeing the opposite: depolarization. Unsurprising given that depolarization is a default stress response in cells.

A lesson here is that understanding how your reporting method works matters.

So, this tells us what actually happened: the acoustic field was disrupting the cell membrane by punching small holes in it. The DiBAC4(3) leaked out through the damage, and the signal dimmed, not because the cells hyperpolarized, but because the reporter escaped.

The data looked right but the biology was wrong. The ability to determine when data is in sync with biology and when it is not is on the scientist. No assay will tell you this. No p-value will flag it. It requires understanding what your tools are physically doing at the level of the system you're measuring and having the intellectual honesty to question a result that confirms your hypothesis just as hard as one that doesn't.

The paper we replicated passed peer review and has been cited a couple of times. We weren’t super bullish on this mechanism going in, but the risk of trying in parallel to the core mechanisms we’re hanging our hat on was low and so we did.

If you’re interested in learning more about what we’re working on, or joining our team, check out 3 newly open positions here.

We’re hiring for integrity, mission alignment and competence - in that order.

-Benjamin Anderson

Follow me on X.

1. Primary cells are directly derived from living tissue with a finite lifespan which give them a higher physiological relevance, while immortalized cell lines are modified to proliferate indefinitely which makes them easier to use but gives you less relevant data.