Hi Rowena,

I think I do get what you are saying but I still think there are things people can do to make sure that what they are seeing is really reproducible in a reasonable way. I think that we are all influenced by the strong culture to publish as quickly as possible that it is very difficult to see how much it influences our choices when considering even hypothetical scenarios.

In response to your example: examination of a material under TEM or SEM using a random selection of areas itself may not be statistically reliable, but independently preparing several samples of the material will improve the likelyhood that what you are seeing is reproducible significantly if, by random selection you see the same features on all samples. Of course that begs the question regarding how do we 'randomly' select areas of a sample and that is really up to the individual experimenter to justify as I know (having done a significant about of microscopy during my post-doc) that it is very easy to focus on the 'interesting' parts of a sample and forget to place them in context. It is no conincidence that modern microscopy methods, such as STM have been the subject of a few recent academic scandals regarding irreproducibility and incorrect interpretation of results.

Even with imaging techniques such as SEM & TEM, you shoud never really be dealing with individual samples when it comes to producing publishable-grade research, as a senior colleague in analytical chemistry said to me recently, "One data point is no data point".

In essence, my fundamental view is that we should, as a community, strive to achieve the highest standards of ethical conduct in research, even if that is challenging and time-consuming, because it strengthens our scientific legacy and makes us better scientists as a result. After all, isn't being the best we can be a worth goal?

Re: Rutherford's gold foil - I'm not sure I can see where the selectivity in data has taken place here - Rutherford's team monitored the distribution of all alpha particles and reported them all, it was the fact that only some were reflected that was the important conclusion here not that they just focussed on the reflected particles.

Hi Chris. I think you've misunderstood some of my points a little. For example, with the gold foil experiment, the point is that selecting which thing to focus on and only using data from that is commonly criticised now as not being reproducible. Let's say, on a TEM or SEM image the data gathered is only a fraction of the sample, and so deemed not enough, even with lots of repeats it might not be "average" or the selection process to get good resolution might mean certain similar areas are examined. You can't say everything should be repeated because the limits of the scope are difficult to determine - obviously repeating experiments as you did during your PhD is critical, but some experiments need a closer look to determine what is "good enough" or "a useful amount". Completely agree with your points on publication; the idea of making peer reviewers replicate experiments still seems ridiculous to me.

I think reproducibility is key and it should be part of the everyday culture of the research lab as it ensures quality and stops wasting more money and time after the fact when things cannot be reproduced (there is always a cost). The chemical community seems to get itself into constant trouble because experimental results are not verified properly prior to publication. So, I would really disagree wih your sentiment that reproducibility is 'optional' in science. During my PhD I was simply turned away at my supervisor's door if I did not have all the data to demonstrate that I had conducted any experiment in at least triplicate to demonstrate reproducibility. This was a good thing too as I personally experienced several 'false positives' that may have made it into the literature if I did not repeat the experiments before writing them up.

I agree that there is a lot of pressure on people to publish single positives and that needs to be addressed, both through challenging the method of governance within academic institutions regarding research ethics (what governance?) and also by ensuring that PhD students are taught, by example, about the right way to conduct experiments and validate results.

Regarding publication: In my view, manuscripts that do not provide enough evidence of reproducibility should be rejected during peer-review - this is a quick and easy way to ensure that all researchers do take the time and spend the money to make sure that the work they are doing is correct. It is not the responsibility of peer reviewers to repeat experiments, it is the responsibility of the author to do this and to be able to convince a peer review that it has been done correctly.

Regarding the seminal experiments you quote at the end of your post; you are indeed correct that Milikan appears to have been selective in which data he used to determine the value of 'e', this had the effect of reducing the statistical error from 2% to 0.5%, a minor but significant change. It may not have effected the results of his experiment, but I doubt such a great scientist would want part of his legacy to used as an example of how not to do science just because he wanted his results to be a bit 'prettier'.

However, I think you misunderstand the results from the gold foil experiment. The predominant model of atomic structure at that time (the 'plum pudding' model) led to the prediction that *all* alpha particles should be transmitted directly through the foil so that fact that Rutherford and his co-workers detected a portion of particles that appeared to have been reflected or scattered was highly surprising as it indicated that finer detailed structure was present within the atom (with some alpha particles interacting with this fine structure while others passed through), this wasn't 'data mining' at all but a genuine unexpected observation that led to the formulation of a new model of atomic structure.