If 20 cooks around the country used the identical recipe to prepare carrot cake, would each baker's creation taste equally delicious? With the multitude of flour, butter and oil brands from which a chef may choose, not to mention performance differences between electric and gas ovens, the answer is probably no.
Similar variability problems occur when researchers in different laboratories must perform and compare identical experiments — with consequences that can be much more serious than a less-than-delectable dessert. However, Human Biology Division investigators and colleagues around the country have discovered that when scientists make their "recipes" a bit more stringent — and their "palates" a bit less discriminating — experiments among individual labs can be highly reproducible.
To reach this conclusion, Dr. Helmut Zarbl and colleagues systematically identified all of the variables that contribute to individual laboratory differences in results from a type of large-scale experiment called a global gene-expression study. The studies permit scientists to simultaneously scan all of the genes in a given tissue to test how they react under different conditions, such as when exposed to a suspected cancer-causing chemical. Each gene-expression experiment generates thousands of data points and requires sensitive equipment and many multi-step procedures.
By identifying potential sources of variability and testing strategies to improve standardization, the researchers were able to dramatically boost the reproducibility of the experiments across labs. The achievement will make it easier for teams of scientists to collaborate on research projects that attempt to understand why and how cancer develops, which will lead to advances in disease prevention, detection and treatment. The finding are published in the May issue of Nature Methods.
Members of the Toxicogenomics Research Consortium, a collaboration funded by the National Institutes of Environmental Health (NIEHS), conducted the analysis, which is the most comprehensive of its kind. The consortium's goal is to transform the field of toxicology — which attempts to explain the nature, effects and detection of poisons and treatment of poisoning — into a predictive science. Zarbl serves as the principal investigator of the Hutchinson Center/University of Washington site of the consortium.
The rationale for the project is that upon exposure to environmental toxins or agents that adversely affect health, certain sets of genes will be turned on or off. Each toxin or other stress would trigger a specific genetic signature or fingerprint that is characteristic of each compound, the amount and duration of exposure, and the individual's own unique genetic blueprint. Researchers involved in the project hope that these signatures can eventually be used to predict the toxicity of chemicals, provide insights into how they cause damage, provide better methods to measure people's exposure to potentially harmful agents and more accurately predict risk from exposure. For example, the signatures might be used to predict an individual's risk of developing diseases such as cancer following exposures to environmental agents like tobacco or asbestos.
"When the NIEHS set up the consortium, one of the goals was to create a database that would be populated with data from experiments looking at the effects of potentially toxic exposures on gene expression," Zarbl said. "However, for such a database to be useful in predicting toxicity, it must be populated with data that are reproducible across labs and experimental platforms. The first question we had was, if every individual laboratory in the consortium was doing its own experiments, would the collective results be reproducible and meaningful?"
To address this, seven individual laboratories performed global gene-expression experiments on two different types of tissue samples using techniques, tools, and data-entry and analysis methods of their choosing. All of the experiments rely on tools called DNA microarrays — also known as gene chips — which are small glass slides spotted with bits of DNA from hundreds or even thousands of different genes. Some types of microarrays are commercially available, while others that are used for more specialized experiments are sometimes manufactured by individual research institutions. For example, the Center's Genomics shared resource has produced genomic microarray chips for a variety of species including humans, as well as more specialized chips for studies on human breast and prostate cancers.
The researchers found that within a single lab, there was high reproducibility among experiments that involved similar types of microarray chips. Reproducibility of experiments that involved different types of microarray chips — even when performed in the same lab — was generally poor, as was the reproducibility of experiments of any kind when compared among different labs. Consistency was greatest among experiments that relied on commercially available chips, although Zarbl noted that arrays produced at the Center performed quite well.
After examining all sources of variability, the researchers found they could markedly improve reproducibility among laboratories not only by standardizing the choice of microarray chip, but also by specifying common methods for preparing and processing samples, and for extracting and analyzing the data.
In particular, Zarbl said that reproducibility was greatly enhanced if data were analyzed in a more general fashion.
The investigative phase
"When we compared individual genes from one experiment to another, we saw a lot of variability," he said. "But if instead, we look at overall patterns by focusing on gene pathways — groups of genes that contribute to the same metabolic function — we see very high reproducibility. This holds true even when we look at experiments that involved noncommercial arrays."
Based on what they have learned, Zarbl said that the Toxicogenomics Research Consortium members are now using a standardized approach on the true investigative phase of their work. This will involve examination of the effects of specific chemical agents in human tissues and in a variety of model organisms, including strains of mice and rats that differ in their susceptibility to cancer.