Praedicat's general acceptance (GA) score measures the degree to which the peer-reviewed scientific literature supports the hypothesis that exposure to a chemical can cause bodily injury. This score is derived from an aggregation of metadata (e.g., study outcome, study type) extracted from articles reporting the results of scientific investigations into the hypothesis. The metadata shown in ChemMeta are:
- Relevance: Whether the article is investigating the causal hypothesis of harm in question
- Study outcome: Whether the article accepts or rejects the stated hypothesis
- Study subject: Whether the study employs a cell, animal, or human experimental model
We compute "confidence scores" for each type of metadata we extract. The gold standard is extraction via human analyst - we are fully confident that our trained analysts will extract the correct metadata from an article. We set "relevance" (1), "outcome confidence" (2), and "subject confidence" (3) to 100 when the corresponding metadata are extracted by human analysts. Our machine-learning algorithms seek to replicate the metadata that human analysts would extract for the same article. Algorithmic results, of course, are inherently probabilistic; in this context confidence scores indicate how certain our algorithms are that the extracted metadata reflect the true metadata. We set the boundary for acceptance at 50 (i.e., there's at least a 50 percent chance that the algorithm classified the metadata correctly). As the scores increase from 50, the confidence in the extracted metadata increases.
Article influence scores
We also now report an article "influence score" (4). The influence score indicates how influential the article is to the GA score to which it contributes compared to the most influential article possible. The most influential studies are human meta-studies published in the most highly cited journals (e.g., Nature, The New England Journal of Medicine). In those cases, the reported influence score is 100. A study with an influence score of, say, 75 (perhaps a human cohort study published in a less frequently cited journal) has 75% as much influence on the hypothesis GA score as the most influential study. Users can sort and filter contributing articles by influence and confidence scores to zero-in on the most important studies within a given literature (5).
GA confidence scores
With this release, users will also now see GA confidence scores (6) reported in the chemical and chemical-harm lists that indicate the overall level of confidence in the reported GA scores given the article-level relevance and confidence scores described above. The GA confidence score aggregates article-level confidence weighted by influence scores. Low confidence studies with low influence scores will dampen the aggregate GA confidence score less than will low confidence studies with high influence scores. A GA confidence score of 100 indicates that the underlying article metadata was extracted entirely by human analysts.
ChemMeta inclusion criteria
Our machine-scale approach to extracting article metadata and scoring scientific literatures is firmly guided by human analysts (e.g., biochemists, epidemiologists, bioengineers, environmental scientists). As such, we require any chemical-harm hypothesis reported in ChemMeta to be validated for relevance by a human analyst. A validated hypothesis must be supported by two or more human-coded relevant in vitro studies or at least one human-coded relevant animal or human study. These inclusion criteria have resulted in omitting 129 chemicals that were previously available in ChemMeta. Those omitted chemicals will continue to be monitored for inclusion in future releases. This release also adds 161 chemicals now determined to have relevant hypotheses of harm.