Methods: Through an innovative approach, we construct ontology-based
knowledge graphs from authentic medical literature and AI-generated content.
Our goal is to distinguish factual information from unverified data. We
compiled two datasets: one from biomedical literature using a "human disease
and symptoms" query, and another generated by ChatGPT, simulating articles.
With these datasets (PubMed and ChatGPT), we curated 10 sets of 250 abstracts
each, selected randomly with a specific seed. Our method focuses on utilizing
disease ontology (DOID) and symptom ontology (SYMP) to build knowledge graphs,
robust mathematical models that facilitate unbiased comparisons. By employing
our fact-checking algorithms and network centrality metrics, we conducted GPT
disease-symptoms link analysis to quantify the accuracy of factual knowledge
amid noise, hypotheses, and significant findings.
Results: The findings obtained from the comparison of diverse ChatGPT
knowledge graphs with their PubMed counterparts revealed some interesting
observations. While PubMed knowledge graphs exhibit a wealth of disease-symptom
terms, it is surprising to observe that some ChatGPT graphs surpass them in the
number of connections. Furthermore, some GPT graphs are demonstrating supremacy
of the centrality scores, especially for the overlapping nodes. This striking
contrast indicates the untapped potential of knowledge that can be derived from
AI-generated content, awaiting verification. Out of all the graphs, the factual
link ratio between any two graphs reached its peak at 60%.
Conclusions: An intriguing insight from our findings was the striking number
of links among terms in the knowledge graph generated from ChatGPT datasets,
surpassing some of those in its PubMed counterpart. This early discovery has
prompted further investigation using universal network metrics to unveil the
new knowledge the links may hold.Comment: 7 Pages, 3 algorithms, 4 tables, and 7 figure