ChatGPT and the Turing Test: Biblical Studies Version

My inventive colleague, Dr. Isaac Soon, recently offered a constructive counterpart to all the handwringing in academia about ChatGPT. Instead of bemoaning ChatGPT’s potential to mimic student papers for fear of widespread cheating, why not use ChatGPT as a teaching and research tool?

Isaac put up on his weblog a process he could supervise, as a teacher of the New Testament, to help students dive deeper and deeper into the study of a text.

With a series of iterative steps, students could get a general sense of the passage, an overview of the main critical question, help with the actual Greek text, and even a bibliography for further study.

ChatGPT as a graduate assistant! The whole exercise seemed almost too good to be true.

And it was.

Soon after he published his blog post, a scholarly friend of Isaac’s tipped him off. One of the sources adduced by ChatGPT was incorrect. Just page numbers, but still.

So Isaac, assiduous scholar that he is, submitted all the references to scrutiny. And he became dismayed.

Real authors—as notable as Paula Frederiksen and James D. G. Dunn—were cited, but with the wrong cities for the publisher cited, or the wrong publisher, or even simply fabricated articles in toto. The more Isaac dug, the more he found bogus material.

ChatGPT really was just mimicking human work. (Indeed, I concluded darkly that ChatGPT demonstrated that it truly was capable of writing student papers—but not in a good way.)

Isaac announced his finding in a rueful update to his blog post, pronouncing himself “idiotic.”

But Isaac Soon is no idiot. He is a young scholar with a master’s degree from Oxford and a Ph.D. from Durham. We turned down a fine candidate with a Princeton Seminary Ph.D. to hire Isaac Soon instead. Barely graduated from his doctoral program, Isaac has already published a handful of journal articles, including one in the Journal of Biblical Literature, and has a book contract with OUP.

And he was fooled.

So here’s to a clever idea for how to use ChatGPT as a research tool—except not yet, it seems. Here’s to a clever scholar who, with his network of learned friends, caught and fixed a public mistake.

And here’s to lying awake at night, still, worrying about ChatGPT after all.