Study Tests AI Refinement of Chinese Texts

A study by Bryce P. Towne published in Discover Artificial Intelligence in 2026 tested whether authors’ target-informed guidance to language models changed how readers aligned with intended meanings or understood Chinese texts. Two controlled experiments compared original texts with versions refined by GPT-4 through ChatGPT and Claude 3 Opus; Study 1 also tested Gemini Advanced. Models received author-supplied target ratings or answer keys that were also used to score outcomes, so the intervention was criterion-linked rather than ordinary copyediting. Participants recruited through Credamo were randomly assigned by the platform to text-version conditions.

In Study 1, 572 participants were assessed for exact-match alignment with authors’ emotional-expression ratings and answers to 15 author-keyed comprehension questions. GPT-4 refinement increased emotional-expression alignment versus original texts (p = .002, signed r = .18). Gemini Advanced reduced alignment at the unadjusted level (p = .010, signed r = −.15), but did not meet the Bonferroni-adjusted threshold of 0.0083; Claude 3 Opus did not differ from original texts. No comprehension differences were detected. Recruitment was extended after initial results were inspected, so Study 1’s inferential analyses are exploratory.

Study 2 included 420 participants and measured exact-match alignment on six scholarly attributes conveyed in the text, plus author-keyed comprehension. Neither GPT-4 nor Claude 3 Opus produced detectable differences in either outcome. The authors say the results depend on the fixed materials, participant samples, prompts, single-output procedure, criterion-linked guidance and scoring rules; they do not establish that all authorial meaning is preserved or that scholarly quality is objective.

A study by Bryce P. Towne published in Discover Artificial Intelligence in 2026 tested whether authors’ target-informed guidance to language models changed how readers aligned with intended meanings or understood Chinese texts. Two controlled experiments compared original texts with versions refined by GPT-4 through ChatGPT and Claude 3 Opus; Study 1 also tested Gemini Advanced. Models received author-supplied target ratings or answer keys that were also used to score outcomes, so the intervention was criterion-linked rather than ordinary copyediting. Participants recruited through Credamo were randomly assigned by the platform to text-version conditions.

In Study 1, 572 participants were assessed for exact-match alignment with authors’ emotional-expression ratings and answers to 15 author-keyed comprehension questions. GPT-4 refinement increased emotional-expression alignment versus original texts (p = .002, signed r = .18). Gemini Advanced reduced alignment at the unadjusted level (p = .010, signed r = −.15), but did not meet the Bonferroni-adjusted threshold of 0.0083; Claude 3 Opus did not differ from original texts. No comprehension differences were detected. Recruitment was extended after initial results were inspected, so Study 1’s inferential analyses are exploratory.

Study 2 included 420 participants and measured exact-match alignment on six scholarly attributes conveyed in the text, plus author-keyed comprehension. Neither GPT-4 nor Claude 3 Opus produced detectable differences in either outcome. The authors say the results depend on the fixed materials, participant samples, prompts, single-output procedure, criterion-linked guidance and scoring rules; they do not establish that all authorial meaning is preserved or that scholarly quality is objective.