5 research outputs found
Detecting The Corruption Of Online Questionnaires By Artificial Intelligence
Online questionnaires that use crowd-sourcing platforms to recruit
participants have become commonplace, due to their ease of use and low costs.
Artificial Intelligence (AI) based Large Language Models (LLM) have made it
easy for bad actors to automatically fill in online forms, including generating
meaningful text for open-ended tasks. These technological advances threaten the
data quality for studies that use online questionnaires. This study tested if
text generated by an AI for the purpose of an online study can be detected by
both humans and automatic AI detection systems. While humans were able to
correctly identify authorship of text above chance level (76 percent accuracy),
their performance was still below what would be required to ensure satisfactory
data quality. Researchers currently have to rely on the disinterest of bad
actors to successfully use open-ended responses as a useful tool for ensuring
data quality. Automatic AI detection systems are currently completely unusable.
If AIs become too prevalent in submitting responses then the costs associated
with detecting fraudulent submissions will outweigh the benefits of online
questionnaires. Individual attention checks will no longer be a sufficient tool
to ensure good data quality. This problem can only be systematically addressed
by crowd-sourcing platforms. They cannot rely on automatic AI detection systems
and it is unclear how they can ensure data quality for their paying clients
Detecting the corruption of online questionnaires by artificial intelligence
Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify the authorship of such text above chance level (76% accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on a lack of interest among bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AI submissions of responses become too prevalent, then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowdsourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients
Detecting The Corruption Of Online Questionnaires By Artificial Intelligence
Online questionnaires that use crowd-sourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial Intelligence (AI) based Large Language Models (LLM) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested if text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify authorship of text above chance level (76 percent accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on the disinterest of bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AIs become too prevalent in submitting responses then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowd-sourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients
DataSheet1_Detecting the corruption of online questionnaires by artificial intelligence.pdf
Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify the authorship of such text above chance level (76% accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on a lack of interest among bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AI submissions of responses become too prevalent, then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowdsourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients.</p