For my senior Capstone in Cognitive Science, I designed and ran a pilot study looking at how blurry regions in images may affect a person’s ability to distinguish AI-generated images from real images. Featured below are the stages involved in creating, running, and drawing conclusions from this study, including my initial study design that was scrapped, how I created the image stimuli, information about the data collected, and my takeaways from conducting this study.
Qualtrics, Adobe Photoshop, Excel pivot table analysis, survey design, data collection, pilot study design, literature reviews
The basis of this study was to explore the question: How do people make distinctions between real images and AI-generated images, and what influences how accurate they are? Initially, I hypothesized that people can accurately identify AI-generated images up until a certain threshold of realism; essentially, when an AI image becomes too realistic, we cannot accurately differentiate them from real images. Given this first hypothesis, I would have needed to define that realism threshold. I began this by identifying certain visual features in images—such as unnatural smoothness, oddly placed blurry regions, and impossible overlapping of objects—that humans might pick up on as indicators that an image isn’t real.
When starting to formulate the study design, I ran into a couple issues. The idea was to have a two-part study, in which subjects would first rank the realism of a breadth of AI and real images based on several different visual features, sorting these images into levels of realism. Then, subjects would be presented with these images again and asked to select whether they were AI generated or real. This way, I could analyze the results of the second part of the study based on the level of realism each image was sorted into. The first issue was simple: I did not have the time or resources to conduct a 2-part study in a few months without sacrificing quality. The second issue concerned the premises of the study, specifically the realism threshold. It occurred to me that the effects of these visual indicators of images being AI-generated may be synergistic, not additive, so a linear spectrum of realism with one static, identifiable threshold wouldn’t make sense.
To address these issues, I narrowed my initial idea. I hypothesized that people’s perception of blurry regions in images helps them accurately distinguish AI images from real images, and, when these regions are removed from an image, people are less accurate in their classifications.
STUDY DESIGN
My final study design was a modified version of the initial study design. First, I collected a breadth of AI generated and real images to use. I then created these into the visual stimuli for this study by cutting out the blurry regions of copies of these images, which served as the altered copies of the images. This gave me 4 types of images to work with: unaltered AI, altered AI, unaltered real, and altered real.
For simplicity’s sake, this study design only utilized the second part of the initial design. Subjects were to be presented with all of the visual stimuli images I created and asked to classify them as real or AI. Results would then be analyzed based on how accurate subjects were in their classifications when presented with altered images versus their unaltered counterparts.
I created this study in Qualtrics; the survey used can be accessed here.
I created the altered versions of these images in Adobe Photoshop by simply cutting the blurry regions of each image out and replacing it with a transparent background. While all images can be viewed through the Qualtrics survey, examples of each type of image are shown below.
DATA ANALYSIS
I analyzed the data collected through Qualtrics using Excel pivot tables and breaking down the average accuracy of responses according to each type of image. Below is the table and graph of these results.
FINAL REPORT
Here is the final report writeup of this study.
Some highlights include:
20 American adults participated in this study.
The Qualtrics survey consisted of 40 questions, each including a single image and prompting the participant to select if they think the image is real or AI.
Participants were more accurate in classifying real images when they were unaltered (blurry regions were present in the image).
The presence of blur in images did not seem to have a significant effect on participants' ability to accurately classify AI images.
TAKEAWAYS
The weaknesses of my original study design helped me with the direction of my final study; honing in on a specific aspect of these images, i.e., blurry regions, made the study design easier to create and yielded better quality results than if I had kept my broader initial idea, where there would have been too many variables at play to create that aforementioned threshold. Embracing simplicity in my study design yielded more precise data.
Something apparent given the results of this study is that my hypothesis wasn’t supported. However, looking for insights from my results that weren’t part of this hypothesis proved to be useful. While the results don’t necessarily indicate that blurry regions in images help us accurately determine whether they’re real or AI, it did suggest that people may have a bias for associating these image alterations with AI imagery. So, keeping an open mind while drawing conclusions about my results may give directions for future research that weren’t initially intended.
Framer 2023
Amsterdam