Report 6: Vision Science & Machine Learning

Post Reply
glegrady
Posts: 203
Joined: Wed Sep 22, 2010 12:26 pm

Report 6: Vision Science & Machine Learning

Post by glegrady » Tue Mar 29, 2022 2:22 pm

Report 6: Vision Science & Machine Learning

MAT 255 Techniques, History & Aesthetics of the Computational Photographic Image
https://www.mat.ucsb.edu/~g.legrady/aca ... s255b.html

Please provide a response to any of the material covered in this week's two presentations by clicking on "Post Reply". Consider this to be a journal to be viewed by class members. The idea is to share thoughts, other information through links, anything that may be of interest to you and the topic at hand.

Report for this topic is due by May 20, 2022 but each of your submissions can be updated throughout the length of the course.
George Legrady
legrady@mat.ucsb.edu

siennahelena
Posts: 8
Joined: Tue Mar 29, 2022 3:33 pm

Re: Report 6: Vision Science & Machine Learning

Post by siennahelena » Sat May 14, 2022 1:30 pm

This week’s classes caught my interest in our discussion of Alfred Yarbus’s work in the 1950s and 1960s on eye-tracking. In summary, Alfred Yarbus was a Russian psychologist who tracked observers’ eye movements as they moved around a scene. In class, we talked specifically about Yarbus’ study of “The Visitor” painting and how different prompts resulted in observers gazing and focusing on different parts of the painting. In addition to this, Yarbus also conducted several eye-tracking studies of observers looking at faces.

Image
Image

Yarbus’ work reminded me of another famous study about culture and point of view conducted several years later by Liang-Hwang Chiu (1972). Chiu examined how children from Western cultures (American) and Eastern cultures (Chinese) interpreted the same scene. Overall, they found that Westerners tended to perceive the world through rules of category membership while Easterners perceive the world through relational groupings of objects. In the study, Chiu had children from the West and from the East look at an image composed of three objects: a cow, a chicken, and grass. He then asked the participants to choose which of the two objects should go together. The Western children chose the cow and chicken because they were both animals while the Eastern children chose the cow and grass because cows eat grass.

Image

After hearing about Yarbus’ work, I thus wondered if any of these cultural difference studies like Chiu’s used similar methods of eye-tracking? The answer is yes, there are several. For example, researchers Hannah Chua, Julie Boland, and Richard Nesbitt (2005) built upon the finding that Westerners pay attention to focal objects and Easterners to relations and context. Using eye-tracking of 36 images with objects in the foreground and realistic backgrounds, they found that American participants were quicker to local at the focal object and spent more time overall looking at it. In comparison, Chinese participants looked more at the background.

Image

In a more recent study, researchers Jennifer Haensel, Tim Smith, and Atsushi Senju (2022) used eye-tracking to examine how Easterners and Westerners differed in their mutual gaze during dyadic social interactions. Contrary to the prior theory that Easterners are more likely to avoid gaze as a cultural normal than Westerners, this study found that Easterners had longer instances of mutual gaze during a story-telling task than Westerners.

Returning to Yarbus’ original work, it could be quite interesting to replicate this study using participants from different cultures. Specifically, I wonder what the eye-tracking results would be if Eastern and Western participants were asked questions while looking at the same scene, the differences in their answers, and their eye-tracking patterns to arrive at each answer.

Sources:
  • Chiu, L.-H. (1972). A Cross-Cultural Comparison of Cognitive Styles in Chinese and American Children. International Journal of Psychology, 7, 235–242.
  • Chua, H. F., Boland, J. E., & Nisbett, R. E. (2005). Cultural variation in eye movements during scene perception. Proceedings of the National Academy of Science, 102, 12629–12633.
  • Haensel, J., Smith, T. J., & Senju, A. (2021). Cultural differences in mutual gaze during face-to-face interactions: A dual head-mounted eye-tracking study. Visual Cognition. Retrieved from https://www.tandfonline.com/toc/pvis20/current
  • Tatler, B. W., Wade, N. J., Kwan, H., Findlay, J. M., & Velichkovsky, B. M. (2010). Yarbus, eye movements, and vision. I-Perception, 1, 7–27.

ashleybruce
Posts: 11
Joined: Thu Jan 07, 2021 2:59 pm

Re: Report 6: Vision Science & Machine Learning

Post by ashleybruce » Mon May 23, 2022 12:43 pm

One thing I was really interested in from this week was also the eye tracking studies. As mentioned in class, Alfred Yarbus was a soviet psychologist who studied eye tracking movements in the 1950s and 60s. In his book "Eye Movements and Vision", he presented the eye movement trajectories after telling subjects to perform different tasks. We can see the results of this study below:
Yarbus_The_Visitor.jpg
One of the most interesting things this study shows is that across the tasks, how much time the eyes linger on the people from the original photo. I find this so interesting because our eye movements are so embedded into our processing of the world around us. Moving our eyes around an image is such a crucial part to comprehending and understanding what we see.

But what if our eyes moved but the image we saw stayed the same? A somewhat weird question, but this is the case for patients who have lost their vision and have "regained" some of it due to eye implant technologies. These patients typically are blind in most or all of their retina. Without going into too much detail about how these implants work, essentially an electrode array is implanted on the retina and can stimulate surviving cells on the retina to produce, what our brain interprets as, "flashes of light". Argus II is one such implant and works by taking video obtained from an external camera, doing some image processing, and stimulating the corresponding electrodes. The components of Argus II can be seen in the image below:
ArgusII.jpg
Those patients who have these implants were once sighted but have gone blind due to various retinal diseases. I bring this up because we can see that the camera is mounted on the sunglasses that the Argus II user wears. Even though the implant is inside the eye, what the patient "sees" changes depending on what the camera sees. So normal eye movements will not change the resulting image, and to change what they see, they have to move their whole head. In my Bionic Vision class in a previous quarter, we got the opportunity to chat with someone who had an Argus II implant inside of his eye. I was able to ask whether or not it was disorienting moving their eyes and not having the image change. They answered that yes, especially near the beginning of having their implant, it took them time to learn that what they saw was dependent on moving their head and not their eyes.

Since the concept of bionic vision is so new, there is a lot of research not just looking into these implants themselves, but also studying eye tracking and movement and how it could potentially be used to aid those with low vision [1][2][3].

References
[1]. https://www.biorxiv.org/content/10.1101 ... 9.220889v1
[2].https://files.zotero.net/eyJleHBpcmVzIj ... nd%20g.pdf
[3]. https://files.zotero.net/eyJleHBpcmVzIj ... 0Sever.pdf

nataliadubon
Posts: 15
Joined: Tue Mar 29, 2022 3:30 pm

Re: Report 6: Vision Science & Machine Learning

Post by nataliadubon » Sun Jun 05, 2022 12:31 am

For this week's lesson plan, I had mentioned a video game that uses a similar eye tracking technology in order to navigate through the storyline and control your virtual character. More specifically, this video game is called "Before your Eyes" and primarily uses your visual blinks in order to perform actions. The following description is a synopsis of the game's tale and visual cues:
Embark on an emotional first-person narrative adventure where you control the story—and affect its outcomes—with your real-life blinks. With this innovative technique you will fully immerse yourself in a world of memories, both joyous and heartbreaking, as your whole life flashes before your eyes.
Essentially, the mechanics of the game revolves around blinking in order to move forward; however, because humans tend to blink roughly 12 times per minute and sometimes uncontrollably, the game uses a metronome at at the bottom of the screen called the Blink Counter. You know your next blink will propel you forward when you see this Blink Counter. There are also items in various scenes that you may locate and interact with to help you remain in a scene for longer. In order for the eye tracker to work, the participant needs a webcam that is connected to their computer. In the beginning of the game, the machine then calibrates to track when the eyes open and close. Participants are ultimately rewards with more clues for the storyline if they're able to control their blink and have their eyes opened longer. However, there are parts where the participant is asked to fully close their eyes. In general, the purpose of using this technology is to add a sense of empathy towards the main character and add an extra sense of immersion to an otherwise 2D screen.

Image

This also opens up the conversation about how machine learning can help with accessibility towards those who may otherwise have difficulty completing physical tasks. Video games often entail pressing buttons rapidly, which becomes difficult for people who have amyotrophic lateral sclerosis (ALS) or other diseases that restrict movement. Of course, people who have trouble using a controller will benefit even more from this new input technique of eye tracking. While we sometimes take for granted how we play video games, many people are unable to appreciate the medium because of restricted controller and input designs. In fact, one of the creators of the game says the following regarding the topic of accessibility in using such technology:
The disarming quality of the control scheme makes [Before Your Eyes] a universal experience. It doesn’t matter what kind of game literacy you have going in — you just need a webcam and a computer that can run it
Sources:

Post Reply