Report 4: Volumetric data, Computational Photography

Post Reply
glegrady
Posts: 203
Joined: Wed Sep 22, 2010 12:26 pm

Report 4: Volumetric data, Computational Photography

Post by glegrady » Tue Mar 29, 2022 2:21 pm

Report 4: Volumetric data, Computational Photography

MAT 255 Techniques, History & Aesthetics of the Computational Photographic Image
https://www.mat.ucsb.edu/~g.legrady/aca ... s255b.html

Please provide a response to any of the material covered in this week's two presentations by clicking on "Post Reply". Consider this to be a journal to be viewed by class members. The idea is to share thoughts, other information through links, anything that may be of interest to you and the topic at hand.


Report for this topic is due by May 6, 2022 but each of your submissions can be updated throughout the length of the course.
George Legrady
legrady@mat.ucsb.edu

siennahelena
Posts: 8
Joined: Tue Mar 29, 2022 3:33 pm

Re: Report 4: Volumetric data, Computational Photography

Post by siennahelena » Fri Apr 29, 2022 3:12 pm

From this week’s class, something that caught my attention was our discussion about the iPhone’s front-facing camera. In particular, the concept of using 3D depth cameras and other imaging technologies to create biometric data is a something that I wanted to explore more.

In the case of Apple, when describing their updated front camera technology works (as we saw in class), they wrote of their TrueDepth system:
“The system features several components dedicated to capturing 3D information for Face ID authentication and Animoji”
On another support page about their Face ID technology, Apple provides a further descriptions TrueDepth camera https://support.apple.com/en-us/HT208108:
“The TrueDepth camera captures accurate face data by projecting and analyzing thousands of invisible dots to create a depth map of your face and also captures an infrared image of your face. A portion of the neural engine of the A11, A12 Bionic, A12X Bionic, A13 Bionic, A14 Bionic, and A15 Bionic chip—protected within the Secure Enclave—transforms the depth map and infrared image into a mathematical representation and compares that representation to the enrolled facial data.”

Related to my prior posts, I am curious about ideas of data ownership and privacy. Something that is particularly fascinating to me about Apple’s Face ID and other similar identification systems is how they create and use biometric data. For instance, facial identification using depth cameras or AI technologies is used in a variety of places including: What’s particularly fascinating for me is how easily this technology captures biometric data whether or not a person provides consent. Using the home security systems as an example, if a person had a Nest Camera and happened to stop by their front door, their camera would be collecting your information without your knowledge. Beyond government surveillance, it is astonishing to think of how many private cameras are constantly collecting information.

However, it is also interesting to think about folk methods for circumnavigating the onslaught of recognition technologies. Below are a couple of examples that I have found suggested to stop face detection:

Juggalo makeup which confuses AI that recognizes faces from contrast points https://www.fastcompany.com/90373952/to ... alo-makeup
Image

Reflectacles fashion sunglasses that use infrared blocking lenses and reflective frames https://www.reflectacles.com/#home
Image

Wearing a mask that has the image of another person https://onezero.medium.com/you-can-fool ... 05ed3a59b0
Image


Overall, there is much to appreciate about these biometric technologies and what they afford in terms of security and convenience. However, as suggested by the methods that people use to “trick” facial recognition, we cannot ignore the deep-seated fears of surveillance and suppression.

ashleybruce
Posts: 11
Joined: Thu Jan 07, 2021 2:59 pm

Re: Report 4: Volumetric data, Computational Photography

Post by ashleybruce » Fri May 06, 2022 3:15 pm

After the hands-on lab using the cameras in processing, I wanted to better explore how the X-box Kinect camera works and the applications these types of cameras bring.

How the camera works is that it transmits a near-infrared light and measures the time it takes for the light to reach the camera again after being reflected off objects [1]. The time it takes for the light to be reflected back is proportional to the distance of the object, so the camera can use that to determine the depth of the object. This type of computation can result in these highly accurate 3D mappings of the objects in the field of view.
depth-point-cloud.png
This is similar to how Lidar works, which is the remote sensing method that self-driving cars work. Lidar - Light Detection and Ranging - uses light pulses to measure the distance of objects to generate precise, 3D maps of the surrounding area.
Lidar.png
As can be seen from both of these images, both the IR camera sensing and Lidar give an accurate 3D model of the surrounding environment. But this is only one part of the process, and a small one at that. Generating these image environments is great, but how does the computer actually do things with them once they are created? For example, if we look at the last image, we, as humans, can see that 2 humans are crossing a crosswalk and (even though we cannot see it) the light is probably red, so the car should be stopped for these people to cross.

But the car doesn’t know that. All it sees are some objects moving across some place in space. This leads to an interesting idea, that even though with Lidar, the car can “see”, it can’t actually “understand” without human intervention. This is where image processing techniques and machine learning come into play for self-driving cars. As that is not the topic for this week, I will hold off on exploring these ideas more for a future week. With this topic coming up though, I want to pose a few questions that this brings.

Since self-driving cars rely on human input, it is to be expected that some human bias gets inherently embedded into the systems of image processing and machine learning that the self-driving cars use. In my past posts, I have brought up the idea of the “true image”, which is an image being a perfect representation of something happening at that moment. With Lidar and IR cameras, it would make sense that the images they produce are true images, as they are a perfect representation of the surrounding environment. But since a computer cannot “see”, we have to tell it what to look for and what things are and to make decisions based on biases we embed into it. This led me to question what exactly it means to call something a “true image”. Since we are all biased in our own ways, is there ever really a true image?


Works Cited
1. https://www.wired.com/2010/11/tonights- ... bject%20is.
2. https://oceanservice.noaa.gov/facts/lidar.html

nataliadubon
Posts: 15
Joined: Tue Mar 29, 2022 3:30 pm

Re: Report 4: Volumetric data, Computational Photography

Post by nataliadubon » Mon May 09, 2022 11:46 pm

Last week we learned about volumetric data, to which I wanted to explore further but within the gaming realm.

As we learned, volumetrics is a computational technique used in three-dimensional computer graphics that adds lighting effects into a rendered scene
When specialized lighting techniques are used to depict a specific perspective in a specified space, volumetric lighting is created. In physical lighting models, it is a modern standard. The result appears as apparent light beams that shine throughout the environment. In video games, volumetric lighting effectively replaces flat lighting. Volumetric lighting gives any video game scene a realistic dimension and enhances the overall appearance.

The volumetric lighting approach has another method that is similar in its execution: volumetric fog. The effect essentially generates a thick fog which aids in creating incredibly atmospheric images which pairs perfectly with volumetric lighting. Although the name implies that the effect is exclusively achieved through fog, this is not the case as technically any other air-like element is capable of material particle occlusion. Overall, both volumetric lighting and volumetric fog work well together to enhance a particular scene.

Below, I've attached a short video that I believe demonstrates the extent of volumetrics very well. This clip is from a video game called "Ghost of Tsushima and has been hailed by both the gamers, programmers, and cinematographers alike for its photorealistic graphics and stunning scenery. I've personally played this game myself and often stopped playing just to take screenshots and video captures of the unbelievably beautiful scenery I often ran across. With technology such as volumetric path tracing, video games have pioneered through the area of computation and aesthetics.
https://youtu.be/my62R7QUf00?t=120

As an addition to volumetrics, ray tracing is similarly a lighting method that adds realism to video games. Its primary focus relies on simulating how real light reflects and refracts and mimicing such light when creating a more lifelike atmosphere. This done to compete against static lighting in more traditional games. Below I've attached some example of ray tracing, with two of the images showing before and afters (you can often successfully guess which side is the "after").

Image

Image

Image

Resources:

Post Reply