Scavenger Hunt: Complex Robot Vision and Machine Learning
Kara Newman and Marjoree Fargas
Our final project will focus on a scavenger hunt for our robot. We plan to use deep learning to teach the robot to recognize complex objects. The robot will start in one point with a fixed camera position and stop to scan around to room to look for objects every five seconds. If the robot recognizes an object, it will move towards it and take a picture to “collect” it as a completed task on the scavenger hunt. In order to complete the hunt, the robot must be able to recognize each object from different angles as it moves around the room. The robot will search for these objects from a list provided. As the robot finds each of these objects, it will make a sound and move onto the next object on the list after sending the image to our computer. After each picture is taken, it will be visualized on a display that will annotate it as “check” off of its scavenger hunt “notebook”. Upon recognizing all of the objects, the robot will do a dance to indicate it has completed its task.
To complete this project, we will be utilizing deep learning, robotic vision, and data collection tools using the Raspberry Pi. We thought this project would be an interesting and unique way to apply our knowledge of robots from the course into an innovative project that combines robotic vision and machine learning. It questions the way robots think and learn in a creative and interesting way. We hope to anthropomorphize the robot by giving it human-like qualities and making it perform tasks like a person would.
The impact complex image detection has on the art community is growing exponentially through the use of machine learning. In a Mashable article produced in March 2018, Rachel Kraus discusses Google’s application of machine learning within the art community through its new app called Arts and Culture that matches selfies to existing works of art. Google searches for similarities between an individual’s face and pieces of art from a large database. Similarly, we will be conducting a Scavenger Hunt that will send the robot into the room looking for an object similar to images we will train it to recognize. A unique element that allows us to experiment with this is the mobility provided by the Arduino and the piCamera, allowing it to scan the room. In this way, our robot will be taking the initiative in physically seeking out these objects and collecting images of them for the purposes of a Scavenger Hunt.
Through Sarthak Jain’s article on the Medium’s website, Jain provides a basic outline for deep learning on Raspberry Pi which will serve as a guideline for our training process. Jain goes into detail on the level of patience and commitment required to train the robot. Through exploring the pyimage website, we have also found creative ideas to experiment with throughout the longevity of this process. We intend on exploring different options for how the images will appear and the manner in which they will be presented to the audience.
Another project we researched for this project was the Facebook AI Scavenger Hunt. This project utilizes reinforcement learning to get a robot to travel around a simulated home and answer complex questions such as “‘What color is the car?’ or ‘Where is the coffee table?’” as it moves. This fascinating project has tremendous relevance for the future role of AI integration into home life.
We expect there to be several challenges as we teach the robot through deep learning to recognize objects with precision. Through resources we found online regarding deep learning through raspberry pi, we will first obtain training data. The challenges may occur during the training process as it is unclear how well the robot will absorb and learn from the data. We intend on challenging the robot with simple and more complex shapes and it is unclear how well it will recognize the more complex objects. We may also encounter some issues with programming the robot to move exactly as desired. We therefore may need to set up a track or boundaries for the robot to ensure it doesn’t wander too far away from the desired targets. The robot may also experience issues with recognizing objects from multiple angles, so this is a something we will need to address in initial learning and troubleshooting.
The theme of a Scavenger Hunt implies a fun, whimsical manner in which we will be applying concepts from lab onto the robot. The innocence of our robot is maintained through its lack of knowledge and dependency often seen in children. Children exploring their surroundings for the purposes of a Scavenger Hunt allows a safe outlet for one to learn about the environment and the objects they are seeking. This will mimic the learning process experienced by a child as we will attempt to teach the robot to seek complex shapes from a predetermined scavenger hunt list. We intend on programming the robot to complete a small dance should it complete the Scavenger Hunt, as an innocent celebration of its accomplishment.
1. https://mashable.com/2018/03/07/google- ... yDIe3Usiql
2. https://medium.com/nanonets/how-to-easi ... 5f29635c74
3. https://www.technologyreview.com/s/6110 ... me-robots/