Robot System for Projection Mapping

Projection mapping refers to the technique or artwork of projecting processed visuals onto objects with three-dimensional shapes. When conducting projection mapping, it is essential to align the projection target with the position of the projector. To achieve optimal visual effects, the position of the imagery needs to be aligned pixel by pixel. If there is any misalignment, it can compromise the appearance of the artwork. While this alignment process may seem straightforward, it often requires adjusting the projection using the projector and visually confirming the results. This can be a laborious and time-consuming task, involving manual adjustments and visual inspections.

The system consists of three main components: a monocular camera (commonly known as a regular camera), a projector, and a computer (PC). While a monocular camera is not necessary for basic projection mapping, incorporating it into the system allows for the calibration of the relative positions between the monocular camera, projector, and projection surface, as well as recognition of the projection surface. Both the monocular camera and the projector are commercially available components that are purchased and used as part of the system.

In the field of sensing, which involves using sensors to capture information about the surrounding environment and object shapes, LiDAR and stereo cameras are well-known technologies. We also attempted to incorporate these devices for sensing and build a projection mapping system. However, we encountered challenges in achieving the level of visual quality that is pleasing to the human eye.

The human eye is incredibly precise, capable of perceiving objects as small as a few millimeters in size, even at a distance of 5 meters, assuming normal visual acuity of 1.0. Unfortunately, with the sensor equipment we owned at the time, such as LiDAR and stereo cameras, the recognition accuracy was approximately 1 cm. When compared to what the human eye can discern, the misalignment was noticeable, making the projected content appear inaccurate.

During our search for more precise sensors, we discovered that conventional cameras, specifically optical cameras, provided the solution we were seeking. Looking back, cameras offer high-resolution imaging capabilities that rival the human eye, and there are many options available in the market. In recent years, web cameras, in particular, have become widely used for remote work and video conferences. It's rare to encounter issues with poor image resolution that hampers visibility of the other person's face. By combining these cameras with projectors and utilizing them for sensing, we have achieved object recognition at a level that appears visually pleasing even to the human eye.

The combination of projector illumination and camera capture for object recognition eliminates the need for recalibrating the positions of the projector and camera. This is an additional advantage of adopting cameras as sensors, as it simplifies the calibration process.

The developed system is divided into three main steps:1. Scanning of the projection target object, 2. Exploration of suitable projection surfaces, and 3. Correction of the projector's position.

Using the scanning results, an exploration is conducted to identify suitable surfaces for projection. Based on the exploration, the artwork images or videos are projected onto surfaces deemed appropriate for projection.

To ensure accurate projection alignment, the captured results from the camera undergo transformation and correction before being projected by the projector. This is necessary because projecting the captured results directly without adjustments would result in misalignment of the projected surface.

We have created a demonstration video of projection mapping using this system. To enhance visibility, we have placed a white cardboard inside a dark room and captured the scanning process on video.

After scanning, we applied an effect to the video that makes the surface appear smooth and eliminates irregularities. The effectiveness of this effect strongly depends on the calibration information, and the quality of the final video reflects the accuracy of the calibration.

We have embedded the video onto a suitable flat surface identified by the algorithm's detection. The region for projection is determined directly from the exploration results, and no manual adjustments or refinements were made.