Digital Twin Realized in the Metaverse: Controlling Real-World Mobile Robots in Real Time

2024.06.21

With the spread of 5G and ICT services, "Digital Twins" are gaining attention, particularly in production sites and urban construction planning. The market size of Digital Twins has been growing annually and is expected to reach approximately 3.9 trillion yen by 2025.

Source: Ministry of Internal Affairs and Communications
https://www.soumu.go.jp/johotsusintokei/whitepaper/ja/r05/html/nd247530.html

SANEI HYTECHS' research and development (R&D) department has developed a unique metaverse platform with Digital Twin functionality as part of its efforts to explore metaverse-related technologies.

https://www.sanei-hy.co.jp/en/rd/metaverse/

In this article, we will introduce the details of the developed platform.

INDEX

What is Digital Twins?
Benefits of Digital Twins
Use Cases of Digital Twins
Development Case Study of a Metaverse Platform

What is Digital Twins?

First, let's introduce an overview of Digital Twins and their use cases.

"Digital Twin technology" replicates the physical world in a virtual space. This technology collects data from the real world using sensors and IoT technologies and reflects it in the virtual space. In the virtual space, the data is analyzed and simulated using AI, and the results are feed back to the real world.

The use of Digital Twins is already expanding in manufacturing, construction, logistics, and urban planning projects.

Benefits of Digital Twins

Predictive Maintenance

By simulating potential equipment deterioration or component failures in the virtual space, Digital Twins can predict the optimal times for replacements or repairs, thereby aiding in maintenance.

Cost Reduction

By creating products or buildings in a virtual space, it leads to cost reduction in terms of expenses and time required for manufacturing, rather than verifying by actually manufacturing in reality.

Remote Communication

By reflecting real-world situations in real-time into virtual spaces, users in different locations can efficiently confirm situations and make decisions together.

Use Cases of Digital Twins

Manufacturing

Predict future production line stoppages based on past experiences to prevent them beforehand.
Simulate safety measures when introducing robots to the production line and determine the optimal placement.

Construction

Simulate multiple construction patterns to formulate the best construction plan.
Reflect real-time site conditions to monitor and support construction and maintenance from remote locations.

Public Works (Urban Planning)

By overlaying social infrastructure and pedestrian flow in real time, urban development plans and traffic congestion mitigation measures can be formulated.
Predict regions likely to experience flooding during typhoons and develop and implement disaster prevention measures.

Development Case Study of a Metaverse Platform

Our company has developed a metaverse platform that realizes Digital Twins. This introduction covers the creation of edge devices to be controlled, the production of 3D CG models, and the design and construction of both the edge device server and the metaverse server.

The overall structure of the developed metaverse platform is shown in the diagram below. For details on the edge devices and servers we created, please refer to the subsequent explanations.

Overview Diagram of a Metaverse Platform with Digital Twin Functionality

Edge Device Creation (Mobile Robots)

We created a mobile robot as an edge device equipped with a 32-bit microcontroller (ESP32). The robot has a ball caster on the front and tires on both sides, each connected to a motor. The microcontroller constantly communicates with the edge device server to control the motors. The chassis was made using our in-house 3D printer, and an AR marker for position estimation is attached on top.

Note: Originally, AR markers are used as reference points to display augmented reality (AR) content. When read by a smartphone or tablet, they allow the user to view AR content through the screen. In recent years, AR markers have also been used to determine the location of autonomous mobile robots.

Created Mobile Robot

3D CG Model Creation

The 3D CG model of the mobile robot was created using Blender and Unity.

Blender: 3D CG Creation Tool

Created 3D CG models of the chassis and various parts, such as the ball caster and screws, from CAD data intended for 3D printing.
Assembled these parts models to create the base 3D CG model of the mobile robot.

Unity: Game Development Engine

Applied coloring (texture and material settings) to the 3D CG model created in Blender to complete the model.
* Texture refers to adding colors or patterns to the model's surface.
Material gives the surface its texture (such as bumps or metallic sheen).

Below is an image showing the process of creating the 3D CG model (the photo is from an earlier design).

Process of Creating the 3D CG Model of the Mobile Robot

Design and Construction of the Edge Device Server

The edge device server uses a USB camera to take wide-angle shots from above, obtaining the position information of the mobile robot on the field.
This position information is continuously sent to the metaverse server.
Additionally, when control commands are received from the metaverse server, the edge device server calculates the control parameters for the motors and sends them to the mobile robot.

Design and Construction of the Metaverse Environment

The metaverse environment was developed in Unity as a standalone application to run on client PCs.
Users can operate the application using arrow buttons to send control commands for forward, backward, and turning movements to the metaverse cloud server.
The cloud server, built using a third-party communication engine cloud service, then transmits these control commands to the edge device server.
The position information of the real mobile robot is continuously feed back from the edge device server and synchronized in real-time with the metaverse application.

Overall Structure of the Metaverse Platform with Digital Twin Functionality

You can see how the real mobile robot is controlled through the metaverse application.

Through our developed metaverse platform, we have demonstrated motor control and position control of mobile robots, making the Digital Twin a reality. In the future, we plan to apply this technology to the automatic harvesting robots in our smart agriculture project (https://www.sanei-hy.co.jp/en/rd/smart_agri/), undertaken by our R&D department.

Experience a trial version of our simulator tool that enables control simulation of robots and drones in a metaverse environment.

We are working on the development of a simulator tool to realize control simulation of robots and drones in a metaverse space. With this tool, users can simulate and operate various devices in a virtual environment. In August 2024, we released a trial version with some functions implemented. This allows users to directly experience this new technology.

For details of the trial version, please see the following link.

https://www.sanei-hy.co.jp/en/news/newsrelease/2024/01133/

We will continue to provide the latest updates on our blog. Please stay tuned.