[ROS 2] Applying VLA to a 4-Axis Robot Arm — Achieving a 69% Success Rate

2026.11.10

  • ROS2
VLA (Vision-Language-Action) models are attracting attention as a core technology for Physical AI. However, many publicly available pre-trained models are trained on data from 6-axis robot arms. Therefore, we verified whether a VLA could operate on a 4-axis robot arm controlled by ROS 2. We trained the model by adding real-world machine data to data generated through simulation. As a result, the success rate for lifting cubes reached 69%.

What is VLA?

VLA is an AI model that handles Vision, Language, and Action in a single model. For example, when instructed to "lift the red cube," it understands the language instruction and recognizes the red cube from the camera image.
On top of that, it directly outputs the robot's actions. Previously, robots required precise programming to recognize the position of objects and determine how to move them. VLA is built by training large amounts of robot motion data based on AI models that understand images and language. Therefore, a key feature is that even if the target object or instructions change slightly, it can compose movements leveraging what it has learned.

Simulation and Fine-Tuning in Isaac Sim

This time, we used "SmolVLA," a compact VLA released by Hugging Face. It is a model that can perform training and inference even on a PC equipped with a standard GPU. First, we tried operating the 4-axis robot arm zero-shot without additional training. Although the base of the arm rotated toward the cube, the subsequent movement was tailored for a 6-axis arm, and it could not grasp the cube. Therefore, we created imitation learning data using NVIDIA's robot simulator "Isaac Sim" and performed fine-tuning. For the training data, we used movements of moving over the cube, lowering straight down to grasp it, and lifting it up. These movements are converted into target angles for each motor using inverse kinematics. After training on data from 148 episodes, the arm was able to grasp cubes in the simulator.
Training Workflow Using Isaac Sim
Training Workflow Using Isaac Sim

Operation Verification on the Actual Hardware

On the actual hardware, VLA inference and arm control were separated across different PCs and connected via ROS2 communication. On the inference PC, camera images are fed into SmolVLA to determine the target angles for each motor. This value is transmitted from the VLA ROS node, and the controller ROS Node on the control PC operates the arm via the control board. The arm's current angles return to the inference side through the reverse path and are utilized for subsequent inference.
Execution Workflow Using Actual Hardware
Execution Workflow Using Actual Hardware

Increasing Real-Machine Success Rate from 42.5% to 69%

The model trained in the simulator did not work on the actual hardware as is. This was due to the significant difference between the training data images and the camera images captured by the actual hardware. Therefore, we introduced variations across episodes in the simulation environment. We varied ambient light brightness, floor color, camera position and angle, and the arm's initial posture each time. As a result of retraining with this data, the actual hardware became able to perform motions close to the training data. The success rate was measured by conducting 200 trials under each condition on the actual hardware.
Condition Number of Episodes Training Steps Success Rate
(1) Simulation 1000 100,000 39% (78/200)
(2) Simulation 4000 200,000 42.5% (85/200)
(3) (2) + Real-machine data 4000 + 100 200,000 + 3,000 69% (138/200)
Even when quadrupling the simulation data, the increase in success rate remained at just 3.5 percentage points. On the other hand, when 100 trials of data collected from the actual hardware were additionally trained on (2), the success rate improved to 69%.
Successfully lifted a cube using a fine-tuned VLA
We successfully fine-tuned a VLA trained on 6-axis robot arm data and deployed it on a 4-axis robot arm. While there is room for improvement with simulation-only training, adding a small amount of real-world data significantly improved the success rate. Going forward, we plan to work on validating other VLA models, training for multiple instructions, and training using terrain data. From the design and implementation of robot systems using ROS2 to simulation and operations system development, we are here to support you. We also welcome inquiries regarding physical AI validation, such as the work described here. For more details, please visit our ROS2 Services page.
Recent articles
Category
Archive