Mapping SpaceMouse Controls to SO-101 Movements

In the previous article, I connected the SpaceMouse to the PC and confirmed that all six types of input could be detected correctly.

https://dev.to/takeofuture/trying-vla-part-5-setting-up-and-testing-a-spacemouse-eio

Forward / Backward
Left / Right
Up / Down
Pitch
Roll
Yaw

During the SpaceMouse test, I confirmed the following input values.

Forward horizontal  = +Y
Backward horizontal = -Y

Left horizontal     = -X
Right horizontal    = +X

Up                  = +Z
Down                = -Z

Forward tilt        = +Pitch
Backward tilt       = -Pitch

Left tilt           = -Roll
Right tilt          = +Roll

Left twist          = -Yaw
Right twist         = +Yaw

An important point here is that these values from the SpaceMouse are not sent directly to individual SO-101 motors.

Conceptually, the flow from the SpaceMouse to the SO-101 looks like this:

SpaceMouse
    ↓
x / y / z / roll / pitch / yaw
    ↓
SpaceMouse Teleoperator
    ↓
target_x / target_y / target_z
target_wx / target_wy / target_wz
    ↓
Inverse Kinematics (IK)
    ↓
SO-101 Joint Positions
    ↓
SO-101

The SpaceMouse plugin treats the 6DoF input from the SpaceMouse as movement of the End Effector in Cartesian coordinates.

The target movement is then converted into the required SO-101 joint angles using IK, or Inverse Kinematics.

In other words, instead of directly specifying something like:

"Move this motor by 5 degrees"

we provide commands such as:

"Move the End Effector slightly forward"
"Move the End Effector slightly upward"
"Rotate the End Effector slightly"

The SpaceMouse provides these commands, and IK calculates how the individual joints need to move.

Mapping Between SpaceMouse and LeRobot Coordinates

There is one thing we need to be careful about here.

The x and y values displayed by the SpaceMouse test do not directly become LeRobot's target_x and target_y.

With the default SpaceMouse plugin configuration, the axes are mapped as follows:

SpaceMouse y     -> target_x
SpaceMouse x     -> target_y
SpaceMouse z     -> target_z

SpaceMouse roll  -> target_wx
SpaceMouse pitch -> target_wy
SpaceMouse yaw   -> target_wz

In addition, the sign of target_y is inverted.

Therefore, based on the SpaceMouse inputs I confirmed in the previous test, the mapping to LeRobot becomes:

Here,

target_x / target_y / target_z

represent End Effector translation, while

target_wx / target_wy / target_wz

represent rotation around the X, Y, and Z axes.

The default SpaceMouse plugin configuration uses:

x_axis="y"
y_axis="x"
y_sign=-1

which is why the axes are mapped this way.

About 6DoF Input and the SO-101

There is another important point.

The SpaceMouse itself is a full 6DoF input device:

X
Y
Z
Roll
Pitch
Yaw

The SO-101 has six servos:

shoulder_pan
shoulder_lift
elbow_flex
wrist_flex
wrist_roll
gripper

However, the final servo, gripper, is used to open and close the gripper.

This means that the robot arm uses five joints to control the position and orientation of the End Effector.

The SpaceMouse can provide a 6DoF target, but this does not mean that the SO-101 can independently realize every arbitrary 3D position and orientation.

LeRobot's IK converts the requested motion into joint movements that are feasible for the SO-101.

For this reason, during the first teleoperation test, I will move the SpaceMouse one axis at a time and verify which direction the End Effector actually moves.

What Are We Actually Controlling with the SpaceMouse?

Before moving the SO-101, let's clarify what we are actually controlling with the SpaceMouse.

The SpaceMouse does not directly control each SO-101 motor one by one.

Instead, what we mainly control is the 3D position and orientation of the robot arm's End Effector.

Using the SpaceMouse, we can:

• Move forward/backward, left/right, and up/down
• Tilt forward/backward and left/right
• Rotate left/right

In other words, we tell the robot:

"Move the End Effector in this direction and orient it this way."

SpaceMouse
    ↓
Specify End Effector
position and orientation
(X / Y / Z / Roll / Pitch / Yaw)
    ↓
LeRobot + IK
(Inverse Kinematics)
    ↓
Calculate how the joints
need to move together
    ↓
Control the SO-101 motors

The user does not need to think in terms of:

Move shoulder_pan by a certain angle
Move shoulder_lift by a certain angle
Move elbow_flex by a certain angle

Instead, we simply operate the SpaceMouse based on where we want the End Effector to move.

LeRobot's IK calculates how the individual joints should move together.

The only exception is the gripper open/close operation.

That operation is mapped directly to the SpaceMouse buttons rather than being calculated by IK.

SpaceMouse 6DoF Motion
    ↓
Position / Orientation
    ↓
Inverse Kinematics
    ↓
Multiple joints move together

SpaceMouse Button
    ↓
Gripper Open / Close
    ↓
Open or close the gripper

So, for the arm's position and orientation, we specify the desired End Effector motion with the SpaceMouse and let LeRobot determine how the joints should move.

For the gripper itself, the open/close commands are directly mapped to SpaceMouse buttons.

Actually Moving the SO-101 with the SpaceMouse

Now let's actually control the SO-101 using the SpaceMouse.

The SpaceMouse plugin provides an IK profile for the SO-101, so teleoperation can basically be started with a command like this.

First, activate the Python virtual environment.

$ source ~/pyenv/lerobot/bin/activate

Then run the teleoperation command.

(lerobot)$ lerobot-teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=takeo_so101 \
  --teleop.type=spacemouse \
  --teleop.adapter.mode=ik \
  --fps=60

Since this is my first time actually controlling the robot with the SpaceMouse, I do not want to start immediately with the normal settings.

Instead, I will begin with more conservative settings.

(lerobot)$ lerobot-teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=takeo_so101 \
  --teleop.type=spacemouse \
  --teleop.adapter.mode=ik \
  --teleop.deadzone=0.1 \
  --teleop.input_timeout_s=0.08 \
  --teleop.adapter.translation_step_m=0.001 \
  --teleop.adapter.rotation_step_rad=0.005 \
  --fps=60

Since I am operating the machine through SSH, I am not adding the GUI visualization option:

--display_data=true

If you are running LeRobot directly from a local Ubuntu desktop, you can add it if needed.

What the Important Options Mean

The following settings are particularly important for this test.

--teleop.adapter.mode=ik

This converts the Cartesian input from the SpaceMouse into SO-101 joint angles using Inverse Kinematics.

--teleop.deadzone=0.1

This ignores very small SpaceMouse inputs around the neutral position.

--teleop.input_timeout_s=0.08

If no new input is received from the SpaceMouse for a certain amount of time, the input is treated as zero.

--teleop.adapter.translation_step_m=0.001

This limits the translational movement per frame, even when the SpaceMouse is pushed to its maximum input.

0.001 m is equal to 1 mm.

--teleop.adapter.rotation_step_rad=0.005

This also keeps the rotational movement small during the initial test.

These settings are introduced in the SpaceMouse plugin README as a "Conservative first run" configuration.

Before Actually Moving the Robot

Before running the command, make sure that:

• The SO-101 is securely fixed to the desk
• The Waveshare board is connected via USB
• The 12V power supply for the servos is connected
• The robot arm is not in an extreme pose
• The arm is positioned so that it will not hit the clamp or desk
• The SpaceMouse is in its neutral position

Also, depending on the configuration, torque may be disabled when the SO-101 program exits.

Therefore, when stopping the program with Ctrl+C, be prepared to support the robot arm so that it does not suddenly drop under gravity.

First, Try Only Forward and Backward Motion

Instead of immediately trying all six axes, I will first test only the forward and backward horizontal movement of the SpaceMouse.

After starting the command, do not touch the SpaceMouse at first.

First, confirm that the SO-101 maintains its current pose.

Then:

Push the SpaceMouse slightly forward
          ↓
Immediately return it to center
          ↓
Check the movement of the SO-101 End Effector

At first, do not push the SpaceMouse all the way or hold it for a long period of time.

Repeat the following:

Push slightly
↓
Return to center
↓
Observe the robot

If forward and backward movement work correctly, test the remaining axes one at a time:

Left / Right
Up / Down
Pitch
Roll
Yaw

What Happens Between the SpaceMouse and SO-101?

For example, when I push the SpaceMouse forward, the previous test showed:

SpaceMouse y = +

The plugin converts this to:

target_x = +

Then IK converts the requested target_x movement into the required joint angles:

target_x
    ↓
shoulder_pan
shoulder_lift
elbow_flex
wrist_flex
wrist_roll

In other words, pushing the SpaceMouse forward does not simply move one particular motor.

Multiple joints move together so that the End Effector moves in the requested direction.

This is the major difference from the previous test, where I directly moved individual motors from Python.

Controlling the Gripper with SpaceMouse Buttons

The SpaceMouse plugin can also use the side buttons on the SpaceMouse.

By default:

button[1] -> Gripper Open
button[0] -> Gripper Close

However, it is better to verify which physical button on your particular SpaceMouse corresponds to button[0] and button[1].

Before connecting the robot, run:

(lerobot)$ lerobot-teleoperator-spacemouse-test

With no buttons pressed, you should see:

buttons=[0, 0]

Then press the left and right physical buttons one at a time.

You should see either:

buttons=[1, 0]

or:

buttons=[0, 1]

This lets you determine which physical SpaceMouse button corresponds to each button index.

Once this is confirmed, the SpaceMouse can be used to control:

End Effector Translation
+
End Effector Rotation
+
Gripper Open / Close

Overall Flow

The complete flow looks like this:

SpaceMouse
   │
   ├─ Forward / Backward / Left / Right / Up / Down
   ├─ Roll / Pitch / Yaw
   └─ Gripper Buttons
   ↓
LeRobot SpaceMouse Teleoperator
   ↓
Cartesian End-Effector Target
   ↓
Inverse Kinematics
   ↓
SO-101 Joint Position Commands
   ↓
SO-101 Follower Arm

The SpaceMouse is not directly connected to the SO-101.

Both the SpaceMouse and the SO-101 are connected to the PC.

LeRobot running on the PC reads the SpaceMouse input and uses IK to convert that input into joint movements for the SO-101.

Once this setup is working correctly, the next step will be to add cameras and record:

Camera Image
+
Robot State
+
Robot Action

as a LeRobotDataset.

That will allow us to start collecting demonstration data for imitation learning.