Notes on my ABB Follow Target Example extension for Omnniverse Isaac Sim 2022.2.0
Supports ABB IRB4600 and 8700
To begin, you need to add and modify a few extensions in Isaac Sim 2022.2.0.
You can hop the installation folder through the Isaac Sim Settings in the OV Launcher.
Extension source directory is here:
~/.local/share/ov/pkg/isaac_sim-2022.2.0/exts
- Copy
follow_target_abb
folder intoomni.isaac.examples/omni/isaac/examples
- In
omni.isaac.examples/config/extension.toml
:
- add
"omni.isaac.abb" = {}
under[dependencies]
- add
"omni.osc.core" = {}
and"omni.osc.server" = {}
under[dependencies]
- add this to the end of the file:
[[python.module]]
name = "omni.isaac.examples.follow_target_abb"
- Copy
omni.isaac.abb
,omni.osc.core
,omni.osc.server
into/isaac_sim-2022.2.0/exts
. - In
omni.isaac.motion_generation
, addomni.isaac.motion_generation/motion_policy/abb
toomni.isaac.motion_generation/motion_policy
- In
omni.isaac.motion_generation
, add this as the last entry inomni.isaac.motion_generation/motion_policy/config.json
"ABB_IRB4600":{
"RMPflow" : "./abb/irb_4600/rmpflow/config.json"
},
"ABB_IRB8700":{
"RMPflow" : "./abb/irb_8700/rmpflow/config.json"
}
- The last step is to manually fix a bug in
omni.isaac.core
.
Go to L141 in omni.isaac.core/omni/isaac/core/tasks/follow_target.py
and change:
self._target.set_local_pose(position=target_position, orientation=target_orientation)
to
self._target.set_local_pose(translation=target_position, orientation=target_orientation)
- Launch Isaac Sim from the OV Launcher or VSCode Terminal. I recommend running in VSCode:
- Open
~/.local/share/ov/pkg/isaac_sim-2022.2.0
VSCode folder. - Terminal > New Terminal to open a new terminal.
./isaac_sim.sh
to run Isaac Sim. This will allow hot reloading and hyperlinks to errors.
- Go to the top toolbar and select Isaac Examples > Manipulation > Follow Target (ABB).
- Press
LOAD
to add the robot to the scene. - Press
START
to begin following the target. - Select the
TargetCube
in theViewport
or through theStage
. - Press
W
/E
when selected toMove
/Rotate
theTargetCube
.
- Calibrate Gestures
- Motion Behavior Params are hard coded for much smaller robot
- Calibrate Speed / Torque
- Tune RMPFlow parameters to avoid high torque errors on robot in RobotStudio
"Coupled Working Range" of Joints 2 & 3 and 5 & 6 cause
Out of Coupled Range
errors - Calibrate Obstacles
- Match to meshes in RobotStudio
RobotStudio Export Geometry not available in free version
This extension can control physical ABB robots using an external EGM_Server
app running on a separate Windows PC (so we can simulate in RobotStudio).
- Connect to the Windows PC over ethernet and configure IPv4 to be on the same network as the Windows PC. For example:
hostname: 192.168.125.100
subnet: 255.255.255.0
- Connect the Windows PC to the Robot Control Box, and configure it's network address to be on the same network as the Windows PC. For example:
hostname: 192.168.125.87
subnet: 255.255.255.0
- Start the
EGM_Server
app on the Windows PC — there's also a desktop shortcut. - Press
PP_to_Main
andPlay
on the Teach Pendant to run theEGM_Main()
task on the robot.
The robot should move to its
[0, 0, 0, 0, 0, 0]
position andTPWrite
that it successfully connected to the EGM Server.
- Connect Isaac Sim to the
EGM_Server
using the Quick Connect panel in Follow Target (ABB).
- Address and Port should connect to the Windows PC running the
EGM_Server
.
The .mod
files for communicating with the real robot are in a directory called egm_template
. Load the program file egm_template.pgf
in RobotStudio and verify the following controller configurations:
You can stream the closest skeleton to Follow Target extension using an external openFrameworks app:
- Use my
ofxAzureKinect
addon forof_v0.11.2_linux64gcc6_release
. - I've modified the
example-bodies
app to broadcast the closest skeleton over localhost.
- Run
$OF_PATH/addons/ofxAzureKinect/example-bodies
on the Linux PC.
#define HOST "localhost"
#define PORT 12345
- In Isaac Sim, in the Follow Target > UDP Connection panel, press
Connect
to receive skeleton data. - Click Enable Tracking to connect the Motion Behaviors and Gestures to the incoming Skeleton.
HANDS UP
: Go HomeSKELETON ZONE IN ROBOT ZONE
: ApproachHAND ZONE IN TOOL ZONE
: Retract
NONE OF THESE GESTURES ARE CALIBRATED FOR THE IRB8700
Requires TouchOSC app and 8BitDo Lite2 controller
- Connect the Gamepad to the iPad over Bluetooth.
- Open TouchOSC and the
Gamepad_Lite2.tosc
file. - Press Play to verify TouchOSC is receiving Gamepad input.
- In TouchOSC, configure the wifi setting to be on the same wifi network as the Linux PC running Isaac Sim.
- In Isaac Sim, in the Follow Target > OSC Connection panel, add the Linux PC wifi address and press Connect.
The gamepad joysticks and X,Y,A,B buttons modify the
TargetCube
when not in MotionBehavior mode.
LEFT_JOYSTICK
: Moves target in XYRIGHT_JOYSTICK
: Moves target in YZA
: Sends target to Reset PositionX
: Sends target to Home PositionY
: Triggers Approach Mode
In testing the Setup Instructions, I added the omni.osc.core
and omni.osc.source
source extensions, so I didn't need to add the deprecated OSC extenstions from the Extension Manager. Leaving these screenshots of my Extension Manager here for posterity: