← Projects
completed

Local LLM-Driven 6-DoF Robotic Manipulation with ROS2 and 3D Vision

A ROS2-based robotic manipulation pipeline integrating a locally deployed LLM via Ollama with 3D vision perception to interpret natural-language commands, reason about object-level tasks, and generate executable motion commands for a robotic arm.

Robot Learning ROS2 LLM 3D Vision Manipulation

Implemented a ROS2-based robotic manipulation pipeline that integrates a locally deployed LLM via Ollama with 3D vision perception to interpret natural-language commands, reason about object-level tasks, and generate executable motion commands for a robotic arm for pick-and-place tasks.

Overview

This project bridges large language models with physical robot control by using a locally hosted LLM (via Ollama) as the reasoning backbone. The system takes natural-language task descriptions, grounds them in a 3D scene representation from RGB-D perception, and outputs structured motion primitives executed by a 6-DoF arm through ROS2.

Key Components

  • Natural Language Interface: LLM interprets free-form task instructions and decomposes them into object-level manipulation steps
  • 3D Vision Perception: RGB-D point cloud processing for object detection, pose estimation, and scene understanding
  • ROS2 Integration: Full pipeline from perception to motion execution via MoveIt2 and ROS2 action servers
  • Local Deployment: Entire LLM stack runs on-device via Ollama — no cloud dependency, suitable for robotics lab environments