Skip to main content

Robotics & Vision-Language-Action Models Portfolio

Context: Delivered while working Full-Time (ARGO DATA) + completing MS CS (UT Austin)

This portfolio showcases my journey in robotics and embodied AI, from foundational kinematics to cutting-edge Vision-Language-Action models for humanoid robots. Each project demonstrates deep technical expertise in multi-modal fusion, real-time control systems, and production-scale robotics deployment.

🤖 2025: VLA-Adapter for NVIDIA GR00T Humanoid

Signal: Robotics / Vision-Language-Action Models / SOTA Implementation

🔗 Model on Hugging Face

BridgeAttention Architecture

🎯 Motivation & Problem Statement

The challenge of controlling humanoid robots like NVIDIA's GR00T requires mapping high-dimensional multimodal inputs (vision + language instructions) to precise 43-dimensional action spaces. Existing VLA models struggle with the complexity of humanoid control due to the semantic gap between natural language commands and low-level joint control.

🔧 Technical Architecture: BridgeAttention Mechanism

Core Innovation

  • BridgeAttention Layer: Novel cross-modal attention mechanism that learns to bridge vision-language understanding with robotic action spaces
  • Frozen Backbone Strategy: Leverages pre-trained SigLIP (vision) and Qwen2.5 (language) while training only the policy head
  • Multi-Head Cross-Attention: 8 attention heads with 512D hidden dimensions for rich multimodal fusion
  • Hierarchical Feature Extraction: Uses last 4 hidden layers from both vision and language models for rich representation

Technical Specifications

  • Vision Input: 224×224 RGB → 768D SigLIP features
  • Language Input: Tokenized instructions → 896D Qwen2.5 embeddings
  • State Input: 43D proprioceptive feedback (joint positions, velocities)
  • Action Output: 43D continuous control signals
  • Policy Head: 2-layer transformer with residual connections

⚡ Data Pipeline & Training Optimizations

Dataset Engineering Challenge

Problem: Limited humanoid teleoperation data (original dataset: ~20k samples)
Solution: Engineered augmented dataset with palm pose trajectories and future trace predictions

  • Data Augmentation Pipeline:
    • Forward kinematics computation for palm pose extraction (7D: position + quaternion)
    • Future trajectory prediction (10-step lookahead) for temporal consistency
    • Semantic data filtering based on task complexity and success rates
    • Final dataset: 124k augmented samples with rich temporal annotations
  • Training Infrastructure:
    • Memory optimization for 15GB VRAM constraint using gradient accumulation
    • BF16 mixed precision training for 2x memory reduction
    • Custom data collator for variable-length sequences
    • Distributed training across multiple GPUs with gradient synchronization

📊 Performance Analysis & Evaluation

Aggregate Metrics

  • Overall MSE: 0.0622 (768 evaluation frames)
  • Overall MAE: 0.118
  • Best Task Performance: g1-pick-apple (MSE: 0.0399)
  • Inference Latency: <50ms for real-time control

Per-Joint-Group Analysis

  • Legs (MSE): Left: 0.0040, Right: 0.0055
  • Arms (MSE): Left: 0.0455, Right: 0.0878
  • Hands (MSE): Left: 0.1253, Right: 0.1154
  • Waist (MSE): 0.0002 (highest precision)

Key Technical Breakthrough

The BridgeAttention mechanism successfully learned to map natural language instructions like "Pick the apple from the table" to precise 43D humanoid control signals, achieving sub-0.1 MSE on complex manipulation tasks while maintaining real-time performance.

🚀 2025: Octo Inference API Deployment

Signal: MLOps / Model Serving / Real-time Robotics

🔗 Live Demo

🎯 Motivation & Production Challenge

Deploying large robotics foundation models (Octo-1.5B parameters) for real-time inference presents unique challenges: memory constraints, latency requirements (<100ms for control loops), and concurrent user handling. The goal was to operationalize cutting-edge robotics research into a production-ready API.

🔧 Technical Implementation

Model Optimization Pipeline

  • Quantization Strategy: INT8 quantization for 50% memory reduction while preserving action precision
  • Dynamic Batching: Adaptive batch sizing based on GPU memory availability and request load
  • Model Compilation: TorchScript compilation for 30% inference speedup
  • Memory Management: Custom CUDA memory pool to prevent fragmentation during long-running inference

Infrastructure & Deployment

  • Container Orchestration: Docker with multi-stage builds for minimal image size
  • Auto-scaling: Kubernetes HPA based on GPU utilization and request queue depth
  • Load Balancing: NGINX with sticky sessions for stateful robot connections
  • Monitoring: Prometheus metrics for latency, throughput, and model accuracy tracking

⚡ Performance Optimizations

  • Inference Pipeline:
    • Pre-processing: Image normalization and tokenization (<5ms)
    • Model forward pass: Optimized attention computation (<40ms)
    • Post-processing: Action denormalization and safety checks (<3ms)
    • Total latency: <50ms P95 for real-time robot control
  • Concurrent Handling:
    • Async FastAPI with connection pooling for 100+ concurrent robot sessions
    • Request queuing with priority scheduling (emergency stops get highest priority)
    • Circuit breaker pattern for graceful degradation under high load

Production Reliability Features

  • Health Checks: Continuous model validation with synthetic test inputs
  • Graceful Degradation: Fallback to simpler control policies when primary model fails
  • Safety Constraints: Real-time action validation against robot joint limits and collision boundaries
  • Logging & Debugging: Comprehensive request/response logging for model behavior analysis

🎯 2024: Vision Model + Agent in Simulated World

Signal: Embodied AI / Sim2Real / Agentic Workflow

🔗 Demo Video

🎯 Motivation & Research Challenge

Building autonomous agents that can perceive, reason, and act in complex 3D environments requires solving the perception-to-action problem with minimal latency. The challenge was creating a ReAct (Reasoning + Acting) agent that could achieve high task completion rates while maintaining real-time performance in Unity ML-Agents environments.

🔧 Technical Architecture

Perception Pipeline

  • Multi-Camera Setup: RGB-D cameras with 640×480 resolution at 30 FPS
  • Object Detection: YOLOv8 fine-tuned on simulation objects (mAP: 0.94)
  • Depth Processing: Point cloud generation and voxel grid mapping
  • Scene Graph: Real-time spatial relationship extraction between objects

ReAct Agent Architecture

  • Reasoning Module: GPT-4 with custom prompts for step-by-step planning
  • Action Space: 12-DOF continuous control (navigation + manipulation)
  • Memory System: Episodic memory with attention-based retrieval
  • LangChain Integration: Tool orchestration and workflow management

⚡ Performance Optimization

  • Latency Engineering:
    • Perception processing: <50ms (GPU-accelerated inference)
    • Reasoning step: <100ms (optimized prompts + caching)
    • Action execution: <30ms (direct motor control)
    • Total decision latency: <200ms P50 (target: real-time control)
  • Multi-Threading Architecture:
    • Perception thread: Continuous image processing and object tracking
    • Reasoning thread: Asynchronous planning with action queue
    • Control thread: Real-time motor command execution
    • Memory thread: Background episodic memory consolidation

📊 Evaluation Results

Task Performance

  • Overall Success Rate: 87.3% (across 500 episodes)
  • Navigation Tasks: 94.1% success rate
  • Manipulation Tasks: 82.7% success rate
  • Complex Multi-Step: 78.9% success rate

Performance Metrics

  • Decision Latency: 187ms P50, 245ms P95
  • Planning Efficiency: 3.2 actions per reasoning step
  • Memory Usage: 2.1GB peak (optimized for edge deployment)
  • GPU Utilization: 78% average (RTX 3080)

Sim2Real Transfer Insights

The agent's modular architecture enabled successful transfer to physical robots. Key factors: domain randomization in simulation, robust perception pipelines, and conservative action policies that account for real-world uncertainties.

🦾 2023: Mobile Robotic Arm (ROS + Deep Learning)

Signal: Physical Robotics / Hardware Integration / ROS

🔗 Demo Video

🎯 Motivation & Hardware Challenge

Integrating deep learning perception models with real-time robotic control systems requires solving synchronization, latency, and reliability challenges. The project involved building a custom wheeled robotic arm platform and developing a ROS-based control architecture that seamlessly bridges AI perception with physical actuation.

🔧 Hardware & Software Integration

Hardware Platform

  • Mobile Base: 4-wheel differential drive with encoders
  • Manipulator: 6-DOF robotic arm (custom designed)
  • Sensors: Intel RealSense D435i, IMU, wheel encoders
  • Compute: NVIDIA Jetson AGX Xavier (32GB RAM, 512-core GPU)
  • Actuators: Dynamixel servo motors with position/velocity feedback

ROS Architecture

  • Perception Node: Real-time object detection and pose estimation
  • Planning Node: MoveIt! integration for motion planning
  • Control Node: Joint trajectory execution and safety monitoring
  • Navigation Node: SLAM-based autonomous navigation
  • Coordination Node: High-level task orchestration

⚡ Technical Challenges & Solutions

  • Inverse Kinematics Synchronization:
    • Problem: Real-time IK solving for 6-DOF arm while maintaining smooth trajectories
    • Solution: Custom KDL-based solver with trajectory smoothing and collision avoidance
    • Performance: <10ms IK computation time, 50Hz control loop frequency
  • Perception-Control Integration:
    • Challenge: Bridging 30Hz perception updates with 50Hz control requirements
    • Solution: Kalman filter-based state estimation with motion prediction
    • Result: Smooth tracking of moving objects with minimal jitter
  • Real-time Deep Learning:
    • Model: YOLOv5s optimized for Jetson (TensorRT acceleration)
    • Inference Time: 25ms per frame (640×480 input)
    • Accuracy: 91.2% mAP on custom object dataset

📊 System Performance

Achieved Metrics

  • End-to-End Latency: 180ms from perception to action execution
  • Manipulation Accuracy: ±2mm positioning error for pick-and-place tasks
  • Navigation Performance: Autonomous navigation in cluttered environments
  • System Reliability: 94.7% task completion rate over 200+ test runs
  • Power Efficiency: 4-hour continuous operation on battery

🔬 Research Contributions

  • ROS-DL Integration Framework: Reusable architecture for integrating deep learning models with ROS
  • Real-time IK Solver: Optimized inverse kinematics with collision avoidance
  • Perception-Control Bridge: Novel approach to synchronizing perception and control loops
  • Edge Deployment: Demonstrated feasibility of complex AI on resource-constrained hardware

📐 2021: Robotics Fundamentals & Forward Kinematics

Signal: Fundamentals / C++ / Mathematical Foundations

Challenges Solved

Built Forward Kinematics visualization engine from scratch without high-level robotics libraries.

Technical Depth

  • Mathematical Foundation: Implemented transformation matrices and kinematic chains
  • Visualization: Real-time 3D visualization of robot configurations
  • C++ Implementation: Low-level implementation for performance and understanding
  • Educational Impact: Used for teaching robotics fundamentals

🔬 Research & Innovation Highlights

Key Technical Contributions

  • BridgeAttention Mechanism: Novel architecture for VLA model adaptation
  • Real-time Control: Consistent sub-200ms latency in robotic control loops
  • Sim2Real Transfer: Successful deployment from simulation to physical robots
  • Production MLOps: Scalable model serving for robotics applications

Technologies & Frameworks

  • Robotics: ROS, Gazebo, Unity ML-Agents, PyBullet
  • Deep Learning: PyTorch, Transformers, SigLIP, Qwen2.5
  • Vision: OpenCV, Computer Vision pipelines, Object Detection
  • Hardware: Custom robotic platforms, sensor integration
  • Deployment: Hugging Face Spaces, Docker, FastAPI

This portfolio demonstrates a comprehensive journey from robotics fundamentals to cutting-edge embodied AI, with a focus on practical implementation and real-world deployment.