This project investigates a real-time motion-capture–driven Mixed Reality (MR) telepresence system for remote physiotherapy training and supervision. The system enables a patient and a physiotherapy expert located in different physical environments to interact through 3D avatars in a shared MR space, with live motion tracking, performance comparison, and adaptive feedback.

The work is structured into three progressive phases, each addressing a critical technical and interaction challenge.

Phase 1 – Real-Time Mocap-to-Avatar Mapping on visionOS

Phase 1 focuses on establishing a robust pipeline for mapping real-time motion capture data to a humanoid avatar rendered on Apple Vision Pro. Initial attempts using WebXR and native RealityKit revealed critical limitations, including the lack of immersive WebXR support on visionOS and the inability of RealityKit to natively ingest external skeleton streams without manual joint-level mapping. To overcome these constraints, the workflow transitions to Unity PolySpatial, which enables native visionOS execution while leveraging Unity’s humanoid animation system. This phase ensures that live mocap data can directly and efficiently drive a skinned avatar with correct skeletal hierarchy, anatomical consistency, and real-time performance, forming the technical foundation of the system.

Phase 2 – Collaborative Mixed Reality Telepresence with Dual Avatars

Phase 1 focuses on establishing a robust pipeline for mapping real-time motion capture data to a humanoid avatar rendered on Apple Vision Pro. Initial attempts using WebXR and native RealityKit revealed critical limitations, including the lack of immersive WebXR support on visionOS and the inability of RealityKit to natively ingest external skeleton streams without manual joint-level mapping. To overcome these constraints, the workflow transitions to Unity PolySpatial, which enables native visionOS execution while leveraging Unity’s humanoid animation system. This phase ensures that live mocap data can directly and efficiently drive a skinned avatar with correct skeletal hierarchy, anatomical consistency, and real-time performance, forming the technical foundation of the system.

Phase 3 – Movement Correlation and Adaptive Feedback for Physiotherapy

Phase 3 introduces motion analysis and adaptive feedback mechanisms to support effective remote physiotherapy training. Patient movements are continuously compared against expert reference motions at the joint and temporal levels to quantify performance deviations. Based on this analysis, the system delivers real-time adaptive feedback through visual, auditory, and spatial cues, guiding the patient toward correct execution. Feedback intensity and modality adapt dynamically to user performance and progression, enabling personalized rehabilitation support. This phase transforms the MR telepresence system from passive visualization into an intelligent, closed-loop physiotherapy training platform.