Kinect with Signs
When learning any language, interactivity plays an important role in the success of the learner’s language acquisition. The teacher relies upon direct and instant feedback channels to assist the learner in developing the necessary listening and production skills. Learning Auslan is no different. In fact, the feedback channel becomes the learner’s friend because access to everyday Auslan is not as simple as turning on the TV or radio.
A unique market exists in our community. Did you know that approximately 90-95% of children who have a hearing loss are born to parents who don't know how to communicate with them? For some, they may have never met an adult with a hearing loss either. This creates some challenging decisions for these parents. We all know that language input at an early age is critical to the child's academic, social and emotional success but how can parents expect to acquire Auslan immediately? Everyone has busy lives and so attending a class every week for a year or more is unfathomable for some parents. Why not bring the tutor to their homes to allow the flexibility they need? Step in the Xbox Kinect!
My Interactive Auslan Coach (MIAC)
Enter modern technology, in particular the Kinect system powered by Microsoft's Xbox. The Kinect uses video capture technology allowing the user to interact with games and activities in the comfort of their own home. It is a uniquely affordable technology which offers the users a way to interact with the programs provided. We have taken this technology to the next level and developed a system to help users learn another language.
Auslan (Australian Sign Language) is a visual language and with the power of the Kinect system we have been able to develop a system to teach sign language with feedback. Current systems for teaching sign language, except in a classroom, cannot provide feedback on the accuracy of the sign the learner has made. Inaccuracies can render a sign incomprehensible. A key element of learning in a classroom is receiving corrective feedback, to ensure that mistakes do not fossilize. Our system can do this. MIAC is like taking a sign language class in your living room. MIAC uses an avatar to display a sign on a large screen, the user makes that sign, this is captured and the image is compared with the stored version of how the sign should be made. If the learner's depiction of the sign is incorrect the system highlights visually the mistakes made. The process is repeated until the sign is correctly made. Below is a diagram of how MIAC works.
The aim is to investigate the potential for using a games console to teach sign language in an engaging interactive form. Our key research questions is:
Can technology such as that designed for games be used to teach sign language?
To address the above question this study is addresing the following sub-questions:
- Q1. Technology: How effective are commercially available games consoles for recording physical actions with sufficient accuracy to provide meaningful feedback particularly for teaching sign language?
- Q2. User Interface: What is the most effective interface to enable intuitive use of a whole body interactive system.
- Q3. Behaviour: What are the modes of behaviour that users assume when learning a physical skill from a computer system that interprets their movements?
- Q4. Computer Aided Learning: What factors influence the accuracy of learning including the rate and sequence of delivery and the importance of the type and timing of feedback?
- Q5. Language Learning: In response to the needs and wants of the deaf, parents of deaf children and professionals in the deafness sector, what are appropriate signs for a basic course in Auslan?
|Dr Kirsten Ellis||Chief Investigator|
|Professor Julie Fisher||Chief Investigator|
|Dr Louisa Willoughby||Chief Investigator|
|Professor Nicole Rinehart||Chief Investigator|
|Dr Jan Carlo Barca||Research Fellow and Project Manager|
|Mr Neil Ray||Partner Investigator|
|Mr Daniel Waghorn||Chief Programmer|
|Ms Madhubhashi Senanayake||Software Developer|
|Ms Shuang Yu||Software Developer|
|Mr Bob Jacobs||Quality Assurance|
|Ms Carol Merlo||Graphical Designer|
We would like to thank our partners for their help and support with this project. In particular Deaf Children Australia who have provided significant supporting funding and critical assistance and advice in the development of MIAC to date.
Organisations that sponsor this research
Room 108, Building 63
Monash University Clayton Campus
For project information please contact us on: kinectWithSigns@monash.edu