Kinect with Signs

Hands

When learning any language, interactivity plays an important role in the success of the learner’s language acquisition. The teacher relies upon direct and instant feedback channels to assist the learner in developing the necessary listening and production skills. Learning Auslan is no different. In fact, the feedback channel becomes the learner’s friend because access to everyday Auslan is not as simple as turning on the TV or radio.

A unique market exists in our community. Did you know that approximately 90-95% of children who have a hearing loss are born to parents who don't know how to communicate with them? For some, they may have never met an adult with a hearing loss either. This creates some challenging decisions for these parents. We all know that language input at an early age is critical to the child's academic, social and emotional success but how can parents expect to acquire Auslan immediately? Everyone has busy lives and so attending a class every week for a year or more is unfathomable for some parents. Why not bring the tutor to their homes to allow the flexibility they need? Step in the Xbox Kinect!

My Interactive Auslan Coach (MIAC)

A demonstration of My Interactive Auslan Coach

Enter modern technology, in particular the Kinect system powered by Microsoft's Xbox. The Kinect uses video capture technology allowing the user to interact with games and activities in the comfort of their own home. It is a uniquely affordable technology which offers the users a way to interact with  the programs provided. We have taken this technology to the next level and developed a system to help users learn another language.

Auslan (Australian Sign Language) is a visual language and with the power of the Kinect system we have been able to develop a system to teach sign language with feedback. Current systems for teaching sign language, except in a classroom, cannot provide feedback on the accuracy of the sign the learner  has  made. Inaccuracies can render a sign incomprehensible. A key element of learning in a classroom is receiving corrective feedback, to ensure that mistakes do not fossilize. Our system can do this. MIAC is like taking a sign language class in your living room. MIAC uses an avatar to display a sign on a large  screen, the user makes that sign, this is captured and the image is compared with the stored version of how the sign should be made. If the learner's depiction of the sign is incorrect the system highlights visually the mistakes made. The process is repeated until the sign is correctly made. Below is a diagram of how MIAC works.

MIAC Flow

Research aims

The aim is to investigate the potential for using a games console to teach sign language in an engaging interactive form. Our key research questions is:

Can technology such as that designed for games be used to teach sign language?

To address the above question this study is addresing the following sub-questions:

  • Q1. Technology: How effective are commercially available games consoles for recording physical actions with sufficient accuracy to provide meaningful feedback particularly for teaching sign language?
  • Q2. User Interface: What is the most effective interface to enable intuitive use of a whole body interactive system.
  • Q3. Behaviour: What are the modes of behaviour that users assume when learning a physical skill from a computer system that interprets their movements?
  • Q4. Computer Aided Learning: What factors influence the accuracy of learning including the rate and sequence of delivery and the importance of the type and timing of feedback?
  • Q5. Language Learning: In response to the needs and wants of the deaf, parents of deaf children and professionals in the deafness sector, what are appropriate signs for a basic course in Auslan?
Presenter

Research team

Name Position
Dr Kirsten Ellis Chief Investigator
Professor Julie Fisher Chief Investigator
Dr Louisa Willoughby Chief Investigator
Professor Nicole Rinehart Chief Investigator
Dr Jan Carlo Barca Research Fellow and Project Manager
Mr Neil Ray Partner Investigator
Mr Daniel Waghorn Chief Programmer
Ms Madhubhashi Senanayake Software Developer
Ms Shuang Yu Software Developer
Mr Bob Jacobs Quality Assurance
Ms Carol Merlo Graphical Designer

Our partners

We would like to thank our partners for their help and support with this project. In particular Deaf Children Australia who have provided significant supporting funding and critical assistance and advice in the development of MIAC to date.

Organisations that sponsor this research

Contact

Room 108, Building 63
Monash University Clayton Campus
Wellington Road
Clayton
VIC 3800
Australia

For project information please contact us on: kinectWithSigns@monash.edu

No content

Word Cloud
Hands

When learning any language, interactivity plays an important role in the success of the learner’s language acquisition. The teacher relies upon direct and instant feedback channels to assist the learner in developing the necessary listening and production skills. Learning Auslan is no different. In fact, the feedback channel becomes the learner’s friend because access to everyday Auslan is not as simple as turning on the TV or radio.

A unique market exists in our community. Did you know that approximately 90-95% of children who have a hearing loss are born to parents who don't know how to communicate with them?  For some, they may have never met an adult with a hearing loss either. This creates some challenging  decisions for these parents. We all know that language input at an early age is critical to the child's academic, social and emotional success but how can parents expect to acquire Auslan immediately? Everyone has busy lives and so attending a class every week for a year or more is unfathomable for some parents. Why not bring the tutor to their homes to allow the flexibility they need? Step in the Xbox Kinect!

My Interactive Auslan Coach (MIAC)

A demonstration of My Interactive Auslan Coach

Enter modern technology, in particular the Kinect system powered by Microsoft's Xbox. The Kinect uses video capture technology allowing the user to interact with games and activities in the comfort of their own home. It is a uniquely affordable technology which offers the users a way to interact with  the  programs provided. We have taken this technology to the next level and developed a system to help users learn another language.

Auslan (Australian Sign Language) is a visual language and with the power of the Kinect system we have been able to develop a system to teach sign language with feedback. Current systems for teaching sign language, except in a classroom, cannot provide feedback on the accuracy of the sign the learner  has made. Inaccuracies can render a sign incomprehensible. A key element of learning in a classroom is receiving corrective feedback, to ensure that mistakes do not fossilize. Our system can do this. MIAC is like taking a sign language class in your living room. MIAC uses an avatar to display a sign  on a large screen, the user makes that sign, this is captured and the image is compared with the stored version of how the sign should be made. If the learner's depiction of the sign is incorrect the system highlights visually the mistakes made. The process is repeated until the sign is correctly made.  Below is a diagram of how MIAC works.

MIAC Flow

Research aims

The aim is to investigate the potential for using a games console to teach sign language in an engaging interactive form. Our key research questions is:

Can technology such as that designed for games be used to teach sign language?

To address the above question this study is addresing the following sub-questions:

  • Q1. Technology: How effective are commercially available games consoles for recording physical actions with sufficient accuracy to provide meaningful feedback particularly for teaching sign language?
  • Q2. User Interface: What is the most effective interface to enable intuitive use of a whole body interactive system.
  • Q3. Behaviour: What are the modes of behaviour that users assume when learning a physical skill from a computer system that interprets their movements?
  • Q4. Computer Aided Learning: What factors influence the accuracy of learning including the rate and sequence of delivery and the importance of the type and timing of feedback?
  • Q5. Language Learning: In response to the needs and wants of the deaf, parents of deaf children and professionals in the deafness sector, what are appropriate signs for a basic course in Auslan?
Presenter

Research Team

Name Position
Dr Kirsten Ellis Chief Investigator
Professor Julie Fisher Chief Investigator
Dr Louisa Willoughby Chief Investigator
Professor Nicole Rinehart Chief Investigator
Dr Jan Carlo Barca Research Fellow and Project Manager
Mr Neil Ray Partner Investigator
Mr Daniel Waghorn Chief Programmer
Ms Madhubhashi Senanayake Software Developer
Ms Shuang Yu Software Developer
Mr Bob Jacobs Quality Assurance
Ms Carol Merlo Graphical Designer

Our Partners

We would like to thank our partners for their help and support with this project. In particular Deaf Children Australia who have provided significant supporting funding and critical assistance and advice in the development of MIAC to date.

Contact

Room 108, Building 63
Monash University Clayton Campus
Wellington Road
Clayton
VIC 3800
Australia.

For project information please contact us on: kinectWithSigns@monash.edu

Organizations that Sponsor this Research