Ronnie Taib, PhD student

Menu: PhD research | Professional research | Publications | Contact me

PhD research

I am part-time PhD student, under Prof. Claude Sammut (CSE) and Dr. Fang Chen (NICTA).

Multimodal user interfaces (MMUI) allow humans to interact with machines using a variety of communication channels such as speech or gesture, and conversely machines to convey information to humans using modalities such as visual graphics and text, or synthesised speech. There is a general agreement on the benefits of multimodal user interaction, i.e. it provides more natural, intuitive and cognitively efficient interaction, but current research falls short of methods to design and evaluate related systems. The main objective of this research is to advance the understanding of how do humans interact multimodally, and to provide a methodology for the creation of MMUI, taking into account the individual variability across users, and to automate some of the processes in order to allow industrial deployment.

Adaptive multimodal user interaction, my topic, refers to adapting a human computer interface's input and output processes to a particular person's productions using several modalities. Detecting specific multimodal interaction patterns (MIP) utilised by that person can improve the automatic recognition of the user's input, as well as help convey information back to that user in a more appropriate way. For example, if the person tends to point at objects before uttering the command he or she wants to perform on these targets, the system may exhibit a similar sequential behaviour. But such mimicking may not be desirable all the time, if at all. This research aims to address fundamental issues such as the correlation between semantic content, temporal arrangements and spatial characteristics of MIP. Based on such findings, suitable strategies can be predicted for input fusion and output generation processes. A strong commitment to industry-deployable results underpins this research; hence it will explore the limitations of traditional user-centred design (UCD) methods for MMUI design, especially the lack of evaluation metrics for such systems. It will propose enhancements leading to the elicitation, capture, analysis and exploitation of multimodal interaction patterns from human subjects. Three major research questions will be addressed by this research:

1. Determination of relevant multimodal interaction patterns, in terms of semantic content, types of modalities and temporal relationships;
2. Prediction of optimal input fusion and output generation schemas according to MIP;
3. Definition of evaluation metrics for MMUI systems, in a UCD context.

Professional research

I am also a senior research engineer within National ICT Australia (NICTA), located at the Australian technology park, Sydney. I am working within the Multimodal User Interaction team, on the Smart Roads and Tranport, Multimodal User Interfaces (STaR-UI) project.

Publications

• Ruiz, N., Taib, R., Shi, Y., Choi, E. and Chen, F. Using Pen Input Features as Indices of Cognitive Load. Proc. 9th International Conference on Multimodal Interfaces (ICMI'07), Nagoya, Japan, Nov. 2007, 315-318.
• R. Taib and N. Ruiz, Integrating Semantics into Multimodal Interaction Patterns. Chapter in Machine Learning for Multimodal Interaction. LNCS 4892, H. Bourlard, S. Renals, and A. Popescu-Belis, Eds. Berlin: Springer-Verlag, 2008, 96-107.
• Chen, F., Taib, R., Choi, E., Shi, Y. and Yee, D. User Interface Design for Traffic Incident Management Systems. Proc. 14th World Congress on Intelligent Transport Systems, (Beijing, China, Oct. 2007), paper 3003, on CD.
• Chen, F., Choi, E., Ruiz, N., Shi, Y. and Taib, R. Design and Evaluation of a Multimodal Operator Interface for Traffic Incident Management Systems. Proc. 10th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, (Seoul, Korea, Sept. 2007), on CD.
• Taib, R., Ruiz, N. Wizard of Oz for Multimodal Interfaces Design: Deployment Considerations. Proc.12th International Conference on Human-Computer Interaction (HCII2007), (Beijing, July 2007), 232-241.
• Choi, E., Taib, R., Shi, Y. and Chen, F. Multimodal User Interface for Traffic Incident Management in Control Room. IET Intelligent Transport Systems, vol. 1, no.1, Mar 2007. (2007), 27-36.
• Shi, Y., Ruiz, N., Taib, R., Choi, E. and Chen, F. Galvanic Skin Response (GSR) as an Index of Cognitive Load. Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI'07), (San Jose, April/May 2007). (2007), 2651-2656.
• Shi, Y., Taib, R., and Lichman, S. GestureCam: a smart camera for gesture recognition and gesture-controlled web navigation. In Proc. 9th international conference on control, automation, robotics and vision (ICARCV'06), (Singapore, 5-8 Dec. 2006). (2006), 2049-2054.
• Ruiz, N., Taib, R., and Chen, F. Examining the redundancy of multimodal input. In Proc. 20th annual conference of the Australian computer-human interaction special interest group (OzCHI'06), (Sydney, Australia, 20-24 Nov 2006). (2006), 389-392.
• Shi, Y., Taib, R., Choi, E. and Chen, F. Multimodal Human-Computer Interfaces for Incident Handling in Metropolitan Transport Management Centre. Proc. IEEE 9th International Conference on Intelligent Transportation Systems (ITSC'06), Toronto, Sept. (2006), 554-559.
• Chen, F., Choi, E., Shi, Y., Taib, R., and Yee, D. User interface design of contacts database for road traffic incident management. In Proc. 8th Asia Pacific Intelligent Transport Systems Forum, (Hong Kong, 10-13 July 2006). (2006), on CD.
• Taib, R. and Ruiz, N. Tangible Objects for the Acquisition of Multimodal Interaction Patterns. In Proc. International Conference on Language Resources and Evaluation (LREC'06), (Genoa, Italy, 24-26 May 2006). (2006), 2540-2545.
• Taib, R. and Ruiz, N. Multimodal Interaction Styles for Hypermedia Adaptation. In Proc. International Conference on Intelligent User Interfaces (IUI'06), (Sydney, Australia, 30 January-1 February 2006). (2006), 351-353.
• Taib, R. and Ruiz, N. Evaluating Tangible Objects for Multimodal Interaction Design. In Proc. 19th annual conference of the Australian computer-human interaction special interest group (OzCHI'05), (Canberra, Australia, 21-25 November 2005). CHISIG of Australia, Narrabundah, Australia, (2005), on CD.
• Chen, F., Choi, E., Ruiz, N., Shi, Y., and Taib, R. User Interface Design and Evaluation for Control Room. In Proc. 19th annual conference of teh Australian computer-human interaction special interest group (OzCHI'05), (Canberra, Australia, 21-25 November 2005). CHISIG of Australia, Narrabundah, Australia, (2005), on CD.
• Taib, R., Shi, Y., Choi, E., Chen, F., Sladescu, M., and Phung, N. Multimodal User Interface Facilitating Critical Data Entry for Traffic Incident Management. In Proc. Multimodal User Interaction Workshop, (Sydney, Australia, 13-14 September 2005). CRPIT, (2005), 55-59.
• Chen, F., Choi, E., Epps, J., Lichman, S., Ruiz, N., Shi, Y., Taib, R., and Wu, M. A Study of Manual Gesture-Based Selection for the PEMMI Multimodal Transport Management Interface. In 7th international conference on Multimodal interfaces (ICMI'05), (Trento, Italy, 4-6 October 2005). ACM Press, New York, NY, USA, (2005), 274-281.
• Ahmed, A., Dwyer, T., Forster, M., Fu, X., Ho, J., Hong, S.-H., Koschützki, D., Murray, C., Nikolov, N. S., Taib, R., Tarassov, A., and Xu, K. GEOMI: GEOmetry for Maximum Insight. In 13th International Symposium on Graph Drawing (GD2005), (Limerick, Ireland, 12-14 September 2005). Springer-Verlag, (2005), 468-479.

Contact

Follow this link to contact me