•  
• 

Introduction

Decades of research suggest that students learn more effectively through one-on-one human tutoring than through any other known method of instruction. Intelligent tutoring research aims to create intelligent systems that equal or surpass the effectiveness of human tutors, and a particularly promising approach involves natural language tutoring systems which engage students in rich dialogue. While great strides have been made, today's natural language tutoring systems do not approach the strategic robustness and flexibility of human tutors. Many exciting challenges remain within tutorial dialogue systems research. These challenges deal with understanding student natural language input, recognizing student goals and plans, and generating good system dialogue moves, all while balancing the cognitive and emotional considerations involved in student learning.

A particularly rich application area for tutorial dialogue research is introductory computing, because in the U.S. alone, a significant shortfall of computing professionals is projected in coming years. Increasing the number of students who obtain degrees in computing is considered of vital importance. At the postsecondary level, the computing pipeline begins with the introductory computing course.

Project Description

The JavaTutor project (NSF; DRL-1007962) aims to create a natural language dialogue system for introductory computer programming. This project, conducted in collaboration with the IntelliMedia group, is based on the notion that dialogue systems can learn robust, flexible dialogue strategies by observing humans. A key goal is to explore the ways in which cognitive and emotional dialogue strategies contribute to student learning and motivation.

Multimodal Data Collection

The JavaTutor project will collect, annotate, and model a groundbreaking corpus of tutorial dialogue and its associated effectiveness measures with respect to cognitive, emotional, and motivational outcomes. The multimodal data collection includes dialogue, problem-solving actions, and student emotional data from various sources including facial video, Kinect body position sensing, and galvanic skin response readings.

The JavaTutor Intelligent System

Models will be learned from the multimodal tutorial dialogue corpus using leading-edge machine learning approaches. These models will directly define the behavior of the intelligent tutoring system. Several versions of this system will be deployed to assess the impact of cognitive and emotional support for student learning. This project is posed to make contributions to the field of intelligent systems and natural language dialogue systems, and to areas of cognitive science and education.

publications

2016
[40]	Gender Differences in Facial Expressions of Affect During Learning. Alexandria K. Vail, Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C.. Lester. Proceedings of the 24th International Conference on User Modelling, Adaptation, and Personalization, Halifax, Canada, 2016, pp. 65-74. [bib] [doi]
[39]	Predicting Learning from Student Affective Response to Tutor Questions. Alexandria K. Vail, Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 13th International Conference on Intelligent Tutoring Systems, Zagreb, Croatia, 2016, pp. 154-164. [bib] [doi]
[38]	The Affective Impact of Tutor Questions: Predicting Frustration and Engagement. Alexandria K. Vail, Joseph B. Wiggins, Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C.. Lester. Proceedings of the 9th International Conference on Educational Data Mining, Raleigh, North Carolina, 2016, pp. 247-254. [bib]
[37]	Reference Resolution in Situated Dialogue with Learned Semantics. Xiaolong Li, Kristy Elizabeth Boyer. Proceedings of the 17th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2016), 2016, pp. 329–338. [bib]
[36]	Do You Think You Can? The Influence of Student Self-Efficacy on the Effectiveness of Tutorial Dialogue for Computer Science. Joseph B. Wiggins, Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. International Journal of Artificial Intelligence in Education, 2016, pp. 1-24. [bib] [doi]
2015
[35]	The Mars and Venus Effect: The Influence of User Gender on the Effectiveness of Adaptive Task Support. Alexandria K. Vail, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 23nd International Conference on User Modelling, Adaptation, and Personalization, Dublin, Ireland, 2015, pp. 265-276. [bib] [doi]
[34]	Choosing to Interact: Exploring the Relationship Between Learner Personality, Attitudes, and Tutorial Dialogue Participation. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the International Conference on Educational Data Mining, Madrid, Spain, 2015, pp. 125-129. [bib]
[33]	A Tutorial Dialogue System for Real-Time Evaluation of Unsupervised Dialogue Act Classifiers: Exploring System Outcomes. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the International Conference on Artificial Intelligence in Education, Madrid, Spain, 2015, pp. 105-114. [bib] [doi]
[32]	Classifying Student Dialogue Acts with Multimodal Learning Analytics. Aysu Ezen-Can, Joseph F. Grafsgaard, James C. Lester, Kristy Elizabeth Boyer. Proceedings of the International Conference on Learning Analytics and Knowledge (LAK), Poughkeepsie, NY, 2015, pp. 280-289. [bib] [doi]
[31]	Understanding Student Language: An Unsupervised Dialogue Act Classification Approach. Aysu Ezen-Can, Kristy Elizabeth Boyer. International Journal of Educational Data Mining (JEDM), vol. 7 no. 1, 2015, pp. 51-78. [bib]
2014
[30]	The Additive Value of Multimodal Features for Predicting Engagement, Frustration, and Learning during Tutoring. Joseph F. Grafsgaard, Joseph B. Wiggins, Alexandria K. Vail, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), Istanbul, Turkey, 2014, pp. 42-49. [bib] [doi]
[29]	Predicting Learning and Engagement in Tutorial Dialogue: A Personality-Based Model. Alexandria Katarina Vail, Joseph F. Grafsgaard, Joseph B. Wiggins, James C. Lester, Kristy Elizabeth Boyer. Proceedings of the 16th International Conference on Multimodal Interaction (ICMI), Istanbul, Turkey, 2014, pp. 255-262. [bib] [doi]
[28]	Toward Adaptive Unsupervised Dialogue Act Classification in Tutoring by Gender and Self-Efficacy. Aysu Ezen-Can, Kristy Elizabeth Boyer. Workshop on Non-Cognitive Factors & Personalization for Adaptive Learning (NCFPAL) in Extended Proceedings of the 7th International Conference on Educational Data Mining (EDM), London, United Kingdom, 2014, pp. 94-100. [bib]
[27]	A Preliminary Investigation of Learner Characteristics for Unsupervised Dialogue Act Classification. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the 7th International Conference on Educational Data Mining (EDM), London, United Kingdom, 2014, pp. 373-374. [bib]
[26]	Predicting Learning and Affect from Multimodal Data Streams in Task-Oriented Tutorial Dialogue. Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 7th International Conference on Educational Data Mining (EDM), London, United Kingdom, 2014, pp. 122-129. [bib]
[25]	Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the 15th Annual SIGDIAL Meeting on Discourse and Dialogue (SIGDIAL), Philadelphia, Pennsylvania, 2014, pp. 113-122. [bib] [doi]
[24]	Adapting to Personality Over Time: Examining the Effectiveness of Dialogue Policy Progressions in Task-Oriented Interaction. Alexandria Katarina Vail, Kristy Elizabeth Boyer. Proceedings of the 15th Annual SIGDIAL Meeting on Discourse and Dialogue (SIGDIAL), Philadelphia, Pennsylvania, 2014, pp. 41-50. [bib]
[23]	Exploring the Relationship between Self-Efficacy and the Effectiveness of Tutorial Interactions. Joseph B. Wiggins, Joseph F. Grafsgaard, Christopher M. Mitchell, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 2nd Workshop on AI-supported Education for Computer Science (AIEDCS), Honolulu, Hawaii, 2014, pp. 31-40. [bib]
[22]	Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes. Alexandria Katarina Vail, Kristy Elizabeth Boyer. Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS), Honolulu, Hawaii, 2014, pp. 199-209. [bib] [doi]
[21]	Exploring the Relationship Between Task Difficulty and Emotion in Online Computer Programming Tutoring. Joseph B. Wiggins, Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE), Atlanta, Georgia, USA, 2014, pp. 721-721. [bib] [doi]
2013
[20]	Learner Characteristics and Dialogue: Recognizing Effective and Student-Adaptive Tutorial Strategies. Christopher M. Mitchell, Eun Young Ha, Kristy Elizabeth Boyer, James C. Lester. International Journal of Learning Technology (IJLT), vol. 8 no. 4, 2013, pp. 382-403. [bib] [doi]
[19]	Automatically Recognizing Facial Indicators of Frustration: A Learning-Centric Analysis. Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, 2013, pp. 159-165. [bib] [doi]
[18]	Evaluating State Representations for Reinforcement Learning of Turn-Taking Policies in Tutorial Dialogue. Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the SIGDIAL Meeting on Discourse and Dialogue, Metz, France, 2013, pp. 339-343. [bib]
[17]	In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the 14th Annual SIGDIAL Meeting on Discourse and Dialogue, Metz, France, 2013, pp. 324-328. [bib]
[16]	Learning Dialogue Management Models for Task-Oriented Dialogue with Parallel Dialogue and Task Streams. Eun Young Ha, Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the 14th Annual SIGDIAL Meeting on Discourse and Dialogue, Metz, France, 2013, pp. 204-213. [bib]
[15]	When to Intervene: Toward a Markov Decision Process Dialogue Policy for Computer Science Tutoring. Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the 1st Workshop on AI-supported Education for Computer Science (AIEDCS), Memphis, Tennessee, 2013, pp. 450-455. [bib]
[14]	Automatically Recognizing Facial Expression: Predicting Engagement and Frustration. Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the International Conference on Educational Data Mining (EDM), Memphis, Tennessee, 2013, pp. 43-50. [bib]
[13]	A Markov Decision Process Model of Tutorial Intervention in Task-Oriented Dialogue. Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED), Memphis, Tennessee, 2013. [bib] [doi]
[12]	Embodied Affect in Tutorial Dialogue: Student Gesture and Posture. Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED), Memphis, Tennessee, 2013, pp. 1-10. [bib] [doi]
[11]	Unsupervised Classification of Student Dialogue Acts With Query-likelihood Clustering. Aysu Ezen-Can, Kristy Elizabeth Boyer. Proceedings of the 6th International Conference on Educational Data Mining (EDM), Memphis, Tennessee, 2013, pp. 20-27. [bib]
[10]	Modeling Student Programming with Multimodal Learning Analytics. Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE), Denver, Colorado, 2013, pp. 736-736. [bib] [doi]
2012
[9]	Multimodal Analysis of the Implicit Affective Channel in Computer-Mediated Textual Communication. Joseph F. Grafsgaard, Robert M. Fulton, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), Santa Monica, California, 2012, pp. 145-152. [bib] [doi]
[8]	Combining Verbal and Nonverbal Features to Overcome the ‘Information Gap’ in Task-Oriented Dialogue. Eun Young Ha, Joseph F. Grafsgaard, Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the SIGDIAL Meeting on Discourse and Dialogue, Seoul, Republic of Korea, 2012, pp. 246-256. [bib]
[7]	From Strangers to Partners: Examining Convergence Within a Longitudinal Study of Task-Oriented Dialogue. Christopher M. Mitchell, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the SIGDIAL Meeting on Discourse and Dialogue, Seoul, Republic of Korea, 2012, pp. 94-98. [bib]
[6]	Toward a Machine Learning Framework for Understanding Affective Tutorial Interaction. Joseph F. Grafsgaard, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the International Conference on Intelligent Tutoring Systems (ITS), Chania, Greece, 2012, pp. 52-58. [bib] [doi]
[5]	Recognizing Effective and Student-Adaptive Tutor Moves in Task-Oriented Tutorial Dialogue. Christopher M. Mitchell, Eun Young Ha, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the Intelligent Tutoring Systems Track of the International Conference of the Florida Artificial Intelligence Research Society (FLAIRS), Marco Island, Florida, 2012, pp. 450-455. [bib]
[4]	Analyzing Posture and Affect in Task-Oriented Tutoring. Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Eric N. Wiebe, James C. Lester. Proceedings of the Intelligent Tutoring Systems Track of the International Conference of the Florida Artificial Intelligence Research Society (FLAIRS), Marco Island, Florida, 2012, pp. 438-443. [bib]
2011
[3]	Predicting Facial Indicators of Confusion with Hidden Markov Models. Joseph F. Grafsgaard, Kristy Elizabeth Boyer, James C. Lester. Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), Memphis, Tennessee, 2011, pp. 97-106. [bib] [doi]
[2]	An Affect-Enriched Dialogue Act Classification Model for Task-Oriented Dialogue. Kristy Elizabeth Boyer, Joseph Grafsgaard, Eun Young Ha, Robert Phillips, James C. Lester. Proceedings of the International Conference of the Association for Computational Linguistics (ACL), Portland, Oregon, 2011, pp. 1190-1199. [bib]
[1]	Modeling Confusion: Facial Expression, Task, and Discourse in Task-Oriented Tutorial Dialogue. Joseph F. Grafsgaard, Kristy Elizabeth Boyer, Robert Philips, James C. Lester. Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED), Auckland, New Zealand, 2011, pp. 98-105. [bib] [doi]