Duration: 8 weeks
My Role: User Experience Researcher
Domain: Education, Feedback
This project explores the use of feedbacks and hints in conversation as a way to help non-native speakers practice conversing in order for them to feel more comfortable socializing with more diverse people. With the research question: “Can instant feedback and hints help international students be better prepared for future conversations and social interactions with native speakers?”, we created an experiment to test how non-native speakers talk to a chat agent and what they gain from it. We split our participants into three groups, one control that involved just a conversation, one with feedback, and one with feedback and hints. All data was collected from the first year international Chinese student population of Carnegie Mellon University. We found that feedback and hints with a chat agent may be a helpful way for students to practice conversations.
A large problem that exists on college campuses around the United States is the siloing of diverse groups, especially between native speakers and non-native speakers. When international students start school, they tend to only interact with other students that speak their same language. While many have been well educated in the English language, they struggle with understanding special native “slang” and other social quirks that are common in day to day interactions. With the application ChitChat, non-native speakers can practice different types of conversation with a chat agent. With these practices, they can receive feedback and hints from the system to help them improve their understanding of native conversation. While this study focuses on helping non-native speakers, it takes an extra focus in international students in college.
Prototype for ChitChat
The question guiding the creation of our system is “Can instant feedback and hints help international students be better prepared for future conversations and social interactions with native speakers?” By providing hints for when the conversation has difficult to understand words or ideas, we hope to help the non-native speaker get past initial comprehension struggles. By being instantaneous, the hints can also allow the participant to learn new information and even apply it later in the conversation. With the instantaneous feedback, corrections to conversation can be made right away. Having these hints and feedback be provided by a system can allow them to more confidently learn and prepare for real life social situations.
Related Work and Theory
Kluger and DeNisi’s study “The Effects of Feedback Interventions on Performance” explains that feedback intervention can beneficial, especially when the intervention is continuous. The study says “In certain situations, FI can yield a large and positive effect on performance. Specifically an FI provided for a familiar task, containing cues that support learning, attracting attention to feedback-standard discrepancies at the task level..” With our system, we are providing feedback to a task that is familiar: normal conversations through a chat app. We give cues to the user when they might need help understanding information and feedback when they have done a good or bad job. The study describes that positive effects of increased task motivation, it may be due to the effect continuous feedback intervention. With our feedback to the users, they are given continuously throughout the conversation.
In Hollan’s “Beyond Being There” study, it is stated that “we must develop tools that people prefer to use even when they have the option of interacting as they have heretofore in physical proximity. We must develop tools that go beyond being there.” What we try to accomplish in ChitChat is to give users similar or even better experience than real world conversation for the same topic. Users can express themselves freely without worrying about judgement or negative reaction from others. This ChitChat App not only serves as an instant feedback tool, but also a window for international students that can broaden their horizons.
We use the Wechat application as our test tool, because it is a very popular application among Chinese international students, and it is easy to form private small chatrooms. We invited Chinese international students who just came to the US or stayed in the US for less than one year to participate the test. The original participants number was 9, but due to time arrangement, there were 6 people participated and completed the test. Below is the description of each step:
Step 1 .
6 people are randomly assigned into three groups – Feedback and Hints Group, Feedback Group, and Control Group. Each group has two chatrooms, and each room is composed by three people (1 participant, Yali, and Julia).
Chat agent Julia sends welcome message and provides two scenarios: Going out to eat and Celebrating holiday. Participant chooses one scenario to start the conversation.
Step 3 .
During the conversation, the Control Group gets no feedback and hints, while the Feedback Group gets feedback from the agent occasionally, and the Feedback and Hints group get both feedback and hints. The differences between feedback and hints are: feedback are given after the person responds/ask questions to the agent, and the hints are given before the person makes response and the pausing time lasts more than 30 seconds. Examples of feedback and hints are shown in the Observation section.
Chat agent Julia wraps up the conversion in 10 minutes, while mediator Yali sends message to end the conversation and sends link of the survey.
Wizard of Oz-ing
Control Group: The participant showed much sadness when the conversation had to end.
Feedback + Hints Group: After the chat agent responded to the participant’s message using native slang “Dope that sounds great!”, a hint was given to explain the meaning of the word “dope” in that specific context. After reading the hint and a few chats later, the participants responds to a situation using dope correctly in context.
In our second study, we tried to create a diverse conversation among participants. We invited the same 6 Chinese students who attend our study last time, and 3 more native speakers to join the conversation as well.
We equally divided 9 participants into three groups, and each group will have one-to-one conversation for a certain topic: travel, movie, and music. Each group is composed by one native speaker and two Chinese students (Figure 3). We also matched the Chinese students based on other dimensions besides race, so that each participant in the same group has at least two different dimensions. Each conversation lasts 10 min, and this time we used Chatbot as the mediator to give feedback and interventions at appropriate times.
We also used “Structure and Messaging Techniques for Online Peer Learning Systems that Increase Stickiness” and the whole concept of the Talkabout systems as a model for our ChitChat program. We modeled the same email that can be seen in figure 1 to help us obtain a perfect attendance rate. Our study proves that the model works very well in terms of improving attendance rate. In our last study, we lost 3 participants due to lateness or not showing up. In this study, all participants showed up on time and one person even rescheduled with us ahead of time to make sure the attendance.
- The results from feedback indicate that instant feedback and hints can help international students be better prepared for future conversations and social interactions with native speakers. Both experimental groups give higher scores (3.5 out of 5) than control group (2.5 out of 5). However, there is no significant difference between the two experiment groups.
- In terms of understandable level, the Feedback and Hints Group gives the highest score (5.0 out of 5) and the Feedback Group gives slightly lower score (4 out of 5), but both experimental groups still give higher scores than control group (3.5 out of 5). So we can draw the conclusion that people understand the chat agent better when hints and feedback are given, and the more feedback and hints are given, the better they can understand the agent.
- One interesting result is the helpful level for feedback. According to the survey data, people rate feedback more helpful (4 out of 5) in the Feedback Group and lower (3.5 out of 5) in the Feedback and Hints Group. It is difficult to explain why feedback is not equally helpful in the experimental groups, perhaps we should collect more data to explore the reason.
- Since our leading research question does not ask how the hints work solely, the effectiveness of hints is not comparable.
Click to see complete report