Having listened to several cognitive psych presentations on ITS I constantly find myself saying “This is a really cool idea, but…”. Intelligent tutoring systems have the inherent drawback of feeling SO artificial and detached and from reality that I often find it is difficult to draw real world applications from them and apply these findings to the classroom or to further inform instruction. I certainly understand the main premise to give learners access to their own free tutor to help them further their academic goals and understandings of concepts but the software appears so very outdated and without a learning environment that feels relatable to the learner, I’m just not quite sure the learning gains could be that extensive.
As far as the paper on learning and confusion goes, I think the premise is a fantastic one to explore as gaining a deeper understanding on how learners deal with confusion and reconcile this confusion is key to creating better learning materials for students on difficult to learn concepts and topic areas. I think the design of study and the conditions really do a good job of introducing and isolating confusion by presenting conflicting information and leaving it up to the learner to make the ultimate decision. I think that this type of design allows learners the opportunity to learn how to weigh evidence and how to come to ultimate decision when confusion of cognitive conflict arises.
As for replication, what I want to see in this paper and in almost all ITS papers that I read, is a conceptual replication using live tutors or teachers. I always would like to see how the concepts introduced within an ITS program can be introduced in the classroom and see how a learner rectifies conflicting information in a typical everyday situation and learning environment. I feel that ITS is a great first step, but only if what is learned through an ITS study can then in turn be applied to the classroom as ITS does not feel like something that will catch on. If anything, ITS feels like something that is outdated but still very useful in informing practice if used as such.
I can definitely see how confusion can lead to better learning. Creating possible “eureka!” moments after wondering “how can that be?” is great both in its learning benefits and the sense of accomplishment that goes along with it. One study they mention distinguishes between the two, eureka and confusion, but it seems to me that they might be related. I think the reward of eureka is what might make confusion so helpful. I do also agree with their theory that it is the deeper thinking engendered by confusion that makes it useful, but I don’t think we can discount the reward aspect of how overcoming a difficult challenge makes us more confident and want to tackle even more challenging problems. Regardless, this idea of discrepant knowledge needing to be resolved is one I’ve encountered several times in my career as a student as well as an aspiring academician and I thoroughly support its implementation.
I wonder if they set up their experiment like a Turing Test, where the human participant doesn’t know the agents are computers. The way the describe it seems a little vague. They named the two computer agents, but I’m not sure if that was to make the human participant unaware they were learning from a program or for some other reason. It seems moreso that the participant knew both agents were computer simulations. I think the way this experiment would be best conducted is if it was performed without the human participant’s knowledge they were interacting with a computer, rather than real people. My reasoning behind this is that if they know that Dr. Williams and Chris aren’t real, they’ll probably give more credence to what Dr. Williams says, thus minimizing confusion, since they can disregard the conflicting information. I know that, personally, at least, I tend to consider the opinions of professors to have more merit than students and peers, since I assume they have more experience in the field. Not to say I dismiss conflicting peer opinions in real life, but in a computer simulation I think I might look at it and think “hmm, I bet the teacher program is right. After all, why would they make a teacher program be wrong?” The false-false trial may help alleviate this tendency, but the fact that it wasn’t very significant supports my case, I think. I think that’s a good opportunity for replication: making a similar study where the student does not know they are talking to computers.
Overall, I think the concept is great, but the technology just isn’t there yet. I think that we need to have programs that don’t rely on forced choice answers and that can reasonably pass for human for this to be adequately utilized in a learning environment.
Again, another great pair of articles! I feel a little silly admitting this, but I had never really heard about AutoTutor until this article. I know it is 2016 and we should all be used to how sophisticated technology is, but these articles were so fascinating to me. The AutoTutor article provided a great deal of detail and background information, which I really appreciated. I know it is not perfect, but I am so impressed that this technology is making such advances and is able to achieve so much. I am excited to see where this research is in say, 10 years.
I also really enjoyed the article on confusion being beneficial for learning. This is a great concept that I hadn’t give much thought too. I think their goal to link confusion and deep learning was ambitious, and their experimental designs and analysis were very advanced and a little over my head. I found it interesting that they had such low reports of confusion but agree that this has to be at least somewhat due to the self-report. They achieved the difference they wanted, but I wonder if there is a better way to measure confusions? Maybe using a think aloud approach or developing a way to gauge confusion with a test or question presented by the experimenter? They go on to discuss this in the limitations, which was laid out and presented very clearly and effectively, in my opinion. There are pictures that show the faces of the participants as they are working through the experiment, perhaps there is a way to use video recording of facial expression to determine degrees of confusion? It really is a rather difficult construct to measure, but I commend the effort and findings of this article, very interesting.
This week’s articles are really interesting because I was amazed by how much auto tutor can do to promote learning. Graesser’s article provides a nice introduction to what auto tutors do to help students learn and I was surprised to know that auto tutor not only can track students’ knowledge, but also can adapt to the emotional status of them. In addition, the trialogue pattern between tutor agent, peer agent and learners is very impressive and I am really interested in seeing how different interacting patterns would influence students’ learning process and learning outcome in the future.
When reading D’Mello et. al., I really like the way they connect confusion as a knowledge or epistemic emotion with cognitive disequilibrium. Piaget believes that it is a biological drive to produce an optimal state of equilibrium between people’s cognitive structure and the environment, and people can either fitting external reality to the existing cognitive structure or changing internal structures to provide consistency with the external reality. Therefore, people learn from the process of moving from equilibrium to disequilibrium and to equilibrium again. Confusion as a emotion plays an very important role in this process. This article really convinced me that for complex leaning tasks, it can provide students opportunity to a deeper processing when inducing confusion. However, I am wondering whether confusion can help students who are not confused at first place and still not confused when seeing the conflicts between agents, will this process influence these students’ learning outcome? Besides, I think one reason why the accuracy is higher in true-false than false-true is that students are more likely to assume that tutor agent is right compare to peer agent based on their experience in the real life. But on the other hand, it is more challenging to come to the correct answer facing false-true conversation, so I wonder whether this pattern can promote a deeper processing compared to other pattern.
Thinking about replication, first I would like to see if more research could replicate the similar results. It would also be interesting to see whether different learning content can lead to different results. In addition, the participants are undergraduates, so I wonder whether this study can be replicated in other age group.
ITS have come along way in the almost 20 years since they were first introduced to the world. I think they are particularly useful in situations where students need tutors and tutors are not available (which happens quite frequently) in classrooms. Students cannot always meet with human tutors after school because of sports or getting rides home or various other reasons. Teachers often don't have the time after school to meet with every student that needs help and finding individuals with the domain knowledge and teaching ability to fill these rolls can be difficult. ITSs are a great way to meet the demand for these needs. They can also be hugely beneficial to adult learners who would like to learn about a given topic but do not have the resources, time, or transportation to partake in the traditional higher education school system or educational outreach programs. ITSs are a great way to reach a variety of people and foster life-long learning. Getting off my soapbox now....
I think that for the D'Mello article it is important to note the fact that these studies were looking at the domain of scientific reasoning which can be somewhat difficult for individuals who have no experience with research design and methodology. The article indicated that no prior experience in the domain was necessary. I think this makes the subject great for inducing and monitoring confusion but I also think it opens itself up for low domain knowledge learners who might not pick up on the contradiction as readily as high domain knowledge learners. Subsequently, this might lead to answering questions less accurately which the researchers used as an indirect way of inferring confusion. In this case, the learner may not have experienced confusion but their data might read as though they did. The author does note that this is a possibility, but I think it should be examined more carefully. Perhaps a replication could involve looking low domain knowledge learners specifically to see if telling patterns arise in the data. Perhaps low domain knowledge learners need a bit more conversational scaffolding to pick out these contradictions and make judgements based on the given information.
I had not heard of ITS before reading these articles. I was a tutor in college, and I'm a little skeptical that it could replace people as tutors. If it could be programmed to "think" like people, then perhaps it would work. I'm interested in seeing how this technology develops. I liked what was discussed about confusion, and it reminded me of Piaget and the role of conflict in his theories. Substitute conflict for confusion, and you have a compelling way to explain how confusion actually leads to learning.
In terms of replication, obviously it would be helpful to directly replicate to make sure the methods and measures are correct. It would also be interesting to apply ITS to various ages of kids, since even young children need help with schoolwork. Another interesting conceptual replication would be using ITS as a resource for a college student doing an independent study. Since they would be more motivated than someone participating in a study for class credit to actually pay attention and learn, there might be slightly different results.
It's important to understand how ITS works, because our society is integrating technology more and more into schools. This is promising technology, and if it's successful it could be extremely beneficial.
In the Graesser article, I was surprised to learn that the AutoTutor agents themselves do not provide much of a benefit on learning, but it is instead what information is presented to the learner and when it is presented that lead to learning gains. It would seem like having the information spoken to you might have some benefit on learning, perhaps through motivation or engagement. I first approached this topic thinking about it as a kind of replication of human tutors, so the part of the article where it talks about how human tutors may not be the “gold standard” stood out to me. Using human tutors may be what we have done all along, but that does not mean that it is providing the best or most optimal results, just like the discussion we had about undergraduate samples versus MTurk samples.
In the D’Mello et al. article, the method, with all of the trues and falses, was at times challenging to follow, but I think the authors did a fairly good job of explaining everything and keeping things straight for their readers considering the complexity of their methods. I have a concern about asking participants a specific “are you confused right now” type question because it might lead them to overanalyze their confusion, and they may feel like they should be confused. It seems like it gives away some information on the research question, which may lead the participants to behave differently. I do like how they had participants in Experiment 1 go back and view the video recording of themselves and rate their affect at specific points throughout the study, but there is the potential that the participant does not remember that specific point in time well enough to remember their affect. I think using a combination of these two methods might be useful. Participants could be prompted at certain points throughout the study to select all affect options that apply to their current state and perhaps rate the strength on a scale.
While I think AutoTutor is a really neat idea, I'm not sure how I feel about conversing with a dummy. It seems almost too artificial and forced for any real takeaways. While I think it's incredible what technology is capable of, I'm not convinced that much can take the place of a real conversation with another human. I don't know how advanced AutoTutor would need to be to eventually be a sufficient substitute for a human tutor. Super interesting topic to argue about though!
As far as the confusion article goes...I was confused. I think it is VERY important to draw a line between being confused because you don't quite understand a topic and being confused because of ignorance of that topic. To me, those are totally different things! I also think the authors may have ignored the fact that students in experiment studies do not always give their best effort. It becomes very easy for a student to, when "confused," give up and move on or pretend to understand the topic at hand. This also happens to students outside of experiments in their various classes. I remember vividly not understanding chemistry. I would be considered "confused" for the sake of this article, when in reality, I was ignorant to the topic of chemistry and didn't care enough to make an effort to excel. That is very different than being confused in the way the article intends.
I had a few issues with the method of this article. First, 2-2.5 hour sessions?!?! I know the study focuses on confusion, but confusion may be pushed into that "not caring" zone after this timeframe. Also, the majority of the participants had not taken courses/had exposure to the topics that were discussed in the study. To me, it seems like this lack of experience with critical thinking/scientific method/etc. would almost always create confusion due to sheer ignorance.
Overall, I think these articles were neat. I also think personality factors should be taken into consideration when looking at the effect of confusion on learning, as we all know people who take a challenge as immediate defeat. I think it might be unfair to say that confusion can benefit all learners. Just another thought!
Since it is closely relate, this week’s articles were quite interesting. The idea that confusion could improve learning seems rather familiar because it has been tested and approved in many different studies. Moreover, I am sure that many students experienced this phenomena by them selves, including me. When you encounter uncertain and confused information, you would try to go deeper into this information and to scrutinize what exactly it is, and then carefully scan your knowledge with a hope that you can solve this trouble with knowledge you already have. Through these procedure, deeper learning could be possible. Needless to say, what kind of prior knowledge already have is crucial. Because if you don’t have any knowledge related new information, you could not even detect you are wrong or lost. In this sense, it seemed weird that there are just small prior knowledge effects on the result in this article. As the authors mentioned, it seems that it is due to their participants recruitment. I think they should have used students who is only qualified that they are enough prior knowledge related testing subjects.
Second article was also interesting. Actually I am not familiar with intelligent tutoring systems and I have been thinking that it would be helpful only contents are simple enough to programming and only an accessory option for supporting main teaching methods. Different from my stereotypic concept, this article gently introduce how far it has been developed and how it will pave its way. In this article, 3-way conversation between learner (human) and programmed tutor and peer, called trialogues, is a key to significantly enhance learning performance. According to authors, this multi-way conversation makes learners think deeper and help enhancing performance. Despite of many positive aspects, however, I still have some concerns that these type of ‘conversation’ seems too easy to disregarded in real studying situation. Reflecting my own experiences, regardless how well this tutoring system is established and they try to make a ‘conversation with me, if I feel bored and skip it, there is no use of this conversation system. Moreover, sometimes this attempt to interact with learner makes leaners more bored because to learners it is not real conversation, rather reading instructions on the screen. Of course, this story is from when I was a child, so nowadays this system would be much better than back than. But, still, I am unclear how reliable and efficient it is in real situation.
In terms of replication, I can’t tell about auto tutor system since I still have worries that whether or not it could get reached certain level of teaching like human teacher. However, about confusion in learning, I would like to see how prior knowledge affect confusion and correction of self-learning. In self-learning, in order to detect confusion (or wrong information) and make it correct is highly rely on how much prior knowledge related with new information is existed. So, I would recruit participants with more strict standard like screening them out if they don’t have enough knowledge related with test subject, or recruiting the students who only took related classes. Testing methods and procedures could be same, just, if possible, I would like to adopt some neurophysiology technique so that I can see more directly what exactly happens in the brain when we confused during learning and when we solve this confusing problems based on prior knowledge.
Admittedly, I don’t know much at all about intelligent tutoring systems, although I have been involved in some SONA studies on campus where I had to engage in some of them. I thought that the Graesser article made a very compelling argument with regards to how technology can oftentimes trump even expert tutors in terms of efficacy. They’re able to assess things from an objective perspective and “learn” about the person that is interfacing with it.
One deficit that I think intelligent tutoring systems fail to take into account is facial expression and the unique human capacity to assess emotional states in different speech intonation. Self-report measures inherently contain error in some of these cases, and someone’s pre and post test answers are not necessarily reflective of valid measurments.. One of the first things we learn about in Piagetian ideology is the idea of conflict and disequilibrium. The article that discussed the role of confusion in complex learning processes nicely complements this. It discussed using self-report pre and post- tests to assess varying domains of confusion. It never occurred to me that confusion could in some cases be induced in order to elicit a certain cognitive state conducive for better learning outcomes.
With regards to replication, I wonder if there could be some sort of physiological or behavioral assessment of confusion, rather than relying on self-report measures. It would be nice to see that being conceptually replicated.
Both articles this week presented some interesting points of consideration in understanding learning. AutoTutor certainly seems to have its benefits; however, as a highly-biased former tutor, I do believe there is something lacking in computer-based interactions. I have not used AutoTutor, so it may be that this system, or related programs, are more complex and, as discussed, do address some of the shortcomings of many computer-aided programs. I do feel that the acknowledged weakness, however, that students cannot effectively ask questions is a very serious one. Additionally,. a great deal of learning, from my anecdotal experience, is assisted by providing analogies, metaphors, or examples which a learner can easily relate to. While AutoTutor can be designed to predict expected misconceptions and concerns, delivering these in the most efficient manner possible, as well as adapting if one approach is unsuccessful, seems beyond practical reach now. This requires some learning about the learner. While there is incredible potential for the Auto Tutor and other intelligent learning technologies, it seems that (again, in my biased and largely uninformed opinion) these technologies are incredible supplemental resources, or maybe viable alternatives to textbooks, but I am skeptical about the possibility of them effectively assuming a lead role in education.
The article on confusion was very interesting and I completely agree that confusion is essential for deeper understanding. I am familiar with conflict detection and dual-process theories of reasoning and this research seems to fit nicely with those ideas. For that reason, it was slightly surprising to me that no reference was made to some of the more pivotal researcher in that area (e.g., Wim De Neys, Steve Sloman). It also suggests that there may be more subtle nuance to this literature than one could extract from a single study (e.g., dual-process focus more heavily on reasoning, rather than memory). It did reflect a similar concern regarding measurement in cognitive science as some of this literature, however. In an (understandable) attempt to overcome the shortcomings of self-reported emotion to more carefully measure confusion, the researchers looked at responses to the forced-choice questions. This is slightly problematic. Using participants’ performance to infer processing relies on the assumption that processing affects performance: the very idea being tested. That is, they assume incorrect responses are the result of confusion and then use this to test whether confusion benefits learning. It is entirely plausible and, in fact, quite reasonable to assume that people answer incorrectly without confusion (some may even be very confident, e.g., Dunning-Kruger). It is also possible that participants are simply responding in line with the tutor in those cases where the tutor is incorrect. In this study, the case being made is very logical and matches what one would expect. However, it does tap into the issue that frequently comes up in psychology: is performance truly the best way to make inferences about processing? In this case, it does not make a great deal of difference. Reliable effects can be produced using certain manipulations. This is not a bad thing: if something works, one should keep doing it. I present no perfect solution to this problem, either. However, it does lead to some of the issues seen with replication (particularly conceptual replications) as the process behind the effect is not fully understood. In this instance, I think a lot could be gained by looking at the overlap between confusion/conflict detection in education, reasoning, and the neuroscientific/biological investigations of this domain (though, I am sure the exclusion of these massive bodies of literature from this study are more the result of practical limitations).
This week’s reading contained a lot of information to take in and analyze! I will do my best to articulate questions and comments the experiments raised. Art’s article was a nice summary of AutoTutor’s capabilities and I wish I would have read it first instead of second, but I got the information either way.
To begin, the second experiment in the D’Mello paper did a good job at addressing issues from the first experiment. At first while reading, I thought the introduction of the online self-report feedback on confusion was great. However, as we saw, it seems that any self-report of confusion is low (only about 25% of trials in the second experiment), despite confused facial expressions or incorrect responses. While I’m sure it would be quite the task, implementing some sort of facial reading software might be a better measure of confusion than self-report, and could ultimately be used in real-time feedback. However, I’m not well-versed in the area and don’t know at what stage that technology is at and how difficult it would be to integrate it into the existing programming.
The main question these experiments left me asking was this: Would these results hold up in the long term? I think it would say a lot towards confusion as being beneficial for learning if these differences remained at a follow-up time (and still be good to know if it didn't), and would be simple to implement in a study like this.
This week reading is very interesting. The author did nice job by showing how auto tutor works which is close to real classroom teacher. The author called this pedagogy as trialogues. However, on the top of overall discussion I am thinking is it possible at certain point other new technology could take over the place of AutoTutor(e.g. iPad and android application or apps)? Because those application almost work same as AutoTutor(i.e.direction, feedback). And the author did mention that The AutoTutor project was launched in 1997 when apps store starts its massive journey around 2009/10 when first iPad lunch. But I feel like there are still a lot to improve in AutoTutor and where all those iPad application are free and open source. The author highlights the status of AutoTutor’s dialogue moves, learning gains, implementation challenges, differences between human and ideal tutors, and some of the systems that evolved from AutoTutor. Current and future AutoTutor projects are investigating three- party conversations, called trialogues, where two agents (such as a tutor and student) interact with the human learner. AutoTutor is a pedagogical agent that holds a conversation with students in natural language and simulates the dialogue moves of human tutors as well as ideal pedagogical strategies. The author study revealed that human tutors revealed that they are prone to follow principles of conversational politeness so they are reluctant to give negative feedback when a student’s contribution is incorrect or vague. Accurate feedback sometimes needs to be sacrificed in order to promote confidence and self- efficacy in the student. This pedagogical agent have helped students learn compared to various control conditions. In the case of AutoTutor, reports covering multiple studies have reported average learning gains significantly (e.g. vary between 0.3 sigma). The methodology of the study looks sounds to me as the author mentioned conducted a series of experiments that attempted to identify the features of AutoTutor that might account for improvements in learning. But for more general reader the author could say more in terms of methodological approach for replication. It is very interesting to see the robustness of the core conversation mechanisms in both AutoTutor and most human tutoring. The author mentioned that many of the core conversation mechanisms in AutoTutor are similar to human tutoring. It is very interesting to see how AutoTutor could change the classroom teaching learning pedagogy in near future.
Considering I've been here at the university for almost 5 years, I have read a lot of AutoTutor papers. I'm not sure what there is to discuss about the Graesser 2016 paper since it surveyed a ton of stuff over the last 15 years. It is a neat tool that has a lot of applications now with the rise of MOOCs.
The paper on confusion was interesting. Having read the intro, I was split on whether confusion while learning should be considered a good or bad thing. I thought that I had previously read that a state of confusion or disbelief was a step in the process of learning something, so it is essentially required in order to learn anything substantial (as opposed to learning something simple, like the name of an object). Is that right? Then according to this, people think there should be a quick intervention. It seems like both sides make sense (confusion is good, but yes there needs to be an intervention), but maybe when to intervene is the question.
I'm confused (ha) as to why purposely inducing confusion would cause more learning though! Maybe that compels people to resolve their own confusion (not knowing something vs knowing that you dont know something).
In terms of their method, I thought the "retrospective affect judgment protocol" was nifty. I've done retrospective stuff before and it gets dicey, but I think what they did is straight forward (in other type of retro type stuff, participants seem to unintentionally make up narratives)
After reading the results and discussion, I don't think the fundamental idea is that confusion induces learning, but rather that they should be challenged to the point of confusion, since that is a sign of learning. Looking forward to the discussion this piece!
Reading through the D’Mello article, I found the use of confusion to stimulate deep-learning interesting. As mentioned in the article, confusion is often subsumed by theories of disequilibrium, such as Piaget’s, though rarely directly addressed. Thinking this over, it would seem at times that I have seen researchers address confusion and disequilibrium as virtually synonymous. However, this could pose a number of issues for replication. Namely, conflating confusion and disequilibrium would muddle the characteristics and classification of each (e.g. confusion is more an emotion, whereas disequilibrium is a much broader state, overarching confusion, which in its earliest manifestations does not even necessarily need to be cognitive in the way one might conceive it at later stages). The article seemed to do a very thorough job delineating how confusion was being addressed and how it was being specifically manipulated in order to create a certain type of confusion (that is task-related rather than peripheral). This would seem particularly useful if wanting to replicate the effects of confusion on learning, particularly applied to different tasks. Further, I believe this did a good job at allowing for the provision of empirical evidence applicable to broader learning theories that at times suffer from issues with measurement and evaluation.
In terms of the Graesser article, I found the development and use of AutoTutor fascinating. While I had a rudimentary understanding of AutoTutor, I was unaware of the many different ways in which it had been applied. In terms of effectiveness, it was somewhat surprising that AutoTutor was roughly as effective as expert tutors. I would have thought it would have been more akin to novice or intermediate tutoring, given common complaints of students regarding a range of online learning-related systems. In terms of replication, it would seem that the beneficial impact of AutoTutor has been well replicated in a range of different disciplines with greatly different content. However, I would be interested to see the longterm influence of AutoTutor on learning. While it may be effective at teaching certain modules over a relatively short period of time, I wonder what happens in more long-term scenarios. For instance, if a student were to use the system regularly over an entire semester or school year, I wonder if some of the benefit might drop off. Specifically, the longer one must be engaged with certain material and methods of learning, the more consistently interesting and engaging the learning context must be. Further, AutoTutor might lack some of the more social and collaborative influences that contribute to complex learning over extended periods of time. I wonder if such potential deficits might be reconciled through integrating live social interactions in with some of the automated tutoring components to create a more comprehensive, class-like experience.
Having listened to several cognitive psych presentations on ITS I constantly find myself saying “This is a really cool idea, but…”. Intelligent tutoring systems have the inherent drawback of feeling SO artificial and detached and from reality that I often find it is difficult to draw real world applications from them and apply these findings to the classroom or to further inform instruction. I certainly understand the main premise to give learners access to their own free tutor to help them further their academic goals and understandings of concepts but the software appears so very outdated and without a learning environment that feels relatable to the learner, I’m just not quite sure the learning gains could be that extensive.
ReplyDeleteAs far as the paper on learning and confusion goes, I think the premise is a fantastic one to explore as gaining a deeper understanding on how learners deal with confusion and reconcile this confusion is key to creating better learning materials for students on difficult to learn concepts and topic areas. I think the design of study and the conditions really do a good job of introducing and isolating confusion by presenting conflicting information and leaving it up to the learner to make the ultimate decision. I think that this type of design allows learners the opportunity to learn how to weigh evidence and how to come to ultimate decision when confusion of cognitive conflict arises.
As for replication, what I want to see in this paper and in almost all ITS papers that I read, is a conceptual replication using live tutors or teachers. I always would like to see how the concepts introduced within an ITS program can be introduced in the classroom and see how a learner rectifies conflicting information in a typical everyday situation and learning environment. I feel that ITS is a great first step, but only if what is learned through an ITS study can then in turn be applied to the classroom as ITS does not feel like something that will catch on. If anything, ITS feels like something that is outdated but still very useful in informing practice if used as such.
I can definitely see how confusion can lead to better learning. Creating possible “eureka!” moments after wondering “how can that be?” is great both in its learning benefits and the sense of accomplishment that goes along with it. One study they mention distinguishes between the two, eureka and confusion, but it seems to me that they might be related. I think the reward of eureka is what might make confusion so helpful. I do also agree with their theory that it is the deeper thinking engendered by confusion that makes it useful, but I don’t think we can discount the reward aspect of how overcoming a difficult challenge makes us more confident and want to tackle even more challenging problems. Regardless, this idea of discrepant knowledge needing to be resolved is one I’ve encountered several times in my career as a student as well as an aspiring academician and I thoroughly support its implementation.
ReplyDeleteI wonder if they set up their experiment like a Turing Test, where the human participant doesn’t know the agents are computers. The way the describe it seems a little vague. They named the two computer agents, but I’m not sure if that was to make the human participant unaware they were learning from a program or for some other reason. It seems moreso that the participant knew both agents were computer simulations. I think the way this experiment would be best conducted is if it was performed without the human participant’s knowledge they were interacting with a computer, rather than real people. My reasoning behind this is that if they know that Dr. Williams and Chris aren’t real, they’ll probably give more credence to what Dr. Williams says, thus minimizing confusion, since they can disregard the conflicting information. I know that, personally, at least, I tend to consider the opinions of professors to have more merit than students and peers, since I assume they have more experience in the field. Not to say I dismiss conflicting peer opinions in real life, but in a computer simulation I think I might look at it and think “hmm, I bet the teacher program is right. After all, why would they make a teacher program be wrong?” The false-false trial may help alleviate this tendency, but the fact that it wasn’t very significant supports my case, I think. I think that’s a good opportunity for replication: making a similar study where the student does not know they are talking to computers.
Overall, I think the concept is great, but the technology just isn’t there yet. I think that we need to have programs that don’t rely on forced choice answers and that can reasonably pass for human for this to be adequately utilized in a learning environment.
Again, another great pair of articles! I feel a little silly admitting this, but I had never really heard about AutoTutor until this article. I know it is 2016 and we should all be used to how sophisticated technology is, but these articles were so fascinating to me. The AutoTutor article provided a great deal of detail and background information, which I really appreciated. I know it is not perfect, but I am so impressed that this technology is making such advances and is able to achieve so much. I am excited to see where this research is in say, 10 years.
ReplyDeleteI also really enjoyed the article on confusion being beneficial for learning. This is a great concept that I hadn’t give much thought too. I think their goal to link confusion and deep learning was ambitious, and their experimental designs and analysis were very advanced and a little over my head. I found it interesting that they had such low reports of confusion but agree that this has to be at least somewhat due to the self-report. They achieved the difference they wanted, but I wonder if there is a better way to measure confusions? Maybe using a think aloud approach or developing a way to gauge confusion with a test or question presented by the experimenter? They go on to discuss this in the limitations, which was laid out and presented very clearly and effectively, in my opinion. There are pictures that show the faces of the participants as they are working through the experiment, perhaps there is a way to use video recording of facial expression to determine degrees of confusion? It really is a rather difficult construct to measure, but I commend the effort and findings of this article, very interesting.
This week’s articles are really interesting because I was amazed by how much auto tutor can do to promote learning. Graesser’s article provides a nice introduction to what auto tutors do to help students learn and I was surprised to know that auto tutor not only can track students’ knowledge, but also can adapt to the emotional status of them. In addition, the trialogue pattern between tutor agent, peer agent and learners is very impressive and I am really interested in seeing how different interacting patterns would influence students’ learning process and learning outcome in the future.
ReplyDeleteWhen reading D’Mello et. al., I really like the way they connect confusion as a knowledge or epistemic emotion with cognitive disequilibrium. Piaget believes that it is a biological drive to produce an optimal state of equilibrium between people’s cognitive structure and the environment, and people can either fitting external reality to the existing cognitive structure or changing internal structures to provide consistency with the external reality. Therefore, people learn from the process of moving from equilibrium to disequilibrium and to equilibrium again. Confusion as a emotion plays an very important role in this process. This article really convinced me that for complex leaning tasks, it can provide students opportunity to a deeper processing when inducing confusion. However, I am wondering whether confusion can help students who are not confused at first place and still not confused when seeing the conflicts between agents, will this process influence these students’ learning outcome? Besides, I think one reason why the accuracy is higher in true-false than false-true is that students are more likely to assume that tutor agent is right compare to peer agent based on their experience in the real life. But on the other hand, it is more challenging to come to the correct answer facing false-true conversation, so I wonder whether this pattern can promote a deeper processing compared to other pattern.
Thinking about replication, first I would like to see if more research could replicate the similar results. It would also be interesting to see whether different learning content can lead to different results. In addition, the participants are undergraduates, so I wonder whether this study can be replicated in other age group.
ITS have come along way in the almost 20 years since they were first introduced to the world. I think they are particularly useful in situations where students need tutors and tutors are not available (which happens quite frequently) in classrooms. Students cannot always meet with human tutors after school because of sports or getting rides home or various other reasons. Teachers often don't have the time after school to meet with every student that needs help and finding individuals with the domain knowledge and teaching ability to fill these rolls can be difficult. ITSs are a great way to meet the demand for these needs. They can also be hugely beneficial to adult learners who would like to learn about a given topic but do not have the resources, time, or transportation to partake in the traditional higher education school system or educational outreach programs. ITSs are a great way to reach a variety of people and foster life-long learning. Getting off my soapbox now....
ReplyDeleteI think that for the D'Mello article it is important to note the fact that these studies were looking at the domain of scientific reasoning which can be somewhat difficult for individuals who have no experience with research design and methodology. The article indicated that no prior experience in the domain was necessary. I think this makes the subject great for inducing and monitoring confusion but I also think it opens itself up for low domain knowledge learners who might not pick up on the contradiction as readily as high domain knowledge learners. Subsequently, this might lead to answering questions less accurately which the researchers used as an indirect way of inferring confusion. In this case, the learner may not have experienced confusion but their data might read as though they did. The author does note that this is a possibility, but I think it should be examined more carefully. Perhaps a replication could involve looking low domain knowledge learners specifically to see if telling patterns arise in the data. Perhaps low domain knowledge learners need a bit more conversational scaffolding to pick out these contradictions and make judgements based on the given information.
I had not heard of ITS before reading these articles. I was a tutor in college, and I'm a little skeptical that it could replace people as tutors. If it could be programmed to "think" like people, then perhaps it would work. I'm interested in seeing how this technology develops. I liked what was discussed about confusion, and it reminded me of Piaget and the role of conflict in his theories. Substitute conflict for confusion, and you have a compelling way to explain how confusion actually leads to learning.
ReplyDeleteIn terms of replication, obviously it would be helpful to directly replicate to make sure the methods and measures are correct. It would also be interesting to apply ITS to various ages of kids, since even young children need help with schoolwork. Another interesting conceptual replication would be using ITS as a resource for a college student doing an independent study. Since they would be more motivated than someone participating in a study for class credit to actually pay attention and learn, there might be slightly different results.
It's important to understand how ITS works, because our society is integrating technology more and more into schools. This is promising technology, and if it's successful it could be extremely beneficial.
In the Graesser article, I was surprised to learn that the AutoTutor agents themselves do not provide much of a benefit on learning, but it is instead what information is presented to the learner and when it is presented that lead to learning gains. It would seem like having the information spoken to you might have some benefit on learning, perhaps through motivation or engagement. I first approached this topic thinking about it as a kind of replication of human tutors, so the part of the article where it talks about how human tutors may not be the “gold standard” stood out to me. Using human tutors may be what we have done all along, but that does not mean that it is providing the best or most optimal results, just like the discussion we had about undergraduate samples versus MTurk samples.
ReplyDeleteIn the D’Mello et al. article, the method, with all of the trues and falses, was at times challenging to follow, but I think the authors did a fairly good job of explaining everything and keeping things straight for their readers considering the complexity of their methods. I have a concern about asking participants a specific “are you confused right now” type question because it might lead them to overanalyze their confusion, and they may feel like they should be confused. It seems like it gives away some information on the research question, which may lead the participants to behave differently. I do like how they had participants in Experiment 1 go back and view the video recording of themselves and rate their affect at specific points throughout the study, but there is the potential that the participant does not remember that specific point in time well enough to remember their affect. I think using a combination of these two methods might be useful. Participants could be prompted at certain points throughout the study to select all affect options that apply to their current state and perhaps rate the strength on a scale.
While I think AutoTutor is a really neat idea, I'm not sure how I feel about conversing with a dummy. It seems almost too artificial and forced for any real takeaways. While I think it's incredible what technology is capable of, I'm not convinced that much can take the place of a real conversation with another human. I don't know how advanced AutoTutor would need to be to eventually be a sufficient substitute for a human tutor. Super interesting topic to argue about though!
ReplyDeleteAs far as the confusion article goes...I was confused. I think it is VERY important to draw a line between being confused because you don't quite understand a topic and being confused because of ignorance of that topic. To me, those are totally different things! I also think the authors may have ignored the fact that students in experiment studies do not always give their best effort. It becomes very easy for a student to, when "confused," give up and move on or pretend to understand the topic at hand. This also happens to students outside of experiments in their various classes. I remember vividly not understanding chemistry. I would be considered "confused" for the sake of this article, when in reality, I was ignorant to the topic of chemistry and didn't care enough to make an effort to excel. That is very different than being confused in the way the article intends.
I had a few issues with the method of this article. First, 2-2.5 hour sessions?!?! I know the study focuses on confusion, but confusion may be pushed into that "not caring" zone after this timeframe. Also, the majority of the participants had not taken courses/had exposure to the topics that were discussed in the study. To me, it seems like this lack of experience with critical thinking/scientific method/etc. would almost always create confusion due to sheer ignorance.
Overall, I think these articles were neat. I also think personality factors should be taken into consideration when looking at the effect of confusion on learning, as we all know people who take a challenge as immediate defeat. I think it might be unfair to say that confusion can benefit all learners. Just another thought!
Since it is closely relate, this week’s articles were quite interesting. The idea that confusion could improve learning seems rather familiar because it has been tested and approved in many different studies. Moreover, I am sure that many students experienced this phenomena by them selves, including me. When you encounter uncertain and confused information, you would try to go deeper into this information and to scrutinize what exactly it is, and then carefully scan your knowledge with a hope that you can solve this trouble with knowledge you already have. Through these procedure, deeper learning could be possible. Needless to say, what kind of prior knowledge already have is crucial. Because if you don’t have any knowledge related new information, you could not even detect you are wrong or lost. In this sense, it seemed weird that there are just small prior knowledge effects on the result in this article. As the authors mentioned, it seems that it is due to their participants recruitment. I think they should have used students who is only qualified that they are enough prior knowledge related testing subjects.
ReplyDeleteSecond article was also interesting. Actually I am not familiar with intelligent tutoring systems and I have been thinking that it would be helpful only contents are simple enough to programming and only an accessory option for supporting main teaching methods. Different from my stereotypic concept, this article gently introduce how far it has been developed and how it will pave its way. In this article, 3-way conversation between learner (human) and programmed tutor and peer, called trialogues, is a key to significantly enhance learning performance. According to authors, this multi-way conversation makes learners think deeper and help enhancing performance. Despite of many positive aspects, however, I still have some concerns that these type of ‘conversation’ seems too easy to disregarded in real studying situation. Reflecting my own experiences, regardless how well this tutoring system is established and they try to make a ‘conversation with me, if I feel bored and skip it, there is no use of this conversation system. Moreover, sometimes this attempt to interact with learner makes leaners more bored because to learners it is not real conversation, rather reading instructions on the screen. Of course, this story is from when I was a child, so nowadays this system would be much better than back than. But, still, I am unclear how reliable and efficient it is in real situation.
In terms of replication, I can’t tell about auto tutor system since I still have worries that whether or not it could get reached certain level of teaching like human teacher. However, about confusion in learning, I would like to see how prior knowledge affect confusion and correction of self-learning. In self-learning, in order to detect confusion (or wrong information) and make it correct is highly rely on how much prior knowledge related with new information is existed. So, I would recruit participants with more strict standard like screening them out if they don’t have enough knowledge related with test subject, or recruiting the students who only took related classes. Testing methods and procedures could be same, just, if possible, I would like to adopt some neurophysiology technique so that I can see more directly what exactly happens in the brain when we confused during learning and when we solve this confusing problems based on prior knowledge.
Admittedly, I don’t know much at all about intelligent tutoring systems, although I have been involved in some SONA studies on campus where I had to engage in some of them. I thought that the Graesser article made a very compelling argument with regards to how technology can oftentimes trump even expert tutors in terms of efficacy. They’re able to assess things from an objective perspective and “learn” about the person that is interfacing with it.
ReplyDeleteOne deficit that I think intelligent tutoring systems fail to take into account is facial expression and the unique human capacity to assess emotional states in different speech intonation. Self-report measures inherently contain error in some of these cases, and someone’s pre and post test answers are not necessarily reflective of valid measurments.. One of the first things we learn about in Piagetian ideology is the idea of conflict and disequilibrium. The article that discussed the role of confusion in complex learning processes nicely complements this. It discussed using self-report pre and post- tests to assess varying domains of confusion. It never occurred to me that confusion could in some cases be induced in order to elicit a certain cognitive state conducive for better learning outcomes.
With regards to replication, I wonder if there could be some sort of physiological or behavioral assessment of confusion, rather than relying on self-report measures. It would be nice to see that being conceptually replicated.
Both articles this week presented some interesting points of consideration in understanding learning. AutoTutor certainly seems to have its benefits; however, as a highly-biased former tutor, I do believe there is something lacking in computer-based interactions. I have not used AutoTutor, so it may be that this system, or related programs, are more complex and, as discussed, do address some of the shortcomings of many computer-aided programs. I do feel that the acknowledged weakness, however, that students cannot effectively ask questions is a very serious one. Additionally,. a great deal of learning, from my anecdotal experience, is assisted by providing analogies, metaphors, or examples which a learner can easily relate to. While AutoTutor can be designed to predict expected misconceptions and concerns, delivering these in the most efficient manner possible, as well as adapting if one approach is unsuccessful, seems beyond practical reach now. This requires some learning about the learner. While there is incredible potential for the Auto Tutor and other intelligent learning technologies, it seems that (again, in my biased and largely uninformed opinion) these technologies are incredible supplemental resources, or maybe viable alternatives to textbooks, but I am skeptical about the possibility of them effectively assuming a lead role in education.
ReplyDeleteThe article on confusion was very interesting and I completely agree that confusion is essential for deeper understanding. I am familiar with conflict detection and dual-process theories of reasoning and this research seems to fit nicely with those ideas. For that reason, it was slightly surprising to me that no reference was made to some of the more pivotal researcher in that area (e.g., Wim De Neys, Steve Sloman). It also suggests that there may be more subtle nuance to this literature than one could extract from a single study (e.g., dual-process focus more heavily on reasoning, rather than memory). It did reflect a similar concern regarding measurement in cognitive science as some of this literature, however.
In an (understandable) attempt to overcome the shortcomings of self-reported emotion to more carefully measure confusion, the researchers looked at responses to the forced-choice questions. This is slightly problematic. Using participants’ performance to infer processing relies on the assumption that processing affects performance: the very idea being tested. That is, they assume incorrect responses are the result of confusion and then use this to test whether confusion benefits learning. It is entirely plausible and, in fact, quite reasonable to assume that people answer incorrectly without confusion (some may even be very confident, e.g., Dunning-Kruger). It is also possible that participants are simply responding in line with the tutor in those cases where the tutor is incorrect.
In this study, the case being made is very logical and matches what one would expect. However, it does tap into the issue that frequently comes up in psychology: is performance truly the best way to make inferences about processing? In this case, it does not make a great deal of difference. Reliable effects can be produced using certain manipulations. This is not a bad thing: if something works, one should keep doing it. I present no perfect solution to this problem, either. However, it does lead to some of the issues seen with replication (particularly conceptual replications) as the process behind the effect is not fully understood. In this instance, I think a lot could be gained by looking at the overlap between confusion/conflict detection in education, reasoning, and the neuroscientific/biological investigations of this domain (though, I am sure the exclusion of these massive bodies of literature from this study are more the result of practical limitations).
This week’s reading contained a lot of information to take in and analyze! I will do my best to articulate questions and comments the experiments raised. Art’s article was a nice summary of AutoTutor’s capabilities and I wish I would have read it first instead of second, but I got the information either way.
ReplyDeleteTo begin, the second experiment in the D’Mello paper did a good job at addressing issues from the first experiment. At first while reading, I thought the introduction of the online self-report feedback on confusion was great. However, as we saw, it seems that any self-report of confusion is low (only about 25% of trials in the second experiment), despite confused facial expressions or incorrect responses. While I’m sure it would be quite the task, implementing some sort of facial reading software might be a better measure of confusion than self-report, and could ultimately be used in real-time feedback. However, I’m not well-versed in the area and don’t know at what stage that technology is at and how difficult it would be to integrate it into the existing programming.
The main question these experiments left me asking was this: Would these results hold up in the long term? I think it would say a lot towards confusion as being beneficial for learning if these differences remained at a follow-up time (and still be good to know if it didn't), and would be simple to implement in a study like this.
This week reading is very interesting. The author did nice job by showing how auto tutor works which is close to real classroom teacher. The author called this pedagogy as trialogues. However, on the top of overall discussion I am thinking is it possible at certain point other new technology could take over the place of AutoTutor(e.g. iPad and android application or apps)? Because those application almost work same as AutoTutor(i.e.direction, feedback). And the author did mention that The AutoTutor project was launched in 1997 when apps store starts its massive journey around 2009/10 when first iPad lunch. But I feel like there are still a lot to improve in AutoTutor and where all those iPad application are free and open source.
ReplyDeleteThe author highlights the status of AutoTutor’s dialogue moves, learning gains, implementation challenges, differences between human and ideal tutors, and some of the systems that evolved from AutoTutor. Current and future AutoTutor projects are investigating three- party conversations, called trialogues, where two agents (such as a tutor and student) interact with the human learner. AutoTutor is a pedagogical agent that holds a conversation with students in natural language and simulates the dialogue moves of human tutors as well as ideal pedagogical strategies. The author study revealed that human tutors revealed that they are prone to follow principles of conversational politeness so they are reluctant to give negative feedback when a student’s contribution is incorrect or vague. Accurate feedback sometimes needs to be sacrificed in order to promote confidence and self- efficacy in the student. This pedagogical agent have helped students learn compared to various control conditions. In the case of AutoTutor, reports covering multiple studies have reported average learning gains significantly (e.g. vary between 0.3 sigma).
The methodology of the study looks sounds to me as the author mentioned conducted a series of experiments that attempted to identify the features of AutoTutor that might account for improvements in learning. But for more general reader the author could say more in terms of methodological approach for replication.
It is very interesting to see the robustness of the core conversation mechanisms in both AutoTutor and most human tutoring. The author mentioned that many of the core conversation mechanisms in AutoTutor are similar to human tutoring. It is very interesting to see how AutoTutor could change the classroom teaching learning pedagogy in near future.
Considering I've been here at the university for almost 5 years, I have read a lot of AutoTutor papers. I'm not sure what there is to discuss about the Graesser 2016 paper since it surveyed a ton of stuff over the last 15 years. It is a neat tool that has a lot of applications now with the rise of MOOCs.
ReplyDeleteThe paper on confusion was interesting. Having read the intro, I was split on whether confusion while learning should be considered a good or bad thing. I thought that I had previously read that a state of confusion or disbelief was a step in the process of learning something, so it is essentially required in order to learn anything substantial (as opposed to learning something simple, like the name of an object). Is that right? Then according to this, people think there should be a quick intervention. It seems like both sides make sense (confusion is good, but yes there needs to be an intervention), but maybe when to intervene is the question.
I'm confused (ha) as to why purposely inducing confusion would cause more learning though! Maybe that compels people to resolve their own confusion (not knowing something vs knowing that you dont know something).
In terms of their method, I thought the "retrospective affect judgment protocol" was nifty. I've done retrospective stuff before and it gets dicey, but I think what they did is straight forward (in other type of retro type stuff, participants seem to unintentionally make up narratives)
After reading the results and discussion, I don't think the fundamental idea is that confusion induces learning, but rather that they should be challenged to the point of confusion, since that is a sign of learning. Looking forward to the discussion this piece!
Reading through the D’Mello article, I found the use of confusion to stimulate deep-learning interesting. As mentioned in the article, confusion is often subsumed by theories of disequilibrium, such as Piaget’s, though rarely directly addressed. Thinking this over, it would seem at times that I have seen researchers address confusion and disequilibrium as virtually synonymous. However, this could pose a number of issues for replication. Namely, conflating confusion and disequilibrium would muddle the characteristics and classification of each (e.g. confusion is more an emotion, whereas disequilibrium is a much broader state, overarching confusion, which in its earliest manifestations does not even necessarily need to be cognitive in the way one might conceive it at later stages). The article seemed to do a very thorough job delineating how confusion was being addressed and how it was being specifically manipulated in order to create a certain type of confusion (that is task-related rather than peripheral). This would seem particularly useful if wanting to replicate the effects of confusion on learning, particularly applied to different tasks. Further, I believe this did a good job at allowing for the provision of empirical evidence applicable to broader learning theories that at times suffer from issues with measurement and evaluation.
ReplyDeleteIn terms of the Graesser article, I found the development and use of AutoTutor fascinating. While I had a rudimentary understanding of AutoTutor, I was unaware of the many different ways in which it had been applied. In terms of effectiveness, it was somewhat surprising that AutoTutor was roughly as effective as expert tutors. I would have thought it would have been more akin to novice or intermediate tutoring, given common complaints of students regarding a range of online learning-related systems. In terms of replication, it would seem that the beneficial impact of AutoTutor has been well replicated in a range of different disciplines with greatly different content. However, I would be interested to see the longterm influence of AutoTutor on learning. While it may be effective at teaching certain modules over a relatively short period of time, I wonder what happens in more long-term scenarios. For instance, if a student were to use the system regularly over an entire semester or school year, I wonder if some of the benefit might drop off. Specifically, the longer one must be engaged with certain material and methods of learning, the more consistently interesting and engaging the learning context must be. Further, AutoTutor might lack some of the more social and collaborative influences that contribute to complex learning over extended periods of time. I wonder if such potential deficits might be reconciled through integrating live social interactions in with some of the automated tutoring components to create a more comprehensive, class-like experience.