Liveblog — CALRG reading group,’Intelligent Tutoring Systems by and for the Developing World’

This week’s CALRG reading group discussed the paper: ‘Intelligent Tutoring Systems by and for the Developing World: A Review of Trends and Approaches for Educational Technology in a Global Context” by Benjamin D. Nye. Below is a liveblog summary of our discussion. Each bullet represents one point made in the discussion (which does not necessarily represent my own views). As always, please excuse any typos or errors as it was written on the fly.

  • I found several issues with the actual research methodology of this paper, and some of that is cultural — it is written by a single American author who ignores any research not in English. There might have been relevant research in other languages, but this is ignored in this paper. In some ways, the paper lends itself to being part of the problem.
  • Sure, this paper could have been much stronger with international collaboration, and inclusion of non-English languages. The author does really even highlight this as a limitation of the paper.
  • The proposed barriers are also not unique to ITS, but rather apply to the use of all technologies in the developing world  — if you have no electricity, you have no technology.
  • One problem I found in this paper is that he never really defines what he means by ITS
  • This paper is mapping the geography of ITS research, but there’s no comparison to other fields. There is no reflection of whether the distribution of papers across the world is normal. You’d probably see the same patterns at any conference, with more of a focus on the US and Europe. You could replace his map with total papers in any discipline and it would look quite similar
  • Some of the barriers he describes are fundamental barriers that need to be developed before you can even consider bringing ITS technologies. If there is no infrastructure in place to support technology, then it is hopeless to try to make it a sustainable option.
  • ITS is not a simple technology to make work in the developing world either
  • Fundamentally the paper makes an assumption that ITS is beneficial to the developing world in the first place
  • Other than brochures from the companies marketing the systems, I’ve never seen anything about how ITS leads to learning gains
  • Link to Project Ceibal (Uruguay, one laptop per child). Their research highlights the need for infrastructure to benefit productivity. For example, simply giving schools good internet connections led to better problem solving.
  • And that research wouldn’t have been picked up in this paper because the author only looked at English publications
  • But maybe what you need is just access to the internet to encourage problem solving? Maybe it’s not related to ITS at all. What is the relationship between the two?
  • Another important question: how does that sustain itself over time? Did the good internet connection lead to better problem solving five years down the line? Or was it about the novelty of it?
  • Recent EDM conference posed an interesting question. ITS researchers have specific questions that they goes after, and their findings keep showing small incremental increases based on these models. The argument has always been that it’s a small effect, but is a lot of people if you bring it to scale. One question asked at the conference: Are we at a point in the community where they need to change what questions we are exploring?
  • Well, the questions you ask are limited by the data you can collect.
  • There was a computer-enhanced learning video from the 70s [edit: anyone have a source for this?], and the same things they said then about new technologies are being said now. If they’re saying the same thing in the 70s, then maybe we need to start asking different questions.
  • Our research is often very tech-led, it’s about what can we do, rather than what do we need.
  • In a way that’s priveledged thinking. We in the UK can be tech-led because we have the infrastructure to support that kind of thinking. This isn’t necessarily possible in some developing countries.
  • Link to Zoran Popović and his work in the ITS domain on math education. He’s considered the environment of question banks, and asked: what would traditional curriculums look like, and what are those pathways? Example: Singapore vs US curriculum styles are two potential paths through these materials. That’s a good use of how ITS informs learning pathways for design.
  • One big question is whether ITS can mark or understand what it is teaching. If so, then maybe it’s not the higher level skills students need to gain, as these sort of skills would not be understandable by a computer
  • From a tutoring perspective, I’ve been playing the Guess the Correlation game, which gives you a scatterplot and you guess the correlation. That feels like a tutoring experience, as it’s low level and grade-able. I view that as focussing on beefing up by fundamentals and small skills/components. This will contribute to fractional gains that will allow you to spend more time on things that you need more time. H
  • It has to be a pretty rich system. Example: at a military school when teaching officers who are going to be generals — they found in six months the staff officers had gamed the system for fantastic marks. They had to trash a 1.5 million system after they figured it out in one semester.
  • The same is true about the GRE (standardised test for postgraduate level study in the US). When the computer marks your written essays, you just have to learn what phrases and structures it views as “good writing” in order to score well.
  • This is another part of the problem of spending money on ITS for the developing worlds — there are inherent problems in the systems themselves, so isn’t it better to use the money for other basic needs (sanitation, etc)?
  • These products are shipped in from the west and there is a financial incentive to that, which means there isn’t a motivation to work with the local environment and stimulating their economy from the ground up
  • One example is Pearson, which is developing common core competencies around the world by getting governments to offload their problems onto their company. It’s a business model.
  • I feel that the best part of this paper was buried: at the end he talks about different models of how these sort of things are created (transferred, homegrown, combo). He begins to make distinctions that the homegrown technologies had different issues or tackled different issues. However, this is relatively hidden at the end of the paper.
  • This paper also highlights the bigger issue of writing culture in research. Obviously things are going on in Russia or China, but it probably is never written up in English. There is a huge barrier to access to research between countries that is obvious in this paper, and the author should reflect on it more.
  • Can someone explain what is the difference between ITS and adaptive learning systems?
  • ITS versus adaptive learning systems: ITS is typically a bank of questions that are mapped to key components of knowledge and ITS gives ‘hints.’ There is no scaffolding, there are hints. Adaptive  learning systems try to have a model of student understanding (i.e. Bayesian knowledge tracing). They try to understand what you know and make a choice on what you go to next.
  • Interesting in both of these systems, there is a notion that if I find the answer in a way the system didn’t expect, it’s cheating
  • But it’s not too different from being in an actual classroom.
  • It seems to deal with the perceived relevancy of what you feel is being taught. When will you ever use it? This will motivate you to “follow the system” or not.
  • One other thing about this paper: How do you determine if the system you’ve exported is actually helping? It is very tricky. Example: I’ve been trying to see how open data is being used in teaching, but no one is making publications on this topic. There are informal accounts, but going about non-published reflections in a rigorous way is tricky.
  • There’s not an internal reflection of what we’re doing either. Often, no one is thinking about ‘what is the point?’ We are reluctant to question the broader social picture and how our research or technology development benefits society.
  • It’s hard to get away from a simple ideology in education: that what we are doing is a ‘good thing’
  • And what needs to be considered is framework are we using to establish ‘good,’ especially in an international context
  • There are hardly any papers that say “this was not a good idea” especially if it was funded
  • One good example of this is Doug Clow and Rebecca Ferguson’s [edit: and Leah Macfadyen and Paul Prinsloo]’s Failathon workshop in the upcoming LAK conference. They are asking academics to come together to talk about what we did wrong and how we can learn from it.
  • The Games Learning and Society Conference also does something similar to this, by getting all stars in the field to show examples of when they had the best set-up and sign for success, but how it went wrong.
  • The military requires advanced degrees to reach certain levels in the hierarchy, but none of the officers will write a paper that’s critical about what they are doing because the guy grading you is the one above you giving you a promotion. The education that they get becomes corrupted and standardises their thinking.
  • But there is a question about providing self determination. There needs to be some level that within our subcultures that can build effective practice.
  • Professionalism means you need to self-police and self-discipline. Unless you critique your own work, you lose the edge of the profession.
  • It’s difficult in most environments to embrace failures
  • Going back to the idea of ITS, how much can they do for areas that are not necessarily fact-based, such as classic literature? How can it support interpretation?
  • Most of these systems are built for STEM people by STEM people.
  • One example of this was from my recent CALRG talk. The computer scientists were trying to find the best location on the map, but the arts people were talking about the process. There was a clear tension. Computer scientists talked about data, art people talked about getting people engaged in the process of interpretation.
  • ITS optimises time on task, getting people through it in a fast and efficient way. How do you get someone through an art gallery in a fast and efficient way? The boundaries of failure and where the models break down is an area that is rich in meaning and exploration. You’re commoditising the systematic mechanical aspect of learning. Where are we fostering the beautiful work that could be going on?
  • Example of going into a museum: watch people with the headset. They follow the sound, and when the headphone tells them to move on. Your experience is over because it wants you to go to the next painting.
  • Has anyone looked at ITS or ADL to copy aspects of what a teacher would do? Rather than pose a solution that gets people through it, but actually copy other aspects that are less obvious. Your starting point should be a teacher — observe them, break down what they do in the classroom.
  • Art Graesser used 10,000 hours of video of tutors to inform his intelligent tutor design. He had a good presentation at LAK 2 years ago where he said we have wonderful theories, but watching 10,000 hours of one on one teaching, it just isn’t there. If you start with the hypothesis that one on one is the best model for learning, looking at 10,000 hours of that experience is good practice to inform these designs.
  • Going back to ITS not supporting things like literature or art — that’s where our culture lies. If we want to transport these to developing cultures, we need to be sure that our systems are not erasing their cultures from education.
  • There is a need to be careful of cultural superiority and imperialism.
  • One of the things that is bothersome of ITS is that it has a decision tree and it decides what the answers are. If I come up with a new answer, it does not accept it, but a professor would engage with my thoughts. There is no incentive in these systems to make intellectual leaps. It is forcing people to fit a mold and that stifles debate.
  • Liz wrote a piece in The Conversation against personalised learning — her argument that the idea behind these systems is making things efficient, but learning is never linear or smooth. You always have roadblocks.
  • Right, these systems are just finding the path of least resistance.
  • My undergrad math experience: There is a clear hierarchy in math classes — the guy second in the program was asked ‘how shall we use computers to better teach the TNB plane?” His response would be to grab the computer by the power cords and swing it to throw it, and as the cords are sticking out, that is the plane. I loved that response. It wasn’t expected, using a physical object for a physics experience, and so was novel and brilliant and I still remember it today. I still think of TNB planes as a computer swinging in the sky.
  • It comes back to is there anything more efficient than someone just talking to you to help you understand? Just by waiting for a computer to turn on, you could have understood it by then in a conversation.
  • What is the expressive capacity of my intelligent tutoring system? Could I identify cultural groupings off of an intelligent tutoring system — then you are landing in the imperialistic approach to education. If the answer is yes, then you are getting to the point where you can connect people that are thinking in a way that is not in the dominant discourse in their culture. Support and facilitate diverse expressing to connect people who can help them down their path to understanding.
  • We think we have to fit in with the culture of who we are trying to help. There is this idea that you can learn from something exotic. You don’t have to remove another culture from it, but you can’t just say it’s all about understanding England. Some sort of balancing act there.
  • Boils down to education about being social engineering — a reflection of the government in power. The conservative government won’t want answers in that databank that won’t support their values. As you move that platform along to different countries, that can be restrictive.
  • We tend to put money into things that are measurable. If you put half of EdTech money into teacher development instead, there would be good results.
  • Both sides have short-comings though (putting all money in teacher training or all money into technology) — need to have a balancing act
  • There is often an ‘ITS-like’ approach to face-to-face instruction  – simply follow the worksheets, and if students do those, they will understand.
  • Professional development programs are often based on hierarchies, that the person at the top of the room has the best ideas.
  • There was a Washington Post article about professional development in Chicago schools. One teacher took a video and the teacher in the front of the room had everyone repeat after him “I will provide students with choice.”
  • The content with the method of delivery needs to match. I don’t think we cut ITS loose, but we should consider the utility of it

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s