Liveblog — CALRG reading group, ‘Learner Performance in Multimedia Learning Arrangements’

This week’s CALRG reading group discussed the paper: ‘Learner Performance in Multimedia Learning Arrangements: An Analysis Across Instructional Approaches,’ by Tessa H.S. Eysink, Ton de Jong, Kirsten Berthold, Bas Kolloffel, Maria Opfermann, and Pieter Wouters.  Below is a liveblog summary of our discussion. Each bullet represents one point made in the discussion (which does not necessarily represent my own views). As always, please excuse any typos or errors as it was written on the fly.

  • Why we chose this paper for CALRG: This is firstly a good paper for PhD students as a basis for their literature reviews, as it defines concepts well and uses good references. Interesting bibliography as it didn’t have the ‘usual suspects’ (i.e. papers typically used in UK research). There is a tendency to only cite things in your own area, so this paper is a good grouping of the literature that we don’t see very often, and it references well-respected international journals. This paper also pulls together 4 substantial contributions to the field, which is something that needs to be done in a PhD thesis literature review.
  • This paper also had me thinking about how I evaluate quantitative papers. Things that demonstrate rigorous quantitative research in this paper: a clear write up about the methods (often if authors don’t do that, it’s because they don’t really know the rationale for what they’ve done), pilot tests existed to test for reliability of their instruments, use of the statistical validations like Chronbach’s alpha
  • This paper made me think: could we do this work here in the OU or in IET? After all, we have access to large groups of students working online, such as OU students and over 3 million FutureLearn students
  • This paper comes from a learning sciences field, which tends to take a parallel track as educational technology. In these fields, there is a tendency to do quantitative research in a rigorous way.
  • This paper is a response to the Kirschner, Sweller & Clark (2006) article, which found that guided instruction is superior to the unguided/minimally guided approach. The Eysink et al article works to highlight that actually it’s more complicated than that. It readdresses the notion that unguided approaches and the task of reviewing examples takes longer, but is more effective.
  • What they’ve done in this paper is ask four world leading institutions to design a learning approach and a learning design based on their expertise, and then tested to see which was the best. This is something we can do in our own research as well. What makes this paper so interesting is that each institution probably thought their approach was best, but the analysis found there were subtle differences.
  • Previous research has looked at the sequencing of exploratory learning: should it be instruction followed by exploratory learning, or the other way around? This is a more interesting question to ask than whether you need to learn on your own or be taught — maybe they both are important, but the sequence is the key.
  • This paper is 7 or 8 years old, and the top education journals are now moving towards more of these types of studies.  If we want to play in these big journals, what can we do to follow those kind of approaches?
  • That depends on the journal, as some are still focused on more traditional educational research, but if you want to publish in Computers & Education, you have to do a quantitative or mixed method study
  • I’m not sure about that. I think that Computers & Education, for example, is keen to publish more qualitative work, but they just don’t tend to get qualitative submissions in good standard
  • You could certainly do this study from a qualitative perspective about the user experiences, and it would still be a good study as long as the methods were rigorous
  • A profound variable in learning is the teacher, so they made a conscious decision in this research to go with technology enhanced learning because that removed the instructor. At the OU, we have a lead in these type of studies because we can do things that are entirely online, which don’t have a variable of different instructors
  • We had this conversation this week in the Leverhulme PhD student training.  Maybe the results you are getting in your research are because of the positive presence of the researcher or the teacher. To do rich qualitative work, you need to control for interaction somehow.
  • In this sense, Tessa Eysink is not necessary tech-oriented, but by adopting educational technology, it allowed her to better answer her research questions in this study
  • This research design has me thinking about what could be built upon this study. For instance, there is research was undertaken in two countries, but there was no cross-cultural evaluation. It also would be interesting to see how these approaches led to learning gains over time. The post-test in this paper was taken right after the activity, but it would be interesting to do it later to determine retention.
  • Question: This research and its findings are interesting, but to what extent has this been practically implemented?
  • That’s a real problem in educational research that needs more attention. There are some efforts –> for example, UCL has been running a series of what research says to address problems in education (note: anyone have a source for this?). We have to consider how can we take the best research findings into the classroom. People often say we don’t have good research about education, but that’s entirely wrong now. We have 10-15 years of high quality research, but it isn’t being filtered down to practice. Yes, more research is needed, but what we know now needs to be translatable into the classroom. One reason for this is teacher training, which needs to incorporate research. Some teachers were trained 30 years ago, when our understanding of education was different.
  • That’s not entirely the case, because you do see dodgy research flooding through schools, like learning styles or benefits of hydration.
  • You’re right. Why does dodgy research take off?
  • A lot of it is very appealing and seems intuitive. These ideas are seductive to the broader audience because they make sense, and are easy to engage with.
  • A lot of it in the marketing of research findings
  • There isn’t really an easy route from research into commercialisation. You can come up with a good idea, but there’s no route towards marketing it or getting your research out there.
  • You have to push to get your findings out there to the general public, and few people do that.
  • Don’t underestimate the impact of things like our Innovative Pedagogy reports
  • As a follow up to this study, the authors have done 3 or 4 more studies to consider changing the sequence or intensity, and all of those papers also got published. Then their funding ran out, and the follow up funding ran out due to changes in the Dutch government, then the researchers moved in a different direction.
  • Moolenaar et al (2010) –> even if schools find method X is better than method Y, it’s the head and the informal head of schools that determine the practices adopted
  • There’s always a 5 or 6 year delay from research to practice. The only way to move research forward is to build on other people’s work. Even if you’re more of a qualitative researcher, you should still read these kinds of quantitative articles (and vice versa), in order to understand the groundbreaking results in the field and see how to connect it to your own work, regardless of methods
  • One criticism of this paper is that it is quite long, it could have done with a bit more headlines and summaries to make it more user-friendly
  • It’s interesting to see that it took a year for corrections to get it to published state. It was a long process.
  • It’s also interesting how they looked at learning gains by splitting it into four areas, as one common problem in research is understanding how to define learning gains. However, their different types of learning gains are not defined too well in this article.
  • I’m also intersted in their translation process between German and Dutch, and how rigorous this was. Who did the translations? What was the source language? There could be differences in understanding based on subtle translation differences.
  • It also seems interesting that they opted to do the study in two countries, especially as they have taken steps to account for so many other variables (subject, age, teacher impact, etc). This isn’t really justified in the paper, and the opportunity to do a cross-cultural comparison isn’t used.
  • One problem with conducting this in two different countries: In Dutch education you have statistics in high school, but statistics are not as emphasised in Germany. This means that students may have had different understandings or skills needed to accomplish this task. However, the difficulty in any research is that there are always mitigating cirumstances. There could be millions of reasons why students engaged in these tasks in certain ways, leading to different results. In a PhD thesis, you’d have to write about these confounding factors.
  • This paper does highlight that good research has international connections, and it’s really exciting that we can connect across countries to study different ideas. We should be using these relations more.
  • Yes, but those international connections were understated, and even ignored, in this paper. They had an opportunity for a really rich cross-cultural comparison, but they didn’t use it.
  • This article was a good addition to CALRG, as it takes on the Kirschner, Sweller and Clark article, as we had talked about this article in a session last year. At that time, we said their findings wouldn’t happen in real life and those are the things that Eysink et al specifically addressed. It shows that from this sort of discussion we are having today, you can  go on and build a foundation for your own research, and decide how to investigate ideas properly.
  • Educational technology is a paradigm war in a way, but it’s good to have this debate and see where the tipping point is, to understand where inquiry learning works and under which conditions.
  • This would be really irritating as a teacher, though, as they don’t or can’t read all the research, but see these conflicts between researchers.
  • Another interesting thing in the research is that it tends to look at one design over another. Why does it have to be one method? Why is more research not looking at combinations of different styles?
  • This paper highlights well that what gurus in the field say is a good starting point for your own papers. Then you build on that previous research, even if your findings are different, by verifying what you find with your analysis.
  • Things we need to think about: How can we build on what we currently know, and how do we make people aware of our findings? How do we bring people together to do the kind of research that this paper highlights?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s