Onderwijskundig ontwerponderzoek in opmars

“The practical impact of educational research should be a requirement for research done in taxpayer-financed institutions, not only the impact factor of the journal.” This is what dr. Thomas Reeves (University of Georgia) and dr. Susan McKenney (University of Twente) put forward in dialogue with research fellows Josephine Lappia (Rotterdam University) and Donald Ropes (INHolland University of Applied Science). Educational Design Research is a very relevant research approach for scientific researchers who besides scientific relevance also want to make an educational impact.

Participants from schools that are engaged in an Educational Design Research project will encounter several benefits, like context specific educational ‘artefacts’ such as learning materials, arrangements, computer programs et cetera that are field-tested and can be re-used. Teachers learn how to co-design and implement the innovative educational materials and arrangements. And for the researcher, the several case studies of which Education Design Research consist, will lead to guidelines about how to design a solution to improve the intended innovation. Apart from this practical relevance, Educational Design Research also aims to extend generalised scientific knowledge about the intended innovation and how it can be applied within a next related educational context.

For Josephine Lappia and Donald Ropes, both research fellows within universities of applied science, this research approach is a very suitable approach for their PhD research projects on respectively Work Related Learning Arrangements and Communities of Practice. For over three years, they have participated in the Design Based Research Group under the aegis of Professor Joan van Aken (Technical University of Eindhoven) and Dr. Daniel Andriessen (INHolland University).

Educational Design Research is more and more used by PhD students at several research universities in the Netherlands. ICO – Research School for Educational Science – offered this academic year for the second time a 5-days course about Educational Design Research that was attended by 25 PhD-students who are to obtain their doctorates within the next four years. Time to speak to two outstanding teachers of this course: Susan McKenney of the University of Twente and Thomas Reeves of the University of Georgia (USA).

Josephine Lappia: I pose this question to you in your roles as teachers, as you are both besides being researchers also teachers. If teachers are attracted to design educational materials, they often also have another or more general passion for design. How is that for you?

Thomas Reeves: It is funny that you are asking me that because I first enrolled at Syracuse University in 1974 hoping to learn how to develop educational television programs because I was impressed by the broadcastings of Sesame Street. This program was only about five years old at that time and I wanted to create such educative materials for young children myself. But then I met Professor Don Ely and he persuaded me to come over to the Instructional Technology program. I now have been working in this field for the last 35 years.

Susan McKenney: Well for me, I did my Masters in Educational Sciences at Twente University, but if I would have the time for a second career I definitely would be in graphic design.

Josephine Lappia: When you are supporting teachers in their work to design educational materials, would you say to teachers that adopting a ‘design goal’ in their developmental work is a good thing to do?

Susan McKenney: As much as possible, I would start by asking teachers what problems they encounter in their work and that they are willing to resolve. Then I will try to link those issues to my research themes. I do not have a solution beforehand. As much as possible, participants are involved in the design of the solution because that is a meaningful and practical way to work.

Josephine Lappia: So in working that way, can you say that you are the creator and the teachers the co-creators?

Susan McKenney: It depends on the stage the designing teachers are in, if they are willing to design solutions by themselves I will scaffold them to do so. But with teachers designing all by themselves without scaffolding I do not have very good experiences, because their designs often lack validation: little or no references to practical or scientific insights at all. Often, the creative energy overrides their own pedagogical content knowledge when they are in design mode. In general, the motivation to design by themselves is not very widespread among teachers, so I usually start with making clear to the teachers what they will gain by being cooperative in the design research project.

Josephine Lappia: In September 2008, I visited several school centred initial teacher training schools (SCITT’s) and the Centre for Learning and Teaching of Newcastle University in the UK. About fifteen years ago, some of the secondary schools there started a process in which teachers were involved in several action research projects supervised by the Centre for Learning and Teaching. The head of that Centre admitted that although the teachers involved in the research projects had liked it a lot, the scientific and practical progress that had been made could not really be pointed out clearly.  Dutch Institutes for Initial Teacher Training and pilot-schools involved in programs like ‘Academic Schools’ and ‘Teaching at School’ (Opleiden in de School) are interested in these kind of references because they want to establish a robust research agenda. Therefore, it is important to ‘forecast‘ the answer to the question: how to ensure that we can point out our scientific and practical progress over the next fifteen years?

Susan McKenney: Was he disappointed in his hope for educational relevance or because of the lack of scientific impact? If it is the last, I would think it is up to him. If the results were lacking educational relevance then that is another situation. It reminds me of a large action research project in San Francisco called the BASRC. Pupil performances where low in that area and this initiative set the bar high in terms of its goals for a positive impact. The project offered schools large amounts of support in implementing the action research projects, starting by individual learning gap analysis. Because the program had so much to offer, the program leaders only invited schools eager to participate. The most remarkable outcome of the program was a huge change in the mindset of teachers in how they can and are capable in influencing pupil performances. I think this is indeed a big result.

Thomas Reeves: What I expect is that in the project of the UK – that I am not familiar with – they used many ‘one turn cycles’ as might be the case in some action research programs. If you do research like that, then of course you have difficulties in transferring knowledge from one cycle to another.

Josephine Lappia: Yes, indeed they reported that teachers were very enthusiastic about their own research, but there was not always as much enthusiasm when it came to the exchange of ideas and knowledge with others, because they did not turn up.

Thomas Reeves: So in that case, adding a design orientation within collaborative school based research projects might be a good idea. In Educational Design Research, you use theory in an effort to better understand the problem before starting the research projects. You re-examine theory in the light of the practical findings as well as additional research findings related to design. Afterwards, it makes transfer of knowledge from one school to the other much easier. In the State of Georgia, we have the Program for School Improvement project in which more than hundred professional schools participate. Results are shared with other schools and those programs work well. But of course, I also know examples of individual outstanding teachers who do not choose to share their expertise with others. Such a teacher might have a passion for Shakespeare and be able to inspire students in all subjects ranging from Maths, through Biology to Language, by providing links to Shakespeare. However, when such an outstanding teacher retires, their educational approach usually turns out to be not transferable because it has not been subjected to research and shared more widely.

Donald Ropes: I have been doing educational design research for three years for my dissertation. This has had some major consequences concerning among other things acceptance of my work. For example, when I go to publish my research, I naturally look to journals that are fitting to my subject and where my peers publish. However, I do not see any articles that reflect an Educational Design Research approach. Most published studies follow the traditional presentation format: theoretical framework, methodology, data analysis and discussion. Traditional reporting formats differ from how Educational Design Research is reported because Educational Design Research follows a different research process. This leads to confusion among editors – and sometimes even my dissertation mentor – because my research cannot really be framed in the traditional manner. This leads me to my question: there seems to be but a few dissertations applying an Educational Design Research approach. The same is true for published studies. Are there any strategies that a design researcher can use to get a design-based research study approved and eventually published?

Susan McKenney: Thomas and I have been involved in several dissertations deploying an Educational Design Research approach. One possible strategy is to write a dissertation based on articles. In other words, break the dissertation into publishable sections. For example, the theoretical framework, individual iterations, and a reflection on the process and outcomes. One of my Ph-D students is doing this now and it seems to work for getting design research in peer-reviewed journals. However, although I have published several articles from my dissertation – which was design based – I know it can be frustrating.

Donald Ropes: One of the problems with breaking up an Educational Design Research study is that the rich descriptions, so important for understanding the context, might suffer. This could effect the conclusions of the research in a negative way.

Susan McKenney: This, indeed, is a challenge. You need to set up your description of your work in such a way that the onus of the generalizability can fall on the consumer. If you don’t know what are the salient characteristics of your intervention, what are the most influential factors within your context, what kinds of people you had to begin with, what other factors were playing into that at the time, then you are not able to articulate that then you limit the consumer-side generalizability of the work. I think that that is a really important part of presenting those results.

Donald Ropes: Another consequence of Educational Design Research is related to the type of knowledge that results from these types of studies. On the one hand we are striving toward generalizable conclusions, but on the other hand the knowledge from DR is often highly situated and developed in specific contexts – as it should be because it also is meant to solve practitioners’ problems. How could Educational Design Research deal with this?

Thomas Reeves: The type of knowledge you developed is more ‘humble’ than say, medicine or chemistry or physics. In fact I think this whole business of generalizability in education is a bit of a stretch sometimes. If you look at the working conditions under which a generalizable principle is supposed to be applied, it actually turns out to be a working hypothesis at best, not a law or principle in the traditional scientific sense. Ideally, educational design research leads to reusable innovations while at the same time contributing to more and more refined design principles that over time can be synthesized into some sort of theory. There is a danger of having unrealistic expectations. Educational scientists are on the same campuses as doctors and physicists and want to be like them, and so there is a push to publish in more and more distinguished journals. But I really think that there is a lack of concern for impact in that approach. And impact is why I am interested in educational design research. I like to call for a change in tenure review criteria so that it would include broader aspects of impact. In the current situation, university teaching staff members are assessed mostly on their publications in peer-reviewed journals, but not on the impact their research has had. In fact, in my opinion, judging the practical impact of educational research should be a requirement for research done in taxpayer-financed institutions.

Donald Ropes: How about the level at which design research should focus? The background to my question lies in the focus of Educational Design Research studies. We need to understand the process of learning – outcomes too, I think – and the means that support this. But what is meant by means? Is this the context? Are the specific interventions making up a whole? Or do we need to look at the whole system?

Susan McKenney: If you are designing an intervention, then the focus selection must also attend to your jurisdiction of change. You have to decide where am I going to be active right now? You need to play the rules of the game you are in and understand how far (and not) your intervention can go. It’s kind of like determining the ‘zone of feasible intervention.’ But I think that you will have to make a choice.

Find additional information here:

Homepage of Susan McKenney, University of Twente, Department of Curriculum Design and Educational Innovation

Homepage of Thomas Reeves, University of Georgia, Department of Learning, Design & Technology

Link to “Communities of Practice in het hbo’ (Communities of Practices within higher vocational education) with summary of PhD research project of Donald Ropes

Link to “Hogeschool Rotterdam start eigen uitgeverij’ (Rotterdam University starts its own publishing) about the book of the first educational design research project of Josephine Lappia

Link to Design Science Research Group (NL) at Linked-IN

Link to ICO – Research School for Education Science

Link to Design Based Research Collective (USA)