5 ways to get retrieval practice wrong and reflections on a new study which shows that when material is complex, testing can stop being a desirable difficulty and start being an undesirable one.
Hi Carl. I found myself tripping over the early paragraph re the student who has 'read something once and understood 60%.'
You mention that asking this student to close the book and (attempt to) retrieve is 'floundering' rather than learning. However, doesn't this contradict the research?
My understanding of the literature (and my longtime experience with Anki) is that the struggle to retrieve fuzzy, partially-encoded material (i.e., that 60%) is exactly what prevents the 'illusion of competence.' If we wait until encoding is 'secure' (near 100%) before we test, aren't we missing the biggest ROI window for retrieval?
Surely identifying the 40% we don't know via a failed retrieval attempt is more valuable than just re-reading the text?
Essentially, the paragraph seems to go against most of my reading and understanding: that testing is not only good, but is literally learning in-and-of-itself.
Did I misinterpret?
--
P.S. I've just read the Redifer paper, and I'm not sure about it. Asking students to 'free recall' a very heavy/dense 2,500-word academic article after a single 15-minute read seems less like a fair test of retrieval practice and more like a 'working memory torture test' that was destined to cause overload.
Why are you using AI to generate your texts? I like your content, it’s really good, and I (and I guess most people, too) would happily take fewer posts in return for the posts being written fully by you and not AI.
It’s super obvious that you are using LLMs, sadly. :(
Based on some recent classroom observations, I too have had thoughts about the limits of retrieval practice, and I found this article thought-provoking. I agree that retrieval practice seems to work best for discrete pieces of information, but at least one of the well-known retrieval practice studies did NOT involve that kind of information.
In 2011, Karpicke and Blunt described a retrieval practice study in which the to-be-learned content was "a science text," and they had undergraduates in four conditions, one of which was retrieval practice. That group engaged in free recall, reading the text and then putting it aside and writing down everything they could remember--and then they did that again. When tested a week later, the retrieval practice group remembered the concepts significantly better. (See https://www.science.org/cms/asset/882cef39-74f8-4177-a127-dd7191b24d4e/pap.pdf).
I don't know whether the science text used in 2011 was less complex than the article in the recent study, or whether (perhaps) in the time between the two studies, students have become less able to understand complex text (we do have some evidence of that).
I do know that when Karpicke tried to repeat the experiment with 4th graders, using a 4th grade level science text, there was no difference between the retrieval practice group and the other groups. Karpicke told me he suspected that there was simply too much information in the text for the 4th graders to be able to retrieve. When he simplified the task (less reading and/or more support for the writing), the retrieval practice group did do significantly better.
So I wonder: have undergrads become more like 4th graders in this respect in the last 15 years? Or maybe the Karpicke results were different because the undergrads engaged in retrieval practice twice, even though I think little or no time elapsed between the two efforts?
Apart from forms of knowledge, isn't this an 'apples & pears'-type comparison in the first place given the extreme age and learning experience gaps between these two cohorts. This occurs frequently in research I read -- such as estimating the efficacy of 'inquiry' learning (in schools), by significantly referencing research into undergraduate (medical) courses.... not distinguishing significant aspects of primary and secondary school learning.. etc etc.
Actually, Karpicke and his team should be commended for checking to see if an intervention that worked with undergraduates would also work with younger students. Most studies, of course, are done using university students -- since those are the subjects most readily available -- and, as you note, it's just assumed that the findings are universal.
In the series of studies I'm describing, in contrast, Karpicke and his team went on to modify the task in various ways and discovered that with more support (e.g., less reading and/or less writing), retrieval practice can in fact be highly effective with young children.
There is so much to mine in this piece, but I'll begin with a statement that goes to the heart of the knowledge building vs. strategy instruction debate, which has created a false binary. We use literary works and knowledge-rich texts to guide students toward understanding by engaging them in interpretation, analysis, and synthesis. The knowledge in the piece is the "what" we teach; making sense of it is the "how" we teach.
"An interpretation of a novel, an analysis of historical causation, a synthesis of competing theoretical perspectives; these are not items stored in memory awaiting collection. They must be constructed in the moment, assembled from understanding rather than retrieved from storage."
“Assembled from understanding” … yes. But in what ways do the cognitive operations underlying this complex and interleaved ‘understanding’ actually function? ie listing concepts such as ‘interpretation’, ‘analysis’ & ‘synthesis’ is one thing; whereas, designing meaningful ‘retrieval’ of complex texts seems to require at least 1. students’ background knowledge (specifically & generally inculcated over time and in relation to topic at hand) 2. a clear grasp of key concepts (more easily amenable to direct teaching/retrieval). All dependent on existing level of familiarity with the specific subject matter, vocabulary & so on. These need to be inculcated/taught specifically in the first place, and this is where some direct ‘retrieval’ practice is useful. For the rest, formative guided practice (ie the ‘how’) - in comprehension, precis, paraphrasing etc - is what builds knowledge and skills over time.
Hi Carl - another incredibly rich article. Thank you. Along similar lines to the comment from shaeda.io I was hoping you could clear up my understanding on the point regarding retrieval only working on 'fully encoded' knowledge. Isn't the point of retrieval to strengthen the trace on key/priority curriculum information that has been partially forgotten? Indeed isn't it an advantage / a necessity for retrieval to be carried out on information that has been partially forgotten (i.e not fully encoded) to a) allow desirable difficulty to be utilised on forgotten information and b) enable the benefits of the 'prediction error' effect to kick in?
Also where does this leave pre-questioning and pre-testing and its supposed benefits? As I understand them, those are a) for the teacher to see what's known and what isn't (this would be true of retrieval in terms of what's been genuinely remembered) b) to position what the learner should be paying attention to? Many thanks
Great stuff. Can't the first four mutations be explained more simply as issues with depth of processing/ Levels of Processing (LoP)? 1. No trace = processing too shallow 2. unlinked curriculum context = processing too shallow, not related to related concepts 3 ambiguous terms = processing to shallow not sufficiently mapped to related terms 4. spaced = processing too shallow insufficient trace to be preserved
I think this article opens up space for more Concept-based Curriculum and Instruction. I always felt that retrieval practice was only for simple, discrete facts in a science classroom. We can teach more for conceptual understanding and use retrieval practice to consolidate learning, and leave comprehension to language work and figuring out.
This is something I'm deeply interested in, as a(retired) teacher of English, Literature, Philosophy & Theory of Knowledge specialising in extending learning for high potential and advanced learners. Clearly, some aspects of each of these - eg stylistic and literary devices, structuring an essay, knowing & understanding the work of major thinkers etc -is open to forms of retrieval practice. Other aspects, relating to conceptual development and building understanding - eg, clarification of commonly misunderstood / non-intuitive scientific phenomena, consideration of what makes an act "fair"/"just" etc - clearly do not fit the same 'retrieval' bill. In these cases, forms of Community of Inquiry and Socratic Dialogue, well-enacted, are truly efficacious in developing students' rich and elaborated cognitive schema and thinking.
Hi Carl. I found myself tripping over the early paragraph re the student who has 'read something once and understood 60%.'
You mention that asking this student to close the book and (attempt to) retrieve is 'floundering' rather than learning. However, doesn't this contradict the research?
My understanding of the literature (and my longtime experience with Anki) is that the struggle to retrieve fuzzy, partially-encoded material (i.e., that 60%) is exactly what prevents the 'illusion of competence.' If we wait until encoding is 'secure' (near 100%) before we test, aren't we missing the biggest ROI window for retrieval?
Surely identifying the 40% we don't know via a failed retrieval attempt is more valuable than just re-reading the text?
Essentially, the paragraph seems to go against most of my reading and understanding: that testing is not only good, but is literally learning in-and-of-itself.
Did I misinterpret?
--
P.S. I've just read the Redifer paper, and I'm not sure about it. Asking students to 'free recall' a very heavy/dense 2,500-word academic article after a single 15-minute read seems less like a fair test of retrieval practice and more like a 'working memory torture test' that was destined to cause overload.
I wonder if this doesn’t address your question: “retrieval practice is a consolidation strategy, not a comprehension strategy”.
So you are asking more about Check for Understanding than about Retrieval Practice? https://chatgpt.com/share/693fe778-5f80-800b-bd50-3e6335c91994
Why are you using AI to generate your texts? I like your content, it’s really good, and I (and I guess most people, too) would happily take fewer posts in return for the posts being written fully by you and not AI.
It’s super obvious that you are using LLMs, sadly. :(
Based on some recent classroom observations, I too have had thoughts about the limits of retrieval practice, and I found this article thought-provoking. I agree that retrieval practice seems to work best for discrete pieces of information, but at least one of the well-known retrieval practice studies did NOT involve that kind of information.
In 2011, Karpicke and Blunt described a retrieval practice study in which the to-be-learned content was "a science text," and they had undergraduates in four conditions, one of which was retrieval practice. That group engaged in free recall, reading the text and then putting it aside and writing down everything they could remember--and then they did that again. When tested a week later, the retrieval practice group remembered the concepts significantly better. (See https://www.science.org/cms/asset/882cef39-74f8-4177-a127-dd7191b24d4e/pap.pdf).
I don't know whether the science text used in 2011 was less complex than the article in the recent study, or whether (perhaps) in the time between the two studies, students have become less able to understand complex text (we do have some evidence of that).
I do know that when Karpicke tried to repeat the experiment with 4th graders, using a 4th grade level science text, there was no difference between the retrieval practice group and the other groups. Karpicke told me he suspected that there was simply too much information in the text for the 4th graders to be able to retrieve. When he simplified the task (less reading and/or more support for the writing), the retrieval practice group did do significantly better.
So I wonder: have undergrads become more like 4th graders in this respect in the last 15 years? Or maybe the Karpicke results were different because the undergrads engaged in retrieval practice twice, even though I think little or no time elapsed between the two efforts?
Apart from forms of knowledge, isn't this an 'apples & pears'-type comparison in the first place given the extreme age and learning experience gaps between these two cohorts. This occurs frequently in research I read -- such as estimating the efficacy of 'inquiry' learning (in schools), by significantly referencing research into undergraduate (medical) courses.... not distinguishing significant aspects of primary and secondary school learning.. etc etc.
Actually, Karpicke and his team should be commended for checking to see if an intervention that worked with undergraduates would also work with younger students. Most studies, of course, are done using university students -- since those are the subjects most readily available -- and, as you note, it's just assumed that the findings are universal.
In the series of studies I'm describing, in contrast, Karpicke and his team went on to modify the task in various ways and discovered that with more support (e.g., less reading and/or less writing), retrieval practice can in fact be highly effective with young children.
There is so much to mine in this piece, but I'll begin with a statement that goes to the heart of the knowledge building vs. strategy instruction debate, which has created a false binary. We use literary works and knowledge-rich texts to guide students toward understanding by engaging them in interpretation, analysis, and synthesis. The knowledge in the piece is the "what" we teach; making sense of it is the "how" we teach.
"An interpretation of a novel, an analysis of historical causation, a synthesis of competing theoretical perspectives; these are not items stored in memory awaiting collection. They must be constructed in the moment, assembled from understanding rather than retrieved from storage."
“Assembled from understanding” … yes. But in what ways do the cognitive operations underlying this complex and interleaved ‘understanding’ actually function? ie listing concepts such as ‘interpretation’, ‘analysis’ & ‘synthesis’ is one thing; whereas, designing meaningful ‘retrieval’ of complex texts seems to require at least 1. students’ background knowledge (specifically & generally inculcated over time and in relation to topic at hand) 2. a clear grasp of key concepts (more easily amenable to direct teaching/retrieval). All dependent on existing level of familiarity with the specific subject matter, vocabulary & so on. These need to be inculcated/taught specifically in the first place, and this is where some direct ‘retrieval’ practice is useful. For the rest, formative guided practice (ie the ‘how’) - in comprehension, precis, paraphrasing etc - is what builds knowledge and skills over time.
"But in what ways do the cognitive operations underlying this complex and interleaved ‘understanding’ actually function?"
Let me know if this comes close to addressing your question: Daniel Willingham Reminded Me That Memory is the Residue of Thought (https://harriettjanetos.substack.com/p/daniel-willingham-reminded-me-that?r=5spuf).
Thank you, much appreciated. I'll take a look now.
Hi Carl - another incredibly rich article. Thank you. Along similar lines to the comment from shaeda.io I was hoping you could clear up my understanding on the point regarding retrieval only working on 'fully encoded' knowledge. Isn't the point of retrieval to strengthen the trace on key/priority curriculum information that has been partially forgotten? Indeed isn't it an advantage / a necessity for retrieval to be carried out on information that has been partially forgotten (i.e not fully encoded) to a) allow desirable difficulty to be utilised on forgotten information and b) enable the benefits of the 'prediction error' effect to kick in?
Also where does this leave pre-questioning and pre-testing and its supposed benefits? As I understand them, those are a) for the teacher to see what's known and what isn't (this would be true of retrieval in terms of what's been genuinely remembered) b) to position what the learner should be paying attention to? Many thanks
Great stuff. Can't the first four mutations be explained more simply as issues with depth of processing/ Levels of Processing (LoP)? 1. No trace = processing too shallow 2. unlinked curriculum context = processing too shallow, not related to related concepts 3 ambiguous terms = processing to shallow not sufficiently mapped to related terms 4. spaced = processing too shallow insufficient trace to be preserved
I think this article opens up space for more Concept-based Curriculum and Instruction. I always felt that retrieval practice was only for simple, discrete facts in a science classroom. We can teach more for conceptual understanding and use retrieval practice to consolidate learning, and leave comprehension to language work and figuring out.
This is something I'm deeply interested in, as a(retired) teacher of English, Literature, Philosophy & Theory of Knowledge specialising in extending learning for high potential and advanced learners. Clearly, some aspects of each of these - eg stylistic and literary devices, structuring an essay, knowing & understanding the work of major thinkers etc -is open to forms of retrieval practice. Other aspects, relating to conceptual development and building understanding - eg, clarification of commonly misunderstood / non-intuitive scientific phenomena, consideration of what makes an act "fair"/"just" etc - clearly do not fit the same 'retrieval' bill. In these cases, forms of Community of Inquiry and Socratic Dialogue, well-enacted, are truly efficacious in developing students' rich and elaborated cognitive schema and thinking.