Iranian Researchers Re-imagine Speaking Rubrics

How do we fairly assess English learners in diverse educational contexts? In this episode of The TOEFL Speaking Prep Podcast for the AI Era, we explore groundbreaking research from Iran that challenges traditional speaking rubrics.

A team of high school teachers and university professors collaborated to develop a localized rubric—one designed specifically for Iranian students preparing for high-stakes exams like the TOEFL. By analyzing existing assessment frameworks, they crafted a system that balances pronunciation, vocabulary, grammar, and fluency in a way that truly reflects student abilities.

But that’s not all. We also examine how AI-powered tools like Speech Rater could revolutionize speaking assessments, making feedback more personalized and accessible. Could this research reshape language testing worldwide?

Tune in for a deep dive into personalized assessment, AI in language learning, and the future of speaking evaluations.

🎧 Listen now and join the conversation!

Source

Free TOEFL Speaking practice:

• My Speaking Score

Free resources:

• TOEFL Speaking Course
• SpeechRater Blog
• SpeechRater Knowledge Base

Social:

• Follow us on Facebook
• Subscribe on YouTube

My Speaking Score serves 000s of users across the globe by helping them data-power their TOEFL Speaking prep.

Transcript

All right. So today we're diving into how speaking skills get evaluated in language learning. 

Okay. And, you know, especially for those taking tests like the TOEFL, we've got some interesting research here that really digs into all this, but there's a twist. It focuses specifically on high school students in Iran who are learning English. 

Interesting. So these students, they aren't native English speakers, obviously. They're learning English in a system where speaking hasn't always been the main focus.

(1:10 - 3:43)
Yeah. So how do you accurately assess their progress? Yeah, that's a great question. That's the challenge that this research is tackling. 

It really makes you wonder, are the, you know, right. Typical rubrics that we see like for TOEFL or IELTS. Yeah. 

Are those actually fair? Yeah. In this context. That is a key question that the researchers explored. 

What they found. Yeah. Is that using those standardized rubrics. 

Okay. Might not be the most effective way to assess these students. Oh, interesting. 

So instead they're proposing something called a local rubric. A local rubric. Yeah. 

Okay. So it's basically an assessment tool. Got it. 

Specifically designed for the needs and the context of these particular learners. So how did they even go about creating this local rubric? So they brought together a team of high school teachers who work with these students every day. Gotcha. 

And university professors who specialize in language acquisition. Okay. So you have this really nice blend of perspectives. 

Like practical and theoretical. Yeah. Yeah. 

So the teachers are bringing that firsthand experience. Yeah. With the students challenges, right. 

And then the professors offer that deep knowledge of assessment techniques. Right. And how to actually create a rubric. 

I bet that led to some pretty fascinating discussions. I'm sure. Yeah. 

Did they all agree on what the rubric should include? You know what's interesting? There was a strong consensus on the core elements. Okay. So pronunciation, vocabulary, grammar, and fluency. 

Those seem pretty standard. Yeah. Right. 

Like the building blocks. Yeah. But I'm guessing the devil's in the details. 

Right. So did they agree on how to score or weigh these elements? Exactly. That's where the debate started. 

Okay. Some argued for a really granular scale. Like the zero to nine that's used in IELTS for very precise evaluation. 

Others preferred a simpler scale. Okay. Like the zero to four in TOEFL. 

Yeah. Arguing that it was less overwhelming for high school students. That makes sense. 

So even among the experts. Exactly. No one size fits all approach. 

Right. It really makes you realize how much thought goes into. Absolutely. 

Drafting these rubrics. They really dug deep into the IELTS and TOEFL rubrics. Okay. 

Analyzing, you know, the strengths, the weaknesses. The teachers especially pointed out how IELTS explicitly includes pronunciation. Yeah. 

As a separate category. Recognizing its importance for those learning English as a foreign language. That makes sense.

(3:44 - 3:50)
Yeah. Especially since pronunciation can be a major hurdle for non-native speakers. For sure.

(3:50 - 4:06)
So did this analysis of these like established rubrics. Yeah. Did that influence their local rubric? Definitely. 

So they basically used this knowledge. Okay. To create a hybrid approach.

(4:06 - 5:16)
Oh, interesting. Combining the best aspects of both IELTS and TOEFL. So it's like taking the best tools and customizing them. 

Yeah. But here's what I keep thinking about. Yeah. 

Even with this perfectly tailored rubric. Right. Getting feedback and improving speaking skills. 

Yeah. Can be a challenge. For. 

Especially when you're preparing for something like the TOEFL. It's high stakes. It's high stakes. 

Yeah. For students worldwide. Absolutely. 

And this research really highlights that global need. Yeah. For assessments that are accurate. 

And adapted to individual learning environments. Yeah. And when you consider the weight that TOEFL scores carry. 

Right. For academic and professional opportunities. Huge. 

The stakes are even higher. That's where I think technology. Specifically AI could really come into play. 

Interesting. Imagine if we could use tools like Speech Rater. Which can analyze pronunciation. 

Yeah. Low T. Vocabulary in real time. Yeah. 

To make that feedback process more personalized. You're thinking along the right lines. Okay. 

So imagine a platform. Yeah. That doesn't just give you a TOEFL speaking score.

(5:16 - 6:26)
Okay. But uses Speech Rater technology. Okay. 

To break down your performance. Yeah. Pinpoint your strengths and weaknesses. 

Yeah. And even recommend personalized exercises. Wow. 

It's like having a personal language coach. Yeah. Available 24-7. 

That would be incredible. Yeah. Especially for students who might not have access to. 

Expensive tutors or language programs. Exactly. This kind of AI powered feedback. 

Yeah. Could be a game changer. Absolutely. 

Preparation. It could democratize access. Yeah. 

To these advanced learning resources. Yeah. Helping individuals overcome language barriers. 

And achieve their goals. Exactly. Whether it's acing the TOEFL or just confidently navigating. 

Yeah. A globalized world. For sure. 

So we've got this research on building a more effective rubric. Yeah. For a specific group of students. 

And how that connects to a broader need. Right. For personalized assessments. 

Absolutely. Especially in the context of high stakes tests like the TOEFL. Yeah. 

And then layering in the potential of AI. Right. To really revolutionize. 

Yeah. How students get feedback. Yeah. 

And prepare for these challenges. It all connects. It feels like we've uncovered this whole ecosystem.

(6:27 - 7:00)
Of interconnected ideas. And it's all stemming from this one research study. That's the beauty of a deep dive. 

You never know where it will lead you. So let's delve a little further into this local rubric. Okay. 

They created. What exactly does it look like? Yeah. And how does it address the specific challenges. 

Yeah. Faced by these Iranian students? That's a great question. Let's unpack the details of this rubric and see how it compares to the more established ones. 

Yeah. This will give us a clearer understanding. Okay. 

Of what makes it unique. Awesome. Let's do it.

(7:00 - 9:57)
All right. So they didn't completely reinvent the wheel. Okay. 

They landed on four main categories. Pronunciation, vocabulary range, grammar use, and then fluency and coherence. Okay. 

Those sound familiar. Yeah. Like those resonate with the core elements we talked about. 

Exactly. But what makes this rubric unique? How do they actually measure these categories? Right. So they use a zero to four scale for each category. 

Okay. With detailed descriptions for each score level. Okay. 

So for instance, a top score in pronunciation means the student can use a variety of sounds accurately. Got it. Convey clear meaning while being easily understood. 

So it's not just about having good pronunciation. Yeah. Right. 

But mastering the nuances of pronunciation. Exactly. It's about being able to actually use those sounds effectively in communication. 

That seems especially relevant. Oh yeah. For those who don't have English as their native language. 

Yeah. It's recognizing that pronunciation is a key part of being understood. Yeah. 

And being able to communicate your ideas. And it's not just individual sounds either, right? It's like rhythm and stress and intonation. Exactly. 

All those things. Yeah. They consider all of those. 

Which all contribute to natural sounding speech. Absolutely. Are those aspects that standardized rubrics maybe don't emphasize as strongly? You know, some do, but some don't. 

Okay. And I think this rubric is really trying to highlight the importance of those super segmental features as we call them. Because they do play a big role. 

Absolutely. In you sound and how you come across. They can completely change the meaning of what you're saying. 

Right. So it's really important for these students to master those as well. So it seems like this rubric is really trying to capture the overall quality. 

Exactly. And effectiveness of the student's spoken English. Yeah. 

Rather than just ticking boxes. Right. It's really about how well can you communicate? It's about the communicative competence. 

Can they actually use English to convey their ideas? In a real world setting. In a real world setting. And that focus on nuance continues in the other category. 

Yes, it does. So for vocabulary range, they assess the student's ability to use a wide variety of words. Yes. 

It's not just. Not just basic vocabulary. Basic vocabulary. 

They're looking for less common words. Yeah. Even figurative language. 

So it's about pushing students. To express themselves more. Richly and precisely. 

This aligns well. Very much so. With the demands of something like the TOEFL. 

Absolutely. Where you need to articulate. Complex ideas. 

Complex ideas effectively. Yes. And the grammar use category also reflects this depth. 

It does. So it's looking at both accuracy. Yeah. 

And then the range of grammatical structures. That's right. That the student uses.

(9:57 - 10:29)
Yeah. A high score there would mean the student can confidently use. Yeah. 

A variety of complex grammatical structures. Okay. Showing a good grasp of English grammar. 

So it's not just about avoiding mistakes. Right. But showing mastery. 

Yes. Different grammatical tools. It's about using grammar. 

Right. As a tool for communication. Yeah. 

Not just something to be afraid of. This reminds me of how AI tools could analyze these specific aspects of speech. And provide targeted feedback. 

Almost like a grammar checker. Right. But for spoken language.

(10:30 - 12:12)
That's a great point. AI could definitely play a role there. Yeah. 

In analyzing grammar usage in real time. Helping learners identify. Yeah. 

And correct errors as they speak. Which would be invaluable. Invaluable. 

For TOEFL preparation. Absolutely. Where grammatical accuracy is. 

Crucial. So crucial for a good score. Yeah. 

Okay. So what about fluency and coherence? Right. What are they looking for there? So this is all about how smoothly and naturally the student speaks. 

Okay. They look at things like pace. Okay. 

Clarity. And how well the student connects their ideas. Okay. 

This is where those ums and ahs that we all struggle with. Right. Can really impact your score. 

Yeah. And it's something that AI could potentially track. Sure. 

And provide feedback on as well. Wow. AI really could revolutionize how we prepare for speaking exams. 

I think so. It has that potential. It's like having this personal coach who's just analyzing. 

Constantly analyzing. Every aspect of your speech. Yeah. 

But before we get too carried away with AI. Let's get back to the research. Okay. 

Did they actually put this rubric to the test? They did. Okay. They had an examiner use the rubric. 

Okay. To assess student speaking. Okay. 

And the results were encouraging. Okay. The rubric was reliable. 

Yeah. In evaluating performance. Okay. 

And providing valuable feedback. So this real world validation is important. It is because it moves beyond just theory. 

Right. Into practice. It's one thing to create a rubric. 

Exactly. But it's another to see it actually used. Used in a real classroom. 

In a classroom and have it be successful. And be effective. This shows they're on the right track.

(12:13 - 15:01)
Absolutely. And this kind of research could pave the way for more nuanced. More effective speaking assessments. 

Yeah. Not just in Iran but globally. I agree. 

I think this study has broader implications. Like imagine adapting this local rubric approach to. To other educational contexts. 

Yeah. I mean we could see more personalized assessments. That truly reflect needs and challenges. 

Yeah. This makes me wonder. Yeah. 

While this research focused on high school students in Iran. Right. Could these principles be applied to anyone? That's a great question. 

Learning English as a foreign language. I think that. Could it even be helpful for native speakers trying to. 

Refine their communication skills. Refine their communication skills. I think the core idea of tailoring assessments. 

Yeah. To the specific needs of learners regardless of their background. Yeah. 

Is incredibly valuable. Right. Whether you're a student in Iran. 

Right. A professional in China. Or a native speaker looking to improve your presentation skills. 

Yeah. That concept of personalized feedback. Yeah. 

Remains relevant. And as we've discussed AI could play a huge role. A huge role. 

Having making this personalization. Yeah. Happen on a larger scale. 

Absolutely. So we're seeing this interesting convergence of ideas. Yeah. 

We have this research advocating for more nuanced. Right. And localized rubrics. 

So we're seeing this interesting convergence of ideas. We have this research advocating for more nuanced and localized rubrics. Right. 

The increasing importance of speaking skills in a globalized world. And the emergence of AI as this powerful tool. Yeah. 

For personalized language learning. It's like all these pieces are fitting together. They are. 

To create this new landscape for language education. It's a landscape full of potential. Yeah. 

Imagine a world where language learning is no longer limited by geography or access to resources. Right. AI could really bridge those gaps. 

Yeah. Providing everyone with the tools they need. Yeah. 

To become confident communicators. That's a really inspiring vision. It is. 

But let's bring it back to you the listener. Yeah. What resonates with you the most from this deep dive? What connections are you making to your own experiences with language learning or teaching? Because at the end of the day research is most valuable when it sparks new ideas. 

And inspires action. Exactly. So thinking about the bigger picture. 

Yeah. What does this research tell us about effective language learning? I think it highlights the importance. Yeah. 

Of developing the ability to communicate meaningfully and naturally. Yeah. In real world situations.

(15:02 - 16:49)
It's about fluency. Fluency. Being able to express yourself confidently. 

Absolutely. Whether you're taking the TOEFL. Right. 

Giving a presentation. Yeah. Or simply having a conversation. 

Exactly. And that requires this holistic approach to learning and assessment. It does. 

It's about recognizing the nuances of spoken language. Yes. Understanding the challenges faced by different learners. 

And providing personalized support. Yeah. That helps them grow. 

It's a journey not a destination. Yeah. It's about continuous improvement. 

Embracing feedback. Yeah. And never being afraid to experiment and push your boundaries. 

And with the support of AI powered tools. Right. This journey can become even more engaging. 

Absolutely. And effective. I agree. 

As we wrap up this deep dive. Yeah. We want to leave you with a final thought to ponder. 

Okay. We've talked a lot about how technology can transform language learning. We have. 

But here's a question. Okay. What role does culture play in all of this? That's a great question to consider. 

Yeah. As the world becomes more interconnected. Yeah. 

It's crucial to understand. Yeah. How cultural differences. 

Yeah. Influence communication styles. Right. 

How do we ensure that language assessments. Yeah. Are sensitive to those differences. 

It's something to really reflect on. It is. Because ultimately effective communication goes beyond just language proficiency. 

Absolutely. It's about understanding and respecting. Yes. 

Different cultural perspectives. Well said. So as you continue your language learning journey. 

Yeah. Keep exploring. Keep learning. 

And keep pushing the boundaries. Yeah. Of what's possible. 

I love it. And remember the world is full of fascinating discoveries. It is. 

Waiting to be made. Absolutely. Happy learning. 

Yeah