This article is Part 2 in a fivepart series. Find Part 1 here.
Interested in discussing these findings with other teachers? Follow the Edmodo Topic for this series!
In this second article in our series examining student performance based on largescale data, we’ll study student responses to some questions on Ratios and Proportions. This way, we can learn how students often apply incorrect reasoning and how teachers can help students learn not just techniques, but how and when to use them correctly.
(In the entire series, we are exploring student thinking based on their responses to questions from the American Mathematics Competition (AMC). These questions were attempted by almost 3 million students over a period of a few months on Edmodo! Such a large data sample provides unique opportunities to gain insights into how students think, partly because they cut across different classrooms, states and countries. Thus, they can help detect underlying thinking patterns across all students.)
In the last article, we signed off with an intriguing question – why do so many students answer 7.5 pounds to the question below?
Attempted: 122283  Skipped: 75 
Coming to the question, the error seems to come about when students use snippets of procedure which they have learnt using the numbers in the question. As often happens, the procedure leading to the wrong answer is actually more complex than the correct one. One approach that leads to the wrong answer is this: Students draw 8 lines or circles to represent the 8 hamburgers. They then draw 3 lines or circles corresponding to the 3 pounds. The correct answer involves reasoning that 24 hamburgers is 3 times 8 and hence the 3 pounds need to be multiplied by 3 as well.
But what some students do is look at the figure and notice that one of the larger circles correspond to a little more than 2 of the smaller circles. Some students mistakenly take it as 2.5 pounds and multiply it by 3 to coming to the final answer of 7.5 pounds. Other students who ‘correctly’ see than 2 2/3 pounds are needed, arrive at the wrong 8pound answer. Note that this flaw leads to both the C and D answers and the total number of students who select C or D together exceed the correct answer percentage illustrating how common these errors are.
But how can we be sure students followed that logic? Our analysis tells us what students answered but not why. Teachers can easily verify the student’s logic and identify core misunderstanding by asking the question in the class and having students either note down or orally share their explanation. The fact that certain options are presented can also influence the children, such questions can be asked without options too to investigate children’s thinking.
We also can’t be sure that all students answered all questions seriously. One way of addressing this is by asking students to optionally share the explanation for their reasoning along with their answers. A surprisingly large number of students do, providing not only an insight in their own words on how they think, but proof that they are not selecting their answer randomly. We can doublecheck by confirming that the percentage of students choosing a particular option does not vary significantly between students who wrote explanations and those that didn’t.
Let’s look at another question from the AMC Grade 10 paper, which is designed to be easy for 10th graders. On Edmodo, however, students had more trouble than we expected. Of the 2,100 students that answered, more students selected a wrong answer (20), than the right answer (18)! This is unusual and merits investigation.
Attempted: 2108  Skipped: 0 
Can we compare how the students who chose the different options fared in the paper as a whole? See the table below:
Option 
Average Total Score in the full paper of students who chose this option 
A 
25% 
B 
51% 
C 
31% 
D 
27% 
E 
28% 
We clearly see that the score of the students who chose the correct option B is significantly higher than those who chose the other options! This means that the students who were stronger in Math overall, chose option B. As it turns out, this question is an excellent test for understanding which students grasp these concepts and are able to correctly apply them across the whole paper.
To understand why so many students choose the wrong option (20), let’s create a new question which looks like and uses the same numbers as the original question:


Do you see the similarity between these 2 questions and how the correct answer in the latter question (18) could be selected by a majority of students for the new question? This also suggests that students are familiar with the method of solving a question and often apply it blindly when they find an opportunity.
Let us look at one more question and try and think about the reasons for a common wrong answer students give.
Attempted: 454188  Skipped: 484 
Over 450,000 students answer this question and about 45% of students get the correct answer. Note that in case students had simply ‘guessed’, each of the options would have close to 20% students selecting it. Clearly, students are thinking and, in this case, largely choosing one of 2 options.
Note also, though asked in a Grade 8 test, the question actually relates to a Grade 6 standard about understanding ratios and using them in reallife situations:
“6.RP.3: Use ratio and rate reasoning to solve realworld and mathematical problems, e.g., by reasoning about tables of equivalent ratios, tape diagrams, double number line diagrams, or equations.”
Yet, 55% of grade 8 students answer it incorrectly.
The graph on the right shows a number of interesting insights about how students answered this question. The horizontal axis represents the score in the ‘overall paper’ of students who attempted this question, while the vertical axis represents the percentage of students. Thus points on the left represent students who scored lower in the overall paper and points on the right those who scored higher. Thus we can see that among students scoring in the 812 range, option D is by far the most popular option (significantly exceeding even the number of students choosing the correct option B!)
The most interesting insight is that there is a clear, welldefined group of students who believe that the ratio of girls to boys in Mr. Raub’s class is 7:4. In fact, among weaker students, that is the most popular option.
Why do you think these students chose the answer 7:4? We shall discuss this briefly in our next article, so please share your thoughts in the comments section below. Remember, you can try asking this question to your students and noting the reasoning they use.
What are the takeaways here for a teacher?
 We see that students usually have a reasoning— even if it’s flawed— for what they answer.
 Concepts that we think are basic or should have been acquired 2 or 3 years earlier are often not reliably learned. They need to be tested for and reinforced.
 Mistakes are often not profound! They can be trivial, even careless. But when they manifest themselves in patterns that are clearly visible across large groups of students, they deserve to be studied, understood and corrected.
 Students often follow a procedure in a mechanical, unquestioning way when they should be checking if the procedure applies in that situation. They apply a technique when a question looks similar without fully understanding the question. And they often skip the final check— whether the answers make sense given the original question— which can point out the error.
If teachers are aware of these errors and then look for them either in students’ written work or in class, it is possible to understand and correct these learning gaps!
Edmodo acknowledges the inputs of Educational Initiatives, Inc. which provided the question performance analysis used in the article.
7 comments. Leave new
good article
nice article
Its the first and foremost duty of any teacher should be to understand the mind of their students. All great teacher had a tremendous connectivity with their students and in that condition only 100% transfer of knowledge can take place. Thanks for posting such an informative post. Cheers.
Thanks for sharing this post. It is very important to be able to analyze data to see what it really means.
These are good things to know and more importantly, not to forget during the teaching learning process.
1.We see that students usually have a reasoning— even if it’s flawed— for what they answer. (asking students their rational is important to understand what they are thinking)
2.Concepts that we think are basic or should have been acquired 2 or 3 years earlier are often not reliably learned. They need to be tested for and reinforced. I( find that our 7th graders are coming to us missing basic skills and knowledge – how do we get them caught up?)
3.Mistakes are often not profound! They can be trivial, even careless. But when they manifest themselves in patterns that are clearly visible across large groups of students, they deserve to be studied, understood and corrected. (data to the rescue!)
4,Students often follow a procedure in a mechanical, unquestioning way when they should be checking if the procedure applies in that situation. They apply a technique when a question looks similar without fully understanding the question. And they often skip the final check— whether the answers make sense given the original question— which can point out the error. ( I think this comes from – getting the answer – not the process)
I can use this article to strengthen my teaching practices.
Just Awesome information boss.
Thanks for sharing valuable post keep posting…!!!
See You
Simply Superb Post…!!!