Saturday, December 25, 2010

Experimentation

When I give presentations about teaching, I always urge the audience members to experiment as much as possible. It is hard to make improvements if you are not willing to try new things. I am always reminded of Albert Einstein’s definition of insanity: doing the same thing over and over again and expecting different results.

Some experiments work and some experiments don’t work. That is just the nature of the game. However, you will never find the winners if you are not willing to risk some losers. Playing it safe is no fun (and provides no benefit).

I tried an experiment with my final exam about two weeks ago. Even now, I am still not sure whether it was a winner or a loser but I found it interesting. I like the fact that I am still thinking about it.

What is the purpose of a final exam? I can think of two reasons. First, it gives the students one last opportunity to influence their grades. There is something about having hope for improvement that keeps students working until the end. Second, the final exam forces the students to review the material and, hopefully, get it better set in their understanding. In other words, they learn more.

I like giving my students an opportunity to improve their grades but my main reason for believing in final exams is that I really want them to leave the semester with all of the knowledge fresh in their minds. The final exam should encourage them to tie all of the material from the semester into a cohesive whole.

Unfortunately, I have often been disappointed in the results of final exams. Students seem overwhelmed by the huge amount of material and flit back and forth during their studies over the various topics without really getting a strong grip on any of it. They just don’t always learn as much as I want from their preparation.

So, at the end of my Intermediate Accounting II test this past semester, I wrote out 49 multiple-choice questions that covered everything that we had discussed that I thought was essential. I tried to gear each question to take about 4 minutes to solve. Although they were designed to be multiple-choice questions, I did not include any answers—just the questions.

Ten days before the final exam, I distributed these questions to my students along with the following speech: “Here is your final exam. These are the 49 questions that I would really love for you to be able to answer on the final exam. When you arrive for the final exam, you will have three hours to answer these questions. I will only make three changes from what you see here:

“1 – I will change the order of the questions.

“2 – I will add four multiple-choice answers to each question along with a “none of the above” answer.

“3 – Most importantly, for each question, I will change one or more of the variables in the question. For example, if the cost is $400,000, I might change that to $500,000. If the life is 5 years, I might change that to 10 years. If the interest rate is 8 percent, I might change that to 10 percent. If the blue method is used, I might change that to the red method. But the question will be fundamentally the same. If you can answer these questions, you should be able to answer all questions on the test.

“If you make sure you can work these 49 questions over the next ten days, you should make 100. But you must understand the problem so well that my changing of the variables will not really slow you down. I realize these are very difficult questions, but they cover the essentials that I want you to be able to work. You’ve got ten days to get these 49 under control.”

I quite honestly was not sure what was going to happen. In the end, the A students missed about 6 of the questions and got 43 correct. The B students missed about 13 and got 36 correct. The C students missed about 20 and got 29 correct. (The D and F students missed more, as you might imagine).

If I had given this test without the pre-test, I am convinced that most would have missed 50 to 100 percent more than they did. Students had clearly gone over the pre-test and learned to work many of the questions. They knew where to focus their attention. However, the number of missed questions was still higher than I had anticipated. Okay, these were 49 extremely tough questions about leases, pensions, cash flows, bonds, deferred taxes, and the like. But I really expected someone to become obsessed and learn them all backwards and forwards and make 100. That didn’t happen. Even with ten days, they just didn’t have enough time for that.

What interested me the most was that this type test had little impact on overall grades. Of all my students, the final exam grade made by 68 percent was within five points of their overall average for the semester. Students with an 82 average made about 82 and students with a 95 average made about 95. Only 32 percent had more than a 5 point difference between this test from their final average. I really had expected a greater number of students to show a greater change.

But, the basic question is still the same-did the students learn more in their studying? That was what I was trying to accomplish. And, I think they did that. Or, at least, I am encouraged enough to try it again. Maybe, this time with 40 questions instead of 49. Maybe, you just can’t do 49 complex questions in three hours even with a ten-day head start.

That’s my most recent experiment and how it worked. What was yours?

5 comments:

  1. Since I'm not a teacher, I don't experiment on making exams or anything, but one thing that would be interesting to find out is whether any of the students had similar mistakes on the study test as they did on the final. Ultimately, you can't really figure this information out (since you didn't see their answers on the study test, you can't compare to their answers on the real final.)

    As a student, I would imagine this happening: without an answer key, no one would be able to improve substantially. Students would answer questions to the extent of their *existing* understanding and ability, with some making predictable, yet consistent mistakes. Since they have no reference guide, they wouldn't even know that they are making mistakes, they wouldn't know even to suspect they are mistaken, and they wouldn't know how to correct those mistakes.

    As a result, when everyone goes into the final, the students who understand the material will continue to understand the material, and student performance will predictably degrade as student understanding degrades. The curve as a whole shift toward less missed questions because of familiarity with the format, not necessarily because of improvement in critical understanding.

    ReplyDelete
  2. Andrew, that is a very astute comment -- one that I had thought about a lot myself. However, I gave the test to the students ten days ahead and had multiple office hours during that period. I told the students that I would not work the questions for them but I would be glad to answer questions about how things should be done. Several students took good advantage of my availability and asked excellent questions. However, much to my surprise, many students did not. Not sure why not.

    Do you think it would have been better to give them an answer without any work shown? "The answer to this question is $324,588 but you have to figure out how I got there." I hate to do too much of the work but I am aware of the problem you have raised. JH

    ReplyDelete
  3. I'd re-assert that if someone doesn't even know that he doesn't know, then there's no chance he's going to go to the office hours.

    The answers without shown work should be helpful and I think that it would lead to higher office hour attendance -- at least for the people who would seek help *if* they knew they needed it.
    I mean, not everyone would care; I'm not denying the apathy and/or laziness of many students.

    BTW, this is essentially what many of my professors do, but with even less effort on their part (or perhaps, more efficiency?). Instead of making a *new* study exam with different numbers, they simply provide answers to answers from previous quizzes, homework problems, in-class workout problems, or maybe even previous exams.

    I think the real issue is figuring a way to work around student psychology. You have motivated students who will ask questions if they don't understand. You don't have to worry about these. You have students who just get the material who don't need to ask questions. You don't have to worry bout these ones either. But what about the students who don't care or don't like the material or the class? For them, a study exam or suggested problems or even knowing where your office *is*...is a bane to their existence. I don't know how to fix that.

    ReplyDelete
  4. Well, I think your argument begins to break down here a bit for me. I have had these students in my class for 14 weeks. If they don't care enough about the material after 14 weeks to even try, then they deserve the grade they get. As I often tell my students, I'm willing to do half the work but only half. If they are not willing to do the other half, they need to be in some other class or not in college at all. I then try to convince them that the material is worth them doing that work. If they still are not interested in learning, then I won't lose any sleep. I am not sure where you go to school but it sounds as if students there either want to fail (not do any work to learn the material) or the teachers pass them without pushing them to do any real work. I'm sorry -- not all schools are like that.

    ReplyDelete
  5. If that is all you got from my last comment...

    ...Nah, I'm just going to walk away from that one.

    He who takes offense when no offense is intended is a fool, and he who takes offense when offense is intended is a greater fool.

    ReplyDelete