Saturday, May 28, 2011

How Do You Know How Your Program Is Doing?

I had a great time last week attending the Teaching Summit put on by the Lebow College of Business at Drexel University. Dr. Thomas Hindelang was a wonderful host and I was so pleased to have the opportunity of presenting the opening keynote address. It is hard to boil a 45 minute speech down to a sentence or two but my primary theme was the disconnect between what colleges and universities claim to accomplish and what the new book Academically Adrift: Limited Learning on College Campuses (Arum and Roksa) says that we are actually accomplishing. I then described six steps that I felt we (as faculty) needed to consider if we were going to fulfill the promises being made to our students.

I might write more about those thoughts later but that speech was not the purpose of this post. After giving my opening remarks, I attended a number of paper presentations at the conference. One of my favorites was Feedback from Alumni and Employers Guiding Assessment of Business Curricula by Ellen Kraft and Diane Holtzman of Richard Stockton College of New Jersey.

I was interested in this paper because it addressed one of my pet peeves about colleges. Over the past 40 years, I cannot tell you how many meetings I have attended where academic programs were discussed and changes suggested with absolutely no statistical backup. No evidence was given that a change was needed and no one could show that the proposal in question might actually provide improvement. It seems that we are often asked to modify programs based on our own intuition of what might work better.

Professors Kraft and Holtzman had helped direct a survey at their school where alumni as well as the employers of those former students were asked to identify characteristics that employees needed for success. Because I didn’t take great notes, I won’t try to explain their methodology. I was just impressed that the folks at that college wanted data to help in assessing the past and planning for the future.

All educational programs must evolve over time. That process, though, should be guided (I believe) by careful data analysis. Too often changes are made that seem to have no basis other than which person can argue his or her position the best (or who can last the longest in the debate).

Since I believe data gathering is essential as a prerequisite for change, I wanted to mention two relatively painless ways that information can be generated for better program decision-making.

(1) – I am a member of the Accounting Department here at the Robins School of Business at the University of Richmond. Last summer, we decided to survey all Accounting students who graduated between May 2006 and May 2009. We didn’t include the 2010 graduates because they had not yet started work. We didn’t go back earlier than 2006 because the program had changed enough over the intervening years that we were not sure the feedback would still be relevant.

We had reached the point where we were tired of making decisions based on what we would hear from former students (almost randomly) at alumni receptions or from an occasional email. The basic questions were obvious: What were we doing well and, more importantly, where did we need to make improvements to better prepare our students for careers after graduation? All faculty members like to believe they are adding value and providing their students with an appropriate education. But how does any teacher really know?

The Alumni Affairs office here on campus was able to provide us with email addresses for well over half of those students who graduated in our designated time period. The Office of Institutional Effectiveness helped create the survey, send it out, and accumulate the returned data.

We asked a number of general questions including:
--How many jobs have you held since graduation?
--In what field are you currently working?
--Have you taken the CPA Exam and, if so, how well have you done?
--Have you attempted any other certification exams such as the CMA Exam?
--Have you attended graduate school?

We then took the former students through a number of different subject areas (income tax, auditing, government accounting, financial accounting, and the like) and asked them to assess whether they felt their education was adequate (a) for their employment and (b) in comparison to students from other schools with whom they worked. We also asked the same questions for three skills: written communications, oral communications, and IT skills.

We wanted to pinpoint areas where changes might need to be made.

Our response rate was quite good and the information has proven to be helpful as we move forward and try to guide the logical evolution of our program. Although the overall results were excellent, we did uncover several smaller areas where a number of individuals indicated that they had been at a disadvantage educationally. Obviously, we have worked on those areas during the past 9 months to rectify the problems.

(2) – Approximately six years ago, our department – once again – wanted to gather information to aid our internal discussions and decisions. Many of our Accounting students continue to live here in the Richmond area after graduation. We invited 16 former accounting majors who had been out of school 1-3 years to return to campus for a one-hour focus group. Because of the number, we had to split them into two focus groups held on separate days. We asked a facilitator to talk with these individuals about their job experiences and how well prepared they felt based on the education they received in the Richmond program. We provided specific questions for the facilitator to discuss with the group. The sessions were audio-taped and a transcript was typed. The facilitator wrote up his assessment of the information that was gathered.

No serious political campaign would ever make decisions without focus group studies. No large business would ever start a major advertising campaign without focus group studies. I am surprised that universities do not make better use of that technique to help assess the quality of the education that their students actually obtain. Once again, as with the email survey that was done more recently, we were pleased with what we heard although the former students did point up areas where they felt they had been at a disadvantage in their jobs.


Perhaps one reason that books like Academically Adrift are written is because universities do not do a better job of talking with their graduates to identify both strengths and weaknesses of the academic programs being offered. Professors Kraft and Holtzman seem to have done a great job of providing useful information for their school.

When is the last time your program made a serious attempt to generate data to indicate the quality of the education that your students are receiving? Either through a survey or through a focus group, information is not that difficult to obtain. Such data can help guide the decisions that are made in the future about modifications needed in your academic programs.

Wednesday, May 11, 2011

In Case You Are Interested - Part II

I have the honor of being the opening keynote speaker on May 20 for "Teaching Summit 2011" being presented at the LeBow College of Business at Drexel University in Philadelphia, PA. They have a great group of papers and presentations scheduled on all aspects of teaching and learning. If you are interested in the possibility of attending, registration information is available at http://www.lebow.drexel.edu/Faculty/Centers/BPTS/index.php

Tuesday, May 10, 2011

Worth Reading

I discovered long ago that I did not have many clever and creative ideas. However, I also discovered that if I paid close attention, I could steal wonderful ideas from some of the greatest minds around.

I was reminded of this recently. One of the readers of this blog sent me a link to a great article on the teaching and learning of math. She said that she thought I would find it interesting because I was always curious about education. And, she was absolutely correct. It is well worth reading and pondering.

http://opinionator.blogs.nytimes.com/2011/04/18/a-better-way-to-teach-math/

Several parts of this article really caught my attention. They are the parts that I’ll probably steal in the future. In fact, here are five thoughts that I plan to steal (borrow) and apply to my own teaching.

(1) – The subject of the story has an absolute certainty that students can learn math. In fact, he says so “’Almost every kid — and I mean virtually every kid — can learn math at a very high level, to the point where they could do university level math courses,’ explains John Mighton.”

I wonder what would happen if we all walked in each day and had that same certainty about our own students.

It is amazing to me how quickly I begin mentally classifying my students as “excellent,” “average,” and “poor.” Do I start treating those students in that way? Do I ask harder questions of the “excellent” students and dumb down the questions for the students I perceive as “poor?” What kind of subtle messages am I sending them? Perhaps in that way, my perceptions become a self-fulfilling prophecy. I know I say to my students often “you can ALL learn this.” Is that my true sentiment or just lip service that I spout because it sounds good?

In the future, I’m going to try to do a better job of helping everyone to excel. Maybe, as he says, I am forcing my own students early in the semester to choose to think of themselves as either smart in accounting (my subject) or dumb in accounting. And, maybe it is that choice that then serves as a driving force for the rest of the semester. If I can change that mindset, can I get better results? Can I convince them that they can all learn accounting and, if so, how will that change them over the rest of the semester?

(2) – I loved what he had to say about “extensive practice.” I get the feeling in education that we show students how to do something once, maybe twice, and then expect them to be experts from thereon. When I was growing up, we would work extensively on math facts in school: 56 divided by 8 is 7, 20 percent of 15,000 is 3,000. We did hours and hours of that type of practice. Sure, it wasn’t exciting. But, as I have said here before, every student in my fifth grade class in 1959 knew more about basic math than every student that I taught this past year in college. Why? That is easy – students today are trained to use calculators and have never really practiced math facts. The next time you are in a college class, ask a student to multiply 12 times 9 without a calculator and you’ll see pure panic and dread. In 1959, every hand would have been raised (in the fifth grade).

Practice really is necessary for learning.

Maybe we all want to avoid the possibility of boredom so we eliminate extensive practice from the learning process. Then, when we get to a test, we (as teachers) are surprised that the students cannot handle the questions we give them. Perhaps what they really need is just more practice. Few things can be learned by doing them merely once or twice.

(3) – “Studies indicate that current teaching approaches underestimate the amount of explicit guidance, ‘scaffolding’ and practice children need to consolidate new concepts.” I really like the idea of “scaffolding” or building a structure that allows students to see new material within a logical framework. Often, professors have worked with their material for so long that they have no need for a structure. To them, the material is virtually self-explanatory as it stands alone. However, students do not have that wide-ranging knowledge as a basis for understanding new material and how it ties in with other material. To me, a course is a large puzzle where every day you introduce new pieces and then work to fit them into the rest of the puzzle so that—by the end—it all makes sense; the total picture becomes clear.

(4) – “’No step is too small to ignore,’ Mighton says. ‘Math is like a ladder. If you miss a step, sometimes you can’t go on. And then you start losing your confidence.’” Once again, we—as teachers—always understand where the process is headed. We know the end of the story before we set out the beginning. In teaching, we may lose track of why we need to explain every minute step along the way. To the student, one missed step can bring the whole learning process to a screeching halt. Some students get lost easily. Of course, we prefer students who can make giant leaps from one concept to the next but, in truth, those are the people who have the least need for a teacher. If you really want to be a teacher, you have to want to help those people who experience the most trouble in making the connections. That is where the real teaching comes in. You have to recognize when a student can only take small steps and then break the process down so they can successfully see the connections.

(5) – “The foundation of the process is building confidence.” If you have taught for more than one day, you will have experienced the student who loses confidence. It is a real shame but those students will come to believe that they are simply not smart enough to understand the topic at hand. Often, after that, no degree of work can win back that student. They are lost. They assume they will do poorly and they do. If you are going to turn all your students into winners, you have to recognize immediately when a student starts to struggle and then work to break the process down into more manageable pieces for that student. You need to constantly work to build confidence.

Yes, teaching can be real work but no one ever said it would be easy.

Thanks for sending the article along to me!!! I definitely managed to steal (borrow) a lot from it.

In Case You Are Interested

I was talking recently with a friend who asked something like: "You have had nearly 33,000 page views on your teaching blog. Which of your entries have drawn the most traffic?"

In truth, some entries have had almost no readers while others have had thousands of page views. Not really sure what makes such extreme differences.

If you want to go back and check to see the most popular entries over the last 16 months or so, here they are.

1 - What Do We Add? July 22, 2010
2 - Big Mistakes March 26, 2011
3 - Need Some Inspiration? September 13, 2010
4 - What Do You Tell Your Students? August 19, 2010
5 - The $10 Million Question January 16, 2011
6 - The Opening August 24, 2010
7 - Using Power Point Slides March 17, 2010

Tuesday, May 3, 2011

My Latest Problem

In every job, problems arise and, unfortunately, there are not always great solutions.

I ran into a problem on Saturday and I wasn’t sure what to do with it. When you are dealing with a group of college students and the decisions you make are important to those people, inconsistencies can drive you crazy.

I have two Intermediate Accounting II classes this semester. At many schools, Intermediate II is considered one of the most challenging courses in the entire university. Consequently, the grade in that class is often viewed as extremely important to the students. The difference of one letter grade can have major implications in the direction of a career.

Our final exam schedule this semester ran from Monday morning at 9 a.m. until Saturday evening. As luck would have it, one of my Intermediate II classes had its final exam in the very first slot from 9 until noon on Monday. The other class had its exam on Friday evening from 7 until 10 p.m. Because my classes were small this semester, I allowed my Intermediate II students to take either exam. I didn’t care. Eight students chose to take the first exam and 25 chose to take the second. Most students seemed uncomfortable trying to take the exam on Monday without sufficient time to prepare. That was fine by me.

I gave the first group 37 problems ranging in time from 3-8 minutes in length. No one left early but everyone seemed to be finished or close to finished by the end. I graded that exam and came up with a raw score for each. I did not curve the exam at that time because I only had 8 tests and wanted to see how the other 25 students did. For convenience, let’s assume that the raw scores ranged from minus 20 to minus 60. I liked the test; I liked the range of raw scores; I liked the length of time that my students took to finish the exam; I liked the distribution from top to bottom.

I would have loved to give the same test to my second class but I worried (especially over a 5-day period) that too much information would get out. I trust my students but I don’t want to put too much temptation out there for them. My guess is that every university worries about cheating.

So, I took each of those 37 problems and changed it slightly – I increased a 6 percent interest rate to an 11 percent rate, I changed a residual value from $10,000 to $25,000, I changed a $5,000 gain to a $9,000 loss. I then rearranged the questions into a different order just in case some student had slipped information out such as “the first question requires you to deal with a 20 percent stock option.”

I honestly believed that I was giving the second group a test that was the equivalent of the first test.

However, the results were so much worse that I was stunned. For the second group, the raw scores ranged from roughly minus 20 (same) to minus 90. Worse still, and this is what really caught my attention, approximately half of the students in the second group did worse than my very worse student in the first group. It was like the two groups took two completely different exams.

My problem became immediately obvious to me. Should the students in the first group get significantly higher grades than the second group or was there something about the second test that made it harder (and that I was not seeing)? I thought I had given comparable tests but maybe not. For example, maybe some of my changes managed to create more complex situations. Or, perhaps changing the order of the questions caused a problem for the students (maybe the first questions were now harder and slowed them down or discouraged them).

And, to make matters even worse, although virtually all of the first group finished the test on time, many of the students in the second group did not come even close to completing it. I had page after page of blanks. It is hard to give any partial credit to a blank sheet of paper.

--Could the better students have all taken the first test rather than the second? If so, then I had a justification for giving them a higher grade. But, for the most part, I couldn’t see any difference in the abilities of the two groups (and I looked very carefully).

--Could the first class have been bright and awake at 9 a.m. and the second class sleepy at 7 p.m.? They didn’t look any different but I could not peak inside of their heads. And, should that make a difference in the final grading?

--Could the first class have been fresh because it was their first final exam and the second class exhausted because it was their fourth or fifth? And, again, if so, should that have any effect on the grading? Should you factor in the time of the test when handing out grades? If two students both make minus 60 can you give one student a different grade because he or she took the test on Friday night after 3-4 other tests over a long week?

--Could some of the changes I made in the questions have subtly changed easy questions (on the first test) into hard questions (on the second test) without my awareness?

Why Why Why is the question I have asked over and over since then? Why were the raw scores so different and what should I do about that in arriving at final test scores?

It is several days later and I still do not have a good answer for that question.

Sunday, May 1, 2011

Congratulations!!! You Did It!!!

Today is one of my very favorite days. I finished my grading. Then, before I turn in the final grades, I write each student who made an A to personally congratulate that individual. For someone who works that hard and did that well, I think something more than an anonymous grade report is appropriate. So, I wrote out the following letter and then emailed it to each student who made an A. I very much like doing this. It makes me smile. I have worked with these folks for months and they responded to my challenge. I get a real thrill from being able to let them know that the work was worth it (in my opinion) because they received the highest possible grade. A is for Excellent!!!!


Mr. or Ms. XXX,

I wanted to let you know that I have finally completed the grading for Accounting 302 and you have earned the grade of A for the spring semester. Congratulations!!! As I am sure you are aware, it is a very challenging course. It is a course that requires a lot of work and requires that the work be done consistently over an entire semester. Most students can be excellent for a week or two but it takes a special talent to perform consistently excellent work for an entire semester at this level of difficulty.

I always believe that every student is capable of making an A but you were one of a small group that actually did earn that A. You were one of the few who managed to make it happen. Good for you! I am proud of you and pleased by how well you did. More importantly, I hope you are proud of yourself.

It was a pleasure working with you this semester. I enjoy having the chance to work with every student but I’m always delighted by those students who have the drive, ambition, and passion necessary to get an A in this course. If I can ever be of assistance in the future, please let me know.

And, finally, I want to request a favor. For the last decade or so, it has been my custom to ask the students who make an A in any of my classes to write a paragraph or two to guide the next group of students. You may remember reading (at the beginning of this semester) the advice given by the A students from last fall. I think it is very helpful for one group of excellent students to explain to the next group how it is done. You can save them a lot of time and experimenting. Students will believe what other students say 100 times more often than they will believe what I tell them. “Here’s how I managed to get an A in Professor Hoyle’s class” can be a great help to the next “crop” of students. I hope, over the next 2-3 days, that you’ll sit down and write out what you did that worked so well and send it to me by email for distribution next fall.

My only two requirements: be serious and be honest. You know how you did it – share that information with future students.

Congratulations!!!! Go out and celebrate!!! YOU DID IT!!!!

Have a great summer!!!! Yeah, work hard but make time to read books and go to museums and check out art galleries and talk to as many people as you can and learn about who they really are. You need a broad education about the wonders of life as well as a great knowledge of accounting.

JH