if text is copied and pasted into Word from e.g. Wikipedia, it can include non-breaking spaces (small raised circles instead of the usual dot seen with show/hide is turned on). These are practically impossible to type by mistake! Similarly, when text is copied and pasted from PDFs, odd line breaks/returns are imported—another give-away.Thought I'd pass that along.
Friday, March 24, 2017
Monday, January 16, 2017
As a new prof I had the singular experience of an undergraduate handing in a paper to the course I was teaching on the sociology of education that, as it turned out, I had written when I had taken the same course a decade or so before. Unbeknownst to me, the paper must have been photocopied by a classmate of mine years before and entered into the frat files. The original title page must have long since been discarded by the frat boys as who knows how many times the paper had been recycled, so my student merrily retrieved it from the files, typed up a new cover page with his name, and handed it in to it's author. It took me a minute to recognize after all those years, but there was something about it that was strangely familiar. And as it occurred to me that it was in fact, my paper, I recognized that the typing had the cracked line through the 'e' that I had had on my typewriter that year. I actually went through the effort of digging through the garage to find the original paper, and yup, it was identical. So that was an 'F' for the student, and an anecdote for me.
The other example that I recall from roughly the same period was a colleague of mine telling of the case of a seminary student in his PhD defense. The originally scheduled external examiner had called in sick, so the supervisor had located a replacement. "You're in luck," he had told the student, "Dr X from the University of Y in Germany is in Calgary for a conference and has agreed to fly up to sit in as your external. It's a great honor because he is apparently quite an expert on your topic." But of course, turns out he really was an expert because when he opened the student's dissertation, he discovered at once it was his dissertation translated from the German. Since one does not get a PhD in theology for translating someone else's doctoral thesis, that was another FAIL. But I have to say, great example of the direct intervention of God against someone unworthy of the doctor of theology, because how else to explain the coincidence? The poor bastard must have thought himself completely safe from discover (this being decades before the internet).
Moral of the story: don't plagiarize or god will get you. .
Sunday, January 1, 2017
Assessment Design / Test Construction: General Principles
Defining Learning Objectives for Assessment
Assessment is about Learning
Appropriate Failure Rate
Directions in Test Questions
Question Order on an Examination
Item Writing 1
Item Writing 2
Item Writing 3
Examples of Poor Test Questions
Internet Memes (Revisited)
These are cartoons or photos you may have seen circulated for humorous effect, but which raise valid points about assessment practices.
My Assessment Pinterest Board (Memes and Cartoons)
Posting Student Work
Question Order on an Examination
Write Better Instructions
Three Minutes with Scott MacLeod
Setting Writing Assignments
ESL Test Answer
ESL Students and Tests
Multiple Choice Tests
Question Order on an Examination
How many alternatives should a multiple-choice question have?
Where is the best spot for the correct answer?
Write Better Instructions
Taking Student Questions
Studying for Tests
Taking Multiple-Choice Tests
Testwiseness Part I
Testwiseness Part II: Why Teach Students to be Testwise
Testwiseness Part III: Do Testwise Techniques Work?
Taking Essay Tests
Test Results and Analysis
The Bell Curve (JohnMighton)
Teaching Reading and Writing
Teaching and Evaluating Writing: Part I
Runté GoH speech on Future of Writing Instruction and Publishing (Audio)
Sunday, December 11, 2016
Another stimulus from the internet. So two principles here:
First is, "you can't mark it wrong if it answers the question asked" so, the solution here is to you congratulate the student on her imagination and authenticity and then ask the student to read you the answer in English so you can tell if they have the concepts you were actually testing for. And then you ask her if she'd care to share the (translated) answer with the class because she probably has a better sense of the actual answer than you do, especially if she asked mom about the family's stories handed down from great-grandmother.
Second, it is a good reminder of an assessment technique to use with ESL students in your classroom (though in this case, the student was obviously bilingual since she is taking the exam in English): it's vital to give the ESL students in your class the opportunity for success. If they are constantly struggling to express themselves in English, the quality of their responses will obviously always be less sophisticated and more error-prone than their native-English peers. So, whenever possible when doing this sort of written assignment, get them to write it in their own language, and have their parent or another adult fluent in their language grade it for content. Because, if they're writing fabulous poetry in Chinese or Italian, you don't want to give them a grade in the poetry unit that says they don't understand poetry. That's just incorrect assessment. English Literature and social studies instructors can make good use of portfolio assessment strategies that allow for the inclusion in the portfolio of pieces in the student's own language, alongside pieces that show the students progress/improvement in writing in English. The inclusion of work in their own language is a huge moral booster for the student—that says, in effect, "I may not have mastered English yet, but I'm still a pretty damn good poet!"—which, if one's purpose is to teach kids an appreciation of poetry, is kind of the whole point of the exercise. Without occasional success and reinforcement, what we are teaching the student is that school is a horrible experience to be avoided whenever possible; that they are a failure at our subject— that poetry, or whatever is not for them; and that they are failures and unworthy generally. I don't know a lot of teachers who signed up so they could oppress and discourage kids, but I have met quite a few who think their job is to enforce English instruction to the point of tears and beyond. Yes, let's have assignments in English, but let's make sure we have some parallel assignments in their own language to remind the student—and frankly, to remind us, the instructors—that these are talented, motivated, and hard-working kids. The reason they only put two line answers to that essay question is that writing two lines in English took them as long and five times the effort as the native-English speaker who wrote three pages. Seeing their eight-page essay in Italian or page in Chinese (Chinese characters are words, remember) with an "B+" from their Uncle reminds everyone that this is not a lazy or inferior student.
It should go without saying that the native language assignments must be optional for the student...at some point, asking a student to do any work in their mother tongue becomes the layering on of unnecessary additional work, which would be both dysfunctional and maybe a little racist. But it should remain an option in every teachers' toolbox.
Friday, December 2, 2016
Friday, November 25, 2016
- Check that you have all the question pages; check the back of the pages to see if there are questions there.
- Read all test or examination instructions and questions.
- Note which question or sections are compulsory, optional, or allow for choice. In the instructions, underline or circle all key words or phrases such as ‘and’, ‘or’, ‘answer three of the following. . . .’
- Make certain you know the number of questions you are required to answer.
- Note the numerical value of each question and the total time allowed for the test or examination. Then convert the numerical value to the appropriate number of minutes per question. Remember to deduct five minutes for preparation and at least ten minutes for reading, correcting and revising. The proof-reading and revision will be made easier if questions are answered on every other line.
- Within each question underline or circle all key words or phrases such as ‘discuss’, ‘list’, ‘explain’, ‘compare’, ‘three causes of’. . . .
- Choose the question you wish to do first. The question you answer first should be the one with which you feel most confident. Spend only the appropriate amount of time, or less, on this question.
While answering the questions:
- For each question, jot down on a separate piece of paper (rough work), the key ideas or facts pertaining to that question.
- Decide which ideas or facts to include in your answer, based on what is required in the question. For example, if the question asks for three facts or concepts, choose from your list of key ideas or facts the three most important facts upon which you can expand. Do not waste valuable time providing information not specifically asked for by this particular question.
- Take into account the marks allotted for each question or sub-question, and decide how much emphasis should be placed on each idea or fact. For example, if a question asks for three concepts or facts, and the total number of marks is twelve, assume that each fact or concept should be developed with four points each in order to add up to 12 – or three points each if marks are also assigned for organization and style.
- Decide on the format of the answer. Will you discuss each concept in a paragraph of its own, or will you compare concepts within a paragraph?
- Use headings or underlining to make it easy for the marker to quickly identify your key concepts.
- Leave some space between the end of each answer and the start of the next one. This space will allow you to add further information if you recall something important, and you have time at the end of the examination.
- Go immediately to the next question of your choice. Start each answer on a new page. Draw an arrow to the next page to make sure marker finds all your answers.
- Make certain you have answered the required number of questions.
- Read all your answers carefully and correct or revise where necessary.
- Always type your answers if given the option; typed papers are worth 10% more than the identical paper handwritten because typing is easier to read and therefore typed essays appear to ‘flow’ better.
Friday, November 18, 2016
Here is a personal anecdote from back when I worked as a full-time Test Development Specialist.
I had been working for a few years as a Test Specialist when the senior manager suddenly realized that there was only one person on staff who knew how the software for grading our exams worked. As it occurred to him that the whole operation could grind to a halt were something to happen to that individual, he transferred me into the computer programming section as the backup programmer. (This made sense to him because I was the only person back in those days—30 years ago!—who owned his own personal computer, so I looked like a computer guy to him.) Since I had no programming training, and since all our computer programs were written in FORTRAN, my boss signed me up for a university-level correspondence course on FORTRAN programming. By the time the course materials had arrived in the mail two weeks later (this before online courses!), however, my boss had come to his senses, hired a team of FORTRAN programmers, and switched me back to writing tests. I therefore had no further need to learn FORTRAN, and I pushed the unopened box of course materials under my bed and forgot about it.
Six months later I received a notice in the mail saying the examination for the FORTRAN course would be at such and such location in a week's time. And I thought, "Well, the course has already been paid for, why not take the test?" Not, you understand, that I had ever opened the box or looked at any of the course materials. I just wondered if I could pass the test on my knowledge of test design.
The examination, if I recall, was 70% multiple-choice and 30% written response. I had no hope of answering the written response part, since I had absolutely no idea how to write a FORTRAN program. I did repeat back the question (because some markers wrongly give at least one mark for a student filling in the space with something) and attempted a line or two of code based on examples from the multiple-choice part of the test, but since I had no idea what it was I was copying, I highly doubt it made any sense at all.
The multiple-choice questions, however, were a different matter. As a test specialist, I was able to examine each question for the tiny flaws that gave the answer away. Examining the alternatives for the odd one out, or for one answer that was significantly longer than the others, and so on, (see previous post for these techniques) I gave it my best college try, even though I often had no idea what the question was asking, let alone what the correct answer might be.
Three weeks later I received a letter saying I had obtained 70% on the test; which considering I had left the written response essentially blank, means I must have aced the multiple-choice. Not bad for a topic I literally knew nothing about!
I subsequently wrote the institution in question pointing out this weakness in their tests, and sometime later they hired me to give a one-day workshop on the do's and don'ts of multiple-choice item writing. There is, therefore, no use asking me how to register for that course; these techniques won't work to pass that test again!
This was probably an extreme case; and, to be fair, such tests are not intended for people who make their living designing multiple-choice tests. Most people would not have found the flaws quite so obvious.
In sharp contrast, I sat down with a copy of Alberta's Grade 12 Mathematics Diploma Examination to see what I could get using these test-wise tricks. In this case, I only managed 23%, slightly less than one would expect by pure chance. This test had been written by test experts and it was impossible for me to get through it without actually knowing the course content (calculus) which I did not.
Most classroom teachers' tests are somewhere in between these two extremes. A student will not likely be able to pass a teacher-constructed exam solely on knowing testwise strategies, but such strategies are likely to improve the testwise student's grade a by significant percentage if the teacher does not know and implement correct test construction technique.
In that context, it is worth noting that teachers should be cautious about using test banks that come with textbooks. Some of the larger textbook publishers do employ professional test developers as editors for their test banks, but many do not. Test banks are sometimes written by the author of the textbook (who likely will not have any expertise in test-construction) but are more often written by graduate students from that discipline desperate for cash (and who definitely do not have test-construction skills) and dashed off over a weekend before the publisher's deadline. In other words, don't assume the test bank is worth anything. At a minimum, any instructor using a test bank written by anyone else should edit the questions before putting them on their test to ensure the questions are less flawed and more relevant to their own classroom. (On the other hand, starting from a flawed test bank is sometimes easier than writing one's own items starting from a blank page. Using a test bank as initial rough draft can be okay, provided one guards against it being all rote memorization. You will probably need to write your own higher level thinking questions, because publishers can't pay grad students enough to come up with those!)