Best practices and tips for open-book, take-home exams

  • 03/23/2020 9:22 AM
    Message # 8849877

    Please post your questions, answers, suggestions and best practices for writing and developing open-book, take-home exams (offered online or not).

  • 03/23/2020 6:38 PM
    Reply # 8851912 on 8849877
    It would seem that for a good question, internet / book access is likely to be more of a distraction than a help. 


    While these are not perfect (is any question "perfect", really) here are examples of exams from the 

    biofundamentals-based course sequence [http://virtuallaboratory.colorado.edu/Biofundamentals-coreBIO/]


    coreBIO part 1 (fundamentals): 

    – http://virtuallaboratory.colorado.edu/Biofundamentals-coreBIO/sample-test/exam%201-2017%20answer%20key.pdf

    –  http://virtuallaboratory.colorado.edu/Biofundamentals-coreBIO/sample-test/MIDTERM3-2017-answerd.pdf


    and coreBIO part 2 (genetical): 

    – http://virtuallaboratory.colorado.edu/Biofundamentals-coreBIO/NB-readings-GENETICS/Midterm-2-answers.pdf

    – http://virtuallaboratory.colorado.edu/Biofundamentals-coreBIO/keynote-2018-pdf/MCDB2222-Midterm-1-2018%20answered.pdf


    and this spring's development biology course: 

    http://virtuallaboratory.colorado.edu/DEVO@CU/Midterm-1-2020%20answered.pdf


  • 03/24/2020 11:37 AM
    Reply # 8854270 on 8849877

    I love open-book, open-everything exams for Intro courses. I’m sorry it has taken this crisis to bring them into vogue, but I bet there will be more of them after this pandemic clears. Hooray for bottlenecks.

    I’ve been giving them for years, and here are a few (hopefully helpful) elements:

    • If you want to grade less but cover more, a ‘grid’ style exam can be helpful. Attached is an example of an Intro exam portion in which four questions are asked about four papers about a week before the open exam. In the actual exam interval, only four total questions (out of 16 options) are actually asked. Students study the entire set of possibilities (in my experience: creatively and in great depth) but the actual grading is only on fraction. I have done this in 5-6 classes of 750-800 students each, albeit with a larger team of TAs. 
    • For more general open exams:
      • You don’t need to get tricky with genius-level questions. If what you want is for students to be able to explain how protein structure impacts the association of a virus with a cell, then you can literally just ask “Explain how protein structure impacts the association of…”. Remind students to be as thorough, detailed, and complete as possible in their 3 sentence limit. 
      • Because the exam is open, you can give tighter formatting demands. I would never ask a student to ‘Limit your answer to 12 words, total’ on in-class work, but with a week to perfect their answers this is a much more reasonable request. And it forces your class to identify exactly what is the most important. 
      • Choose topics that are in the news or relevant to student ideas. They have more time to parse the complexity, and students will come back years in the future and thank you for teaching them about transcriptome editing…even if you never really taught it but simply used it as basis for an open-exam question.
      • Questions without known answers can still be graded usefully, and they really challenge students. You can ask students to ‘Imagine that an expedition into the corona of the sun requires a new biomolecular substance. What kind of biomolecule would you choose, and why? Use your answer to show how much you understand about the molecular forces comprising common biomolecules.’ Good luck googling that! It is possible that 20 students will hold a focused group review session and develop a well-comprised conceptual core of an answer that uses your class material in novel, fundamentally sound ways. If this is what your students are doing with their time, you are winning no matter how the exam goes. Plus, their likelihood of plagiarizing is no worse than it would have been for a stock question.
      • Make ‘em draw. Opportunities to schematize what is known and unknown make students grapple with the material beyond regurgitating your notes back to you. Plagiarism is more obvious in many ways, but the depth has to be there to really show their understanding.
      • There will be differentiation in the scores! The real determiner of the variance is how we grade. As long as the questions are challenging, you can choose to grade in a way that will produce variation. Open exam answers for full points should be very, very good. You can also give grading descriptions like “correct answer will score 7-9 points based on quality, with scores of 10 out of 10 reserved for the very best answers in the entire class”. 
      • Out of ideas for how to make a topic tough? Just join it with some other topic. If the same exam question asks students to combine aerobic respiration with developmental signaling then you will really see who understands the material beyond artificial boundaries of those two separate exam chapters. 
      • Rubrics are fantastic for grading mid-complexity types of questions, use them…but for really complex answers, they can be more stilted than you need. Don’t be afraid to give notes about how you graded a question afterwards in combination with photocopied examples of the absolute best answers (with identifiers removed, of course). Seeing the quality of high-stakes peer work can be transformative. Your post-exam feedback for creative answers and complex questions can have some useful grey area that is motivating for students who (too often) receive messages that science is about memorizing facts and not about creative understanding.
      • For all of these suggestions, less is more. If you give 20-30 challenging questions in class, then you probably want to start with a mix of 10 of your typical questions and 2-4 highly complex questions designed specifically for an open exam. 


    I hope something in these quick-scratched notes is useful to you. I’d be happy to talk more. Stay healthy everyone,

    -Ben Wiggins

    1 file
  • 03/24/2020 11:39 AM
    Reply # 8854276 on 8849877

    I just gave a MT exam last week that was online and open book. Required a lot of time and effort to ensure that each question wasn’t easily googleable by copying and pasting the stem of the MCQ into Google and revising the wording until there were no longer any hits on exam and flash card sites. And all of these sites where students have posted screen shots of online exams or photos of their paper exams. 

    Anyways, expect to take a few hours going through that. 

    Also, ensure you explain to your students that open book does not mean easy. My students made the erroneous assumption that they were going to be able to look up each answer in their textbook. 

    Also, some students have indicated to me that for whatever reason, they find online exams more cognitively taxing than paper and pencil exams. Maybe because reading a screen is more fatiguing than reading paper? Is it more cognitively taxing to mouse an answer choice rather than scribbling a bubble MCQ answer form with a pencil?

    Stay well, take care. 

    Neil Haave

  • 03/24/2020 11:40 AM
    Reply # 8854278 on 8849877

    our questions look similar to what I do now, for in person exams.  I am less worried about the internet being helpful as I am about students' colleagues.  Given my students are across all the time zones of the world, I cannot have exams open for a short time.  As students will definitely be collaborating on exams, I am wondering how to make exams a valuable experience.  For example, I am thinking about how to write questions for which there is no one right answer and students can describe their thinking (e.g., something with synthesis or evaluation of evidence were multiple answers can be supported and argued for). Would this be effective?  Are there other options?  

    If folks have examples or further insight, I would be very appreciative.

    Thank you,

    Jennifer Doherty



  • 03/24/2020 11:40 AM
    Reply # 8854280 on 8849877

    This has been an interesting conversation with some very creative and useful tips.  As Ben notes, perhaps this event will be a catalyst for ongoing transformations in instructional and assessment practices.

    As you contemplate your own particular choices, keep in mind a lesson that many faculty have learned the hard way about doing anything transformative with your instruction:  Be clear with your students that, whatever your choices, you will do everything to ensure that they are treated fairly.  A major impediment that many faculty identify to adopting reformed practice is that students object.  What they object to has having the rules changed and being unsure about the impact on them, not the particular changes.

    So, whatever new and innovative approach you take, it is likely to further compound your students' already high stress level.  Now is not the time to suddenly transition from low level Bloom MC questions to high level open-ended questions that require synthesizing materials in ways that YOU never taught them how to do.

    Keep in mind backward design: identify the goals, write the exam, then do the instruction.  If you haven't done that from the beginning of the course, you can't suddenly upend the exam in a last minute attempt to prevent cheating.   If you have been giving exams for which students can google the answers, then you should just continue and realize that this is your problem, not your students' problem.  They will get inflated grades and you will have learned a lesson about instructional design.

    Best of luck everyone,

    Mark Urban-Lurain

  • 03/24/2020 11:41 AM
    Reply # 8854281 on 8849877

    I'm definitely in agreement with Mark. ID the outcomes, then the assessement, then the instruction. 

    I do want to address two things: 

    First, the outcomes were ID'd months ago. Instructors should not be expected to create an entirely new course in a week. Nor should they be shamed for doing their best with the situation at hand. 

    Second, Bloom's is often misinterpreted. If you read Bloom's original work, he specifically states that ALL levels of learning are equally important. 

    I teach an animal diversity course - while there are many higher level concepts to address, students do not have the lower level content knowledge to manipulate yet. Many of my questions are google-able. But, if my students go into the field, they need basic content knowledge when they are diving, in remote jungles, or simply when an animal is too fast. They need to know WHAT to look for before they can use a search engine. Many of us teach anatomy. Again, higher-level questions can be written, but sometimes "what is this structure?" is the most straight-forward way to measure student knowledge. I like the drawing idea (it's my area of research), but I've spent years developing objective rubrics for drawings. 

    I know that everyone wants to do what's best for our students. But let's also be kind to ourselves. Everyone I know is working full-throttle to adapt, and helping out peers, and keeping a positive outlook.

    So good job everyone! And excellent food-for-thought in this thread.

    Thank you for helping!

    Jennifer Landin

  • 03/24/2020 1:16 PM
    Reply # 8854587 on 8849877

    Re: Neil's concerns above:

    Also, some students have indicated to me that for whatever reason, they find online exams more cognitively taxing than paper and pencil exams. Maybe because reading a screen is more fatiguing than reading paper? Is it more cognitively taxing to mouse an answer choice rather than scribbling a bubble MCQ answer form with a pencil?

    One solution to this is having students work on paper and then convert to PDF and upload. Gradescope is a great tool for grading PDFs that I strongly recommend (free complete access for courses created by June 30 2020). You can either have students fill out a template (as we did for in-person and some online exams) or tag their PDF with the page for each question (as we've done for homework and now an online exam).

    Gradescope also has a great guide for converting paper assignments to PDF with smartphones that can be useful even if you don't use Gradescope.

    One thing to be aware for any tool used, especially assessments, is FERPA compliance - you might want to check with your institution on what's allowed to protect students' privacy if you're providing names, emails, grades, ID numbers. I personally encountered some FERPA concerns from our administrators when I first used Gradescope, but now the UCs have a contract and it's embedded in our Learning Management System.

    Kristin


SABER is a 501(c)(3) non-profit organization.
Powered by Wild Apricot Membership Software