M. Support integration of diagnostic assessment into short-format training

Summary:🚲

Short-format training is more effective when learners must demonstrate they understand course material during the instruction and instructors adjust their teaching to address misunderstandings.

Assessment promotes effective instruction and provides evidence that learning objectives are met by learners. Assessments during the learning process (e.g., exercises, homework, quizzes) are formative or diagnostic, and help instructors and learners measure the progress of learning. Assessment at the end of an instructional unit (e.g., final exams, lab practicals) are summative and demonstrate that learning has been achieved. SFT generally lacks formal diagnostic assessment.

The short duration of SFT may disincentivize the inclusion of diagnostic assessment, but SFT instructors may also lack training and experience in designing effective assessments that are feasible for short time frames. For example, assessment can be done in advance of the course with information from course registration that describes learner's goals and professional experiences. Polls and questions during instruction help learners and instructors gauge and correct understanding. Diagnostic assessments are an accepted best practice for effective instruction 1, and instructors need to be supported in multiple ways to understand and leverage this practice.

How might this work:🚲

Training (such as developed by The Carpentries 2 and GOBLET 3) helps SFT instructors understand the value of diagnostic assessment and how to integrate actionable assessments into their teaching. This training has been implemented effectively in SFT for years and could be generalized for broader use. Open resources for developing and utilizing diagnostic assessments can reduce instructor burden in designing their own, but instructors and instructional designers still need to learn about the importance and utility of diagnostic assessment, and some may need to learn how to deploy, and interpret it. Metrics of success include increasing numbers of instructors developing and including diagnostic assessment and documenting how results prompt them to adjust their teaching.

Benefits to the learners:🚲

  1. Repeated diagnostic assessment can demonstrate objectively to learners that they have changed from being unable to solve a problem to being able to solve it.
  2. Instructors who recognize the importance and utility of diagnostic assessment make learners true partners in learning 4.
  3. Learners are more engaged when they are actively participating through multiple diagnostic assessment tasks where performance is improving.
  4. Learners can learn to diagnose their strengths and weaknesses, assess their progress, target areas for improvement, and identify what they may need to learn next.

Incentives to implementers:🚲

For Instructors

  1. Sharing existing resources for diagnostic assessment in the SFT context would reduce instructor implementation burden.
  2. Instructors have a better understanding of where instruction needs to be adjusted or supplemented in response to diagnostic assessment.
  3. Instructors can have increased confidence that learners are achieving learning objectives.

For Instructional Designers

  1. SFT programs that are more interactive can develop a reputation for effectiveness.

Barriers to implementation:🚲

  1. The idea and deployment of diagnostic assessment may seem disruptive to SFT instructors and learners.
  2. Funding, evaluation expertise, and time will be needed to develop general resources for diagnostic assessment in SFT. Effective diagnostic assessment depends to some extent on group size. This complicates the development, deployment, and use of diagnostic assessment by learners and instructors alike. There is no “one assessment fits all” approach.
  3. Instructors will need support in finding balance between covering material in time available and integrating assessments.
  4. Training must also train instructors to adjust their instruction in response to assessment feedback.
  5. Assessments should also be interesting for learners to use (e.g., incorporating relevant contexts or culture where possible).

Survey question🚲

After reading the survey instructions and consent page and the recommendation above, please rate and leave feedback using the survey questions below:

Create your own user feedback survey

  1. Irons, A., & Elkington, S. (2021). Enhancing Learning through Formative Assessment and Feedback (2nd ed.). Routledge. https://doi.org/10.4324/9781138610514. ↩

  2. Karen Word, Maneesha Sane, Kelly Barnes, Sarah M Brown, François Michonneau, Christina Koch, Rayna M Harris, Erin Becker, Brian Ballsun-Stanton, Gerard Capes, Toby Hodges, Serah Njambi, Sarah Stevens, Zhian Namir Kamvar, Angela Li, Ariel Deardorff, Pradeep Eranti, SherAaron Hurt, Aleksandra Nenadic, … Ashwin Vishnu Mohanan. (2021). carpentries/instructor-training: The Carpentries Instructor Training November 2021 (v2021.11.18-1). Zenodo. https://doi.org/10.5281/zenodo.5709383. ↩

  3. Via A, Palagi PM, Lindvall JM et al. Course design: Considerations for trainers – a Professional Guide [version 1; not peer reviewed]. F1000Research 2020, 9:1377 (document) https://doi.org/10.7490/f1000research.1118395.1. ↩

  4. Robeva, R.S., Jungck, J.R. & Gross, L.J. Changing the Nature of Quantitative Biology Education: Data Science as a Driver. Bull Math Biol 82, 127 (2020). https://doi.org/10.1007/s11538-020-00785-0. ↩