Posted in Eleanor Cawley

Why Write a Rubric? Focused Feedback, with Consensus Across Raters that’s why!

Focused Feedback

After many discussions with other therapists wanting to learn how to write a rubric. I decided that a blog post was needed to clarify why a rubric is important, not only to ourselves but most importantly, to our clients, other professionals and parents/caregivers. As I began my research for this blog, I came across the term “focused feedback.” I think that it is a great way to express the intent of a rubric. I think, though, that a rubric is more than just feedback to the client or ourselves. I am thinking in lateral terms, from therapist to therapist or teacher for example.  A rubric provides clearly understandable expectations for goal progress, from professional to professional, professional to parent, professional to student or client. A well written rubric is easy to understand by all parties, including the client and caregivers, with clearly identified expectations for progress without having to search for more information. My book, Using Rubrics to Monitor Outcomes in Occupational Therapy, discusses, in more detail, the importance of writing rubrics clearly.


Across Raters

When this term is used, mainly in research, it generally means inter-rater reliability or the degree of agreement among raters. A rater is someone that is using your rubric to assess the client’s performance. Think about times when you are not available, someone is covers for you. It may also be that a classroom aide is collecting data using your rubric. Data collection on all goals occurs throughout the day and is the responsibility of everyone working with the student. You want all the raters to be consistent, considered reliable and see the same progress that you do regardless of their training. Raters can vary in judgment without clear and consistent guidelines [definitions]. According to Hung and Wang (2012), individuals judged by the most severe raters are at a disadvantage, because their ratings are always lower. Conversely one might assume the same is true of a lenient rater in that the same disadvantage may occur because their ratings were always higher. Neither the severe or the lenient rater actually provide usable data. The only way to accomplish this is to have clear and concise definitions for scoring. It is difficult to gain a consensus between raters when the definitions for scoring are vague. Typically, one word definitions [such as fair, poor, and good, maximum, moderate and minimal] are very subjective, leaving the rater to guess at what that means. It the heat of a classroom or therapy room, it is difficult to flip back and forth between explanations and rubrics, possibly leading to non-compliance of a rater or that dreaded subjective assessment. This really drives home the point of doing a thorough task analysis, keeping the rubric clear and concise with definitions for scoring defined within the rubric itself.

I recently assess a student for assistive technology. One of the staff concerns was the student’s lack of compliance with the current device.  Upon interview, the student revealed sensory issues [sound and proprioceptive feedback] with the use of a standard QWERTY keyboard. The student also revealed a preference for a virtual keyboard, such as on a tablet. The student typed equally as well on a standard keyboard and a virtual keyboard so that switching devices would most likely not cause a problem.  So a ‘Compliance Rubric’ was developed to determine if the student would use the device in class. I would like you to look at two rubrics exploring her compliance. Check out the definitions for scoring. Which rubric do you think will reveal more accurate data on compliance?

Rubric 1

Criteria Score 0 Score 1 Score 2 Score 3
Student will bring  tablet to class Refused or maximum assistance Moderate assistance Minimal assistance Independent

Rubric 2

Criteria Score 0 Score 1 Score 2 Score 3
Student will document that tablet is requested Student refused to document in agenda, staff will document Student documents the request for tablet with 2 or 3 prompts Student documents the request for tablet with one prompt Student documents the request for tablet without prompts
Student will bring tablet to class Student does not have the tablet with and refuses to retrieve it when prompted Student does not have tablet but will retrieve it with prompts Student does not have tablet but will ask to retrieve  tablet without prompts Student brings tablet to class without prompting
Student will use  tablet to work on assignments in class Student refuses to use the tablet to complete assignment Student will open the tablet and activate the app with prompts Student will open the tablet and activate the app without prompts Student will open tablet, activate the app and begin to work on assignment without prompts

It is very difficult to assess compliance without clear and concise expectations. One of the criterion that should be considered is, does the student know that the tablet is expected in class during a particular period on a particular day? We need to clearly indicate that this student was notified of this expectation. Then and only then can we have the expectation that the device will be there and make judgments on her ability to comply with the request.

Data Collection

Data collection should also be on the same page with a location to provide notes provided. This can be on the back of the page or in a separate notebook. When the data collection table is not on the same page, we run the risk of either delayed entry or no data collected at all. The data collection table needs to be as clear and concise at the rubric itself. It should include the date, time or period, the criteria being assessed and a place to insert the score.

compliance data collection

Compliance Rubric


I am an occupational therapist with 18 years of experience in the pediatric sector, much of that time as an independent contractor. I am very passionate about my work and my writing. My degrees include a Bachelor’s of Science in Health Sciences and a Master’s of Science in Occupational Therapy from Touro College. Since graduating as a non-traditional student, I have worked in a variety of settings throughout the life span but settled in the area of school-based therapy. My interests lie in the area of using technology to support independence and I often train students to use programming not only to monitor their own goal progress but also support educational, vocational and life skills. Another area of particular interest is documentation. As an independent contractor for many years, I feel that it is important to align methods of documenting goal progress with educators for greater consistency and understanding when writing for an IEP. It is better to plan a format for documentation used in the IEP, such as for assessment and goal progress and that a rubric, in many ways, fulfills the need for consistency in documentation across all domains. Combining my interest in technology and documentation, I use Microsoft OneNote to maintain all documentation. I create a digital notebook for each student or patient with any forms required uploaded as templates which can then be completed, and saved automatically. I strongly believe in student centered approach to therapy. Students must be active participants in developing goals and documenting progress. In order to help students understand their progress, I teach my students to develop electronic portfolios and to use spreadsheet programming with graphs to collect data and view progress, whenever possible.

One thought on “Why Write a Rubric? Focused Feedback, with Consensus Across Raters that’s why!

Comments are closed.