I started at Questar as a scorer several years ago, and I feel a sense of pride in how Questar has always hand-selected seasonal scoring staff, who remain loyal, dedicated, and reliable in a world that has gone test crazy. Now, as the manager of handscoring at Questar, I continue to ask our pool of graduate students, house managers, retired educators, and business professionals why they keep coming back season after season to score student responses. The answer is simple: They want to make a difference.
I often think back to one of my first evening shifts several years ago when I noticed a young grade school teacher crying. When I asked her what was wrong, she said she was nervous about being responsible for giving the students the correct score, despite the training she had been given to use the rubric and training materials. That moment has stuck with me as I strive to ensure that my team takes great care to give each and every student a score that is the most thoughtful and trustworthy — just as I would expect nothing less from the company scoring my own daughter’s tests.
Becoming a scorer at Questar is not simply answering an ad in the newspaper, showing up for work, and applying scores at random to faceless, nameless student tests. It begins with a process built on years of educational experience in providing sound practices to train people to score with accuracy and integrity. An introduction to Questar’s philosophy, an application process, an evaluation of content performance, and one-on-one interviews are just the beginning of how we hand-select individuals to work on our projects. The rigorous training that ensues once a person even gets their foot in the door ensures that no one is scoring who is not fully trained and qualified to assign scores to the students who each response represents. Once scoring begins, scorers are then continuously monitored on their accuracy, rates of agreement, and drift.
Handscoring student responses requires smart recruiting, rigorous training, and diligent monitoring. After reaching out to our existing talent pool, if we need to recruit additional scorers, we deliberate with careful consideration which media will gain us the best potential scorers. We use different shift models to capitalize on various resource schedule restrictions, and rely heavily on word of mouth and referrals to help us fill in the gaps.
Our expert staff of scoring directors and team leaders monitor the scorers’ performance on a day-by-day, hour-by-hour basis to ensure that we offer our clients exceptional human scoring service. Our feedback is presented in a positive and useful way by our trained staff and consistently results in increased quality and performance.
It’s time we illuminate what happens behind scoring room doors, if for no other reason than to remedy any misconceptions people have about who is scoring student tests. The “ability” in account[ability] includes all of us — not just educators.