Psychometrics

Defensible exams build confidence and certainty.

Valid, reliable, and fair exams guided by psychometric science and a vital mission: to ensure candidates are competent and qualified.

The science of testing. Defensible and fair exams.

Psychometrics is the science of designing, planning, and implementing defensible assessments. Our psychometricians bring technical expertise and a client-focused attitude to create processes and products that help ensure students, candidates, and professionals are assessed fairly and reliably.

Measuring competence. Verified test results that you can defend.

Our testing experts use industry best practices to produce well-designed exams, following the stages of our Assessment Lifecycle. Our exams are fair and defensible, from content definition and blueprinting through to exam delivery and reporting.

Our approach. Your results are strongest when we work together.

Psychometric science is complex, but at YAS, we approach it with a spirit of openness and curiosity. Where some might guard their deep knowledge of test science, we share our expertise, experience, and leadership freely with the industries and clients we serve. Working together, with a common understanding, we build an assessment that is defensible, relevant, fair, and perfectly tailored to your needs.

Our testing experts (psychometricians) follow a series of industry best practices known as the Assessment Life Cycle.

These rigorous methods – and the precise and proven science behind them – allow us to design, develop, and deliver assessments that are valid, fair, and legally defensible.

Overview Defining competencies and creating specifications Item development Assembling examination form(s) Test administration Item and test analysis Scoring and reporting Standard setting

At Yardstick Assessment Strategies (YAS), our testing experts (psychometricians) follow a series of industry best practices known as the Assessment Life Cycle. These rigorous methods – and the precise and proven science behind them – allow us to design, develop, and deliver assessments that are valid, fair, and legally defensible.

These methods aren’t some well-guarded industry secret – they’re drawn from the available research literature. In line with this idea, we don’t believe in keeping our methods hidden from our clients. Instead, we treat our clients as partners; and will involve you at every step of the examination process – from item development all the way to examination delivery and analysis. Our experts will work with you to ensure that you understand the assessment development process, and feel supported and engaged every step of the way.

Stage 1.
Defining competencies and creating specifications

Whether we’re conducting a job analysis, building a competency profile, or defining a curriculum, our testing experts will consult with subject matter experts, practitioners, and published materials to strategically identify the knowledge and skills that are required for safe,  competent and ethical performance in a given subject area.

We use this knowledge to design an examination “blueprint” (also known as a table of specifications), which identifies the specific knowledge and skills that will be assessed on your examination; and how heavily each area should be weighted on the examination to accurately reflect the breakdown of knowledge and skills required for competent practice.

Our specialized services include:

  • Competency profiling and job task analysis
  • Surveys and consultation
  • Blueprint development

Stage 2.
Item development

If you want to accurately measure someone’s qualifications, having the right number and right type of questions on your examination is essential. Filling an examination with accurate, valid, well-written questions is careful work, requiring meticulous review and revision.

Luckily, our testing experts are available to help you with this, and will coach you through the item development process. This includes training your subject matter experts on how to write effective examination questions; evaluating your draft items to ensure they are fair, clear, and conform to item writing best practices; and project managing the item development process to ensure that the items being written and approved for your examination conform to the target specifications in your examination blueprint.

Our specialized services include:

  • Item writing workshops and guides
  • Professional item development
  • Item development project management
  • Item translation services

Stage 3.
Assembling examination form(s)

An effective examination must be carefully designed. It needs to have the right number of questions to effectively assess the subject area; it needs the right breakdown of items to be reflective of the examination blueprint; and it needs to include questions that assess candidates using the right distribution of cognitive levels, question formats, and target populations in order to be reflective of the topic area being assessed.

This may seem like a daunting and exacting process, but our testing experts will work with you to design examination forms that fit these specifications. To do this, YAS draws questions from the item bank that we help you create in order to make sure that any examination forms fit the unique constraints of your examination. We can even automate this process for you using specialized software.

Our specialized services include:

  • Automated test assembly (ATA)
  • Facilitation of the examination validation process
  • Practice test development

Stage 4.
Test administration

Once the examination forms are assembled, candidates finally have the opportunity to demonstrate what they know and can do. To make sure this process runs smoothly, our psychometricians work with our examination administration team to ensure that your examination is administered in a way that that securely handles sensitive information, consistent with the stakes (or seriousness) of your examination.

Our specialized services include:

  • Paper to computer-based transition
  • Collusion detection
  • Exam delivery in test centres
  • Remote proctoring services

Stage 5.
Item and test analysis

After the examination is administered, our testing experts will review the results to ensure that both the exam, and each item that comprises it, work as they should. In doing so, our testing experts can evaluate the overall reliability of your examination; and identify any items that are too easy, too hard, or fail to discriminate between competent and not-yet-competent candidates. Based on these post-administration findings, we will consult with you to decide whether to retain, revise, or remove questions for future assessments, in order to maximize the effectiveness of your assessment moving forward.

Our specialized services include:

  • Classical Test Theory (CTT) item analysis
  • Item Response Theory (IRT) item analysis
  • Differential Item Functioning (DIF) analysis

Stage 6.
Scoring and reporting

After candidates finish writing your examination, it is important to ensure that their results are calculated efficiently and accurately; and reported back to them as soon as possible. At YAS, our testing experts will work with you to get your examination scored; and will create customized reporting solutions that break down candidate scores based on specific competencies, regions, or other categories of interest.

If you use different examination forms, our testing experts can also statistically equate candidate scores across the different form. This ensures that – even if candidates receive different items – their overall scores are comparable, improving the fairness of the assessment process.

Our specialized services include:

  • Redundant scoring using different forms of software to ensure accurate calculations
  • Statistical equating between examination forms
  • Customized candidate, school, and national results reporting

Stage 7.
Standard setting

Credentialing examinations and other “criterion-referenced” assessments determine a candidate’s success based on whether or not their score on the examination meets or exceeds a set passing mark (or “cut score”). This cut score serves as the dividing line between candidates who are “competent” versus candidates who are “not-yet-competent”.

This is not a decision to be made lightly. A valid, fair, and legally defensible examination requires a well-informed cut score. At YAS, our testing experts will work with your subject matter experts and stakeholders to determine a passing mark for your examination that is informed by the knowledge and skills that your examination is meant to assess, as well as the difficulty of the examination and items that comprise it.

Our specialized services include:

  • Consultation and training on various standard setting methods (e.g., Angoff, Ebel, Bookmark, Borderline)
  • Facilitation of standard setting sessions with subject matter experts

Want to make your high-stakes examination defensible?

Talk to us