Subject Matter Experts (SME) in Exam Development

Subject matter experts (SME) are an important part of the process in developing a defensible exam, especially in the world of credentialing and certification.  A SME is someone with deep expertise on the content of the examination, whatever that might be.  If the test is on repairing a certain brand of X-ray machine, you need a group of SMEs with that specific training and experience.  They will use this knowledge to contribute important aspects of the credential, from writing items to strategic organizational management to marketing.

There are several ways that their input is required on the exam.

1. Serving on the Certification Committee (if relevant) to decide important things like eligibility pathways
2. Serving on panels for psychometric steps like Job Task Analysis or Standard Setting (Angoff)
3. Writing and reviewing the test questions
4. Answering the survey for the Job Task Analysis

They also provide input on many matters related to the organization.

1. Serving on other committees, such as Education or Finance
2. Writing/creating learning materials
3. Teaching workshops or courses
4. Marketing and selling the credentials
5. Strategic guidance for the entire organization (serving on the Board)

Who are Subject Matter Experts?

A subject matter expert (SME) is someone with knowledge of the exam content.  If you are developing a certification exam for widgetmakers, you need a panel of expert widgetmakers, and sometimes other stakeholders like widget factory managers.  For example, I once worked on exams for ophthalmic technicians; some of the SMEs were ophthalmic technicians, and some were clinic managers, but some of the SMEs (and most of the nonprofit board) were ophthalmologists, the medical doctors for whom the technicians worked.

You also need test development staff and psychometricians.  Their job is to guide the process to meet international standards, and make the SME time the most efficient.

The subject matter experts should also be distributed amongst other demographics. Depending on your test, this might be geographical region, type of role, specialty, training, or native language.

Recruiting SMEs

The recruitment of subject matter experts can be an issue for some organizations. Those with large budgets can pay honorariums, or even pay for travel to exotic locations for meetings to accomplish the tasks below. Small organizations typically rely on volunteers who do the work for the benefit of their profession. There are some non-financial benefits such as networking and the prestige of adding “Served on national certification committee” to your CV, but this is not always enough.

Example: Item Writing Workshop with SMEs

The most obvious usage of subject matter experts in exam development is item writing and review. Again, if you are making a certification exam for experienced widgetmakers, then only experienced widgetmakers know enough to write good items.  In some cases, supervisors do as well, but then they are also SMEs.

An item writing workshop typically starts with training on item writing, including what makes a good item, terminology, and format. The psychometrician or exam development manager will provide the training to the rest of the group.  Item writers will then author questions, sometimes alone and sometimes as a group or in pairs. 

For higher stakes exams, all items will then be reviewed/edited by other SMEs. Items might be projected on a screen at the front of the room, and the SMEs discuss the content and alignment.

Example: Job Task Analysis

Job Task Analysis studies are a key step in the development of a defensible certification program.  It is the second step in the process, after the initial definition, and sets the stage for everything that comes afterward.  Moreover, if you seek to get your certification accredited by organizations such as NCCA or ANSI, you need to re-perform the job task analysis study periodically. JTAs are sometimes called job analysis, practice analysis, or role delineation studies.

The job task analysis study relies heavily on the experience of Subject Matter Experts (SMEs), just like Cutscore studies. The SMEs have the best tabs on where the profession is evolving and what is most important, which is essential both for the initial JTA and the periodic re-set of the exam. The frequency depends on how quickly your field is evolving, but a cycle of 5 years is often recommended.

JTAs are described in greater detail in this dedicated blog post.

Example: Cutscore studies

When the JTA is completed, we have to determine who should pass the assessment, and who should fail. This is most often done using the modified Angoff process, where the SMEs conceptualize a minimally competent candidate (MCC) and then set pass/fail point so that the MCC would just barely pass.  There are other methods too, such as Bookmark or Contrasting Groups.

For newly-launching certification programs, these processes go hand-in-hand with item writing and review. The use of evidence-based practices in conducting the job task analysis, test design, writing items, and setting a cutscore serve as the basis for a good certification program.  Moreover, if you are seeking to achieve accreditation – a third part stamp of approval that your credential is high quality – documentation that you completed all these steps is required. Performing these tasks with a trained psychometrician inherently checks a lot of boxes on the accreditation to-do list, which can position your organization well for the future. When it comes to accreditation— the psychometricians and measurement specialists at Assessment Systems have been around the block a time or two. We can walk you through the lengthy process of becoming accredited, or we can help you perform these tasks a la carte.