CutScoreInstructions

Here are the instructions for helping with a Cut-Score Study.

Introduction

The purpose of this study is to use specialized ratings from expert Linux system administrators and users to determine what the passing score should be for our exams.

The work we do in this study is critical to ensuring that the cut-scores for the exams are fair. For the study to be successful, it is critical that the Subject Matter Expert (SME) understand the process and what they are asked to do. The purpose of this document is to accomplish this requirement. If you have any questions at all, please let us know.

Conventions

Throughout, this document will refer to LPI Director of Exam Development. G. Matthew Rice is LPI's Director of Exam Development and he can be contacted at mrice at lpi dot org.

The Linux Professional Institute will be referred to as LPI and the various exams will be referred to by their number code.

Materials

In order to participate in a study, an SME will have the following materials.

1. These instructions.
2. Access to an on-line application for rating the items.
   The URL will be e-mailed to the SME upon commencement of a study.
3. Access to the (potential) exam questions.
   These are the actual questions on the current or future forms of the exams. Due to the nature of this access, an NDA will be required to participate in this study with LPI.

Protecting Confidentiality

In order to participate in this study, an SME must have access to a large subset or the entire set of questions for an exam. The disclosure of the exam questions is covered by the non-disclosure agreement (NDA) that is signed with LPI before access can be provided. It is absolutely essential that an SME protect the confidentiality of this material. LPI exams cost tens of thousands of US dollars/Euros to develop and, if confidentiality is broken, the entire effort needs to be repeated at great expense.

In particular, please strictly adhere to these guidelines:

1. Do not, in any manner, share the test questions with anyone. Do not disclose item content to co-workers, supervisors, family members, students, the media, other LPI judges or reviewers, or anyone else. Avoid all forms of unauthorized disclosure including in writing, by voice, and by any electronic means. If you need to discuss the contents of any of the questions, you should only do so with the LPI Director of Exam Development. In the past, LPI has provided a mailing list for the discussion of items during cut-score studies. In these unsecured communications, please avoid disclosing any actual item content and, if content is disclosed, please limit it to the minimum necessary.
2. Do not make copies of the questions as they are presented in the web application. Practice safe web browsing and clear your cache regularly.
3. When communicating outside of the online application, please refrain from discussing specific content on the exam items. If such discussion is required, please use GPG encryption for any e-mails.
4. If you ever suspect that the confidentiality of the material might be broken, please report it to the LPI Director of Exam Development immediately.

Procedures

 Some adjustments to this procedure have been made to facilitate performing this review over the internet and via e-mail.

Step 1: Preparation: At this stage, you will receive your instructions and be given an opportunity to familiarize yourself with the process, online application and ask questions. This step began when you received this document. Time permitting, it might be helpful for you to review the items by cycling through them online and noting your answers. You can then compare your answers to the answer key as you make your ratings.

Step 2: The first step is to refine the description of the Minimally Qualified Candidate (MQC). If you have any comments on this description, which is included in later in this document, please bring them up immediately and/or via e-mail.

The current MQC definitions are listed below.

Step 3: Calibration Round: The purpose of this stage is to practice the rating procedure and to ensure that all judges are using relatively similar standards for issuing judgments. A small sample of the item pool will be used for this. You'll be asked to rate each item and comment on the reasoning you used in assigning each rating. You will receive feedback on whether your ratings are generally consistent with the other judges, or generally harder or easier.

Step 4: Initial Judgment Round: At this stage, you will provide judgments for each item in the pool. LPI staff will compile the ratings. Items for which there is no general agreement about difficulty among the judges will be included in a second judgment round.

This rating is the precentage of Minimally Qualified Candidates that would, in your estimation and experience, get this item correct. Remeber, you are not the MQC, but have to put yourself in their shoes. Please enter the number as a percentage in increments of 5 and as a whole number (not a decimal).

It is important to make your comments on why you chose a rating very clear. Other people will use those comments to decide if they should adjust their rating in the Final Judgment Round.

If you have an objection to the question, now is the time where it can be flagged. Please fill in the rating as if it was appropriate, in case the issue can be resolved.

Step 5: Final Judgment Round: At this stage, you can return to the rating application and a subset of the items will be presented for rating. These are items that had no general agreement on the difficulty along with important comments on why various ratings were set. After considering the comments, please provide a final rating for the item. There is no need for agreement on the ratings. If you decide that you still believe your rating is appropriate, leave your initial rating in the final rating field and continue on to the next item. Otherwise, put your new rating in the final rating column.

The Minimally Qualified Candidate

The method of rating that LPI has chosen involves imagining a person who is just barely qualified to pass the LPI exam in question. This person is referred to as the Minimally Qualified Candidate (MQC). If this person knew any less about system administration, then he or she would not be qualified, but they understand enough to be just barely qualified. It is important to define this MQC and make all ratings relative to this person. This person represents the cut-score in a meaningful way; anyone as qualified or more qualified than the MQC should pass while anyone less qualified than the MQC should not pass.

Note that by "minimally qualified", we mean someone who is at the bottom of the competence scale for job performance, not a typical or outstanding job holder.

Your first task will be to refine the MQC description that will be used in the study. Below is a list of the descriptions for various LPI certifications and exams. If you have any comments on them, please make them to the exam coordinator.

MQCs:

• Linux Essentials
• LPIC-1 (101 and 102)
• LPIC-2 (201 and 202)