CutScoreInstructions
Here are the instructions for helping with a Cut-Score Study.
Introduction
The purpose of this study is to use specialized ratings from expert Linux system administrators and users to determine what the passing score should be for our exams.
The work we do in this study is critical to ensuring that the cut-scores for the exams are fair. For the study to be successful, it is critical that the Subject Matter Expert (SME) understand the process and what they are asked to do. The purpose of this document is to accomplish this requirement. If you have any questions at all, please let us know.
Conventions
Throughout, this document will refer to LPI Director of Exam Development. G. Matthew Rice is LPI's Director of Exam Development and he can be contacted at mrice at lpi dot org.
The Linux Professional Institute will be referred to as LPI and the various exams will be referred to by their number code.
Materials
In order to participate in a study, an SME will have the following materials.
- These instructions.
- Access to an on-line application for rating the items.
The URL will be e-mailed to the SME upon commencement of a study. - Access to the (potential) exam questions.
These are the actual questions on the current or future forms of the exams. Due to the nature of this access, an NDA will be required to participate in this study with LPI.
Protecting Confidentiality
In order to participate in this study, an SME must have access to a large subset or the entire set of questions for an exam. The disclosure of the exam questions is covered by the non-disclosure agreement (NDA) that is signed with LPI before access can be provided. It is absolutely essential that an SME protect the confidentiality of this material. LPI exams cost tens of thousands of US dollars/Euros to develop and, if confidentiality is broken, the entire effort needs to be repeated at great expense.
In particular, please strictly adhere to these guidelines:
- Do not, in any manner, share the test questions with anyone. Do not disclose item content to co-workers, supervisors, family members, students, the media, other LPI judges or reviewers, or anyone else. Avoid all forms of unauthorized disclosure including in writing, by voice, and by any electronic means. If you need to discuss the contents of any of the questions, you should only do so with the LPI Director of Exam Development. In the past, LPI has provided a mailing list for the discussion of items during cut-score studies. In these unsecured communications, please avoid disclosing any actual item content and, if content is disclosed, please limit it to the minimum necessary.
- Do not make copies of the questions as they are presented in the web application. Practice safe web browsing and clear your cache regularly.
- When communicating outside of the online application, please refrain from discussing specific content on the exam items. If such discussion is required, please use GPG encryption for any e-mails.
- If you ever suspect that the confidentiality of the material might be broken, please report it to the LPI Director of Exam Development immediately.
Procedures
Some adjustments to this procedure have been made to facilitate performing this review over the internet and via e-mail.
Step 1: Preparation: At this stage, you will receive your instructions and be given an opportunity to familiarize yourself with the process, online application and ask questions. This step began when you received this document. Time permitting, it might be helpful for you to review the items by cycling through them online and noting your answers. You can then compare your answers to the answer key as you make your ratings.
Step 2: The first step is to refine the description of the Minimally Qualified Candidate (MQC). If you have any comments on this description, which is included in later in this document, please bring them up immediately and/or via e-mail.
The current MQC definitions are listed below.
Step 3: Calibration Round: The purpose of this stage is to practice the rating procedure and to ensure that all judges are using relatively similar standards for issuing judgments. A small sample of the item pool will be used for this. You'll be asked to rate each item and comment on the reasoning you used in assigning each rating. You will receive feedback on whether your ratings are generally consistent with the other judges, or generally harder or easier.
Step 4: Initial Judgment Round: At this stage, you will provide judgments for each item in the pool. LPI staff will compile the ratings. Items for which there is no general agreement about difficulty among the judges will be included in a second judgment round.
This rating is the precentage of Minimally Qualified Candidates that would, in your estimation and experience, get this item correct. Remeber, you are not the MQC, but have to put yourself in their shoes. Please enter the number as a percentage in increments of 5 and as a whole number (not a decimal).
It is important to make your comments on why you chose a rating very clear. Other people will use those comments to decide if they should adjust their rating in the Final Judgment Round.
If you have an objection to the question, now is the time where it can be flagged. Please fill in the rating as if it was appropriate, in case the issue can be resolved.
Step 5: Final Judgment Round: At this stage, you can return to the rating application and a subset of the items will be presented for rating. These are items that had no general agreement on the difficulty along with important comments on why various ratings were set. After considering the comments, please provide a final rating for the item. There is no need for agreement on the ratings. If you decide that you still believe your rating is appropriate, leave your initial rating in the final rating field and continue on to the next item. Otherwise, put your new rating in the final rating column.
The Minimally Qualified Candidate
The method of rating that LPI has chosen involves imagining a person who is just barely qualified to pass the LPI exam in question. This person is referred to as the Minimally Qualified Candidate (MQC). If this person knew any less about system administration, then he or she would not be qualified, but they understand enough to be just barely qualified. It is important to define this MQC and make all ratings relative to this person. This person represents the cut-score in a meaningful way; anyone as qualified or more qualified than the MQC should pass while anyone less qualified than the MQC should not pass.
Note that by "minimally qualified", we mean someone who is at the bottom of the competence scale for job performance, not a typical or outstanding job holder.
Your first task will be to refine the MQC description that will be used in the study. Below is a list of the descriptions for various LPI certifications and exams. If you have any comments on them, please make them to the exam coordinator.
MQCs:
Objectives
LPI publishes objectives for each level of exams. These objectives describe the exam content for that level. Reviewing the objectives for the level would also be helpful. You can find the objectives at:
Objectives:
Making Ratings
Overview
You will be assigning ratings for each question. You will read the question and examine the answers and make your best guess about the chances of the MQC getting this item correct. You will formulate this rating as a probability from 0 to 100 and you will record it.
LPI will average all the ratings to estimate the probability that the MQC would pass each item. The sum of these probabilities is the expected number of correct answers for the MQC. This expectation will be used to set the cut-score.
Detail
Since your reference should be the MQC, you MUST familiarize yourself with the FINAL MQC description. The links to them should be above.
For each exam question, you are asked to provide a specialized judgment about how likely the MQC would be able to answer the question correctly.
Record your rating as a percentage ranging from 0 to 100. Keep in mind that with multiple choice questions, a certain portion of examinees will guess the correct answer even if they don't know it. The following table shows the probability of answering a multiple choice question correctly without no knowledge of the content:
| # OF CHOICES | % |
|---|---|
| 3 | 33 |
| 4 | 25 |
| 5 | 20 |
This table shows, for example, that a multiple choice item with 5 choices will, on average, be answered correctly by .20 (20%) of the examinees who have no knowledge at all of the question content.
Because of this "guessing effect", in general, the lowest rating you should consider for multiple choice questions is the value in the table above. For example, ratings for a 5-response multiple-choice item should usually range from 20 to 100 because even the hardest question will be answered correctly by 20% of the people through guessing alone. (The only way we might expect less than 20% to answer the question correctly is if it is a "trick question". We tried to avoid these kind of questions on this exam.)
Also note that this guessing effect is not applicable to fill-in-the-blank type questions. In general, the test-taker either knows the answer or does not. Some examinees will get these correct by guessing, but this is not the blind kind of guessing that is possible with multiple-choice questions. So, the ratings on fill-in-the-blank questions may range from 0 to 100.
If one or more of the incorrect responses are implausible, then the guessing rate may be much higher than indicated in the above table. For example, on a five-option item where three incorrect responses can be discarded, the guessing rate is 50%.
Note that the rating you provide should not be a rating of the quality of the item. It should be a judgment about the percent of barely qualified job holders that will answer the item correctly. Since we are currently testing these items, we cannot change them and your rating should be about the question as it appears (even if there is a mistake).
For each item, you should also provide a comment. The comment should be related to the judgment (comments about the quality or wording of the item are welcome for future use, but we cannot change the items at this time). Any comments you provide will be made available to LPI staff and anonymously to other judges in this study. You should definitely make a comment when you make a very extreme rating or when you are unsure of your rating. You do not need to comment every item but it helps if you do.
Examples
For these examples, some of the questions may seem silly, and that is intention. They are intended to illustrate the guidelines covered above.
Q1: What is 1 + 1? a. 2 b. 1 c. 3 d. 11
Suggested rating: 100
It is reasonable that 100% of job holders could correctly answer this question.
Q2: What was the date on which this question was written? a. Mar 10 2000 b. Mar 14 2000 c. Mar 21 2000 d. Mar 22 2000 e. Mar 23 2000
Suggested Rating: 20
No one (except for the writer!) would know the correct answer to this question, obviously. However, since it is a multiple choice question with 5 responses, anyone attempting it will have a 20% chance of answering it correctly.
Q3: (Fill in the blank) What unix command is most often used for setting the permissions on a file? ________
Suggested Rating: 97
Almost all minimally qualified candidates should know that the answer is chmod, but there could be a very small number who do not know this, who cannot recall, misread the question, make a simple mistake, mistype the response, etc.
