CutScoreInstructions - Revision history

GMatthewRice: /* Detail */

2012-05-30T14:19:51Z

Detail

← Older revision		Revision as of 14:19, 30 May 2012
Line 223:		Line 223:
	a comment when you make a very extreme rating or when you are unsure of		a comment when you make a very extreme rating or when you are unsure of
	your rating. You do not need to comment every item but it helps if you do.		your rating. You do not need to comment every item but it helps if you do.

	~~Please type your ratings into the ratings column if the spreadsheet.~~


	==Examples==		==Examples==

GMatthewRice: /* The Minimally Qualified Candidate */

2012-05-29T16:19:26Z

The Minimally Qualified Candidate

← Older revision		Revision as of 16:19, 29 May 2012
Line 134:		Line 134:
	* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_2 LPIC-2 (201 and 202)]		* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_2 LPIC-2 (201 and 202)]

			==Objectives==

			LPI publishes objectives for each level of exams. These objectives
			describe the exam content for that level. Reviewing the objectives
			for the level would also be helpful. You can find the objectives at:

			Objectives:
			* [http://wiki.lpi.org/wiki/LinuxEssentials Linux Essentials]
			* [http://wiki.lpi.org/wiki/LPIC-1_Objectives LPIC-1 (101 and 102)]
			* [http://wiki.lpi.org/wiki/LPIC-2_Objectives LPIC-2 (201 and 202)]

	=Making Ratings=		=Making Ratings=

GMatthewRice: /* Making Ratings */

2012-05-29T16:17:54Z

Making Ratings

← Older revision		Revision as of 16:17, 29 May 2012
Line 136:		Line 136:

	=Making Ratings=		=Making Ratings=


			==Overview==

			You will be assigning ratings for each question. You will read the question
			and examine the answers and make your best guess about the chances of the
			MQC getting this item correct. You will formulate this rating as a
			probability from 0 to 100 and you will record it.

			LPI will average all the ratings to estimate the probability that the
			MQC would pass each item. The sum of these probabilities is the expected
			number of correct answers for the MQC. This expectation will be used to
			set the cut-score.


			==Detail==

			Since your reference should be the MQC, you MUST familiarize yourself
			with the FINAL MQC description. The links to them should be above.

			For each exam question, you are asked to provide a specialized judgment
			about how likely the MQC would be able to answer the question correctly.

			Record your rating as a percentage ranging from 0 to 100. Keep in mind
			that with multiple choice questions, a certain portion of examinees will
			guess the correct answer even if they don't know it. The following table
			shows the probability of answering a multiple choice question correctly
			without no knowledge of the content:

			{\|
			! # OF CHOICES !! %
			\|-
			\| 3 \|\| 33
			\|-
			\| 4 \|\| 25
			\|-
			\| 5 \|\| 20
			\|}

			This table shows, for example, that a multiple choice item with 5 choices
			will, on average, be answered correctly by .20 (20%) of the examinees
			who have no knowledge at all of the question content.

			Because of this "guessing effect", in general, the lowest rating you
			should consider for multiple choice questions is the value in the table
			above. For example, ratings for a 5-response multiple-choice item should
			usually range from 20 to 100 because even the hardest question will be
			answered correctly by 20% of the people through guessing alone. (The only
			way we might expect less than 20% to answer the question correctly is
			if it is a "trick question". We tried to avoid these kind of questions
			on this exam.)

			Also note that this guessing effect is not applicable to fill-in-the-blank
			type questions. In general, the test-taker either knows the answer or
			does not. Some examinees will get these correct by guessing, but this
			is not the blind kind of guessing that is possible with multiple-choice
			questions. So, the ratings on fill-in-the-blank questions may range from
			0 to 100.

			If one or more of the incorrect responses are implausible, then the
			guessing rate may be much higher than indicated in the above table.
			For example, on a five-option item where three incorrect responses can
			be discarded, the guessing rate is 50%.

			Note that the rating you provide should not be a rating of the quality of
			the item. It should be a judgment about the percent of barely qualified
			job holders that will answer the item correctly. Since we are currently
			testing these items, we cannot change them and your rating should be
			about the question as it appears (even if there is a mistake).

			For each item, you should also provide a comment. The comment should be
			related to the judgment (comments about the quality or wording of the
			item are welcome for future use, but we cannot change the items at this
			time). Any comments you provide will be made available to LPI staff and
			anonymously to other judges in this study. You should definitely make
			a comment when you make a very extreme rating or when you are unsure of
			your rating. You do not need to comment every item but it helps if you do.

			Please type your ratings into the ratings column if the spreadsheet.


			==Examples==

			For these examples, some of the questions may seem silly, and that is
			intention. They are intended to illustrate the guidelines covered above.

			'''Q1:''' What is 1 + 1?
			a. 2
			b. 1
			c. 3
			d. 11

			Suggested rating: 100

			It is reasonable that 100% of job holders could correctly answer this
			question.

			'''Q2:''' What was the date on which this question was written?
			a. Mar 10 2000
			b. Mar 14 2000
			c. Mar 21 2000
			d. Mar 22 2000
			e. Mar 23 2000

			Suggested Rating: 20

			No one (except for the writer!) would know the correct answer to this
			question, obviously. However, since it is a multiple choice question with 5
			responses, anyone attempting it will have a 20% chance of answering it
			correctly.

			'''Q3:''' (Fill in the blank) What unix command is most often used for
			setting the permissions on a file? ________

			Suggested Rating: 97

			Almost all minimally qualified candidates should know that the answer is
			chmod, but there could be a very small number who do not know this, who
			cannot recall, misread the question, make a simple mistake, mistype the
			response, etc.

GMatthewRice: /* The Minimally Qualified Candidate */

2012-05-29T16:04:22Z

The Minimally Qualified Candidate

← Older revision		Revision as of 16:04, 29 May 2012
Line 133:		Line 133:
	* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_1 LPIC-1 (101 and 102)]		* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_1 LPIC-1 (101 and 102)]
	* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_2 LPIC-2 (201 and 202)]		* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_2 LPIC-2 (201 and 202)]


			=Making Ratings=

GMatthewRice: /* Procedures */

2012-05-29T16:02:21Z

Procedures

← Older revision		Revision as of 16:02, 29 May 2012
Line 53:		Line 53:

	=Procedures=		=Procedures=

			Some adjustments to this procedure have been made to facilitate performing this review over the internet and via e-mail.

			'''Step 1:''' Preparation: At this stage, you will receive your instructions
			and be given an opportunity to familiarize yourself with the process,
			online application and ask questions. This step began when you received
			this document. Time permitting, it might be helpful for you to review
			the items by cycling through them online and noting your answers. You can
			then compare your answers to the answer key as you make your ratings.

			'''Step 2:''' The first step is to refine the description of the Minimally
			Qualified Candidate (MQC). If you have any comments on this description,
			which is included in later in this document, please bring them up
			immediately and/or via e-mail.

			The current MQC definitions are listed below.

			'''Step 3:''' Calibration Round: The purpose of this stage is to practice
			the rating procedure and to ensure that all judges are using relatively
			similar standards for issuing judgments. A small sample of the item pool
			will be used for this. You'll be asked to rate each item and comment on
			the reasoning you used in assigning each rating. You will receive
			feedback on whether your ratings are generally consistent with the other
			judges, or generally harder or easier.

			'''Step 4:''' Initial Judgment Round: At this stage, you will provide judgments
			for each item in the pool. LPI staff will compile the ratings. Items for
			which there is no general agreement about difficulty among the judges will
			be included in a second judgment round.

			This rating is the precentage of Minimally Qualified Candidates that would,
			in your estimation and experience, get this item correct. Remeber, you are
			not the MQC, but have to put yourself in their shoes. Please enter the
			number as a percentage in increments of 5 and as a whole number (not a
			decimal).

			It is important to make your comments on why you chose a rating very clear.
			Other people will use those comments to decide if they should adjust their
			rating in the Final Judgment Round.

			If you have an objection to the question, now is the time where it can be
			flagged. Please fill in the rating as if it was appropriate, in case the
			issue can be resolved.

			'''Step 5:''' Final Judgment Round: At this stage, you can return to the rating
			application and a subset of the items will be presented for rating.
			These are items that had no general agreement on the difficulty along with
			important comments on why various ratings were set. After considering
			the comments, please provide a final rating for the item. There is no
			need for agreement on the ratings. If you decide that you still believe
			your rating is appropriate, leave your initial rating in the final rating
			field and continue on to the next item. Otherwise, put your new rating
			in the final rating column.


			=The Minimally Qualified Candidate=

			The method of rating that LPI has chosen involves imagining a person
			who is just barely qualified to pass the LPI exam in question.
			This person is referred to as the Minimally Qualified Candidate (MQC).
			If this person knew any less about system administration, then he or
			she would not be qualified, but they understand enough to be just
			barely qualified. It is important to define this MQC and make all
			ratings relative to this person. This person represents the cut-score
			in a meaningful way; anyone as qualified or more qualified than the MQC
			should pass while anyone less qualified than the MQC should not pass.

			Note that by "minimally qualified", we mean someone who is at the bottom
			of the competence scale for job performance, not a typical or outstanding
			job holder.

			Your first task will be to refine the MQC description that will be
			used in the study. Below is a list of the descriptions for various LPI
			certifications and exams. If you have any comments on them, please make
			them to the exam coordinator.

			MQCs:
			* [[LinuxEssentials#Minimally_Qualified_Candidate_Description\|Linux Essentials]]
			* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_1 LPIC-1 (101 and 102)]
			* [http://www2.lpi.org/eng/certification/the_lpic_program/lpic_2 LPIC-2 (201 and 202)]

GMatthewRice: /* Protecting Confidentiality */

2012-05-29T15:33:33Z

Protecting Confidentiality

← Older revision		Revision as of 15:33, 29 May 2012
Line 50:		Line 50:
	# When communicating outside of the online application, please refrain from discussing specific content on the exam items. If such discussion is required, please use GPG encryption for any e-mails.		# When communicating outside of the online application, please refrain from discussing specific content on the exam items. If such discussion is required, please use GPG encryption for any e-mails.
	# If you ever suspect that the confidentiality of the material might be broken, please report it to the LPI Director of Exam Development immediately.		# If you ever suspect that the confidentiality of the material might be broken, please report it to the LPI Director of Exam Development immediately.


			=Procedures=

GMatthewRice: Created page with "Here are the instructions for helping with a Cut-Score Study. =Introduction= The purpose of this study is to use specialized ratings from expert Linux system administrators ..."

2012-05-29T15:31:53Z

Created page with "Here are the instructions for helping with a Cut-Score Study. =Introduction= The purpose of this study is to use specialized ratings from expert Linux system administrators ..."

New page

Here are the instructions for helping with a Cut-Score Study.

=Introduction=

The purpose of this study is to use specialized ratings from expert
Linux system administrators and users to determine what the passing
score should be for our exams.

The work we do in this study is critical to ensuring that the cut-scores
for the exams are fair. For the study to be successful, it is critical
that the Subject Matter Expert (SME) understand the process and what
they are asked to do. The purpose of this document is to accomplish this
requirement. If you have any questions at all, please let us know.

==Conventions==

Throughout, this document will refer to LPI Director of Exam Development.
G. Matthew Rice is LPI's Director of Exam Development and he can be
contacted at mrice at lpi dot org.

The Linux Professional Institute will be referred to as LPI and the
various exams will be referred to by their number code.

==Materials==

In order to participate in a study, an SME will have the following
materials.

# These instructions.
# Access to an on-line application for rating the items.<br/>The URL will be e-mailed to the SME upon commencement of a study.
# Access to the (potential) exam questions.<br/>These are the actual questions on the current or future forms of the exams. Due to the nature of this access, an NDA will be required to participate in this study with LPI.

==Protecting Confidentiality==

In order to participate in this study, an SME must have access to a large
subset or the entire set of questions for an exam. The disclosure of the
exam questions is covered by the non-disclosure agreement (NDA) that is
signed with LPI before access can be provided. It is absolutely essential
that an SME protect the confidentiality of this material. LPI exams cost
tens of thousands of US dollars/Euros to develop and, if confidentiality
is broken, the entire effort needs to be repeated at great expense.

In particular, please strictly adhere to these guidelines:

# Do not, in any manner, share the test questions with anyone. Do not disclose item content to co-workers, supervisors, family members, students, the media, other LPI judges or reviewers, or anyone else. Avoid all forms of unauthorized disclosure including in writing, by voice, and by any electronic means. If you need to discuss the contents of any of the questions, you should only do so with the LPI Director of Exam Development. In the past, LPI has provided a mailing list for the discussion of items during cut-score studies. In these unsecured communications, please avoid disclosing any actual item content and, if content is disclosed, please limit it to the minimum necessary.
# Do not make copies of the questions as they are presented in the web application. Practice safe web browsing and clear your cache regularly.
# When communicating outside of the online application, please refrain from discussing specific content on the exam items. If such discussion is required, please use GPG encryption for any e-mails.
# If you ever suspect that the confidentiality of the material might be broken, please report it to the LPI Director of Exam Development immediately.