Getting ready

If the packages are not ready installed on your computer use the function install.packages(). Then we include the libraries:

library(eRm)
library(iarm)
library(PP)
library(rms)
library(r2r)
library(splines)

Targeting

Targeting indicates the degree to which the study population is outside the target range of the scale items. Using the function PCM() from the R-package erm we can fix the mean of item difficulties at zero using the option sum0 = TRUE. For a well-targeted scale (not too easy, not too difficult) the mean person locations should then also be around zero. A positive mean value for person locations indicates that the sample is located at a higher level than the items, while a negative value suggests the opposite. If many respondents are at the margins, then the scale is not properly targeted. Targeting can be verified graphically and numerically. We will demonstrate both approaches.

Typically, the analysis of the quality of the targeting of the scale for a population, first compares the mean person ability estimate with the mean item difficulty. With good targeting both should be close to zero. We consider the colitis data set containing a sample (N=678) of cases and controls (variable status) who were interviewed by nurse or postal questionnaire (variable interv). Information on gender (variable sex) and a global self rated health item (variable health) is also included.

  • 11 items about problems: anxiety, anger, hopeless, lonely, restless, dependent, tense, touchy, shame, depress, crying scored 0,1 (1: problem is present)
  • 3 positively worded items: happy, confident, goodhumour scored 0,1 (1: not happy)

We download the data set from the home page and save it in an R-object called data.

data <- read.table(url("http://publicifsv.sund.ku.dk/~kach/scaleval/colitis.txt"),
    header = TRUE)

Let us define a new object items featuring only the actual items (columns 6-19) and complete cases.

items <- data[, 6:19]
items[items == "."] <- NA
items <- items[complete.cases(items), ]
items <- as.data.frame(sapply(items, as.numeric))

First, we compute the parameter estimates of a PCM using the PCM() function with sum0 = TRUE.

fit <- PCM(items, sum0 = TRUE)

We extract the difficulty parameter of the items (=Location in eRm) and calculate the mean and sd.

i.dif <- fit$betapar
mean.dif <- mean(i.dif)
sd.dif <- sd(i.dif)

We find the ability of the persons using the person.parameter() function and calculate the mean and sd.

pp <- person.parameter(fit)
abil <- pp$theta.table[, "Person Parameter"]
mean.abil <- mean(abil)
sd.abil <- sd(abil)

We show the rounded mean and sd difficulties and abilities.

summary.targeting <- data.frame(Mean_Difficulty = mean.dif, SD_Difficulty = sd.dif,
    Mean_Abilities = mean.abil, SD_Abilities = sd.abil)

round(summary.targeting, 3)
  Mean_Difficulty SD_Difficulty Mean_Abilities SD_Abilities
1               0         1.816           0.38        1.692

Reliability

Reliability, in the context of Modern Test Theory, is an estimate of the precision of the person ability estimates. The Person Separation Reliability, calculates the proportion of person variance that is not due to error. The concept of person separation reliability is very similar to reliability indices such as Cronbach’s \(\alpha\). The interpretation of the person separation reliability may vary, but in general it is agreed on that the reliablity is very good if the PSR is close to 1, good if > 0.85 and sufficient if > 0.7.

The PSR reported by most Rasch computer programs is calculated by subtracting the ratio of the sample mean square person measure error (\(MSE_p\)) to the sample person measure variance (\(SD_p^2\)) from one. The formula (Wright and Masters, 1982, p.106) is:

\[PSR=1-\left[\frac{MSE_p}{SD^2_p}\right]\]

The R-package eRm informs about the persons separation with the function SepRel() which can be applied on the person.parameter() output.

SepRel(pp)
Separation Reliability: 0.7299

The scale at this stage, has very high reliability.

The function test_prop() in R-package iarm makes available reliability and also targeting indices:

  1. The separation reliability : identical to the PSI computed with SepRel() in the R-package eRm
  2. Test difficulty:
  3. Test target index: the minimum standard error of measurement divided by the mean standard error of measurement, which is expected close to one for good targeting
  4. Test information: the ratio of the mean and max test information

We check the data distribution.

round(test_prop(fit), 3)
Separation Reliability        Test difficulty            Test target 
             0.7298994              0.1490000              0.5380000 
      Test information 
             2.3132049 
Separation Reliability        Test difficulty            Test target 
                 0.730                  0.149                  0.538 
      Test information 
                 2.313

The graphical testing of targeting includes the person item map as implemented in eRm with the function plotPImap(), i.e., plots of the distribution of item difficulty parameters and person ability parameters.

plotPImap(fit, sorted = TRUE)

In the top panel the grey bars form a bar chart showing the distribution of the person locations. Each item has its own line and the item location is shown as a dot. Note that this is different for polytomous items.