|
This example illustrates
calibration and scoring of a test or scale containing 20 multiple
category items. Syntax is shown below.
EXAMPLE 1: ARTIFICIAL EXAMPLE: MONTE CARLO DATA
GRADED RATING SCALE MODEL, NORMAL RESPONSE FUNCTION: EAP SCALE SCORES
>COMMENT ;
>FILE DFNAME='PSLDAT\EXAMPL01.DAT',SAVE;
>SAVE PARM='PSLDAT\EXAMPL01.PAR',SCORE='EXAMPL01.SCO';
>INPUT NIDCH=10,NTOTAL=20,NTEST=1,LENGTH=(20),NFMT=1;
(46X,10A1,/,20A1)
>TEST1 TNAME=SCALE1,ITEM=(1(1)20),NBLOCK=1;
>BLOCK1 BNAME=SBLOCK1,NITEMS=20,NCAT=4, CADJUST=0.0;
>CAL GRADED,NQPTS=30,CYCLE=(25,2,2,2,2),
NEWTON=5,CRIT=0.005,ITEMFIT=10;
>SCORE EAP,NQPTS=30,SMEAN=0.0,SSD=1.0,NAME=EAP,PFQ=5;
The simulated data represent
responses of 1000 examinees drawn randomly from a population
with a mean trait score of 0.0 and standard deviation of 1.0.
As the default for SAMPLE on INPUT is 1000, all generated
data will be used as input by default.
The generating trait value
of each examinee is used as the case ID. The case ID is 10
characters long and is indicated as such using the NIDCH keyword
on the INPUT command. It is also reflected in the format statement
as 10A1.
Data are read from the
file exampl01.dat in the examples folder using
the DFNAME keyword on the FILES command.
All 20 items are used in
a single test (NTEST=1 on INPUT command, with LENGTH=20).
All 20 items have common categories and are assigned to the
same BLOCK (NBLOCK=1 on TEST; NITEMS=20 on BLOCK).
All items have four categories
(NCAT=4 on BLOCK command) and varying difficulties and discriminating
powers. The graded model
is assumed (GRADED on CALIB command); A logistic response
model (LOGISTIC on CALIB command) is requested. The choice
between a logistic or normal response function metric is effective
only if the graded response model is used. The response function
of the graded model
can be either the normal ogive or its logistic approximation.
Graded is the default. If logistic is selected, the item parameters
can be in the natural metric of the logistic ogive. Natural
is the default. For the normal metric, set SCALESCALE_keyword_on_CALIB_command
equal to 1.7. Neither LOGISTIC nor SCALE are needed when PARTIAL
is selected. Because the generalized model allows for varying
item discriminating powers, both a slope and threshold is
estimated for each item. The CADJUST keyword on the BLOCK
command is used to set the mean of the category parameters
to 0 as simultaneous estimation of slope parameters and all
category parameters is not obtainable.
The ITEMFIT keyword is
used to set the number of frequency score groups for the computation
of item fit statistics to 10. Note that there is no default
value for the ITEMFIT keyword.
The CYCLES keyword specifies
25 EM iterations, with maximum 2 inner EM iterations for the
item and category parameter estimation. Five Newton-Gauss
iterations are requested (NEWTON=5 on CALIB). A convergence
criterion of 0.005 is specified using the CRIT keyword on
CALIB.
30 quadrature points are
to be used in the EM and Newton estimation instead of the
default of 10 for cases where LENGTH less or equal to 50 in
the INPUT command. The calibration procedure depends on the
evaluation of integrals using Gauss-Hermite quadrature. In
general, the accuracy of numerical integration increases with
the number of quadrature points used.
The score estimation method
is specified (EAP option on SCORE command). Scale scores for
each subtest are estimated by the Bayes (EAP) method, and
their posterior standard deviations serve as standard errors.
The scores, which are rescaled
to zero mean and unit standard deviation in the sample (SMEAN
and SSD on SCORE), are saved in the file exampl01.sco
using the SCORE keyword on the SAVE command.
The PFQPFQ_keyword_on_SCORE_command
keyword is specified. This keyword is usually used to make
ML scores more computable but would also improve EAP estimates
somewhat.
In
addition, the estimated item parameters are saved in the file
exampl01.par (PARM keyword on the SAVE command).
The first three records
of the data file exampl01.dat are shown below.
Samp Group 1 1 00001 1.0 .44739
42444232223343433332
Samp Group 1 1 00002 1.0 -.93465
12221121122324121432
Samp Group 1 1 00003 1.0 -.56465
32212212213342314121
Two lines of data are given
for each respondent. On the first line, information concerning
generation of the data is given. The generating value for
each respondent is the last entry given on the first line:
it is used here as the case ID. In the format statement
(46X,10A1,/20A1)
46 columns on the first
line are skipped, after which the case ID is read as a character
string of length 10. The '/' indicates that following information
should be read on the second line of data. From this line,
the 20 item responses are read. Item responses are given in
the first 20 columns of this line, and are read as character
values of length 1 each. In the format statement this is indicated
by 20A1.
Although the data for each
respondent are spread out over 2 lines, the format statement
in the syntax file
occupies just one line, and thus NFMT is set to 1 on the INPUT
command.
At the beginning of the
output for
Phase 0, the syntax file is echoed. Information on the number
of tests, items, and type of model to be fitted as interpreted
by PARSCALE is also given.
EXAMPLE 1: ARTIFICIAL EXAMPLE: MONTE CARLO DATA
GRADED MODEL, NORMAL METRIC: EAP
SCALE SCORES
>COMMENT ;
>FILE DFNAME='PSLDAT\EXAMPL01.DAT',SAVE;
>SAVE PARM='PSLDAT\EXAMPL01.PAR',SCORE='EXAMPL01.SCO';
>INPUT NIDCH=10,NTOTAL=20,NTEST=1,LENGTH=(20),NFMT=1;
SINGLE MAIN
TEST IS USED.
NUMBER OF
ITEMS: 20
FORMAT OF
DATA INPUT IS
(46X,10A1,/,20A1)
>TEST1 TNAME=SCALE1,ITEM=(1(1)20),NBLOCK=1;
BLOCK CARD: 1
>BLOCK1 BNAME=SBLOCK1,NITEMS=20,NCAT=4,CADJ=0.0;
>CAL GRADED,SCALE=1.7,NQPTS=30,CYCLE=(25,2,2,2,2),
NEWTON=2,CRIT=0.00001,ITEMFIT=10;
MODEL SPECIFICATIONS
======================
NORMAL OGIVE
- GRADED ITEM RESPONSE MODEL IS SPECIFIED.
SCALE CONSTANT 1.70 FOR
SLOPE PARAMETERS.
This section of the output
file contains information on the settings to be used during
the item parameter estimation in Phase 2.
CALIBRATION
PARAMETERS
======================
MAXIMUM
NUMBER OF EM CYCLES: 25
MAXIMUM INNER EM CYCLES: 2
MAXIMUM CATEGORY ESTIMATION CYCLES: 2
MAXIMUM ITEM PARAMETER ESTIMATION
CYCLES: 2
MAXIMUM NUMBER OF NEWTON CYCLES: 2
CONVERGENCE CRITERION FOR EM CYCLES: 0.0000
CONVERGENCE CRITERION FOR SLOPE: 0.0000
CONVERGENCE CRITERION FOR THRESHOLD: 0.0000
CONVERGENCE CRITERION FOR CATEGORY: 0.0000
CONVERGENCE CRITERION FOR GEUSSING: 0.0000
ORDER OF INNER EM CYCLES: CATEGORY - ITEM PARAMETERS
ESTIMATION ACCELERATOR: NO (DEFAULT)
RIDGE METHOD: NO (DEFAULT)
No prior distribution was
requested in the CALIB command, and consequently the default
prior, a normal distribution on equally spaced points, will
be used (DIST=2 on CALIB). The number of quadrature points
to be used during item parameter estimation was set to 30
(NQPT on CALIB). The program-generated quadrature points and
weights are printed to the Phase 0 output
file, as shown below.
THE FIXED
PRIOR DISTRIBUTION FOR LATENT TRAITS
MEAN : 0.0000
S.D. : 1.0000
QUADRATURE
POINTS AND PRIOR WEIGHTS (PROGRAM-GENERATED NORMAL APPROXIMATION):
1 2 3 4 5
POINT -0.4000E+01 -0.3724E+01 -0.3448E+01 -0.3172E+01 -0.2897E+01
WEIGHT 0.3692E-04 0.1071E-03 0.2881E-03 0.7181E-03 0.1659E-02
6 7 8 9 10
POINT -0.2621E+01 -0.2345E+01 -0.2069E+01 -0.1793E+01 -0.1517E+01
WEIGHT 0.3550E-02 0.7042E-02 0.1294E-01 0.2205E-01 0.3481E-01
11 12 13 14 15
POINT -0.1241E+01 -0.9655E+00 -0.6897E+00 -0.4138E+00 -0.1379E+00
WEIGHT 0.5093E-01 0.6905E-01 0.8676E-01 0.1010E+00 0.1090E+00
16 17 18 19 20
POINT 0.1379E+00 0.4138E+00 0.6897E+00 0.9655E+00 0.1241E+01
WEIGHT 0.1090E+00 0.1010E+00 0.8676E-01 0.6905E-01 0.5093E-01
21 22 23 24 25
POINT 0.1517E+01 0.1793E+01 0.2069E+01 0.2345E+01 0.2621E+01
WEIGHT 0.3481E-01 0.2205E-01 0.1294E-01 0.7042E-02 0.3550E-02
26 27 28 29 30
POINT 0.2897E+01 0.3172E+01 0.3448E+01 0.3724E+01 0.4000E+01
WEIGHT 0.1659E-02 0.7181E-03 0.2881E-03 0.1071E-03 0.3692E-04
TOTAL WEIGHT:
1.00000
MEAN : 0.00000
S.D. :
0.99970
The control settings to
be used during calibration is followed by settings to be used
during the scoring phase (Phase 3). The EAP method of scoring
is requested (EAP option) and, as in the calibration phase,
30 quadrature points were requested. Since no prior distribution
was requested using the DIST keyword, by default a normal
distribution on equally spaced points will be used (DIST =
2 on SCORE). Note that the DIST keyword applies only when
EAP scoring has been selected.
>SCORE EAP,NQPTS=30,SMEAN=0.0,SSD=1.0,NAME=EAP,PFQ=5;
PARAMETERS
FOR SCORING AND TEST AND ITEM INFORMATION
====================================================
METHOD OF
SCORING SUBJECTS: EXPECTATION A POSTERIORI
(EAP;
BAYES ESTIMATES)
TYPE OF
PRIOR: NORMAL APPROXIMATION
NUMBER OF
QUADRATURE POINTS 30
SCORES WRITTEN TO FILE EXAMPL01.SCO
QUADRATURE
POINTS AND PRIOR WEIGHTS (PROGRAM-GENERATED NORMAL APPROXIMATION):
1 2 3 4 5
POINT -0.4000E+01 -0.3724E+01 -0.3448E+01 -0.3172E+01 -0.2897E+01
WEIGHT 0.3692E-04 0.1071E-03 0.2881E-03 0.7181E-03 0.1659E-02
6 7 8 9 10
POINT -0.2621E+01 -0.2345E+01 -0.2069E+01 -0.1793E+01 -0.1517E+01
WEIGHT 0.3550E-02 0.7042E-02 0.1294E-01 0.2205E-01 0.3481E-01
11 12 13 14 15
POINT -0.1241E+01 -0.9655E+00 -0.6897E+00 -0.4138E+00 -0.1379E+00
WEIGHT 0.5093E-01 0.6905E-01 0.8676E-01 0.1010E+00 0.1090E+00
16 17 18 19 20
POINT 0.1379E+00 0.4138E+00 0.6897E+00 0.9655E+00 0.1241E+01
WEIGHT 0.1090E+00 0.1010E+00 0.8676E-01 0.6905E-01 0.5093E-01
21 22 23 24 25
POINT 0.1517E+01 0.1793E+01 0.2069E+01 0.2345E+01 0.2621E+01
WEIGHT 0.3481E-01 0.2205E-01 0.1294E-01 0.7042E-02 0.3550E-02
26 27 28 29 30
POINT 0.2897E+01 0.3172E+01 0.3448E+01 0.3724E+01 0.4000E+01
WEIGHT 0.1659E-02 0.7181E-03 0.2881E-03 0.1071E-03 0.3692E-04
TOTAL WEIGHT:
1.00000
MEAN : 0.00000
S.D. : 0.99970
The values assigned to
the rescaling constants SMEAN and SSD in the SCORE command
are shown:
SET NUMBER : 1
SCORE NAME : EAP
NUMBER OF ITEMS : 20
RESCALE CONSTANT: MEAN = 0.00 S.D. = 1.00
ITEMS : 1 2 3 4 5 6 7 8 9 10
11 12 13 14 15 16 17 18 19 20
0001 0002 0003 0004 0005 0006 0007 0008 0009 0010
0011 0012 0013 0014 0015 0016 0017 0018 0019 0020
Input and output
files as requested with the DFNAME keyword on the FILES command
and the PARM and SCORE keywords on the SAVE command are listed:
FILE ASSIGNMENTS
AND DISPOSITIONS
=================================
[INPUT FILES]
SUBJECT
DATA INPUT FILE PSLDAT\EXAMPL01.DAT
SINGLE-SUBJECT DATA
NO
CASE WEIGHTS
[OUTPUT
FILES]
ITEM PARAMETERS
FILE PSLDAT\EXAMPL01.PAR
SUBJECT SCALE-SCORE FILE EXAMPL01.SCO
[SCRATCH
FILES]
PARSCALE SYSTEM BINARY DATA FILE Exampl01.MFL
TEMPORARY FILE Exampl01.T99
TEMPORARY FILE Exampl01.T98
TEMPORARY FILE Exampl01.T97
TEMPORARY FILE Exampl01.T96
To allow the user to verify
that data have been read in correctly from the raw data file,
the first two records from the data file are echoed in the
output.
The INPUT RESPONSES fields give the original responses while
the RECODED RESPONSES reflect any recoding of the responses.
Recoding of responses is controlled by the ORIGINAL and MODIFIED
keywords on the BLOCK command.
INPUT AND
RECODED RESPONSE OF FIRST AND SECOND OBSERVATIONS
OBSERVATION
# 1
GROUP: 1
ID: .4473
INPUT RESPONSES: 4 2 4 4 4 2 3 2 2 2 3 3 4 3 4 3 3 3 3 2
RECODED RESPONSES:4 2 4 4 4 2 3 2 2 2 3 3 4 3 4 3 3 3 3 2
OBSERVATION
# 2
GROUP: 1
ID: -.9346
INPUT RESPONSES: 1 2 2 2 1 1 2 1 1 2 2 3 2 4 1 2 1 4 3 2
RECODED RESPONSES:1 2 2 2 1 1 2 1 1 2 2 3 2 4 1 2 1 4 3 2
Finally, the number of
observations to be used in the analysis is recorded; by default,
all observations will be used. The number of observations
to be used can be manipulated using the SAMPLE or TAKE keywords
on the INPUT command.
[MAIN TEST:
SCALE1 ]
1000 OBSERVATIONS READ FROM FILE: PSLDAT\EXAMPL01.DAT
1000
OBSERVATIONS WRITTEN TO FILE: Exampl01.MFL
The title given in the
TITLE command and name assigned to the test in the TEST command
in the syntax file are echoed in the output
file.
EXAMPLE 1: ARTIFICIAL EXAMPLE: MONTE CARLO DATA
GRADED MODEL, NORMAL METRIC: EAP
SCALE SCORES
MAINTEST:
SCALE1
The master file created
during Phase 0 is used as input. Note that the master file
exampl01.mfl may be saved using the MASTER keyword
on the SAVE command for use as input in a subsequent analysis
(MFNAME keyword on FILES command). The keywords TAKE and SAMPLE
on the INPUT command controls the number of records read from
the raw data file. As the default value of SAMPLE is 1000,
neither keyword was used and all data were used by default.
1000 OBS.(WEIGHTS: 1000.000) WERE READ FROM Exampl01.MFL
Summary item statistics
for the 20 items are given next. Since no not-represented
(NFNAME on FILES) or omit key (OFNAME on FILES) was used,
no frequencies or percentages are reported under the 밡OT
PRESENT?or 밢MIT?headings. Under the 밅ATEGORIES?heading,
frequencies and percentages of responses for each of the 4
categories are given item-by-item. Cumulative frequencies
and percentages for the categories over all items are given
at the end of the table.
Note that, if empty categories
are encountered, the user has to recode the corresponding
items of which this occurs before proceeding with the analysis.
SUMMARY
ITEM STATISTICS
=======================
BLOCK NO.: 1 NAME:
SBLOCK1
---------------------------------------------------------------
ITEM | TOTAL NOT OMIT| CATEGORIES
| PRESENT |
| | 1 2 3 4
---------------------------------------------------------------
0001 | |
FREQ.| 1000 0 0| 194 303 313 190
PERC.| 0.0 0.0| 19.4 30.3 31.3 19.0
| |
0002 | |
FREQ.| 1000 0 0| 204 284 310 202
PERC.| 0.0 0.0| 20.4 28.4 31.0 20.2
| |
0003 | |
FREQ.| 1000 0 0| 206 308 285 201
PERC.| 0.0 0.0| 20.6 30.8 28.5 20.1
?br>
0020 | |
FREQ.| 1000 0 0| 305 211 212 272
PERC.| 0.0 0.0| 30.5 21.1 21.2 27.2
| |
---------------------------------------------------------------
CUMMUL.| |
FREQ.| | 4844 5186 5204 4766
PERC.| | 24.2 25.9 26.0 23.8
---------------------------------------------------------------
Item means, initial slope
estimates, and Pearson and polyserial item-test correlations
are shown in the next table.
Pearson
The sample product-moment
correlation of the test score,
and m-category polytomous
item score, 
is the point polyserial correlation
, where
where n is the sample
size, 
is the mean test score and
, the mean item score.
In this example n
= 1000. For item 1,
so that
Also
so that
Polyserial correlation
The polyserial correlation
can be expressed in terms of the
point polyserial correlation as
where
Initial slopes and location
The polyserial correlation
estimates the item factor loading, 
, say. If the arbitrary scale of the item latent variable,
, is chosen so that the variance
equals 1, then
where 
is the factor score with mean
0 and variance 1, and the error, 
, has mean 0 and variance 
.
For purposes of MML parameter
estimation in IRT, it is convenient to rescale the item latent
variable so that the error variance equals 1. The factor loading
then becomes the item slope,
This provisional estimate
of the slope is then used as the starting value in the iterative
EM solutions of the marginal maximum likelihood equations
for estimating the parameters of the polytomous item response
models. The initial locations shown in the last column of
the table are the averages of the category thresholds for
each item.
Initial item-category threshold
parameters
Item-category threshold
parameters can be calculated once the polyserial coefficients
have been obtained. The expression for the threshold parameter
in terms of the cumulative category proportions and the biserial
correlation coefficient (Lord & Novick, 1968) as
with 
the biserial correlation for item
j and 
the z score that cuts of
proportion of the cases to item
j in a unit-normal distribution; that is
where 
is the frequency of the categorical
response for item j and category k. These provisional
thresholds of the categories serve as starting values in MML
estimation of the corresponding item parameters. For the rating
scale model, whether all items have the same thresholds, the
category proportions are computed from frequencies accumulated
over all items; i.e. ,
In Muraki뭩 (1990) formulation
of the rating scale model, the category threshold parameter,
, is expressed as a deviation from the item threshold parameter,
; that is
under
the constraint that
In the context of the rating
scale model,
is referred to as a "location" parameter. The INITIAL LOCATION column provides the values
of the average of the category thresholds for each item.
---------------------------------------------------------------------------
BLOCK | RESPONSE TOTAL SCORE | PEARSON & | INITIAL INITIAL
ITEM | MEAN MEAN | POLYSERIAL | SLOPE LOCATION
| S.D.* S.D.* | CORRELATION
|
---------------------------------------------------------------------------
SBLOCK1 | | |
1 0001 | 2.499 49.892 | 0.778 | 1.488 -0.009
| 1.009* 14.754* | 0.830 |
2 0002 | 2.510 49.892 | 0.797 | 1.628 -0.028
| 1.030* 14.754* | 0.852 |
3 0003 | 2.481 49.892 | 0.785 | 1.545 0.020
| 1.031* 14.754* | 0.839 |
4 0004 | 2.515 49.892 | 0.805 | 1.695 -0.045
| 1.037* 14.754* | 0.861 |
5 0005 | 2.511 49.892 | 0.811 | 1.739 -0.031
| 1.032* 14.754* | 0.867 |
6 0006 | 2.137 49.892 | 0.728 | 1.293 0.844
| 1.037* 14.754* | 0.791 |
7 0007 | 2.118 49.892 | 0.735 | 1.336 0.863
| 1.033* 14.754* | 0.801 |
8 0008 | 2.144 49.892 | 0.754 | 1.426 0.765
| 1.029* 14.754* | 0.819 |
9 0009 | 2.136 49.892 | 0.736 | 1.329 0.838
| 
is the scoring corresponding
to the cumulative proportion, 
of the k-th response
category to item j (for item 1, for example, the
cumulative proportions are 0.194, 0.497, and 0.81 for categories
1,2, and 3), 
is the standard deviation of
item scores for item j (1.009 for item 1), and
is the ordinate in the normal
distribution in the point 