Package 'maat'

Title:	Multiple Administrations Adaptive Testing
Description:	Provides an extension of the shadow-test approach to computerized adaptive testing (CAT) implemented in the 'TestDesign' package for the assessment framework involving multiple tests administered periodically throughout the year. This framework is referred to as the Multiple Administrations Adaptive Testing (MAAT) and supports multiple item pools vertically scaled and multiple phases (stages) of CAT within each test. Between phases and tests, transitioning from one item pool (and associated constraints) to another is allowed as deemed necessary to enhance the quality of measurement.
Authors:	Seung W. Choi [aut, cre] , Sangdon Lim [aut] , Luping Niu [aut] , Sooyong Lee [aut] , M. Christina Schneider [ctb], Jay Lee [ctb], Garron Gianopulos [ctb]
Maintainer:	Seung W. Choi <[email protected]>
License:	GPL (>= 2)
Version:	1.1.0.9000
Built:	2024-09-05 05:09:21 UTC
Source:	https://github.com/choi-phd/maat

Help Index

Multiple Administrations Adaptive Testing
Class 'assessment_structure': assessment structure
Bound grades within a specified range
Grade operator: add or subtract
Phase operator: move to next phase
Test operator: move to next phase
Create an assessment structure
Create a single module
Class 'examinee': a single examinee
Update a constraints object to exclude administered items
Format the output of maat
Calculate adaptivity indices from an examinee list object
Get administered items per test
Calculate bias from an examinee list object
Get item exposure rates from an examinee list
Get item names per grade
Grade operator: difference between two grades
Calculate RMSE from an examinee list object
Calculate standard error from an examinee list object
Load multiple modules
Simulate multi-stage multi-administration adaptive test
Example item pools
Class 'module': a module
Class 'output_maat': a simulation output
Extension of plot()
Extension of print()
Remove item data from examinee list
Extension of show()
Simulate an examinee list
Simulate theta values
Update the assessment-level theta of an examinee object
Update the grade slot of an examinee object
Update the item data slot of an examinee object
Update the routing log of an examinee object
Update the current module of an examinee object
Update the current phase of an examinee object
Update the current test of an examinee object
Update the theta used for routing of an examinee object
Update theta estimates using combined responses from a test

Multiple Administrations Adaptive Testing

Description

Multiple Administrations Adaptive Testing

Details

maat package is based on the assessment framework involving multiple tests administered throughout the year using multiple item pools vertically scaled and multiple phases (stages) of computerized adaptive testing (CAT) within each test allowing for transitioning from one item pool (and associated constraints) to another between phases as determined necessary by a selected transition policy to enhance the quality of measurement.

The current version of maat supports three administrations (Fall, Winter, and Spring) with two phases within each administration (Phase 1 and Phase 2), for six modules in total administered over the course of a year.

Within each administration, students begin Phase 1 at the grade of record. One exception to this is that if a student's final $\theta$ from the previous administration was above the 'advanced achievement' cut score of the grade of record, then the student begins Phase 1 of the following administration in an above-grade item pool. For example, if a Grade 3 student's final $\theta$ from the Fall administration was $\theta = 1.1$ and the 'advanced achievement' cut score for Grade 3 was $\theta = 1.0$ , then the student begins Phase 1 of the Winter administration in a Grade 4 item pool.

Within each administration, at the completion of Phase 1, business rules are used to determine whether a student is routed to an on-grade or off-grade item pool in Phase 2.

Detailed descriptions of the assessment design are available in the vignette.

Class 'assessment_structure': assessment structure

Description

assessment_structure is an S4 class to represent an assessment structure.

Slots

n_test: a numeric, the number of test administrations.
n_phase: a numeric, the number of phases within each test.
route_limit_below: the number of grades to allow routing below, relative to the grade of record. If the grade of record is G4 and this is 1, then routing to G3 is allowed but not to G2.
route_limit_above: the number of grades to allow routing above, relative to the grade of record. If the grade of record is G4 and this is 2, then routing to G6 is allowed but not to G7.
test_routing_restrictions: R1: If grade is G-1 in the last phase of any administration, ignore achievement level and always change grade by +1. R2: If grade is G in the last phase of any administration: If achievement level is Beginning, do not decrease grade. R3: If grade is G+k in the last phase of Administration k: If achievement level is Advanced, do not increase grade.

Bound grades within a specified range

Description

boundGrade is a function for keeping the grade within a specified range. boundGrade checks the relative grade compared to the grade of record. If the current grade is outside the allowed bound, the grade that is within the bound in the same direction is returned.

Usage

boundGrade(
  current_grade,
  grade_of_record,
  route_limit_below,
  route_limit_above
)
boundGrade(
  current_grade,
  grade_of_record,
  route_limit_below,
  route_limit_above
)

Arguments

`current_grade`	the current grade. This must be formatted as `G?`, where `?` is a number.
`grade_of_record`	the grade of record. This must be formatted as `G?`, where `?` is a number.
`route_limit_below`	the number of grades to allow routing below, relative to the grade of record. If the grade of record is G4 and this is 1, then routing to G3 is allowed but not to G2.
`route_limit_above`	the number of grades to allow routing above, relative to the grade of record. If the grade of record is G4 and this is 2, then routing to G6 is allowed but not to G7.

Value

the grade after the range limit is applied

Examples

boundGrade("G2", "G1", 0, 2) # G2
boundGrade("G3", "G1", 0, 2) # G3
boundGrade("G4", "G1", 0, 2) # G3
boundGrade("G5", "G1", 0, 2) # G3

boundGrade("G2", "G1", 0, 2) # G2
boundGrade("G3", "G1", 0, 2) # G3
boundGrade("G4", "G1", 0, 2) # G3
boundGrade("G5", "G1", 0, 2) # G3

Grade operator: add or subtract

Description

changeGrade is an operator for grade values.

Usage

changeGrade(grade, delta)
changeGrade(grade, delta)

Arguments

`grade`	a string containing the current grade in the form `G?`, where `?` is a number.
`delta`	a number containing the relative change in grade to apply. `0` retains the current grade as-is.

Value

a string containing the new grade.

Examples

changeGrade("G4",  0) ## G4
changeGrade("G4",  1) ## G5
changeGrade("G4", -1) ## G3
changeGrade("G10", 1) ## G11

changeGrade("G4",  0) ## G4
changeGrade("G4",  1) ## G5
changeGrade("G4", -1) ## G3
changeGrade("G10", 1) ## G11

Phase operator: move to next phase

Description

changePhase is an operator for phase values.

Usage

changePhase(phase, assessment_structure)
changePhase(phase, assessment_structure)

Arguments

`phase`	a string containing the current phase in the format `P?`, where `?` is a number.
`assessment_structure`	an `assessment_structure` object.

Value

a string containing the new phase.

Examples

## assessment uses two phases
changePhase("P1", assessment_structure_math) ## P2
changePhase("P2", assessment_structure_math) ## P1

## assessment uses two phases
changePhase("P1", assessment_structure_math) ## P2
changePhase("P2", assessment_structure_math) ## P1

Test operator: move to next phase

Description

changeTest is an operator for test values.

Usage

changeTest(test, phase, assessment_structure)
changeTest(test, phase, assessment_structure)

Arguments

`test`	a string containing the current test in the format `T?`, where `?` is a number.
`phase`	a string containing the current phase in the format `P?`, where `?` is a number.
`assessment_structure`	an `assessment_structure` object.

Value

a string containing the new test.

Examples

## assessment uses two phases
changeTest("T1", "P1", assessment_structure_math) ## T1
changeTest("T1", "P2", assessment_structure_math) ## T2

## assessment uses two phases
changeTest("T1", "P1", assessment_structure_math) ## T1
changeTest("T1", "P2", assessment_structure_math) ## T2

Create an assessment structure

Description

createAssessmentStructure is a function for creating an assessment_structure object that defines the structure of the assessment.

Usage

createAssessmentStructure(
  n_test,
  n_phase,
  route_limit_below,
  route_limit_above,
  test_routing_restrictions = c("R1", "R2", "R3")
)
createAssessmentStructure(
  n_test,
  n_phase,
  route_limit_below,
  route_limit_above,
  test_routing_restrictions = c("R1", "R2", "R3")
)

Arguments

`n_test`	a numeric, the number of test administrations.
`n_phase`	a numeric, the number of phases within each test.
`route_limit_below`	the number of grades to allow routing below, relative to the grade of record. If the grade of record is G4 and this is 1, then routing to G3 is allowed but not to G2.
`route_limit_above`	the number of grades to allow routing above, relative to the grade of record. If the grade of record is G4 and this is 2, then routing to G6 is allowed but not to G7.
`test_routing_restrictions`	the restrictions for between-test routing. (default = `c("R1", "R2", "R3")`)

Value

an assessment_structure object.

Examples

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 1,
  route_limit_above = 2
)

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 1,
  route_limit_above = 2
)

Create a single module

Description

createModule is a function for creating a module object based on the item pool, attribute, and constraints.

Usage

createModule(constraints, item_pool, item_attrib, passage_attrib)
createModule(constraints, item_pool, item_attrib, passage_attrib)

Arguments

`constraints`	constraints data. A `data.frame` or a csv file name to be used in `loadConstraints`.
`item_pool`	item pool data. A `data.frame` or a csv file name to be used in `loadItemPool`.
`item_attrib`	item attribute data. A `data.frame` or a csv file name to be used in `loadItemAttrib`.
`passage_attrib`	passage attribute data. A `data.frame` or a csv file name to be used in `loadStAttrib`.

Value

a module object.

Class 'examinee': a single examinee

Description

examinee is an S4 class to represent a single examinee.

Slots

examinee_id: the ID of examinee.
current_grade: the current grade the examinee is in. Updated using updateGrade.
current_phase: the current phase the examinee is in. Updated using updatePhase.
current_test: the current test the examinee is in. Updated using updateTest.
current_module: the current module the examinee is in. Updated using updateModule.
grade_log: grades that the examinee belonged at each module position. Updated using updateLog.
phase_log: phases that the examinee belonged at each module position. Updated using updateLog.
test_log: tests that the examinee belonged at each module position. Updated using updateLog.
module_log: modules that the examinee belonged at each module position. Updated using updateLog.
n_module: the number of modules the examinee received. This is the number of module positions.
true_theta: a vector containing the true theta (if simulated) of the examinee, for each module position.
initial_theta_in_module: a vector containing initial thetas used in each module.
prior_par_by_module: a list containing prior parameters used for each module.
estimated_theta_by_phase: a list containing estimated thetas and SEs using items in each phase.
estimated_theta_by_test: a list containing estimated thetas and SEs using combined items in each test. Updated using updateThetaUsingCombined.
estimated_theta_for_routing: a list containing estimated thetas and SEs that were used for routing. Updated using updateThetaForRouting.
estimated_theta: a list containing estimated theta and SE using all responses from all modules. Updated using updateAssessmentLevelTheta.
alpha: the alpha value used to compute lower and upper bounds.
selection_theta: a list containing selection thetas in each module position.
interim_theta: a list containing interim thetas and SEs in each module position.
administered_items: a list containing administered items in each module position.
administered_stimuli: a list containing administered stimuli in each module position.
response: a list containing the examinee response in each module position.
item_data: a list containing item_pool of administered items.
routing_based_on: a vector containing the routing was based on estimated_theta_by_phase or estimated_theta_by_test at each module position.

Update a constraints object to exclude administered items

Description

The function excludeAdministeredItems produces a new constraints object that excludes administered items from being selected.

Usage

excludeAdministeredItems(constraints, administered_items)
excludeAdministeredItems(constraints, administered_items)

Arguments

`constraints`	a `constraints` object.
`administered_items`	item names of previously administered items.

Value

a constraints object that also constrains the administered items to be excluded.

Examples

## Not run: 
require(TestDesign)

cfg <- createShadowTestConfig(
  MIP = list(solver = "lpsymphony")
)
constraints <- constraints_reading
solution <- Shadow(cfg, constraints, true_theta = 0)
administered_items <- solution@output[[1]]@administered_item_index
administered_items <- solution@constraints@pool@id[administered_items]
administered_items

updated_constraints <- excludeAdministeredItems(constraints, administered_items)

solution <- Shadow(cfg, updated_constraints, true_theta = 0)
administered_items <- solution@output[[1]]@administered_item_index
administered_items <- solution@constraints@pool@id[administered_items]
administered_items ## entirely different from above

## End(Not run)
## Not run: 
require(TestDesign)

cfg <- createShadowTestConfig(
  MIP = list(solver = "lpsymphony")
)
constraints <- constraints_reading
solution <- Shadow(cfg, constraints, true_theta = 0)
administered_items <- solution@output[[1]]@administered_item_index
administered_items <- solution@constraints@pool@id[administered_items]
administered_items

updated_constraints <- excludeAdministeredItems(constraints, administered_items)

solution <- Shadow(cfg, updated_constraints, true_theta = 0)
administered_items <- solution@output[[1]]@administered_item_index
administered_items <- solution@constraints@pool@id[administered_items]
administered_items ## entirely different from above

## End(Not run)

Format the output of maat

Description

formatOutput is a function for formatting the output examinee object of the function maat for analysis.

Usage

formatOutput(examinee_list, digits = 3)
formatOutput(examinee_list, digits = 3)

Arguments

`examinee_list`	the output from `maat`.
`digits`	digits to round theta values. (default = 3)

Value

a data frame containing:

p_ID: the person ID.
test_phase_ID: the module position. If we have 3 tests with 2 phases in each test then the range of test_phase_ID is 1 to 6.
initial_grade: the initial grade of the person.
final_grade: the final grade of the person after completing all modules.
grade_ID: the grade at the module position.
phase_ID: the phase at the module position.
test_ID: the test at the module position.
module_ID: the module ID at the module position.
final_theta_est: the grand final estimated $\theta$ after completing all tests.
final_SE_est: the standard error of grand final estimated $\theta$ after completing all tests.
theta_by_phase: the final estimated $\theta$ after completing each phase.
SE_by_phase: the standard error of final estimated $\theta$ after completing each phase.
combined: whether items were combined with the previous phase to obtain the theta estimate.
true_theta: the true $\theta$ in each module position.
item_ID: the item IDs of administered items.
ncat: the number of categories of administered items.
IRT_model: the IRT models of administered items.
item_par_1: the first item parameter of each administered item (e.g., for 1PL, this is item difficulty)
item_par_2: the second item parameter of each administered item (e.g., for 1PL, this is 'NA')
item_resp: the item response on each administered item.
momentary_theta: the momentary (interim) $\theta$ estimate obtained after each item administration in CAT engine.
momentary_SE: the standard error of momentary (interim) $\theta$ estimate obtained after each item administration in CAT engine.

Calculate adaptivity indices from an examinee list object

Description

getAdaptivityIndex is a function for calculating adaptivity indices from the output of maat.

Usage

getAdaptivityIndex(x)
getAdaptivityIndex(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

a data frame containing adaptivity indices by test and also for all tests combined.

Get administered items per test

Description

getAdministeredItemsPerTest is a function for extracting the administered items stored in the examinee objects.

Usage

getAdministeredItemsPerTest(x)
getAdministeredItemsPerTest(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

a list containing administered items in each test and also for all tests combined.

Calculate bias from an examinee list object

Description

getBias is a function for calculating the bias of ability estimates of the simulation results.

Usage

getBias(x)
getBias(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

a list containing bias by test and also for all tests combined.

Get item exposure rates from an examinee list

Description

getItemExposureRate is a function for building an item exposure rate table.

Usage

getItemExposureRate(x)
getItemExposureRate(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

the table of item exposure rate.

Get item names per grade

Description

getItemNamesPerGrade is a function for extracting item names from a module list.

Usage

getItemNamesPerGrade(module_list)
getItemNamesPerGrade(module_list)

Arguments

module_list

a module list from loadModules.

Value

item names per grade.

Examples

getItemNamesPerGrade(module_list_math)

getItemNamesPerGrade(module_list_math)

Grade operator: difference between two grades

Description

getRelativeGrade is an operator for grade values.

Usage

getRelativeGrade(current_grade, initial_grade)
getRelativeGrade(current_grade, initial_grade)

Arguments

`current_grade`	a string containing the current grade in the form `G?`, where `?` is a number.
`initial_grade`	a string containing the initial grade in the form `G?`, where `?` is a number.

Value

the grade difference of the current grade relative to the initial grade.

Examples

getRelativeGrade("G4", "G3") ## 1
getRelativeGrade("G5", "G3") ## 2
getRelativeGrade("G2", "G3") ## -1

getRelativeGrade("G4", "G3") ## 1
getRelativeGrade("G5", "G3") ## 2
getRelativeGrade("G2", "G3") ## -1

Calculate RMSE from an examinee list object

Description

getRMSE is a function for calculating root mean square error (RMSE) for the simulation results.

Usage

getRMSE(x)
getRMSE(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

a list containing RMSE by test and also for all tests combined.

Calculate standard error from an examinee list object

Description

getSE is a function for calculating the standard error of the estimates.

Usage

getSE(x)
getSE(x)

Arguments

`x`	an `output_maat` object from `maat`.

Value

a list containing SE by test and also for all tests combined.

Load multiple modules

Description

loadModules is a function for creating multiple module objects from a specification sheet.

Usage

loadModules(fn, base_path = NULL, assessment_structure, examinee_list)
loadModules(fn, base_path = NULL, assessment_structure, examinee_list)

Arguments

`fn`	the full file path and name of a csv file containing module specifications.
`base_path`	(optional) the base path to be prepended to the file paths contained in the module specifications sheet.
`assessment_structure`	an `assessment_structure` object.
`examinee_list`	an examinee list from `simExaminees`. Used to determine the range of required modules.

Details

The module specification file is expected to have the following columns:

Grade a string containing the grade in the form G?, where ? is a number.
Phase a string containing the phase in the form P?, where ? is a number.
ItemPool the file path of a file that contains item pool data. This must be readable with loadItemPool.
ItemAttrib the file path of a file that contains item attribute data. This must be readable with loadItemAttrib.
PassageAttrib the file path of a file that contains passage attribute data. This must be readable with loadStAttrib.
Constraints the file path of a file that contains constraints data. This must be readable with loadConstraints.

Value

a module list containing module objects. Each module can be accessed using module_list[[grade]][[test]][[phase]].

Examples

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 0,
  route_limit_above = 2
)
examinee_list <- simExaminees(
  N             = 5,
  mean_v        = c(0, 0, 0),
  sd_v          = c(1, 1, 1),
  cor_v         = diag(1, 3),
  assessment_structure = assessment_structure
)

fn <- system.file("extdata", "module_definition_MATH_normal_N500_flexible.csv", package = "maat")
pkg_path <- system.file(package = "maat")
module_list <- loadModules(
  fn,
  base_path = pkg_path,
  assessment_structure = assessment_structure,
  examinee_list = examinee_list
)

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 0,
  route_limit_above = 2
)
examinee_list <- simExaminees(
  N             = 5,
  mean_v        = c(0, 0, 0),
  sd_v          = c(1, 1, 1),
  cor_v         = diag(1, 3),
  assessment_structure = assessment_structure
)

fn <- system.file("extdata", "module_definition_MATH_normal_N500_flexible.csv", package = "maat")
pkg_path <- system.file(package = "maat")
module_list <- loadModules(
  fn,
  base_path = pkg_path,
  assessment_structure = assessment_structure,
  examinee_list = examinee_list
)

Simulate multi-stage multi-administration adaptive test

Description

maat is the main function for simulating a multi-stage multi-administration adaptive test.

Usage

maat(
  examinee_list = examinee_list,
  assessment_structure = NULL,
  module_list = NULL,
  config = NULL,
  cut_scores = NULL,
  overlap_control_policy = NULL,
  transition_policy = "CI",
  combine_policy = "conditional",
  transition_CI_alpha = NULL,
  transition_percentile_lower = NULL,
  transition_percentile_upper = NULL,
  initial_theta_list = NULL,
  prior_mean_policy = "mean_difficulty",
  prior_mean_user = NULL,
  prior_sd = 1,
  verbose = TRUE
)
maat(
  examinee_list = examinee_list,
  assessment_structure = NULL,
  module_list = NULL,
  config = NULL,
  cut_scores = NULL,
  overlap_control_policy = NULL,
  transition_policy = "CI",
  combine_policy = "conditional",
  transition_CI_alpha = NULL,
  transition_percentile_lower = NULL,
  transition_percentile_upper = NULL,
  initial_theta_list = NULL,
  prior_mean_policy = "mean_difficulty",
  prior_mean_user = NULL,
  prior_sd = 1,
  verbose = TRUE
)

Arguments

`examinee_list`	an examinee list from `simExaminees`.
`assessment_structure`	a `assessment_structure` object.
`module_list`	a module list from `loadModules`.
`config`	a `config_Shadow` object. Also accepts a list of `config_Shadow` objects to use separate configurations for each module. Must be from 'TestDesign' 1.3.3 or newer, and its `exclude_policy$method` slot must be `SOFT`.
`cut_scores`	a named list containing cut scores to be used in each grade. Each element must be named in the form `G?`, where `?` is a number.
`overlap_control_policy`	overlap control is performed by excluding administered items from being administered again within the same examinee. `all` performs overlap control at all module positions. `within_test` performs overlap control only within each test. `none` does not perform overlap control.
`transition_policy`	`CI` uses the confidence interval to perform routing. `pool_difficulty_percentile` uses item difficulty percentiles of all items in the `item_pool` argument to perform routing. `pool_difficulty_percentile_exclude_administered` uses item difficulty percentiles of all items in the `item_pool` argument to perform routing, excluding all previous items administered to the examinee. `on_grade` does not permit any transition. (default = `CI`)
`combine_policy`	This is only applied when `module_position %% 2 == 0` (at Phase 2, which is the end of each test). `conditional` uses the combined theta (using items from the previous module combined with the current module), if the examinee was in the same grade in Phases 1 and 2. If the examinee was in different grades in Phases 1 and 2, then the theta estimate from Phase 2 is used. `always` uses the combined theta. `never` uses the theta estimate from Phase 2. (default = `conditional`)
`transition_CI_alpha`	the alpha level to use when `transition_policy == "CI"`.
`transition_percentile_lower`	the percentile value (between 0 and 1) to use for the lower routing when `transition_policy == "difficulty_percentile"`.
`transition_percentile_upper`	the percentile value (between 0 and 1) to use for the upper routing when `transition_policy == "difficulty_percentile"`.
`initial_theta_list`	(optional) a list containing initial thetas to use in each module position.
`prior_mean_policy`	This is only effective at the beginning of each test. This determines what value is used as the prior mean. `mean_difficulty` uses the mean item difficulty of the current item pool. `carryover` uses the routing theta from the previous module. For Phase 1 of the first test, user supplied values are used if available. Otherwise, the mean item difficulty of the current item pool is used. `user` uses user-supplied values in the `prior_mean_user` argument. (default = `mean_difficulty`)
`prior_mean_user`	(optional) user-supplied values for the prior mean. Must be a single value, or a vector for each grade.
`prior_sd`	user-supplied values for the prior standard deviation. This is only effective at the beginning of each test. This is utilized regardless of `prior_mean_policy`. Must be a single value, or a vector for each grade. (default = `1`)
`verbose`	if `TRUE`, print status messages. (default = `TRUE`)

Value

an output_maat object from the simulation.

Examples


library(TestDesign) # >= 1.3.3
config <- createShadowTestConfig(
  final_theta = list(method = "MLE"),
  exclude_policy = list(method = "SOFT", M = 100)
)
# exclude_policy must be SOFT

examinee_list <- maat(
  examinee_list          = examinee_list_math,
  assessment_structure   = assessment_structure_math,
  module_list            = module_list_math,
  overlap_control_policy = "all",
  transition_CI_alpha    = 0.05,
  config                 = config,
  cut_scores             = cut_scores_math
)

library(TestDesign) # >= 1.3.3
config <- createShadowTestConfig(
  final_theta = list(method = "MLE"),
  exclude_policy = list(method = "SOFT", M = 100)
)
# exclude_policy must be SOFT

examinee_list <- maat(
  examinee_list          = examinee_list_math,
  assessment_structure   = assessment_structure_math,
  module_list            = module_list_math,
  overlap_control_policy = "all",
  transition_CI_alpha    = 0.05,
  config                 = config,
  cut_scores             = cut_scores_math
)

Example item pools

Description

Example data for a 6-module assessment.

Details

assessment_structure_math an assessment_structure object defining 3 tests with 2 phases in each test. Also defines routing limits as G - 1 and G + 2, where G is the starting grade.
examinee_list_math a list of examinee objects. The number of examinees is 10. This can be created using simExaminees.
module_list_math a list of module objects. This can be created using loadModules.
cut_scores_math a list of theta cut scores. This is used in the cut_scores argument of the maat function.

Class 'module': a module

Description

module is an S4 class to represent a module.

Slots

module_id: the ID of the module.
constraints: a constraints object.

Class 'output_maat': a simulation output

Description

output_maat is an S4 class to represent a simulation output.

Slots

examinee_list: a list of examinee objects.
assessment_structure: an assessment_structure object.
module_list: a module list from loadModules.
config: the list of config_Shadow objects used in the simulation for each module.
cut_scores: the cut scores used in the simulation.
overlap_control_policy: the policy used in the simulation.
transition_policy: the policy used in the simulation.
combine_policy: the policy used in the simulation.
transition_CI_alpha: the transition parameter used in the simulation.
transition_percentile_lower: the transition parameter used in the simulation.
transition_percentile_upper: the transition parameter used in the simulation.
initial_theta_list: the starting theta values used in the simulation.
prior_mean_policy: the policy used in the simulation.
prior_mean_user: the prior parameters used in the simulation.
prior_sd: the prior parameters used in the simulation.

Extension of plot()

Description

Extension of plot()

Usage

## S4 method for signature 'output_maat'
plot(
  x,
  y,
  type,
  examinee_id = 1,
  cut_scores = NULL,
  theta_range = c(-4, 4),
  main = NULL,
  box_color = "PaleTurquoise"
)
## S4 method for signature 'output_maat'
plot(
  x,
  y,
  type,
  examinee_id = 1,
  cut_scores = NULL,
  theta_range = c(-4, 4),
  main = NULL,
  box_color = "PaleTurquoise"
)

Arguments

`x`	x
`y`	y
`type`	the type of plot. `route` plots the number of examinees routed to each path across the course of entire assessment. `correlation` produces a scatterplot of thetas across administrations. `audit` plots interim thetas over modules for a single examinee.
`examinee_id`	the examinee ID to plot.
`cut_scores`	(optional) a named list containing cut scores for each grade.
`theta_range`	the theta range to use in scatter plots when `x` is an examinee list.
`main`	the figure title to use in scatter plots when `x` is an examinee list.
`box_color`	the cell color to use when `type` is `route`. (default = `PaleTurquoise`)

Value

the route plot.

Examples


library(TestDesign)
config <- createShadowTestConfig(
  final_theta = list(method = "MLE"),
  exclude_policy = list(method = "SOFT", M = 100)
)
examinee_list <- maat(
  examinee_list          = examinee_list_math,
  assessment_structure   = assessment_structure_math,
  module_list            = module_list_math,
  overlap_control_policy = "all",
  transition_CI_alpha    = 0.05,
  config                 = config,
  cut_scores             = cut_scores_math
)

plot(examinee_list, type = "route")
plot(examinee_list, type = "correlation")
plot(examinee_list, type = "audit", examinee_id = 1)


library(TestDesign)
config <- createShadowTestConfig(
  final_theta = list(method = "MLE"),
  exclude_policy = list(method = "SOFT", M = 100)
)
examinee_list <- maat(
  examinee_list          = examinee_list_math,
  assessment_structure   = assessment_structure_math,
  module_list            = module_list_math,
  overlap_control_policy = "all",
  transition_CI_alpha    = 0.05,
  config                 = config,
  cut_scores             = cut_scores_math
)

plot(examinee_list, type = "route")
plot(examinee_list, type = "correlation")
plot(examinee_list, type = "audit", examinee_id = 1)

Extension of print()

Description

Extension of print()

Usage

## S4 method for signature 'module'
print(x)
## S4 method for signature 'module'
print(x)

Arguments

`x`	an object to display the content.

Remove item data from examinee list

Description

removeItemData is a function to remove the item data from the examinee objects for the reduction of file size.

Usage

removeItemData(examinee_list)
removeItemData(examinee_list)

Arguments

examinee_list

a list containing examinee objects.

Value

a list containing examinee objects, with item_data data stripped for compact storage.

Extension of show()

Description

Extension of show()

Usage

## S4 method for signature 'module'
show(object)
## S4 method for signature 'module'
show(object)

Arguments

object

an object to display the content.

Simulate an examinee list

Description

simExaminees is a function for generating a list of examinee objects.

Usage

simExaminees(
  N,
  mean_v,
  sd_v,
  cor_v,
  assessment_structure,
  initial_grade = "G4",
  initial_test = "T1",
  initial_phase = "P1"
)
simExaminees(
  N,
  mean_v,
  sd_v,
  cor_v,
  assessment_structure,
  initial_grade = "G4",
  initial_test = "T1",
  initial_phase = "P1"
)

Arguments

`N`	the number of examinees.
`mean_v`	a vector containing the mean of each dimension.
`sd_v`	a vector containing the standard deviation of each dimension.
`cor_v`	a correlation matrix.
`assessment_structure`	an `assessment_structure` object. This can be created using `createAssessmentStructure`.
`initial_grade`	the initial grade for all examinees. The grade must exist in `module_list`. Also used as the grade of record when the initial phase and test points to a module position greater than 1. (default = `G4`)
`initial_test`	the initial test for all examinees. (default = `T1`)
`initial_phase`	the initial phase for all examinees. The phase must exist in `module_list`. (default = `P1`)

Details

Each dimension of mean_v, sd_v, cor_v represents a test level. For example in a three-test structure (see the assessment_structure_math example data), these arguments must have three dimensions.

Value

a list of examinee objects.

Examples

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 1,
  route_limit_above = 2
)
examinee_list <- simExaminees(
  N      = 100,
  mean_v = c(0, 0, 0),
  sd_v   = c(1, 1, 1),
  cor_v  = diag(1, 3),
  assessment_structure = assessment_structure
)

assessment_structure <- createAssessmentStructure(
  n_test  = 3,
  n_phase = 2,
  route_limit_below = 1,
  route_limit_above = 2
)
examinee_list <- simExaminees(
  N      = 100,
  mean_v = c(0, 0, 0),
  sd_v   = c(1, 1, 1),
  cor_v  = diag(1, 3),
  assessment_structure = assessment_structure
)

Simulate theta values

Description

simTheta is a function for generating a theta matrix based on the given sample size, mean, standard deviation, and correlation matrix.

Usage

simTheta(N, mean_v, sd_v, cor_v)
simTheta(N, mean_v, sd_v, cor_v)

Arguments

`N`	the number of examinees.
`mean_v`	a vector containing the mean of each dimension.
`sd_v`	a vector containing the standard deviation of each dimension.
`cor_v`	a correlation matrix.

Details

simTheta calls mvrnorm internally.

Value

a theta matrix.

Examples

o <- simTheta(
  N      = 100,
  mean_v = c(0, 0, 0),
  sd_v   = c(1, 1, 1),
  cor_v  = diag(1, 3)
)

o <- simTheta(
  N      = 100,
  mean_v = c(0, 0, 0),
  sd_v   = c(1, 1, 1),
  cor_v  = diag(1, 3)
)

Update the assessment-level theta of an examinee object

Description

updateAssessmentLevelTheta is a function for updating examinee objects after completing all modules. updateAssessmentLevelTheta computes the assessment-level theta. Estimation options are based on the final_theta slot of the config object.

Usage

updateAssessmentLevelTheta(examinee_object, config)
updateAssessmentLevelTheta(examinee_object, config)

Arguments

`examinee_object`	an `examinee` object.
`config`	a `config_Shadow` object. The `final_theta` slot is used.

Value

an examinee object with its estimated_theta slot updated.

Update the grade slot of an examinee object

Description

updateGrade is a function for determining the grade an examinee is routed to.

Usage

updateGrade(
  examinee_object,
  assessment_structure,
  module_position,
  cut_scores,
  transition_policy = "CI",
  transition_CI_alpha,
  transition_percentile_lower,
  transition_percentile_upper,
  item_pool
)
updateGrade(
  examinee_object,
  assessment_structure,
  module_position,
  cut_scores,
  transition_policy = "CI",
  transition_CI_alpha,
  transition_percentile_lower,
  transition_percentile_upper,
  item_pool
)

Arguments

`examinee_object`	an `examinee` object.
`assessment_structure`	an `assessment_structure` object.
`module_position`	the current module position, ranging from `1` to `6`.
`cut_scores`	a named list containing cut scores to be used in each grade. Each element must be named in the form `G?`, where `?` is a number.
`transition_policy`	`CI` uses the confidence interval to perform routing. `pool_difficulty_percentile` uses item difficulty percentiles of all items in the `item_pool` argument to perform routing. `pool_difficulty_percentile_exclude_administered` uses item difficulty percentiles of all items in the `item_pool` argument to perform routing, excluding all previous items administered to the examinee. `on_grade` does not permit any transition. (default = `CI`)
`transition_CI_alpha`	the alpha level used when `transition_policy == "CI"`.
`transition_percentile_lower`	the percentile value (between 0 and 1) used for the lower routing in percentile-based transition policies.
`transition_percentile_upper`	the percentile value (between 0 and 1) used for the upper routing in percentile-based transition policies.
`item_pool`	the `item_pool` object to determine difficulty range in percentile-based transition policies.

Details

Currently the routing rules are hard-coded in the function. See the vignette for a description of routing rules.

Value

an examinee object with its current_grade slot updated.

Update the item data slot of an examinee object

Description

updateItemData is a function for updating examinee objects after completing a module.

Usage

updateItemData(examinee_object, module_position, solution)
updateItemData(examinee_object, module_position, solution)

Arguments

`examinee_object`	an `examinee` object.
`module_position`	the current module position.
`solution`	an `output_Shadow_all` object.

Details

updateItemData updates the item_data slot with an item_pool object that contains administered items in the module.

Value

an examinee object with its item_data slot updated.

Update the routing log of an examinee object

Description

updateLog is a function for updating examinee objects after completing a module. updateLog updates logs with grades, phases, tests and modules.

Usage

updateLog(examinee_object, current_module_position)
updateLog(examinee_object, current_module_position)

Arguments

`examinee_object`	an `examinee` object.
`current_module_position`	the current module position.

Value

an examinee object with its grade_log, phase_log, test_log, and module_log slots updated.

Update the current module of an examinee object

Description

updateModule is a function for updating examinee objects after completing a module. updateModule assigns an module object from the supplied list to match the grade and the phase the examinee is in.

Usage

updateModule(examinee_object, module_list)
updateModule(examinee_object, module_list)

Arguments

`examinee_object`	an `examinee` object.
`module_list`	a module list from `loadModules`.

Value

an examinee object with its current_module slot updated.

Update the current phase of an examinee object

Description

updatePhase is a function for updating examinee objects after completing a module. updatePhase updates the phase by calling changePhase.

Usage

updatePhase(examinee_object, assessment_structure)
updatePhase(examinee_object, assessment_structure)

Arguments

`examinee_object`	an `examinee` object.
`assessment_structure`	an `assessment_structure` object.

Value

an examinee object with its current_phase slot updated.

Examples

## assessment uses two phases

examinee <- examinee_list_math[[1]]
examinee@current_phase ## P1

examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_phase ## P2

examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_phase ## P1

## assessment uses two phases

examinee <- examinee_list_math[[1]]
examinee@current_phase ## P1

examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_phase ## P2

examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_phase ## P1

Update the current test of an examinee object

Description

updateTest is the function for updating the new test ID in an examinee object.

Usage

updateTest(examinee_object, assessment_structure)
updateTest(examinee_object, assessment_structure)

Arguments

`examinee_object`	an `examinee` object.
`assessment_structure`	an `assessment_structure` object.

Value

an examinee object with its current_test slot updated.

Examples

## assessment uses two phases

examinee <- examinee_list_math[[1]]
examinee@current_test  ## T1
examinee@current_phase ## P1

examinee <- updateTest(examinee, assessment_structure_math)
examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_test  ## T1
examinee@current_phase ## P2

examinee <- updateTest(examinee, assessment_structure_math)
examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_test  ## T2
examinee@current_phase ## P1

## assessment uses two phases

examinee <- examinee_list_math[[1]]
examinee@current_test  ## T1
examinee@current_phase ## P1

examinee <- updateTest(examinee, assessment_structure_math)
examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_test  ## T1
examinee@current_phase ## P2

examinee <- updateTest(examinee, assessment_structure_math)
examinee <- updatePhase(examinee, assessment_structure_math)
examinee@current_test  ## T2
examinee@current_phase ## P1

Update the theta used for routing of an examinee object

Description

updateThetaForRouting is a function for updating examinee objects after completing a module. updateThetaForRouting determines what type of theta estimate is used to perform routing.

Usage

updateThetaForRouting(examinee_object, current_module_position, combine_policy)
updateThetaForRouting(examinee_object, current_module_position, combine_policy)

Arguments

`examinee_object`	an `examinee` object.
`current_module_position`	the current module position.
`combine_policy`	This is only applied when `module_position %% 2 == 0` (at Phase 2, which is the end of each test). `conditional` uses the combined theta (using items from the previous module combined with the current module), if the examinee was in the same grade in Phases 1 and 2. If the examinee was in different grades in Phases 1 and 2, then the theta estimate from Phase 2 is used. `always` uses the combined theta. `never` uses the theta estimate from Phase 2. (default = `conditional`)

Value

an examinee object with its estimated_theta_for_routing slot updated.

Update theta estimates using combined responses from a test

Description

updateThetaUsingCombined is a function for updating examinee objects after completing a module. updateThetaUsingCombined adds final theta estimates using all administered items in the test. A test may consist of multiple phases.

Usage

updateThetaUsingCombined(examinee_object, current_module_position, config)
updateThetaUsingCombined(examinee_object, current_module_position, config)

Arguments

`examinee_object`	an `examinee` object.
`current_module_position`	the current module position.
`config`	a `config_Shadow` object. The config for obtaining final estimates is used.

Value

an examinee object with its estimated_theta_by_test slot updated.