This logfile accompanies the paper "On multi-level modeling of data from repeated measures designs: A tutorial" by Hugo Quené and Huub van den Bergh (2004).

The multilevel analyses reported here were done with the program MLn for multilevel analysis (version 1.0a). This is a command-line version of the program MLn, with output suitable for offline inspection as presented here. The program has since been modernized into a version for MicroSoft Windows, MLwiN (version 1.10). In the latter program, the user can open a command-line window and input the commands below; program output is written to a separate window.

In the log below, user input commands are indicated by blue and bolder type, like this.

Program output is indicated by black monospace type, like this.

Comments and annotations are given in italic sans-serif type, with indented left margin, like this.

MLN - Software for N-level analysis.

All commands can be abbreviated to their first 4 characters.

The command HELP gives a list of all possible commands. The command HELP with another command as parameter, e.g. "HELP DINPUT" yields a short explanation of that command.

The command HELP gives a list of all possible commands. The command HELP with another command as parameter, e.g. "HELP DINPUT" yields a short explanation of that command.

dinp c1-c4

250000 spaces left on worksheet Type file name ->f:\qb108.dat

read Disk INPut, and store data from file into columns 1 to 4 of worksheet. The listing of the Disk INPut file is abbreviated here.

1.7074 1 1 1 -.55131 1 2 1 -.75213 1 3 1 ... 1.7583 12 7 3 -.18328 12 8 3 -.72555 12 9 3

names c1 'score' c2 'subj' c3 'trial' c4 'treatm'

Rename columns to sensible variable names.

NB: This program requires that data are sorted in hierarchical order, e.g. first by subject and within subjects by trial. This was already done in the data file. Data can be sorted in MLn by CALCulating an auxiliary variable from the sort key variables, and then SORTing all columns using the auxiliary variable.

NB: This program requires that data are sorted in hierarchical order, e.g. first by subject and within subjects by trial. This was already done in the data file. Data can be sorted in MLn by CALCulating an auxiliary variable from the sort key variables, and then SORTing all columns using the auxiliary variable.

note construct CONST vector, filled with one's only

NOTE helps to add comment into the logfile.

This program does not add constant variances into the model by default; these have to be specified by the user. An auxiliary column containing the value unity (1) everywhere is needed for this purpose. This auxiliary column is CALCulated here from an already existing column.

This program does not add constant variances into the model by default; these have to be specified by the user. An auxiliary column containing the value unity (1) everywhere is needed for this purpose. This auxiliary column is CALCulated here from an already existing column.

calc c5=c1/c1

name c5 'const'

name c5 'const'

names

The NAMEs command without columns as argument returns summary information about the columns in the worksheet, as an aid to check data integrity.

Name n missing min max 1 SCORE 108 0 -4.7446 5.7573 2 SUBJ 108 0 1.0000 12.000 3 TRIAL 108 0 1.0000 9.0000 4 TREATM 108 0 1.0000 3.0000 5 CONST 108 0 1.0000 1.0000 6 C6 0 7 C7 0 ... 19 C19 0 20 C20 0

resp c1

expl 1 c5

expl 1 c5

The independent (EXPLanatory) and dependent (RESPonse) variables must be specified. The RESPonse variable is in column 1. The command to include or exclude EXPLanatory variables works as a toggle. The first parameter "1" forces inclusion of the constant variable in column 5 into the model.

iden 1 c3 2 c2

The hierarchical structure must be specified. Level-1 units (trials or occasions, lowest level) are IDENtified in column 3. Level-2 units (subjects, highest level) are IDENtified in column 2.

fpart 1 c5

The fixed part of the model must be specified. Here the grand mean is specified as the only predictor, by choosing the CONST variable in column 5 as the only predictor for the scores. The command to include or exclude explanatory variables in the Fixed PART also works as a toggle.

setv 1 c5

setv 2 c5

setv 2 c5

The random part of the model must be specified. A constant variance at both levels is specified here, by choosing the CONST variable in column 5 as the only predictor for the variance at each level.

sett

Show SETTings of the current model: specifications of the dependent and independent variables, hierarchical structure, etc.

EXPLanatory variables in CONST FPARameters CONST RESPonse variable in SCORE FSDErrors : uncorrected RSDErrors : uncorrected MAXIterations 20 TOLErance 2 METHod is RIGLS BATCh is ON IDENtifying codes : 1-TRIAL, 2-SUBJ LEVEL 2 RPM CONST CONST 1 LEVEL 1 RPM CONST CONST 1

The output of the SETTings command lists the explanatory and response variables, and gives the variance-covariance matrix at each level specified by the IDENtify command. Here the matrices only contain the CONST column, yielding constant variances and no covariances at both levels.

This is the**empty model** mentioned in the paper.

This is the

start

START the iterative estimation of the coefficients in the current model. The program is set here to iterate until a certain convergence criterion is achieved (BATCH ON). The command BATCH OFF allows inspection of estimates after each iteration.

Convergence achieved

Only a single iteration was needed to obtain good estimates for this simple model.

fixed

PARAMETER ESTIMATE S. ERROR(U) PREV. ESTIMATE CONST 0.2371 0.4401 0.2371

The command FIXED shows the estimates for the fixed part of the model, together with the standard error of that estimate (and the estimate in the previous iteration). These are given in units of the dependent variable. The single constant estimate corresponds with the grand mean, corrected or unbiased for the hierarchical random effects in the model.

random

LEV. PARAMETER (NCONV) ESTIMATE S. ERROR(U) PREV. ESTIM CORR. ------------------------------------------------------------------------------- 2 CONST /CONST ( 1) 2.056 0.9437 2.042 1 ------------------------------------------------------------------------------- 1 CONST /CONST ( 2) 2.408 0.3475 2.408

The command RANDOM shows the estimates for the random part of the model, together with the standard error of that estimate (and the estimate in the previous iteration). These are given in variance units. The single constant estimate corresponds with the pooled between-subjects and within-subjects variance, corrected or unbiased for the hierarchical random effects in the model.

aver c1

The AVERage command shows the raw, uncorrected average of the dependent variable in the first column. Note that the standard error of the mean in this single-level, disaggregated model is considerably lower than the standard error of the mean as estimated in the fixed part of the two-level model (listed above), as explained in the paper.

Count = 108 Average = 0.23708 S.D. = 2.0764 S.E.M. = 0.19980

like

249114 spaces left on worksheet -2*log(lh) is 426.334

The log-likelihood indicates the amount of "stress" or "deviance" between the model and the actual data. Smaller values generally indicate a better fit for the model (except for proportion data).

note construct dummies for treatment conditions

dummy c4 c11-c13

dummy c4 c11-c13

In order to analyze the **cell means** model mentioned in the paper, we need dummy variables for each of the 3 levels of the Treatment factor in column 4. The resulting dummy variables are stored in columns 11 to 13.

full set of 3 dummies created

names c11 'TrA' c12 'TrB' c13 'TrC'

expl 1 c11-c13

fpart 0 'const'

expl 1 c11-c13

fpart 0 'const'

Include the new dummy columns as EXPLanatory variables in the new model. Default behavior is to include these explanatory variables in the fixed part but not in the random part, unless specified explicitly. Exclude the column named 'const' (column 5) from the Fixed PART of the new model.

sett

EXPLanatory variables in TRA TRB TRC CONST FPARameters TRA TRB TRC RESPonse variable in SCORE FSDErrors : uncorrected RSDErrors : uncorrected MAXIterations 20 TOLErance 2 METHod is RIGLS BATCh is ON IDENtifying codes : 1-TRIAL, 2-SUBJ LEVEL 2 RPM CONST CONST 1 LEVEL 1 RPM CONST CONST 1

start

Convergence achieved

fixed

PARAMETER ESTIMATE S. ERROR(U) PREV. ESTIMATE TRA -0.1798 0.4807 -0.1798 TRB -0.2194 0.4807 -0.2194 TRC 1.11 0.4807 1.11

These coefficients are the estimated cell means for the treatment conditions.

random

LEV. PARAMETER (NCONV) ESTIMATE S. ERROR(U) PREV. ESTIM CORR. ------------------------------------------------------------------------------- 2 CONST /CONST ( 1) 2.099 0.9435 2.085 1 ------------------------------------------------------------------------------- 1 CONST /CONST ( 1) 2.02 0.2916 2.02

These coefficients are the estimated variances at both levels. Note that the within-subject variance at level 1 is less than in the empty model, because within-subject differences can now in part be attributed to treatment conditions.

input c41

1 -1 0 0 1 0 -1 0 0 1 -1 0

ftest c41

1 -1 0 0 1 0 -1 0 0 1 -1 0

ftest c41

In order to test pairwise comparisons among the coefficients in the fixed part of the model, we have to set up appropriate contrast weights. There are 3 coefficients in the fixed part of this model. For each contrast, 3 weights are specified for the 3 coefficients, followed by the expected value of the contrast under H0 (i.e. zero). The sequence of 3 weights plus expected value is given for three pairwise comparisons A-B, A-C and B-C. All weights for all contrasts in the fixed part are INPUT in a single auxiliary variable (in column 41), so that they are tested simultaneously. Data INPUT into column 41 is terminated by an empty line. The FTEST command performs the actual testing in the fixed part using the weights in column 41.

CONTRASTS TRA 1.00 1.00 0.00 TRB -1.00 0.00 1.00 TRC 0.00 -1.00 -1.00 result 0.04 -1.29 -1.33 chi square ( 1 df) 0.01 14.83 15.76 +/-95% c.i.(sep.) 0.66 0.66 0.66 +/-95% c.i.(sim.) 0.94 0.94 0.94 chi sq for simultaneous contrasts(3 df) = 20.40

As explained in the paper, each contrast is tested by evaluating the amount of variance associated with that contrast, using a chi-square test statistic.

cprob 14.8 1

0.00011954

Calculate the Chi-square PROBability for one of the contrasts, with chi-square = 14.8 and df=1, p=.00012. Treatment C differs significantly from treatments A and B, which are not significantly different themselves.

like

249107 spaces left on worksheet -2*log(lh) is 407.5

The log-likelihood (deviance) for the cell means model is lower than for the empty model (see above), indicating a better fit for the cell means model.

expl 0 c5

expl 1 c11-c13

expl 1 c11-c13

Make sure that the CONST variable is excluded as an EXPLanatory variable from the model (toggle off, constant vector in column 5). Make sure that the dummy variables TrA, TrB and TrC are included in the model (toggle on, dummies in columns 11 to 13). The latter command automatically includes these dummies in the fixed part of the model.

setv 1 c11-c13

setv 2 c11-c13

setv 2 c11-c13

The random part of the model must be adjusted. The dummy variables in columns 11 to 13 are specified as predictors for the variance at level 1 and level 2. The default behavior of the program is to include not only the variances, on the diagonal of the variance-covariance matrix, but also the covariances, off the diagonal. This can be verified by inspecting the current SETTings for the multilevel model.

sett

EXPLanatory variables in TRA TRB TRC FPARameters TRA TRB TRC RESPonse variable in SCORE FSDErrors : uncorrected RSDErrors : uncorrected MAXIterations 20 TOLErance 2 METHod is RIGLS BATCh is ON IDENtifying codes : 1-TRIAL, 2-SUBJ LEVEL 2 RPM TRA TRB TRC TRA 1 TRB 1 1 TRC 1 1 1 LEVEL 1 RPM TRA TRB TRC TRA 1 TRB 1 1 TRC 1 1 1

note Remove covariances between trials and treatments

Since treatments were not crossed with trials in this experimental design, there are no covariances between treatments, at level-1 (trials). The covariance terms between treatments at the lowest level (trials) must therefore be excluded.

clre 1 c12 c13

clre 1 c11 c13

clre 1 c11 c12

clre 1 c11 c13

clre 1 c11 c12

CLeaR a single Element in the variance-covariance matrix for level 1, viz the covariance between columns 12 and 13 (TrA and TrB). The other two commands work in similar fashion to remove or clear the other covariance elements.

sett

EXPLanatory variables in TRA TRB TRC FPARameters TRA TRB TRC RESPonse variable in SCORE FSDErrors : uncorrected RSDErrors : uncorrected MAXIterations 20 TOLErance 2 METHod is RIGLS BATCh is ON IDENtifying codes : 1-TRIAL, 2-SUBJ LEVEL 2 RPM TRA TRB TRC TRA 1 TRB 1 1 TRC 1 1 1 LEVEL 1 RPM TRA TRB TRC TRA 1 TRB 0 1 TRC 0 0 1

This is the correct model, with a full variance-covariance matrix at level-2 (between subjects) and variances only at level-2 (trials or occasions). Let's go!

start

Convergence achieved

fixed

PARAMETER ESTIMATE S. ERROR(U) PREV. ESTIMATE TRA -0.1798 0.4739 -0.1798 TRB -0.2194 0.5184 -0.2194 TRC 1.11 0.6383 1.11

The fixed estimates are the same as in the cell means model above.

random

LEV. PARAMETER (NCONV) ESTIMATE S. ERROR(U) PREV. ESTIM CORR. ------------------------------------------------------------------------------- 2 TRA /TRA ( 1) 2.411 1.097 2.396 1 2 TRB /TRA ( 1) 0.9261 0.8869 0.9202 0.35 2 TRB /TRB ( 1) 2.903 1.312 2.885 1 2 TRC /TRA ( 1) 3.3 1.408 3.279 0.989 2 TRC /TRB ( 1) 0.826 1.164 0.8208 0.226 2 TRC /TRC ( 1) 4.617 1.986 4.588 1 ------------------------------------------------------------------------------- 1 TRA /TRA ( 2) 0.8513 0.2458 0.8513 1 TRB /TRB ( 2) 0.9635 0.2781 0.9635 1 TRC /TRC ( 2) 0.8186 0.2363 0.8186

The random estimates show interesting properties, especially at level-2 (between subjects), as explained in the paper. First, between-speaker variances differ widely among treatments (2.4, 2.9 and 4.6), although not significantly so (see below). Second, covariances among treatments are not constant among pairs of treatments. Hence there is no compound symmetry and no sphericity. Variance within subjects (level-1) seems to be homogeneous across treatments.

input c42

1 0 -1 0 0 0 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 0 0 1 0 0 -1 0 0 0 0

rtest c42

1 0 -1 0 0 0 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 0 0 1 0 0 -1 0 0 0 0

rtest c42

The different between-speaker variances among treatments can be evaluated by means of pairwise comparisons, similar to pairwise comparisons in the fixed part of the cell means model (see above). We have to set up appropriate contrast weights. There are 9 coefficients in the random part of this model (6 at level-2 and 3 at level-1). For each contrast, 9 weights are specified for the 9 coefficients, again followed by the expected value of the contrast under H0 (i.e. zero). The sequence of 9 weights plus expected value is given for three pairwise comparisons A-B, A-C and B-C. All weights for all contrasts in the random part are INPUT in a single auxiliary variable (in column 42), so that they are tested simultaneously. Data INPUT into column 42 is terminated by an empty line. The RTEST command performs the actual testing in the random part using the weights in column 42.

CONTRASTS TRA /TRA :> 1.00 1.00 0.00 TRB /TRA :> 0.00 0.00 0.00 TRB /TRB :> -1.00 0.00 1.00 TRC /TRA :> 0.00 0.00 0.00 TRC /TRB :> 0.00 0.00 0.00 TRC /TRC :> 0.00 -1.00 -1.00 TRA /TRA := 0.00 0.00 0.00 TRB /TRB := 0.00 0.00 0.00 TRC /TRC := 0.00 0.00 0.00 result -0.49 -2.21 -1.71 chi square ( 1 df) 0.09 3.11 0.54 +/-95% c.i.(sep.) 3.19 2.45 4.57 +/-95% c.i.(sim.) 6.69 5.14 9.59 chi sq for simultaneous contrasts(9 df) = 3.88

cpro 3.11 1

0.077813

Calculate the Chi-square PROBability for this contrast, with chi-square = 3.11 and df=1, p=.078. Although the between-speaker variance under treatment A is considerably smaller than under treatment C (2.4 vs 4.6), this difference is not significant. Nevertheless the assumption of homoschedasticity is probably violated in these data.

calc b1 = 3.3^2 / (2.411*4.617)

0.97830

note this is correlation in subj means between treatmts A and C

The covariance amount of 3.3 can be regarded as unstandardized correlation in subject means between treatments A and C. Here the standardized correlation is CALCulated, yielding r=0.978. This standardized correlation is also given (value 0.989) in the output for the random part, in the last column of the output table (above).

ftest c41

The full model still has only 3 coefficients in its fixed part. For the pairwise comparisons we can re-use the contrast weights that were set up above for the cell means model, and stored in column 41. Again, each contrast is tested by evaluating the amount of variance associated with that contrast, using a chi-square test statistic.

CONTRASTS TRA 1.00 1.00 0.00 TRB -1.00 0.00 1.00 TRC 0.00 -1.00 -1.00 result 0.04 -1.29 -1.33 chi square ( 1 df) 0.00 20.28 3.28 +/-95% c.i.(sep.) 1.14 0.56 1.44 +/-95% c.i.(sim.) 1.63 0.80 2.05 chi sq for simultaneous contrasts(3 df) = 22.91

cprob 3.28 1

0.070129

The third contrast, between treatments A-C, yields a chi-square value of 3.28 with df=1, with a Chi-square PROBability of p=.070, which is not significant at alpha=.05. Adding random coefficients has led to larger standard errors for the cell means estimates in the fixed part, as compared to the cell means model. In turn, this has led to more conservative testing of the pairwise comparisons: the third contrast was significant in the cell means model but not in the full model.

like

249058 spaces left on worksheet -2*log(lh) is 355.759

The log-likelihood (deviance) for the current full model is lower than for the empty model or for the cell means model (see above), indicating a better fit for the current model.

save

Type file name ->

f:\qb108.ws

249028 spaces left on worksheet

SAVE the current worksheet for later use. The worksheet contains the data in columns 1 to 4, auxiliary variables, and estimated coefficients in columns 96 to 99.

stop

STOP the current session, and terminate the program MLn.

2004.02.20 HQ