- [2007.01.08] Here at last are your grades, with apologies for the overlong delay.
In the past weeks I've re-read (almost) every assignment, contrary to my intention to grade only a sample of your work.
Now I'm still searching a few missing papers. If you think something is wrong then please let me know. Total grades will be computed later this week, using adjusted weights of 70:30 for weekly:final assigments.

In order to clarify a few frequent errors in your assignments, I'll also write some post-mortem notes to be added to this page, later this week. - [2006.11.08] As promised, I've added my annotated answers for session 7 (last year, this was session 8).
- [2006.10.09] Second extra class for SPSS will be Fri 13 Oct, 9:00 to 11:30, Janskerkhof 13, room K.060 (basement). Prepare questions and problems!
- [2006.10.03] Extra class for SPSS intro will be Fri 6 Oct, 9:00 to 11:30, Janskerkhof 13, room K.060 (basement). We will more or less follow the outline of this Dutch workbook for SPSS from my Statistics intro course.
- [2006.09.24] Added a hyperlink (to session 1) to a posting about Gabby guys: the effect size. This is also related to the notes on effect size on this page (see session 4).
- [2006.09.05] Classes will meet on
**Friday**12-15h, in the James Boswell Institute, room 113. More details below. - [2006.08.22] We will use a Yahoo bulletin board to exchange information in this course. The bulletin board or 'group' for this course is at uk.groups.yahoo.com/group/mer200607. Please subscribe to (join) this group, by visiting the group webpage and following instructions given there. More instructions are available below, see the first session in the schedule. PS: This is a members-only group, meaning that only members can access its contents. The teacher has to approve your subscription for membership.

Trans 10, room 1.31

office hours Tue 14:00-15:30 and by appointment

- Rietveld, T. & Van Hout, R. (2005).
Statistics in Language Research: Analysis of Variance. Berlin: Mouton de Gruyter. ISBN 3-11-018581-4.

Further abbreviated as RvH. - Additional reading materials; these will be distributed either in class, or through my pigeonhole (postvak), at Trans 10, room 1.21.

Fri | 12:00-15:00 | James Boswell Inst, room 113 |

The course will be taught in English.

- make assignments about the topics covered in the last meeting;
- hand in your assignments (see below), by Tuesdays 18:00h at the latest;
- review and judge the assignments of a fellow student, by Thursday 12:00h at the latest;
- read and study new materials.

Put your work in one document per week, preferably in PDF, since that ensures correct display of figures and tables. Place your document in a folder on your personal webpage at UU — check the CIM help pages for instructions. Make sure your document is accessible over the web.

Now send a short message to the group bulletin board, with a short description of your work and with a working hyperlink (URI) to your document on your webpage. (You don't have to enclose the URI in tags, just copying the right

Retrieve the document of your victim for this week, and write a review of her/his work in a separate document. Place your review on your personal webpage, and again announce its location on the group bulletin board, by Thursday 12:00h at the latest.

On Thursday afternoon or evening, you should read the peer review of your assignment. Notice that strict timing is required to make this schedule work!

Peer review, commenting the work of a peer or colleague, is a serious business. You can learn more about it through these web pages:

- You Lost Me In The Third Paragraph, about "gracious criticism" (Writing Center, George Mason Univ, Fairfax, VA);
- Responding To Other People's Writing (Writing Center, Univ North Carolina, Chapel Hill, NC).

This means that your assignments and reviews will

Experimenting. General methodology. Design of experiment. Refresher on inferential statistics. How to peer-review.

- RvH: Chapters 1 and 2 (and 3).

- This course requires and presumes that you already have previous knowledge of statistics, equivalent with an introductory course in statistics. You may test yourself by means of this tentamen of the Statistiek course.
- Make sure that you have an account on the Solis UU netwerk.
- Join the course bulletin board for this class. By way of experiment, we'll use a Yahoo! group for exchanging documents and information. You'll need a Yahoo! ID to join this group, so you may need to sign up or register as a Yahoo! user. The homepage for this class group is uk.groups.yahoo.com/group/mer200607. Visit this homepage, and join the group [just press the Join button], so you can tell others about your assignments, reviews, etc.

More information about how to submit and distribute your work is given above. - Browse the various websites listed below. Make sure to browse the Research Methods Knowledge Base.

Your elaborations on the questions below have to be placed on your personal webpage, and announced on the group webpage, as described above. Write clearly, correctly, and concisely. Make a PDF (preferred) or PS or HTML document, or plain ASCII text file, with a maximum length of about 2000 words. (I've made a short explanation about how to make a non-proprietary document, in Dutch.)

- Visit your institutional library (e.g. Letterenbibliotheek). Take a recent printed issue (2005 or 2006) of an experimental journal (in phonetics, psycholinguistics, etc.), such as Language and Speech, Journal of Phonetics, Speech Communication, Journal of Memory and Language, Phonetica, and select an article that reports an experimental study.

(a) Which questions does the study attempt to answer?

(b) Which independent and dependent variables are involved in the study?

(c) Describe the design of the experiment. - A researcher wants to know whether the vowel duration in stressed vowels is longer than in unstressed vowels. There are two groups of participants, and the researcher is interested in their difference (e.g. L1 and L2 speakers). The target vowels occur in the first vs. the third syllable of three-syllable words. To prevent strategic behavior (what's that?), a speaker may not produce words with different stress patterns: all words produced by a single speaker need to have the same stress pattern.

Provide a possible design for this experiment. Indicate which factors are between or within subjects, dependent or independent, etc. Make a graph or table to illustrate your design. - Answer the following questions in RvH section 2.9: Exercises 2, 3, 4, 5, 6, 8.
- In preparation for next week, also read RvH Chapter 3.
- This last assignment is not for peer review but for independent study. Now is the perfect time to brush up your statistical skills. Answer the tentamina of my Statistics course (see above). Afterwards, check your answers with those provided on the course webpage. Determine what parts of your statistics proficiency are still deficient. Design a plan of action, to remedy your shortcomings during this teaching period.

- Research Methods Knowledge Base by William M. Trochim, Cornell University, Ithaca, NY (The Web Center for Social Research Methods)
- Statistics Every Writer Should Know by Robert Niles, journalist at the Los Angeles Times.
- Rice Virtual Lab in Statistics by David Lane, Rice University, Houston, TX. This website also contains HyperStat Online, an online introduction to statistics.
- WISE Project, Web Interface for Statistics Education, at Claremont Graduate University, Claremont, CA.
- StatPages.Net by John C. Pezzullo, Georgetown University, Washington DC. A treasure trove of helpful links and programs.
- web-based tools for statistical computation, by Richard Lowry, Vassar College, Poughkeepsie, NY.
- GraphPad QuickCalcs, easy online statistical calculators.
- Gabby guys: the effect size, by Mark Liberman, Univ of Pennsylvania, PA.
- Lucian Freud, Grosses Interieur W 11 (nach Wattau) (1981/'83).

ANOVA: general principles, one-way, post-hoc test, power.

**Readings**:
RvH: Chapters 3 and 4.

Your answers and solutions to the questions below have to be handed in as described above. As always, write clearly, correctly, and concisely.

- Answer the following questions in RvH, section 3.10: Exercises 2, 5.
- Answer the following questions in RvH, section 4.11: Exercises 1, 2, 3, 4, 7, 8.

ANOVA: multifactorial designs, interaction, fixed vs random factors, error terms.

**Readings**:
RvH: Chapters 5 and 6.

Your answers and solutions to the questions below have to be handed in as described above. As always, write clearly, correctly, and concisely.

- Answer the following questions in RvH, section 5.13: Exercises 1, 3, 5, 6.
- Answer the following questions in RvH, section 6.9: Exercises 3, 4, 5.
- In a study of cardiovascular risk factors, joggers who run at least 15 miles per week were compared with a control group described as "generally sedentary". Both men and women participated in this study. The design is a 2x2 between-subjects ANOVA, with Group and Sex as factors. There were 200 participants for each combination of factors. One of the dependent variables is the rate of heartbeat of a participant, after 6 minutes on a treadmill, expressed in beats per minute.

Data from this study are available here in SPSS format, or as plain text (the latter file contains variable names in the first line).

(a) What do you think of the construct validity? Please comment.

(b) Is is allowed to conduct an analysis of variance on these data? Motivate your answer with relevant statistical considerations.

(c) Conduct a two-way ANOVA on these data.

(d) Write a summary of the results of this study, including the (partial) effect size η and η^{2}. Draw your conclusions clearly.

(e) From each cell (combination of factors), draw a random sample of n=20 individuals, out of the 200 in that cell. Explain how you have performed the random sampling. Repeat the two-way ANOVA on this smaller data set.

(f) Discuss the similarities and differences in results between (b) and (d).

This exercise is adapted from: Moore, D.S., & McCabe, G.P. (2003). Introduction to the Practice of Statistics (4th ed.). New York: Freeman. Example 13.8, pp.813-816.

ANOVA: Repeated Measures, post-hoc tests.

- RvH: Chapter 8
- my manual about ANOVA in SPSS.

Compare these notes from similar courses in experimental research methods, at other universities:

- Repeated Measures ANOVA, from Durham University, Durham, England — with instructions for SPSS.
- Repeated Measures ANOVA, from University of Guelph, Canada — with instructions for SPSS.
- Repeated Measures ANOVA Using SPSS Manova, Univ of Texas, Austin, TX — with solid background.
- MANOVA and GLM in SPSS, van UCLA, Los Angeles, CA — about differences between these two methods for repeated measures ANOVA.
- A bluffer's guide to sphericity, by Andy Field.

Your answers and solutions to the questions below have to be handed in as described above. As always, write clearly, correctly, and concisely.

- Answer the following questions in RvH, section 8.15: Exercises 1, 2, 3, 4, 5.

If we are comparing *two* groups of means, as in a pairwise *t* test, then the effect size *d* is defined as: *d* = (m_{1}-m_{2})/s (Cohen, 1969, p.18; m represents mean).

A value of *d*=.2 is regarded as small, *d*=.5 as medium, *d*=.8 as large. It is left to the researcher to classify intermediate values (ibid., p.23-25).

The difference in body length between girls of 15 and 16 years old has a small effect size, just as male-female differences in sub-tests of an IQ test. "A medium effect size is conceived as one large enough to be visible to the naked eye," e.g. the difference in body length between girls of ages 14 and 18. Large effect sizes are "grossly perceptible", e.g. the difference in body length between girls of ages 13 and 18, or the difference in IQ between PhD graduates and freshman students.

If we are comparing *k* groups of means, as in an *F* test (ANOVA), then the effect size *f* is defined as: *f* = s_{m}/s, where s_{m} in turn is defined as the standard deviation of the *k* different group means (ibid., p.268). If *k*=2, then *d*=2*f* (ibid., p.278). These rules apply only if all groups are of the same size; otherwise different criteria apply.

A value of *f*=.10 is regarded as small, *f*=.25 as medium, *f*=.40 as large.
Again, it is left to the researcher to classify intermediate values (ibid., p.278-281).

Small-sized effects can also be meaningful or interesting. Large differences may correspond to small effect sizes, due to measurement error, disruptive side effects, etc. Medium effect sizes are observed in IQ differences between house painters, mechanics, carpenters, butchers. Large effect sizes are observed in IQ differences between house painters, mechanics, carpenters, (railroad) engine drivers, and lab technicians.

Adapted from: Cohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences (1st ed.). New York: Academic Press.

regression, error of measurement, reliability.

- Ferguson, G. A., & Takane, Y. (1989). Statistical Analysis in Psychology and Education (6th ed.). New York: McGraw-Hill. Chapter 24 "Errors of Measurement", pp.466-478.
- optional: Trochim, W.M. (2002). Measurement. In: Research Methods Knowledge Base (Web Center for Social Research Methods).

Your answers and solutions to the questions below have to be handed in as described above. As always, write clearly, correctly, and concisely.

- Answer the following questions:
Ferguson & Takane, Chapter 24: Exercises 1, 2.

- After you've done this week's reading, make a trip through the Reliability Test Maze, consisting of 10 tricky questions. Make notes of your thoughts and answers, and whether your answer was right or wrong. For the questions you've answered WRONG, you have to say why you came to that answer, and why it was wrong. Discuss these thoughts in a paragraph, one paragraph for each wrong answer. Also indicate the number of questions you've answered correctly.

- Test Reliability, by Lucy Jacobs, Indiana University

Let us assume that we have 2 observations for each of 5 persons. These observations are about the perceived body weight, as judged by two 'raters' or judges, x1 and x2. The data are as follows:

person x1 x2 1 60 62 2 70 68 3 70 71 4 65 65 5 65 63Because we have only two measures (variables), there is only one pair of measures to compare in this example. Very often, however, there are more than two judges involved, and hence many more pairs.

First, let us calculate the correlation between these two variables x1 and x2. This can be done in SPSS with the `Correlations` command (Analyze > Correlate > Bivariate, check Pearson correlation coefficient).
This yields r=.904, and the average r (over 1 pair of judges) is the same.

If you need to compute r manually, one method is to first convert x1 and x2 to Z-values [(x-mean)/s], yielding z1 and z2. Then r = SUM(z1×z2) / (n-1).

This value of r corresponds to Cronbach's Alpha of (2×.904)/(1+.904) = .946 (with N=2 judges).
Cronbach's Alpha can be obtained in SPSS by choosing Analyze > Scale > Reliability Analysis. Select the "items" (or judges) x1 and x2, and select model Alpha.
The output states: `Reliability Coefficients [over] 2 items, Alpha = .9459` [etc.]

If the same average correlation r=.904 had been observed over 4 judges (i.e. over 4×3 pairs of judges), then that would have indicated an even higher inter-rater reliability, viz. alpha = (4×.904)/(1+3×.904) = .974.

Exactly the same reasoning applies if the data are not provided by 2 raters judging the same 5 objects, but by 2 test items "judging" a property of the same 5 persons. Both approaches are common in language research. Although SPSS only mentions items, and inter-item reliability; the analysis is equally applicable to raters or judges, and inter-rater reliability.

Note that both judges (items) may be inaccurate. A priori, we do not know how good each judge is, nor which judge is better. We know, however, that their reliability of judging the same thing (true body weight, we hope) increases with their mutual correlation.

Now, let's regard the same data, but in a different context. We have one measuring instrument of the abstract concept x that we try to measure. The same 5 objects are measured twice (test-retest), yielding the data given above. In this test-retest context, there is always just one correlation, and the idea of inter-rater reliability does not apply in this context. We find that r_{xx}=.904.

This reliability coefficient r = s^{2}_{T} / s^{2}_{x} . This provides us with an *estimate* about how much of the total variance is due to variance in the underlying, unknown, "true" scores. In this example, 90.4% of the total variance is estimated to be due to variance of the true scores. The complementary part, 9.6% of the total variance, is estimated to be due to measurement error. If there were no measurement error, then we would predict perfect correlation (r=1); if the measurements would contain only error (and no true score component at all), then we would predict zero correlation (r=0) between x1 and x2.

In this example, we find that

s_{e} = s_{x} × sqrt(1-.904) = sqrt(15.484) × sqrt(.096) = 1.219

check: s^{2}_{x} = 15.484 = s^{2}_{T} + s^{2}_{e} =
s^{2}_{T} + (1.219)^{2},

so s^{2}_{T} = 15.484 - 1.486 = 13.997

and indeed r = .904 = s^{2}_{T} / s^{2}_{x} = 13.997 / 15.484.

Supposedly, x1 and x2 measure the same property x. To obtain s^{2}_{x}, the total observed variance of x (as needed above), we cannot use x1 exclusively nor x2 exclusively. The total variance is obtained here from the two standard deviations:

s^{2}_{x} = s_{x1} × s_{x2}

s^{2}_{x} = 4.18330 × 3.70135 = 15.484

In general, a reliability coefficient smaller than .5 is regarded as low, between .5 and .8 as moderate, and over .8 as high.

Multiple regression, multivariate analyses.

- Moore, D.S., & McCabe, G.P. (2003). Introduction to the Practice of Statistics (4th ed.). New York: Freeman. [ISBN 0-7167-9657-0]. Chapter 11 "Multiple Regression", pp. 708-745. [book companion website].
**optional**: Devore, Jay & Peck, Roxy (2001) Statistics: The Exploration and Analysis of Data (4th ed.). Pacific Grove, CA: Duxbury. [ISBN 0-534-35867-5.] Chapter 14 "Multiple Regression Analysis", pp. 553-610.- optional: chapter on Multiple Regression, from the excellent online statistics textbook at StatSoft, Inc.

Your answers and solutions to the questions below have to be handed in as described above. As always, write clearly, correctly, and concisely. File your work in folder "six" of the group webpage.

- Answer the following questions:
Moore & McCabe, Chapter 11: Exercises 2, 3, 16, 33.

Data for the last two questions are available here in plain text format (the first line of this file contains variable names).

For questions 16 and 33 the FORWARD method is most appropriate.
This means that you start with an empty model (only intercept b_{0})
to which predictors are added step by step. After each addition of a predictor,
you check whether the model performs significantly better than before
(e.g. by checking whether R^{2} increases).

The questions are about the **increment** in R^{2} by **adding** a predictor.
The relevant information is easier to find in the SPSS output if you specify
the FORWARD method.

As a bonus, you could check what happens if you exclude case #51 from the
data set, e.g. by marking it as a missing value. This is quite easy if you
keep the regression command in a Syntax window for repeated use.

For admission to a university, two things are taken into account: (a) your average grades in the final years of high school (HSM, HSS, HSE), and (b) your score in a national admissions exam, like the Dutch CITO test (Scholastic Aptitude Test, SAT). Top-class universities, like Harvard, Yale, Stanford, etc., use both parameters in selection. You have to be the best in your class (but your classmates are strongly competing for this honor), plus you need a minimal score on your SAT.

During your academic study, all your grades and results contribute to your Grade Point Average (GPA), a weighted average grade. This GPA is generally used as an indication of academic achievement and success. The authors attempt to predict the GPA from the previously obtained indicators (a) and (b).

Take a sample of fathers, and note their body length (X). Wait for one full generation, and measure the body length of each father's oldest adult son (Y). Make a scattergram of X and Y. The best-fitting line throught the observations has a slope of less than 1 (typically about .65). This is because the sons' length Y tends to "regress to the mean" — outlier fathers tend to produce average sons, and average fathers also tend to produce average sons. Galton called this phenomenon "regression towards mediocrity". Thus the best-fitting line is a "regression" line because it shows the degree of regression to the mean, from one generation to the next. (Note that any slope larger than 0 suggests an hereditary component in the sons' body length, Y.)

Questions: Which variable has the larger variance, X or Y? Does the variation in body length increase or decrease (regress) over generations? Why?

- Why is it called regression, by Ann Lehman and John Sall (SAS Institute).
- Galton, Pearson, and the Peas, by Jeffrey M. Stanton (Syracuse University).

r

- Ferguson, G. A., & Takane, Y. (1989). Statistical Analysis in Psychology and Education (6th ed.). New York: McGraw-Hill. p495.

There will be no class this week, because of Fall Break (herfstvakantie).
This gives you a chance to catch up on reading materials, etc.
There is a refresher assignment (see below) but there will be **no peer review** of this assignment!

Here is one refresher assignments, outside the normal peer-review process. Your answers and solutions to the question below do NOT have to be handed in, but you're invited to submit them into folder "extra" on the group webpage. As always, write clearly, correctly, and concisely.

- In a
**fictitious**study, the effect of a growing potion was investigated. The growing potion was administered in 5 different dosages (of 1, 3, 5, 10, and 20 units per day), to 10 men and to 10 women for each dosage, during 15 days. The dependent variable is the increase in body length of a participant, after 15 days, in cm.

Data from this study are available here in SPSS format, or as plain text (the latter file contains variable names in the first line).

(a) Import these data into SPSS or a statistical package of your choice. Make a graph of the increase in body length, for each of the 10 conditions. (Hint: In SPSS use a "clustered boxplot".) Discuss what the graph shows.

(b) Conduct a two-way ANOVA on these data, with Sex and Dosage as two "fixed" factors. Include measures of effect size and of power in your report.

(c) What is the range of generalisation over dosages, in the ANOVA in (b)? Discuss the external validity of the dosage factor.

(d) Conduct a two-way ANOVA, but now with Dosage as "random" factor. (Hint: SPSS does not handle "mixed" models like this one very well. It's probably easiest to calculate the F-ratios by hand, using the ANOVA results obtained under (b) above.)

(e) What is now the range of generalisation over dosages, in the ANOVA in (d)? Again discuss the external validity of the dosage factor.

(f) Discuss the similarities and differences in results between the two ANOVAs in this assignment. Does the growing potion have a different effect on men and women?

logistic regression, GLM, modelling.

- Moore, D.S., & McCabe, G.P. (2003). Introduction to the Practice of Statistics (4th ed.). New York: Freeman. [ISBN 0-7167-9657-0]. Chapter 15 "Logistic Regression", only available here. [book companion website].
**optional**: Generalized Linear Models, van StatSoft, Inc — be warned that this is not an easy text! Concentrate on the first part, until "Types of Analyses". The sections on matrix algebra may be skipped. Make notes about your questions and problems with this text.

- Logistic Regression, from UCLA, Los Angeles, CA.
- GLM in SPSS, from UCLA, Los Angeles, CA.
- Generalized Linear Models (GLZ), by Edward F. Connor, San Francisco State University, CA.
- GLM in SPSS, from Universiteit Gent.
- Logistic Regression, by Michael T. Brannick, University of South Florida, Tampa, FL.

Your answers and solutions to the questions below have to be handed in as described above, by Tuesday 18:00. As always, write clearly, correctly, and concisely.

- Answer the following questions: Moore & McCabe, Chapter 15: Exercises 8, 10, 12, 25.

In order to speed up your work on exercise 15.25 in SPSS, I've put the data on the web, in a plain text data file. The first line contains the names of the variables. Data (N=2900) start on line 2, and are coded as follows:hospital: 0=hosp.A, 1=hosp.B; outcome: 0=died, 1=survived; condition: 0=poor, 1=good.

Variables are separated by commas.

In your logistic regression, the variables`hospital`and`condition`must be treated as categorical variables. For easier interpretation of the results, I prefer to use the zero codes as references or baselines (in SPSS choose`Reference: First`).

SPSS does not provide you with 95% confidence intervals; you need to calculate these by hand. The Wald statistic in the SPSS output is the same as the test statistic for β as defined on p.46 in the reading material. - For this week, there will be
**no peer review**to write, since we're pressed for time in the final week of this teaching period.

As promised, here are my annotated answers for session 7.

As always, the revised paper should be (as much as possible) a running text, not a collection of incomplete sentences and statistical output.

In the revised version you have to accommodate the comments of your reviewer — if you agree of course. Also use the reading materials and hyperlinks provided.

You may discuss the reviewer's comments in the text of your revised version. But perhaps you find it easier to write a coherent (revised) text on your own, plus a second document with revision notes, in which you discuss the reviewer's comments explicitly, stating which comments you have taken into account, which comments you have ignored, and why.

**Deadline is Thursday 16th November 2006, 23:59 h**.

- Maxwell, S.E. & Delaney, H.D. (2004) Designing Experiments and Analyzing Data: A model comparison perspective (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates. ISBN 0-8058-3718-3. [very good, but not an easy book].
- Levin, I.P. (1998) Relating Statistics and Experimental Design: An introduction. Thousand Oaks, CA: Sage. Sage University Papers Series on Quantitative Applications in the Social Sciences; 07-125. ISBN 0-7619-1472-2.
- Carver, R.H. & Nash, J.G. (2005) Doing Data Analysis with SPSS version 12.0. Belmont, CA: Brooks/Cole. ISBN 0-534-46551-x.
- Kirkpatrick, L.A. & Feeney, B.C. (2005) A Simple Guide to SPSS for Windows/ for Version 12.0. Belmont, CA: Thomson Wadsworth. ISBN 0-534-61006-4.
- StatSoft, Inc. (2004) Electronic Statistics Textbook. Tulsa, OK: StatSoft. URL: http://www.statsoft.com/textbook/stathome.html [clear and concise chapters about most statistical topics].
- Also check the hyperlinks listed under session 1.
- Also check the webpage of my statistiek course [in Dutch].

© 2003-2007 HQ 2007.01.10