Error of Measurement: Validity and the Place of Consequences

The introduction of consequences of test use as a source of validity evidence has prompted a long and contentious debate. This presentation will: 1) summarize some common aspects of current validity theory; 2) demonstrate the error of incorporating consequences into the concept of validity; and 3) describe a reconceptualization of validity that accommodates consequences and illustrates how efforts aimed at gathering evidence to support test score inferencesÑvalidationÑcan be distinguished from attention to gathering support for justification of a specific test use. A framework will be presented that identifies seven dimensions on which validation and justification for test use differ.

Psychometric Properties of Scores on Mixed-Format Tests using IRT

Methods for estimating psychometric properties of scores on mixed-format tests using item response theory (IRT) methods are presented. Four real-data examples examine the following: (a) effects of weights associated with each item type on reliability, (b) comparison of conditional standard errors of measurement and reliability of different scale scores, (c) evaluation of the equity property of equating, and (d) comparison of the effects of using unidimensional and multidimensional IRT models for estimating reliability and conditional standard errors of measurement.

[No Title/Abstract]

See the American Educational Research Journal article at
http://aer.sagepub.com/cgi/content/abstract/46/1/203

This talk will have two parts: First, and concretely, I'll offer a trail of bread crumbs that mark out an elementary path back home to Bayesian thinking, including a class activity. Second, and abstractly, I'll offer my best long view of where I think our curriculum is headed. In particular, I'll address my sense that our current happy curricular picnic has only begun to register the unwelcome skunk scent of the digital revolution.

Click here to see the slides.
Click here to for the handout.

The Database for Assessment of National STEM Education Research (DANSER) is an emerging project geared toward developing a multi-disciplinary on-line STEM education assessment environment. This database, funded by NSF, is designed to support teachers of undergraduate STEM courses, assessment instrument authors, and education researchers. Multiple areas of undergraduate STEM education are included in this project, and examples from one area (statistics) will be shared.

In studies of both adults and children, struggling readers demonstrate anomalous brain activation patterns at posterior regions of the left hemisphere during phonological processing tasks. Struggling readers also display what appears to be a compensatory reliance on frontal lobe sites and right hemisphere systems. Intervention studies have examined the influence of intensive phonological remediation in young at-risk children and have revealed substantial gains in both reading scores and the development of these left hemisphere reading systems for children afforded this treatment. The presentation will describe the neuroimaging research, the intensive phonological interventions, and the outcome data on the interventions.

Learn about Search Institute's research on how multiple dimensions of human development and community contexts affect student success and development. This research includes the widely used framework of developmental assets as well as more recent research on the processes of human thriving, each of which has implications for education, family, and community development. This session will provide an overview of this research as well as emerging research priorities. The presentation will set the stage for dialogue about mutually beneficial collaborations between University of Minnesota and Search Institute on fundable research projects.

Based in Minneapolis, Search Institute is a leading innovator in discovering what children and adolescents need to become caring, healthy, and responsible adults. It applies this knowledge to motivate and equip everyone in society - youth and adults, schools, youth-serving systems, and others - to take part in creating a world where all young people are valued and thrive.

Search Institute is an independent, nonprofit, nonsectarian organization whose mission is to provide leadership, knowledge, and resources to promote healthy children, youth, and communities. It was founded in 1958 and has been promoting positive change on behalf of young people for 50 years.

The topic is the fitting of nonlinear growth curves to binary criminal offending data of male participants in the Pittsburgh Youth Study (PYS). Two models for binary data are considered, the population-averaged model and the subject-specific model (random effects model). It is argued the latter has advantages for model selection as it can be estimated with maximum likelihood. Several forms of the age-crime curve based on second-order conventional and fractional polynomials were considered. Random intercepts growth curve models were fit to the PYS binary theft and violence data based on official records and self-report. Results show that asymmetric fractional polynomial curves fitted best for the official records measures, whereas conventional quadratic polynomials fitted best for the self-report measures. The results are discussed within the context of developmental criminology and related to a number of curve fitting issues with longitudinal data.

This talk will address selected issues that are infrequently discussed in the literature but critical for planning large scale assessments of numeracy and quantitative literacy, for interpreting their results, and for translating assessment results into actionable knowledge. Examples and lessons learned from the international Adult Literacy and Lifeskills survey, NAEP, and PISA will be used to illustrate some key points.

Valerie Ruhe will present on her upcoming book with Professor Bruno Zumbo. The book is entitled"Evaluation in Distance Education and E-Learning: The Unfolding Model." Adapted from Messick's framework, the Unfolding Model responds to recurring calls to adopt a professional approach to evaluation in distance education.

The topics include: the current context, the gaps in traditional evaluation models, Messick's framework, using the Unfolding Model to conduct evaluation studies, findings from two authentic case reports, and future directions.

Early social experiments in the 1960s encountered significant technical and logistical problems, leading some researchers to prefer other methodologies. During the last 10 years, however, experiments have re-emerged as a more widely-used methodology. This talk will review the events that prompted this renaissance, and then examine progress in the use of several different kinds of designs: the randomized experiment, the regression discontinuity design, and the simple nonequivalent comparison group design with a pretest. For two quasi-experimental designs, empirical studies now suggest that they can provide estimates of effects that are as good as those from randomized experiments, although we still have much to learn about the conditions under which this optimistic conclusion might hold.

Click here to view presentation.

We all recognize the destabilizing effect that computers are having on what we do, and, potentially at least, on how we think about what we do. The resulting changes bring both good news and bad news. Much of the good news is well known: we no longer have to teach all those boring computational rituals that used to give statistics such a bad name; we can use real data in our classes, and leave the dirty work to silicon. The bad news is only beginning to sink in: teaching and learning mere technique is a lot easier than teaching and learning whatever it is that we should be doing instead, now that computers have so drastically cheapened the value of mere technique.

This talk will offersome opinions about what we should teach, and why: (1) Scrap distribution theory in favor of simulation. (2) Teach testing and intervals based on simulation. (3) Teach a lot more design. (4) Teach Bayes without his Theorem.

Click here to view presentation.

An important goal of education is to prepare students to adapt to new circumstances. Given the centrality of this educational goal, it is surprising that education has not developed assessments of adaptiveness. Instead, most measures of learning emphasize efficiency - the ability to apply one's knowledge quickly and without error. This is different from adaptation, which often involves letting go of immediate efficiencies to learn new ways of doing things. This talk will describe instruction on the statistical topic of variability that targets a specific type of adaptiveness; namely, the preparation for future learning. It will demonstrate that measures of preparation for future learning can capture positive outcomes from curricula that emphasize student innovation. At the same time, it will demonstrate that innovation-oriented curricula can also lead to efficient outcomes if done well.

The R package has a number of advantages for both teachers and students of statistics. Aside from the fact that it's free and multi-platform, it's a professional-level system that can be used at all levels of statistics study and research. This seminar will introduce R and demonstrate the power of the language for facilitating exploration of data and statistical concepts.

[No Title/Abstract Available]

QME students, and interested Ed Psych students, are invited to join QME faculty in a brown bag lunch presentation and discussion on how to quickly and smoothly clear the hurdles of prelims, prospectus, and dissertation. With advanced planning and strategic decisions the time period between the completion of coursework and the completion of the doctorate can be shortened. Learn what has helped and what has hindered the progress of QME students.

First, Michael Harwell, QME director of graduate studies, will provide an overview of the steps needed for completion of the degree. Then, Ernest Davenport, Mark Davidson, and Joan Garfield will share case studies of students who minimized (and maximized) the amount of time and energy needed to complete their degrees. Following these presentations questions and stories from students will be welcomed.

We hope to see all students - from those just beginning of their degrees to those close to receiving their doctorates - at this discussion.

[No Title/Abstract Available]

Conventional formulas for assessing the statistical significance and confidence bounds for correlation coefficients are often highly misleading in psychological research. The equations that you learned in Stat 101 were designed for manifest (observed) variables. Since at least the time of Cronbach and Meehl (1955), behavioral scientists have recognized the importance of latent variables in theory construction and testing. Statistical bounds on correlations between manifest variables are almost always smaller than the associated theoretical bounds on latent correlations. Moreover, the bounds on latent correlations do not get smaller with increasing sample size. Structural Equation Modeling cannot solve this problem. In this talk I will describe new methods for computing the theoretical bounds on latent correlations and provide examples showing why these bounds should be routinely computed in research dealing with latent constructs.

[No Title/Abstract Available]

[No Title/Abstract Available]

[No Title/Abstract Available]

Listen to three graduates of the Educational Research Methodology Program share their experiences on life and work after completing a PhD.

Simpson's Paradox occurs for two states when their difference in scores has the opposite sign of the score differences for each of the state subgroups. Simpson's Paradox is a specific manifestation of statistical confounding. The paradox has been understood for many years but is usually regarded as simply a curious anomaly. The purpose of this paper is to show the influence of Simpson's Paradox in NAEP data. NAEP public-school data are analyzed for 2000 Grade 4 Math and 2002 Grade 8 Reading. Conditions for a Simpson's reversal are presented. Approximately 100 instances of Simpson's Paradox are found per data set based on the influence of three confounders: family income, school location and race/ethnicity. In analyzing the influence of race/ethnicity two approaches are used. A straight forward approach generated 64 Simpson's reversals in the NAEP 2002 Grade 8 reading data of which 18 involve initial differences that are statistically significant. A more liberal approach generated 117 Simpson's reversals in the same data set of which 52 involve initial differences that are statistically significant. Either way these results support the claim that Simpson's Paradox is not rare in NAEP data. As a percentage of all pairs of state differences in the same data that are statistically significant, 4% are reversed using a conservative approach while 10% are reversed using a more liberal approach. All Simpson's reversals - whether statistically significant or not - are argued to have 'journalistic significance' because of their political significance. Recommendations include ordering the data by key confounders as an adjunct when reporting results. The failure to allow adjustments for confounders can lead to a serious misinterpretation of the results which in turn can lead to questionable policies.

[No Title/Abstract Available]

[No Title/Abstract Available]

[No Title/Abstract Available]

[No Title/Abstract Available]

[No Title/Abstract Available]

Dr. Becker will discuss recent advances that allow multivariate methods to be applied in meta-analysis. Refreshments will be provided. Please plan to attend!

Behavior sometimes develops in phases. When it does, the rate of change is distinctively different in one time period than in another. One possibility is the mixed-effects model adapted to multiphase data. Basically the structure is a low-order spline with random coefficients, but there are many variations on the theme. An interesting component of the model is the change-point, the time when development switches from one phase to another. In some problems, the change-point is known to the investigator and the objective of the analysis is to describe the process in each phase that is identified a priori. In other contexts, the change-point is unknown and must be estimated. In a mixed-effects model, the change-point can be a random coefficient. This possibility allows different individuals to make the transition between phases at different ages. Some examples are reviewed to illustrate the variety of repeated measures data that develops in phases. All the examples presented can be analyzed with software that is widely available.