Assessing Non-cognitive Constructs in Education

Non-cognitive constructs are important predictors of academic achievement and behavioral adjustment that transcend socio-economic factors.
 
Although terminology can vary because of discipline-specific definitions, some of the most commonly researched non-cognitive constructs can be grouped into four domains. “The first broad group of non-cognitive constructs includes the beliefs that students hold about themselves as learners, the nature of learning, the fairness or supportiveness of the school environment, and their attitudes towards different disciplinary areas and towards school in general” (Lipnevich, MacCann, & Roberts, 2013, p. 2). Essentially, beliefs manifest reality. Second are social and emotional qualities as they define interpersonal and intrapersonal coping skills leading to success. Furthermore, habits and processes such as time management and meta-cognitive skills such as self-monitoring do, indeed, effect academic achievement.  The final domain is broad personality traits and narrow personality facets. Broad personality traits include extraversion, agreeableness, conscientiousness, neuroticism and openness, with conscientiousness and openness to experience showing robust relationship with academic achievement. Narrow personality facets underlie each broad category of personality traits and may be more predictive than the broad domains.
 
There are a variety of assessment methods each with strengths and weaknesses that can affect test characteristics.
 
Traditional approaches include self- and other-report rating scales and interviews.
 
Self-assessments are the most widely used non-cognitive assessment. At a broad level, they measure “personality, values, beliefs, and affect. Examples of specific constructs that may be assessed with self-reports include communication skills, time management, teamwork, leadership, self-regulation, self-efficacy, and altruism” (Lipnevich, MacCann, & Roberts, 2013, p. 3). Although self-assessments are a relatively pragmatic, cost-effective, and efficient way of gathering information, they can have many issues. The TpB (Theory of Planned Behavior) has been particularly effective in serving as a framework for the development of assessments of individuals’ attitudes; it posits that the central determinant of volitional behavior is one’s intention to engage in that behavior (Lipnevich, MacCann, & Roberts, 2013, p. 4).
 
Other-ratings are assessments made by other people. This is a well-established way to gather information. Although they preclude socially desirable responding, they are prone to rating bias. Self and other-ratings do not always converge, with other-ratings being the more accurate predictor of academic achievement. Other-rating assessments include teacher and parent-ratings which are used due to concerns that children lack cognitive or psychological ability to self-rate. Parent ratings of behavior (BASC-2) accurately capture a range of behavior problems. Conscientiousness is also accurately assessed with these ratings. “A more external perspective might be viewed as a more objective reporting of the facts and so might result in more accurate measurement of the construct” (Lipnevich, MacCann, & Roberts, 2013, p. 6). Another other-rating assessment is the letter of recommendation that includes information about past performance and the writer’s opinion as an essay. The lack of standardized format undermined the ability to compare letters; this led to a standardized letter of recommendation: The Educational Testing Service (ETS)® Personal Potential Index has shown initial evidence of reliability and validity. Biographical data (biodata), explored for college admissions, is a standard methodology for assessing opportunity to learn. Questions about an individual’s past behavior, activities, or experiences are offered in multiple-choice or open format. It is effective for measuring personality and may be less fakeable than standard self-report scales. Another other-rater assessment is the common interview. This frequently-used assessment can be structured with prepared questions or unstructured allowing adaptive or intuitive switching of topics. There are three types of structured interviews:
 
The behavioral description interview involves questions that refer to the candidate’s past behavior in real situations. The situational interview uses questions that require that interviewees imagine hypothetical situations (derived from critical incidents) and state how they would act in such situations. The multimodal interview combines the two approaches and adds unstructured parts to ensure high respondent acceptance. (Lipnevich, MacCann, & Roberts, 2013, p. 8)  
 
Structured interviews are good performance predictors and incrementally valid above mental ability with behavior description interviews showing higher validity than situational interviews. In academic settings, interviews are moderately effective predictors of meaningful outcomes.
 
Novel non-cognitive assessments include situational judgement tests (SJTs), day reconstruction, implicit association tests (IATs), and conditional reasoning tests (CRTs).
 
Situational judgement tests (SJTs) present individuals with a situation and the opportunity to choose their typical response from a list of items using multiple-choice, constructed response, or ratings. The advantages of self-rated SJT’s include an ability to assess more subtle and complex judgement processes, as well as many relevant attributes of applicants; consequently, overcoming the validity ceiling found for conventional cognitive assessments. They also have reduced subgroup differences, can be used in training to provide feedback on competencies, and are less susceptible to faking. SJT’s use in academic settings is on the rise because there is increasingly more evidence for their construct validity. Multimedia SJT’s have more criterion-related validity than written SJT’s for predicting interpersonal skills and appear more effective when measuring student’s affective characteristics. “The researchers suggest that self- and parent-evaluations may index the frequency of different types of emotion management strategies” (Lipnevich, MacCann, & Roberts, 2013, p. 11).  SJT’s are promising non-cognitive assessments because of their high validity and suitability for younger children and those with limited verbal ability.
 
Day Reconstruction Method (DRM) explores how people use their time. Although DRM is an intrusive assessment that requires a significant expenditure of energy, it does appear to evaluate difficult-to-capture characteristics. It may also be generalizable to high school and college populations. Ultimately, the DRM has been “substantially correlated with a self-assessment of psychological well-being and a situational judgment test of emotional management” (Lipnevich, MacCann, & Roberts, 2013, p. 11).
 
Analysis of writing samples is based on the idea that what and how we speak and write reflects our personality.
As far as faking, people can and do fake responses in non-cognitive assessments.  The attempt to address this behavior with “lie scales” has returned inaccurate results. That being said, “using non-cognitive scores to screen out wildly inappropriate applicants is still a useful exercise” (Lipnevich, MacCann, & Roberts, 2013, p. 12). Although in their infancy, Implicit Association Tests, Conditional Reasoning Tests, Forced-choice Assessment and the Bayesian Truth Serum are designed to reduce the possiblity for faked responses.
 
The Implicit Association Test (IAT) is popular for researching attitudes. IAT’s record reaction time to classify stimulus pairs with. They have high internal consistencies, but lower test-retest reliabilities. More research is needed due to controversy about what the IAT measures as well as the significant amount of method-specific variance in this assessment.
 
Conditional Reasoning Tests (CRTs) look like reading comprehension or logic, but measure world-view, personality, biases, and motives. CRT’s are used to measure aggression or achievement motivation, but this is not reliably replicated assessment. Cognitive difficulty restricts CRT’s to populations at or above high school, and it is not known if they are subject to faking.
 
“The basis for forced-choice testing is that test-takers are forced to choose between two or more statements that are equal in evaluative content (i.e., are equally socially desirable) but differ in terms of their descriptive content (i.e., measure different personality traits)” (Lipnevich, MacCann, & Roberts, 2013, p. 13) through pair comparisons, rank-ordering, and multidimensional forced-choice. Forced choice assessments may be less fakeable than standard rating scales while showing strong relationships with performance. They do not, however, reveal accurate measures for test-takers who are high on multiple personality dimensions. A multi-dimensional pair-wise preference (MDPP) measure may address this issue by first determining both social desirability and item parameters of many items presented in conventional format, and then using social desirability ratings and item parameters to develop pairs of statements that act as a pair-comparison judgement. The MDPP method creates scores that differentiate between people and appear resistant to faking. Consequently, this method is being more widely used in academic settings.
 
The Bayesian Truth Serum (BTS) calculates how often people endorse possibly undesirable items. A test-taker must provide two pieces of information for every item (e.g. what behavior is most like them and what percent of the population would choose an answer). Because of this, the BTS may be subject to frame-of-reference effects. Complexity of BTS might only be accurate after the mid-teens with cognitively normal populations.
 
Potential uses of non-cognitive assessments include high-stakes assessments, developmental scales, and interventions.
 
“High-stakes applications of non-cognitive tests in education include diagnosis and selection. Non-cognitive assessments have an important role to play in augmenting traditional cognitive assessments aimed at diagnosing learning disorders and difficulties” (Lipnevich, MacCann, & Roberts, 2013, p. 14). They can also be used in selection for college, graduate school, preparatory school, gifted classes, and honors or advanced placement tracks. Because of the potential for response distortion, non-cognitive assessments should be used in conjunction with other key factors.
 
Non-cognitive assessments can also be used to track students’ development over time. Schools could track trends at specific levels. District, state or national comparisons allow evidence for policy decisions in education.
 
“One of the strengths of using non-cognitive tests as developmental scales is the potential for interventions or training” (Lipnevich, MacCann, & Roberts, p. 15). Large-scale assessments can be considered while developing policy. Additionally, scores may serve as tailored feedback providing suggestions for action plans.
 
REFERENCE
Lipnevich, A.A., MacCann, C., and Roberts, R. (2013). Assessing non-cognitive constructs in Education: A review of traditional and innovative approaches. Oxford Handbook of Child Psychological Assessment
 
INSIGHTS
My proposal presentation is an international-data-supported policy recommendation for a California principal to use a non-cognitive assessment to obtain information that will allow the school to provide tailored support for students in order to improve their academic performance. Comparing the scores of those students with other demographically similar students will provide data that could be used to justify the decision to use the non-cognitive assessment throughout the school. Those results can then be presented to legislators to lobby for state-wide reform in academic policy and practices.
https://www.linkedin.com/pulse/improving-academic-achievement-developing-skills-k-erica-eberhardt/
​​