Research has shown that cognitive and noncognitive skills play an essential role in success in life, and large investments are made in training programs aimed at improving skills—US$1 billion a year by the World Bank alone. Development economists and policymakers are increasingly interested in measuring skills and personality to predict performance, improve targeting of beneficiaries, and reduce poverty. Rachid Laajaj and Karen Macours asked whether our measurement of these skills is working in developing countries.
Skills are complex and hard to measure. To capture them, psychometricians have developed scales (sets of questions) that have been tested and validated almost exclusively in developed country settings. Often these scales are then used without prior validation in developing country contexts.
To address this lack of validation, Laajaj and Macours conducted survey experiments to test a wide range of commonly used measures of cognitive, noncognitive, and technical skills on rural populations in Kenya and Colombia. They tested the reliability (consistency of answers over time) and validity of the scales, checking correlation between different measures of the same skill, or whether it predicts outcomes that are expected to be affected by the skill.
The team found that cognitive skills can be measured quite accurately, but commonly used scales of noncognitive and technical skills suffered from large measurement error. The measures of noncognitive skills are perhaps the most worrying because some measurement error tend to be systematic. For example, people with less education tend to have a stronger “agreement bias,” the tendency to agree more often, even to contradictory statements. Also, which enumerator administers the questionnaire can have a strong influence on answers.
Laajaj and Macours provide recommendations to be cautious about interpretations when there is no evidence of the reliability and validity of a skill measure in a given context, and to improve the measures. For example, to address the “agreement bias” issue, one needs to ask questions with opposite meaning in order to correct for that bias. To limit the consequences of the enumerator effect on conclusions from the analysis, they should be well balanced between different populations. Improving skills measurement is a must for better policies that improve skills and their contribution to development in more places around the world.
Read the study in the Journal of Human Resources: “Measuring Skills in Developing Countries,” by Rachid Laajaj and Karen Macours.
Rachid Laajaj is an associate professor at the Universidad de Los Andes. Karen Macours is a professor at Paris School of Economics and INRA researcher.