Paper presented at the United Nation Washington City-group
on disability measurement. Washington,
Feb 2002
LANGUAGE AND CULTURAL ISSUES IN SURVEYS OF THE EUROPEAN UNION
A short description and an example of fundamental problems and practical solutions
H. Van Oyen1, S. van Buuren2, J.
van den Berg3, J. Bonte3
1: Scientific Institute of Public Health, Brussels
2: TNO Prevention and Health, Leiden
3: Statistics on Health and Safety, Eurostat
1. Executive summary
At about fifty years ago the European Commission set up Eurostat, the statistical office responsible for the execution of the statistical programmes of the European Union (EU). The network of Eurostat and statistical authorities is called the European Statistical System (ESS). Within this framework arrangements and agreements are made on data collection, standards and instruments, analysis and reporting. It is a network of fruitful collaboration and mutual professional exchange and support.
When Eurostat was established (bearing another name) there were only six Member States (MS) using five different languages. With the enlargement of the EU -now there are 15 MS- Eurostat had to cope with increasing language difficulties and cultural differences. This became particularly important for social statistics, which in many cases are based on population surveys. Particularly during the last decade these statistics showed a great increase, both in domains covered and in the number of subjects per domain. The statistics are sometimes based on voluntary participation of MS, sometimes on a legal basis (compulsory). Especially when regulations (legal basis) are involved great care should given to definitions, classifications, questionnaires, etc, as they should reflect in different languages the same underlying concepts.
In some cases this proved to be nearly impossible as even the equivalent words in different languages do not cover the same underlying concept. This leads unavoidably to different answers and statistical results for the same definitions and/or classifications. Cultural and language differences are reflected in the data outcomes. So practical solutions had to be found. In EU it is more and more said that it is not always possible to establish standard questions, which could then, after straightforward translations, be applied in every culture and language of Europe (or the world).
Gradually a conceptual approach leading to operational approaches was developed which is becoming more and more accepted. First of all an agreement is made on a conceptual framework (conceptual level), next on reference standard instruments (input level) and finally national operational standards (input level) are developed and finally conversion and other methods (output level) are needed for analysing the results, both at national level for trendanalysis and at international level for comparing the data of the MS.
An example of this conceptual and operational approach is the module on disability included in the Labor Force Survey 2002 (LFS 2002).
2. Introduction
In the EU national health surveys are conducted at regular intervals or have been conducted at irregular times in 14 of the 15 MS. Eurostat started collecting these national data aiming at EU data on perceived health and related topics. So far data were collected for 12 items:
1 chronic conditions (general question)
2 self perceived health
3 long-term physical disability (specific items)
4 activity limitations/temporary disability
5 height and weight
6 current smoking
7 former smoking
8 consumption of alcohol
9 physical activity
10 in patient care (hospitalisation)
11 out patient care (GP, dentist, out patient specialists)
12 use of medicines
In this context the items 1 to 4 are particularly relevant for language and cultural influences. Questions on the items 6 to 9 have similar problems but experience from other statistical domains can provide analogue solutions. The items 10 to 12 will also show differences not bound to the language and culture but due to the different organisation of the health care delivery systems in the MS.
In the ESS the European Community Household Panel (ECHP) also contains a small module on health. It is a voluntary survey covering a range of issues on living conditions, e.g. on educations, professional activities, household. The ECHP is a fully harmonised EU wide survey conducted in every MS using the same questionnaire. The English questionnaire is the standard reference and the questions are translated in the national languages. So here the differences in questionnaires can no longer be a reason for strange differences in outcomes. However cultural differences in combination with language remains the major confounding factors in the results.
An example is the results on perceived health (Figure 1a) from the
ECHP. There are substantial differences
between some MS that can hardly be understood. It is unlikely that the
differences are so big as shown in the figures. Part of the difference can be
traced by small differences in questions, e.g. a four points scale for
perceived health instead of a recommended 5 points scale. However we are becoming also more and more
aware of the substantial cultural differences leading to difference answers on
the same questions or to pure language problems. For instance what to do if in
a language ‘health’ is a pure positive notion and can not be used in a question
about ‘less good health’; in such a case a word for word translation leads to
misunderstanding for the general public even if it well understood in technical
scientific circles. Another example is given on “severe disability” (severely
hampered in daily activities) (figure 1b).
Figure 1a – not available
Figure 1b – not available
3. The labour force survey
(LFS) as an example
The labour force survey is another fully harmonised EU wide survey. It is mainly designed to monitor labour force developments and the related social factors in the MS of the EU. It is based on a regulation, which means that it is compulsory for MS. Each MS has to collect every year data complying with a minimum set of survey methods and according to a specific list of variables and applying well defined classes (codes). Every year there is a possibility to add to the survey a module for data collection in more detail on a specific subject. In 2002 the module is on disability.
The regulation concerning the LFS module
on disability is in annex 1. This regulation is published in each official
language of the EU. It shows the list of items and variables and the codes to
be applied. The formulation of the questions is left to the MS. So here the standard list of items and codes (at
input level) to be applied is defined (compulsory), but in the regulation there are no descriptions of the variables,
no coding instructions and no standard
questionnaire.
The next step was to make a description of the concepts for each item. The descriptions were agreed by all MS in the appropriate working groups: see annex 2, explanatory notes (main author: Howard Meltzer of ONS-UK). These explanatory notes include the structure of the module, the rationale of the variables (descriptions and/or concept) and coding instructions, which were also agreed by all MS. They also have been translated in all official national languages of the MS. So there exists also a reference standard concept and coding instructions (at input level) agreed by all MS.
The next step left open was the design of
a reference standard questionnaire. Until now the design of the questionnaire,
which means also the wording, was left entirely to the MS. For the module on
disability it was proposed to provide MS with a reference questionnaire in English, which later was also translated
in all official languages. The purpose was to ease the work of the colleagues
in MS in charge of the LFS module and to avoid misunderstanding in translations
and/or wording, and also to help interviewers in better understanding there
work during the interview. The reference standard questionnaire is given in
annex 3 and it was accepted as a reference by MS.
In addition it is accepted that at national level the operational questionnaires need some adaptations in order to reflect the real concept and meaning of the agreed international reference standard. This leads to the so-called national operational standard.
It should be noted that it is important to have not only translations from the standards, (concepts, classifications, coding instructions and questionnaires) which are usually in English, to the other languages, but also back from the national language to English. This allows for checking the correctness of the translations into the national languages, but taking into account that some ‘deviations’ may be reflecting to a large extent cultural backgrounds.
4.
Conversion techniques (RC)
Conversion techniques are rather new and have great potential for overcoming problems in incomparability of outcomes of data based on different classifications and/or scalings (questions) many times reflecting cultural differences. This is even more valid for the next step of the conceptual framework as described in paragraph 3 above.
Response conversion (RC) is a novel statistical technique that assists in comparing data in several circumstances:
1.
items on the same topic having different formulations;
2.
new items replacing old ones, thereby causing a trend gap;
3.
items with identical formulations but subject to cultural response
biases.
RC was developed within the
framework of the Health Monitoring Program (HMP) of the European Commission,
and its introduction as a routine tool is currently being considered. The goal
of the HMP is to provide relevant and timely information about the health in
each Member State (MS). To avoid unnecessary duplication, the system has to be
fed by existing data that are collected by the individual MS. Incomparability of
information is a major problem in this context. Each MS has its own tradition
in collecting and processing data, and changing established ways of working is
not so easy.
The potential of RC was
demonstrated for walking and dressing disability. Van
Buuren et al. (2001a)
identified 81 different items for measuring walking disability, and 56
different items for measuring dressing disability. The items used different
question formulations, used different response categories, and were posed in
different languages. RC consists of two steps (Figure 2). The first step
involves the construction of a conversion key. This is a relatively complex
activity, but needs to be done only once. For walking and dressing disability,
a conversion key could be constructed that included nearly all of the available
items. In the second step, one uses the conversion key to convert prevalence
information from individual MS into a common scale. This step is simple, and
can be repeatedly done on a routine basis as new information arrives. In this
way, estimates of walking and dressing disability in different MS could be
derived on a common scale.
The reason why the technique works is that it systematically exploits
any overlap in existing information through a statistical model based on
well-established Item Response Theory. RC works in the majority of cases,
except when the linking information is based on an internal frame of reference.
An example is "perceived health", where the frame of reference is
subjective and internal. Under these circumstances it is not possible to
compare without the presence of an external calibration. However, most items as
used in the field of disability have an external frame of reference, for
example "distance" in walking disability. Subject to the normal
constraints of validity, such items can be directly compared by the
RC-approach. A project is currently on its way to identify what other forms of
disability could benefit from the technique. The annex 4 provides more
information about the technology (Van Buuren et al, 2001b).
The most important asset of RC is that it allows the expression of existing information onto a common
scale. The values on the common scale can subsequently be used to compare and
monitor health indicators of different countries. RC thus allows setting up a
health monitoring system without the need to drastically change established
ways of working.
Figure 2. Steps in Response
Conversion
5.
Conclusions
The EU has gained over the last decade, through the ESS and in close collaboration with its MS, a substantial experience in language and cultural issues related to the health domains in surveys. The make-up of the EU, currently 15 MS, implies different language and cultures. This forced the ESS to search for creative and practical solutions to deal with the language and cultural related bias on the inference of the data collected. The results of the different EU-wide organised surveys have important policy implications. Therefore confounding due to language and culture should be understood and minimised.
The experience of the ESS over time indicates that the problem of language and cultural bias has to be dealt with both at the data production (the input-side) and at the data manipulation (the output-side) as they are complementary.
At the input side there is the stepwise process based on 3 pillars and linking the conceptual and operational approach. The important conclusion is that the primary focus has to be given in the first step to the consensus on the ‘concept’. The linguistic concordance and cultural agreement have to be developed after the conceptual framework. Once the consensus is reached on the concept, the concept is translated into a reference instrument. When agreed upon, the reference instrument is adapted to language and culture and leads to the construction of operational instruments at the level of MS.
At the same time a substantial effort was done within the ESS to develop statistical techniques to solve the problem of linguistic and cultural bias within existing data. Tools such as the described Response Conversion were developed to take into account and to do a final adjustment to possible left-over language and cultural confounding.
Reference
Van Buuren S, Eyres S, Tennant A, Hopman (2001a). Response conversion: A new technology for comparing existing health information.
TNO report 2001.097. TNO Prevention and Health, Leiden
(http://www.iph.fgov.be/epidemio/disab/EDM2002_08.pdf).
Van Buuren S, Hopman-Rock M (2001b). Revision of the
ICIDH Severity of Disabilities Scale by data linking and item response theory. Statistics in Medicine, 20, 1061-1076.
Annex 1
COMMISSION REGULATION (EC) No 1566/2001
of 12 July 2001
implementing Council Regulation N°577/98 on the organisation of a labour force sample survey in the Community concerning the specification of the 2002 ad hoc module on employment of disabled people.
THE COMMISSION OF THE EUROPEAN COMMUNITIES,
Having regard to the Treaty establishing the European Community,
Having regard to Council Regulation n° 577/98 on the organization of a labour force sample survey in the Community[1], and in particular Article 4(2) thereof,
Whereas:
Commission Regulation (EC) n° 1626/2000[2] has drawn up a programme of ad hoc modules to the labour force survey covering years 2001 to 2004, which includes an ad hoc module on employment of disabled people.
In accordance with Article 4(2) of Regulation (EC) n° 577/98 the detailed list of information to be collected in an ad hoc module shall be drawn at least 12 months before the beginning of the reference period for that module.
There is a need for a comprehensive and comparable dataset on the labour market situation of people with disabilities, as referred to in the Council Resolution of 17 June 1999 on equal opportunities for people with disabilities[3] , which calls upon the Commission to work together with the Member States, in particular within the framework of the European Employment Guidelines and in accordance with the mainstreaming principle, to monitor and analyse the development of the employment of people with disabilities on the basis of comparable data.
The measures provided for in this Regulation are in accordance with the opinion delivered by the Statistical Programme Committee established by Council Decision 89/382/EEC, Euratom[4],
HAS ADOPTED THIS REGULATION:
Article 1
The detailed list of information to be collected in the year 2002 by the ad hoc module on employment of disabled people is laid down in the Annex to the present Regulation.
Article 2
This Regulation shall enter into force on the seventh day following its publication in the Official Journal of the European Communities.
This Regulation shall be binding in its entirety and directly applicable in all Member States.
Done at Brussels, […]
For the Commission
Pedro SOLBES MIRA
Member of the Commission