A review of European best practices on monitoring instruments (bijlage bij 28760,nr.99) - Hoofdinhoud
Inhoudsopgave
| Officiële titel | A review of European best practices on monitoring instruments (bijlage bij 28760,nr.99) |
|---|---|
| Documentdatum | 06-01-2020 |
| Publicatiedatum | 14-01-2020 |
| Nummer | 2020D00115 |
| Kenmerk | 28760, nr. 99 |
| Externe link | origineel bericht |
| Originele document in PDF |  |
Monitoring participation in adult learning
programmes
A review of European best practices on monitoring instruments
This report discusses the key issues to be addressed when setting up and using a
system to monitor participation in adult basic education. It focuses on the data
indicators that could be collected, possible tools for data collection, and the role of the
different stakeholders. It also considers the stage after data collection and reflects on
how to analyse, report, and act upon the information. For each question, it describes
the alternatives available, their strengths and weaknesses, as well as the choice made
by different European countries.
PUBE
2 |
1
Introduction
Improving the competences of low-skilled adults is crucial, yet their participation in adult learning education is lower than other groups. Individuals with low levels of basic skills are particularly
disadvantaged in life. For instance, they are more likely to become unemployed and have low earnings
(OECD, 2013 [1] ; OECD, 2016 [2] ). Low-skilled adults also have lower levels of trust, participate less actively in the democratic process and in community life, and experience worse health outcomes (OECD,
2016 [2] ). However, according to PIAAC data, low-skilled adults are less likely than medium and highskilled individuals to participate in adult learning by more than 20 percentage points (OECD, 2019 [3] ).
Monitoring participation in adult learning programmes constitutes a fundamental element of any
initiative aimed at engaging more individuals in adult learning. It is a necessary first step to evaluate
the effectiveness of the programmes, both in terms of overall reach and in terms of targeting. Participation figures can constitute an important tool for all the stakeholders involved in the non-formal
education sector, including training centres themselves, as this information may enable them to learn and reflect on the practices, and potentially develop adjustment measures for a better impact.
However, collecting such data can prove difficult. Indeed, the provision of basic skills training is, in
most countries, highly decentralized and fragmented. As a result, data collection is rarely harmonized,
even within a country. Regarding formal education, when training can be related to a national or international qualification structure and deliver official diplomas, the existence of enrolment or graduation
records makes it easier to compile participation statistics. However, when it comes to non-formal education, such data rarely exists. To put in place the appropriate infrastructure for data collection, the
different stakeholders must join forces, reflect on what information should be gathered, how to setup the data collection process in practice, and agree on the use of the data.
This report presents the different steps to set up, implement, and use a system to monitor participation in adult basic education. After a short introduction to the issue of participation in adult basic education in the
Netherlands, the first section discusses the type of data to be collected, and the level of details to be included. The second section describes the data collection process in practice, and details several
implementation aspects. Finally, the third section considers the stage after data collection and reflects on how to analyse, report, and act upon the information collected. For each question, the report describes
the alternatives available, their strengths and weaknesses, as well as the choices made by various EU countries. As the focus is on adult basic education, several programmes described in the report concern
non-formal education. However, best practices regarding participation in formal education are also included when systems for data collection have the potential to guide the reflection for the development
of a similar system for non-formal training.
| 3
2
Setting the scene
The Netherlands ranks among the top performers in the the Survey of Adult Skills (PIAAC) (OECD,
2017 [4] ) which measures adults’ proficiency in literacy, numeracy skills, and problem solving in
technology-rich environments. Yet, a significant share of adults still lag behind their peers. Figure 1
shows that 13% of adults in the Netherlands have low foundation skills (they score of Level 1 or below in the survey), i.e. can only understand simple life- and work-related texts and instructions. Looking at
literacy and numeracy together, data from the same survey suggest that the number of adults with scoring low in literacy, numeracy or both was over 1.7 million in 2012 (OECD, 2013 [1] ).
Figure 1. Adults with low skills levels in literacy
%
70
60
50
40
30
20
10
0
Note: Adults (25 to 65 year-olds) scoring at or below level 1 in literacy. Belgium refers to Flanders only, United
Kingdom to England only.
Source: PIAAC (2012, 2015).
Participation of the general population in adult education in the Netherlands is well above the OECD
average. Figure 2 shows that more than half of the adults has participated in formal or non-formal education and training over the past year. However, OECD (2017 [4] ) notes that the participation of lowskilled adults in formal education and non-formal learning is low and concerning.
4 |
Figure 2. Participation in formal or non-formal education, 25 to 65 year-olds
%
70
60
50
40
30
20
10
0
Note: Adults (25 to 65 year-olds) participating in formal or non-formal education. United Kingdom refers to England only.
Source: PIAAC (2012, 2015).
The Dutch government has developed several programmes to address these challenges, the most recent being Tel mee met Tall (Count on Skills). Created in 2016, this initiative aims at reducing the
number of native Dutch speakers with low levels of literacy. It relies on Language Hub or Language Point centres established by municipalities. However, given that not all centres nor all municipalities collect
information on participation, it is difficult to assess the actual scope of the programme and whether it reaches the intended target group. The development of a harmonized data collection system is
particularly complicated given the highly decentralized nature of the initiative.
Other countries have developed similar initiatives and faced similar challenges as to monitoring progress.
The programmes presented in this report are summarized below and details are provided in sections 3 to 5.
-
•In France, several interesting initiatives exist:
o The programme Compétences-Clés (Key Competences) was created in 2009
and brought together different schemes (social integration and fight against illiteracy (IRILL), workshops of personalized pedagogy (APP) and initiation to the use of Internet (NSI)) that existed independently until then. In the context of this programme, participation data was collected via the extranet Rosace.
o The Regional Database for Vocational Trainees (Base régionalisée des stagiaires de la formation professionnelle – Brest) was created in 2003 to gather
information on job seekers starting a vocational training, when they receive a salary or are covered by Social Security during the training.
o The platform Agora, is currently in development and aims at gathering all the
data concerning adult education.
| 5
-
•In Slovenia, until 2016, in order to measure the extent of further non-formal education, a survey (ŠOL-NAD) was sent to all identified providers of non-formal adult education and
training (excluding the public administration), such as adult education centres, schoolbased units, company-based units, NGOs and others. However, this system suffered from a number of challenges related to under-coverage of education providers and lack of harmonization with other databases on adult education. Created in 2011 to overcome these issues, the CEUVIZ database (Central Register of Participants in Education) contains information on participation in all publicly recognized programmes, including for adult learners.
-
•In Ireland, the Education and Training Boards provide adult literacy and numeracy
programmes. Each local service offers a variety of different programmes such as family learning, English to Speakers of Other Languages (ESOL), workplace basic education, Intensive Tuition in Adult Basic Education (ITABE). Some of these programmes are
accredited at levels 1 – 5 on the National Framework of Qualifications; others are nonaccredited.
Participation in these different programmes is monitored. More specifically, the Funding Allocations Requests Reporting (FARR) system collects participation data and is part of the Program and Learner Support System (PLSS) that includes a National Course Database, a Course Calendar Scheduling System, and a Learner Database.
-
•In Sweden, municipalities offer adult education (Komvux) in the form of courses at the
basic and upper secondary level. The Komvuxdatabasen contains statistics on
participation in these courses. Adult education is also offered in the form of liberal
education, in folk high schools 1 and study associations 2 .
-
•Denmark has a long tradition of collecting data on publicly regulated education (Jensen
and Rasmussen, 2011 [5] ). Participation data for several programmes started as early as
in the 1970’s (for example attendance of cooking and textile schools, or attendance of
adult vocational programmes). Concerning general adult education, data collection
started in the 1980’s (Statistics Denmark, 2018 [6] ). The database for Adult Education
and Continuing Training contains data for part-time adult education. The Student database contains information for all full time education, including but not restricted to, adult education. No data exists at the national level for private education.
-
•In Norway, the Skills Plus programme aims at improving basics skills for the working
life. 3 The programme relies on employers, as they have to apply for funding for their
employees’ training. Training providers include study associations, and public or private
suppliers. A database has been created for monitoring purposes.
1 Folk high schools offer non-formal adult education and have a long background in Scandinavia. The first originated
in Denmark in the 19 th century. Usually, there are no entrance qualifications, grades, or leaving examinations.
Courses are offered in vocational training, physical education, foreign language, the arts, and subjects of general
interest in literature and social science, and may last from several weeks to one year. In Sweden, there are
approximately 150 folk high schools; two-third are are run by various popular movements, organizations and
associations (NGOs), while the remaining third are run by county councils or regions. Each folk high school develops
the content and direction of their own courses. There are two types of courses: general or specific. General courses
are an alternative to municipal adult education (komvux) and can lead to studies at a university or university college.
They are comparable to secondary school or upper secondary school. After the completion of each course, the
student receives a student assessment (studieomdöme). Specific courses are offered areas such as music, art or
recreational leadership. At the end of the course, students receive a course certificate.
2 Study associations are the second component of the Swedish liberal adult education. The ten study associations in
Sweden organize study circles around different subjects, such as artistic subjects, languages and social studies.
3 In 2006, the Norwegian government established a national program called Basic Competence in Working Life
(BCWL; Program for basiskompetanse i arbeidslivet (BKA)). In 2015, the Norwegian Government extended the basic
skills training to other target groups with a similar program for the voluntary sector (Basiskompetanse i frivilligheten
(BKF)). From 2016, the two programmes were merged into one program titled SkillsPlus.
6 |
-
•Finland also collects statistics on the education system as a whole, including for liberal
(non-formal) adult education. Figures are available on the website Vipunen 4 .
| 7
3
Deciding on what indicators to collect
Number of participants vs number of participations
In most initiatives, the unit of observation or counting unit is actually not the participant itself but the
participation. When this is the case, an individual enrolled in more than one course will be recorded or counted several times. Unless participation to a given programme precludes enrolment in another programme, adding up participation in different programmes does not necessarily yield a total number of
participants as individuals. Recording numbers of participation instances instead of numbers of
participants tends to inflate participation numbers artificially if the issue is not properly acknowledged or taken into account.
O In France, the Regional Database for Vocational Trainees (Base régionalisée des stagiaires de la formation professionnelle – Brest) gathers information on job
seekers starting a vocational training course, when they receive a salary or are covered by Social Security for the duration of the training. Data is collected from
the organization that pays the participant ’s salary (not the training fees). Data
providers include the Public Employment Services (PES), regional councils, and other state organizations. One caveat of this strategy is that the unit of observation of the resulting database is not an individual but a participation case, as the different data providers do not have a harmonized system for identifying individuals. Furthermore, some participation cases (those that do not give right to a remuneration or social security protection and those not officially declared to the unemployment services by job seekers) are not recorded in the database.
O In Ireland, the Education and Training Boards transmit information on
participation thanks to the Funding Allocations Requests Reporting (FARR) system. Data is broken down by type of program attended, and by type of accreditation.
O In Slovenia, data collected by the survey ŠOL-NAD included the number of
participation instances in continuing education by region, type of programme (recognised and non-recognised, field of education) and gender; programmes by region, type and length; and providers by region, municipality, number of employees and their education. As continuing education was not monitored at the individual level, it was not possible to identify duplicates and accurately estimate the share of adults who participate in training in a given period (Savarin, 2016 [7] ). Nowadays, the CEUVIZ database allows for the identification of participants.
In order to be able to count participants instead of only participation cases, a unique identifier should be attributed to or collected from individuals at the time of their first registration in the centre. Another
question concerns the possibility to identify individuals across training centres. Some individuals may attend training sessions in different centres, especially as they may relocate. To be able to identify these
individuals, it would be necessary to harmonize the attribution of unique identifier across training centres, or use a pre-existing national identification number. While the latter solution has several advantages
(such as the possibility to link participation data with other databases, see Section 5), asking for this
8 |
personal information may constitute an important barrier to participation for participants who value anonymity when visiting learning centres.
O In Sweden, adult education is offered by municipalities (Komvux) in the form of
courses at the basic and upper secondary level. Participation statistics are collected both at the participant level and at the participation level. Individuals that have participated to more than one course are identified thanks to their personal identity number (personnummer). The number of participation instances is thus higher than the number of participants. Statistics on course completion
are presented at the participation level, while statis tics on individuals’
characteristics such as age are presented at the participant level.
A second challenge concerns the possibility of complementing the information on the number of participants with data on the number of hours of training undertaken. Indeed, a short introductory
program lasting only a couple of hours is by no means equivalent to a comprehensive course spanning several months. To get a finer picture of adult learning programmes, it is important to count both the number of participants as well as the number of hours for each participant. Few European countries,
however, monitor hours of training.
O An exception is Denmark, where the unit of observation can be a participant or a
participation instance, and where number of hours are collected for part-time education. Denmark then computes the number of full-time equivalents to ease comparability across programmes. The Danish authorities also record information on the different course modules when relevant.
Information on interest, registration, attendance, drop-out, and completion
When recording the number of participants in adult learning programmes, it can be interesting to make a distinction between the number of interested individuals, the number of registered participants, and the
number of those that complete the course. Figures on number of interested individuals can provide information on the general awareness among the target population and the attractiveness of the courses
offered. This is particularly meaningful since this information is scarce in general and the limited available evidence on the effectiveness of awareness campaigns suggests that while they usually raise the
general population’s knowledge of the programmes, they are less successful in reaching those with low
skills (European Commission/EACEA/Eurydice, 2015 [8] ; OECD, 2019 [9] ).
Furthermore, comparing figures on registered participants and course completers can be useful to determine whether dropout or critical delays in completion are major issues. Indeed, while dropout may
be the result of someone starting a new job or identifying a more suitable training option, an important number of dropouts in a given programme usually signals low quality. However, it is not clear where to
set the threshold for an acceptable number of dropouts. This would require a thorough analysis of benchmarks in the sector.
Similarly, attendance rates and patterns of attendance may help policy-makers and institutions identify current weaknesses and anticipate potential future problems, or provide benchmarks for quality
assurance purposes. However, monitoring attendance in addition to registrations will demand more efforts to training centres and teachers. Furthermore, it is not clear how to monitor attendance of drop-in
courses.
O In Slovenia, the CEUVIZ database (Central Register of Participants in
Education) contains information on participation in publicly recognized programmes, including for adult learners. It contains enrolment information such as information on institution, programme, grade, mode of education (full-time or part-time), date of enrolment and completion.
| 9
O In Ireland, the FARR system records the number of learners enrolled, the
number of early leavers (i.e. leaving the course prematurely), the number of course completers (distinguishing between different type of certification obtained), as well as information on course completers progression.
O In Sweden, the Komvuxdatabasen for municipal adult education provision
contains information on number of participants that have completed, canceled, or continue their education. It also contains information on grades. The information for each year is provided by subject, level, and municipality.
O In Denmark, the database for Adult Education and Continuing Training contains
a variable indicating if the course is completed and thus allows identifying dropouts.
Detailed participants’ characteristics and level of aggregation
In addition to recording participation information, several countries also collect individual characteristics
such as basic demographic information (age, gender), and socio-economic status (education and skills, and employment status, etc.). Having access to detailed information on participants is crucial to
understand who the people attending the courses are, and whether the objectives of the programme in terms of targeting are achieved. The type of demographic and socio-economic variables to collect thus
also depends on the objectives of the programme in terms of targeting.
Furthermore, by comparing the population attending the programme and the general population, it is
possible to infer whether one group of the population is under-represented, and thus gain insights on potential barriers to participation. For instance, research based on survey data shows that several
individual characteristics, including age, level of skills, wage, and to a lesser extent gender, are correlated with low participation in such programmes (OECD, 2019 [3] ).
O In France, the database created in the context of the programme Compétences Clés (Key Competences) included participants’ details such as age categories,
gender, education levels, socio-professional status, employment status, disability status, and other information indicating whether individuals benefited from welfare payments or lived in a priority area. Motivations to attend the program, as well as plans for the future were also elicited.
The database Brest for vocational training of job seekers also includes
’ demographic and socio-economic variables. However, the
participants information comes from several data sources with different definitions and coverage. The comparability and use of such information is therefore limited. This example underscores the importance of communicating a precise definition
of each variable that will be collected, not only regarding participants’
characteristics but also for information on attendance, dropout, completion, etc.
O In Ireland, the Program and Learner Support System (PLSS) includes a National
Course Database, a Course Calendar Scheduling System, and a Learner
Database. The later contains participants’ characteristics such as gender, birth
date, nationality, employment status, and whether the individual receives welfare benefits.
O In Sweden , the database for Adult Education contains individuals’ characteristics
such as age, education, gender, country of birth.
O In Norway, the database created to monitor the programme Skills Plus contains
information on participants such as gender, formal education, but also industry of their employer.
O The website Vipunen contains statistical data on Finland's education system as
a whole. Available indicators include information on applications, admissions,
10 |
pass rates, time to complete a degree or qualification, class size. However, only some of these indicators are relevant for adult basic education systems. Regarding liberal (non-formal) adult education, available statistics are the number of students and number of teaching hours compared to population over 25 years old. Data includes detailed personal characteristics of participants such as gender, place of residence, age group, education level, income group, main type of activity, mother tongue, municipality of education, nationality.
The question of whether and what individual characteristics will be collected is closely related to the
issue of aggregation. If participants’ detailed characteristics are collected (such as gender, age, or socioeconomic
background), the level of observation will be the participant or the participation and learning
centres will have to collect and transmit microdata. On the contrary, if only the total number of participants or participations (as well as dropouts and completers) are of interest, centres may provide
directly aggregated figures, by type of programme. Collecting comprehensive details on participants will enable to draw a more precise picture of the sector. However, this may come at the cost of an increased
burden for training centres, possibly diverting them from their first mission of improving adults’ skills and
labour market outcomes.
Table 1 summarizes the different initiatives presented in this section and the type of data that is collected in each case. One of the most comprehensive system in terms of details on participations and
participants is the PLSS service implemented in Ireland in 2016. Indeed, the system allows the identification of participants (as opposed to participation cases only), the number of hours, course
completion, as well as detailed participants characteristics, such as gender, birth date, nationality, employment status, and other socio-economic variables, when most of the other initiatives collect only
some of these variables. However, when new systems are put in place, they usually include as many details on courses and participants as possible. In particular, most countries record information on
dropout and course completion. To collect information on participants’ characteristics, two different
strategies have been developed: a number of countries directly collect the variables at the time of
registration in training (Ireland), while others rely on the national identification number of participants (Slovenia, Denmark).
Table 1. Type of information used to monitor participation in basic adult education
Country Name of the initiative or Possibility to identify Number of Information Detailed participants’ database participants hours on course characteristics
completion France Database Brest No Yes No Yes, but limited use Slovenia Database CEUVIZ Yes - Yes No but possibility to collect
additional participants’ characteristics
Ireland PLSS Yes Yes Yes Yes
Sweden Komvuxdatabasen Yes No Yes Yes
Denmark Database for Adult Education - Yes Yes No but possibility to collect
and Training additional participants’
characteristics
Norway Database for Skills Plus - - - Yes
Finland Website Vipunen - Yes Yes Yes
| 11
4
Setting-up the data collection
process
Data collection methods
As far as data collection is concerned, a number of choice need to be made as to: i) who is responsible
for providing the data (e.g. training providers vs training participants); ii) in what format the data is transmitted (e.g. paper or electronic format); iii) the frequency of data collection. These issues are
addressed below.
Responsibility for data provision
A key issue to be addressed when collecting participation data concerns who will be responsible for recording the information. When registering for a training course, participants may be asked to fill in a
form to provide personal details. However, for candidates to courses in basic literacy skills, having to fill in a form prior to the training could deter them from applying or registering to a course and may represent
a major obstacle to their participation. Data reliability would be maximised if training centres could be in charge of data inputting. This would also simplify the data collection process given that training centres
would be in charge of recording information on the training programme in which the person is enrolled, number of hours of training attended, or course completion.
Format of data transmission
A related question arises on the type of support used for data collection. If training participants are asked
to fill in the form themselves, it might be easier to use paper forms, as their digital skills are often also low. However, this means that training centres will have to bear the cost of the data digitization process.
If training centres input the data themselves, and if digital tools are available, they are usually preferred to avoid typesetting errors and costs related to transcription.
O In France, for the programme Compétences-Clés, a particular platform was
developed (the extranet Rosace). It was used in the first instance by professionals prescribing training to low skilled individuals to refer them to a particular training centre and programme. Training providers also inputted information in the system. Participants were thus never asked to provide the information themselves and paper forms were not used.
O In Slovenia, until 2016, the ŠOL-NAD questionnaire was sent to all identified
providers of non-formal adult education and training (excluding the public administration), such as adult education centres, school-based units, companybased units, NGOs and others. Providers were identified on the basis of records of the Ministry of Education, Science, Culture and Sport, the Adult Education Centre of Slovenia and the Business Register of Slovenia according to the
12 |
standard classification of the principal activity. 800 providers received the survey in 2016. Since responses were not mandatory, only 517 providers returned the survey (OECD, 2018 [10] ). Training providers provided information about their activity using data taken from their own records of activities, operating income and costs. Paper questionnaires were sent by mail. However, this strategy led to under-coverage of training institutions, as data reporting was burdensome and voluntary. Nowadays, the CEUVIZ database gathers administrative data from institutions. Since reporting is mandatory, the coverage of CEUVIZ is much higher.
O In Ireland, the FARR system to track aggregate numbers of participations and
course completers is fully digitized. In contrast, detailed information on participants for the Learner Database is collected using a paper form filled by learners themselves. Training centres then digitize the information. Recently, an electronic version of the form has been made available to participants.
O In Sweden, each municipality can choose how to transmit their information to
Statistic Sweden. If the municipality has purchased services from an IT system provider that developed a special module for reporting statistics on adult education, they can transmit their information directly to Statistics Sweden. When there are changes in the information requirements, the statistical office communicates with the different data providers so that they modify the systems accordingly. The alternative is to fill in an Excel file provided by the statistical institute: the municipality needs to download the file and report the information manually. Statistics Sweden does not favour the latter solution as it is more prone to reporting error and poorer data quality.
O In Denmark, the statistical institute receives data on general and preparatory
adult education from the Ministry of Education. For other forms of adult education, such as courses in folk high schools, training providers send the data directly to the statistical office. 80% of folk high schools use an IT system, and 20% of them upload Excel files on a platform.
Frequency of data collection
Another issue relates to the frequency with which participation data is collected and entered into the
system. More frequent data collection allows for a more accurate and up-to-date picture of training participation and a better understanding of training trajectories. However, the more frequent the data
collection, the more costly and time-consuming it will be for training centres. The optimal frequency should strike the right balance between granularity and administrative cost and will also depend on the
use of the data, as discussed in the next section.
O In France, for the database Brest, the different providers transmit data every
month.
O In Ireland, the FARR system is open for data entry and editing four times a year:
at the beginning of the year for the planning period, in June to enter information
for the period January – May, in October for the period June - September period,
and in January of the next year the period October-December.
O In Denmark, data collection happens once a year.
O In Sweden, the National Agency for Education compiles and reports data once a
year. Since 2009, the statistics are published per calendar year instead of per academic year in order to better reflect the flexibility that exists in the municipal adult education.
O In Germany, the Alpha Monitor survey has been conducted every two years
since 2013.
| 13
Coordination of the monitoring system
This section presents the governance arrangements in different countries regarding the monitoring of adult learning programmes.
O In France, the programme Compétences-Clés was organized by the Ministry of
Employment and co-financed by the state and the European Social Fund. It was implemented by the Regional Directorates for Enterprises, Competition Policy, Consumer Affairs, Labour and Employment (DIRECCTE) through public procurement. Data collection was the responsibility of DARES (the Statistical Unit of the Ministry of Employment).
The Brest database for vocational training for job seekers is also managed by DARES and gathers information from several data providers such as the unemployment services, regional councils, and other organizations.
O In Slovenia, the Council of Experts for Adult Education is the body in charge of the monitoring and the evaluation of the Master Plan for Adult Education (2013 –
2020) (DIMA project, 2016 [11] ). Until the development of an harmonized system with the database CEUVIZ, various ministries and agencies separately collected and analysed the data on adult learning provided by a wide range of actors, including adult-learning providers and employers, resulting in high heterogeneity in data quality and the lack of comprehensive information. (OECD, 2018 [10] ). Nowadays, the CEUVIZ database is the responsibility of the Ministry of
Education, Science, and Sport ( MIZŠ).
O In Ireland, SOLAS and ETBI are in charge of the development of FARR and
PLSS databases. SOLAS is the state organization responsible for funding, planning and coordinating Further Education and Training in Ireland. Education and Training Boards Ireland (ETBI) is the national representative association for the sixteen Education and Training Boards. An Education and Training Board (ETB) is a statutory local education body that administers most adult education
and some secondary education. 5 One unit in SOLAS is responsible for the
development of the PLSS and FARR databases. The FARR system is used to support the Funding Allocations Request and thus to plan the allocation of funding and grants.
O Statistics Sweden handles the collection of data on behalf of the Swedish
national agency for education for formal education. For liberal adult education (folk high schools and study associations) Statistics Sweden is directly in charge of the official statistics.
O In Denmark, the Ministry of Education is responsible for data collection
regarding general and preparatory adult education, adult vocational programmes and vocational colleges. It then transmits the information to Statistics Denmark that compiles the data and creates the different databases (Adult Education and Continuing Training, and the Student database). Statistics Denmark receives directly data on course activity and participation from the other providers: folk high school, university colleges, academies of professional higher education and universities.
A Danish committee on education (Kontaktudvalgetet for Uddannelse) composed of users of the data, representatives from selected ministries, labour market organisations and NGOs in the education sector meets every year to discuss the statistics. This type of discussions where the different stakeholders have the chance to provide feedback on the data collection process and on the statistics
5 In 2013, they replaced the existing system of Vocational Education Committees (VECs) that had been in place
previously.
14 |
produced could prove instrumental to ensure the involvement of all stakeholders in the project.
O In Norway, funding for Skills Plus comes from the Ministry of Education and
Research. The agency responsible for the practical implementation of the programme and for the documentation of its results is Kompetanse Norge, the Norwegian Agency for Lifelong Learning, part of the Ministry of Education and Research. Data describing participation in adult learning in general is collected
and administered by various institutions, including Statistics Norway.
O In Finland, statistics published in Vipunen are based on data and registers
collected by Statistics Finland, the Ministry of Education and Culture, the Finnish National Agency for Education. Data on the cost and operation of municipal comprehensive school education (basic education) are gathered from municipalities. The Ministry of Education and Culture and the Finnish National Agency for Education are jointly responsible for the statistics presented in the
website.
Table 2 reviews the different arrangements for data collection. In most countries, the national statistical institute is one organization in charge of data collection. In some cases it is the main responsible body. Almost all initiatives rely on data inputted by training providers and not by participants themselves, with the exception of Ireland. Similarly, the vast majority use a fully digitized system for data transmission. The Swedish example is particularly noteworthy, in that it offers great flexibility to data providers (municipalities) regarding the format for data transmission
(municipality’s own platform or Excel files). The Slovenian experience is also particularly
insightful for the development of monitoring systems in other contexts, as they recently underwent an important reform for the collection of data on adult education, transitioning from a system where data was collected by various actors with little coordination to a comprehensive system that tracks participation in all publicly recognized education programmes.
| 15
Table 2. Details on the data collection process
Country Name of the Organizations in Who inputs the Frequency Paper form or initiative or charge of data information? electronic system database collection and only?
responsible body (in bold)
France Database Brest DARES Pôle Emploi (PES), Monthly Electronic system regional councils,
other organizations
Slovenia Database CEUVIZ Ministry of - - Electronic system Education, Science,
and Sport (MIZŠ) Ireland PLSS SOLAS and ETBI Training participant 4 times a year Online system for
for PLSS, centers for FARR. FARR Paper form and digitization by training centers for PLSS,
with a move towards the use of an
electronic form. Sweden Komvuxdatabasen Statistics Sweden Municipalities Once a year Municipalities’ own
system Denmark Database for Adult Ministry of Training providers: Once a year Electronic system
Education and Education and folk high school, Training Statistics Denmark university colleges, academies of
professional higher education and universities
Norway Database for Skills Kompetanse Norge - - - Plus and Statistics Norway
Finland Website Vipunen Statistics Finland, the Training providers - Electronic system Ministry of from their registers
Education and Culture, the Finnish National Agency for
Education
16 |
5
Making the most of the data
The process of monitoring adult learning participation does not stop at data collection and several issues
arise in the post-collection phase, including: data protection and data quality; data analysis, and publication of results; and the linking of the data on participation with other data sources.
Issues of data protection and quality
As soon as individual information is recorded, questions of data protection, confidentiality and security should be addressed. The issue will be particularly stringent if the data contains personal identifiable
information, and less important if the data is sufficiently aggregated. In any case, the publication or transmission of data should not permit the direct identification of individuals.
Regarding data quality, recommendations on organisation and management of quality can be found in
the Code of Practice for European Statistics. 6 Implementation guidelines are presented in the Quality Assurance Framework of the European Statistical System. 7 The principles of the Code of Practice of the
European Statistics are listed in Box 1 and the interested reader can refer to the two publications for
more details.
f729c75878c7 (Accessed 30th September 2019)
7 https://ec.europa.eu/eurostat/documents/64157/4392716/ESS-QAF-V1-2final.pdf/bbf5970c-1adf-46c8-afc3-
58ce177a0646 (Accessed 30th September 2019)
| 17
Box 1. The Principles of the Code of Practice of the European Statistics
The European Statistics Code of Practice lays down general principles developed to ensure a
common data quality framework for the European Statistical System and is used for the development,
production and dissemination of European Statistics. The different principles touch upon varied topics
such as the institutional environment, the statistical processes and the final output. They are listed below:
Institutional Environment
-
•Professional Independence
-
•Coordination and cooperation
-
•Mandate for Data Collection and Access to Data
-
•Adequacy of Resources
-
•Commitment to Quality
-
•Statistical Confidentiality and Data Protection
-
•Impartiality and Objectivity
Statistical processes
-
•Sound Methodology
-
•Appropriate Statistical Procedures
-
•Non-excessive Burden on Respondents
-
•Cost Effectiveness
Statistical output
-
•Relevance
-
•Accuracy and Reliability
-
•Timeliness and Punctuality
-
•Coherence and Comparability
-
•Accessibility and Clarity
More details on these principles and the indicators that can be used are presented in the Code of
Practice and in the implementation guidelines.
Source: https://ec.europa.eu/eurostat/documents/4031688/8971242/KS-02-18-142-EN-N.pdf/e7f85f07-91db
4312-8118-f729c75878c7 and https://ec.europa.eu/eurostat/documents/64157/4392716/ESS-QAF-V1-
2final.pdf/bbf5970c-1adf-46c8-afc3-58ce177a0646 (Accessed 30th September 2019)
In order to allow for meaningful analyses, it is also important to ensure that there will be as little missing
and erroneous information as possible. To this end, the use of electronic systems for data collection can prove instrumental. Indeed, with mandatory fields functions, the respondent will not be able to submit the
data without the necessary information being provided. Furthemore, it will also be easier to ensure that data is collected in the expected format, for instance by using drop down menus or some sort of input
validation for free-form writing.
Once data is collected, it is important to ensure its validity. To this end, data quality standards should be
set. For instance, it should be decided in advance which data to keep, which to get rid of and which to correct. Similarly, statistical offices or units usually agree on a plan for data correction.
18 |
o Statistics Denmark undertakes several steps to check and ensure data quality.
Statistics on the Danish population's participation in courses and adult are in general regarded as of high quality (Jensen and Rasmussen, 2011 [5] ; Savarin, 2016 [7] ). First, Statistics Denmark inspects missing and inconsistent data. For instance, data for each school or training centre is compared with past data for the same school and with current data for similar schools to detect anomalies. In case of missing or inconsistent data, Statistics Denmark may contact the course provider for explanation or correction of data. Other checks include making sure that only valid codes are used, start and end date for each course are consistent, etc.
Denmark has also established a Working Group on Quality and a central quality assurance function reporting the Working Group with suggestions for further improvements. The Working Group assess, decides and implements the suggestions as deemed necessary.
However, one caveat should be noted. Indeed, adult education curriculum have evolved substantially over time. While necessary to adapt to changing labour market needs, these modifications make comparability over time difficult.
Use of the data and publication of results
The value of data lies in its use and data on participation in adult learning courses has the potential to inform decision-making at all levels. For instance, training centres can use information on their own
participants for self-assessment. Furthermore, data analysis at the local and national levels can help better understand whether the intended targeting has been achieved.
O In France, the data produced with the extranet Rosace for the programme
Compétences Clés was used by the funding organizations for monitoring purposes (French state and European Social Fund). The aim was also to improve the prescription and supervision of the training. Data inputting in the system was seen an integral part of the services of the training organizations and was one of the conditions for their payment.
O Data collected for the database Brest is used to inform analyses of job seekers
training carried out by DARES.
O In Slovenia, results from the survey ŠOL-NAD were published in the SI-STAT
database of the Slovenian Statistical Office, and the data served as a basis for several publications of the Statistical Office. Nowadays, access to the CEUVIZ
data is restricted to the schools and the responsible Ministry ( MIZŠ). This
database is used for policy-making to follow key education goals and objectives, and allocate public funds and for research.
O The aim of the systems put in place in Ireland is to enable the production of
appropriate evidence regarding adult learning programmes, including proper counterfactual impact evaluation. At the request of training centres, the system can generate a range of reports that can be viewed in PDF, exported as a CSV or Excel file that can be used by training centres for self-assessment. Actually, the systems have been developed not only to monitor participation but also to help providers schedule courses and manage learners. Some statistics are also published in the FET Services Plan, available on SOLAS website.
O In Sweden, national statistics as well as figures by school are presented on the
National Agency for education website. 8
(accessed on 27 th August 2019)
| 19
O Statistics Denmark publishes the main figures online ( http://www.statbank.dk ).
Further statistics can be requested to Statistics Denmark's Costumers Centre. Micro-data is available for statistical analyses and research purposes via the Division of Research Services in Statistics Denmark. The data serves as a basis for several analyses published in Danish.
O In Norway, Kompetanse Norge publishes the most important statistics in an
annual report, The Vox Mirror. These and other figures can also be found online in the Skills Norway Statistics Bank.
O In Finland, the data is used to plan central government transfers to local
government for comprehensive school education (basic education), upper secondary education and polytechnic education. Statistics are also published
online on a dedicated website for education statistics. 9
Development of a fully integrated system and possibility to combine and link
data
Linking-in other data sources could potentially enrich the data. However, several issues arise. A first
issue concerns the existence of multiple data-gathering systems across different types of adult trainings (formal, non-formal, informal). Ideally, a fully integrated system can help collect data on participation in
different programmes.
Second, in order to fully exploit the potential of data on participation and participants in adult trainings, it
would be useful to link them with other data sources such as employment and earning records. This would necessitate collecting the national identification number of participants. However, participants may
not feel comfortable with disclosing this information and this may constitute an important barrier to participation. Furthermore, this would make the question of data protection even more important.
O In France, the platform Agora, currently in development, aims at gathering all the
data concerning adult education.
O In Slovenia, data for formal adult education (CEUVIZ) is fairly comprehensive. This database does not contain detailed participants’ characteristics but thanks
to a Personal Identification Number (PIN) it is possible to collect additional
participants’ characteristics such as gender, date and place of birth, address,
citizenship from the Central Residents Register.
O In Ireland, information collected for PLSS includes the Personal Public Service
Number of participants. This number, given by the Department of Employment Affairs and Social Protection, is a unique reference number that is needed for all dealings with public service agencies It is thus possible and planned to integrate information on participation with databases from Revenue, the Department of Employment and Social Protection, the Department of Education and Skills and others.
O In Sweden, the database collected in the context of municipal adult education
can be complemented with the Longitudinal integrated database for health insurance and labour market studies (LISA) to obtain information on individual characteristics, transfer payments and earnings.
O In Denmark, the database for Adult Education and Continuing Training includes
general and preparatory single courses such as Dyslexia Education (OBU, since the school year 2007/2008), Preparatory Adult Education (FVU, since the school year 2000/2001), General Adult Education (AVU, since the 1978/79 school year),
9 https://vipunen.fi/en-gb/ th (accessed on 27 August 2019)
20 |
Higher Preparatory Single Subject (since 1978/79), the Higher Preparatory Exam (since the school year 1991/1992). It also includes participation in folk high schools (including cooking and textile schools), Danish language courses for foreigners, courses for the adult vocational programmes (AMU), open education at universities, university colleges, business academies (former academies of professional higher education), and open education at vocational colleges. The different education registers (including for adult education) contain the CPR number of individuals, the unique personal identification number of Danish residents that allow them to access a broad range of public services. Thanks to
this number, it is possible to link information on individuals’ education across years, and to retrieve participants’ characteristics such as socio-economic
background.
Table 3 discusses the choices made in different countries in the post-collection phase.
Concerning the comprehensiveness of data on adult education, the possibility to link data with additional data sources, and the quality of the data, Denmark particularly stands out, followed closely by Sweden, Ireland, and Slovenia.
Table 3. Other challenges and ways ahead
Country Name of the Integrated system Possible to link Publicly available Data use initiative or data with other statistics
database data sources France Database Brest Integrated system in No No Data analysis by
development (Agora) DARES Slovenia Database CEUVIZ Yes Yes No Data used to follow
key education goals and objectives,
allocate public funds and to inform
scientific research. Ireland PLSS Yes Yes Yes Centres selfassessment,
planning of funding, impact evaluations
Sweden Komvuxdatabasen Yes Yes Yes - Denmark Database for Adult Yes Yes Yes Micro-data available
Education and for research purposes Training
Norway Database for Skills - - Yes Basis for statistics on Plus formal and non-formal education and training
Finland Website Vipunen Yes - Yes Data used to plan government transfers
| 21
6
Conclusions
Developing and implementing a monitoring system for participation in adult training presents a number of
challenges and open questions regarding 1) the type of information to collect, 2) the methodology and business process, and 3) the use of data that will be made and other questions arising after data
collection.
For the first point, it is important to define what participation exactly means: does it refer to the mere
interest or registration in a training programme, or does it imply sufficient attendance or programme completion? The unit of observation also needs to be discussed: will training centres provide aggregate
figures or transmit microdata? Should number of participation or participants be counted? Furthermore, if
detailed participants’ characteristics are of interest, it is possible to collect the information directly or to
retrieve the data from already existing databases. The data may also be complemented with information on the training programme, such as subject, content, number of hours.
Regarding the data collection process itself, the support and frequency, as well as its voluntary or compulsory nature, must be decided upon. Furthermore, a main responsible body may be designated. In
this case, it could be an already existing organization, or an entity newly created for this specific purpose. The potential role of the statistical institute needs to be discussed, more particularly given the need for
data protection and quality. Finally, it is also important to reflect in advance on the use of data after its collection, as the discussion has the potential to inform previous steps such as the type of information to
collect, and to think about the possible integration and harmonization of the monitoring system for nonformal education with other systems that could exist for formal education.
The present report has presented the choices made in different countries regarding these questions. The diversity of the initiatives show that there is no one-size-fit-all solution and that the development of a
monitoring system for participation should carefully take into account the peculiarities of the adult education sector in the country.
It is also important to remember that participation statistics should be used as a tool and not an end in itself. Participation statistics are only output indicators of adult education programmes. To get a better picture of the effectiveness of adult learning programmes, it is necessary to also measure outcome and
impact indicators, such as learning outcomes, and career development and social inclusion information.
Gathering information on outcomes and impact along output indicators is crucial to make sure that measuring participation does not distort the incentives of training providers away from the initial policy
goal. Indeed, greater or more targeted participation should not be the only objective, especially as it could come at the expense of training quality.
22 |
References
Aranđelović, D. (2017), Methodological Explanation: Basic Education For Youth And Adults, [12]
https://www.stat.si/StatWeb/File/DocSysFile/8340/09-126-ME.pdf (accessed on 2019 August 25).
DIMA project (2016), State of the Art in Adult Education Strategies, Policies, and Tools.. [11]
European Commission/EACEA/Eurydice (2015), Adult Education and Training in Europe: [8]
Widening Access to Learning Opportunities.
Jensen, V. and A. Rasmussen (2011), “Danish education registers”, Scandinavian Journal of [5]
Public Health, Vol. 39/7_suppl, pp. 91-94, http://dx.doi.org/10.1177/1403494810394715 .
OECD (2019), Getting Skills Right: Engaging low-skilled adults in learning, [9]
http://www.oecd.org/employment/emp/engaging-low-skilled-adults-2019.pdf .
OECD (2019), Getting Skills Right: Future-Ready Adult Learning Systems, Getting Skills Right, [3]
OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311756-en .
OECD (2018), Skills Strategy Implementation Guidance for Slovenia: Improving the [10]
Governance of Adult Learning, OECD Skills Studies, OECD Publishing, Paris,
https://dx.doi.org/10.1787/9789264308459-en .
OECD (2017), OECD Skills Strategy Diagnostic Report: The Netherlands 2017, OECD Skills [4]
Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264287655-en .
OECD (2016), Skills Matter: Further Results from the Survey of Adult Skills, OECD Skills [2]
Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264258051-en .
OECD (2013), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD [1]
Publishing, Paris, https://dx.doi.org/10.1787/9789264204256-en .
Savarin, A. (2016), Methodological Explanation: Continuing Education, Statistical Office of the [7]
republic of Slovenia, https://www.stat.si/StatWeb/File/DocSysFile/8339/09-030-ME.pdf (accessed on 25 August 2019).
Statistics Denmark (2018), Documentation of statistics for courses and Adult Education, [6]
https://www.dst.dk/en/Statistik/dokumentation/documentationofstatistics/courses-and-adulteducation (accessed on 25 August 2019).