OECD PISA 2006 (Programme for International Student Assessment)
Research identification sheet ID: DAYR 016
Author of research: OECD, via the consortium lead by the Australian Council for Educational Research (ACER), expert groups for particular areas (reading literacy - REG, mathematics literacy - MEG, science literacy - SEG), national coordination centres and national project coordinators, as well as Governing Board for PISA study (PGB)
Data collection period: spring 2005 (field trial), 13. - 24. 3. 2006 (main study)
Data collection: ŠPÚ (ŠPÚ was responsible for adaptation and translation of tools: tests, questionnaires, manuals for school coordinators and administrators, instructions for study administration and evaluation, sampling in cooperation with ÚIPŠ and ACER, data entry procedures, making databases and outcomes on the national level)
ŠPÚ research team: Paulína Koršňáková, PhD. (OECD PISA 2006 National Project Manager, SVK representatives in PGB and SEG 2006 member), Ing. Martin Swan and Mgr. Jana Kováčová (data entry managers, responsible for statistical outcomes on the national level, Mr. Swan until the year 2006, Mrs. Kováčová from the year 2007)
57 countries over the world participated in OECD study PISA 2006: all 30 OECD member countries and 27 other, partner countries. Slovak Republic took part in the OECD study PISA for the second time. PISA tasks are not derived from the curricula and they were not aimed to find out, how students in respective country mastered the content-based knowledge. PISA tasks were developed according to the model of lifelong learning to assess three domains of students´ literacy – reading, mathematical and scientific literacy (the core of the PISA 2006 study). In the OECD PISA study, the literacy means ability to use knowledge and competencies gained during the compulsory education as well as ability to present own ideas and opinions. The main survey of the PISA 2006 study was held from 13th to 24th of March 2006 in all Slovakian regions. More than 4731 students (born in the 1990 year) from 189 schools (ISCED 2 and ISCED 3, out of them 13 with the Hungarian language of instruction) took part in the main survey. Representative and stratificated selection of schools and students from the Slovak Republic offers a possibility to generalise obtained results within the Slovak educational system and to compare them with the results of the other participating countries. The study results on the national level are presented in national report PISA 2006 – Slovensko available on ŠPÚ web site ( PISA Slovensko 2006 - Národná správa ).
PISA field trial, organized one year ago in May 2005, involved 1800 pupils from 30 schools. That survey was focused on verification of the testing tools like test items, tests, questionnaires, with the aim to obtaining all types of students´ responses and improve instructions on their evaluation.
1. Sample Construction:
Required target population for the OECD study PISA consists in each country from 15-year-old students (more precise from the 15 years and three months to the 16 years and two months old students) attending at least the 7 grade (counted from the beginning of the school attendance).
Required target population is defined to provide a picture of educational system results at the end of compulsory school attendance. Definition of 15-year-old population depended also on the date of survey. In Slovakia was the main survey administrated within the period from 1 March to 31 May and the students born in 1990 were tested.
It was two-level stratified selection.
Explicit variables were: region (8 regions), type of school (basic school, secondary school – general, secondary school - other), and school size -according to a number of students born in 1989 (large - 35 and more, small - 18 - 34, very small up to 17).
Implicit variables were: language (Slovak language, Hungarian language), school subtype (basic school, special basic school, 8-year grammar school, 4-year (secondary) grammar school, secondary technical school, secondary vocational school, united secondary school).
Based on these variables 26 strata were created (PISA_STRAT)
01 - basic schools in Bratislava Region (large)
02 - secondary schools general in Bratislava Region (large)
03 - secondary schools other in Bratislava Region (large)
04 - basic schools in Trnava Region (large)
05 - secondary schools general in Trnava Region (large)
06 - secondary schools other in Trnava Region (large)
22 - basic schools in Košice Region (large)
23 - secondary schools general in Košice Region (large)
24 - secondary schools other in Košice Region (large)
25 - small schools from all regions
26 - very small schools from all regions
Certain groups of schools and students could have been excluded from the target population; however criteria for exclusion were very strict. According the technical standards of the OECD PISA it was possible to exclude the whole schools all together attended by maximum of 2.5% students and another 2.5% students could be excluded within schools. The whole schools exclusion was possible from the technical reasons (0.5%) – logistically, and/or financial too demanding survey administration. In the Slovak Republic were in such case excluded the schools with two different languages of instructions: Ukrainian language of instruction (8 schools – 38 students) and Bulgarian language of instruction (2 schools – 19 students). The PISA survey administration in those schools would require tests and questionnaires translated into Ukrainian, resp. Bulgarian language. Another 2% of students could have been excluded via exclusions of whole schools attended EXCLUSIVELY by students with special educational needs (severely challenged students). In the case of Slovak Republic some special basic and vocational schools were excluded (all together 232 schools – 1 187 students). As for the within school exclusions of students, it was allowed to exclude certain students with special needs or limited knowledge of the language of instruction. The total target population was set on 75 883 students, which was 96.75 percents from the whole 15-year-olds population. To define the sample for the Slovak Republic this number was used. Schools were ranked according to their size, proportionally towards the whole population. The defined number of sampled schools was 193 schools with 5255 students.
To get an image of the target population (population of 15-year-old students in Slovakia) we used data provided by ÚIPŠ. It was a database of all basic (ISCED 2) and secondary (ISCED 3) schools together with number of 15-year-old students in the school year 2004/2005 (actually 1 year older than the target population).
To be successful in administration of the OECD survey PISA, 85 percent from sampled schools had to participate. 35 students were sampled in each of those schools (in the case of the small and very small schools all students were sampled if they attended at least the grade 7). The participation of 80 percent of chosen students was required.
4731 students from 189 schools participated in the main survey of the OECD study PISA 2006.
2. Methodology of Data Collection:
14 test booklets were administered in the main survey of the OECD study PISA 2006 to assess the reading, mathematical and scientific literacy. Tests were administered according the given procedure in the language of instructions by trained teachers (school associates – school coordinators and administrators). Each student had 120 minutes to solve his/her test and worked on four groups of units. In the year 2006, from two to three of these groups assessed scientific literacy; some units contained also items focusing on students´ opinions (they were not contributing to the final test score). The other groups contained units assessing reading and mathematical literacy of student. The PISA test contained items requiring open constructed responses as well as multiple choices or complex multiple choices. Items within one unit were related to a stimulus (a text and/or graphics).
After finishing tests students filled out also questionnaires (45 minutes) focused on students´ background, interests and motivation in the field of science education. ICT questionnaire was the last part of the Student Questionnaire.
Directors of participating schools filled out School Questionnaire focused on the context of education and school infrastructure.
Outcomes of the SLOVAK SCHOOL SYSTEM ARE statistically significantly UNDER THE AVERAGE of OECD in all of the three domains: reading, mathematical and also scientific literacy.
The average score achieved by Slovakia in the year 2006 was 466 points (level 2 out of five possible levels). Compared to the year 2003 was the score of Slovakia 3 points lower (469) in the year 2006, but the overall achievement did not change statistically significantly (also the OECD average was 6 points lower). The final ranking of Slovakia was almost unchanged:
from 23rd to 26th position among 30 OECD member countries,
from 31st to 35th position among all 57 participated countries (except of the USA, because in that country there occurred some technical problems related to the administration of the reading domain).
Note: In the year 2003 the Slovak Republic ranked between the 29th to 31st places, but just 41 countries participated in the OECD study PISA 2003 (data of 40 countries were proceeded, the Great Britain did not fulfilled the standard requirement for the minimum sampled school participation).
The average score achieved by Slovakia in the year 2006 was 492 points (level 3 out of six possible levels), just a few points bellow the OECD average (498 points in the year 2006), but however the difference was statistically significant. The score of Slovakia in the year 2003 was 498 points, which was enough to achieve the OECD average (500 points in the year 2003). The final ranking of Slovakia was almost unchanged:
from 17th to 23rd position among 30 OECD member countries,
from 23rd to 30th position among all 57 participated countries.
Note: In the year 2003 the Slovak Republic ranked between the 18th to 25th places, but just 41 countries participated in the OECD study PISA 2003 (data of 40 countries were proceeded, the Great Britain did not fulfilled the standard requirement for the minimum sampled school participation).
The average score achieved by Slovakia in the year 2006 was 488 points (level 3 out of six possible levels). As the scientific literacy was the main domain in the year 2006, the framework of the scientific literacy was elaborated and changed; it is not possible to compare the averages for the 2003 and 2006 years. The new group of items assessing the knowledge about science was introduced (items related to the scientific enquiry and scientific explanations). This influenced significantly the achievement of the Slovak Republic (under the OECD average, 500 points in the year 2006). There is a final ranking position of Slovakia:
from 20th to 25th position among 30 OECD member countries,
from 26th to 34th position among all 57 participated countries.
Socio-economic background in Slovakia
The socio-economic background of students was measured by an ESCS index in the OECD study PISA 2006. (The index was constructed on the bases of parents working positions, highest educational level achieved by mother and/or father and household possessions). In each country participated in the study PISA 2006 there was a higher difference (variability) among its students as between the most and less achieving countries. In each OECD country in average 33% of this difference (variability) could be contributed to the between schools variability. The between school variability contribution was much higher in Slovakia – almost 41%.
Like in the year 2003, also in the year 2006 was found a significant influence of students´ socio-economic background to their test achievement. In Slovakia this influence was statistically significantly higher compared to the average of the OECD. The score difference in achievement of Slovak students with better socio-economic background was 45 points for one unit of ESCS (the OECD average was 40 points).
Information and Communication Technologies in Slovakia
IT availability and the exploitation of IT significantly improved. Schools are much better equipped as in the year 2003. Significant increase occurred mostly in the case of Slovak basic schools, where the average number of the PC doubled (PISA 2003: 12 PC per 1 Basic School, PISA 2006: 24 PC per 1 Basic School). An increase of a PC use by students in schools is also an important finding (the use of PCs for the communication and the work on Internet was significantly increased).
The recommendations for the reading and mathematical literacy improvements formulated in relation to the OECD study PISA 2003 results analyses are still valid (for details see DAVM 009, even more is available in Slovak language in an electronic version of the PISA thematic reports focused on mathematical and reading literacy).
5. Data Processing:
The data obtained from questionnaires are presented in ziacky2006.sav and skolsky2006.sav. To use the students´ data, it is necessary to weight data with the final student weights (w_fstuwt). Except the final student weights there are also 80 replicate weights, which serve for computation of standard errors. This model is based on two-stage sampling and uses Fay’s modification of the BBR (Balanced Repeated Replication).
Cognitive data obtained from test booklets are presented in the file skognitiv2006.sav (scored). Items included in assessment had different difficulties. In survey OECD/PISA there is used so called Item response theory (IRT) for estimation the difficulty and competency. IRT is a mathematical model which provides the estimation of probability, that the answer of a student to a particular question is correct. This model creates a continuum (so called Rasch scale) on which both student performance and item difficulty will be located and a probabilistic function links these two components. On the basis of Rasch model five plausible values (PV1 – PV5) were attached to each student for each domain separately (they are included in the file ziacky.sav). To analyze the students´ data, which are concerned to performance, it is necessary to use the five sets of PVs.
More detailed information related to the theoretical background is available in English language:
PISA 2003 SPSS data manual (manual for SPSS users)
PISA 2003 SAS data manual (manual for SAS users)
PISA 2006 results are available in/on:
PISA Slovensko 2006 - Národná správa (4.26 MB) (the National Report in the Slovak language)
PISA 2006 results (wide spectrum of information in English on the OECD web pages)
Data files for SVK in SPSS: