PISA and TIMSS are like apples and oranges

How successful are Australian students? Based on the results of the last two Programme for International Student Assessment (PISA) tests, Premier Bob Carr argues that our students are the best in the world.

The results of the first PISA test, sponsored by the OECD to test 15-year-old students in areas such as literacy, were released late 2001 and Australian students were ranked among the best performers.

The second round of the PISA test results were released early December last year. Out of the 41 countries in reading literacy Australia was ranked number 2; in scientific literacy we were ranked number 4; and in mathematics literacy Australian students came 5th.

Based on PISA, it appears that Premier Carr has got it right. Unfortunately, such is not the case. A week after the PISA results were released, the results of a second international test became public. Unlike the PISA test, the results of the Trends in Mathematics and Science Study (TIMSS), undertaken by the International Association for Evaluation of Educational Achievement (IEA), shows a much bleaker story. In fact, rather than ranking with the best, Australian students performed poorly, worse still, we were overtaken by a number of countries that we had outperformed in previous TIMSS tests.

Australian year 4 maths students were beaten by 15 other countries and 13 countries outperformed our year 8 maths students. Even worse, on comparing the results of the 2004 TIMSS tests with earlier TIMSS tests, Australia’s results remained static while students’ results in the USA, England and New Zealand dramatically improved.

Which test carries more weight? The first thing to note is that PISA adopts many of the progressive education fads such as fuzzy maths and whole language that have bedevilled Australian education. TIMSS, on the other hand, measures more traditional classroom content on the assumption that students need to master essential learning such as demonstrating an understanding of fractions and decimals and the relationship between them.

Similar to Australia’s progressive approach, PISA argues that learning should not be restricted to right and wrong answers where students are made to learn a set body of knowledge. Instead, education must deal with clichés like real-life problems and life-long learning.

As a result, as noted in the OECD’s Executive Summary of the 2000 PISA results, the traditional view of literacy is redefined, in the “edubabble” much loved by those committed to education fads, as:

Knowledge and skills that reflect the current changes in curricula, moving beyond the school-based approach towards the use of knowledge in everyday tasks and challenges.

Based on PISA, it appears that education in Australia is on the right track and that standards are high. A closer view of the PISA test shows that the opposite is the case as the test, when compared to TIMSS, is substandard and flawed.

Firstly, even though the reading test is about literacy, students are not penalised for faulty spelling, grammar and punctuation. If Australian students were corrected the majority would have failed.

To quote from the ACER’s Australian report: “Errors in spelling and grammar were not penalised in PISA … It was the exception rather than the rule in Australia to find a student response that was written in well-constructed sentences, with no spelling or grammatical error.”

While most employers and parents expect students in the real world to be able to spell correctly, the PISA approach is to accept answers like the following as correct and to give students full marks: “because before than it disapeared completly and at that time it reapeared (sic)”.

A second problem with PISA is the way the test adopts fuzzy maths in opposition to the more traditional approach. The PISA test ignores student ability to master mathematical operations and calculations, preferring to focus on problem solving and guesswork.

The results of fuzzy or real-world maths are clear to see. Not only do many of our universities now have remedial classes for first year science and maths students, but, as a visit to the local market demonstrates, most teenagers are incapable of mental arithmetic.

Unlike PISA, with its faddish approach, the TIMSS test focuses on what is taught in terms of essential mathematics and science content. TIMSS also identifies effective classrooms by analysing the characteristics of those systems that perform best. Effective classrooms are those where teachers actually teach, instead of “facilitating”, and where more time is spent on whole class work instead of students working individually or in groups.

Successful systems like Singapore and the Netherlands also have succinct syllabus documents that focus on essential learning, especially during the early years, and which give teachers a clear and easy to follow road map of what it to be taught. There is regular testing and the assumption is that students should only progress after they have mastered the set work. This approach is the opposite of what happens in most Australian classrooms.

NSW curriculum, for example, as noted by the Vinson Report, is cumbersome and bureaucratic and teachers complain that in attempting to cover so much ground students miss out on the basics. More traditional skills like mental arithmetic, rote learning times tables and mastering algorithms like long division are ignored in favour of calculators and problem solving and students are promoted from year to year without a clear sense of whether they have passed or failed.

In most English classrooms, phonics has given way to the whole language approach where, instead of learning the relationship between letters and sounds, students are taught to look and guess.

The result? According to the 1996 national literacy test, initiated by the Howard Government, approximately 30 per cent of primary students failed to meet the minimum standard considered essential if they were to cope with future learning.

Last year, a second survey carried out by the Australian Council for Educational Research, concluded that approximately one third of year 9 students lacked the literacy skills needed to cope with the demands of the senior school curriculum.

Significant for Australia is that countries like the USA and England that have improved their performance in TIMSS have dropped progressive fads in favour of a more academic approach. An approach based on teachers teaching, students knowing right and wrong answers and mastering the basics.