Are International Educational Tests Good Accountability Tools?

incentives, education, School accountability, educational tests, TIMSS

Using TIMSS to Inform Policy and Practice at the Local Level

The Third International Mathematics and Science Study (TIMSS)-1995 and its successor, TIMSS-1999, provide education researchers, policymakers, and practitioners with rich, comparative data designed to help better understand the performance of educational systems. As such, TIMSS is a valuable tool in current efforts to improve mathematics and science instruction and to educate students in the United States to global standards of excellence. What can we learn from this ambitious and unprecedented international effort to provide meaningful, useful data for the reform of mathematics and science instruction? It is important to reflect on this question as we assess the promise and challenges of using TIMSS-type data in particular, as well as the broader national effort to use data to guide school improvement in general. The first in CPRE’s series of Policy Briefs about TIMSS (Dunson, 2000) looked at initial efforts to make use of TIMSS-1995 data. In this Policy Brief, we take a closer look at the ways in which TIMSS-1995 and TIMSS-1999 data have helped to inform changes in policy and practice as schools, districts, and states respond to the call for improvement in mathematics and science achievement. This Policy Brief was prepared to complement CPRE’s effort to address this question in a TIMSS Policy Forum, held in Washington, DC, in May 2002. This forum convened TIMSS Benchmarking jurisdiction representatives, teachers, administrators, policymakers, researchers, and technical assistance providers to share successful strategies and ongoing challenges in taking full advantage of TIMSS data. This Policy Brief is based primarily on data collected in structured interviews with Using TIMSS to Inform Policy and Practice at the Local Level By Deborah I. Nelson administrators and teachers in 10 TIMSS Benchmarking jurisdictions (referred to as “Benchmarkers”). These jurisdictions within the United States participated in the TIMSS- 1999 Benchmarking Study, committing their own resources and time in order to receive data from a representative sample of their own eighth-grade students. TIMSS Benchmarkers thus have international comparative data on their students’ achievement and their system variables. Our purposive sample included equal representation from states, districts, and consortia with a variety of demographic characteristics. While TIMSS data have been used nationwide, Benchmarking jurisdictions are notable for their existing commitment to reform of mathematics and science programs and for the fact that they have access to their own local TIMSS- 1999 data. This Brief is designed to facilitate networking and continued learning from TIMSS; it focuses on Benchmarkers’ experiences, but is relevant for anyone interested in using TIMSS to improve mathematics and science instruction. Strategies are reported in summary form. Actual TIMSS data and analysis are not discussed in detail, but related references are provided at the end of this Brief

ANALISIS SOAL PENILAIAN TENGAH SEMESTER (PTS) GENAP KELAS V SD BERDASARKAN PROPORSI DOMAIN KOGNITIF TRENDS INTERNATIONAL MATHEMATICS AND SCIENCES STUDY (TIMSS) : Penelitian Analisis Isi secara Kuantitatif pada Soal Penilaian Tengah Semester (PTS) di Kelas V Semester Genap SDN Ragunan 07 Pagi

Matematika dianggap sebagai mata pelajaran yang sulit untuk dipelajari dan diajarkan. Kurangnya pembangunan konsep yang baik pada siswa menjadi salah satu faktor matematika dirasa sulit, sehingga hal ini berdampak kepada hasil belajar siswa yang masih tergolong rendah, dan jika dilihat berdasarkan hasil TIMSS dari tahun ke tahun menunjukkan kemampuan matematika siswa di Indonesia masih tergolong rendah. Kurangnya latihan dalam memecahkan soal yang berstadar TIMSS menjadi salah satu pemicu rendahnya kemampuan siswa di Indonesia. Tujuan dari dilakukannya penelitian ini adalah untuk menganalisis proprosi soal Penilaian Tengah Semester (PTS) kelas V semester genap SD Ragunan 07 Pagi berdasarkan domain kognitif TIMSS 2019. TIMSS merupakan pengujian berskala internasional yang digunakan untuk mengukur kemajuan kemampuan dalam matematika dan sains. Domain kognitif TIMSS 2019 terdiri dari domain pengetahuan (knowing), penerapan (applying), dan penalaran (reasoning). Metode penelitian yang digunakan yaitu kuantitatif dengan desain penelitian analisis isi/konten. Instrumen yang digunakan dalam penelitian ini yaitu soal Penilaian Tengah Semester yang diambil dari SDN Ragunan 07 Pagi. Hasil proprosi dari penelitian ini pada domain kognitif yang terdapat pada pada soal Penilaian Tengah Semester (PTS) yaitu, (1) domain pengetahuan (knowing) sebesar 69,9%, sedangkan menurut TIMSS sebesar 40%; (2) domain penerapan (applying) sebesar 16,6%, sedangkan menurut TIMSS sebesar 40%; (3) domain penalaran (reasoning) sebesar 13,3%, sedangkan menurut TIMSS sebesar 20%. Hasil tersebut menunjukkan bahwa proporsi pada soal Penilaian Tengah Semester (PTS) belum sesuai dengan proporsi domain kognitif TIMSS 2019. ----- Mathematics is considered as a subject that is difficult to learn and teach. Lack of good concept development in students is one of the mathematical factors that is considered difficult, so this has an impact on student learning outcomes that are still relatively low, and when viewed based on TIMSS results from year to year shows the mathematical ability of students in Indonesia is still relatively low. The lack of practice in solving questions with the TIMSS standard is one of the triggers for students' low ability in Indonesia. The purpose of this study is to analyze the problems of Mid-Semester Assessment (PTS) class V even semester SD Ragunan 07 Pagi based on the 2019 TIMSS cognitive domain. TIMSS is an international scale test used for the application of testing benefits in mathematics and science. The 2019 TIMSS cognitive domain consists of the knowledge (knowing), application (applying), and reasoning (reasoning) domains. The research method used is quantitative research design content analysis. The instrument used in this research was a matter of Mid-Semester Assessment taken from SDN Ragunan 07 Pagi. The results of the proxies from this research on the cognitive domain contained in the Mid-Term Assessment (PTS) problems, i.e, (1) the domain of knowledge (knowledge) of 69.9%, whereas according to TIMSS by 40%; (2) the domain of application (application) of 16.6%, whereas according to TIMSS by 40%; (3) the domain of reasoning (reasoning) of 13, 3%, whereas according to TIMSS by 20%. These results indicate that the proportion in the Mid-Semester Assessment (PTS) questions does not match the proportion of the 2019 TIMSS cognitive domain

Matematik Öğretmenlerinin Ev Ödevleri ve Sınıf İçi Değerlendirmelere Yönelik Görüşleri: TIMSS 1999 ve TIMSS 2007 Dönemleri

The purpose of this study is to compare the responses to the common items regarding opinions about homework and in-class assessment in “TIMSS Mathematics Teachers’ Questionnaire” which were given by the mathematics teachers who participated to the TIMSS 1999 and TIMSS 2007 assessments. The sample from Turkey consists of 204 math teachers in TIMSS 1999 and 146 teachers in TIMSS 2007. The graphs which were generated from Mathematics Teachers’ Questionnaire data were compared in TIMSS 1999 and TIMSS 2007. Besides, the data were analyzed with z-test whether there were significant differences between the response percentages of the teachers’ opinions in TIMSS 1999 and 2007 periods. According to the findings, the frequency of homework, the amount of time to do homework and the giving frequency of type of homework such based on doing problem and routine exercises did not change, and checking homework and using homework as a contribution to student’ marks increased from 1999 to 2007. In addition, importance given by the teachers to the national-scaled exams and exams of their own has increased more in time.Bu çalışmanın amacı, TIMSS 1999 ve TIMSS 2007 durum belirleme çalışmalarına katılan Türk matematik öğretmenlerinin “TIMSS Matematik Öğretmen Anketi”nde yer alan ev ödevleri ve sınıf içi değerlendirmeye yönelik ortak maddelere verdikleri cevapların yıllara göre karşılaştırılmasıdır. Araştırmanın Türkiye örneklemini 204 (TIMSS 1999) ve 146 (TIMSS 2007) matematik öğretmeni oluşturmaktadır. Elde edilen verilerden karşılaştırmalı grafikler oluşturulmuş ve öğretmenlerin ortak anket maddelerine verdikleri cevap yüzdeleri bakımından uygulama dönemleri arasında manidar farklar olup olmadığı yüzdeler arası farkın manidarlığını test etmede kullanılan z–testi ile sınanmıştır. Bulgulara göre, öğretmenlerin ödev verme sıklık ve süresi, algoritmik işlem gerektiren soru çözme, rutin alıştırma türü ev ödevleri verme durumu dönemler arasında değişme göstermemiş, ancak, öğretmenlerin ödevlerin kontrolü ve ödev sonucunu not olarak kullanma sıklığı zamanla artmıştır. Ayrıca zamanla öğretmenlerin öğrenci başarısını değerlendirmede ulusal düzeyde yapılan testlere ve kendi hazırladığı sınıf içi sınavlara daha çok önem verdiği saptanmıştır

Pengembangan Soal Matematika Tipe Timss Menggunakan Konteks Rumah Adat Untuk Siswa Sekolah Menengah Pertama

Penelitian ini bertujuan untuk: (1) menghasilkan soal-soal tipe TIMSS yang valid dan praktis menggunakan konteks Rumah Adat di SMP; (2)melihat efek potensial soal-soal tipe TIMSS terhadap kemampuan penalaran matematika siswa di SMP. Penelitian ini merupakan penelitian pengembangan (development research). Subjek penelitian adalah siswa kelas VIII-7 SMP Negeri 9 Palembang sebanyak 32 orang. Pengumpulan data menggunakan analisis dokumen, dan tes soal tipe TIMSS. Semua data yang dikumpulkan dianalisis secara deskriptif kualitatif. Hasil analisis data menyimpulkan bahwa (1) penelitian ini telah menghasilkan suatu produk soal tipe TIMSS menggunakan konteks Rumah Adat untuk siswa kelas VIII SMP yang valid dan praktis. Valid tergambar dari hasil penilaian validator, dimana semua validator menyatakan sudah baik berdasarkan content, konstruk, dan bahasa. Selain itu kevalidan soal tipe TIMSS ini tergambar setelah dilakukan analisis validasi butir soal pada siswa sebanyak 26 orang non subjek penelitian. Praktis tergambar dari hasil uji coba small group dimana sebagian besar siswa dapat menyelesaikan soal tipe TIMSS yang diberikan. (2) prototype soal tipe TIMSS yang dikembangkan memilki efek potensial yang positif terhadap kemampuan penalaran matematika siswa. Hasil yang diperoleh rata-rata kemampuan penalaran matematika siswa dalam menjawab soal matematika tipe TIMSS sebesar 56 yang termasuk kategori baik

What Explains Differences in International Performance? TIMSS Researchers Continue to Look for Answers

The Third International Mathematics and Science Study (TIMSS)-1995 and its successor, TIMSS-1999, provide researchers, policymakers, and educators with an unprecedented opportunity to explore the possible relationships between educational policies and student achievement in an international context. Due to the amount and complexity of the TIMSS data, analyses have proceeded in phases, each providing a more detailed and sophisticated understanding of these relationships. This Brief is the third in the Consortium for Policy Research in Education’s (CPRE) series reporting on the policy implications of TIMSS data. It summarizes results of recently completed TIMSS research that explores in greater detail questions raised in initial analyses. Earlier analyses of TIMSS data were based largely on observed differences between instructional variables in the United States and in high-achieving TIMSS nations. This was a kind of “benchmarking” approach in which certain practices were hypothesized to increase student achievement. Some of the research summarized in this Brief continues this earlier work by exploring additional differences between policy and practice in the United States and in other TIMSS nations. Other studies expand on this work by looking for instructional variables that might explain differences in student achievement across a broader range of TIMSS nations. Results do not yield simple lessons and may thus be frustrating for those who hoped that TIMSS might prescribe specific recommendations for policy and practice. Recent research does, however, continue to raise important questions about the relationships between policy, practice, and student achievement in an international context. The summary presented here highlights key issues that emerge from this work

ANALISIS KEMAMPUAN PEMAHAMAN KONSEPTUAL, PEMECAHAN MASALAH, DAN PENALARAN ADAPTIF SISWA KELAS V SEKOLAH DASAR DALAM MENYELESAIKAN SOAL TIMSS

Tujuan dari penelitian ini adalah untuk mengetahui tingkat kemahiran matematis khususnya kemampuan pemahaman konseptual, pemecahan masalah, dan penalaran adaptif dalam mengerjakan soal Trends in International Mathematics and Science Study (TIMSS). Soal TIMSS yang digunakan adalah soal TIMSS tahun 2007, 2011, dan 2015 dengan objek penelitiannya adalah seluruh siswa kelas V pada salah satu SD Swasta yang terletak di kota Bandung. Metode yang digunakan dalam penelitian ini adalah survey cross-sectional. Teknik pengumpulan data menggunakan metode tes, observasi, wawancara dan dokumentasi. Melalui penelitian ini diperoleh hasil bahwa tingkat kemampuan siswa dalam menyelesaikan soal TIMSS adalah 553 atau berada pada kategori tinggi (high). Rata-rata tingkat kemampuan pemahaman konseptual dan kemampuan pemecahan masalah siswa kelas V Sekolah Dasar dalam menyelesaikan soal TIMSS berada pada kategori sedang. Sedangkan tingkat kemampuan penalaran adaptif siswa kelas V Sekolah Dasar dalam menyelesaikan soal TIMSS didominasi pada kategori rendah. ;--The purpose of this research is to determine the students competency level of mathematical proficiency, especially the ability of conceptual understanding, problem solving, and adaptive reasoning when solving the problems of Trends in the International Mathematics and Science Study (TIMSS). The problems used are TIMSS 2007, 2011, and 2015 which carried out in one of the private Elementary School in Bandung. The method used in this study is a cross-sectional survey. Data is collected through written test, observation, students interview and documentation. Through this study the results obtained that the level of students' ability to solve TIMSS questions is 553 or in the high category. The average level of conceptual comprehension and problem solving ability of 5th grade students in completing TIMSS questions is in the moderate category.The level of adaptive reasoning ability of 5th grade elementary school students in completing TIMSS questions was mainly classified in the low category

Cross-National Surveys of Learning Achievement: How Robust are the Findings?

International surveys of learning achievement and functional literacy are increasingly common. We consider two aspects of the robustness of their results. First, we compare results from four surveys: TIMSS, PISA, PIRLS and IALS. This contrasts with the standard approach which is to analyse a single survey with no regard as to whether it agrees or not with other sources. Second, we investigate whether results are sensitive to the choice of item response model used by survey organisers to aggregate respondents' answers. In both cases we focus on countries' average scores, the within-country differences in scores, and on the association between the two. There is mixed news to report