Home > About
About SACMEQ
The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) is an international non-profit developmental organisation of 15 Ministries of Education in Southern and Eastern Africa that decided to work together to share experiences and expertise in developing the capacities of education planners to apply scientific methods to monitor and evaluate the conditions of schooling and the quality of education, with technical assistance from UNESCO International Institute for Educational Planning (IIEP).
The 15 Ministries of Education that constitute the SACMEQ network are Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania (Mainland), Tanzania (Zanzibar), Uganda, Zambia and Zimbabwe. SACMEQ has completed two major education policy research (SACMEQ I and SACMEQ II) Projects between 1995 and 2005. The third (SACMEQ III) Project that commenced in 2007 is scheduled to be completed by 2010.
- Download an animated presentation of SACMEQ (6 MB).
- Origins of SACMEQ
- SACMEQ’s Membership, Governance, and Management
- SACMEQ's Mission
- The Evolution of SACMEQ and Its Research and Training Programmes
- The Use of SACMEQ’s Reports and Information Resources
- Four Evaluations of SACMEQ
- Why has SACMEQ Been So Successful?
- Visualization of Research Results on the Quality of Education
About SACMEQ
Member countries
Ministers meetings
National coordinators
Photo gallery
Research medals
SACMEQ in the news
External friends
Useful links
Indicators and Reports
SACMEQ indicators
SACMEQ reports
Technical reports
Research and Training
Papers & theses
Training modules
Training workshops
SACMEQ III
Project update
Scientific committee
SACMEQ data
Data archive application
Visualization
Research visualization
StatPlanet Map Maker
In 1989 Zimbabwe’s Minister for Education and Culture and the Director of the IIEP agreed that a major research and training project (the “Indicators of the Quality of Education Study”) should be undertaken in Zimbabwe during 1990 in order to (a) assess the quality of education provided by Zimbabwe’s primary schools, (b) involve the staff of the Ministry’s Planning Unit in integrated research and training activities, and (c) provide meaningful advice related to policy concerns expressed by senior Ministry decision-makers.
The project resulted in a research report (Ross and Postlethwaite, 1991) that was used to review a range of important education policy issues and to provide baseline information for comparison with later studies of the quality of education in Zimbabwe. The report was also used as the central theme of a series of IIEP follow-up training workshops for educational planners and researchers from Zimbabwe and several nearby countries.
During 1992 the educational researchers and planners that had participated in these workshops commenced a dialogue within their Ministries of Education and with IIEP staff on the training needs required in order to expand and strengthen the capacity of their Education Planning Units to monitor and evaluate the quality of their education systems. At the very outset of these discussions it was agreed that the most effective modality for this type of training was not a traditional “university-style” arrangement – in which theories and methods were explained in lectures and then illustrated with examples delivered during tutorial-style practical sessions. Instead, it was decided that a “learning-by-doing” approach should be adopted in which the most modern educational policy research methodologies would be applied in a cross-national fashion to “real educational policy problems” in order to provide hands-on experience in the generation of information that can be employed for making informed decisions aimed at improving the conditions of schooling and the quality of education.
This dialogue eventually resulted in the preparation of a proposal (Moyo et al., 1993) that would address these important research and training challenges via the establishment of an association of Ministries of Education known as SACMEQ (the Southern and Eastern Africa Consortium for Monitoring Educational Quality).
SACMEQ’s first cross-national cooperative activity was to work with the IIEP on the preparation of a research-based policy report that was entitled “From Educational Research to Educational Policy: An Example from Zimbabwe” – which was authored by teams of educational planners and researchers from ten countries (Ross, 1995). This report received a very positive reaction from many African Ministers of Education, and in late 1995 the SACMEQ Consortium was officially launched and given continuing long-term support through the generous assistance of the Government of the Netherlands.
The SACMEQ Consortium now has 15 members: Ministries of Education in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania (Mainland), Tanzania (Zanzibar), Uganda, Zambia and Zimbabwe. The linkages among the different elements of the SACMEQ Consortium during 2008-2009 are presented below.
Linkages among Different Elements of SACMEQ Consortium
(2008-2009)
Broad policy decisions concerning SACMEQ's mission and its programmes of research and training are set down by the SACMEQ Assembly of Ministers of Education – which meets every two years in Paris at the time of the UNESCO General Conference. The details of SACMEQ’s administration and finance are coordinated by the SACMEQ Managing Committee, a sub-committee of the SACMEQ Assembly of Ministers, which holds an annual meeting in Africa and is available, as required, for more frequent consultation.
SACMEQ’s daily operational activities are managed by the SACMEQ Coordinating Centre which is staffed by the SACMEQ Director and the SACMEQ Data Processing Manager. The SACMEQ Coordinating Centre provides administrative and technical support for the SACMEQ National Research Coordinators and their Deputies. The SACMEQ Coordinating Centre also works with partners to obtain funding for the cross-national co-operative components of SACMEQ’s research and training programmes. These components include: training workshops; technical inputs to sampling and measurement requirements; the production of specialized software systems for sampling, data analysis, and data cleaning; the design and production of technical and field manuals; and meetings of the SACMEQ Scientific and Measurement Committees.
SACMEQ Ministries of Education
Each SACMEQ Ministry of Education is responsible for (a) managing its own within-country SACMEQ research activities (for example: the appointment of SACMEQ National Research Coordinators and their Deputies, consultations with curriculum and research specialists concerning the construction of tests and questionnaires, and the training and logistics associated with SACMEQ data collections), and (b) covering its own within-country costs associated with participation in SACMEQ research projects (for example: salary coverage for SACMEQ National Research Coordinators and their Deputies, costs associated with the printing and distribution of data collection instruments, travel and per diems for data collection teams, and payments for staff involved in data entry and data cleaning operations).
SACMEQ National Research Coordinators
SACMEQ’s National Research Coordinators are drawn from various divisions within the Ministries of Education. For example, they could be from the Planning Division (Botswana, Lesotho, Malawi, Namibia, Swaziland, Tanzania (Zanzibar), Uganda, Zambia, Zimbabwe), the Examinations Centre (Kenya), the Human Resources Department (Mauritius), the Institute for National Educational Development (Mozambique), the Office of the Principal Secretary (Seychelles, Tanzania (Mainland)), or the Quality Assurance Division (South Africa).
The work of the SACMEQ National Research Coordinators and their Deputies is supported by other within-country staff – especially during (a) the conduct of SACMEQ data collections, and (b) the conduct of data entry and data cleaning operations. For example, during the implementation of the main data collection for SACMEQ’s most recent research study (the SACMEQ III Project) it was necessary to train and deploy a total of 2,250 Data Collectors to make field visits in teams of three people to around 2800 schools across the SACMEQ countries.
SACMEQ's partners and external friends
The IIEP’s role with respect to the SACMEQ Consortium has been concerned with the provision of training and technical inputs to the SACMEQ Coordinating Centre and the SACMEQ National Research Coordinators and Deputies. Other “External Friends” have also contributed in these areas – particularly the International Association for the Evaluation of Educational Achievement (IEA), the Aga Khan Foundation (Kenya Office), the National Centre for Educational Research and Training (NCERT) in Malawi, and the Assessment Research Centre in Australia.
From 1995 to 1998 SACMEQ operated as an organization without a permanent home base – with the main points of contact among the SACMEQ countries being regular training workshops arranged by the IIEP for the SACMEQ National Research Coordinators and their Deputies. In 1998 this changed when SACMEQ established an office for the SACMEQ Director, and later the SACMEQ Data Processing Manager, within the UNESCO Regional Office in Harare, Zimbabwe.
In March 2006 the Government of Zimbabwe officially registered SACMEQ as “an international non-profit, developmental organization with a membership consisting of Ministries of Education in Southern and Eastern Africa”. Certain funding agreements that were made during 2007 resulted in the channelling of funding for SACMEQ’s research and training programmes via the IIEP, and it therefore became more efficient to locate the SACMEQ Coordinating Centre, on a temporary basis, in Paris during the period 2008-2009. In 2010 the SACMEQ Coordinating Centre will return to an African location that is yet to be determined by the SACMEQ Assembly of Ministers.
SACMEQ's main mission was set down by the SACMEQ Assembly of Ministers as follows:
“To undertake integrated research and training activities that will expand opportunities for educational planners and researchers to: (a) receive training in the technical skills required to monitor, evaluate, and compare the general conditions of schooling and the quality of basic education; and (b) generate information that can be used by decision-makers to plan the quality of education.”
This mission statement contained two key elements that have been employed to design and guide SACMEQ’s research and training programmes for over a decade. Firstly, the main target audiences for SACMEQ’s activities were “educational planners and researchers”. Second, the substantive content of SACMEQ’s activities was divided into two main areas: “training” (in technical skills) and “information generation” (for use by decision-makers).
It is important to note that the SACMEQ Assembly of Ministers intentionally placed “training” ahead of “information generation” in SACMEQ’s mission statement in order to send a clear message that SACMEQ should be seen primarily as a capacity-building initiative, and not mainly as a mechanism for the provision of educational planning information.
In the words of two SACMEQ Ministers of Education, the SACMEQ Consortium should (a) “be recognized as an innovative and successful approach to human resource development rather than as an indicators production factory”, and (b) “be required to concentrate on expanding and deepening training activities within the SACMEQ countries - with the task of information generation being treated as a useful by-product arising from the approach of focussing SACMEQ training workshops on real education policy problems”.
Within the SACMEQ Consortium these somewhat subtle distinctions about SACMEQ’s mission are considered to be very important. However, due to the high profile and wide usage of SACMEQ’s research reports and data archives, many individuals and organizations outside the SACMEQ Consortium have assumed that SACMEQ’s main objective is to generate educational planning information in the same manner that is expected of other international indicator production initiatives such as the OECD’s PISA Project and the IEA’s TIMSS and PIRLS Projects. This kind of “mission confusion” has occasionally resulted in SACMEQ being judged by the degree to which it has delivered useful information according to tight time schedules, rather than by its outstanding performance in improving the capacity of educational planners and researchers to monitor, evaluate, and compare the general conditions of schooling and the quality of education.
The Evolution of SACMEQ and Its Research and Training Programmes
The nature and operations of the SACMEQ Consortium have evolved over the past 20 years in harmony with the increased scope and complexity of the four research and training projects in which it has been involved. This evolution, which has been shown in diagrammatic form below as four main phases, commenced from humble beginnings as an experimental IIEP research and training project conducted in Zimbabwe, and has developed into an independent and officially registered African inter-governmental organization that is widely respected for the relevance, importance, and impact of its research and training programmes.
SACMEQ’s evolution began with an “Innovate” phase (in which the IIEP worked with a single country on an innovative experimental educational research and training project designed to address a major educational challenge related to planning the quality of education). In the next phase the IIEP encouraged several countries in the same region to “Collaborate” (by working together on a mutually beneficial multi-country replication of the earlier experiment that drew upon lessons learned). This was followed by a phase that was designed to “Consolidate” the control of participating countries over the whole of the research and training process (by working together, sharing their experiences, and learning from each other). Finally there was the “Launch” phase (in which the participating countries assumed overall control of research and training directions, and began to make their own professional decisions about who assisted them and how they were to be assisted).
Each of these phases has been accompanied by major shifts in the power relationships between the IIEP and SACMEQ – with the IIEP gradually surrendering its controlling influence over SACMEQ’s research and training directions. The IIEP commenced its work with SACMEQ as the “initiator” of the experimental project in Zimbabwe (Phase 1), and then gradually reduced its profile by first acting as a “facilitator” for the SACMEQ I Project (Phase 2), and then as “the external friend” for the SACMEQ II Project (Phase 3), and finally as “one of several external friends” for the SACMEQ III Project (Phase 4).
In Phase 1, from 1989 to 1994, the IIEP worked with the Zimbabwe Ministry of Education to conduct a national survey of the conditions of schooling and the quality of education, and then to report the results of this research in a report written by teams of educational planners drawn from countries surrounding Zimbabwe. During this phase the IIEP controlled all research and training decisions because the SACMEQ Consortium existed only as a kind of “informal interest group” and there was no SACMEQ Coordinating Centre.
In Phases 2 and 3, from 1995 to 2004, the SACMEQ Consortium focussed on the delivery of integrated research and training programmes in support of two cross-national educational policy research studies (the SACMEQ I and SACMEQ II Projects). These studies examined the conditions of schooling and the quality of education across Southern and Eastern Africa, and in particular they collected accurate and detailed information on the reading and mathematics performance of Grade 6 students. The SACMEQ I Project was conducted during the period 1995 to 1998 and covered around 1000 schools and 20,000 students in seven countries (see References). The SACMEQ II Project was conducted during the period 1999 to 2004 and covered around 2000 schools and 40,000 students in fourteen countries (see References). The IIEP was located in the central controlling position for the SACMEQ I Project in Phase 2 because, just as in Phase 1, there was no SACMEQ Coordinating Centre. This situation changed in Phase 3 when the SACMEQ Coordinating Centre was established and accepted as the main coordinating body for the SACMEQ II Project, and the IIEP moved to the outer fringes of the Consortium as an external friend.
Phase 4 commenced in 2005 when the SACMEQ Consortium started to work on the design and the implementation of the SACMEQ III Project. This study followed the general research directions of the first two SACMEQ projects by focussing on an examination of the conditions of schooling and the quality of education. The SACMEQ III Project involved data collection in around 2800 schools from around 70,000 pupils, 6,000 teachers, and 2800 school principals. At the request of the SACMEQ Ministers of Education, an additional research component was included in this study in order to provide information that could be used to generate more effective educational planning responses to the challenges associated with the HIV-AIDS pandemic.
In Phase 4 the SACMEQ Coordinating Centre took the central role in managing the SACMEQ III Project – thereby consolidating SACMEQ as an independent entity that was owned and operated by the participating Ministries of Education. This final phase removed the IIEP’s controlling linkages with the SACMEQ Coordinating Centre and the SACMEQ Ministries – with the IIEP’s role being reduced to the status of one of SACMEQ’s several external friends that are available to assist – but only when requested to do so.
The well-planned 20-year odyssey of SACMEQ towards independence represents a major achievement on behalf of the IIEP and other partners (particularly the Government of the Netherlands) because it has given the SACMEQ Coordinating Centre sufficient time and opportunities to develop the kinds of human resources within the SACMEQ countries that are required to design, implement, and manage research and training directions in accordance with the wishes of the participating Ministries of Education. This important transformation has taken a great deal of effort and dedication – and it provides hard evidence that there are no “quick-fix shortcuts” to success for the establishment of worthwhile, sustainable, and client-owned education development initiatives in developing countries.
The Use of SACMEQ’s Reports and Information Resources
It was not until 2004 that SACMEQ’s reputation for research and training excellence became more widely known - both within Africa and many other countries. This change occurred because the SACMEQ Ministers of Education decided in July 2004 that it would be possible for educational planners and researchers from around the world to have access to the SACMEQ Data Archives. Before that date no SACMEQ country would permit access to its SACMEQ project data by researchers from other countries – even by other researchers from the SACMEQ countries. Since that date there has been a massive growth of interest in using the SACMEQ Data Archives for conducting educational policy research studies and for the preparation of postgraduate theses.
One interesting “indicator” of the spread of SACMEQ’s reputation and the degree to which its research results are used is that in July 2004 an Internet search (via AltaVista) for the “SACMEQ” acronym provided around 250 “hits”, but by September 2008 this figure had grown to over 23,000. In addition, a recent search for “SACMEQ” by using the Google Scholar option (which targets published reports and articles) resulted in a listing of over 450 reports and publications.
In the discussion presented below some examples have been provided to illustrate the use of SACMEQ research and training resources by ministries, international organizations, bilateral organizations, universities, and individual researchers and planners.
(a) Use of SACMEQ By Ministries of Education
The research reports from SACMEQ’s first two educational policy research projects (see References) have been used extensively within the participating countries as a resource for education sector studies and as baseline information that can be employed in policy discussions and debates about the conditions of schooling and the quality of education. Murimba (2005), Nzomo and Makuwa (2006), and Oduol (2006) have provided reviews of the use of these reports by the SACMEQ Ministries of Education.
(b) Use of SACMEQ by International/Bilateral Organizations
SACMEQ’s Data Archives and reports have been used by organizations such as UNESCO (2004; 2005; 2006; 2007; and 2008), the World Bank (2006), the Asian Development Bank (Hirosato, 2001), and the DFID Edqual Project (Yu and Thomas, 2007) in order to discuss and evaluate policy options related to the quality of education.
(c) Use of SACMEQ by Universities
In 2005 the IIEP arranged seminar presentations for professors and graduate students in African universities in order to demonstrate how to use the SACMEQ Data Archive for the preparation of masters theses and doctoral dissertations. Since that time there has been a steady flow of postgraduate work based on SACMEQ data at institutions such as the University of Bourgogne, Bristol University, the University of Cape Town, the University of Gothenburg, the University of Melbourne, the University of Pretoria, the University of Stellenbosch, the University of Western Australia, and the University of the Western Cape.
(d) Use by Individual Educational Planners and Researchers
The SACMEQ Data Archive and SACMEQ Website are very popular with educational planners and researchers from many countries. Both of these resources contain valuable research data, research tools, publications, software, and training materials. In late 2008 there were around 400 “Officially Registered” users of the SACMEQ Data Archive – located in ministries of education, universities, donor organizations, national and international agencies, non-government organizations, and a variety of research establishments. There are known to be at least another 200 “Unofficial” users. In 2007, in response to requests from a number of African universities, a full set of SACMEQ’s training modules in the area of quantitative research methods was placed on SACMEQ and IIEP Websites. It is estimated that hundreds of downloads of the modules have occurred – by users in universities, teachers colleges, and research units located in both developed and developing countries.
During 2004 SACMEQ was awarded the Comenius Medal by the International Bureau of Education at the 47th International Conference on Education in Geneva. The Comenius Medal celebrates the life and work of Jan Amos Komensky, a 17th Century philosopher, theologian, and teacher who devoted his life to the improvement of education through innovation. The citation for the award congratulated SACMEQ for:
“having made possible a highly fruitful type of collaboration among 15 Ministries of Education (Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania (Mainland), Tanzania (Zanzibar), Uganda, Zambia, and Zimbabwe) in the field of innovative educational research”
(http://www.ibe.unesco.org/comenius/comenius.htm)
In the period 2005 to 2008 four external evaluations were conducted in order to examine SACMEQ’s research and training programmes, and also to take stock of SACMEQ’s impact on education policy and planning in the participating countries.
SACMEQ was evaluated in 2005 as part of a World Bank-sponsored evaluation of the research and training programs conducted by the International Institute for Educational Planning (IIEP). The extensive discussion of SACMEQ’s initiatives and impact presented in the evaluation report concluded with the following statements:
“Our overall assessment is that SACMEQ is an innovative and very effective form of capacity and institution-building, which is producing results in the form of reliable research findings relevant for policy.
There is a strong sense of ownership among the 15 Ministries of Education in the member countries, and the research findings have had an impact on policy in several countries. There is growing recognition in these Ministries that policy making should be based on reliable evidence on school conditions and quality, and there is strong evidence of the impact on national research teams of training and assistance provided by the IIEP.”
(Oxenham and Woodall, 2005, page 50).
(b) Second External Evaluation
A second external evaluation of SACMEQ was conducted in 2006 for UNESCO Headquarters. The evaluation report described SACMEQ as follows:
“The Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ) is considered to be a particularly successful initiative in monitoring educational quality and, based on the SACMEQ experience, IIEP has been approached to initiate other such networks.
The SACMEQ project has progressed in a number of stages: an initial capacity-building project during which IIEP staff trained educational planners from one Ministry of Education to undertake a national study; subsequent joint work with seven ministries of education, with IIEP and member state research teams working as equal partners to replicate the integrated research and training; and SACMEQ operating as an autonomous international organization governed by an Assembly of Ministers of Education and undertaking a 15 nation study on the quality of education.
The characteristics of the SACMEQ project which contributed to its success include: long-term donor support; a decade of hard work and commitment; and a shared need for information on educational quality.”
(Davis and Mutch, 2006, page 82)
(c) Third External Evaluation
A third evaluation of SACMEQ was undertaken in 2007-2008 by an external evaluation team commissioned by UNESCO’s Internal Oversight Service. In the “Overall Comments” section of their report the evaluators noted that:
“SACMEQ is a well established, successful cross-national data collection and capacity building effort. Overall, the technical soundness of the SACMEQ project is of good quality. The aims of the study are clearly stated and are relevant for all participating countries, and the defined target population is appropriate.
The achievement tests have been built upon a solid framework and blueprinting process that were designed to address the policy questions expressed by the participating countries. The content of the SACMEQ questionnaires (pupil, teacher, and school head) provides excellent coverage of the study’s overall aims.
Participants in SACMEQ workshops have expressed great satisfaction with the training and technical support that they receive. Evidence from these participant comments indicates that the data they receive from SACMEQ meet their needs for policy development.
SACMEQ needs to be congratulated and commended for excellent organizational arrangements, effective quality control and quality assurance procedures, and assessment capacity development in the participating countries.”
(Ercikan et al., 2008, pages 5-6)
(d) Fourth External Evaluation
A fourth external evaluation of SACMEQ was undertaken during 2008 at the request of the Netherlands Ministry of Foreign Affairs. The evaluator’s report commenced with the following “General Observations”:
“SACMEQ has been able to develop into a major regional programme for the monitoring of educational quality with an outstanding reputation for the sophistication of its research procedures, the reliability of its outcomes, its relevance for educational development in the region, and the professionalism and dedication of its staff at all levels.
SACMEQ is widely known in the Ministries of Education of the region, among a number of international donor and technical agencies, and among specialised organisations and experts in the field of educational measurement and assessment at the international level. In these environments SACMEQ is highly respected for the quality of its work.
Policy makers and professionals in the region view SACMEQ primarily as an African programme that plays a major role in educational development, especially (at this point) at the basic education level. If anything, there is a common view that this role needs to be maintained, to be enhanced and deepened, and to be further institutionalised at country and at regional levels.
(Hoppers, W., 2008, page 1)
Why has SACMEQ Been So Successful?
Many governments, organizations, universities, and agencies have conducted studies that are concerned with monitoring and evaluating the quality of education in developing countries. However, the reports produced from these studies have only occasionally captured the attention of senior decision-makers in Ministries of Education. In contrast, the educational policy reports published for the SACMEQ I Project and SACMEQ II Project (see References) have featured prominently in presidential and national commissions on education (for example, Zimbabwe and Namibia), prime ministerial and cabinet reviews of educational policy (for example, Zanzibar), national education sector studies (for example, Kenya and Zambia), and reviews of national education master plans (for example, Mauritius).
The high quality of SACMEQ’s reports and associated SACMEQ research materials have also been recognized by major universities (for example, Harvard and Melbourne University), research organizations (for example, the International Association for the Evaluation of Educational Achievement), international organizations such as UNESCO’s EFA Global Monitoring Report Division), and other prestigious associations (for example, the International Academy of Education).
This prominence of SACMEQ's policy research programme at the highest decision-making levels poses an important question: "What makes SACMEQ so successful?"
The answer to this question can be found in certain key features of the design and implementation of SACMEQ's research and training programmes. These are concerned with (a) a co-operative "working style" derived from ownership and an integrated research and training programme, (b) the use of an education policy research cycle based on the high priority policy concerns of decision-makers, and (c) a determination to employ the world's best research methodologies.
(a) A Co-operative Working Style"
While “what SACMEQ does” is extremely important, many experts believe that two key features related to “how SACMEQ works” have been central to its consistent capacity to generate high quality research and training programmes. First, the participating Ministries of Education (via decisions taken during the biennial SACMEQ Assembly of Ministers Meeting) have ownership of the scope and future directions of SACMEQ's research and training programmes. Second, SACMEQ integrates research and training activities through a "learning-by-doing" strategy aimed at long-term institutional capacity building.
These characteristics have fostered a “co-operative working style" in which SACMEQ National Research Co-ordinators have been able to work together, to share expertise, and to learn from each other. An important outcome from this co-operation has been that SACMEQ projects have employed the same technical procedures across participating countries with respect to research objectives, target populations, sample designs, data collection instruments, field work, basic data analyses, and policy report formats. These common technical procedures have ensured that SACMEQ's reports offer meaningful policy advice that ranges from within-country usage to cross-country and regional applications.
(b) A Research Cycle Based on the Policy Concerns of Decision-Makers
SACMEQ’s research and training activities are embedded within a general framework known as the “SACMEQ Education Policy Research Cycle” (see Appendix A). The cycle has three main steps (see outer circle of Appendix A): “Research”, “Development”, and “Evaluation”. Within these main steps there are 8 activities – commencing with the “General Policy Concerns (of Decision-Makers)”, and then moving to “Specific Research Questions”, “Data Collection and Analysis”, and so on (see inner circle of Appendix A). The cycle ends with “Policy Reform and Agenda for Action” and “Programme Implementation” - and then the cycle starts again.
One of the most important features of the SACMEQ Education Policy Research Cycle is its starting point. Each of SACMEQ’s major research projects have commenced with a series of consultations with key decision-makers in participating Ministries of Education in order to define their “General Policy Concerns”. The information gathered during this process is debated and refined by the SACMEQ National Research Co-ordinators in order to define a short list of “Specific Research Questions” that can be used to guide research design.
The use of an initial survey of general policy concerns to control, and often drastically limit, aspects of research design differs a great deal from the approach taken by wide-ranging, curiosity-driven, university-style educational research projects. In particular, the use of policy concerns to drive the whole research process: (i) accelerates the completion of research by restricting data collections to information that can be used to address specific policy concerns, (ii) prevents research from degenerating into wasteful long-term "fishing expeditions" that offer diffuse and/or highly technical findings on topics that are of negligible interest to decision-makers, and (iii) permits the policy concerns of decision-makers to shape the research process - which increases the likelihood that research results will be used to make informed decisions.
The final chapter in each SACMEQ national policy report provides a meta-analysis of the various policy suggestions presented in earlier chapters. In all SACMEQ national reports these suggestions are grouped into five main themes: consultations with staff, community, and experts; reviews of existing planning procedures; data collections for planning purposes; educational policy research projects; and investment in infrastructure and human resources.
Within each theme the policy suggestions are classified according to the department or unit within the Ministry of Education that should be responsible for implementation. The time required for implementation is also estimated according to three broad categories: "short"-around 3 to 9 months, "medium" - around one to two years, and "long" - around three to five years. Finally, each policy suggestion is linked to approximate cost estimates ranging from "low" (activities that could be accommodated within existing budgets) to "high" (major investments in capital works and human resources).
Prior to the publication of SACMEQ reports, all "draft" policy suggestions are discussed and clarified with senior decision-makers. This is an important step because it provides critical feedback on both the wording and feasibility of the suggestions. For example, a single poorly-worded suggestion could confuse, or even enrage, a decision-maker who might then immediately develop doubts about the credibility and utility of all other material in the policy report. Similarly, a suggestion for action to be taken by a Ministry of Education may not be feasible because the suggested action either lies within the domain of another Ministry, or is far too expensive to implement, or is subject to constraints (linked with administrative, logistical, or legal barriers) that prevent implementation.
SACMEQ's technique of employing the general policy concerns of senior decision makers in Ministries of Education as a foundation for research design and research reporting was endorsed by the International Academy of Education in a publication that reviewed the benefits and limitations of participating in international studies of educational achievement (Beaton et al, 1999).
(c) A Determination to Employ the World’s Best Research Methodologies
Some large-scale educational survey research studies conducted in developing countries have been based on inappropriate methodologies. These studies have employed erroneous sampling procedures, poorly-designed data collection instruments, inadequate field work arrangements, faulty data entry and data cleaning procedures, and inaccessible reporting formats (Ross et al., 1990).
When SACMEQ was initially launched in 1995, the SACMEQ National Research Co-ordinators decided to make every effort to identify and use the world's best research methodologies - rather than to "make do" with low quality substitutes. Three important areas where the SACMEQ I and SACMEQ II projects have employed state-of-the-art methodologies have been related to the critical areas of data preparation, sample design, and test construction.
(i) In the area of data preparation, SACMEQ has been using a new version of the Windows Data Entry Manager (WINDEM) software system developed by the International Association for the Evaluation of Educational Achievement (IEA) for computer-based data entry. This has been supplemented with IIEP’s specialized JANITOR data cleaning software – which conducts a wide range of data accuracy checks to ensure that information contained in questionnaires and tests is correctly transferred to computer-readable formats.
(ii) In the area of sample design, SACMEQ decided to apply the official technical standards (Martin et al., 1999) set down by the IEA. These consist of strict rules concerning the selection of samples and the evaluation of their technical precision. In order to meet these standards, SACMEQ has been involved in the development of two important pieces of computer software. These are: IIEPSAMP - a computer program that facilitates the design and selection of large-scale stratified two-stage complex probability samples of schools and students, and IIEPJACK - a computer program that undertakes the computer-intensive calculations required for an assessment of the accuracy of complex survey samples. Many requests have been received from both developed and developing countries to obtain access to these valuable research tools.
(iii) In the area of test construction and test scoring, SACMEQ has been undertaking analyses of test data with the RUMM software produced by the University of Western Australia (Andrich, Sheridan and Luo, 2003). This software supports both classical and item response theory models, and is used for the construction and scoring of tests. In the SACMEQ Projects the application of item response theory models facilitated the generation of valid comparisons of student reading and mathematics achievement across countries both within and across the three SACMEQ projects. This was achieved by calculating student scores on the same underlying scales of reading and mathematics - even though these scores had been derived from different (but overlapping) sets of test items. This methodology also provided "fairer cross-national comparisons” because students could be measured only with respect to test items that assessed "essential" elements of their own country's curriculum.
(iv) In the area of presenting and visualizing research results, SACMEQ has been using the StatPlanet interactive visualization software through which data can be presented in map, graph and table form.
© 2010 SACMEQ |