Re-advertised: Terms of Reference
Evaluation of the Ministerial Commitment on comprehensive sexuality education and sexual and reproductive health services for adolescents and young people in Eastern and Southern African (ESA)
1. Background and Context:
A commitment for positive health outcomes for all young people in East and Southern Africa was endorsed and affirmed in 2013 by Ministers of Education and Health from 20 ESA countries. Together they agreed to work collaboratively towards a vision of young Africans who are global citizens of the future, who are educated, healthy, resilient, socially responsible, informed decision-makers, and have capacity to contribute to their community, country, and region. The countries that affirmed the commitment are Angola, Botswana, Burundi, Democratic Republic of Congo, Ethiopia, Kenya, Lesotho, Madagascar, Malawi, Mauritius, Mozambique, Namibia, Rwanda, Seychelles, South Africa, South Sudan, Swaziland, Uganda, United Republic of Tanzania, Zambia, Zimbabwe[1].
The ESA Commitment, as it is known, is a response to the circumstances of the region’s adolescents and young people aged 10 to 24 years and numbering around 199 million. They face many sexual and reproductive health (SRH) challenges, including early and unintended pregnancy, HIV and sexually transmitted infections (STIs), gender-based violence (GBV) and child marriage, discrimination and low access to quality friendly health services – all of which can undermine education opportunities, especially for girls, and affect future health and opportunities.
The ESA Commitment has time-bound targets agreed upon by member states which paved the way for actions to scale up delivery of sexuality education and related health services; supported joint action around developing programmes and sharing information; integration of services and reinforced linkages and referrals between schools and health services; and fostered an overall approach which facilitates access and equity and strengthens national responses to HIV and adolescent sexual and reproductive health and rights (ASRHR).
The ESA Commitment process was co-led by the East African Community (EAC) and the Southern African Development Community (SADC), with support from the United Nations Educational, Scientific and Cultural Organization (UNESCO), the Joint United Nations Programme on HIV and AIDS (UNAIDS), the United Nations Population Fund (UNFPA) and other UN partners. The Commitment also drew on the support of Civil Society Organizations (CSOs), and religious and youth leaders. To drive the ESA Commitment at regional level, a High-Level Group (HLG) was created, composed of regional leaders in education, sexual and reproductive health and rights (SRHR), HIV prevention, and development. Assisting the HLG is a Technical Coordinating Group (TCG), whose key task is to provide technical, administrative, and financial support to the ESA Commitment process. Having developed a Regional Accountability Framework (RAF) that breaks down the ESA Commitment targets into several processes and outcome indicators, the TCG has been instrumental in tracking progress through a harmonized monitoring and evaluation (M&E) system.
2. Purpose and scope of the Evaluation
After six years of implementation, an evaluation of the Commitment is needed to assess the effectiveness of the Commitment in achieving targets and improving outcomes for young people. The overall objective of the evaluation is to assess the results, take stock of progress and generate knowledge and evidence from the ESA Commitment implementation experience in 21 member countries and their respective Ministries of Health and Education. The overall purpose of the evaluation is to assess the processes and achievements made through the ESA Commitment efforts in order to draw lessons that will inform the rationale for the extension of the ESA Commitment to 2030 to align with Agenda 2030. The evaluation is intended to be forward looking and will provide information on the nature, extent and where possible, the effect of the ESA Commitment to the sexual and reproductive health and rights of adolescents and young people in East and Southern Africa, while also forming a baseline for a possible extension of the commitment beyond 2020.
The evaluation will cover the entire implementation period from 2013 to 2020. It will assess the progress made in the specific areas of the ESA Commitment Accountability Framework, review the achievements or lack thereof in meeting the targets set for 2015 and 2020 and assess the efficacy of the multi-sectoral mechanisms employed to realize the Commitment. While the evaluation will cover the experiences of each member country, in-depth study will be conducted in ten (10) purposefully selected countries[2] from East Africa and Southern Africa and best practices will be documented and disseminated. The size sample of key informants and focus groups discussions as well as survey respondents should total to a minimum of 100 per each one of the 10 countries, where the in-depth analysis will be conducted. It is recommended the following estimation to guide the definition of the sample:
Minimum of 10 Key informant interviews for each one of the 10 selected countries;
A minimum of 2 FGDs with a maximum on 8 participants for each one the of the FDGs in each one of the 10 selected countries;
A minimum of 74 responses to the survey that will be administered on-line to students, teachers, parents, community leaders, CSOs and traditional and religious leaders.
Specific objectives of the evaluation include the following:
1. Relevance
The evaluation should assess the design and focus of the ESA Commitment Accountability Framework and review the extent to which the objectives of the Commitment are consistent with the needs and priorities of adolescents and young people, the implementation partners, and key stakeholders within the member states. Questions to be answered here include, but not limited to the following:
How has the ESA Commitment influenced the development of national ASRHR policy, strategy and plans?
How has the ESA Commitment influenced national priorities aiming at fulfilling adolescent and young people sexual and reproductive health rights?
To what extent has the target group been involved in the ESA commitment coordination processes in the country? Have the target populations/primary beneficiaries been reached? Why or why not?
To what extent are the interests, voices and priorities of adolescents and young people taken into consideration in planning and implementation of the interventions?
Has a participatory/coordination methodology been applied as a means to achieve a larger degree of ownership by the countries?
How do stakeholders and target groups perceive the ESA Commitment and contributions made toward improving the sexual reproductive health and rights of adolescents and young people? What is the perceived value going forward?
To what extent are issues of rights to Sexual and Reproductive Health (SRH) and HIV prevention for young key populations, particularly LGBTI considered?
Did the implementation of the ESA Commitments benefit from the support of the partners? If yes, who are the key stakeholders?
2. Effectiveness
The evaluation will assess how the ESA commitment ‘operationalization’ processes that were put in place at regional and national level have been effective in coordinating the achievement of the commitment in the different countries. Questions to be answered include the following:
To what extent did the Technical Coordinating Group mechanism contribute in meeting results?
What results were achieved (quality and extent)? How were the results achieved? How do they respond to the targets set in the accountability framework?
What factors contributed to effective achievement of results, across the different country contexts?
How effective has the Accountability Framework been in responding to the needs of the beneficiaries?
What challenges were faced during implementation of the commitment and how can they be used to improve future plans in accessing services to the target group?
What are the future intervention strategies and issues?
Are there any examples of unintended results (positive or negative) from project implementation?
3. Efficiency
The evaluation will assess the efficiency of ESA Commitment implementation in terms of how country investments in the areas of the commitment have converted to results. Questions to be answered include the following:
Have countries budgeted for the realization of the targets set by the accountability framework? If so, to what extent is the investment justified by its actual results so far?
To what extent have countries been able to coordinate all the relevant initiatives under the umbrella of the ESA Commitment?
What is the added benefit of a regional commitment / HLG/ TGC to achieving targets of the Commitment?
Have the interventions been brought to scale for optimal impact?
Are the national coordination mechanisms multi-sectoral in nature and do they include planning and fiscal Ministries such as Finance and Economic Planning?
What challenges if any have been experienced in project implementation?
4. Sustainability
The evaluation should also examine the sustainability of national interventions designed to achieve the ESA Commitment’s targets. Questions to be answered include the following:
What is the likelihood of continuation and sustainability of the key interventions undertaken by countries and partners to achieve the ESA Commitment targets beyond the year 2020?
What are the strategies put in place at the national level to sustain the implementation of key interventions beyond 2020?
Are the results achieved under the umbrella of the ESA Commitment sustainable at national level?
What was the degree of involvement of private sector/civil society organizations in the implementation of ESA commitment as they are major service providers in health and education sectors?
Are the ESA Commitment interventions/targets integrated into and prioritized in the national development strategies and UN Sustainable Development Cooperation Framework at the Country level?
What are the opportunities for sharing and exchanging of best practices for replication and scale up within the countries in the ESA regions?
5. Emerging issues
The evaluation should not only look into the results achieved against the accountability framework but will also generate the necessary information for a possible extension including emerging issues relevant to the core of the commitment and not yet included. Questions to be answered include the following:
If you had to recommend for the extension of the ESA Commitment beyond the year 2020, what are key areas of work that you would like to include? Why?
Please give us a list of three (3) main areas of work that need to be included in the ESA commitment to ensure the full realization of adolescents’ sexual and reproductive health and the principle of leaving no one behind?
Are there good practices/lessons learned that have emerged from the synergies and complementarities among the participating countries in form of South-South Cooperation?
3. Methodology for Evaluation
This evaluation will be conducted virtually. No field-work is expected in the countries or communities due to the uncertainty due to the COVID-19 pandemic. A mixed-methods approach is desired for this evaluation. It is expected that the evaluation will gather both quantitative and qualitative data on the ten commitments, the nine targets of the commitment and the individual elements of the Accountability Framework. As such, the evaluation is expected to use a combination of methods, including but not limited to the following:
Desk study and review of all relevant documentation including the ESA Commitment documents, annual work-plans, annual progress reports, mid-term review report, reports of the High-Level Group and the Technical Coordinating Group
Desk study, review and sub-analysis of relevant secondary data, including Demographic and Health Surveys (DHS), Multiple Indicator Cluster Surveys (MICS) and other population surveys and studies, for all ESA Commitment member states
In virtual depth interviews to gather primary data from key stakeholders using a structured methodology and interviews with relevant key informants
Virtual Focus Group discussions with beneficiaries and other stakeholders
On-line administration of surveys to the selected sample.
A Results-Based Management approach will be applied considering not only progress toward the targets set, but quality and the logic of the commitment, as well as its consequences. The approach would allow us to analyze why intended results have or have not been achieved. It will help to identify gaps and bottlenecks and enable assessment of specific causal contributions of outputs to outcomes, examine the implementation process and explore unintended results. The results-based approach will also ensure the measurement of relevance of the action and ownership of the programme and it will offer recommendations for improvement.
4. Guiding Principles for the Evaluation
It is requested that the evaluation be conducted within the prescripts of the following four broad sets of evaluation standards as guiding principles for the consultancy, namely: propriety standards, feasibility standards, accuracy standards and utility standards:
The propriety standards are ethical standards meant to ensure that evaluations are conducted with due regard for the rights and welfare of affected people. The most basic of the propriety standards is that evaluations should never violate or endanger human rights. Evaluators should respect human dignity and worth in their interaction with all persons encountered during the evaluation and do all in their power to ensure that they are not wronged.
• The feasibility standards are intended to ensure that evaluations are realistic and efficient. To satisfy these requirements, an evaluation must be based on practical procedures, not unduly disrupting normal activities, and be planned and conducted in such a way that the co-operation of key stakeholders can be obtained. They should also be efficient.
• The accuracy standards are meant to ensure that the information produced by evaluations is factually correct, free of bias, and appropriate to the evaluation issues at hand.
• The utility standards, finally, are meant to ensure that evaluations serve the information needs of their intended users: to be useful, evaluations must be responsive to the interests, perspectives and values of stakeholders.
A human rights-based approach should be employed to bring into focus not only the relevance, effectiveness, efficiency and sustainability of activities carried out but also the processes of project implementation. Particular attention should be given to the principles of inclusion, participation, equality and non-discrimination, and accountability as addressed in project activities.
5. Duration of the Evaluation / Timeframe Activities Deliverables Timeframe (# of days) Preparation Briefing with Technical Coordinating Group Minutes of meeting 1 – September 2020 Review all relevant data sources and prepare an inception report to be submitted to the TCG The inception report will detail: methodology;availability of data sources, by commitment areas and Countries;schedule of activities and timeline per country;draft data collection tools;a data analysis matrix which links the questions in the data collection tools to the outcome areas/indicators/questions. Draft inception report including tools available for comments 7 Submit the final Inception report and quality assurance plan with all comments integrated Final inception report available 4 Data Collection Literature review of available documents, survey reports and published studies on adolescents and young people relevant to the scope of this assignment 10 Qualitative and quantitative data collection, including data capture and processing (Virtually) 30 Data Analysis and Reporting Analyze data collected and prepare draft report Draft evaluation report available for review by TCG and stakeholders 16 Integrate comments from TCG and stakeholders in draft report and share draft 4 Presentation of the draft report. Comments made by the key stakeholders will inform the final report 1 Produce a final evaluation report incorporating all comments received and a final PowerPoint presentation summarizing the report. 5 Total of days and estimated timeframe 78 (From September to November 2020)
6. Expected Deliverables
Deliverable 1: An inception report which contains the objectives and scope, description of methodology/methodological approach, data collection tools, data analysis methods, key informants/agencies, review questions, performance criteria, work plan and reporting requirements including ethical approval requirements and tools for submission. It should include a clear matrix relating all these aspects and a desk review with a list of the documents consulted as well as a quality assurance plan.
Deliverable 2: Draft report to be shared with key stakeholders for comments whose structure follows Introduction, Methodology, Analysis, Key challenges/Opportunities, Lessons Learned, Key Recommendations, Conclusions and Annexes.
Deliverable 3: Presentation of the draft report: develop and present a PowerPoint presentation showing preliminary findings, lessons learned and recommendations to the ESA Commitment’s key stakeholders. Comments made by the key stakeholders will inform the draft report.
Deliverable 4: Final evaluation report incorporating all comments received and a final PowerPoint Presentation summarizing the report.
Deliverable 5: Master presentation of the findings and recommendations.
7. Required expertise and qualification
I. Firm/Entity
The firm should fit or exceed the following profile(s)
Mandatory
It is mandatory for Firm/Entity to have a minimum 5 years of global/international experience in programme/ project evaluation. Firm/entity with less than the required number of years of experience will be disqualified
It is mandatory for Firm/Entity to be registered in Africa, or have affiliation with an Africa-based Firm/Entity with presence in the majority or all the countries where the evaluation will be undertaken.
Proven experience in submitting research protocols for ethical approvals in countries, supported by at least two references
Desirable
It is desirable for Firm/Entity to have designed and implemented a minimum of three (3) evaluations at regional level in the field of adolescent and young people sexual and reproductive health and rights
It is desirable for a Firm/Entity to have worked in the past on a project/assignment with the UN System or Regional Economic Communities
It is desirable for a Firm/Entity to have at least one (1) partner firm in half of the target countries (5 or more).
The firm will be required to submit one sample of previous similar work produced and proof of registration
Team leader
The team of consultants should have the following profile(s).
Mandatory
It is mandatory for the Team Leader to have at least an advanced degree in one of the following fields: Public health, Demography, Development Studies, Health Economics, Social Sciences, or other related studies; a PhD will be an added advantage;
It is mandatory for the proposed Team Leader to have minimum 10 years of programme and project evaluation experience at regional/global/international level.
Desirable
It is desirable for firm to propose a Team Leader with experience in the field of adolescent and young people sexual and reproductive health and rights at the regional and international level
It is desirable for the Team Leader to have proven experience and skills in developing policy, strategic documents and conducting complex evaluation in Africa at regional and national levels within UN, EAC and SADC
It is desirable that the Team Leader has experience relating to evaluation methodologies and techniques to be used, supported by at least three (3) references
III. Senior Evaluation Expert (specify if different from Team Leader)
It is mandatory for the Senior Evaluation Expert to possess a Master’s Degree in Population, Demography, Statistics, Public Health, Development Studies or other related studies;
It is mandatory for the Senior Evaluation Expert to have 7 years of experience in project/program evaluation at regional or national level;
It is mandatory for the Senior Evaluation Expert to demonstrate proven experience in conducting reviews and evaluations involving adolescents and young peoples’ sexual and reproductive health and rights, supported by at least three (3) references
It is desirable for the Senior Evaluation Expert to demonstrate experience and skills in knowledge base creation and ability to develop systems for improved performance, supported by at least two (2) references
All interested consultants/firms are requested to submit a proposal:
Explaining their competencies to meet the requirements of the assignment;
Explaining, in detail, the proposed methodology to be used in carrying out the assignment, including sampling strategy (not just sample size but also urban, rural, age, sex disaggregation, etc.);
Providing the a workplan with expected duration of the assignment and dates of availability; roles and competencies of core team members;
Providing a detailed professional budget in USD (Indicate daily professional rates and days);
Attaching brief technical bio data of core team members;
Providing evidence of similar work undertaken recently (Not more than 5 years old) and references.
8. Management Arrangement
The Evaluation Team will report to the Technical Coordinating Group under the leadership of SADC and EAC. M&E Advisors from the participating UN agencies will provide technical guidance on the evaluation and ensure independence of the evaluation process, and that policy is followed. UNESCO, UNFPA, UNAIDS, WHO, UNDP and UNICEF will manage the evaluation and provide logistical support under the overall guidance of SADC Secretariat and the East Africa Community.
9. Financial bid
The financial bid will have to include the following items (not exhaustive list) and ensure alignment with the proposed methodology:
Human resources costs
Ethical approval costs (expedite processes to be considered)
Data collection and analysis costs
Desk review and sub-analysis of relevant data costs
Quality assurance plan costs
Overheads costs
Important: Upon recruitment of the successful Firm/Entity, deliverables will be broken down based on each agency’s financial contribution. The Firm/Entity will then enter into bilateral contractual agreements with each individual agency.
10. Submission of proposal
Interested Consultants to submit technical and financial proposals via email to vacancies.harare@unesco.org no later than 4 September 2020 (Midnight CAT). Emails to reflect Ref: HAR/ED/HIV/ESA Commitment Evaluation (Re-advertised)
[1] Rwanda did not officially endorse the commitment. However, they have been active members of the initiative also regularly reporting on progress
[2] The purposively selected countries are: Angola, DRC, Eswatini, Malawi, Mauritius, Tanzania, South Africa, Zambia, Rwanda, and Uganda