International Consultant for Country Programme Evaluation, Timor Leste

UNFPA Timor-Leste is seeking two (2) International Consultants for the Evaluation of UNFPA Timor-Leste Country Programme 2021-2025

 

 

The UNFPA is the leading UN agency on sexual and reproductive health and youth and adolescents’ development. Its mission is to deliver a world where every pregnancy is wanted, every childbirth is safe and every young person’s potential is fulfilled. 

UNFPA Timor-Leste’s programmes consist of targeting universal access to sexual and reproductive health and reproductive rights and the three transformative results -ending preventable maternal deaths, ending the unmet need for family planning, ending gender-based violence and all harmful practices. Our priority actions are thus three-folded: reducing maternal and perinatal mortality and morbidity, through the strengthening the primary health care system to deliver a high-quality integration of maternal and newborn health care including antenatal care (ANC) and postnatal care (PNC), Emergency Obstetric and Newborn Care (EmONC) Services, Family Planning (FP), HIV/STI, and Gender Based Violence (GBV). Reducing the unmet need for family planning and ensuring the safety and dignity of all women, girls, adolescents, youth and key populations, through the strengthening of laws against gender-based violence and the implementation of comprehensive sexuality education programmes that promote human rights and gender equality. Advocating for the availability, quality, timeliness and accuracy of statistical data that serve policy and decision-makers in addressing population and development issues, through the strengthening of the national capacity for the production and dissemination of quality databases, studies and investment cases in demographic disparities, socio-economic inequalities, health economics analytics, adolescents and youth and gender-based violence.

The UNFPA Timor-Leste country programme is implemented from 2021 to 2025. At the penultimate year, it is mandatory for the country office to conduct a final country programme evaluation per UNFPA Evaluation Policy to assess progress and evaluate the areas where the UNFPA Timor-Leste’s implementation of the ICPD Plan of Action is lagging. The results of the evaluation will be a crucial step for designing the new country programme document, and consequently protecting the gains and identifying ways to accelerate transformative changes. The CPE will draw conclusions and provide a set of actionable recommendations for the next programme cycle.

The main audience and primary intended users of the evaluation are: (i) The UNFPA Timor-Leste CO; (ii) the Government of Timor-Leste; (iii) implementing partners of the UNFPA Timor-Leste CO; (iv) rights-holders involved in UNFPA interventions and the organizations that represent them (in particular women, adolescents and youth); (v) the United Nations Country Team (UNCT); (vi) Asia and Pacific Regional Office (APRO); and (vii) donors. The evaluation results will also be of interest to a wider group of stakeholders, including: (i) UNFPA headquarters divisions, branches and offices; (ii) the UNFPA Executive Board; (iii) academia; and (iv) local civil society organizations and international NGOs. The evaluation results will be disseminated as appropriate, using traditional and digital channels of communication.

 

The evaluation will be managed by the evaluation manager within the UNFPA Timor-Leste CO, with guidance and support from the regional monitoring and evaluation (M&E) adviser at the APRO, and in consultation with the evaluation reference group (ERG) throughout the evaluation process. A team of independent external evaluators will conduct the evaluation and prepare an evaluation report in conformity with these terms of terms of reference. 

 

Duration and working schedule:

The consultants are expected to work up to 45 days upon signing of the contract until 31 July 2024.

 

Place where services are to be delivered:

The consultants will work both remotely and on mission to Timor-Leste.

The consultant is expected to be on one-time mission of 2 weeks. 

When on mission, the consultant will be based at the UNFPA office. For remote-based support, the consultants will closely coordinate with the UNFPA M&E team and Management.

 

Delivery dates and how work will be delivered:

The Individual consultants expected to produce the following deliverables in electronic form.

 

Deliverables	Due Date	Payment  (%)
Inception report using template annex 1	2 weeks upon the signing of the contract	20%
First draft of the evaluation report	1 week after the field mission	40%
Final evaluation report and a PowerPoint presentation of the evaluation results	4 weeks after the field mission	40%

 

Monitoring and progress control, including reporting requirements:

Regular consultation (face-to-face, online meetings, e-mails) and sharing of information and documents to the M&E and the technical review committee

 

Supervisory arrangements:

The consultant will be guided by the UNFPA Country Representative and closely coordinated by CPE team leader and the Evaluation Managers.

 

Inputs / services to be provided by UNFPA:

UNFPA will provide a work station when the consultants are on mission to Timor-Leste.

 

 

 

I. OBJECTIVES AND SCOPE OF THE EVALUATION

 

Objectives

The overall purpose of the Country Programme Evaluation (evaluation) for 2021-2025 is to: (i) enhance the accountability of UNFPA for the relevance and performance of the fourth country programme, (ii) provide the existing knowledge-base with evidence and lessons learnt to serve the design of the next programming cycle and the acceleration of the implementation of the ICPD PoA

The specific objectives of evaluation include:

• To provide a comprehensive and updated analysis of the CP4 context, focusing on multisector needs, population changes and dynamics (including youth bulge, ageing population or climate change), political and social dynamics between multiple level stakeholders, and capabilities and resources    
• To provide an independent assessment of the relevance, effectiveness, efficiency and sustainability of UNFPA support and progress towards the expected outputs and outcomes set forth in the results framework of CPD4 including in humanitarian settings, emphasizing on the coherence and functional relations with the implementing partners and analysing positive and negative intended and unintended results and challenges
• To provide an assessment of the role played by the UNFPA country office (CO) in the coordination mechanisms of the United Nations Country Team (UNCT) with a view to enhancing the United Nations collective contribution to national development results and UNFPA contribution to the UNSCDF result groups
• To draw key lessons learnt from the past and current cooperation and provide a set of clear and forward-looking options leading to strategic and actionable recommendations for the next programming cycle

 

Temporal scope

The Country Programme Document (CPD) and Country Programme Action Plan (CPAP) cover the period between 2021 to 2025. The evaluation will cover the period starting from 2021 to Q2 2024 (until the data collection).

 

Geographical scope

The evaluation will cover all implementation activities by UNFPA and implementing partners at national and sub-national levels.

 

Programme scope

This evaluation will cover the following thematic areas of the CP4: Sexual Reproductive Health and Rights, Adolescents and Youth, Gender Equality and Women’s empowerment, and Population & Development and Humanitarian Action.

 

The evaluation will cover cross-cutting aspects such as human rights based approach, integration of the LNOB principle, disability inclusion, communication, partnerships and resource mobilization. Besides the assessment of the intended impact of the country programme, the evaluation also aims to assess UNFPA’s positioning in Timor-Leste to address the emerging issues and other megatrends such as youth bulge and ageing population.    

III. UNFPA COUNTRY PROGRAMME

 

The UNFPA Fourth Country Programme Document (CPD4) for Timor-Leste for 2021-2025, detailing the planned collaboration between UNFPA and the Republic Democratic of Timor-Leste (RDTL) is aligned with the Timor-Leste’s Strategic Development Plan (SDP 2011 – 2030), the International Conference on Population and Development Programme of Action (ICPD PoA), the Sustainable Development Goals (SDGs), and the 2021-2025 United Nations Strategic Development Cooperation Framework (UNSDCF) for Timor-Leste. The CPD4 was developed in consultation with government of Timor-Leste (GoTL) and partners taking into account the shift to a new business model of working in lower middle-income countries, such as Timor-Leste, i.e. focusing on advocacy and policy dialogue/advice, capacity development,  knowledge management and some projects on service delivery.

 

The third country programme evaluation highlighted several key achievements: development of key technical and policy documents; development of in-school teaching materials on sexual and reproductive health and rights (SRHR), gender and gender-based violence (GBV) prevention; approval of the National Action Plan on Gender-Based Violence (NAP-GBV); and undertaking of the 2015 population and housing census and 2016 Demographic and Health Survey. 

The evaluation identified lessons learned and made recommendations for developing the capacity of the Ministry of Health in safe motherhood, family planning, addressing gender-based violence, and improving adolescent sexual and reproductive health. It also recommended for continued strengthening of integrated sexual and reproductive health systems, including the logistics management capacity of the Ministry of Health, and technical support on collecting data, with increased emphasis on raising data literacy to enable the Government to obtain, interpret and utilize the data for policy and planning.

Drawing on the experience of previous programmes, the CPD4 supports the UNSCDF strategic priorities 3 (Early Childhood Development and Life-long Learning Outcomes and Skills) and 4 (High-quality Healthcare and Well Being). The overall goal of the 4th CPD in Timor-Leste is to support national efforts to achieve universal access to sexual and reproductive health and reproductive rights, in line with the UNFPA transformative results to end maternal deaths, unmet need for family planning, and gender based violence and harmful practices. It responds to the principle of leaving no one behind, focusing on women, adolescents and youth, particularly those living in rural areas, people with disabilities and key population groups.

 

The Implementation of CPD4 has been quite challenging due to COVID-19 pandemic which affected all countries around the world without exception.  The restrictions imposed by the host country hampered the movements of staff from IPs and UNFPA during the implementation of programmes from mid-2020 when the COVID-19 pandemic hit the country. Major structural, conjectural and behavioural changes were observed across the world subsequent to the pandemic crisis and Timor-Leste could not escape from those shifts.

 

 

IV. EVALUATION CRITERIA

 

In accordance with the methodology for CPEs outlined in the UNFPA Evaluation Handbook (see section 3.2), the evaluation will examine the following four OECD/DAC evaluation criteria: relevance, effectiveness, efficiency and sustainability. It will also use the evaluation criterion of coherence to assess the extent to which the UNFPA Timor-Leste CO harmonized interventions with other actors, promoted synergy and avoided duplication under the framework of the UNCT. Furthermore, the evaluation will use the humanitarian-specific evaluation criteria of coverage and connectedness to investigate: (i) to what extent UNFPA has been able to provide life-saving services to affected populations that are hard-to-reach; and (ii) to work across the humanitarian-peace-development nexus and contribute to building resilience.

 

The evaluation is expected to answer these key preliminary evaluation questions under each criterion:

Relevance	EQ1. To what extent the Country Programme is aligned with the UNFPA strategic plan 2022-2025 priorities and accelerators and with relevant national SDG targets? EQ2. To what extent has UNFPA ensured that the varied needs and capacities of targeted  populations, including women and girls, adolescents and youth, survivors of gender-based violence, people at risk of and living with HIV, and also of government and civil society organizations at national and local levels, have been taken into account in both the planning, implementation and monitoring of all UNFPA-supported interventions under the country programme action plan? EQ3. To what extent has the country office been able to respond to changes in national needs and priorities, caused by gaps in policies and data, protocols or external factors, or to shifts caused by crisis or major political changes?
Effectiveness	EQ4. To what extent have interventions led and supported by UNFPA changed the access to, and use of quality human-rights based integrated sexual reproductive health (maternal health, family planning, HIV/STI) services and gender-based violence response mechanisms? EQ5. To what extent has UNFPA ensured that the needs of young people (including adolescents) in all their diversities (age, location, gender, sexual orientation, ability, employment, marital status etc.) have been addressed in the planning and implementation of all UNFPA-supported interventions?
Efficiency	EQ6. Did UNFPA get the value for money for its intervention vis-à-vis the results achieved?
Sustainability	EQ7. To what extent has UNFPA been able to support implementing partners and beneficiaries (rights-holders), in developing capacities and establishing mechanisms to ensure ownership and the durability of effects?   EQ8. To what extent did UNFPA achieve government ownership of various interventions (BEmONC, Safe Spaces, ANC-PNC, Family Planning, HIV, Census)?
Coherence	EQ9. To what extent is the UNFPA country office benefited from coordinating with other United Nations agencies and partners in the country to ensure complementarity, particularly in the event of potential overlaps?
Coverage & connectedness	EQ10. To what extent has UNFPA humanitarian action appropriately targeted at the varied needs of population groups facing life-threatening suffering, particularly those within groups that are hard-to-reach and “furthest-behind”?   EQ11. To what extent has UNFPA established linkages between a short-term emergency intervention and the recovery phases to build capacity and resiliency of the humanitarian partners and beneficiaries?
Connectedness	EQ12. To what extent were activities of a short-term emergency nature carried out in a context that takes longer-term and interconnected problems into account?

 

 

V. METHODOLOGY AND APPROACH

 

The evaluation team shall develop the evaluation methodology in line with the evaluation approach and guidance provided in the UNFPA Evaluation Handbook. The CPE will be conducted in accordance with the UNEG Norms and Standards for Evaluation, Ethical Guidelines for Evaluation, Code of Conduct for Evaluation in the UN System, and Guidance on Integrating Disability Inclusion, Human Rights and Gender Equality in Evaluations. The methodology that the evaluation team will develop builds the foundation for providing valid and evidence-based answers to the evaluation questions. The methodological design of the evaluation shall include in particular: (i) a theory of change; (ii) a strategy for collecting and analyzing data; (iii) specifically designed tools for data collection and analysis; (iv) an evaluation matrix; and (v) a detailed evaluation work plan and agenda for the field phase.

 

The evaluation matrix is centerpiece to the methodological design of the evaluation (see Handbook, section 1.3.1, and Tool 1: The Evaluation Matrix, pp. 138-160 as well as the evaluation matrix template in Annex C). The evaluation matrix will be drafted in the design phase and must be included in the design report and in the annexes of the final evaluation report.

 

Finalization of the evaluation questions and related assumptions. Based on the preliminary questions and the theory of change underlying the CP, the evaluators are required to refine the evaluation questions. In their final form, the questions should reflect the evaluation criteria and clearly define the key areas of inquiry of the CPE. The evaluation questions must be complemented by a set of critical assumptions that capture key aspects of how and why change is expected to occur, based on the theory of change of the CP. This will allow the evaluators to assess whether the preconditions for the achievement of outputs and the contribution of UNFPA to higher-level results, in particular at outcome level, are met. The data collection for each of the evaluation questions and related assumptions will be guided by clearly formulated quantitative and qualitative indicators. If needed, the TOC can be revised to support the evaluation.

 

Sampling strategy. Building on the initial stakeholder map and based on information gathered through document review and discussions with CO staff, the evaluators will develop the final stakeholder map. From this, using concrete selection criteria, the evaluation team will select a sample of stakeholders at national and sub-national levels who will be consulted through interviews and/or group discussions during the data collection phase and also select a sample of sites that will be visited for data collection.

 

Data collection. The evaluation will consider primary and secondary sources of information. Primary data will be collected through semi-structured interviews with key informants, as well as group discussions with service providers and rights-holders and direct observation during visits to selected sites. Secondary data will be collected through review of documents and from administrative databases, such as DHIS2 and LMIS. A case study will be considered for specific topics as relevant.

 

Data analysis. The evaluators must enter the qualitative and quantitative data in the evaluation matrix for each evaluation question and each assumption. Once the evaluation matrix is completed, the evaluators should identify common themes and patterns that will help to answer the evaluation questions. The evaluators shall also identify aspects that should be further explored and for which complementary data should be collected, to fully answer all the evaluation questions and thus cover the whole scope of the evaluation (see HBK sections 5.1, 5.2). The methods for data analyses are expected to be explained in the CPE design report.

 

Validation mechanisms. The mechanisms to ensure the validity of collected data and information include (but are not limited to) systematic triangulation of data sources and data collection methods, regular exchange with the evaluation managers at the CO, internal evaluation team meetings to corroborate data and information for the analysis of assumptions, the formulation of emerging findings and the definition of preliminary conclusions; and the debriefing meeting with the CO and the ERG at the end of the field phase. (see Handbook, section 3.4.3, section 4.2). During the field phase, besides interviews, focus group discussions and workshops, the evaluators may conduct a collective corporate workshop with UNFPA’s staff particularly for intersected programmes (SRH & Gender & Youth, Youth and Population Dynamics).

 

Theory-based approach. The CPE will adopt a theory-based approach that relies on an explicit theory of change, which depicts how the interventions supported by the UNFPA Timor-Leste CO are expected to contribute to a series of results (outputs and outcomes) that contribute to the overall goal of UNFPA. The evaluation team will be required to verify the theory of change underpinning the UNFPA Timor-Leste CPD4 and to determine whether changes at output and outcome levels occurred (or not) and whether assumptions about change hold true.

 

Participatory approach. The CPE will be based on an inclusive, transparent and participatory approach, involving a broad range of key partners and stakeholders at national and sub-national levels. Particular attention needs to be paid to involve beneficiaries from the groups furthest behind, including PWDs.

Mixed-method approach. The evaluation will primarily use qualitative methods for data collection, including document review, interviews, focus group discussions, case studies, and observations during field visits, where appropriate. The qualitative data will be complemented with quantitative data to minimize bias and strengthen the validity of findings. Quantitative data will be compiled through desk review of documents, websites and online databases, and possibly through surveys, to obtain relevant financial data and data on key indicators that measure change at output and outcome levels.

 

 

VI. EXPECTED OUTPUTS

 

The evaluation team will produce the following deliverables:

• An inception report using the template of Annex 1 including (as a minimum): a) a stakeholder map; b) the evaluation matrix (including the final list of evaluation questions and indicators); c) the overall evaluation design and methodology with a detailed description of the data collection plan for the field phase; and d) a detailed evaluation work plan and agenda for the field phase. Maximum 70 pages
• PowerPoint presentation of the design report. The PowerPoint presentation will be delivered at an ERG meeting to present the contents of the design report and the agenda for the field phase. Based on the comments and feedback of the ERG, the evaluation manager and the regional M&E adviser, the evaluation team will develop the final version of the design report.

• PowerPoint presentation for debriefing meeting with the CO and the ERG. The presentation provides an overview of key emerging findings of the evaluation at the end of the field phase. It will serve as the basis for the exchange of views between the evaluation team, UNFPA Timor-Leste CO staff (incl. senior management) and the members of the ERG who will thus have the opportunity to provide complementary information and/or rectify the inaccurate interpretation of data and information collected.

• A draft evaluation report using the template of Annex 2 (potentially followed by a second draft, taking into account potential comments from the evaluation technical committee and ERG);
• A final evaluation report;
• A PowerPoint presentation of the results of the evaluation for the dissemination events.

All deliverables will be in the English version of the final draft evaluation report is required.

 

 

VII. PROFILE OF THE EVALUATION TEAM

 

The evaluation team will be composed of a Team Leader and two Team Members as specified below. It will consist of two (2) international expert and one (1) national expert who demonstrated expertise and experience in

• evaluation of other country programmes – at least three previous ones
• experience in programme formulation (at least two previous experiences, preferably with the UN)

 

Team leader – international consultant

The competencies, skills and experience of the evaluation team leader should include:

• Master’s degree in public health, social sciences, demography or population studies, statistics, development studies or a related field.
• At least 10 years of experience in conducting evaluations in the field of international development, in Maternal and Reproductive Health, Gender and Women’s Empowerment, Population and Development and evaluation methodology
• Extensive experience in leading complex evaluations commissioned by UN organizations and/or other international organizations and NGOs.
• Demonstrated expertise in one of the thematic areas of the CP covered by the evaluation (see expert profiles below).
• In-depth knowledge of theory-based evaluation approaches and ability to apply both qualitative and quantitative data collection methods and to uphold high quality standards for evaluation
• Ability to ensure ethics and integrity of the evaluation process, including confidentiality and the principle of do no harm.
• Ability to consistently integrate human rights and gender perspectives and disability inclusion in all phases of the evaluation
• Excellent management and leadership skills to coordinate the work of the evaluation team, and strong ability to share technical evaluation skills and knowledge.
• Experience working with a multidisciplinary team of experts.
• Excellent ability to analyze and synthesize large volumes of data and information from diverse sources.
• Excellent interpersonal and communication skills (written and spoken).
• Work experience in/good knowledge of the region and the national development context of Timor-Leste
• Fluent in written and spoken in English. Portuguese or Bahasa Indonesian is an advantage.

 

Two Thematic Experts – (1 national and 1 international consultants)

The competencies, skills and experience of the thematic expert should include:

• Relevant Master’s degree in the programmatic area of the evaluation (see details below)
• Substantive knowledge of the programmatic area of the evaluation such as

knowledge of SRHR, including HIV and other sexually transmitted infections, maternal health, and family planning

• Substantive knowledge of adolescent and youth issues, in particular SRHR of adolescents and youth
• Substantive knowledge on gender equality and the empowerment of women and girls, GBV and other harmful practices
• Substantive knowledge on the generation, analysis, dissemination and use of housing census and population data for development, population dynamics, migration and national statistics systems
• 5-7 years of experience in conducting evaluations, reviews, assessments, research studies in the field of international development
• Ability to ensure ethics, integrity and confidentiality of the evaluation process, incl. do no harm principle
• Ability to consistently integrate human rights and gender perspectives, and disability inclusion in all phases of the evaluation
• Solid knowledge of evaluation approaches and methodology and demonstrated ability to apply both qualitative and quantitative data collection methods
• Excellent analytical and problem-solving skills
• Experience working with a multidisciplinary team of experts.
• Excellent interpersonal and communication skills (written and spoken).
• Work experience in/good knowledge of the national development context of Timor-Leste
• Familiarity with UNFPA or other UN organizations’ mandates and activities will be an advantage.
• Fluent in written and spoken in English. Portuguese is an asset

 

 

VIII. BUDGET AND PAYMENT MODALITIES

 

 

The exact number of workdays and workload distribution will be proposed by the evaluation team in the design report and will be subject to the approval of the evaluation manager.

 

The evaluators will receive a daily fee according to the UNFPA consultancy scale based on qualifications and experience.

The payment of fees will be based on the submission of deliverables, as follows:

• 20%: Upon approval of the design report
• 40%: Upon submission of a draft evaluation report of satisfactory quality
• 40%: Upon approval of the final evaluation report and the PowerPoint Presentation of the evaluation results

In addition to the daily fees, the evaluators will receive a daily subsistence allowance (DSA) in accordance with the UNFPA Duty Travel Policy, using applicable United Nations DSA rates for the place of mission. Travel costs will be settled separately from the consultancy fees.

 

 

Note: The full ToR with annexes will be made available to the consultants who have been recruited to conduct the CPE.

 

 

Interested candidates should send cover letter, P11, technical and financial proposal to this email address: timor-leste.office@unfpa.org

• A technical proposal outlining objectives, formulation of questions for the review, methodology, sample questionnaire for kick-off workshop, KII and FGD, a list of literature to be reviewed, an outline of the report and a workplan;
• A financial proposal for 45 working days required for this CPE;