Consultancy Health Equality / Climate Resilient WASH

Improved and sustained health, gender equality, and well-being of rural and peri-urban communities in KP Province through inclusive, sustainable and climate-resilient WASH (LIFE-Pakistan)

Terms of References (TORs) for End of the Project Evaluation

1.      Introduction

 IRC is one of the world’s leading humanitarian and development organizations, working with communities in over 40 developing countries. The core of IRC’s mandate is to support and empower people who have been affected by conflict and disaster. IRC’s portfolio consists of government, multilateral, and foundation funding that spans short-term and protracted humanitarian response through long-term development initiatives. Alongside other thematic areas, IRC has extensive global expertise in health and reproductive health and is a leader in gender-focused programming.

IRC has been working in Pakistan since 1980 and delivers a wide range of programming to marginalized/underserved communities across the country. Programs focus on Health and WASH, Disaster Risk Reduction, Protection and Women’s Protection and Empowerment, Livelihoods, and Education.

2.      Program Description

International Rescue Committee (IRC) is implementing the “Leveraging Inclusive and Climate Sensitive WASH for Empowerment – LIFE II” project in three districts of Khyber Pakhtunkhwa province of Pakistan with an overall goal, “Improved and sustained health, gender equality, and wellbeing of rural and peri-urban communities in KP Province, through inclusive, sustainable and climate resilient WASH". The  Australian Government’s Department of Foreign Affairs and Trade (DFAT) funded the project under the Water for Women (WfW) Fund. In district Swat, the project is being led by the “Environmental Protection Society” (EPS), while in Peshawar and Buner districts, it is being self-implemented by IRC. The project intended to support 45 villages, 10 selected health facilities and 15 schools in climate-resilient, inclusive WASH across 3 districts. At the Outcome Level, there are four End of Project Outcomes (EOPO), as elaborated below.  

EOPO1: Provincial WASH governance systems are strengthened, with greater emphasis on inclusive and climate-resilient WASH.

EOPO 2:  Increased equitable, universal access to and use of sustainable climate-resilient WASH services, particularly for marginalized communities and community members.

EOPO 3: Strengthened climate-resilient gender equality and social inclusion in households, communities, and institutions.

EOPO 4:  Strengthened use of new evidence innovation and practice in sustainable gender-sensitive, inclusive and climate-resilient WASH by Communities, Local Government, other CSOs, and national and international WASH actors.

3. Evaluation Purpose

The purpose of this evaluation is

• To assess the overall performance of the project and to evaluate end-of-program outcomes against the Theory of Change, the project design, and JMP Indicators.
• To review the recommendations of the project's previous phase and assess the extent to which these were implemented.
• The Evaluation is intended to assess and triangulate the evidence of the project results.
• The evaluation seeks to capture both intended and unintended results for all four components.

4. Primary Information Users

The primary audience for final evaluations is.

• Project, AMU and MEAL teams, including the Technical Team, Project Coordinator, Component Managers, and MEAL Coordinator/Managers, to reflect on and learn from how the project has performed and what actions must be taken while designing similar future programs.
• Senior Management of the International Rescue Committee.
• The Fund Coordinator, GHD and DFAT will also use the project evaluation results and findings for future WASH programs, fund synthesis, learning, and decision-making.

5. Scope of the evaluation

The scope of this end of the project evaluation will cover all the three-project districts. i.e., Swat, Peshawar, and Buner covering the entire project duration.  The evaluation will integrate climate resilience, inclusion, and governance throughout its methodology and all deliverables, including the final report. More specifically, the evaluation is expected to cover the following components.

5.1. Evaluation criteria and Evaluation questions.

This evaluation will be carried out using “The Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Criteria[1] and will assess project performance against relevancy, effectiveness, efficiency, impact, and sustainability. Similarly, according to the updated DAC, Coherence will be assessed to measure how well the intervention was a good fit. The evaluation questions have been set considering the WfW MEL framework, but the project team will have an opportunity to refine questions further during the inception phase of the evaluation.

The evaluation is expected to address all the questions detailed below to the greatest extent possible. The evaluator may adapt the evaluation criteria and questions, but any fundamental changes should be agreed upon between the IRC management and the evaluator. The evaluation instruments (to be summarized in the inception report) should identify the general and IRC priority areas. Suggested evaluation criteria and questions are summarized below.

Evaluation criteria

Main evaluation questions and sub-questions

Effectiveness

• To what extent were the four End of Project Outcomes (EOPOs) achieved as planned?
• How effectively did the project strengthen provincial WASH governance systems, particularly regarding inclusivity and climate resilience (EOPO 1)?
• To what extent did the project increase equitable access to and use of sustainable WASH services for marginalized groups (EOPO 2)?
• How successful was the project in improving climate-resilient gender equality and social inclusion in households, communities, and institutions (EOPO 3)?
• How effectively was new evidence, innovation, and practice disseminated and adopted by communities, local government, and other WASH actors (EOPO 4)?
• What intended and unintended, positive, and negative results were produced by the intervention?
• Are there any external factors that hindered or facilitated the achievement of the project?
• How effective was the partnership with the local organization in achieving the project’s outcomes, particularly in ensuring the inclusion of marginalized groups and delivering climate-resilient WASH services?

Relevance

• To what extent were the project's objectives aligned with the needs and priorities of the target communities, particularly marginalized groups, in the context of climate change?
• How relevant were the project activities addressing gaps in WASH services, gender equality, and social inclusion in the target districts?
• Did the project adapt to any emerging challenges or contextual changes (e.g., climate shocks, local governance changes)
• How effectively did the program learn from the previous phase of the project and inform the required program redesign?

Coherence (new DAC criteria)

• How well did the project align with and complement other local, national, and international WASH or climate resilience interventions in the districts?
• Were there any synergies or overlaps with other ongoing initiatives, and how were they managed to avoid duplication of efforts?

Efficiency

• Were resources (human, financial, time) used efficiently to achieve the project’s intended outcomes?
• Were there any delays or resource constraints, and how were they managed to minimize the impact on project delivery?
• How efficient was the coordination between the project partners (e.g., local government, CSOs, schools, and health facilities)?
• To what extent has the project delivered value for money?

Impact

• What tangible changes have been observed in the WASH governance systems at the provincial and community levels as a result of the project?
• How has the project impacted access to climate-resilient WASH services, especially for marginalized communities?
• To what extent has the project influenced gender equality, social inclusion, and the use of innovative WASH practices?

Sustainability

• How likely are the project outcomes (e.g., strengthened governance, and access to WASH services) to be sustained after the project’s completion?
• What measures were put in place to ensure that the innovations and practices introduced will continue to be used by communities, local government, and other WASH actors?
• To what extent did the project build local capacity (e.g., in WASH governance, service delivery, gender mainstreaming) to ensure long-term sustainability?
• To what extent are the community structures (IWJ) likely to continue in the future to address the community WASH issues?

6. Methodology

The project evaluation will employ a mixed-methods approach, combining quantitative and qualitative data collection techniques to assess the project in line with the DAC criteria and the specific evaluation questions detailed earlier. The consultant will be tasked with developing a comprehensive methodology, ensuring it captures critical aspects of the project's performance, such as relevance, effectiveness, efficiency, impact, and sustainability.

This methodology should include creating customized data collection tools, such as surveys, interviews, and focus group discussion guides, to gather insights from various stakeholders. These tools must facilitate the collection of quantitative data to measure outcomes and perceptions and qualitative data to delve into participants' experiences and contextual factors influencing the project's outcomes.

The consultant will submit the proposed methodology, outlining sampling approaches, data analysis methods, and a detailed timeline. This methodology will be finalized during the assignment's inception phase, following a thorough review and approval, before the commencement of data collection activities.

7. Evaluation Deliverables

The evaluation team is expected to deliver:

a.     Inception Report – The inception report will outline the findings from the desk review with key questions raised determining the scope of the work, the intended work plan of the analysis, the sampling method to be used, data collection and reporting plans with draft data collection tools keeping in view evaluation criteria, objectives and questions. The inception report should include a proposed schedule of tasks, activities, and deliverables, as well as the travel and logistical arrangements for the team.

b.      Debriefings/feedback sessions to management/relevant information users—The team will plan a feedback session and report its preliminary findings to IRC Management five days after fieldwork. The team leader will incorporate any comments about factual inaccuracies and present the full draft report to the evaluation management team within one week of the debriefing meeting.

c.       Draft report—A draft report identifying key findings, conclusions, clear recommendations (operational or strategic level, etc.), and lessons for the current and future operation, considering the debriefing session's outputs.

d.      Presentation to the IRC Technical Team—After producing the draft report, the Consultant is expected to present the findings to the technical team for validation and further improvement of the report.  

e.       Final report—The final report will contain a short executive summary (no more than two pages) and a main body (Max. 30 pages); it must cover the background of the intervention, evaluation methodology, including evaluation objectives, data sources, data collection and analysis and critical limitations, Findings by district and by component (disaggregated by Gender, disability and district), Lessons learned in all four components, including Govt. partnerships and clear recommendations (are to be supported by a specific set of findings and must be action-oriented, specific, and actionable). The report should also contain appropriate appendices.

8. Expected Timeframes:

The evaluation study will be completed within 40 days of the agreement signing. The table below provides an approximate guide for estimated days required for the completion of key activities, and the consultant will provide a separate breakdown for filed activities and expenses for information collection by the field enumerator:

Task

Time Frame (Days)

Desk review, Finalization and translation of data collection tools / guiding questions

05 days

Inception report (Final tools / guiding questions, methods, sample size, analysis plan, reporting template, etc.) meeting with the project team

05 days

Training of field teams on methodologies, tools, ethics and data punching

03 days

Primary Information collection at 3 target districts, including stakeholder interviews

12 days

Data cleaning, entry, analysis, and report draft

07 days

Submit draft Report and in-person presentation to stakeholder team

03 days

Final Report and Presentation to management

05 days

9.       Financial Disbursement

The agreed budget will be transferred to the consultant in tranches based on deliverables. The details are below.

Milestone/Deliverable

% of Among to be released

Approval of Inception Report

40%

Approval of the final draft of the report

60%

10.   Intellectual Property Rights

IRC will own all products arising from this assessment. Without written authorization, the consultant may not present any analytical results as his or her work or use the assessment results for private publication.

11.   Quality & Ethical Standards

The consultant should take all reasonable steps to ensure that the assessment is designed and conducted to respect and protect the rights and welfare of the people and communities involved and to ensure that the assessment is technically accurate and reliable, is conducted transparently and impartially, and contributes to organizational learning and accountability. The assessment team will also adhere to the IRC Way and Code of Conduct.

12.   Qualification of Consultant/ Consultancy firm

The following qualifications, competencies, and skills are required to carry out this assignment successfully: Specific skills and experience sought:

• A qualified team with at least a team lead must hold a post-graduate degree in relevant fields such as Research Studies, Social Work, Gender Studies, Development Studies, or other related disciplines.
• Expertise in collecting, analyzing, and reporting qualitative and quantitative WASH-related data.
• Proven experience working with diverse stakeholders, including government officials, donors, youth, civil society organizations, women and adolescent girls, and persons with disabilities (PWD), and the ability to engage effectively with stakeholders at various levels.