1. How can evaluations be independent if they are conducted by the Evaluation Office?
- The Evaluation Office reports to the UN Environment Executive Director, whereas all operational staff in UN Environment (managing and overseeing projects) report through their Division Directors to the Deputy Executive Director. In this way, the work of the Evaluation Office is independent of the operational side of UN Environment.
2. If the Evaluation Office is independent, why does it use external consultants to undertake evaluations?
- External evaluation consultants introduce an additional level of independence and impartiality to the evaluations since they are external to the organization. In addition, successful evaluation of the broad spectrum of activities that UN Environment undertakes can only be done by contracting external experts with the specific technical expertise for each of the subject-specific projects. The Evaluation Office also uses external evaluation consultants because the demand for evaluation is far greater than could be met by Evaluation Office professionals alone.
3. How are consultants selected for evaluations?
- Selection criteria for evaluation consultants are determined separately for each evaluation. This assessment takes into consideration the thematic area, regional focus, language requirements, strategic importance and complexity of the project to be evaluated. Evaluation consultants’ CVs are then screened against these criteria and the strongest candidates are invited to go through a competitive interview process. Evaluation consultants are also carefully vetted for conflict of interest to ensure their independence from the project, UN Environment Divisions and key partners involved in the project.
4. How can I become an evaluation consultant for the Evaluation Office?
- The Evaluation Office welcomes Curriculum Vitaes for qualified evaluation consultants. You can register on the UN INSPIRA portal and add your details to the Consultants General Roster, selecting the Evaluation category. If your Curriculum Vitae/profile meets the selection criteria for a specific evaluation, Evaluation Office staff will contact you for more information.
5. What kind of evaluations does the Evaluation Office do?
- UN Environment regards evaluation as a key component in effective Results Based Management. The Evaluation Office has responsibility for managing and conducting evaluations of projects and sub-programmes as well as thematic evaluations and other studies of management interest in the organization. The Evaluation Office carries out project evaluations across all funding sources including projects supported by the Global Environment Fund. Each sub-programme is evaluated once during two Medium Term Strategy cycles. The Medium Term Strategy can be evaluated at three stages during teh four year cycle: formative stage, mid-term and end of cycle. At the end of each biennium, the Evaluation Office prepares a biennial Evaluation Synthesis Report, summarizing the performance for the organization through trends and patterns observed from completed evaluations at all levels. The predominant focus on UN Environment evaluations is on strategic relevance and the achievement of sustainable and up-scalable results and impact, along with factors that affect such performance.
6. How do you decide what gets evaluated?
According to the Evaluation Policy, all UN Environment projects need to allocate funds for evaluation and be evaluated when they finish. This can take place either by doing a single project evaluation or clustering thematically related projects and conducting a portfolio evaluation. The Evaluation Office prepares a biennial evaluation work plan in correspondence to on-going programming and project cycles. Reviewed by the Senior Management Team and approved by the Executive Director, the plan takes into consideration responsibilities and mandatory requirements of ongoing and planned evaluations: a 'long list' of possible evaluations is collated and the minimum funding required for each evaluation is estimated; criteria are applied to score and rank the prospective evaluation activities and priorities are clearly set:
- Highest priority - Strategic evaluations (for example Medium Term Strategy, sub-programme and thematic) and the preparation of mandated documents such as the biennial Evaluation Synthesis Report
- High priority - Evaluation of projects that contribute to strategic evaluations
- Preference - Mandatory evaluations and / or those evaluations with critical time constraints
- Preference - Completed projects against on-going projects;
- Preference - Projects / portfolios with larger total operational budgets;
- Preference - Projects where the magnitude / distribution of benefits/effects are expected to be high, (and among these further prioritise those where attribution of benefits/effects is most feasible);
- Mid-term evaluations of projects are considered only where the projects have been reported as being at risk, are jointly implemented with other agencies, or are of high strategic significance (or potential reputational risk) to the organization.
7. What happens to the evaluations when the evaluation report is finalized?
- The Evaluation Office communicates the evaluation findings to the project stakeholders and publishes finalized evaluation reports on this website. The evaluation recommendations are converted into a Recommendations Implementation Plan together with the project team and the Evaluation Office tracks the compliance of implementation of the recommendations every six months for 18 months, as appropriate. Up-dates on the implementation status of recommendations are regularly sent to the Deputy Executive Director and levels of compliance with recommendations are summarized in the biennial Evaluation Synthesis Report.
8. What are good lessons and recommendations?
- Good lessons and recommendations are integral components of a good evaluation report, since they often serve as the ‘take-home’ messages that continue their lives after evaluations are completed. Whereas recommendations provide concrete action points to UN Environment and its partners on how to solve a particular problem affecting the project, lessons are more general and describe good practices with a wider application. All lessons and recommendations should be clearly derived from the findings of the evaluation, and this context should be briefly presented when formulating them. Recommendations should be feasible to implement within the timeframe and resources available and clearly indicate who should do what and by when. Since the purpose of a recommendation is to solve a problem, it is okay to propose alternate solutions to the problem when relevant, and to assess the pros and cons of each option. Lessons are also derived from real project experiences but they propose action with a wider applicability by promoting practices that were found to be beneficial or discourage practices that were found to cause problems to successful project implementation.
9. How are evaluation terms of references developed?
- The Evaluation Office has developed a template for evaluation terms of references that is based on internationally accepted evaluation norms and standards from Organisation for Economic Co-operation and Development/Development Assistance Committee, the United Nations Evaluation Group, and the Global Environment Facility Evaluation Office, among others. The Evaluation Office regularly reviews the ToR against experience from evaluations, new trends in evaluation, feedback from evaluation consultants and changes and needs within the organization. The template is updated so that it remains relevant and ensures that evaluations are of high quality and respond to organizational and donor needs. The final terms of reference are specifically tailored for each evaluation by Evaluation Office staff, in collaboration with the project team. Whilst for accountability reasons, the standard criteria against which projects are evaluated remain the same, tailoring the terms of reference enables the evaluation consultant to focus attention on issues particularly important to a given project. This increases the relevance and utility of the evaluation findings.
10. How do you assess the quality of evaluation reports?
- The Evaluation Office uses a set of questions to assess the quality of the zero-draft and the final evaluation report as well as the quality of the entire evaluation process. These questions are annexed to the evaluation terms of reference and presented in the final evaluation report. The main purpose of this assessment is to increase the transparency of the evaluation process and to enable the evaluation audience to read the evaluation report against this quality assessment. Recording the report quality at zero-draft and final report stage allows the Evaluation Office to better capture this review and revision process. In addition, the Evaluation Office staff is heavily involved in the evaluation process and may add its own ratings as a second column in the ratings table in cases where they find that evaluation ratings are not sufficiently grounded in evidence, reflect internal inconsistency, or have not been established in a similar way among different evaluation consultants.
11. Where can I find information on official UN rules for consultants?
- UN Environment works under the rules and regulations set by the United Nations Secretariat. Information on official rules and regulations for consultants can be found in a document issued by the United Nations Secretariat – Administrative instructions for consultants and individual contractors, ST/AI/2013/4.
12. Who evaluates the Evaluation Office?
- The Evaluation Office is part of the United Nations Evaluation Group, which has a Framework for Professional Reviews of the Evaluation Function of United Nations Organizations. In line with this Framework, the Evaluation Office is also periodically peer-reviewed by members of the United Nations Evaluation Group. The last peer-review took place in 2012 and the peer-review report is available on this webpage.
- The evaluation reports of UN Environment projects supported by the Global Environment Facility go through a quality assessment by the Global Environment Facility Evaluation Office who periodically publish statistics of the quality of evaluation reports produced by the different Global Environment Facility Implementing Agencies. The latest reports are available here.
14. Why do Evaluation Office staff have coffee together EVERY DAY?
- We think that sharing a morning coffee moment with colleagues is a great way to catch up on what’s going on in each other’s lives but also to informally catch up on work-related issues. It really helps to build a good team spirit!
- Collegues working in or visiting Nairobi can find the Evaluation team at Coffee Station 1 (between New Office Facility Block 1 and 2) everyday at 10am.