Understanding barriers to the use of evaluations, as well as factors that enhance effective utilisation of findings, is therefore essential. We actively seeks to maximise evaluation utilisation by focusing on five key questions, namely:
Often, investors are obsessed with measuring the ‘impact’ of their investments. Reporting on the impact of the programme is a fair client expectation at face value, but measuring impact is sometimes unrealistic as it may not be immediately observable upon project implementation or completion. At Tshikululu we advise our clients to consider appropriate evaluation types depending on the project implementation stage and information available to make an informed judgement of the programme.
Specifically, social investors should ideally conduct an evaluability assessment prior to evaluating programmes. This assessment helps evaluators determine if it is worthwhile to evaluate. In instances where it might be too early, we advise to give time for end beneficiaries to properly experience the intervention before investing in an often costly evaluation. Data gaps are also a reality; sometimes there simply isn’t enough data/evidence for impact assessments. In these cases, social investors should opt for an outcomes evaluation as opposed to impact evaluation.
Evaluators often face resistance from programme partners and/or beneficiaries as they can be viewed as ‘police’ whose aim is to identify and pick on what’s going wrong. It’s important for evaluators to build rapport with the implementation team and potential evaluation participants (while of course not getting so close that the independence of the evaluator may be questioned). Most importantly, participants and organisations should understand that the evaluation process is meant for learning and not to “get anyone in trouble”.
In that way, there is a better chance of establishing trust, which will improve the validity of findings. Validity, in turn, enhances the uptake of evaluation recommendations.
When engaging end-beneficiaries, substantial power dynamics may come into play. For example, beneficiaries may be concerned that providing negative feedback will lead to the loss of funding or resources. This results in beneficiaries painting a perfect picture of the programme or saying what they think the evaluator ‘needs’ to hear. By making use of various participatory approaches that enable the beneficiaries to “own” their impact stories without fear, we aim to improve validity of qualitative findings even further.
Tshikululu invests in building transparent relationships with our partner organisations. These relationships are characterised by honest conversations and feedback. In instances where it makes sense to exit a programme, we notify the organisation and ensures that there is a well-communicated and structured exit process, including an exit grant where appropriate. This process can help to eliminate fear from the partner, and manage inevitable donor-donee power dynamics.
The participation of all relevant stakeholders in an evaluation is a key enabler for effective utilisation. Without meaningful participation of the key players, buy-in and support for both the process, from inception to close-out, the findings may be limited. Evaluation stakeholders may include clients, end-beneficiaries, programme partners and/or sector experts. These stakeholders can be involved at different stages of the evaluation process.
As an example, it is important to involve the client during evaluation design as they need to provide input on the methodology and instruments. During data collection, the partner must be involved for accurate feedback on implementation as they have the on-the-ground programme experience. The end-beneficiaries are best placed to report on what the impact of the programme is, as they were the ultimate recipient of the intervention.
Charlene Lackay, Nation Builder 21 Oct 2019
Lengthy reports that are too complicated and full of technical jargon run the risk of not being used. Ever wonder what happens to your 100-page evaluation report? It could be sitting on someone’s shelf collecting dust or perhaps acting as a nice doorstop! It’s important to produce simplified, accessible and comprehensible evaluation reports. If people can read, enjoy and understand your evaluation report- utilisation is improved. We achieve this by providing concise versions of evaluation reports in addition to a detailed final report.
Evaluators sometimes don’t think about the practicality of their recommendations – they take a “Rolls Royce” approach when there is only budget, time or energy for a Volkswagen! In order to improve evaluation use, recommendations must be feasible and consider the amount of time and resources required to implement. “As a means of enhancing utilisation, Tshikululu workshops recommendations with the client to establish the extent to which they are feasible and economic.
Although there may be external factors that lead to evaluations not being utilised (e.g. change of government policies, change in social investor strategy), evaluators do have many ways of influencing uptake of findings for the better. By taking a utilisation-focused approach when carrying out evaluations, we aim to do just that.
