Today we will talk about models of monitoring and evaluation of projects that cannot be fully measured only by numerical KPI due to the strong influence of social effects. This problem is faced by many non-profit projects, governmental projects, as well as commercial ones, which are undergoing significant scaling up, have corporate social responsibility programs in their portfolio, conduct complex marketing campaigns or are seriously engaged in the issue of involving their large staff. There is no single way to evaluate such projects. However, framework methodologies are being developed that allow you to realize your own monitoring and evaluation plan for such organizations. Today we will meet one of them. It is called “Developmental Evaluation.” This method appeared in the literature in 2008 for the first time at the McConnell Foundation.
What is this method used for?
This is not just a table for evaluating some indicators, as we are used to seeing in KPI systems. The problem to be solved is much deeper. Here it is important for us to answer the question: what barriers are facing the project, what difficulties are there, what resources are needed, which way should it develop and what remains to be learned in the process.
When will this method be useful?
If the project is being implemented in a highly changing, volatile reality. For example, it depends on the moods of certain social groups that are influenced by a mass of factors. Today you are developing a product that has just come into fashion. Let it be delicious burgers cooked using new technology. And tomorrow, fashion is changing and heading towards a healthy lifestyle. Were you ready for this? Or you produce a product that, according to recent studies, is very environmentally friendly, and conscious consumers buy it despite the cost. But a month later, a study comes out that completely refutes your basic message.
Project results are unpredictable, and the relationships between its constituent parts are non-linear. I think you often come across this phenomenon when conducting marketing campaigns. It would seem that the advertising message is well thought out, but there is a social effect that completely changes the result. Just recall the consequences of the campaign of a well-known manufacturer with the slogan “Change from the needle of male approval to the face.” The marketer (well, unless he is conscientious and not a bully) did not plan at all to get instead of sales growth lengthy discussions of the problem of women’s rights in social networks and the media.
The problem you are trying to solve is complex and depends on the cooperation of various organizations and population groups. This is very characteristic for non-profit projects. For example, you deal with poverty. Not only the impacts are not linear: sometimes charitable help instead of a direct result gives the opposite effect in the form of people’s unwillingness to work and leave this circle, but even without the joint efforts of the authorities, citizens, banks and business, the problem cannot be solved. At the same time, each of them has its own, sometimes overlapping interests and uneven resources.
Your project is innovative. Such enterprises constantly need to analyze public opinion, predict technological and fashion trends. What was sold well half a year ago can go out of use very quickly. Remember at least a cryptocurrency hype that has come to naught.
Data collection
Gathering information to use the method is very difficult. Here it’s not enough to take the numbers from the database; one survey or feedback form is not enough. You need to build relationships with your customers, partners or employees, depending on who the study is dedicated to. You need to collect information bit by bit: from questionnaires, formal and informal conversations, analysis of their requests. Gathering information will take considerable time, so you need to start as soon as possible. You will need to understand how your customers think, what are their prejudices, decision-making models. The focus of this study is not reports for the top manager, but the search for areas for development and training.
What kind of employee is needed to apply the method
In addition to experience in evaluating and understanding your area of activity, he will need the following qualities:
Strategic and systemic thinking
The ability to highlight patterns in social systems
The ability to build trust, actively listen and ask questions
The ability to separate one’s beliefs and one’s worldview from analysis
Method Plan
First of all, it is very important to pose the right questions for research. You cannot learn everything and everything around. You need hypotheses that you want to accept or refute in order to make further decisions. For example, will the trend be stable in the coming year? What factors can level it? What real wishes will your marketing message bring to the public? What happens if you take a specific step? How will people’s behavior change? These are very broad questions. Your task is to clarify them as much as possible during the study.
Focus on what the purpose of the study is. What problem are you trying to solve? Who is the main catalyst for your audience? How to understand that the work is done and the problem is solved? What resources are needed for this? What are the pain points? Where is the energy of people concentrated? Where is the greatest potential for your influence? What are the barriers in your path (organizational, financial, social)? Who else is working on this issue and how do they do it? How are these organizations interconnected? What have people already tried to do in this direction and what lessons have they learned? Who is your audience? How are roles distributed in it? What are their expectations and fears? How do they interact with each other? How are decisions made?
Of course, you will not find the answer to all these questions if you do not build relationships with the audience. In this method, informal methods of communication work much better than formalized data collection in the form of surveys and questionnaires. However, remember that you still need to consider the time you spend. You cannot go to the cinema 24 hours a day with your clients and talk about work. Focus on the goals of communication, learn to ask questions on time, not overwhelming, but spending a minimum of time. This is art.
At the third stage, when you formulated the questions and received a different form of a pool of answers to them, you need to build a concept for further education. It should answer two key questions: which groups need training and what exactly do they need to understand? Education is here in the broad sense. This is new information, behaviors, theories and research results, marketing messages, and not just textbooks and lectures. It is about your leverage over the audience and possible changes in its behavior. While deciding issues of a social nature or the involvement of staff, you should think about training in the classical sense. What should your employees learn to be interested? What should your partners learn in order for their actions to lead to significant social effects?
This is the hardest part of the job. You need to depict your complex system, put all the information received on the diagram. Depict on it all the persons involved and the relationship. Your task is to see repeating patterns, patterns that manifest themselves in many people. The difference between patterns and standard logical models such as “if … then …” is that if a change occurs in one part of it, the entire map may be rebuilt in an unexpected way. Your task is to identify such possible influences and their consequences. Perhaps you will draw several maps, focusing on the main points where changes can occur (both those that depend on you, those that are not dependent on you, or even a combination of them).
Imagine playing with a ball of kittens. If you pull one of them, then in the classical logical model, you will take out the kitten that you pulled and nothing else will happen. In reality, you can get a couple of kittens out of the ball at once, and the rest will begin to move and move.
Are there deviations from the patterns you have identified in the system? What could be the reason? It is interesting to compare: how the methods of data collection affected your results. Do the data collected in different ways show you the different relationships in the system? Did you find amazing results in the data? Do you have any further questions?
Your next task is to determine: what is the optimal system state for you? Everything here is neither linear nor ambiguous. After all, 100% involved employees with a three-fold increase in pay are unlikely to suit you? To do this, you need to determine which system you consider ideal. And which of the maps you draw are as close as possible to them. What strategy will lead you to them? And how will you measure progress: are you really on the road in the right direction? What are the markers of progress? What are the barriers on your way?
Always evaluate group dynamics in the process. Your pattern cards may change during the implementation of the strategy. Perhaps unforeseen factors appeared, or you did not take into account all the relationships.
Simple example
Suppose it’s important for you now to solve the problem of involving staff in the work. You feel that their initiative is falling, there is no interest. Let’s start with the formulation of hypotheses:
Which of the ongoing complex of measures are most effective for involving people (this should be a specific closed list, for example: raising wages, holding corporate holidays, arranging a gym in the office, a clearer explanation of the mission and vision of the company, and so on).
What exactly do employees need to increase engagement (again, a closed list is needed: increased pay, a sense of ownership, training, career growth, and so on).
Of course, you can follow the path of formal surveys. However, they will have distorted results for two reasons: some people will not tell the truth, the survey results will not take into account the interconnections of possible events. For example, what happens if you increase people’s salaries, but will make them bear more responsibility and do routine work instead of creative work? And if in addition you give them opportunities to study? Do I need opportunities to learn without putting them into practice at work? The relationships here are far from linear.
That is why it is important for you to talk to people in person and, possibly, informally. Your task is to identify key patterns: which changes are always a plus, which can be significantly limited due to the influence of other factors, and which are a waste of money. Your ideal system in this case is employees involved at full capacity and not receiving a salary. It is clear that you will not reach such a state, but you can come closer.
Cons of this method
Of course, the method is very difficult. Its results strongly depend on the degree of development of systemic thinking of the employee responsible for the analysis. Moreover, his personal beliefs, things that he has long believed in, can distort the results. That is why the analysis is ideal to perform in a group. Beliefs rooted in society can also influence results.
In addition, the analysis takes considerable time. It is necessary to find a balance between time spent, especially on the collection of information, and the usefulness of research as such. However, many strategic issues of your organization without a systematic approach and drawing maps simply can not be solved.
