Friends talking about evaluation
In honour of World Refugee Day, Lauren and Pyching Marma (friend of String Theory) discussed doing evaluation with and for refugee and asylum seeker people. Pyching was a LEEP intern at ASRC and an evaluation intern at Clear Horizon. He works in programs helping newly arrived migrant communities build lives in Australia and is a producer at SBS radio. You can learn more about Pyching here: https://www.linkedin.com/in/pychimong-marma/
Hello Pyching! Thank you for agreeing to do this interview with us! Do you want to tell our readers about who you are?
I work at Settlement Services International as a Jobs Victoria advocate. I also work for SBS Radio - I do that for my passion. I have been in Australia since 2015, and since then, I have spent about half of my career working with refugee and asylum-seeker communities.
Do you have any hobbies?!
I rarely get spare time, I have a strong interest in politics and economics, but I don’t get much time to read newspapers. I often watch political shows online, but I reckon they can be super boring. Luckily, SBS sends me news updates often and in real time. I love reading and writing, but I don’t have time for them!
What was it like coming to a new country and learning about evaluation in a new way? What went well, and what did not go well? What lessons did you learn that you want other people to know about?
What I learned about evaluation is that it is a bunch of social research and is just a profession for those who can afford it! It takes advanced English language skills and academic skills. Migrants in Australia face challenges to survive, to earn their livelihood. They would rather choose certificate courses to earn quickly- that is why most carers (aged care, childcare, disability) are from migrant backgrounds. It is highly unlikely that a migrant would pursue an evaluation course and compete in a niche job market. It takes too long and is too hard to make a living.
In your experience seeing evaluation in Australia, what have you observed that is good and bad?
I would not say good or bad. I would say it is about effectiveness after the completion of a program. Evaluators often follow methodology strictly, but they must go beyond the methodology. They should also consider those who are not in the program. Like my current program I work for, any typical evaluator…. would collect data from participants, facilitators, and stakeholders. But what if I take interviews from a grocery seller, maybe a neighbour, someone who is passing by, who is seeing the benefits? I am talking about broadening the scope. Not only those who are delivering program, but I am also now seeing about the program through the lens of a third person not part of this program; what do they see?
As part of my job, I must go to the stores, to the market, to community centres, to a train station... The people who see me in those places often say the least in evaluation. Like, the store person who saw me when I was connecting with people - he has not said a single thing to the evaluator, but he has seen many things. People who appear insignificant, like a person at a tiny grocery store, I was delivering my program right in front of him. These people don’t get asked about the program. But they have seen a lot. Sometimes, evaluators do ask them, and they don’t stay much, but if you stay five minutes and maybe offer a latte, they might say something different.
What advice would you have for people who are evaluating programs that reach people from refugee and asylum seeker backgrounds?
It depends on the subject matter of the evaluation (i.e., whether it is a health program, economic program, religious or cultural program). But whatever the program, the evaluator should be trained in evaluation techniques and be aware of how to behave in a culturally appropriate manner. The evaluator should also be trained in trauma-informed evaluation processes/ trauma-informed service delivery. You must be cautious that the participants’ trauma is not triggered and that their emotion/feelings are not hurt.
What advice would you have for people working in evaluation who want to bring people from refugee and asylum seeker backgrounds onto their teams as researchers?
You should only hire people competent in evaluation. For instance, if someone is trained overseas but lacks local knowledge, they might benefit from being given a chance or opportunity to prove their knowledge in the local context. Do not offer evaluation roles as an act of charity. If a person does not qualify, give them charity payment, but don’t give them the job of an evaluator.
People should be allowed to work with independence and not be undermined! We know the background and circumstances of our communities' socio-economic, mental, social, and psychological issues better than others; - this has to be recognised. Let them find a way to apply their knowledge.
Do you see any way that evaluation could be more accessible?
Putting aside all the theories of evaluation, like MSC, theory of change, or whatever it is, any ordinary person can see the change, whether any intervention is working. It does not require any degree to evaluate project implementation. People can understand that this project is working. That is common sense.
If you want to dig down, we need theories, we need approaches, we need methods. But I think that common sense comes first. No technology perceives human beings. Human is the first evaluator. If they are honest, they can say, ‘This is bad. This is a waste of money. This is worth it.’
My interpretation of what you are saying is that evaluation could be more accessible if it were less academic all the time.
That is correct. But even still, institutions give the overarching arrangements.
You must have seen a lot of evaluations in your work.
Yes. Evaluation is a great thing. After completion of a program, an evaluation needs to be done. And there are chosen systems to assess these programs. But what we see of evaluation in Third World countries and what we see in Australia is different. It is like seeing a coin from a different side. We have grown up with evaluations in our home countries - since Third World countries run the programs funded by First World countries. It can be said that First World countries are consultant exporters. For example, World Bank or ADB gives a loan with the condition that loan recipients will employ their consultants for evaluation. M&E is taught in Third World countries as a subject of Development Studies, not an independent discipline.
In Third World countries, they see external consultants as a threat. External evaluators come to see, come to judge, and can make people’s lives a misery. For example, I saw 360 million USD spent on a community project that they said would change lives. This program did not go well from my viewpoint. I have hardly seen evaluators in Third World countries be honest about what they have found. I am sad to say, but this is reality. You see Professors or people who are very academic. They are doing this for money. I don’t know what drives them. They get money, and the donor gets money. The people who evaluate the program, they do not speak the truth – there is like a leniency towards whom they are hired for and for the people who are where they are from.
Some people who are evaluators can see from the perspectives of people who live in the community, who are poor and hungry. If I were hungry, I would not bother with university lectures or evaluations. I would not care what the United Nations said to me. I would only bother with bread and water. I would only care about the money.
In First World countries, the evaluator has a different role. The difference is what defines us, what makes us human beings. We all have some unconscious bias, we have a tendency, we lean into people, and we lean into beliefs. It takes us to the question; how can we avoid these leniencies? That is a hard question.
Given the challenges you describe, can evaluation become more inclusive of the voices of people from refugee and asylum seeker backgrounds? How do we as a profession need to change?
To be honest, I don’t know because I am not an evaluator. I can’t tell what needs to be changed. I don’t feel like I know what needs to be changed in Australia.
An evaluator can confirm, but because of human error, humans also conform. Humans cannot go beyond that. Technology is the last weapon humans can use to confirm - with maths, AI, and machines. AI will be able to do that with evaluation eventually.
And technology is accessible to everyone. Every time I use Google Mail, Facebook, or something entertaining. Even if I try to search for something on Google, a form pops up. It is accessible to everyone to use. Technology is pervasive in every part of our life. The corporations have those tools to gather our data and tools to evaluate. They have those algorithms. What we are going to do, and what are we not going to do? Is the Australian Evaluation Society ready to deal with this?
There might be some refugees and asylum seekers who are seeking an opportunity to explore AI or technology, which will be prominent in the future. We are going to lose talent if we send people back. Change is a constant. Whether it is a refugee or a migrant, Australia must welcome them.
There are people whom Australia must welcome, who have different skills, all the sciences that make Australia great, and will make Australia great again, we should accept. This is the truth, and this is always going to be the truth.
