Realistic Evaluation

 I have decided to share a piece of my research on Realistic Evaluation

1. THE HISTORY OF REALISTIC EVALUATION

Whereas evaluation has become a ubiquitous practice in civil societies, governments and a topical issue in the scholastic fraternity, the present generation of practitioners owe a lot to the think tanks in various successive ages who polished the subject to what it is today. Evaluation is defined as “the use of social research procedures to systematically investigate the effectiveness of social intervention programs”, (Rossi, Lipsey & Freeman, 2004). Alkin (2004) defined it as “social research applied to answering policy-oriented questions”. This paper looks at the birth of this approach from its conception, development as it passes through many hands, its core fundamental principles, how it was used in a case study and finally its inherent weakness.

Astbury and Leeuw (2010) indicated that Edward Suchman had highlighted the importance of opening and empirically testing the black box of social programs in the year 1967. Black box is when practitioners view the programs in terms of the program’s effects and explain clearly the components making up a program (Kazi, 2003). Roy Bhaskar in 1977 and Rom Harre in 1972 laid the foundations of realistic evaluation although some of the elements were still lacking such as explicitly opening and explaining the black box.

Astbury and Leeuw (2010) indicated that Weiss’ writings in 1972, 1995, 1997 and Wholey in 1979, 1983 and Chen in 1989 and 1990 corroborated the proceeding generation of proponents of realistic evaluation by further emphasising the need for understanding the theories underpinning all social programs. In 1987 Chen and Rossi introduced the term mechanism in explaining the causality relationships that were in social programs and by the year 1990 Chen would popularise the term mechanism. In 1990 Michael Scriven would iterate the need to unpack the black box and inspect categorically the components of the black box.

Ray Pawson who is one of the co-founders of realistic evaluation has a background in sociology and applied much of his research in criminology (Astbury, 2013). By 1978 it had become clear that he had become an open critique of empiricist measurement and explanation methods that were used in Sociology. In 1980, while working as a lecturer his writings were already gravitating towards realistic evaluation as an alternative logic to realistic epistemology. 

Astbury (2013) posits that Pawson’s work was further galvanised with his collaboration with Nick Tilley. In 1982 they had their first joint publication, Monstrous Thoughts, in which they delved into the weakness of subjective epistemology. They challenged relativism and argued that an alternative philosophy was needed for knowledge development which would avoid “epistemological extremes of positivism and radical constructivism”.

Astbury (2013) reveals that a nascent crop of scholars were beginning in Europe which espoused the realistic evaluation methodology.  This movement was further strengthened by the writing of other philosophers like Mary Hesse in 1974, Rom Harre in 1972, Roy Bhasker who wrote the Realist theory of science in 1974 and another one in 1974 called the Possibility of Naturalism. In 1994 Pawson and Tilley produced a very crucial piece of work entitled “What works in evaluation research”. Although the writing seemed to awaken criminologists to the dangers of experimental design as evaluation methods, their works would later on pave way for a more definitive school of thought.

Pawson and Tilley’s work continued to build its momentum. In 1992 they presented their arguments in the Yearbook of Correctional Education. In 1993, the duo would present their lines of reasoning at a bigger gathering of the British Sociological Association where they presented the ‘OXO, Tide, Brand X and New Improved Evaluation’. In 1994 they wrote contending that the quasi-experimental approach to evaluation had a dearth in explaining causality (Pawson and Tilley, 1994).

In 1994 Pawson and Tilley did an experiment of the practical use of the realistic evaluation in a closed circuit television (CCTV) (Pawson and Tilley, 1994). This article demoed how evaluation could be conducted in a real life setting. Although targeted at criminology, they further demonstrated for the first time context –mechanism- outcome pattern configuration which would later form the spine of their ideology.

Astbury (2013) reveals that in 1995 Pawson and Tilley presented their work at the International Evaluation Conference held in Vancouver, Canada. Two years later, a milestone would be reached as Pawson and Tilley published their first official book together, Realistic Evaluation (1997). In 2006, Pawson added another dimension to realistic evaluation which is systematic reviews. These are essential components in informing policy development as opposed to stand alone evaluations. Evidence based policy making uses lessons gleaned from evaluations to make relevant policies and subsequently practice that which is advocated and this happens on a continual basis.

1. PRINCIPLES OF REALISTIC EVALUATION

Realistic evaluation is based on the premise of realism which asserts we live in a real world and everything change that happens in real and can be really measured (Pawson and Tilley, 2004). Realistic ontology probes questions such as “what works for whom in what circumstances and in what respects, and how (Pawson and Tilley, 2004). Realistic evaluation seeks to improve knowledge gleaned from the interactions of mechanism and contexts to derive certain outcomes (Byng, 2011).

• Programs are theories

According to Pawson and Tilley (2004), realistic evaluation is an amalgamation of theory driven approaches. Being theory incarnate, any program implementation is in essence testing the theory on what a particular change occurred in a program, even though theories may not be so explicit. In realistic evaluation, an evaluation is then to make theories within a program applicable by coming up with hypotheses clearly showing what would work in a particular circumstance. It is the real implementation of a program that tests these hypotheses. The programs start in the heads of some policy architects who then pass them on the hands of evaluation practitioners and sometimes they end up in the hands of program subjects. Normally, such conjectures begin with a notion of what causes certain behaviours or some social inequalities and then hypothesise variations made on these trends. New ideas are then brought into the same systems to destabilise it and then rebalance it. The “Dishy David Beckham” theory is a good example of a program instituted to give growing teenagers with good role model. This was after some health education theorists had cited that televisions and magazines were full of ills and bad examples of stars that would influence that teenagers into many social ills.

• Programs are active

Pawson and Tilley (2004) discovered that when a combination of reasoning and resources is employed in an intervention, a change is imminent. The outcome produced would subsequently require active involvement individuals. Pawson and Tilley (2004) give the following scenario of fluoridation of water and publicly brushing teeth twice a day. The former is a good example of a passive programme which works whenever one drinks water and applies arbitrarily to the whole population. It does not require anyone to actively engage with it, but even while inactive or ignorant of it one drinks water that is fluoridated. However the later needs action. One has to choose to brush teeth twice a day, a direct heed of a health promotion or may choose not to follow the message conscientiously. Program interventions are inherently active and require persons involved to be active participants in the program.

• Social programs are embedded

Realistic evaluation assumes the principle of embeddedness (Kazi & Rostila, 2002). Causal powers are not embedded in the behaviours and events variables of individuals but in relations and organizational structure making up a social system (Kazi & Rostila, 2002). Events and social conditions are altered by the interactions of the total social system of social relationships (Pawson and Tilley, 2004). Realistic evaluation takes a look a social system that could be families, cultures or community and delves into researches to find causalities of certain trends. It is worth a while to note that these systems interact with many other societal factors and that all factors are embedded at different levels. That said, it is imperative to observe that one phenomenon may have a plurality of causes and lead to many other consequences to that effect (Westhorp, Prins, Kusters, Hultink, Guijt & Brouwers, 2011).

While program inputs can result in a great change in the society, the impact can also be amplified depending on the circumstances of the denizens involved. Pawson and Tilley (2004) give an outline of a prison educational and reformation program in which the success of the program is dependent upon  the capacities of trainees and facilitators, the relationships between these two entities, capability of the prison to support the program as well as society’ ability to support the ex-convicts upon their return. Clearly the success of the program depends on many other factors lying at different strata of programs.

• Social systems are open

Pawson and Tilley (2004) revealed that realistic evaluation postulates that social programs operate in on open system implying that programs change a system and a system may change a program. A plethora of external factors make the programs permeable and elastic. Programs operate in an uncertain environ which is subject to unprecedented changes owing to politics, technological changes, personal whims and a lot of many other unexpected events. In essence, programs cannot be static or assumed to operate in a constant mode and such externalities always see programs changing in their nature. Although the principle of realism has an allowance to self-transformational programs, it is imperative for proponents of realistic evaluation to know how such externalities affect the program delivery. It is the work of a practitioner to produce empirically verified causal relations between contexts and mechanisms (Morén, & Blom, 2003). The CCTV was installed as a deterrent mechanism to crime, however it was no sooner that the CCTV was installed than the thieves devised other means of circumventing detection by the system. The thieves could use other corners where cameras were not focussing. Other interviewed offenders revealed that cognitively, they told themselves that the picture quality was too poor for it to prosecute them and hence every program is open to other changes as is seen in the risk perception of these offenders (Gill& Turbin, 1999).

• Context-mechanism-outcome-pattern-configuration

According to Pawson and Tilley (2004), realistic evaluation assumes that programs are complex interventions being implemented in a complex web of social reality. In social science complex matters are broken down by frameworks into simpler key processes and elements. Realistic evaluation as a complex process is simplified into, Context, Mechanism Outcome Configurations (CMO).

If certain mechanisms are to be active in a particular context the program will end up with a certain outcome. Mechanisms may interact with their context and have outcome A and intersect in another setting and result in Outcome B. Clearly, evaluation needs a good knowledge of causal factors and that outcomes are not just some linear function of any given combination of context and mechanism (Pawson and Tilley 2004).

• Context

A context refers to the environmental matters that enable a mechanism which range from organisation, staffing, history, pedigree, culture and beliefs (Pawson and Tilley, 2004). It is generally assumed that the mechanisms are active under certain conditions or contexts. It is equally important for practitioners to know about context that enable certain mechanism in order to know the possible outcome. These are responsible for triggering mechanism in a program and for constraining the working of mechanism.  The question, for whom does this program work talks to contextual factors (Westorp et al., 2011).

• Geographical settings

In Australia people of Aboriginal tribe were synonymous with petrol sniffing (Pawson and Tilley, 2004). Programs were designed to curb that social ill and the intervention included building cultural pride amongst this populace as mechanism. What context could trigger Aboriginal pride? This would best work in a context of a community that was located far from the petrol stations.

• Community settings

Pawson and Tilley (2004) give the scenario of property marking. In a certain program that wanted to discourage theft, engraving the name of the owner on the property was proposed. However it was seen that mechanism of crime deterrence would only work contingent to a particular context. The context needed a smaller community with higher rates of publicity. This would mean should a property item go missing with a name, it would soon be discovered courtesy of good communication and that community being small. On the contrary a large community cannot deter crime even if property was tagged.

• Historical periods

In social programs such as crime prevention, context plays a pivotal role. When a crime prevention method is put in place it works for a short period before criminals invent a new method to avoid detection (Gill& Turbin, 1999). Older methods’ context cannot offer deterrent mechanism for crime, whereas new methods may for a while curb crime.

• Past events

Pawson and Tilley (2004) indicate that September the 11th 2001 has since changed the perception of Muslims especially by Americans. They have become acrimoniously identified with terrorism and treated wih contempt in some circles. The uptake of services becomes difficult in areas where racism against them exists. If ever there are any social program that needs to be aimed at them their response to it is difficult given the context of Muslims shunning away from active community participation due to their fears.

• Settings

Pawson and Tilley (2004) argue that some settings just make it impossible to fire some mechanism. Empowering rape victims in a prison set up is hampered by the very prison setting. Empirical evidence seems to suggest that empowering prisoners who are rape victims was made hard because the very context of a prison was intrinsically disempowering.

• Population groups

Pawson and Tilley (2004) pointed out that gender plays a pivotal in social programs. In certain agricultural programs due to the nature of the trade, very physical tasks are done by men and thereby limiting the role of women to support in cooking and other simpler task that support men. In such a context where women play a passive role triggering mechanism of equipping them with agricultural sustenance skills is hampered by their subordinate role in farming.

• Mechanism

Pawson and Tilley (2004) postulated that realistic mechanism refers to the hidden causal factors in a program that is likened to the movements of the hands of a watch which are a result of many processes running underneath. It is argued that it is not programs that work in essence but resources that are employed which in turn enable their subjects to work. In a social program, there is a glut of processes, influences, powers that work in certain context to generate an outcome and are called causal mechanisms. It must be noted that mechanisms cannot be observed but can only be hypothesized and tested in any social program. Furthermore mechanisms do not work alone in a program but can be triggered in certain contexts which the evaluator must know. A realistic evaluator is not only focused on an outcome but in attributing a certain mechanism to the outcome. The evaluator asks himself or herself the question, what caused this outcome? What triggered this mechanism? In what context was this mechanism fired? Pawson and Tilley had a case study for a CCTV installation in the car park where crime was rampant.  This intervention triggered a number of mechanisms.

1. Deterrence mechanism

Potential offenders could feel that they could now be seen on cameras and then choose not to offend again. In this case the CCTV deters criminal activities (Pawson and Tilley, 2004).

1. Displacement mechanism

The offenders may choose to ply their criminal trade elsewhere because of the perceived risk should they continue to operate in this particular car park. In case where they decide to carry on with crime, they may be displaced to another car park without a CCTV (Pawson and Tilley, 2004).

1. Target hardening mechanism

The CCTV may attract car owners who are risk averse and very cautions enough to put away items that attract thieves. This mechanism makes stealing difficult on its own and so coupled with other mechanism such as detection, may make criminals shun this place. In a nutshell, decision making is affected by people’s attitudes, experiences and resources.  That said a combination of reasoning and resources available makes up a mechanism.

Reasoning + Resources = Mechanism (Pawson and Tilley, 2004).

• Outcome patterns

Programs are carried out in very different contexts and this implies that different mechanism will be activated by different contexts (Pawson and Tilley, 2004). There is a highly likelihood of very different combinations of contexts and mechanisms which will give rise to different outcomes. A combination of various mechanisms and contexts will bring along a glut of intended and unintended consequences to the program. It is therefore expected that the programme will have a number of possible outcome patterns accruing to the contexts and mechanism combinations.

Pawson and Tilley (2004) look at possible outcomes following a CCTV installation in a car park. Car owners will feel safer and in that parking and the outcome ) -will be increase in turnover. The deterrent mechanism will result in ) –a decrease in crime rate. Increased car owners moving in and out of the car creates a natural surveillance. This will make car owners ) slack and the car park busy ).

It is the thrust of realistic evaluation to continuously refine their programs through theory testing (Pawson and Tilley, 2004). All the possible outcomes of realistic evaluation try to come up with a context mechanisms combination that has optimal outcome and sustainable as well. For instance, a property marking program to reduce burglary can only work if all property is well marked, in a small area where criminals have few alternatives, a community with tracing potential and very good publicity avenues (Laycook, 2008).

1. A practical use of the realistic evaluation methodology

Martin Gill and Vicky Turbin implemented a realistic evaluation as defined by Pawson and Tilley (1997) in a closed circuit television in England. This case study looks at many other aspects of crime prevention including neighbourhood watch and electronic article surveillance. Prior to this study, many evaluations had tended to focus primarily on an outcome of evaluation and relegated mechanism and contextual factors. In this case study, Gill and Turbin focus on evaluating crime prevention strategies within a particular context. This case study was conducted in the United Kingdom (UK) in 1999 in a jeans and casual store over a year for a company with ten stores and over two hundred and fifty employees (Gill & Turbin, 1999).

• The data collection methods

Gill and Turbin (1999) outlined that the project involved data collection for analysis and subsequently for evaluation purposes. The four main sources for the collection of data were the customers, staff, regular stock takers and the shop thieves.  In this period of 12 months, 480 customers were interviewed. At each shop, 120 customers were interviewed before installation and another 120 during the installation of the CCTV. On top of that interviews were done to the employees of the 2 shops in three phases. The first one was done before CCTV installation, second one was done whilst the CCTV was in store and the third one was done after the CCTV had been uninstalled from the shop.

Interestingly, 38 shop thieves were also interviewed.  Of these, 5 were from probation services and the other 31 and the other 2 were already participating in the same research. The flaw in this sample of thieves is that it not representative in the sense that not all thieves participated for obvious reasons that they were never caught and some did not admit the offences levelled against them. However it was the aim of the study to get to understand the offenders view.  Their responses would then be compared to the other data gleaned from customers, staff and stock takers to get the full picture of the impact of the CCTV (Gill & Turbin, 1999).

• CCTV and contextual issues

A focus on conceptual issues around the CCTV plays a crucial role in realistic evaluation. The interviews were done on staff members to gain an insight into a number of issues ranging from attitude towards a number of security measures and experiences of crime at their workplaces (Gill & Turbin, 1999). The interview duration was 1 hour and all were tape recorded. A total of 25 staff members were interviewed before the CCTV installation and after the installation this figure rose to 27 (Gill & Turbin, 1999). After the removal of the CCTV a total of 9 were interviewed and most notably, they had been interviewed before CCTV and while CCTV was there and the rationale was to get changes in their perception at each stage of the CCTV phase.

From the interviews it emerged that the shops from the locations experienced the same problems of theft especially considering the frequency at which such matters occurred (Gill & Turbin, 1999).   The responses showed that the staff at Leeds had arrested more offenders than their counterparts in Sheffield. Both shops alluded that more than once per week there were reports of theft. Most notably, staff also showed that the crime problems also posed serious concerns from their perspective. It emerged that some staff had chosen not to confront the thieves for their personal safety. There were also the sentiments from the staff that the other staff members were not concerned about the problems of thieves and others blamed the then company policy for being too lenient for these offenders. It appears that the thieves knew of these flaws and would capitalize on that to the point of being for bold daringly frequenting the same shop over and over.

• The testing the CMO configurations

In this case study, the mechanisms were developed before the data collection was started(Gill & Turbin, 1999). However it is worth to note that they were not so directly related to the contexts or the potential outcomes of the CMO configurations which is a weakness of this particular approach. Although CMO configurations may be so easy to come up with, it is plausible to postulate that researchers may miss some uncommon ones or inadequate data may leave the researchers with no option but to reject them.

This case study had a number of CMOs as indicated below;

• CMO 1

This mechanism shows that the CCTV gave the staff the bravado to approach the offender because they were backed by the video footage. Subsequent to that, the next step of approaching the thief would deter them from stealing from the shop. It is imperative to note that is only in the context of staff that feel threatened by thieves where this mechanism is fired (Gill & Turbin, 1999). The aforementioned is further corroborated by the comments from the staff in the interviews who felt that their confidence to approach has been bolstered by the CCTV. “You know it's [CCTV] there if anything is going to happen. It

just makes you feel, you know, more comfortable approaching the situation” (Gill & Turbin, 1999).

• CMO 2

The introduction of CCTV reduced the staff vigilance because they had the idea that it would reduce the levels of theft(Gill & Turbin, 1999). However it was noted that theft started increases because staff vigilance had decreased. The staff members confessed in one of the interviews that they had high expectations of what the CCTV could do until they saw the continuing of theft in the shop even right in front of the CCTV. Some staff said that they felt a huge relief because they would no longer have to deal with thieves again. It was later on discovered by the staff that the CCTV was only an aid to security and not an absolute measure on its own.

• CMO 3

This CMO suggests that CCTV increased customer satisfaction although there was not much evidence to support that. More so, there would be a natural surveillance because if there were many customers thieves would be naturally deterred to steal from such an environment. This mechanism could only be activated by a situation in which the customers are already having a low satisfaction level. Although the evidence of increased customer satisfaction was not substantiated, the data gathered revealed that the number of customers increased though there is no safe basis to attribute this to CCTV(Gill & Turbin, 1999).

• CMO 4

This CMO suggests that CCTV enhances the deployment of staff to area where suspicious behaving individuals are. When they are there they can easily offer them help thereby deterring would be thefts. However this is only applicable in contexts in which the staff members are unable to identify suspicious behaviour. Most of the staff members had said that they were able to identify the suspects from the CCTV and yet this mechanism could work the other way (Gill & Turbin, 1999).

• CMO 5

This CMO suggests that CCTV assists in identifying trouble areas which are targeted by offenders. The context in which this mechanism works is where staff is unaware of any trouble areas. When the CCTV was installed the staff already knew these areas making this mechanism not to work and this was common on both shops (Gill & Turbin, 1999).

• CMO 6

This CMO works in context where there is lack of evidence to prosecute an offender and is triggered by mechanism in which a CCTV clearly captures the offenders in the act (Gill & Turbin, 1999). The outcome can only be more certain if the offender is caught red handed in the shops and then detained and handed over to the police. If the picture is unclear and the offender has escaped apprehension prosecuting the accused may almost be impossible. If this mechanism works coupled with this context there will be successful convictions and more offences would be deterred. There is quite empirical evidence to prove that this works although there was one case the offender was caught in another shop elsewhere and in another, the police were called to come and arrest the offender.

• CMO 7

This CMO seems to suggest that if customers dislike store surveillance and they notice that in the shop are CCTV monitors they are likely to go and buy elsewhere (Gill & Turbin, 1999).

The results are likely to have a ripple effect; there will be less natural surveillance which will compensated by the fact that there will be more staff visible in the shop for security purposes. However CMO 3 seems to prove that wrong.

• CMO 8

From the interviews conducted on 480 customers it appears that as many as many 65% do not even see the CCTV from a sample of 64 customers or 35% saw the CCTV (Gill & Turbin, 1999). Of the 480, 70% welcomed the CCTV as a good initiative and 4.5% showed dislike of the CCTV and 25.2% did not have an opinion. There was definitely evidence that the bulk of the clientele like the CCTV disproving the CMO as incorrect. More notably the CMO 7 and 8 are unlikely to be triggered because 65% of the customers did not even see the CCTV.

1. CRITICISM LEVELLED AGAINST REALISTIC EVALUATION

Whereas the works on Pawson and Tilley (1997) had critiqued the works of Popper and Campbell it is comes as no surprise when their methodology is also found to be having flaws. The realistic evaluation methodology has flaws as Gill and Turbin used it in the CCTV case study. It would appear the CMO configurations pose a lot of challenges especially is a real life scenario as was experience by Gill and Turbin.

• The choice of mechanisms used in an evaluation

In their case study, Gill and Turbin outlined that although they chose a number of mechanism that would operate in particular contexts, still there was a limitation to that.  Some of the mechanism were quite difficult define precisely as to come to a conclusive outcome. In their case studies they clearly outlined that it was hard for the mechanism to cover all probable ways in which the CCTV was going to have an impact on the store thereby compromising the final outcome of the evaluation. It was also noted that another weakness of realistic evaluation was that one needed a very thorough understanding of the processes involved in a particular context in order to know appropriate mechanisms for a particular evaluation. As the item 3.3 below reveals, contextual factors are not easily identified before the start of the process making it harder even.

• Difficulty in data collection

In their case study it was outlined that in some cases CMO configuration’s data is very difficult to obtain. It is one thing to identify a mechanism to use in a particular context and yet another to have done for it. Depending on the case study, the nature of the study done by Gill and Turbin was particularly difficult to get offenders who or those accused because it is normal for offenders to circumvent any incriminating evidence.

• Challenges with contextual issues

From a theoretical perspective the question on what works in what circumstances appears easy on paper but in reality it is more complicated than postulated in theory. The challenge is that some of the matters will come to light way after the research had started. In this case it was noted late that the staff attitudes towards the whole CCTV system and their expectation about the CCTV were crucial effect on the outcome. When the staff expected the CCTV to do everything, it infact made their less alert to theft and then theft started increasing and compromised the outcome. Such contextual matters are hard to know beforehand and when not factored into account compromises the efficacy of a realistic evaluation.

• Limited scope of realistic evaluation

It appears that realistic evaluation is suitable for the evaluation of micro projects and not macro projects (Pedersen, Rieper & S?rensen, 2005). Policy makers, senior government officials, civil societies cannot use the realistic approach at a national level and worse still at an international level to measure the effect of an intervention. It appears that small shops, car parks and other micro projects have been the recipients of this evaluation.

• There is a limitations to targets

The CMO of realistic evaluation limits the targets to individuals rather than corporate bodies and structures (Pedersen et al., 2005). Criminals , social workers, victims, customers have often been the target of the data collection and sampling methods whereas aggregated entities such as companies, markets, bodies have not been included due to the limited nature of this evaluation method (Pedersen et al., 2005).

• Limited to social aspects

Realistic evaluation seems to be limited to social aspects such as measuring crime reduction through certain mechanism in particular context (Pedersen et al., 2005). Other fields that are technical or which may involve sophisticated technological inventions and improvements are quite difficult to evaluate because of the inherent nature of realistic evaluation.

• Lack of specificity

Realistic evaluation looks at a lot of CMO configurations. It looks at an outcome of one particular context and mechanism. That said, it looks at different dimensions through which problems may be addressed such as crime, education and unemployment can be addressed at a particular time and in a particular location (Pedersen et al., 2005). However there are no CMOs made up of players in different ages and locations to look at the social problems.

• Low participants in evaluation

It has been realized that there a few key players in the issues that are being evaluated by the CMO (Pedersen et al. 2005). Now this makes it hard to get willing participants in the data collection methods and may skew the data in a certain way if there is less representation by a certain key group.

• Lack of highly Influential individuals

Most of the people that participate in the evaluation and data collection are ordinary people who are on the ground such as the policemen, social workers, teachers, nurses (Pedersen et al., 2005). However the minister who sits in the cabinet or the police chief, or a president cannot be considered in matters that happen on the ground. A police minister cannot give his opinion on how safe he feels in chasing after robbers because it is not in his scope of work and he has no idea. These are very influential people and yet they are left out of the data collection or interviews for the CMO and yet they make big decisions about social issues affecting the society.

1. CONCLUSION

Realistic evaluation is an evaluation methodology polished by Ray Pawson and Nick Tilley and looks at what works particular contexts, a characteristic idiosyncratic this methodology. The two authors owe a lot to a preceding generation who continually worked on the subject, until it was passed to them as a rough diamond which they later polished into what it is now.  Although this methodology worked in certain case studies conducted by explaining the black box paradox, it has its own weaknesses that other authors have revealed.

REFERENCE

Alkin, M. C. (Ed.). (2004). Evaluation roots: Tracing theorists' views and influences. Sage.

Astbury, B. (2013). Some reflections on Pawson’s Science of Evaluation: A Realist Manifesto. Evaluation, 19(4), 383-401.

Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363-381.

Byng, R. (2011). What makes realistic evaluation? Fam Med, 43(2), 112-113.

Carlsson, S. A. (2003). Advancing Information Systems Evaluation (Research): A Critical Realist Approach. Electronic Journal of Information Systems Evaluation, 6 (2), 11-20.

Coryn, C. L. (2007). Revisiting realistic evaluation. Journal of MultiDisciplinary Evaluation, 2(3), 143-149.

Gill, M., & Turbin, V. (1999). Evaluating Realistic Evaluation: Evidence from a Study of CCTV. Crime Prevention Studies, 10, 179-199.

Kazi, M. (2003). Realist evaluation for practice. British Journal of Social Work, 33(6), 803-818.

Kazi, M. A. F. & Rostila, I. (2002). The Practice of realist evaluation in two countries. Paper presented at the European Evaluation Society Conference, Sevilla, Spain, October 10-12. 2002, p1-19.

Kazi, M., & Rostila, I. (2002). The practice of realist evaluation in two countries. In European Evaluation Society Conference.

Laycock, G. (2008). Special Edition on Crime Science. Policing, 2(2), 149-153.

Morén, S. & Blom, B. (2003). Explaining Human Change.On Generative Mechanisms in Social Work Practice. Journal of Critical Realism, 2 (1), p37-60.

Pawson R and Tilley N (1994) What works in evaluation research? British Journal of Criminology 34(3): 291–306.

Pawson, R. and Tilley, N. (1997) Realistic Evaluation, London: Sage Publications.

Pawson, R., & Tilley, N. (2009). Realist evaluation. H.-U. Otto, A. Polutta & H. Ziegler (red.), Evidence-based practice: Modernising the knowledge base of social work, 151-180.

Pedersen, L. H., Rieper, O. & S?rensen, E. M. (2005). Is Realist Evaluation a realistic approach for complex reforms? Paper presented at the Nopsa conference in Reykavik, Iceland, on 11–13 August 2005, p1-16.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach. Sage.

Stame, N. (2002). Why a movement for theory in evaluation? Fifth Conference, Seville, 10-12 October 2002, p1-17.

Tilley, N.  (2000). Realistic evaluation: An overview. PDF paper. Presented at the Founding Conference of the Danish Evaluation Society, September 2000.

Virtanen, P. & Uusikyl?, P. (2002). Exploring the Missing Links Between Causes and Effects. New Conceptual Framework for Understanding Micro- Macro Conversions in Programme Evaluation. A paper to be presented in 5th EES Conference in Seville, Spain, 10-12 October 2002, p1-15.

Westhorp, G., Prins, E., Kusters, C. S. L., Hultink, M., Guijt, I. M., & Brouwers, J. H. A. M. (2011). Realist Evaluation: an overview. In Report from an Expert Seminar with Dr. Gill Westhorp. Wageningen UR Centre for Development Innovation.