Evaluation of a Report on the BladeRunner and Picasso Café projects

This is an analytical review of an existing evaluation report that was based upon an evaluation at the BladeRunner and Picasso Café projects located in Vancouver, BC, Canada.?The original report was prepared by the Social Research and Demonstration Corporation as a pilot study of youth programs for several governmental boards and various other interested parties.?While the BladeRunner project also operated in several other cities, the report concentrated on issues central to the operation in Vancouver and only addressed the outlying programs in a cursory fashion.?The Picasso Café is a single site program.?The original report was prepared after an evaluation of the programs was undertaken during 1998 and 1999.?It was issued in March 2001 (https://www.srdc.org/english/publications/eyouth.pdf).?I do not know the reason for the substantial delay between the completion of fieldwork and the issuance of the final report.

The principal goal of the original evaluation was to identify areas within the programs that could contribute to the success in “mainstreaming” troubled street youth.??BladeRunners is a project whose primary mission is to convert street youth into productive members of society by providing them with mentoring, job skills, and other social service support programs in a construction – manual labor trades environment.?Whereas, the Picasso Café is a similar, yet less structured program situated in a small restaurant that was designed to provide opportunities so that street youth may obtain industry training within a semi-formal setting.?The program at Picasso Café contained less mentoring and the social programming was less intensive then what is available through the BladeRunner program.

The research question addressed in the evaluation is a classic example of an attempt to understand what is going on inside the “black box” at the programs.?The evaluation was not an attempt at assessing the net impact of the programs or the full effect of the program outcome on either one participant or on society, in general.?The stated goal was to “help policy-makers and program practitioners both develop a better understanding of the issues involved in evaluating programs for disadvantaged youth and gain insights into some of the factors which might contribute to program success” (p. 3).??Because of the overall foci on factors that could lead to programmatic successes, the report has an air of implementation research study and dedicated considerable space to the underlying activities at the program management level.?

The paper also drew upon a lesson-drawing theme from the policy literature.?While not quoting a specific source, the idea that the evaluation of the two programs could lead to greater understanding of good policy initiatives is more like lesson drawing and less like an outcome or impact study.?I think this unique approach to sharing research information is a grand gesture towards improving the utility of evaluations for a wider population.?Unfortunately, the evaluation did not delve in any significant manner into the politics or economic issues associated with the programs.

The unit of analysis for the evaluation was the entire program, not the individuals receiving the program interventions or society at large.?The individual participants were not analyzed in any attempt to understand individual human behavior or the impact/outcome of the program on individuals.?Individuals were interviewed, observed, surveyed, and limited outcome analysis was performed (general descriptions of workers who had remained on the job, received pay raises, etc) that helped to give meaning to the success factors for the whole program, but the report was – clearly – about the programs and not the underlying individuals.?

The evaluation used case study methodology to get in-depth understanding of the programs and to better use qualitative techniques in an environment where - due to a limited number of observations - statistics and quantitative research would not be statistically significant.?The researchers also made attempts to create or validate a counterfactual by analyzing the data and interviewing dropouts, rejected participants, and other individuals who might have “negative” or less motivated responses (i.e.; no longer employed by the program, etc).??In addition to intensive review of records maintained by the programs, the evaluators also visited the programs several times, participated in group discussions, interviewed participants, job providers, and other interested parties including governmental officials.?The evaluation also briefly discussed the emergent technique as a method of analysis; basic level research was performed to help create deductive inference towards theory or guiding principles.?After the basic understanding was generated through initial interviews and preliminary research, more in-depth reviews were completed to help provide a more crystalline picture of the overall operations.

The case study process was not randomly applied to a variety of either programs or individual participants.?The programs were, in fact, chosen for review because of their highly esteemed positions in the community.?Success was expected at both programs and the fiscal difficulties and restricted access to records at the Picasso Café were not expected and, therefore, received greater scrutiny.?

The evaluation clearly indicated that the goal was not to fully assess the net impacts or program outcomes, but to help policy makers (and other stakeholders) better understand the underlying principles so that new programs would be more successful, the reviewed programs could focus their efforts more efficiently, and to disseminate to the wider public a snapshot of the successful initiatives from each program. The findings were grouped into three areas; program participation and content; program management and structure; and relationships between the program and external institutions.?

Program participation compared the programs with other youth mentoring and job development type programs.?Key areas of success included turning points – a term for when a street youth has an epiphany or otherwise figures out that their path to greater opportunity is not usually found in an inner-city alleyway; few street youth are able to successfully complete any program without first realizing that they need help.?The report mentioned a common theme in these types of programs whereby the participants frequently “take two steps forward and one back” thus, making sustained efforts that much more important.?Other key areas of success included screening, attractiveness of the program, and mentoring.

By screening applicants, the program officers could increase success by ensuring that the applicants had, indeed, made a life altering change and that efforts in redeeming the participants would not be wasted on those individuals deemed less deserving of the full attention and efforts of the program initiative.?This selection screening, obviously, increases success rates by weeding out undesirable applicants.?To some extent, the ethics of not providing support to every applicant could be questioned, but the true reason that screening is effective is that it helps to allocate scarce resources to those individuals more prepared to succeed.?

While screening keeps out the bad seed, having an attractive program attracts butterflies.?Both the BladeRunner and the Picasso Café benefited from having a program that is attractive to that part of the population potentially seeking work and/or a turnaround in their lives.?Construction and foodservice jobs are reasonably plentiful and have relatively low barriers to entry.?It is clear that neither program would have the success that they have had if the job programs they offered were for sanitation workers or pottery repair technicians.?The allure of the jobs available to program participants upon completion helps to maintain the success and attractiveness of the work programs at both BladeRunner and Picasso Café.

To some extent, the greater success of the BladeRunner program was attributed to a better and more thorough mentoring program.?The literature in the area of mentoring indicates that intensive and continued mentoring is more successful then infrequent or sporadic mentoring.?The Picasso Café did have some structures and components for providing mentoring, but staff turnover, management difficulties and other operational issues prevented the Picasso Café from providing as extensive mentoring program as did the BladeRunner program.?The counselors at BladeRunner were frequently former BladeRunners and many also had extensive industry experience with more of a vested interest or even elevated interest approaching stakeholder status.?Additionally, the BladeRunner program was co-situated with other social programs that allowed for greater integration into the lives of the participants.?The duration of the BladeRunner training program was also longer then the average tenure at Picasso Café.??

The program issues were the central focus of the report, but the evaluation did address the management and structure of each program.?The BladeRunner project was located at a variety of other locations outside Vancouver and those outlying locations were included in the review to the extent that they either contributed or detracted from the program success.??Whereas the Picasso Café is a single site program and interaction with “franchised” outlying programs was not an issue.??The key factor in program management was at the BladeRunners, the local coordinators were very involved in the programs.?By visiting the work sites and taking a very intense interest in the success of the program and the individuals, the coordinators were more successful then their counterparts at Picasso Café.?Additionally, the BladeRunner program was more stable in financing and employee turnover rates were lower.?Picasso Café had suffered financial difficulty and even closed for a short period of time.?Both projects required intervention from the funding agencies at local and national levels, but it seemed that the troubles had more of a negative effect on the Picasso Café’s operations.

Lastly, the findings referred to the relationships with the external community.?Youth work programs motivate employers under three different theories.?Altruism, individual self-interest, and collective self-interest are the three principal arguments raised in the paper.?The literature indicated that altruism rarely is sufficient motivation for a program to achieve sustainability.?The motivations must include some interest for either the self or the community.?Difficulty arises in the self-interest argument when free riders are considered.?In the restaurant, program participants remained “employees” of the café for the duration of the program until they were released to participate in a final assignment at a local restaurant.?The restaurants would be reluctant to invest in a program participant who would leave to another competing facility.?So, in the café, the goal was to permanently place the participants.?However, the unionized nature of the construction industry allowed the concept of collective interest to motivate the potential employers.?Once the BladeRunner was “in” the union, it benefited the union and all the unionized workers.?There was greater incentive to participate for the unionized workshops.?It would be interesting to see how a Picasso Café would fare in a highly charged union restaurant town such as Detroit or New York.?Additionally, at some of the other BladeRunner sites where unionized construction is not as abundant, the job placement success factor is not as high as it is in Vancouver.

The case study approach used to review these two programs left many unanswered questions and did not allow for uncontroverted statements as to the precision of the assessment. ?Issues such as counterfactuals or alternate hypothesis and many internal validity bias issues were left unanswered or with little detail to support the findings or assessment as presented.?Granted, the review was done on a preliminary basis with limited funding, but the depth of the understanding was left weakened by the limitations.

Because the review was somewhat of a pilot study designed to help understand the programs with a cursory intention of providing limited guidance and not as a full-blown impact or outcome assessment, the inaccuracies, to some extent, can be overlooked.??The report highlighted many of the deficiencies in design and cost (both time and money) was a primary limiting factor in expanding the assessment and performing a more thorough evaluation.?

The project was a review undertaken over a period of several months with follow-up activity several months later.?The overall time committed to the project barely exceeded one year.?With youth development and job work projects, it is difficult to make complete assessments of the program success without more in-depth panel studies over a much longer time period.?The failure to incorporate greater temporal analysis into the review was a major flaw (and funding was given as the primary restriction from expansion of the review).?The report did reflect upon some indicators of success.?Yet, to fully assess an impact or outcome, pre-test and post-test analysis would need to have been incorporated into the design with far more follow-up detail.?

Because the report was not trying to make a causal inference as to the actual program outcomes or net impact, internal validity issues are less important.?The evaluators were simply trying to focus their efforts on identifying some key components of the programs that lead to some success in improving the lives of troubled youth.?However, some general threats to internal validity might have some bearing on the discussion.??Obviously, something other then the job program could have changed the lives of the youth participants.?The report focused on the concept of turning points and that youth had to have had an epiphany of sorts before being able to really turn things around.?Maybe this turning point is what improved their lives and not the actual program.?To some extent, history and maturation also fall into this category.?Because the youth participants were actually growing both personally and professionally, that growth and maturity might have had more impact then the program.?Also, eliminating drug and alcohol dependence were two a priori stipulations for entry.?Removing two barriers to success from the participant’s lives prior to entry into the programs might also have had a significant impact that was not quantified in the evaluation.?Also, because the youth were mostly at the bottom of life’s cycle, any improvement or regression to the mean would have resulted from almost any program intervention.

In addition to the individual factors that have influenced internal validity, the research instrument and methodology were flawed in that they did not seek out additional counterfactual opportunities such as program dropouts and rejected participants (mortality, attrition, selection bias).?The failure to incorporate some of these attributes into the research design surely made the findings somewhat flawed or suspect.?Because the programs were in “competition” with each other, there could have been some societal threats to validity such as rivalry or competing interests between the two factions. ?Additionally, the interviews and focus on the operating managers at the programs influenced the outcome because all the participants had vested interest in the positive outcome of the final report.

In addition to the internal flaws mentioned above, difficulty at the Picasso Café site - with staff turnover and access to data key challenges - made it less likely that all the available information was reviewed.?In order to gain a countervailing perspective, the investigators made some efforts to contact “retired” participants, refused or removed participants, and others who might have had a negative voice or could provide a different viewpoint.?Also, some of the meetings were held at conflicting schedules and the interviewers missed some scheduled meetings.?A similar difficulty with access to data or participants was not reported at the BladeRunner Projects.?However, BladeRunner has had some difficulty in the past when the program supervisors or contracted agencies were changed.?

The unit of evaluation was the program itself.?So, the sample size of the two programs would mike it difficult to make widespread generalizations to the outside community of youth programs.?Additionally, with the two programs, access to a wide variety of individuals was limited. The programs have less then 100 participants and as a subset of the “troubled youth” in Vancouver, exposure to less then 100 participants is not likely to be statistically significant.

The intervention in both programs includes positive support mechanisms that if withheld would not cause grave harm to omitted participants, so the failure to provide program intervention to everyone who seeks help is not an issue.?It also seems that both programs make every reasonable effort to accommodate all willing and able applicants.?Also, the program intervention is not at an investigational stage, unproven or other dangerous process, or otherwise risky to participants that would require special user-participant notification.?The only ethical dilemma was to screen out some potential applicants who might have benefited from the program intervention; however, to narrow the pool of applicants solely to focus resources on those deemed to be most suitable to intervention is not, in itself, unethical.

The principal foucs of the case study method was to intensively review via qualitative research, the two programs.?Using traditional observations, questionnaires, focus groups and review of documents, the evaluation was able to grasp the underlying tenets and reasons that contributed to the success (or shortcomings) of each program.?However, the addition of more statistical data (which was not collected by the programs and, therefore, mostly not available) would have provided greater validity to the findings.?Because of the small number of participants and the even smaller number of programs, statistical significance would have been difficult to ensure anyway.??The inclusion of nominal other economic factors and a more in-depth discussion of the political arena surrounding the programs would have also provided a better triangulation of methods.

The evaluation prepared by SDRC on the two Vancouver programs was, indeed, a very rich and thorough description of the two youth work development initiatives.??However, due to constraints in time, cost, and access to data, the report could have been even more thorough and detailed in several other areas.?The principal shortcoming is that of longevity.?For us to fully understand the true effects and outcomes of a youth intervention, long-term panel data needs to be collected and interpreted to help understand what the program did in the context of the youth’s lives over the long-run and not just the relatively short period they received treatments.?Additionally, the evaluation should have incorporated more discussion of a counterfactual.?Interviewing program dropouts and those who were rejected for entry would have been very illustrative in helping to understand the net effects and overall outcome of each program.?By creating a quasi control group, the concepts associated with the “bad apples” would have been considered instead of just the rosy outlook presented by the ongoing programs that had a significant vested self-interest in making their programs seem attractive and successful.

In addition to making attempts to take the research over longer periods of time and to include other non-participants in the study, the evaluation team could have used other methodologies and social disciplines to a greater extent.?By using concepts incorporated from economics, they looked at the self-interest and free rider problem inherent in the hiring of the intern-trainees at both programs.?However, failing to go further into economics left out other ideas and concepts that could have been incorporated into the discussion.?Similarly, they could have incorporated a much greater understanding of the socio-political dynamics at play in both funding the programs and how the community sees the program as a part of the greater society.?Some suggestions would include interviews of the government officials and other community organizations in the region.?Also, to investigate how the other stores and businesses near the sites and other contractors, suppliers, and other industry representatives saw the programs.??One last component would have been to make an attempt to understand the youth; why were they disadvantaged and how would social service organizations better prepare intervention programs at an earlier stage in their lives instead of when many youth already forgone conclusions.