Jeff Sheldon on a Theory-Driven Evaluation Study of the Wired With Wisdom Pilot Project: Executive Summary

Introduction

A few years ago I conducted an evaluation study of the Wired With Wisdom pilot project, an Internet safety intervention developed by Web Wise Kids (WWK) of Santa Ana, California.  When I was first hired by WWK I didn’t know much if anything about on-line safety issues or that there were organizations working with parents and students to create safe Internet environments in homes. Needless to say, it was illuminating. This piece is the executive summary of the final evaluation report I wrote for WWK staff and the United States Department of Justice’s Office of Juvenile Justice and Delinquency Prevention (OJJDP). One of the innovative aspects of this evaluation is that it’s an early example of the theory-driven approach derived from Theory-Driven Evaluation Science. As a brief primer, program theory is the set of assumptions about the manner in which the program relates to the social benefits it is expected to produce and the strategies and tactics the program has adopted to achieve its goals and objectives. Within program theory we can distinguish impact theory – the nature of the change in social conditions brought about by program action and process theory – the program’s organizational plan and service utilization plan. In the theory-driven approach the impact model explicates the impact theory by describing the assumptions about causal processes through which an intervention is supposed to work, showing why the problem will respond to the action (i.e., Wired With Wisdom); the components of the impact model include the intervention/treatment, determinants (i.e., mediators and moderators), and proximal, intermediate, and distal goals/outcomes. Similarly, the process model explicates the process theory by prescribing the actions that are required to solve the social problem (i.e., unsafe Internet environment), showing the components and activities that program designers and key stakeholders see as necessary for program success; the components of the process model include the intervention and service delivery protocols by which the implementing organization assesses, enhances, and ensure its capacity, and by which the program implementers recruit, train and maintain both competency and commitment.   That said, if there is interest I will parse the final report so you can see the theoretical models used to inform the evaluation design. As always, I welcome your comments to refute, offer alternative explanations, or confirm and add to what I’ve written; references are provided if you’re interested in the source material. Read on…   

Background 

According to many observers, Wired With Wisdom is a risk prevention program with immense potential for improving Internet safety through the education and skill - building of parents, caregivers, and teachers. Even so, these same observers agree the program will have negligible impact unless parents and caregivers can be successfully recruited to engage in its use and act on the education and training it provides. It seems intuitive that parents with children who use the Internet at home would be concerned about Internet safety and would want to take action to minimize risk. Yet, as theory dictates and reality confirms there is more to successful recruitment of parents than simply making a program available at no cost and urging its use. To test these assumptions Web Wise Kids was funded by the Department of Justice’s Office of Juvenile Justice and Delinquency Prevention to pilot Wired With Wisdom over a two – year period and evaluate: 1) the sources and methods of parent recruitment, and 2) the outcomes of program use. The objectives of the evaluation were three – fold. First, evaluating recruitment strategies through the lens of social science and program theory to determine which sources and methods within those strategies influenced parent use of Wired With Wisdom; second, evaluating the outcomes of program use; and third, making recommendations for the future adoption of successful recruitment strategies by schools of similar type and parent demographics.

Findings

Contrary to the literature on prevention program enrollment indicating enrollment rates of 20 to 25 percent of an eligible population, 157 out of a pool of approximately 3,246 (4.84 percent) eligible parents across the six school sites registered to use Wired With Wisdom. Of the 157 parents who registered, 102 of the approximately 3,246 eligible parents (3.14 percent) used the program. Confirming Brown et al.’s (2004) study on parent recruitment into prevention programs, this study found that those who logged onto the web site were a self-selected parent sample with high income and education levels. Contrary to Gordon’s (2000) findings that many parents remain ignorant of how technology works and are fearful of technology, based on self –ratings about their understanding of the Internet most parents seemed to be savvy about computer technology and the Internet and therefore unafraid to use the program. As Spoth and Redmond (1993, 1995, 2000) pointed out, prior use of a parenting resource such as the Internet or other on-line programs predicts future use of those services. This may in part be explained by the demographics of those who used Wired With Wisdom, all of whom indicated there is at least one computer in the home with Internet access. Confirming Garvey et al.’s (2006) findings, parents also used the program because: 1) they wanted to learn more about their children’s Internet related behavior; 2) there were benefits to participation; and 3) they perceived their children as being at elevated risk for some sort of behavioral problems related to Internet use. Barriers to using Wired With Wisdom were minimized because: 1) parents did not need to commit a great deal of time to the program compared to other prevention programs; 2) they could do it on their own time; and 3) they could do it in their own homes (Snell-Johns et al., 2004; Spoth et al., 1995). For those who completed demographic data, but did not actually use Wired With Wisdom, Heinrichs et al.’s (2005, 2006) explanation is that beyond fear, other important variables may have been the demand of getting into the program (i.e., registering and completing the survey), the recruitment context, or recruitment methods. 

Lessons Learned

In general, parents can be influenced by their children to use Wired With Wisdom thereby creating a safe Internet environment if: 1) the program is couched as a school – based activity, and 2) they are asked to take part. If, however a child insists they use the program, insistence per se would likely have little influence on the parent’s decision - making process. For most parents a letter from the school principal, other parents, and the endorsement of a parent/parent-teacher organization likely would not be highly influential unless parents have positive feelings toward and trust in the principal, the other parent(s), or the organization.  Regardless of who tells them about or endorses the program, the more important influence seems to be the trust rather than the feelings parents have in those people and organizations. Certain correspondence methods would also likely be influential. A letter from someone associated with the school (e.g., the technology teacher) would be more influential than not, a phone call would likely not be influential, but an e-mail would be more influential than either a letter from someone associated with the school or a phone call. Regardless of initial correspondence source or method, most parents would not need a follow-up; it likely would not be detrimental if follow – ups were made. Whichever correspondence method is used, if it is positively worded it will be more influential than either one worded negatively or one that is purely informational. A correspondence that is both positively worded and informational would probably be the most influential. Another strong source of influence would be parents’ perception of risk. Most parents would be influenced by their concern about their child’s on-line activities and would be likely to use Wired With Wisdom because they have their children’s best interest in mind. Although their children’s best interest would not be the only source of influence it would be an important factor in their decision making process. Parents would also be influenced by perceived benefits to using the program. Classroom incentives for the children of parents who use the program would be of negligible influence. Parents would likely be influenced if they believed they would know as much about the Internet as their children resulting from use of the program. Parents who take part in programs that address pressing social problems would likely be willing to use Wired With Wisdom especially if it is understood to be a program that addresses pressing social problems. However, many parents might be influenced to use the program simply because of its perceived inherent qualities. Parents likely would not use Wired With Wisdom unless they understood its purpose, were sure it would be useful to their family or had time. Feeling intimidated by the Internet, unless they have little experience with it, and being embarrassed about trying the program because they do not already know much about it would be negligible barriers.

Conclusions

Like most applied research or evaluation studies, the findings from this study are simultaneously discouraging and encouraging. The discouraging news is that the combined effect of the limitations made it difficult to determine with any degree of precision the sources and methods of influence within a parent recruitment strategy that generalize to parents across all demographics and school types. The encouraging news is that the vast majority of parents in the sample were concerned about Internet predation and other risks associated with Internet use. More importantly, they wanted to address those risks and viewed Wired With Wisdom as a way to take action in that regard. Insight was gained about how schools of different types (i.e., public and private) recruit parents into prevention programs, and the sources and methods within their respective parent recruitment strategies that influence parental use of a prevention program such as Wired With Wisdom. The common – sense recommendations that came out of this study can be used to enhance existing parent recruitment efforts at other schools to increase the likelihood more parents will use the program to create a well - managed home Internet environment and safety plan. Last, and probably the single most important outcome of this project, 102 more parents used Wired With Wisdom and benefited from the education and training it provides. The outcomes of program use are unknown, but it is hoped that there are 102 homes with a well – managed Internet environment and a safety plan in place, and that there are as many if not more children safe from predation, bullying and other risk factors.       

Recommendations

The following recommendations are based on the cross - case analysis about known sources and methods of influence and the findings from school sites viewed as exemplars in their ability to recruit parent use of Wired With Wisdom. Recommendations are offered as general guidelines for other school sites regardless of parent demographics or school type and as potential guidelines in making policy decisions. The over-arching recommendation is to use a simultaneous, three – pronged approach to parent recruitment - students, trusted others and e-mails/letters – paying particular attention to the content of correspondence parents receive about the program, targeting small sub-samples of parents one at a time and making frequent contact over a short, limited period of time. 

1. During the process of informing students about the program at school, instill a sense of excitement, enthusiasm and urgency that they will transmit to their parents.
2. Coach students how to approach their parents. Having them ask, not insist, and making it very clear the program is a school – related activity will help students influence their parents.
3. Any personal – based sources of information should come from people or organizations viewed by parents as highly trusted members of the school community.  
4. Trusted others include the school principal (head of school), the technology teacher and head of the parent/parent –teacher organization. Who correspondences come from is a judgment call, but its importance should not be underestimated and time should be taken in their consideration. 
5. The best method for contacting parents is e-mail. However, some parents either do not have access to e-mail or a list of parent e-mails is incomplete so a letter from a trusted person associated with the school (e.g., principal, head of school, technology teacher, et al…) should be sent home. Trusting hand-delivery to students is not recommended.    
6. Follow – up beyond the initial correspondence is unnecessary. However, it would not be detrimental provided an availability of resources and follow-up is limited in number and frequency.
7. Written correspondence should be worded positively (i.e., not using scare tactics such as “if you don’t use this program your children will meet certain peril and doom”) and be informational (e.g., “we have an opportunity to take matters into our own hands through becoming better educated about Internet risk through Wired With Wisdom”).
8. In written correspondences, make sure parents are informed about the risks that come from Internet use through use of facts and real – life stories/anecdotes. 
9. In written correspondences use language that taps into the natural concern parents have for their children and their tendency to act in their children’s best interest. Be clear that use of Wired With Wisdom will help them address their concerns and its use is in their children’s best interest.  
10. In written correspondences let parents know another benefit that comes from using the program is the acquisition of knowledge that equal the knowledge their already children have about the Internet.
11. In written correspondences, especially the first one, state that Internet predation amongst other risks is a pressing social problem and that Wired With Wisdom is a constructive way of addressing that problem.
12. In written correspondences, especially the first one, make sure the purpose of Wired With Wisdom is clearly stated.
13. In written correspondences, especially the first one, make sure parents are told, specifically, how the program will be useful to their families.
14. In written correspondences, especially the first one, make sure parents know how much time each module takes, the estimated total time and that modules don’t have to be done at once; their use can be parsed out in 20 – 30 minute blocks of time at any time of day or night from any computer with access to the Internet.
15. Concentrate on small sub-samples of the target, grade – level, parent population one at a time (e.g., parents with last names beginning A – H, J – S, T – Z, etc…) to ensure each group gets maximum recruitment effort.    
16. Make frequent contact over a limited time - period. 
17. It is unnecessary to provide classroom incentives. Resources would be better used for program promotion.

References

Brown, J. B., Winzelberg, A. J., Abascal, L. B., & Taylor, C. B. (2004). An evaluation of an Internet-delivered eating disorder prevention program for adolescents and their parents.

Garvey, C., Julion, W., Fogg, L., Kratovil, A., & Gross, D. (2006). Measuring participation in a prevention trial with parents of young children. Research in Nursing & Health, 29, 212 – 222.

Gordon, D. A. (2000). Parent training via CD-ROM: Using technology to disseminate effective prevention practices. The Journal of Primary Prevention, 21(2), 227 – 251.

Heinrichs, N. (2006). The effects of two different incentives on recruitment rates of families into a prevention program. The Journal of Primary Prevention, 27(4), 345 – 365.

Heinrichs, N., Bertram, H., Kuschel, A., & Hahlweg, K., (2005). Parent recruitment and retention in a universal prevention program for child behavior and emotional problems: Barriers to research and program participation. Prevention Science, 6(4), 275 – 286.

Snell-Johns, J., Mendez, J. L., & Smith, B. H. (2004). Evidence-based solutions for overcoming access barriers, decreasing attrition, and promoting change with underserved families. Journal of Family Psychology, 18, 19–35.

Spoth, R., & Redmond, C. (1993). Study of participation barriers in family-focused prevention: Research issues and preliminary results. International Quarterly of Community Health Education, 13(4), 365–388.

Spoth, R., & Redmond, C. (1995). Parent influence to enroll in parenting skills programs: A model of family context and health belief predictors. Journal of Family Psychology, 9(3), 294 – 310.

Spoth, R., & Redmond, C. (2000). Research on family engagement in preventive interventions: Toward improved use of scientific findings in primary prevention practice. Journal of Primary Prevention, 21, 267–284.

Spoth, R., Redmond, C., Haggerty, K., & Ward, T. (1995). A controlled parenting skills outcome study examining individual difference and attendance effects. Journal of Marriage and the Family, 57, 449–464.