When is dissemination of an Internet-delivered intervention successful?
Before determining if an Internet-delivered intervention has been disseminated successfully, one should first determine when an intervention is successfully disseminated. This is not as straightforward as it may seem. Absolute figures may cause unrealistic optimism, since thousands of visitors are not uncommon for websites in general. Relative figures, on the other hand, may cause unrealistic pessimism, since they may be very low if one relates number of visitors to Internet penetration rates. To estimate the reach of a recruitment approach and the generalisability of results, it is important to report the target group, the number exposed to recruitment, the number who respond, the number eligible, and the number who actually participate (Graham, Bock, Cobb, Niaura, & Abrams, 2006). Furthermore, one should determine a final target in advance. There is, however, no “default” cut off point for such a target to determine successfulness of dissemination, since this may differ per intervention (e.g., an intervention aimed at reducing alcohol use among high school students versus party drug use among school drop-outs). Issues such as the general accessibility of the target group and the prevalence of behaviours among the target group should be taken into account when determining a final target for successfulness of dissemination.
The determination if an Internet-delivered intervention has been disseminated successfully does not only depend on your final target, but also on the denominator you choose to estimate reach. In a study by Graham et al. (2006), for example, Internet users were recruited based on use of the terms quit(ting) smoking or stop(ping) smoking in a major search engine. When a user clicked on a link to the intervention in the results of a search engine query, an intercept page appeared inviting them to participate in the study of Graham and colleagues. If they accepted, three questions were asked to determine preliminary eligibility. Reach estimates vary depending on the denominator selected: 2.7% of all Internet users seeking cessation information; 6.9% of those who demonstrated preliminary interest in the study; 13.7% of those who were eligible to participate; 21.1% of those eligible and recruited; and 51.3% of those consented.
This also raises an interesting statistical issue; since it is difficult to identify the population to which samples refer when there is no clear sampling method (i.e., everybody can access your intervention). Questions need to be asked to determine eligibility to participate. It remains unclear, however, whether missing data regarding eligibility is due to visitors’ perception of non-eligibility (e.g., parents visiting the intervention because they are interested in the subject or non-smokers accidentally hitting on a website aimed at smoking cessation) or actual drop-out of potential participants. Appropriate methods of analysis are needed to deal with the vast amount of missing data (Christensen, Griffiths, & Korten, 2002) and to determine the recruited sample size from your target group.
The difficulties regarding measurement of online reach are not limited to the field of health promotion. The Interactive Advertising Bureau Europe (IAB Europe) and the European Interactive Advertising Association (EIAA) have announced to develop a measurement standard for website reach. In this project, denominated as Measurement of Interactive Audience Project (MIA Project), several online measurement methods will be evaluated (Marketing Online, 2007).
Implications for practice
It is recommended to report the target group of your intervention, the number exposed to recruitment, the number who responded, the number eligible, and the number who actually participated to determine whether dissemination has been successful. These numbers should be compared with a final target which should be determined in advance and which depends on the intervention (e.g., its subject and target group). Despite this dependence on the intervention itself, one may look at other Internet-delivered interventions to get an idea about the order of magnitude of a suitable final target.
Implications for future research
Analyses regarding data from Internet-delivered interventions are not as straightforward as they may seem, since data can be missing for several reasons (e.g., visitors which do not perceive the intervention as being relevant to them once they logged on or visitors whose needs are fulfilled halfway the intervention). Although it is probably too conservative to consider all missing data as drop-outs, this is recommended as long as appropriate methods of analysis are lacking. Furthermore, due to the openness of Internet as a medium, not all visitors of your intervention are necessary members of your target group (e.g., parents, people who are generally interested in the topic of your intervention). These “participants” should be excluded from data analysis.
References:
Christensen, H., Griffiths, K. M., & Korten, A. (2002). Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. Journal of Medical Internet Research, 4, e3.
Graham, A. L., Bock, B. C., Cobb, N. K., Niaura, R., & Abrams, D. B. (2006). Characteristics of smokers reached and recruited to an internet smoking cessation trial: a case of denominators. Nicotine & Tobacco Research, 8, S43-48.
Marketing Online. (2007). Europese standaard voor meten online bereik in de maak [European standard for measuring online reach in preparation]. Retrieved October 26, 2009, from http://www.marketingonline.nl/nieuws/ModuleItem52230.html.