Evaluation

Quantitative Evaluation

This national Initiative provides an opportunity to conduct a multi-site evaluation with common data elements across sites. The 15 demonstration sites have collaborated with EPPEC to conduct core evaluation activities and will continue to do so throughout the intervention projects’ 4-year period. On an ongoing basis, sites will report client, provider, organizational setting and intervention characteristics, as well as the results of behavior change surveys and other core measures.

Data from the sites are centrally collected at EPPEC to document implementation and outcomes of the interventions over time for both provider and patient change, and to assess the cost of the interventions.

Expectations for Data Collection

The multi-site evaluation plan encompasses data collection across sites on multiple levels. Collection of core indicators is required at the organizational, intervention, provider and patient levels. Collection of core indicators is also encouraged for all trainings conducted as part of funded interventions. Because the burden of data collection on each site and on the respondents is a concern, the core indicators required are a focused and efficient set. Depending on individual plans for evaluation, sites may participate in selected optional data collection activities including measurement of training outcomes and in-depth measurement of intervention content.

Qualitative evaluation

Objectives

  1. Document each phase of the intervention process (planning, implementation, outcome) across the 15 sites in order to determine what factors facilitate or deter the success of an intervention, and to determine best practices guidelines for future replicable models.
  2. Assess the feasibility and appropriateness of the intervention from a range of perspectives including clinic administrators, health care providers and patients.
  3. Assess patients’ beliefs and perceptions of Prevention with Positives interventions (e.g., strengths, weaknesses, and satisfaction) and what they consider to be the most critical program facilitators for behavior change.

Methods

The following methods of data collection will be used for the qualitative process evaluation:

  • Site Assessment – by thoroughly reviewing each intervention proposal.
  • Key Informant Interviews – individual in-depth interviews with intervention staff and patients/clients.
  • On-site Clinic Observations – observing the clinic environment and patient flow.
  • Monthly conference call observations – logging problems and resolutions as they emerge.

Data Collection Timeframe

Data collection will be carried out in two distinct phases.

  • In Year 1, EPPEC staff conducted on-site pre-implementation interviews with the principal investigators and intervention staff in each of the 15 sites between January and April of 2004.
  • In Years 2 and 3, EPPEC staff will collect data (via an annual site visit and participant-observation of cross-site conference calls) for the process evaluation.

Site visits are expected to average 2–3 days depending on the intervention and clinic size, and may occur concurrent to the site visits conducted by EPPEC Site Liaisons. Site staff shall assist EPPEC staff in making arrangements to facilitate data collection. Sites will be responsible for assisting in the recruitment of appropriate individuals for interviews e.g., providers, key staff members and patients, as well as providing appropriate space to conduct the interviews.

Data Analysis

EPPEC’s qualitative team has begun to code and analyze transcribed interview data. The multiple forms of data will provide detailed information about each of the site’s Prevention with Positives interventions, effective program elements, organizational features and a descriptive context for understanding the quantitative data.

Last modified: January 20, 2011