Lessons Learned and Recommendations

All projects benefited from formative work and process evaluation in that each gained information about clients and service provision at their respective agencies. Although significant outcome findings varied, valuable services were provided for clients in need, and productive working relationships were formed between local service providers and researchers.

Based on our experience working on other collaborative projects, the SCHEP experience reflected several challenges to moving a successful consortia forward. The following lists key elements of what did and did not work in undertaking this collaborative consortium. We conclude with recommendations for future collaborative efforts.

  • Populations at risk: The SCHEP consortia served four distinct populations at high risk for HIV transmission in California: drug using street youth, women of color, alternative school high risk youth, and gay men.
  • Funding intervention plus evaluation: Not only did the projects reach targeted high-risk communities in California, funding evaluation in addition to program intervention is a cost-effective way of monitoring interventions, building CBO infrastructure, promoting researcher collaboration, and extending resources in the community.
  • Formation of the consortia: Despite administrative hurdles and barriers in start up, the project brought together two large bureaucracies and successfully forged links with service providers and researchers.
  • Project management: The CAPS Program Manager played a critical role in providing overall guidance for projects, providing program design consultation, meeting facilitation, and project interaction. Consortia leadership is needed throughout and flexibility is a key requirement to make a collaborative effort work. We found that having one key contact in the Project Coordinator was essential to maintaining communication with each participant, as well as important in establishing subcontracts, handling accounting and budget matters, helping problem solve, and organizing consortia meetings.
  • Technical assistance: CAPS was able to provide differing levels of assistance to projects throughout the contract period, especially in the areas of group communication, program design consultation, data collection, budget management and contract assistance.
  • Flexibility: Flexibility of funders allowed projects to modify research designs as necessary.
  • Project redesign: Most projects had to modify their designs at the beginning of the contract period. All four projects engaged in some form of project redesigning, mostly due to the short turnaround time given to respond to the RFP and the shortened length of the contract period (21 months).
  • Contingency funds: These discretionary funds proved to be an excellent source for supplementing essential intervention-related activities that could not be anticipated within the original project budget proposal. Contingency funds were distributed to cover incentives, survey activities, and related intervention activities to three of four projects. This fund was important for providing flexibility when project design had to be modified.
  • Collaboration: Collaboration between researchers and service providers facilitated information sharing, helped build CBO infrastructure and incorporate evaluation into program activities.
  • Communication: Individual and group meetings, correspondence and reporting provided consistent and timely information on project collaborative working relationships.
  • Lessons learned: Lessons learned from this incarnation can lead us to a more effective and useful version of collaborative intervention and evaluation efforts.

Though the results of the SCHEP collaboration process are positive, we did experience some obstacles in bringing together different organizations from geographically distant parts of the state.

  • Formation of the consortia: Managing the project through two large bureaucracies required extensive administrative legwork in order to help the subcontracts proceed smoothly. This challenge was most evident in the first period of the contract with the establishment of signed subcontracts and the limits imposed on billing periods.
  • Uneven funding periods: The project timeline was as follows: 4 months, 12 months, 5 months. These uneven periods without the possibility of rollover of money from one period to the next didn’t match the needs of programs or research activities. This caused projects to try to spend down artificially in one project period, while operating under-funded in the next.
  • Timeline: The turnaround time for proposals (one month over the winter holidays) didn’t allow for application by a large pool of participants from whom to select four projects (Nine projects applied, four were funded).
  • Delay in contract signing: The delay in contract signing by the State Office of AIDS hindered projects in getting off the ground as originally anticipated.
  • Oversight from the funder: Overall programmatic contract monitoring from the State Office of AIDS was absent for the majority of the SCHEP project. The original program monitor left his position in April 1996, one month after projects were funded, and his replacement was not hired and assigned until the last months of the project (July 1997). We received contract and administrative support from the State throughout, but lack of programmatic oversight prevented us from seeking out expertise at the State level.
  • Political process: Political process at the State level caused a reversal in the decision to fund Harm Reduction Central in Los Angeles after the State participated in proposal review, selection, negotiation, verbal approval and participation in SCHEP orientation. Five months into the contract period the State rescinded approval. Los Angeles’ project was able to successfully complete a modified evaluation despite this problematic setback. This situation points to the difficulties of political controversy in conducting scientific endeavors. This also highlights a need for early identification of potential problems by the funder.
  • Lack of unifying theme: Serving four distinct at-risk populations presented significant barriers to information sharing and building ties between projects. Although occasional project meetings held with participants consistently received good reviews, consortium participants tended not to contact each other in between meetings for any purpose. We found that CAPS became the initiator of information sharing from one project to another, rather than the projects collaborating to solve problems or share information together.
  • Orientation period: The orientation period for projects was not adequate to both unify the consortium and properly assess areas where front-end TA could have helped.
  • Electronic communication: Electronic communication between projects and SCHEP management was never properly established and remained nonexistent throughout the contract period.

Programmatic suggestions and project design:

  • Focus the project on one issue and make it a unifying theme for all consortia participants. For example, require a topic or research population in the proposal and fund four projects which are focused on different aspects of that topic or population.
  • Ensure that program intervention and evaluation fits into local community planning defined needs.
  • Allow adequate lead/proposal submission time. Allow at least two years from the beginning to the end of the evaluation projects. The contract between the State (funder) and the University (administrator) needs to extend several months before the research for adequate planning, RFP and proposal review time, and several months after the research for dissemination activities.
  • Capitalize on site visits as a means of understanding project context, providing timely problem-solving and offering technical assistance.
  • Continue to support flexibility in research design and periodic TA for projects.
  • Dissemination and process-oriented lessons learned should be a group focus.

Collaborative partners:

  • Require that researchers have HIV/AIDS prevention research experience.
  • Require that CBOs have some familiarity with program evaluation and research.
  • Flexible project management and project coordinator positions are important in the operation of the consortium.
  • Define the role of the funder(s) in consortia formation, consortia process and technical assistance.
  • Training/orientation period should be adequately timed in order to assess participants’ evaluation knowledge and skills; provide time to refine research questions and allow for provision of TA, if needed.
  • Group interaction should be frequent throughout the project period. For example, rather than one site visit, four group meetings and a monthly newsletter, hold an extended orientation period, convene at least three group meetings, plan at least two site visits during the length of the project, and schedule regular group updates via conference calls and newsletter.

Funds:

  • Reserve supplemental funds for computer systems training and setting up email or related, effective electronic communication.
  • Contingency funds should be made available for projects in order to fund unexpected costs in intervention and research activities.

Last modified: February 24, 2011