|
 |
 |
| ||||
|
Table of contents
Figures and Tables
[ Top of Page ] IntroductionThis is a report of the evaluation of the Stakeholder Forum on Hearing Enhancement and Assistive Listening systems. The Forum evaluation is an integral part of the ongoing evaluation of the Demand-Pull model of technology transfer. The report presents the background, methods and results related to the evaluation of the Forum conducted on-site. T2RERC, and its partner organizations that sponsored the forum, owe the success of the event to the multiple stakeholders who enriched our self-study process by giving valuable information at various stages of this event. During the months prior to this two-day meeting, experts on Hearing Enhancement that we had identified as appropriate and valuable gave us high quality input through our interview sessions. This information framed the "whitepapers" under the four themes we had identified as significant to the problem of technology transfer for hearing enhancement. These papers fed the discussions that occurred later, at the Forum, by a larger group of stakeholders invited to participate. Their mixed perspectives generated the information needed for synthesizing "problem statements", the intended outcomes of the Forum. [ Top of Page ] BackgroundThe Demand-Pull Model and its Best PracticesDemand-Pull is one of the two approaches to technology transfer implemented and tested by T2RERC. An earlier section of this document described the Demand-Pull project's protocols. The methods under these protocols as well as the theoretical framework that drives them are drawn from best practices in industry and marketing research fields. Among such practices are: use of Kano Model for collecting and using primary market data; Quality Function Deployment for product planning; Structured focus groups and surveys as data gathering tools; and an array of subordinate tasks and practices ranging from sampling and recruiting to managing communication with and among stakeholders at significant project stages. The specific practices under the Demand-Pull model represent applications of these best practices to our project's context in a way that addresses specific disability issues and stakeholder characteristics. As we implement our project to transfer technology, our evaluation efforts focus on field-testing and validating our best practices. The validated practices are duly incorporated into the model and contribute to its ongoing refinement. The Forum - why, what and howPurpose: The Demand-Pull activities that preceded the Forum identified (a) current needs in hearing enhancement technology and (b) the relevant market and industry information, which were documented in White papers and an industry profile. The purpose of the Forum was to have this previously identified information validated by a larger stakeholder expert group interacting in formal, moderated discussion sessions. The stakeholders brought their multiple perspectives to systematically address and discuss issues surrounding current hearing technologies, and came to a consensus about valid technology needs expected to advance the state of the practice regarding hearing enhancement products. SignificanceThe Stakeholder Forum is a significant step in our Demand-Pull process of technology transfer, since it generates, validates and converges information relevant to significant technology needs. After the Forum, we develop them into statements of problems in need of technical solutions. These problem statements not only indicate current limitations in the features and functions of products (hearing enhancement, in this case), but also point to business opportunities for advanced technology solutions. In addition, they establish design and performance for these solutions and the technical specifications needed to improve the products. In a later step, other stakeholders - advanced technology developers - respond by proposing technology solutions to the stated problems in lieu of improving the products. The importance of the Forum's outcomes to the transfer process is evident. And hence the importance of ensuring the quality of the problem statements resulting from the Forum, making sure that they represent significant needs of the hearing enhancement industry. Our on-site forum evaluation constitutes our effort to achieve this by ensuring the quality of the inputs as well as the processes that produced these Forum outcomes. Procedures: Before the ForumWe began our systematic approach to quality assurance far in advance of the Forum itself. During the months prior to the Forum, manufacturing, research, clinical and consumer experts on hearing enhancement provided input through interview sessions conducted by the T2RERC. These sessions explored the current status of product features and functions and on the capabilities of their underlying technologies. Analysis of this information, along with a literature review, allowed us to identify about twelve potential topics in need of further exploration. Then, working with our partner RERC on Hearing Technology at Lexington Center in New York City, we selected the four Technology Areas deemed most important for making significant advancements in the state of the technology underlying hearing enhancement devices. The RERC on Hearing then helped us develop the "white-papers" for each Technology Area, which presented the current state of knowledge and practice for the specific Technology Areas. All Forum participants received the white papers in advance so they arrived with the same current knowledge about each Technology Area. Procedures: At the ForumParticipation in the Forum was strictly by invitation. The Forum brought together 62 stakeholders-each one expert in an aspect unique to Hearing Assistive Technology. This diverse group of stakeholders included - advanced technology developers, consumers, nationally and internationally known researchers, hearing enhancement product manufacturers, clinical and technical experts, and experts in third party reimbursement. They brought specific knowledge about hearing aid technology and assistive listening systems, their underlying technologies, or their application and use. We designed the Forum in the basic mold of our previous Forum on Wheeled Mobility, with refinements from evaluations of that forum. Trained moderators from the T2RERC led structured discussions among the participants, and their varied perspectives generated the consensus statements needed to create problem statements, the intended outcomes of the Forum. The Four Technology Areas on which the discussions concentrated were:
We conducted the four sessions simultaneously each day, corresponding to the four Technology Areas. Thus there were two sessions for each Area, one on each day. Participants took part in two different sessions, one on each day, depending on their expertise and interest. The interactions were highly structured, with moderators using pre-prepared scripts to guide the discussions. A scribe recorded notes on a personal computer for reference and display, another recorded points on a flipchart and a technical consultant aided the moderator. To ensure that our participants with hearing-impairments could fully engage in the discussions, each room had a CART system for real-time captioning and an assistive listening system with microphones for each participant. The room layouts ensured that there were no visual barriers between speakers. For each Technology Area, the consensus that emerged from the discussions of the two sessions were systematically merged, summarized and reported at a general session at the end of the Forum. [ Top of Page ] Evaluation MethodologyPurposeAs mentioned earlier, validation of the Demand-Pull model of technology transfer provides the broader context for the Forum's evaluation. Quality assurance is vital for ongoing refinement and consolidation of best practices under the model. Since the Stakeholder Forum processes represent our current understanding of best practices tested in actual use, assuring the quality of its inputs, process and outcomes, and observing these best practices in action, are integral parts of our Forum activity. It supports the ongoing improvement of both the Forum and the Demand-Pull model's cycle of operations. Thus, the purpose of the Forum evaluation was to:
Best practices in focus: what was different this yearThe evaluation focused on the validity of both input-related and process-related best practices. Such practices involved the development of the white papers, the selection and recruitment of the participant sample, selection and training of the moderators, forum structure and organization involving staff, logistical and technical support, and on-site environmental adaptations. A special set of these best practices represented improvements incorporated from valuable participant suggestions from last year's forum. These included: (a) Improved Session formats -We maintained and strengthened our structured format for the sessions. In particular we refined and brought scripts to a more standardized form, and trained the moderators in their use. (b) Enabling Consumer to participate more effectively: In the interest of drawing their best contributions, we included a pre-training session for our consumer participants using a format that simulated the sessions. (c) Raising consumer sample quality: We expanded our consumer recruitment efforts to include participants who were not only "information-rich" (expert) consumers, but also articulate contributors. (d) Raising stakeholder interaction effectiveness: We strove for better technical/laymen communication both by training our moderators and by appropriate room lay out at the sessions (e) More accessible forum environment: We selected a better venue for this year's Forum. We had all meeting rooms on a single floor potentially more accessible to participants and staff alike. Procedures for collecting, analyzing and using informationWe obtained information about the quality of our best practices from two sources. Internally, our project staff gave us their self-evaluative perceptions and judgments of the forum performance. Our external source was our participating stakeholders. They judged the validity of the Forum sessions they attended and reported their satisfaction levels about them. They also judged the quality of the Forum as a whole at the end. Additionally, a team of two evaluators made formal, on-site observations of the Forum in session for process monitoring. All evaluative information was gathered both by formal and informal modes as described in the next section. We carried the on-site evaluator observations back and forth between live sessions, reinforcing their strengths and correcting process errors. Also, we analyzed stakeholder responses and comments at the end of the day and fed them back to the moderating teams immediately, enabling them to modify their second day sessions in accordance with the needs perceived on the first day. The participants' insights into our strengths and weaknesses during the Forum, as well as our on-site observations enabled us to monitor the processes, improve them while they were still being conducted, and keep them appropriately focused and directed. In addition to guiding on-site modifications in our on site processes, systematic triangulation between the foregoing sets of data also helped us identify improvements for future events and the relevant demand-pull steps. InstrumentsSurveys during and at the close of the Forum: Participant stakeholders evaluated the quality of each individual session against their own expectations, using a survey form. They expressed their satisfaction levels about the session they attended by answering questions on a 5 point rating scale about various session aspects. They evaluated the session for content (topics relevant? discussions deep enough?), purpose (achieved?) and personal satisfaction (felt comfortable? able to contribute?). Participants also responded to open-ended questions, making additional evaluative comments. Fig.1 presents this survey and its items. Our participating stakeholders also made final evaluations of the Forum using a separate survey at the conclusion [see Fig.2]. They rated Forum organization, the quality and appropriateness of the white papers and the accessibility of venue each on a 5 point rating scale. Besides, they indicated the extent to which the Forum met our expectations for them by way of: exposing them to the state-of-the-art technology; facilitating networking, partnerships and collaborations; indicating untapped business opportunities; enabling them to shape the future of the hearing aid and assistive listening industries. Formal On-site Observation Checklist: Our "on-site" evaluation is an important part of the Forum. The team of two evaluators made direct observation of the interactions at every session, circulating from room to room, and recording the unique features of each session. We used a formal checklist [See Fig. 3] to observe what styles the moderators used, the way they used the audio-visual aids, and how effectively their team coordinated its roles to monitor discussions, clarify technical content and summarize key points. Informal On-site Observations: In addition to the formal observation by the evaluation team, there were also informal observations reported by the organizers. The spontaneous comments they captured from the stakeholders outside the sessions corroborated or clarified information that came from the other sources. Formal Self-evaluative feedback: Critical comments from project staff, including the moderators, scribes, evaluators and on-site technical team, also served to clarify or enhance the information collected from other sources regarding Forum effectiveness. [ Top of Page ] Results-What we found and how we used itAnalyses of the foregoing information revealed that the Forum was a success, an experience considered gratifying by participants and hosts alike. The results are detailed below.
Stakeholders Evaluate Forum SessionsIn all, 52- 57 participants consistently filled out the evaluation forms expressing their satisfaction levels about the session they attended and making additional comments. Overall perceptions were very positive about all sessions. Table 1 summarizes and compares the ratings across the four sessions. On the whole, moderators got higher average ratings than last year - ranging from 4.1 to 4.5 [compared to 3.8 to 4.2 from last year]. However, individual evaluations varied due to participant mix and differences in moderator styles. They ranged from reasonably satisfactory (3.4 points on a 5 point scale) to highly satisfactory (4.9 points). Two moderators scored higher ratings the second day, and the other two scored higher on the first day. Table 2 lists and groups open stakeholder comments on the quality of each session. They revealed satisfaction about most of the aspects, in particular our improved practices that related to: our structured and standardized session formats, and moderator quality; level of expertise and participation of the participating consumers; quality of participating stakeholders and contributions. Stakeholders Evaluate Forum PerformanceOn the End-of-the-Forum survey, participant evaluations of the organization of the Forum were higher than last year. Table 3 summarizes the ratings on this survey. Rating averages were 4.7 points (vs. 4.1 last year) for adequacy of the facilities; 4.8 points (vs. 4.0 last year) for accessibility of the facilities; and 4.6 points (vs. 4.2 last year) for background information provided by the white papers. The perceived effectiveness of white papers by the stakeholders is noteworthy. White papers represent significant intermediary outcomes in the Demand-pull process and feed as major input for the Forum discussions. It is also worth noting that accessibility arrangements were acceptable. This was one of the improvements we made from our last year's feedback, incorporated into this year's Forum. Our selection of venue with all meeting rooms on a single floor did in fact make a big difference in accessibility also for our own project staff with functional limitations. The table shows also that perceptions of the stakeholder groups this year were not much different from those of last year's regarding the benefits they took away with them. Different stakeholder groups valued different aspects of the forum as its strength, but the opportunity to network with and learn about the ideas of other stakeholder groups was upheld as the Forum's strength by all groups. Thirty-nine (39) out of the 45 survey responses (about 87%) indicated this as a benefit from the Forum. Networking and quality of the interactions also made a repeated appearance in the open-ended comments participants made pointing to forum strengths. Next in order to networking was "exposure to new or innovative technology" acknowledged as a benefit by thirty (or two-thirds) participants. In terms of proportions, this included most (10 out of 12) technology producers, over half (12 out of 19) of customers, so also (6 out of 11) of manufacturers and 2 out of the 3 attending resource providers. Being able to identify direction for new product development (58%), being able to identify need for new technology (50%) and being able to identify new business opportunities (one-third) were next in that order. Stakeholder comments presented in Table 4 corroborate the above results. Informal Stakeholder EvaluationsParticipant comments gathered during informal interactions with them described the various activities of the Forum with remarks such as: "the most focused meeting, with the clearest objectives", "the most constructive group meeting [he] had ever participated in," "the mix of stakeholders provided a level of insight not previously encountered", 'the sessions were illuminating". The perception on the whole was that the entire program was well run and in a professional manner. Most of the specific comments mentioned as "strengths" of the Forum reflected our best practices improved from last year, reinforcing our decision to incorporate them. On the other hand, we derived "lessons" from this year's experience in the constructive suggestions some participants made about our practices. In-house Input - Self-study CommentsSelf-evaluative feedback from project staff, including the moderators, scribes, evaluators and on-site technical team, also pointed to a high level of satisfaction on everyone's part. These comments were re-grouped under the following categories: Pre-Forum activities and Consumer recruitment; Locale of Event; On-site Project support; Forum organization and coordination; Registration of participants; Use of AV aids; Breakout sessions - structure and interactions; and Reporting Session. The "strengths" under each category reinforced both features that we maintained and those we improved from last year. At the same time, we acknowledge and appreciate several constructive comments related to improving the process further. These "lessons" related to: early consumer recruitment; early dispatch of white papers to participants; better attendance keeping; optimal room and group size; improved survey administration; effective flip-chart work; better collaboration with partner RERC; reschedule for more time at sessions as well as for report preparation; and participant cost reimbursement policies. Most of the findings discussed so far were corroborated and complimented by our on-site process observations. For example, bringing scripts to a more standardized form helped our two new moderators -one a stronger subject matter expert and the other a more experienced facilitator- to be on track at various points. Having all meeting rooms on a single floor made a big difference in accessibility. The quality of interactions in sessions revealed the quality of the participant mix we were able to achieve in the sample. [ Top of Page ] Final ConsiderationsFormal feedback about the Forum processes from staff and participants alike helped us make on-site improvements to raise the performance level of the sessions and of the Forum as a whole. Moderators made the appropriate changes from one session to the next, and we controlled, monitored and made better use of audio-visual resources. On the other hand, all end-of-the process feedback was recorded as "lessons" for future events and will be used in our ongoing planning. Back at the T2RERC, problem statements are now in development, using the rich material generated at the Forum. As the old adage goes, "the proof of the pudding is in the eating." Current perceptions by the staff and the participants about Forum effectiveness need corroboration from evaluations of the outcomes from the Forum and their value to the stakeholders. We conclude by acknowledging our gratitude to our project sponsor, the National Institute on Disability and Rehabilitation Research (NIDRR), our partner RERC on Hearing Technology at the Lexington Center in Queens, NY, our co-sponsor the North East Region of the Federal Laboratory Consortium, the Research Triangle Institute (RTI) that provided us expertise on tech transfer from the federal labs, and, in a very special way, all of the stakeholders who participated in the Forum. [ Top of Page ] |
|
||
|
| |
