|
 |
 |
| ||||
|
Table of contents
Figures and Tables
[ Top of Page ] IntroductionThis is a report of the evaluation of the Stakeholder Forum on Communication Enhancement. The Forum evaluation is an integral part of the ongoing evaluation of the Demand-Pull model of technology transfer. The report presents the background, methods and results related to the evaluation of the Forum conducted on-site. [ Top of Page ] BackgroundThe Demand-Pull Model and its Best PracticesDemand-Pull is one of the two approaches to technology transfer implemented and tested by T2RERC. An earlier section of this document described the Demand-Pull project's protocols. The methods under these protocols as well as the theoretical framework that drives them are drawn from best practices in industry and marketing research fields. Among such practices are: use of Kano Model for collecting and using primary market data; Quality Function Deployment for product planning; Structured focus groups and surveys as data gathering tools; and an array of subordinate tasks and practices ranging from sampling and recruiting to managing communication with and among stakeholders at significant project stages. The specific practices under the Demand-Pull model represent applications of these best practices to our project's context in a way that addresses specific disability issues and stakeholder characteristics. As we implement our project to transfer technology, our evaluation efforts focus on field-testing and validating our best practices. The validated practices are duly incorporated into the model and contribute to its ongoing refinement. The Forum - why, what and howPurpose: The Demand-Pull activities that preceded the Forum [see project description, p.2] identified (a) current needs in Communication enhancement technology and (b) the relevant market and industry information, which were documented in White papers and an industry profile. The purpose of the Forum was to have this previously identified information validated by a larger stakeholder expert group interacting in formal, moderated discussion sessions. The stakeholders brought their multiple perspectives to systematically address and discuss issues surrounding current hearing technologies, and came to a consensus about valid technology needs expected to advance the state of the practice regarding hearing enhancement products. SignificanceThe Stakeholder Forum is a significant step in our Demand-Pull process of technology transfer, since it generates, validates and converges information relevant to significant technology needs. After the Forum, we develop them into statements of problems in need of technical solutions. These problem statements not only indicate current limitations in the features and functions of products (communication enhancement, in this case), but also point to business opportunities for advanced technology solutions. In addition, they establish design and performance for these solutions and the technical specifications needed to improve the products. In a later step, [see project description, p. 2] other stakeholders - advanced technology developers - respond by proposing technology solutions to the stated problems in lieu of improving the products. The importance of the Forum's outcomes to the transfer process is evident. And hence the importance of ensuring the quality of the problem statements resulting from the Forum in the sense that they represent significant needs of the AAC industry. Our on-site forum evaluation constitutes our effort to achieve this by ensuring the quality of the inputs as well as the processes that produced these Forum outcomes. Procedures: Before the ForumWe began our systematic approach to quality assurance far in advance of the Forum itself. During the months prior to the Forum, manufacturing, research, clinical and consumer experts on hearing enhancement provided input through interview sessions conducted by the T2RERC. These sessions explored the current status of product features and functions and on the capabilities of their underlying technologies. Analysis of this information, along with a literature review, allowed us to identify about twelve potential topics in need of further exploration. Then, working with our partner RERC on Communication Enhancement we selected the four Technology Areas deemed most important for making significant advancements in the state of the technology underlying hearing enhancement devices. The RERC then helped us develop the "white-papers" for each Technology Area, which presented the current state of knowledge and practice for the specific Technology Areas. All Forum participants received the white papers in advance so they arrived with the same current knowledge about each Technology Area. Procedures: At the ForumParticipation in the Forum was strictly by invitation. The Forum brought together 75 stakeholders-each one expert in an aspect unique to Communication Enhancement Technology. This diverse group of stakeholders included - advanced technology developers, consumers, nationally and internationally known researchers, hearing enhancement product manufacturers, clinical and technical experts, and experts in third party reimbursement. They brought specific knowledge about AAC systems, their underlying technologies, or their application and use. We designed the Forum in the basic mold of our previous Forums on Wheeled Mobility and Hearing Enhancement, with refinements from evaluations of that forum. Trained moderators from the T2RERC led structured discussions among the participants, and their varied perspectives generated the consensus statements needed to create problem statements, the intended outcomes of the Forum. The Four Technology Areas on which the discussions concentrated were:
We conducted the four sessions simultaneously each day, corresponding to the four Technology Areas. Thus there were two sessions for each Area, one on each day. Participants took part in two different sessions, one on each day, depending on their expertise and interest. The interactions were highly structured, with moderators using pre-prepared scripts to guide the discussions. A scribe recorded notes on a personal computer for reference and display, another recorded points on a flipchart and a technical consultant aided the moderator. The room layouts ensured that there were no visual barriers between speakers. For each Technology Area, the consensus that emerged from the discussions of the two sessions were systematically merged, summarized and reported at a general session at the end of the Forum. [ Top of Page ] Evaluation MethodologyPurposeAs mentioned earlier, validation of the Demand-Pull model of technology transfer provides the broader context for the Forum's evaluation. Quality assurance is vital for ongoing refinement and consolidation of best practices under the model. Since the Stakeholder Forum processes represent our current understanding of best practices tested in actual use, assuring the quality of its inputs, process and outcomes, and observing these best practices in action, are integral parts of our Forum activity. It supports the ongoing improvement of both the Forum and the Demand-Pull model's cycle of operations. Thus, the purpose of the Forum evaluation was to:
Best practices in focus: what was different this yearThe validity of the forum's best practices related to input and process was at the focus of the evaluation. These practices involved the development of the white papers, the selection and recruitment of the participant sample, selection and training of the moderators, forum structure and organization involving staff, logistical and technical support, and on-site environmental support. They reflected improvements based on evaluation findings and lessons we drew from valuable participant suggestions at our previous forums. Some of the lessons were: (a) To maintain our structured format for the sessions, which helped our two new moderators to be on track at various points. (b) To focus on recruiting consumers who were articulate contributors besides being "information-rich" (expert) consumers. Professional listservs were particularly helpful in widening our reach and enabling us to recruit internationally. (c) To respond to the special challenge this year of ensuring full inclusion of our literally "voiceless" consumers as discussants with other stakeholder groups through a prior feasibility study. We conducted this in conjunction with our consumer panel interviews that generated our white papers. We implemented two methods of group discussions, on-line and face-to-face, and compared the quality of their output to assess their ability to elicit consumers' contributions as expert users. We found the quality of the output to be equivalent for our purposes, which verified the feasibility of their face-to-face participation in our Forum processes. We further ensured their full engagement in the discussions, by sending each user the discussion questions ahead of time and moderating the sequence of participant responses so as to prioritize and maximize the opportunity for end user responses. (d) To keep our pre-Forum consumer training session open to other stakeholders as well, with a view to help "partners in communication with AAC users". Here we used a format that simulated the actual sessions. Procedures for collecting, analyzing and using informationTwo sources gave us the needed information about the quality of our best practices. Internally, our project staff gave us their self-evaluative perceptions and judgments of the forum performance. Our external source was our participating stakeholders. They judged the validity of the Forum sessions they attended and reported their satisfaction levels about them. They also judged the quality of the Forum as a whole at the end. Additionally, a team of evaluators made formal, on-site observations of the Forum in session for process monitoring. All evaluative information was gathered both by formal and informal modes as described in the next section. We carried the on-site evaluator observations back and forth between live sessions, reinforcing their strengths and correcting process errors. Also, we analyzed stakeholder responses and comments at the end of the day and fed them back to the moderating teams immediately, enabling them to modify their second day sessions in accordance with the needs perceived on the first day. The participants' insights into our strengths and weaknesses during the Forum, as well as our on-site observations enabled us to monitor the processes, improve them while they were still being conducted, and keep them appropriately focused and directed. In addition to guiding on-site modifications in our on site processes, systematic triangulation between the foregoing sets of data also helped us identify improvements for future events and the relevant demand-pull steps. InstrumentsSurveys during and at the close of the Forum: Participant stakeholders evaluated the quality of each individual session against their own expectations, using a survey form. They expressed their satisfaction levels about the session they attended by answering questions on a 5 point rating scale about various session aspects. They evaluated the session for content (topics relevant? discussions deep enough?), purpose (achieved?) and personal satisfaction (felt comfortable? able to contribute?). Participants also responded to open-ended questions, making additional evaluative comments. Fig.1 presents this survey and its items. Our participating stakeholders also made final evaluations of the Forum using a separate survey at the conclusion [see Fig.2]. They rated Forum organization, the quality and appropriateness of the white papers and the accessibility of venue each on a 5 point rating scale. Besides, they indicated the extent to which the Forum met our expectations for them by way of: exposing them to the state-of-the-art technology; facilitating networking, partnerships and collaborations; indicating untapped business opportunities; enabling them to shape the future of the hearing aid and assistive listening industries. Formal On-site Observation Checklist: Our "on-site" evaluation is an important part of the Forum. Four observers, one in each breakout session, recorded the unique features of the interactions by direct observation, using a formal checklist [See Fig. 3]. They observed what styles the moderators used, the way they used the audio-visual aids, and how effectively their team coordinated its roles to monitor discussions, clarify technical content and summarize key points. The evaluator coordinated their work and delivered on site feedback to the moderating teams, carrying observations back and forth between live sessions, reinforcing their strengths and correcting process errors. Informal On-site Observations: In addition to the formal observation by the evaluation team, there were also informal observations reported by the organizers. The spontaneous comments they captured from the stakeholders outside the sessions corroborated or clarified information that came from the other sources. Formal Self-evaluative feedback: Critical, written comments from project staff, including the moderators, scribes, evaluators and on-site technical team, also served to clarify or enhance the information collected from other sources regarding Forum effectiveness. [ Top of Page ] Results-What we found and how we used itAnalyses of the foregoing information revealed that the Forum was a success, an experience considered gratifying by participants and hosts alike. The results are detailed below.
Stakeholders Evaluate Forum SessionsTable 1 shows the breakdown of stakeholders who responded to our surveys. Between 24 and 32 participants per technology area (two sessions) consistently filled out the evaluation forms, totaling 57 the first day and 54 the second. They indicated their satisfaction levels about the sessions and made additional comments. Table 2 summarizes and compares the ratings across the four sessions. Overall perceptions were very positive about all sessions. Evaluations varied due to participant mix and differences in moderator style. Overall averages for the areas ranged from reasonable satisfaction (3.3 points on a 5 point scale) to high satisfaction (4.4 points), reaching as high as 4.8 for one individual session. Generally, the sessions scored higher ratings the second day. As per our previous experience with the two forums, this explains the participants' adjustment to the uniqueness of the Forum discussion protocols from one day to the next. On the whole, sessions scored high points on comfort of participation (4.3), content relevance (4.0), session being well organized and run (4.0) and clarity of moderator instructions (4.0). Sessions achieving their purpose, although not as highly rated, reached a satisfactory level (3.5). Not surprisingly, as participants' open-ended comments revealed, this is somewhat a result of the challenging nature of AugCom technology as well as the high level of expertise and expectation of the participants having to work within the time constraints of the event. Corroborating this interpretation, participants' rating of depth of discussion, clarity of purpose, their own feeling of having contributed was in the same vicinity (3.7). Stakeholders Expressed Satisfaction with the ForumA total of 45 participants responded to the end-of-the-forum survey (Table 1). Participant evaluations of the overall organization of the Forum (Table 3) showed satisfaction on their part. Rating averages were 4.4 points both for adequacy of the facilities and their accessibility. . Although accessibility was not a major issue with the hotel, participants' open comments pointed to the hotel services being "below par", and incongruent with the efficiency of the hosting organization. Rating mean was 3.7 for background information provided by the white papers. Open-ended comments praised the white papers but suggested it be timelier. Stakeholders Valued Many Benefits from the Forum On the end-of-the-forum survey participant also told us what benefits they took back with them. As in the past two years, they valued networking opportunities leading to partnerships, collaborations and new business above all else. The following are the principal benefits indicated by the participants both in their ratings (Table 3) and in their voluntary, open-ended comments (see Document 1) that evaluated the sessions (first survey) and the Forum (the End-of-forum survey). 1. The opportunity to network with and learn about the ideas of other stakeholder groups was upheld as the Forum's strength by forty out of the 45 survey responses (about 90%). Open-ended comments also repeatedly (19 out of 45) mentioned this as the forum's strength. 2. The opportunity to interact with AAC users was a particularly valued benefit. 10 out of 45 volunteered comments such as, "I got a chance to meet other stakeholders. I am very happy that end users are involved, and it needs to continue that way". (See Document 1) 3. The opportunity to share ideas with stakeholder experts especially in areas other than their own area of expertise was important to many (10 out of 45). To one expert "It was great to have the opportunity to interact in a positive fashion with other stakeholder groups - was quite helpful" , while others were happy "to learn what others in the field were thinking". (See Document 1 for more) 4. Many (16 comments) mentioned Information, knowledge and learning that resulted from the Forum as a benefit. Some examples are: "The knowledge of technology is coming up. I learnt that the future will [would] look life a little bit better"; "Hearing what else was being done in research and might be available to be made use of"; "Knowledge from the technology side and manufacturer side about new up and coming technology"; "Discussions of barriers fellow developers have encountered"; "Seeing some of the brains and the developing to start new products of technology". 5. Other benefits: Twenty-seven of the 45 respondents (or 60%) acknowledged that the Forum helped them "identify need for new technology". This includes 9 of the 11 researchers, over half of the technology producers and of customers, 2 of the 6 manufacturers and 2 of the 5 attending resource providers. Of the 45 respondents, 19 acknowledged being able to "identify direction for new product development", 16 acknowledged exposure to new technology; whereas 13 acknowledged being able to identify new business opportunities. Informal Stakeholder EvaluationsThe
above findings were consistent with the informal observations reported
by our organizing staff that captured spontaneous stakeholder comments
outside the sessions. These comments described the various activities
of the Forum with remarks such as: "It was a very good forum. I liked
it a lot"; "I thought that Lunch was a little slow"; "I am enjoying myself
a lot"; "The white papers were good. They were well done and had all the
information; if I have to ever go back and read the other white papers
from other groups, it would give me all the information I need for understanding";
"It was a very good forum. I am so glad somebody is doing this kind of
work"; "I know that (you use our feedback and improve your work). I am
very impressed. You are a set of very dedicated people and it is visible".
The perception on the whole was that the entire program was well run and
in a professional manner. Most of their specific comments on "strengths"
of the Forum reflected and reinforced our best practices improved from
last year, while we also derived "lessons" from this year's experience
in their constructive suggestions about our practices. In-House Input - Self-study CommentsThe above evaluative information was discussed by the project staff, who separately made self evaluative comments that corroborated the stakeholder evaluations, and recognized the challenges and shortcomings that led to their suggestions for improvement. We are now formalizing the "lessons" that emerged from the experience as input for our next Forum. [ Top of Page ] Final ConsiderationsFormal feedback about the Forum processes from staff and participants alike helped us make on-site improvements to raise the performance level of the sessions and of the Forum as a whole. Moderators made the appropriate changes from one session to the next, and we controlled, monitored and made better use of audio-visual resources. On the other hand, all end-of-the process feedback was recorded as "lessons" for future events and will be used in our ongoing planning. We owe our success, in a very special way, to all of the stakeholders who participated in the Forum, and made it an experience gratifying to participants and hosts alike. Their enthusiasm and commitment in working with us has gone beyond the Forum, many sending their expanded contributions as post-forum messages. The problem statements, our post-forum outcomes, are developed from the rich material generated at the Forum, and incorporate the information that was converged to. They point to problems in need of technical solutions, indicating both the current product limitations and the corresponding technical specifications. In a later step, other stakeholders - advanced technology developers - will respond by proposing the necessary technology solutions to the problem that should improve these products. We conclude by acknowledging support and input from our project sponsor, the National Institute on Disability and Rehabilitation Research (NIDRR), our partner RERC on Communication Enhancement and our co-sponsor the South East Region of the Federal Laboratory Consortium. [ Top of Page ] |
|
||
|
| |
