Assistive Technology Transfer Update

Abstract

Dr. Vathsala Stone explains how the Third Stakeholders Forum was carried out including preparation beforehand, the actual forum, and subsequent evaluations directly following the conclusion of the forum.

[ Top of Page ]

Full Text

Over seventy individuals with expertise in areas crucial to the invention, production and use of Augmentative Communication technology and products came together for a two-day Forum in June this year at the Adam's Mark Hotel in Buffalo, New York, at the invitation of the Rehabilitation Engineering Research Center on Technology Transfer (T²RERC). The participating experts were all stakeholders in T²RERC's effort to transfer communication enhancement technology into the marketplace for persons with disabilities, and they consisted of researchers, advanced technology producers, manufacturers and funding agency representatives, and especially product customers that included augmented communicators, clinicians and caregivers. [Details of the process by which the Stakeholders Forum on Communication Enhancement (SFCE) identified valid technology needs that are expected to advance the state of the practice in Augmentative and Alternative Communication (AAC) devices is discussed elsewhere in this publication.]

Third in a series of professional forums sponsored by T²RERC to support our Demand-Pull project, this Forum was designed in the basic mold of our previous Forums on Wheeled Mobility and Hearing Enhancement technologies, with refinements from evaluations of those experiences. Observing best practices for the Demand-Pull model of technology transfer in action and assuring their quality through on-site evaluation and feedback is vital for their ongoing refinement and consolidation, and has become an integral part of the Forum itself.

Current Best practices: what was improved from last year
Based on our past evaluation findings and lessons, we incorporated the following improvements in this year’s Forum: (a) We maintained our structured format for the sessions, which helped our two new moderators to be on track at various points. (b) We focused our consumer recruitment on participants who were articulate contributors besides being "information-rich" (expert) consumers. Professional listservs (databases of Internet users) were particularly helpful in enabling us to recruit internationally. (c) A special challenge this year was ensuring full inclusion of our literally "voiceless" consumers as discussants with other stakeholder groups and drawing their best contributions as expert users. (The steps through which we ensured they could fully engage in the discussions are described in the article "Accessing User Input, Online and In-Person".) (d) We kept our pre-Forum consumer training session open to other stakeholders as well, and used a format that simulated the actual sessions.

Before the Forum: assuring input quality
In order to achieve quality outcomes for the Forum, one must ensure the quality of the inputs that produce them. Careful research and evaluation went into developing the "White Papers" during the months prior to the Forum, which formed the bases for discussion at the Forum.

Forum Participants:
Participation in the Forum was strictly by invitation. Our seventy-five (75) attendees included 15 advanced technology developers, 16 consumers, 22 nationally and internationally known clinical and technical researchers, 8 augmentative communication product manufacturers, 9 resource provider representatives, and 5 guest scholars. They brought specific knowledge about the input, output and processing systems of devices, their underlying technologies, their application and use.

At the Forum: on-site evaluation and feedback
Trained moderators from the T²RERC led the discussions at the Forum by the diverse group of stakeholders. Their varied perspectives generated the consensus statements needed to create problem statements, the intended outcomes of the Forum.

Our "on-site" evaluation was an important part of the Forum. Four observers, one in each breakout session, recorded the unique features of the interactions by direct observation, using a formal checklist. They observed what styles the moderators used, the way they used the audio-visual aids, and how effectively their team coordinated its roles to monitor discussions, clarify technical content and summarize key points. The evaluator coordinated their work and delivered on site feedback to the moderating teams, carrying observations back and forth between live sessions, reinforcing their strengths and correcting process errors.

Participant stakeholders evaluated the quality of each individual session against their own expectations, using a survey form. We analyzed their responses and comments at the end of the day and fed them back to the moderating teams immediately, enabling them to modify their second day sessions in accordance with the needs perceived on the first day. The participants’ insights into our strengths and weaknesses during the Forum enabled us to monitor and improve the processes and keep them appropriately focused and directed. Stakeholders also made final evaluations of the Forum using a separate survey at the conclusion.

Stakeholders Evaluate the Forum:
About 24 to 32 participants per technology area (over two sessions) consistently filled out the evaluation forms expressing their satisfaction levels about the sessions they attended and making additional comments. They evaluated them for content (topics relevant? discussions deep enough?), purpose (achieved?) and personal satisfaction (felt comfortable? able to contribute?). Overall perceptions were very positive about all sessions. Evaluations varied due to participant mix and differences in moderator styles. Overall averages for the areas ranged from reasonable satisfaction (3.3 points on a 5-point scale) to high satisfaction (4.4 points), reaching as high as 4.8 for one individual session. Generally, the sessions scored higher ratings the second day. As per our previous experience with the two forums, this explains the participants’ adjustment to the uniqueness of the Forum discussion protocols from one day to the next. On the whole, sessions scored high points on comfort of participation (4.3), content relevance (4.0), session being well organized and run (4.0) and clarity of moderator instructions (4.0). Sessions achieving their purpose, although not as highly rated, reached a satisfactory level (3.5). Not surprisingly, as participants’ open-ended comments revealed, this is somewhat a result of the challenging nature of "AugCom" technology, as well as the high level of expertise and expectation of the participants having to work within the time constraints of the event. Corroborating this interpretation, participants’ rating of depth of discussion, clarity of purpose, their own feeling of having contributed was in the same vicinity (3.7).

Participants valued many benefits from the Forum:
A total of 45 participants responded to the end-of-the-forum survey and gave us feedback on what benefits they took back with them. As in the past two years, they valued networking opportunities leading to partnerships, collaborations and new business above all else. The following are the principal benefits indicated by the participants both in their ratings and in their voluntary, open-ended comments.

1. The opportunity to network with, and learn about the ideas of, other stakeholder groups was upheld as the Forum's strength by forty out of the 45 survey responses (about 90%). Open-ended comments also repeatedly (19 out of 45) mentioned this as the forum's strength.
2. The opportunity to interact with AAC users was a particularly valued benefit; (10 respondents made specific comments noting this).
3. The chance to share ideas with stakeholder experts, especially in areas other than their own area of expertise, was important to many (10 out of 45).
4. Many (16 comments) mentioned Information, knowledge and learning that resulted from the Forum as a benefit.
5. Other benefits: Twenty-seven of the 45 respondents (or 60%) acknowledged that the Forum helped them "identify need for new technology". This includes 9 of the 11 researchers, over half of the technology producers and of customers, 2 of the 6 manufacturers and 2 of the 5 attending resource providers. Of the 45 respondents, 19 acknowledged being able to "identify direction for new product development"; 16 acknowledged exposure to new technology; whereas 13 acknowledged being able to identify new business opportunities.

Stakeholders appreciated the high quality of participant expertise:
The groups commended us on the high quality (level of expertise, diversity of background and level of participation) of Forum Participants. While there were 37 (about 90%) positive comments, the only suggestion for improvement in 7 other comments was to have more of them.

Stakeholders applauded the quality of the content and the stimulating nature of discussions but also made suggestions for improvement:
While participants (7 comments) made remarks such as "This was fantastic", "Sitting and thinking with colleagues, bringing up issues, taking in others’ perspectives – it was very stimulating", they also alluded to the discussions being "frustrating". Most of the concerns were about difficulty in covering the topics, the challenging nature of the technology, and the fact that time constraints imposed limitations on how well the experts from diverse backgrounds were able to get their contributions in. A few suggested a rethinking of the sessions’ structure and the model or deliberately preparing participants about what to expect in terms of the discussion protocols. Others praised the session moderators, although suggesting a more flexible and non-directive Moderator model that would allow for some brainstorming and discussion while keeping the participants focused and on track. They also reinforced the importance of having moderators with expertise in the topic/field.

Stakeholders expressed satisfaction with the Forum:
Participant evaluations of the overall organization of the Forum showed satisfaction on their part. Rating averages were 4.4 points both for adequacy of the facilities and their accessibility. Although accessibility was not a major issue with the hotel, participants’ open comments pointed to the hotel services being "below par", and incongruent with the efficiency of the hosting organization. Rating mean was 3.7 for background information provided by the white papers. Open-ended comments praised the white papers but suggested it be timelier.

The above findings were consistent with the informal observations reported by our organizing staff who spoke to stakeholders outside the sessions. The perception on the whole (judging by comments such as "It was a very good forum. I am so glad somebody is doing this kind of work") was that the entire program was run well and in a professional manner. Most of their specific comments on "strengths" of the Forum reflected and reinforced our best practices improved from last year, while we also derived "lessons" from this year’s experience in their constructive suggestions about our practices.

In-house Comments:
The above evaluative information was discussed by the project staff, who separately made self-evaluative comments that corroborated the stakeholder evaluations, and recognized the challenges and shortcomings that led to their suggestions for improvement. We are now formalizing the "lessons" that emerged from the experience as input for our next Forum.

Where do we go from here?
We owe our success, in a very special way, to all of the stakeholders who participated in the Forum, and made it an experience gratifying to participants and hosts alike. Their enthusiasm and commitment in working with us has gone beyond the Forum, many sending their expanded contributions as post-forum messages. The problem statements are now in development, using the rich material generated at the Forum. They will incorporate the information that was converged to, and will point to problems in need of technical solutions, indicating both the current product limitations and the corresponding technical specifications. In a later step, other stakeholders – advanced technology developers – will respond by proposing the necessary technology solutions to the problem that should improve these products.

We conclude by acknowledging support and input from our project sponsor, the National Institute on Disability and Rehabilitation Research (NIDRR), our partner RERC on Communication Enhancement and our co-sponsor the South East Region of the Federal Laboratory Consortium.

[ Top of Page