Title: Stakeholders share voice and expertise at forum on communication enhancement
Author: Vathsala I Stone
Publication: Assistive Technology Transfer Update: Vol. 3 Issue 2 (Summer)
Dr. Vathsala Stone explains how the Third Stakeholders Forum was carried out including preparation beforehand, the actual forum, and subsequent evaluations directly following the conclusion of the forum.
[ Top of Page ]
Over seventy individuals with expertise in areas crucial
to the invention, production and use of Augmentative Communication technology
and products came together for a two-day Forum in June this year at the Adam's
Mark Hotel in Buffalo, New York, at the invitation of the Rehabilitation
Engineering Research Center on Technology Transfer (T2RERC). The participating
experts were all stakeholders in T2RERC's effort to transfer communication
enhancement technology into the marketplace for persons with disabilities,
and they consisted of researchers, advanced technology producers, manufacturers
and funding agency representatives, and especially product customers that
included augmented communicators, clinicians and caregivers. [Details of
the process by which the Stakeholders Forum on Communication Enhancement
(SFCE) identified valid technology needs that are expected to advance the
state of the practice in Augmentative and Alternative Communication (AAC)
devices is discussed elsewhere in this publication.]
Third in a series of professional forums sponsored by T2RERC to support
our Demand-Pull project, this Forum was designed in the basic mold of
Forums on Wheeled Mobility and Hearing Enhancement technologies, with refinements
from evaluations of those experiences. Observing best practices for the
Demand-Pull model of technology transfer in action and assuring their
on-site evaluation and feedback is vital for their ongoing refinement and
consolidation, and has become an integral part of the Forum itself.
Current Best practices: what was improved from last year
Based on our past evaluation findings and lessons, we incorporated the
following improvements in this year’s Forum: (a) We maintained our structured
format for the sessions, which helped our two new moderators to be on track
at various points. (b) We focused our consumer recruitment on participants
who were articulate contributors besides being "information-rich" (expert)
consumers. Professional listservs (databases of Internet users) were particularly
helpful in enabling us to recruit internationally. (c) A special challenge
this year was ensuring full inclusion of our literally "voiceless" consumers
as discussants with other stakeholder groups and drawing their best contributions
as expert users. (The steps through which we ensured they could fully engage
in the discussions are described in the article "Accessing User Input,
Online and In-Person".) (d) We kept our pre-Forum consumer training
session open to other stakeholders as well, and used a format that simulated
the actual sessions.
Before the Forum: assuring input quality
In order to achieve quality outcomes for the Forum, one must ensure the
quality of the inputs that produce them. Careful research and evaluation
developing the "White Papers" during the months prior to the
Forum, which formed the bases for discussion at the Forum.
Participation in the Forum was strictly by invitation. Our seventy-five
(75) attendees included 15 advanced technology developers, 16 consumers,
and internationally known clinical and technical researchers, 8 augmentative
communication product manufacturers, 9 resource provider representatives,
and 5 guest scholars. They brought specific knowledge about the input,
output and processing systems of devices, their underlying technologies,
At the Forum: on-site evaluation and feedback
Trained moderators from the T2RERC led the discussions at the Forum
by the diverse group of stakeholders. Their varied perspectives
generated the consensus
statements needed to create problem statements, the intended outcomes
of the Forum.
Our "on-site" evaluation was an important part of the Forum.
Four observers, one in each breakout session, recorded the unique features
of the interactions by direct observation, using a formal checklist. They
observed what styles the moderators used, the way they used the audio-visual
aids, and how effectively their team coordinated its roles to monitor discussions,
clarify technical content and summarize key points. The evaluator coordinated
their work and delivered on site feedback to the moderating teams, carrying
observations back and forth between live sessions, reinforcing their strengths
and correcting process errors.
Participant stakeholders evaluated the quality of each individual
session against their own expectations, using a survey form.
We analyzed their
responses and comments at the end of the day and fed them back
to the moderating teams
immediately, enabling them to modify their second day sessions
in accordance with the needs perceived on the first day. The
into our strengths and weaknesses during the Forum enabled us to monitor
and improve the processes and keep them appropriately focused and directed.
Stakeholders also made final evaluations of the Forum using a separate survey
at the conclusion.
Stakeholders Evaluate the Forum:
About 24 to 32 participants per technology area (over two sessions)
consistently filled out the evaluation forms expressing their
about the sessions they attended and making additional comments.
them for content (topics relevant? discussions deep enough?),
purpose (achieved?) and personal satisfaction (felt comfortable?
perceptions were very positive about all sessions. Evaluations
varied due to participant mix and differences in moderator
averages for the areas ranged from reasonable satisfaction
(3.3 points on a 5-point
scale) to high satisfaction (4.4 points), reaching as high
as 4.8 for one
individual session. Generally, the sessions scored higher
ratings the second day. As per our previous experience with the two
the participants’ adjustment to the uniqueness of the Forum discussion
protocols from one day to the next. On the whole, sessions scored high points
on comfort of participation (4.3), content relevance (4.0), session being
well organized and run (4.0) and clarity of moderator instructions (4.0).
Sessions achieving their purpose, although not as highly rated, reached a
satisfactory level (3.5). Not surprisingly, as participants’ open-ended
comments revealed, this is somewhat a result of the challenging nature of "AugCom" technology,
as well as the high level of expertise and expectation of the participants
having to work within the time constraints of the event. Corroborating this
interpretation, participants’ rating of depth of discussion, clarity
of purpose, their own feeling of having contributed was in the same vicinity
Participants valued many benefits from the Forum:
A total of 45 participants responded to the end-of-the-forum survey and gave us feedback on what benefits they took back with them. As in the past two years, they valued networking opportunities leading to partnerships, collaborations and new business above all else. The following are the principal benefits indicated by the participants both in their ratings and in their voluntary, open-ended comments.
- The opportunity to network with, and learn about the ideas of, other stakeholder groups was upheld as the Forum's strength by forty out of the 45 survey responses (about 90%). Open-ended comments also repeatedly (19 out of 45) mentioned this as the forum's strength.
- The opportunity to interact with AAC users was a particularly valued benefit; (10 respondents made specific comments noting this).
- The chance to share ideas with stakeholder experts, especially in areas other than their own area of expertise, was important to many (10 out of 45).
- Many (16 comments) mentioned Information, knowledge and learning that resulted from the Forum as a benefit.
- Other benefits: Twenty-seven of the 45 respondents (or 60%) acknowledged that the Forum helped them "identify need for new technology". This includes 9 of the 11 researchers, over half of the technology producers and of customers, 2 of the 6 manufacturers and 2 of the 5 attending resource providers. Of the 45 respondents, 19 acknowledged being able to "identify direction for new product development"; 16 acknowledged exposure to new technology; whereas 13 acknowledged being able to identify new business opportunities.
Stakeholders appreciated the high quality of
The groups commended us on the high quality
(level of expertise, diversity of background
Forum Participants. While
there were 37 (about 90%) positive comments,
the only suggestion for improvement in 7
have more of
Stakeholders applauded the quality of the
content and the stimulating nature of
also made suggestions
While participants (7 comments) made
remarks such as "This was fantastic", "Sitting
and thinking with colleagues, bringing up issues, taking in others’ perspectives – it
was very stimulating", they also alluded to the discussions being "frustrating".
Most of the concerns were about difficulty in covering the topics, the challenging
nature of the technology, and the fact that time constraints imposed limitations
on how well the experts from diverse backgrounds were able to get their contributions
in. A few suggested a rethinking of the sessions’ structure
and the model or deliberately preparing
participants about what to expect in terms
of the discussion protocols. Others praised
moderators, although suggesting a more
flexible and non-directive Moderator model
that would allow
for some brainstorming and discussion while
keeping the participants focused and on
track. They also
reinforced the importance of having moderators
with expertise in the topic/field.
Stakeholders expressed satisfaction with
Participant evaluations of the overall
organization of the Forum showed satisfaction
averages were 4.4 points
adequacy of the facilities
and their accessibility. Although accessibility
was not a
major issue with the hotel, participants’ open comments pointed to the hotel services
being "below par", and incongruent
with the efficiency of the hosting organization.
mean was 3.7 for background information
provided by the white papers. Open-ended
the white papers but suggested it be timelier.
The above findings were consistent
with the informal observations reported
who spoke to stakeholders
outside the sessions.
The perception on the whole (judging
by comments such as "It was a very
good forum. I am so glad somebody is doing this kind of work") was
that the entire program was run well and in a professional manner. Most of
their specific comments on "strengths" of the Forum reflected
and reinforced our best practices improved from last year, while we also
derived "lessons" from this year’s
experience in their constructive suggestions
about our practices.
The above evaluative information
was discussed by the project staff,
self-evaluative comments that
corroborated the stakeholder evaluations,
and recognized the challenges
their suggestions for improvement.
We are now formalizing the "lessons" that
emerged from the experience as input for
our next Forum.
Where do we go from here?
We owe our success, in a very special
way, to all of the stakeholders
who participated in the
an experience gratifying
and hosts alike. Their enthusiasm
in working with us has gone beyond the Forum,
many sending their expanded
contributions as post-forum
messages. The problem statements
are now in development, using
material generated at the Forum.
They will incorporate
that was converged to, and
problems in need of technical
both the current product limitations
and the corresponding technical
specifications. In a later step, other stakeholders – advanced technology developers – will respond by proposing the necessary technology solutions to the problem that should improve these products.
We conclude by acknowledging support and input from our project sponsor, the National Institute on Disability and Rehabilitation Research (NIDRR), our partner RERC on Communication Enhancement and our co-sponsor the South East Region of the Federal Laboratory Consortium.
[ Top of Page