Achieving ongoing improvement through continuous self-study is key to quality assurance in any program. When this effort is integrated with external feedback from its "stakeholders" [see below], as is the practice at the Rehabilitation Engineering Research Center on Technology Transfer (T2RERC), the program also validates itself in its broader context. As we at T2RERC implement our efforts to transfer technology through our Demand-Pull and Supply-Push models, we value, and systematically collect, information from our stakeholders at strategic and significant points during the process cycles of the two models. The Stakeholder Forum on Wheeled Mobility that was held at Pittsburgh, Pennsylvania on May 25 and 26, 1999 is one such landmark, an event significant to the project's demand-pull process.
Ongoing Evaluation at Each Stage of the Forum
T2RERC, and its partner organizations that sponsored the Forum, [see other articles in this publication] owe the success of the event to the multiple stakeholders who enriched our self-study process by giving valuable information at various stages of this event. During the months prior to this two-day meeting, experts on Wheeled Mobility that we had identified as appropriate and valuable gave us high quality input through our interview sessions. This information framed the "white papers" under the four themes we had identified as significant to the problem of technology transfer for wheeled mobility [See column 3, page 2]. These papers fed the discussions that occurred later, at the Forum, by a larger group of stakeholders invited to participate. Their mixed perspectives generated the information needed for synthesizing "problem statements", the intended outcomes of the Forum.
Targeted outcomes cannot be achieved at the desired quality level unless we ensure the quality of the processes that produced them. Thus, on-site, "formative" evaluation formed an important part of the Stakeholder Forum. And once again, we gained insights from our participant stakeholders who mirrored the strengths and weaknesses of the Forum processes by giving us evaluative information during the Forum. They enabled us to walk with the processes, improve them while they were still on the run and keep them appropriately focused and directed.
Mixed stakeholder groups - product consumers (end-users with disabilities), product producers (manufacturers), technology producers, and resource providers (government officials) - participated in each of the four sessions that ran simultaneously each day. A team of two evaluators made direct observation of the interactions at these sessions, circulating from session to session and recording the unique features of each session. They observed what styles the moderators used, the way they used the audio-visual aids, and how effectively the team coordinated its three roles in order to monitor discussions, technical content and summarizing notes. They carried their observations back and forth between the live sessions, reinforcing their strengths and correcting process errors. Additionally, all participants evaluated the quality of each individual session, on a survey form, against their own expectations. We analyzed their responses and comments at the end of the day and fed them back to the moderating teams immediately, which enabled them to modify their second day sessions in accordance with the needs perceived on the first day.
Evaluations Provided by the Stakeholders
In all, 46 participants consistently filled out all the evaluation forms expressing their satisfaction levels and making additional comments. They evaluated each individual session on: content (if the topics were relevant, if the discussions went deep enough); purpose (how well it was achieved); and personal satisfaction (about feeling comfortable and being able to contribute). Overall, perceptions were very positive about all sessions, although not uniformly high across all of them, given the participant mix and differences in moderator styles. The evaluations ranged from reasonably satisfactory (3.5 points on a 5 point scale) to highly satisfactory (4.8 points). They were generally higher on the second day, indicating improvement of session performance and thus validating the value of on-site feedback to moderators.
Different stakeholder groups valued different aspects of the Forum as its strength. But the opportunity to network with, and learn about, the ideas of other stakeholder groups was upheld as the Forum's strength by all groups. 42 of the 46 survey responses, or 91%, indicated this as a benefit from the Forum.
Consumers appreciated learning about the state of the art of the technologies, and felt the "need to have more of these forums", and "looked forward to seeing some of the ideas in the future chairs being manufactured." One comment read, "Thank you very much for inviting me. Perhaps we can do this again. ... the groups that I participated in were very technical but I believe that I was able to understand at least 75% of the information. I increased my knowledge and have a better understanding of what it takes to unite technology and materials".
Technology Producers appreciated the opportunity to "meet a diverse group of people with common mobility interest", describing the Forum as "unifying personnel of different fields under one roof which will and should help getting a proper network".
The Manufacturer group, in addition, appreciated the opportunity of exposure to the consumer group, being particularly impressed with the "mix of manufacturers, researchers, clinicians, and the most important group, the users."
A Resource Provider, in appreciating the networking opportunity, acknowledged getting "good insight to issues dealing with wheel chairs".
Summing Up the Responses for an Overall View
Two other benefits or impacts of the Forum that the total group of participants acknowledged were: exposure to new or innovative technologies (63%) and being able to identify the needs for new products or technology (57%). 18 participants (out of 46) felt they were able to identify new business opportunities and 17 felt they helped shape the direction for new product development.
Overall, the participants indicated very high satisfaction with the effective organization throughout, in particular with the helpful, friendly efficiency of the organizers (4.9 points). They considered the background information provided by the white papers as very appropriate and sufficient (4.2 points). Travel logistics as well as comfort and accessibility of the accommodations were also rated high (4 to 4.4 points), although specific stakeholders gave constructive suggestions for achieving perfectly accessible meeting and dining arrangements in the future.
Individual comments from different stakeholders that corroborated the above results described the event as "extremely professional", and pointed out as a strength of the Forum our involving the RERC on Wheeled Mobility in Pittsburgh as our partner. Special mentions were also made of: the "overall structure" of the Forum, the "motivation" of the moderators, the "model" we followed in the sessions that converged discussion to summaries, the quality of the background materials, among other things. As one Resource Provider observed, "providing a structure for groups to work within", having "focused objectives" and having "facilitators" were the strengths of our sessions.
We also received self-evaluative feedback from the organizing members themselves. The moderators from AZtech and Independent Living Center (ILC), both in Buffalo, and Research Triangle Institute (RTI) of North Carolina, the graduate students from the RERC at Pittsburgh, and the professionals from T2RERC sent their perceptions and evaluative comments electronically to the evaluation team, subsequent to the event. This information pointed to general satisfaction on everyone's part, the entire program considered professional, well run and effective. Having an on-site support system with resources and a work area, as well as getting continuous evaluation and feedback on site, were particularly upheld as strengths. On the other hand, there were a few constructive comments related to either maintaining or improving "an already excellent process". These include: maintaining the present structured format of the sessions, making technical/laymen communication stronger by moderator training, more efficient recruiting and preparation of consumers for better validation of technical solutions, and rethinking ways of efficiently involving our distant partners.
After cross-analyzing the self-study information with the participating stakeholder perceptions, much as we happily conclude the high success level of the Forum, we also recognize the value of the constructive suggestions made. In a spirit of improving our outcomes by enabling stakeholders to make their highest quality inputs, we acknowledge them as "lessons" for our future events and pledge our ongoing efforts towards excellence.
[ Top of Page ]