Forum Proceedings

Stakeholder Forum on Technology for Visual Impairment (TVI): Evaluation Report Contents

1. Introduction
2. Background
   • The Demand-Pull Model and its Best Practices
   • The Forum–Why, What and How
3. Evaluation Methodology
   • Purpose
   • Best Practices in Focus: What Was Different in 2003
   • Procedures for Collecting, Analyzing and Using Information
     • Instruments
4. Results – What We Found and How We Used It
   • Stakeholders Evaluate Forum Sessions
   • Stakeholders Evaluate the Forum Performance
   • Stakeholders Valued Many Benefits from the Forum
   • Informal Stakeholder Evaluations
   • In-house Input – Self-Study Comments
5. Final Considerations
6. Conclusion
7. Tables and Figures
   • Table.1: Distribution of Responses to the TVI Stakeholder Forum surveys
   • Table.2: TVI Stakeholder Forum Session Ratings
   • Table.3: Participant Ratings of the TVI Stakeholder Forum
   • Table.4: Summary of Comments on the TVI Stakeholder Forum
   • Fig.1: TVI Stakeholder Forum Session Survey
   • Fig.2: Overall Survey of the TVI Stakeholder Forum

Introduction

This report concerns the evaluation of the Stakeholder Forum on Technology for Visual Impairment, conducted as part of an ongoing evaluation of the Demand-Pull model of technology transfer. The report presents the background, methods and results related to the Forum evaluation conducted both on-site and immediately following the event.

Background

The Demand-Pull Model and its Best Practices

Demand-Pull is one of the two approaches to technology transfer implemented and validated by T²RERC. An earlier section of this document (p. 2-5) described the Demand-Pull project's protocols. The theoretical framework and the methods that drive these protocols are drawn from best practices in industry and marketing research. Among such practices are: applying the Kano Model for collecting and using primary market data, Quality Function Deployment for product planning, structured focus groups and surveys as data gathering tool, and an array of tasks, which range from sampling and recruiting to managing communication with and among stakeholders. Our specific practices represent applications of these best practices to our project's context in a way that addresses specific disability issues and stakeholder characteristics. As we implement our project to transfer technology, our best practices get field-tested and validated. The validated practices are then duly incorporated into the technology transfer model and contribute to its ongoing refinement.

The Forum – Why, What and How

Purpose: The Demand-Pull activities (see p. 2-4) that preceded the Stakeholders Forum identified (a) current needs in visual impairment technology and (b) the relevant market and industry information, which were compiled into four White Papers and an Industry Profile. The purpose of the Stakeholders Forum was to have this previously identified information validated by a larger stakeholder expert group interacting in formal, moderated discussion sessions. The stakeholders brought their multiple perspectives to systematically address and discuss issues surrounding current vision technologies, and came to a consensus about valid technology needs expected to advance the state of the practice regarding assistive technology products for visual impairment.

Significance: The Stakeholder Forum is a significant step in our Demand-Pull process of technology transfer because it generates, validates, and converges information relevant to significant technology needs. After the Stakeholders Forum, we develop the information obtained into statements of problems in need of technical solutions. These problem statements not only indicate current limitations in the features and functions of products for visual impairment, but also point to business opportunities for advanced technology solutions. In addition, they establish design and performance targets for these solutions and the technical specifications needed to improve the products. In a later step, as described earlier (p. 2-4), other stakeholders, specifically advanced technology developers, respond by proposing technology solutions to the stated problems in lieu of improving the products.

Given their importance to the transfer process, it is imperative to ensure that the problem statements resulting from the Stakeholders Forum validly represent the significant needs of people with visual impairment. Our on-site forum evaluation constitutes our effort to achieve this by ensuring the quality of the inputs as well as the processes that produced the Stakeholders Forum outcomes.

Procedures - Before the Forum: We began our systematic approach to quality assurance far in advance of the Forum itself. During the months prior to the Stakeholders Forum, manufacturing, research, clinical and consumer experts on technology for visual impairment provided input through interview sessions conducted by the T²RERC. These sessions explored the current status of product features and functions and the capabilities of their underlying technologies. Analysis of this information, along with a literature review, allowed us to identify thirteen potential topics in need of further exploration. Then, working with Smith-Kettlewell Eye Research Institute, our partner RERC, we selected the four Technology Areas deemed most important for making significant advancements in the state of the technology underlying visual impairment. They were:

1. Wayfinding
2. Access to Consumer Electronics
3. Access to Textual Information
4. Access to Graphical Information

We then developed a White Paper for each technology area, which presented the current state of knowledge and practice for that specific area. All Stakeholders Forum participants received the White Papers in advance so they arrived with the same current knowledge about each technology area.

Procedures - At the Forum: Participation in the Stakeholders Forum was strictly by invitation. The Forum brought together 43 stakeholders, each one expert in an aspect unique to visual impairment technology. This diverse group of stakeholders included advanced technology developers, consumers, nationally and internationally known researchers, vision technology manufacturers, clinical and technical experts, and experts in third party reimbursement. They brought specific knowledge about technologies for visual impairment or their application and use.

T2RERC staff designed the Stakeholders Forum on Visual Impairment in the basic model of the previous Stakeholders Forums on Wheeled Mobility, Hearing Enhancement and Communication Enhancement, with refinements from evaluations of those forums. Trained moderators from the T2RERC led structured discussions among the participants. Their varied perspectives generated the consensus statements needed to create problem statements which were the intended outcomes of the Stakeholders Forum. The discussions concentrated on the four technology areas mentioned earlier.

We conducted four sessions simultaneously each day, corresponding to the four technology areas. Thus there were two sessions for each area, on each day. Participants took part in two different sessions, one on each day, depending on their expertise and interest. The interactions were highly structured, with moderators using prepared scripts to guide the discussions. A scribe, with a working knowledge of the relevant technical terminology, recorded notes on a personal computer for reference and display. Another scribe recorded points on a flipchart. The room layouts facilitated discussion and ensured that there were no visual barriers between speakers. In addition, ample space was provided for use of assistive devices and guide dogs. For each technology area, the consensus that emerged from the discussions of the two sessions were systematically merged, summarized and reported at a general session at the end of the Stakeholders Forum on Visual Impairment.

Evaluation Methodology

Purpose

As mentioned earlier, validation of the Demand-Pull model of technology transfer provides the broader context for the Stakeholders Forum evaluation. Quality assurance is vital for ongoing refinement and consolidation of best practices. The Stakeholder Forum processes represent our current understanding of best practices that have been tested through actual use. This enables T²RERC staff to assure the quality of inputs, process and outcomes from the Stakeholders Forum. Observing these best practices in action is an integral parts of our Forum activity. It supports the ongoing improvement of both the Stakeholders Forum and the Demand-Pull Projects cycle of operations. Thus, the purpose of the Forum evaluation was to:

1. Evaluate Stakeholders Forum processes and provide support to its monitoring and on-site improvement.
2. Evaluate Stakeholders Forum inputs and structure and provide support to ongoing planning of the Forum and all related model activities.
3. Evaluate, screen and incorporate both existing and newly identified best practices.

Best Practices in Focus: What was Different in 2003

The evaluation focused on the validity of the Stakeholders Forum's best practices related to input and process. These practices involved the development of the White Papers, the selection and recruitment of the participant sample, selection and training of the moderators, Forum structure and organization involving staff, logistical and technical support, and on-site session management. The current practices reflected improvements based on evaluation findings and lessons we drew from valuable participant suggestions at our previous Forums. Some of the lessons were: (a) To maintain our overall structured format for the sessions, which brought the needed uniformity for information convergence, helping our new moderators to be on track at various points, (b) To focus on recruiting consumers who were articulate contributors in addition to being expert consumers, (c) To respond to the special challenge this year of ensuring full inclusion of our consumers with visual impairments as discussants with other stakeholder groups through a prior feasibility study. This study involved the use of laptops with screen reading and screen magnification software and headsets during a consumer panel so that participants could actively follow along with the outline of the discussion and have access to the notes taken on a laptop during the session. The outcome of this study was positive in terms of benefits of inclusion, yet technological constraints made it unfeasible for use at the Forum. Comments elicited from participants in the study groups stated, in general, that the process was useful but not necessary. We tried to ensure full engagement in the discussions, by sending each user the Forum materials ahead of time via electronic, Braille, large print or regular print copies. In addition we provided the medium (laptop, slate and stylus, and the like) each participant needed for communicating their written feedback to us at the Forum. We invited all participants of the Stakeholders Forum to the "orientation" session originally designed only for consumers at the beginning of the Stakeholders Forum with a view to help them better understand the Stakeholders Forum process and expected outcomes.

Procedures for Collecting, Analyzing and Using Information

Two sources gave us the needed information about the quality of our best practices. Internally, our project staff gave us self-evaluative perceptions and judgments of the Forum performance. Our external source was the participating stakeholders. They judged the validity of the Forum sessions they attended and reported their satisfaction levels on formal surveys. They also judged the quality of the Forum as a whole, at its conclusion. Additionally, a team of two evaluators made informal, on-site observations of the Forum in session for process monitoring. We used the information they collected between live sessions, reinforcing their strengths and correcting process errors. Also, we analyzed stakeholder responses and comments at the end of the day and fed them back to the moderating teams immediately, enabling them to modify their second day sessions in accordance with the needs perceived on the first day. The participants' insights into our strengths and weaknesses during the Forum, as well as our on-site observations enabled us to monitor the processes, improve them while they were still being conducted, and keep them appropriately focused and directed.

In addition to guiding on-site modifications in our processes, systematic triangulation between the foregoing sets of data also helped us identify improvements for future events and the relevant demand-pull steps.

Instruments

Surveys during and at the close of the Forum: Participant stakeholders evaluated the quality of each individual session against their own expectations, using a survey form. They expressed their satisfaction levels about the session they attended by answering questions on a 5-point rating scale about various session aspects. They evaluated the session for content (topics relevant? discussions deep enough?), purpose (achieved?) and personal satisfaction (felt comfortable? able to contribute?). Participants also responded to open-ended questions, making additional evaluative comments. Fig.1 presents this survey and its items.

Our participating stakeholders also made final evaluations of the Forum using a separate survey at the conclusion of the Forum [see Fig.2]. They rated Forum organization, the quality and appropriateness of the White Papers and the accessibility of venue each on a 5-point rating scale.

In addition, they indicated the extent to which the Forum met our expectations for them by way of: exposing them to the state-of-the-art technology, facilitating networking, partnerships and collaborations, indicating untapped business opportunities, and enabling them to shape the future of visual impairment technology.

Formal On-site Observation Checklist: The T²RERC's on-site evaluation is an important part of the Stakeholders Forum. Our practice last year was to have four observers, one in each breakout session, and record the unique features of the interactions on a formal checklist. They observed what styles the moderators used, the way they used the audio-visual aids, and how effectively the teams coordinated their roles to monitor discussions, clarify technical content and summarize key points. The evaluator coordinated their work and delivered on-site feedback to the moderating teams, carrying observations back and forth between live sessions, reinforcing their strengths and correcting process errors. This year, a team of two accomplished the same, using the checklist as a guide, but not as a formal recording device.

Informal On-site Observations: In addition to the focused observation by the evaluation team, there were also informal observations reported by the organizers. The spontaneous comments they captured from the stakeholders outside the sessions corroborated or clarified information that came from the other sources.

Formal Self-Evaluative Feedback: Critical, written comments from project staff, including the moderators, scribes, evaluators and on-site technical team, served to clarify or enhance the information collected from other sources regarding effectiveness of the Stakeholders Forum.

Results – What We Found and How We Used It

Analyses of the foregoing information revealed that the Forum was a success; an experience considered gratifying by participants and hosts alike.

4.1 Stakeholders Evaluate Forum Sessions

Table 1 illustrates the breakdown of stakeholders who responded to our surveys. Between 18 to 22 participants per technology area (including the two sessions) consistently filled out the evaluation forms, totaling 41 the first day and 37 the second. We call to attention slight discrepancies between these numbers and the original distribution of stakeholders to individual sessions (See Appendix on Forum Staffing and Participants by Day). This is due to participants changing the sessions they attended, based on their revised area of interest. The respondents indicated their satisfaction levels about the sessions and made additional comments.

Table 2 summarizes and compares the ratings across the four sessions. Although evaluations varied due to participant mix and differences in moderator styles, overall perceptions were very positive about all sessions. Averages for the areas ranged from reasonable (3.7 points on a 5 point scale) to high (4.4 points) satisfaction, reaching as high as 4.8 for one individual session. Generally, the sessions scored higher ratings the second day. As per our previous experience with the three forums, this explains the participants' adjustment to the uniqueness of the Forum discussion protocols from one day to the next. On the whole, sessions scored high points on clarity of moderator instructions (4.4), comfort in participation (4.3), session being well organized and run (4.1) and content relevance (4.1). Open-ended comments from the participants indicated the need for sessions to be in an open format, taking into account the complexity and breadth of technologies for visual impairment. Further, in view of the high level of expertise and urge to brainstorm the sessions in depth, participants expressed concern over time constraints of the event. It is not surprising therefore, that their ratings showed lower, yet reasonable, levels of satisfaction, ranging from 3.7 to 3.9, with regards to the discussions addressing important topics, achieving depth as well as purpose, and the participants being able to contribute to the discussions.

Stakeholders Evaluate Forum Performance

A total of 25 participants responded to the end-of-the-Forum survey (Table 1). Participant evaluations of the overall organization of the Forum (Table 3) showed satisfaction on their part. Adequacy of the Forum facilities and their accessibility were well appreciated by the participants with average ratings of 4.9 and 4.75 respectively. Participants informally commended the accessibility and cordial atmosphere of the hotel, which was an accomplishment this year over the past Forums. Participants also attested to the quality of the White Papers. In addition to a rating of 4.2 for background information provided by the White Papers, participants volunteered open-ended comments to acknowledge the usefulness of the background information and script for the focus group sessions. Nonetheless, they also made suggestions to make the papers more open-ended.

Stakeholders Valued Many Benefits from the Forum

On the end-of-the-Forum survey, participants also told us what benefits they took back with them. As in the past three years, they valued networking opportunities leading to partnerships, collaborations and new business above all else. The following are the principal benefits indicated by the participants both in their ratings (Table 3) of specific benefits and in their voluntary, open-ended comments (see Table 4). These responses were collected for both to the first survey that evaluated the sessions and to the end-of-Forum survey that evaluated the Forum.

1. The opportunity to network with and learn about the ideas of other stakeholder groups was upheld as the Forum's strength by 20 out of the 25 survey responses (about 80%). Open-ended comments also consistently mentioned this as the Forum's strength.
2. The opportunity to interact with other experts was considered invaluable by the participants. Comments such as, "Being able to spend concentrated quality classroom time with various stakeholders really learning their opinions and viewpoints" validates this benefit. (See Table 4)
3. The opportunity to share ideas with stakeholder experts especially in areas other than
   their own area of expertise was important to many participants. One of the experts commented, "The opportunity to listen to others in the R&D field, discuss needs and prospects with consumers was exciting. I am not sure how much information was presented to anyone, but the discussions were lively and potentially help focus researchers on true needs of blind people." (See Table 4 for more)
4. Many participants mentioned the Information, knowledge and learning that resulted from the Forum as a benefit. Some examples are: "Finding out what others were doing and developing and getting consumer perspectives;" "Interacting with the varied disciplines and consumers. Learning about projects underway;" "I learned a tremendous amount from the experiences and input from others in this group…."
5. Other benefits: Thirteen of the 25 respondents (or 52%) acknowledged that the Forum helped them "identify need for new technologies." Of the 25 respondents, 12 acknowledged being able to "identify direction for new product development," 12 acknowledged exposure to new technology, and eight acknowledged being able to identify new business opportunities. (Table 3).

Informal Stakeholder Evaluations

The above findings were consistent with the informal observations and reports by our organizing staff that captured spontaneous stakeholder comments outside the sessions. These comments described the various activities of the Forum with remarks such as:

• "The RERC meeting was delightful to attend and I was very pleased to be a part of it. I thought the event was very well run and productive. I appreciate all your efforts and also am thankful for your invitation."
• "Whoever identified the hotel did an excellent job. The Braille menus in the room were thoughtfully provided making it much easier to know what one wanted to eat prior to going to the dining room. The willingness of the hotel staff to assist in any way possible made our stay much more comfortable and enjoyable. The meeting rooms were comfortable and it was easy to hear anyone speaking at any time."
• "I felt that my participation and interactions with forum personnel and participants was as full and complete as could be hoped for."
• "...I feel that I had as much opportunity to express myself orally in the discussion groups as any of the other participants. I feel that in both forums in which I participated the person conducting the discussions went out of their way to make sure that everyone had ample opportunities to express themselves and contribute to the discussion…."
• "…Thanks again for your attention to the overwhelming needs of all the conference participants…."

The perception on the whole was that the entire program was well run and run in a professional manner. Special appreciation of end-users alluded to our efforts at making the forum inclusive. Most of their specific comments on strengths of the Forum reflected and reinforced our best practices improved from last year, while we also derived lessons from this year's experience in their constructive suggestions about our practices.

In-house Input – Self-Study Comments

Our project staff separately made self-evaluative comments that corroborated the stakeholder evaluations, consolidating the strengths and recognizing further room for improvement. Comments collected from project staff form the basis for the lessons learned from the Stakeholders Forums.

Final Considerations: Where Do We Go From Here?

Feedback about the Stakeholders Forum processes from both staff and participants served to make on-site improvements to raise the performance level of the sessions and of the Forum as a whole. Moderators made the appropriate changes from one session to the next, and we controlled, monitored and made better use of audio-visual resources. Additionally, all end-of-the process feedback was recorded as "lessons" for future events and will be used in our ongoing project planning.

The T²RERC owes the success of the Stakeholders Forum on Visual Impairment to all of the participants. With the help of the stakeholders, the T²RERC was able to make it a gratifying experience for participants and hosts alike. The enthusiasm and commitment of the participants have shown in working with the T²RERC has gone beyond the Stakeholders Forum. Many sent expanded contributions as post-forum messages.

The T²RERC has developed problem statements and post-Forum outcomes, using the rich material generated at the Forum. Much of that material has been incorporated into this document. Through the problem statements, the T2RERC points to problems in need of technical solutions, indicating both the current product limitations and the corresponding technical specifications. In a later step, other stakeholders (i.e., advanced technology developers) will respond by proposing the necessary technology solutions to meet the needs of people with visual impairments.

Conclusion

The T²RERC would like to acknowledge the support and input from our project sponsor, the National Institute on Disability and Rehabilitation Research (NIDRR), Smith-Kettlewell Eye Research Institute, and the Far West Region of the Federal Laboratory Consortium.

[ Top of Page ]