Framework for Integrated Synchronous And Asynchronous Collaboration (ISAAC)

 

Introduction

The ISAAC project was a collaborative technologies research effort, headed by Larry S. Jackson, at the National Center for Supercomputing Applications (NCSA) at the University of Illinois, Urbana-Champaign (UIUC). ISAAC was primarily funded by the Intelligent Collaboration & Visualization program of the Information Technology Office of the Defense Advanced Research Projects Agency under grant number N66001-96-C-8511. The research team included work by NCSA staff, the UIUC Department of Computer Science, the Image Formation and Processing Group at the Beckman Institute at UIUC, VOSAIC, Inc., and Crystaliz, Inc. The project was active from August 1996 through September 2000.

Objective

ISAAC research focused on making computer-assisted collaboration in real-time decision support, science, engineering, and education domains more natural, more powerful, and more responsive to the multi-modal communications needs of users.

Historically, trying to work together via extant computer tools has been chronically limiting. The ISAAC collaboration system enables users to function in an information environment without regard for the temporal nature of the tools they use. Through a series of modality boundary crossing efforts, ISAAC users are able to retrieve, forward, and re-use information regardless of the synchronous or asynchronous nature of the information source or destination.

Ongoing collaborative work may include sections that are necessarily or coincidentally real-time, as well as asynchronous contributions. ISAAC group members unable to participate in portions of an ongoing collaboration will not be deprived of information. ISAAC functions through a combination of techniques, including capture and replay of real-time discussions and analyses, automated indexing of captured material, incorporation of asynchronously generated materials into the collaboration stream, and asynchronous notifications and summaries of real-time events.

Principal Efforts

The ISAAC research addressed these major issues:

"Non-Invasive" Collaboration Enablement for Single-User Applets
The Habanero event interception mechanism was designed to operate as closely as possible to the Java event stack itself. Further, when Java methods for object serialization became available, they were retrofitted into Habanero. As such, automated assistance in the conversion of single-user Java Applets to Habaneroized "Hablets" was possible via a conversion wizard. A major design goal was that a user's investment in a single-user tool should be very heavily leveraged in the creation of the collaborative version. As such, any unique Habanero "API" requirements were deliberately minimized.
Sharing State, not View
The Habanero sharing mechanism shares programmer-defined state encapsulation objects upon programmer-selected events. These events can be semantic events, disjoint from events arising from the windowing system. When received, the remote collaborator's program may act upon these events in any manner the programmer wishes. For example, displays could be done entirely differently.
In this manner, a 2-D Habanero client, running on a 130 MHz Pentium laptop under Windows-95 was successfully interactively linked with a collaborative 3-D immersive CAVE application in October 1996. The total bandwidth required (sharing state, not view) was 15,000 bits per second. Such bandwidth and graphics/processing reduction clearly demonstrate the viability of this design in DoD field conditions, and in other conditions of limited processor/bandwidth such as Distance Education applications.
In a companion effort, we developed versions of a VisAD dataset visualizer as part of our Java3D work, one of which is used with a 6 Degree-of-Freedom wand and stereo glasses, and the other with a 2-D panel display. Both versions can interact over the shared data and visualization, but the views and user experiences are entirely dissimilar.
Consistent Sharing Mechanism Across Tool Types
Habanero/ISAAC employs a state sharing mechanism based on a "serialized Java object protocol". As such, objects from within software tools can be very readily transferred to concurrent copies of a tool running on a colleague's machine. This single-protocol approach allows all Habanero event-based tools to utilize the ISAAC record/search/replay mechanism. Redundant, and/or tool-specific mechanisms need not be developed.
Expansion of the Habanero "Session" object model
The initial Habanero synchronous "Session" design metaphor was insufficient to include asynchounous contributions and support for disconnected users. A revised object model was developed that separates the notion of connectedness from the notion of continuing interest. That is, just because a user must disconnect does not imply the user has become disinterested. Thus, we have adopted a "standing committee" metaphor, in preference to our former "conference call" metaphor. In a standing committee, full membership is retained even when the committee is not in the meeting room.
ISAAC utilizes a number of options in re-establishing a coherent collaboration state among the various participants. For example, state dumping mechanisms used to establish tool state for a late-joining colleague can support resynchronization of a reconnected colleague, bypassing all the discussion and interaction that has occurred while disconnected. If such behavior is not desirable, adaptations of our replay mechanism are useful to support review of the occurrences during the disconnected period.
Increasing Natural Tool Interoperability Regardless of Temporal Mode
A longstanding taxonomy of collaborative tools is based on two binary questions: (1) Is the tool usable in the same or, in different locations? (2) Is the tool used synchronously or asynchronously? This gave rise to a dichotomy in work practices wherein the work was first done, and then discussed. Collaboratively interactively doing the work itself was often a foreign concept. We have completed a series of measures designed to facilitate information flow across historical boundaries of space and time simultaneity.
  1. Real time collaborative tool use can be recorded, the archives can be searched, and the result can be replayed for participants not originally present. These archives can be posted to the Web as multimedia files in their own right, replayable via Web browser using a Habanero-specific MIME encoding.
  2. Event summaries are produced for a disconnected participant, and then made available via non-Habanero devices such as e-mail notification and the generation of personalized web pages that include expandable thumbnail images, document icons, and the like.
  3. E-mail, with optional attachments, may be sent to a running collaboration session from a disconnected participant, making that information available in real-time to the connected participants.
  4. Documents may be shared via publication via a database, in the usual manner.
  5. Files may be shared in real time via a multi-platform tool similar in nature to Microsoft's Windows Explorer. Comparable file sharing security features are provided.
  6. On-the-fly software distribution, suitable even for the distribution of repairs to Habanero itself, is supported via another real-time sharing tool.
  7. Real-time meetings may be convened by clicking links on Web pages. MIME-encoded Habanero meeting directives are passed via the Web browser to Habanero, registered as a "helper app".
  8. Real-time tools such as whiteboards and text editors may be used to edit documents for asynchronous uses, including uses entirely separate from Habanero/ISAAC.
Construction of an event, indexing, and metadata database and query package
This system serves as the primary object store for ISAAC. This facility had to be sufficiently fast to send/receive in real-time the products of a Habanero session. This facility would also, in the future, serve as the storage host for the creation of derivative informational works from captured content, including providing versioning, branching, and annotation facilities.
Maximize Platform Support
As large distributed organizations (e.g., the US military, the NCSA Alliance, large corporations, industry consortia, or systems involving the public) cannot effectively mandate use of a single computer platform or operating system software version, Habanero/ISAAC was implemented in Java, to the maximum extent practicable. Depending on the version of the Java Developer's Kit of the moment, Habanero/ISAAC has been operational on Microsoft Windows (95, 98, NT, 2000), IBM OS/2, Linux, Macintosh, Solaris (for Sparc and Intel architectures), HP, and SGI workstations, to our knowledge.

Additional Efforts

The ISAAC research also addressed these issues:

Incorporation of video feature analysis and audio segmentation algorithms
Information extracted from captured VTC materials by such means could be incorporated into the database system to augment indexing and segmentation capabilities that are available during user queries. Work on segmentation of meeting audio by spectral cues and work on tracking facial features was done in the first half of the ISAAC work. Vendor-specific hardware/software issues ultimately precluded economical integration of this work.
Construction of a representative videoteleconferencing (VTC) capability
An experimental system was developed, based on commercial off-the-shelf (COTS) hardware and software materials, and on software technologies contributed by VOSAIC, Inc. This system was sufficient to provide representative VTC data rates to stress-test capture and replay systems. Due to continual shifting of the VTC marketplace and Java support libraries, keeping this functionality current ultimately proved prohibitively expensive.
Construction of a streamed data storage facility
As COTS database offerings did not have sufficient power to support such data rates, audio and video streams had to be stored to the local file system for our replay experiments.
Development of a Timed Event Protocol to synchronize replay involving streamed data.
If only stream start times are synchronized in replays, communications via other modes (i.e., other streams, or discrete events from the various tools) will not necessarily duplicate at replay the sequence of presentations to the original participants. Alternative stream formats were utilized, interleaving video and audio packets so as to very well approximate synchronization. More general coordination of streams with discrete events was investigated.
Multicasting
We ported Secure Reliable Multicasting to Java in December 1996, incorporating features for late-joining (a late joiner presumably does not want to receive all packets since the start of the group's history, particularly if that may span months!), and for guaranteed packet ordering. A central server hosts arbitrators, of several design variations, that impose "floor control" rules on the actions of participants. Once approved, these actions can be radiated via multicast to potentially very many participants.
Security and Privacy
We incorporated SSL (based on SSLava from Phaos, Inc.) and X.509 certificates in some versions of Habanero, in connection with a research project of USAF's Rome Laboratory.
Distance Education
We supported collaborative technology projects in distance education, particularly those of NCSA's Education, Outreach and Training effort, Quineabaug Valley Community College, CT, and MITRE (Bedford, MA).
Telemedicine
In a spin-off from the ISAAC videoteleconference work, NCSA and the UIUC Department of Speech and Hearing are engaged in a telemedicine trial project. Televideofluoroscopy is being used in evaluation trials concerning remote diagnosis of oropharyngeal dysphagia. This application is demanding in that evidences of the condition require dynamic visualization via fluoroscope. As qualified diagnosticians are rare, telemedicine is particularly economically attractive, and would improve the availability of care.

ISAAC Technology Transfer

Commercial Licenses
Habanero was once licensed for commercial purposes, but the startup folded.
Government, Research, and Private Use
Commercial licensing notwithstanding, Habanero/ISAAC remain available for US government use, for research purposes, and for private non-commercial use. Installable versions of the software are provided on the Habanero homepage. An application for source license is also linked off the Habanero homepage.

References

Principal ISAAC Researchers

Acknowledgements

In addition to the research partners listed above, the ISAAC project has directly benefited from sponsorship and collaboration with the following companies, agencies, universities, and research institutes. Their contributions to this project are gratefully acknowledged.