Sensing Activity in Video Images

Alison Lee
NYNEX Science & Technology
White Plains, NY 10604, USA
+1 914 644-2322

Kevin Schlueter
University of Toronto
Toronto, ON M5S 1A4, Canada
+1 416 978-0835

Andreas Girgensohn
FX Palo Alto Laboratory
Palo Alto, CA 94304, USA
+1 415 813-7244


Video-based awareness tools increase familiarity among remote group members and provide pre-communication information. Low-cost iconic indicators provide less but more succinct information than video images while preserving privacy. Observations of and feedback from users of our video awareness tool suggest that an activity sensing feature along with a variety of privacy options combines advantages of both the video images and iconic indicator approaches. We introduced the activity sensing feature in response to user requests. It derives activity information from video images and provides options to control privacy and improves the usability of video-based awareness tools.


Activity sensing, awareness, differences, privacy, usability.


A classic problem in CSCW is supporting group and collaboration awareness in distributed work groups. Awareness is thought to increase familiarity among co-workers as well as provide important pre-communication information on presence, interruptibility and availability. This problem was first solved by allowing group members to see video images of each other. Xerox Portholes [2], Telepresence Postcards, and NYNEX Portholes use periodically refreshed video snapshots while Montage [5] uses brief reciprocal glances.

Recently, a number of techniques have been proposed which use low-cost iconic indicators as an alternative to video images [1, 3, 6]. These indicators sacrifice a great deal of the information available in video awareness tools but provide more succinct information about presence, and availability while, in some cases, ensuring more privacy.

Observations of and feedback from users of our video awareness tool suggest that it is unnecessary to sacrifice the rich information provided by video images in order to achieve the advantages of the indicator approach. To support our point, we describe a capability in NYNEX Portholes that provides activity sensing and increased user control over privacy. We developed this capability in response to our user requests. We believe that this capability provides an improvement over both the traditional video-based awareness tools and iconic indicators because it meets three crucial user requirements. This paper describes the activity sensing feature, the three user requirements, and how the activity sensing feature meets these requirements.

Figure 1: Portholes Viewer


In our Portholes system, a video image for each user is snapped once every five minutes. The images for the last hour (12 of them) are kept. The activity sensing capability uses a image difference mechanism to derive a measure for the amount of change (1 <= change <= 100) between two successive video images. This measure is represented as a red horizontal bar at the bottom of the video image and can optionally be used to order the images in the Viewer (Figure 1). The red bars have the added benefit of making images with lots of activity "jump out" so that users can easily do a visual scan for activity. The "sort by amount of activity" feature displays images with lots of activity earlier in the matrix. Therefore, users can use a smaller Portholes window and still see all colleagues who are in their offices.


Video images are used in our awareness tool because the visual channel provides many of the pieces of information that people find valuable for awareness. Experiences with Xerox Portholes [2] indicate that sightings, presence and availability are some information that users remember and seek out. In addition, video provides as much information as possible so that we could get maximal feedback about what is useful and what needs to change. After more than two years of use at three NYNEX sites and three distant universities, we have accumulated many user requirements. We present the three that relate to the activity sensing feature.

Increased Activity Awareness

Our users require an indication or a notification of the activity or changes in a co-worker's work area. However, different users seek different kinds of activity information (e.g., presence, absence, availability, interruptibility). Displaying a matrix of the current video images of one's co-workers is not always sufficient (e.g., to determine whether a co-worker has just left, the user has to examine the co-worker's last 12 images). Thus, to make activity or changes accessible, we developed the activity sensing feature.

User Controls on Privacy and Information Disclosure

Figure 2: Activity Bar Chart

Although our users are willing to disclose awareness information to their co-workers, the rich information in the video images raises concerns about privacy, the amount of information disclosed and cameras in the work place. Some users do not want to provide specific activity information but only activity changes. To address these concerns, we use activity sensing to provide a range of privacy options. Users wanting maximal privacy or minimal disclosure can simply show an activity bar chart which gives a single integrative view of the changes over the last hour. Users willing to disclose more but who are uncomfortable with detailed video images can provide activity bars along with images blurred at one of two levels (e.g., "black" image in Figure 1). Some users have mentioned that they are more comfortable with using the activity sensing feature with a blurred image because unflattering images are imperceptible.

Usability of Portholes Viewer

The Portholes Viewer (which displays the video images matrix) requires screen space even when used as a backdrop. However, when a Portholes community is large (many video images), a usability problem is created because not all images are visible in the Viewer and some may be obscured by other windows on the desktop. The activity-based ordering of images combined with setting a cut-off for images to display addresses this problem. The combination of features turns the Portholes Viewer into a visibility filter which preserves valuable screen real estate and optimizes scanning and peripheral monitoring. We have observed that some users are further exploiting this combination by using a reduced size Viewer window that is positioned at an unobstructed corner of their desktop.


Our image differencing algorithm, which is the foundation for activity sensing, differs somewhat from those used elsewhere. Telepresence Postcards, for example, has a Sense option that allows a user to be alerted when a change in the current image exceeds a set threshold. While Postcards also uses an image differencing algorithm to support this option, our algorithm is more complex because it attempts to ignore changes that are simply due to shifts in illumination and video digitization noise. To accomplish both, we gamma correct and blur the image slightly. Next, we compute two different image difference metrics:

In our experience, the first metric is very good at detecting changes in images due to people moving as well as people entering and leaving the frame. However, it is quite sensitive to changes in image brightness due to actual illumination changes or the automatic brightness in some cameras. The second metric is quite insensitive to changes in image brightness. By taking the smaller of the two values, we get a measure that is less sensitive to illumination changes.


We are presently exploring two approaches to improving the activity sensing. The first identifies other candidate statistics for the difference metrics that describe the distributions for the differences between corresponding pixels. The second, inspired by [4], determines the presence or absence information using reference video images.

NYNEX Portholes is a vehicle to explore, through use, the value of video awareness tools for distributed work groups and the rich set of awareness information available in video images. Our users have told us that they are willing to provide certain information but they want to exercise different degrees of control over the amount and form of the information disclosed. Activity sensing permits this while also enhancing usability. Our experiences strongly suggest that this is a worthy addition to video-based awareness tools.


The research was performed while all the authors were at NYNEX.


  1. M. Ackerman and B. Starr. Social Activity Indicators: Interface Components for CSCW Systems. In UIST'95 Conference Proceeding, ACM Press, pp. 159-168, 1995.

  2. P. Dourish and S. Bly. Supporting Awareness in a Distributed Work Group. In CHI'92 Conference Proceeding, ACM Press, pp. 541-547, 1992.

  3. S. Greenberg. Peepholes: Low Cost Awareness of One's Community. In CHI'96 Conference Companion, ACM, pp. 206-207, 1996.

  4. S.E. Hudson and I. Smith. Techniques for Addressing Fundamental Privacy and Disruption Tradeoffs in Awareness Support Systems. In CSCW'96 Proceeding, ACM Press, pp. 248-257, 1996.

  5. J.C. Tang and M. Rua. Montage: Providing Teleproximity for Distributed Groups. In CHI'94 Conference Proceedings, ACM Press, pp. 37-43, 1994.

  6. T. Wax. Red Light, Green Light. In CSCW'96 Conference Companion, ACM, pp. 1-2, 1996.