Key results

• Evaluation techniques and quantitative metrics based on realistic scenarios
• List of common patterns, techniques and guidelines
• List of key projects, systems and papers
• Agreement on some key open problems and well understood approaches for ubiquitous systems
• Decision to start a community portal to host this information, along with coursework and links for new members of the community

Action Items

• Rodger/Ade to set up initial Wiki (Partly done:-)
• Rodger/Mike - summarize results and tidy up google spreadsheets - done see PatternsTechniquesGuidelines and Projects and Systems. We should now maintain these on the wiki.
• Decide on future workshops and possible conference call.

Summary of Outcomes

Reflecting on the past

Several in the group felt that as a result of the workshop they had the opportunity to reflect on past experience, both their own, and others. Because of this, some thoguht that they didn't, nor did they expect to generate any "new" ideas, but did have a better idea of the tools and systems to build on.

Some participating in the evaluation breakout section did feel that they had a better understanding of some useful metrics for evaluation including quantifying the "noise" or "annoyance factor" that a system (or lack of a system) can be evaluated against.

Overall, the group felt it was comforting to find agreement on research areas that are "done" or that are "half-baked". Half-baked is the idea that not only is a problem not adequately solved, it may not be a problem worth solving.

An interesting point discussed was that the world has markedly changed since Mark Weiser first presented his vision of ubiquitous computing in 1991. Specifically, the web was introduced, and since then has evolved to include not only desktop computers but also mobile devices, and small fixed devices including sensors, actuators and everyday objects. Some felt that the idea of the "mobile web" and web-ish things is a huge deal and it should be a bigger thing to this community than it is.

Not invented here, and Grand Challenges

Another issue facing the community is the notion of NIH - we should get over it and move forward. Almost by definition, ubicomp has been perpetually in the future, but here we are trying to make it real, today. If it is happinging today, what is it? Is it just the notion of mapping the physical world to virtual resources?

Considering grand challenges: emerging coutries may be opening new opportunities for ubicmp such as addressing non-leteracy. Usage patterns and user interfaces for these countries may be very different. Other grand challenges may include the use of sensors on phones for environmental monitoring, gathering information for grand purposes to benefit humanity.

Patterns and Systems

Regarding patterns and systems: the group felt that existing systems can be used to illustrate best practices and patterns used. These patterns should be reflected apon, and shouldn't be "new". We should encourage organizers of conferences to inlcude patterns as papers, papers that are more pragmatic than scientific.

Evaluation

Regarding Evaluation: the group felt that keeping track of what systems use certain well understood patterns would help with evaluation. Systems could be ranked and discussed using blogs and commentaries, and practitioners should be recognized for their use of "best practices".

The group discussed the use of scenarios. Many thought it would be difficult to have only a few that was both broad and specific enough to be used for evaluation, but perhaps a set of themes or classes of scenarios could be developed, with sub-scenarios found to be useful for evaluating very narrow aspects of systems such as location sensing. We should also consider areas often ignored such as managability. These metrics need to be quantified in some way. Overall, enumuerating the categories for pervasive/ubisys evaluation would be a great start.

Test beds should also be considered as an evaluation tool. We need virtual and physical testbeds, and to collect data sets. A key requirement is repeatability. Provjects like those at Dartmouth and Placelab built up a large database, and we need to market and publicize thse datasets.

Moving Forward

To move forward, the group felt that it was important to get the infomratin discussed on line (www.ubisys.org). This can then be populated with information from the related workshops. It should be possible to put together a bibliography of links, lists of patterns, coursework, etc.

An online presence could also help us report our work. Not necessarily in a completely published state, but at least noted as "published". We could start an online conference model. Others can contribute, and become collaborators in "ubisys" technical reports, as well as other artifacts like video and images.

We could start off with a list of design patterns and best practices, then use these to create list of scenarios. RFPs could be created for papers, asking for collaborators.

How do we get people involved? Workshops are good, but are we spread too thin? Could we bootstrap the establishment of a more formal Ubisys community through a multiday event? If so, could we publish 2+ papers from the event? Perhaps 2 workshops a year is enough? What about a somewhat regular conference call?

Action items from this discussion would be that we would look into funding for the group, announce a community web site and start populating it with our community ideas.