The Mobile Memex

by

R.M. Vaandrager

Submitted to the Department of Computer Science

in partial fulfillment of the requirements for the degree of

Master of Science in Information Science

at the

VU Amsterdam

October 2003

c© PinkRoccade 2003. All rights reserved.

Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Department of Computer Science

Oct 31, 2003

Certified by. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Anton Eliens

Thesis Supervisor

2

The Mobile Memex

by

R.M. Vaandrager

Submitted to the Department of Computer Scienceon Oct 31, 2003, in partial fulfillment of the

requirements for the degree ofMaster of Science in Information Science

Abstract

Converging trends in mobile information technology open the doors for new kinds of applicationsand services that deal with personal and contextual information. This thesis present a conceptualframework and component architecture for a personal information system that serves as an auto-biographical database of an individuals life that can be searched for information, relations, contextand meaning. The information system is able to acquire, aggregate, annotate and archive the user’scontext and content. It does this by using sensors and other system resources available in commonpersonal information devices such as the watch, mobile phone and PDA. These portable devices areencapsulated in an ad-hoc body area network (BAN) worn by the user.

We identify how contextual information can be structured, stored and made available at the appli-cation and service level. One such application is a personal contextual search engine that enablesthe user to access archived content by formulating search queries based on context. Retrieved infor-mation can in turn serve as contextual cues that can aid the user in the process of retrieving pastexperiences with a high degree of accuracy. The search engine application also serves as the basisof a proof of concept demonstrating the feasibility of the system.

3

4

Acknowledgments

I would like to thank all the people that contributed to the completion of this thesis:

Thiel Chang, my supervisor and PinkRoccade R&D, for offering me a place to do my research, forencouraging me to choose my own subject and the active coaching during the entire timespan of the(extended) research project.

Anton Eliens at the faculty of Computer Science at the VU Amsterdam for accepting my researchproposal and the frequent meetings during which I gained new insights and the renewed focus Ineeded so much.

Martijn v/d Welie for evaluating the M3X prototype and his valuable suggestions on how to furtherimprove it.

Tim van Galen, for being an good companion and for listening to my many moments of frustrationduring our walks in the park and the animal-farm when things didn’t quite go the way i envisioned.

Jelle Gerbrandy, for his excellent insights and help on UML modeling and the essential dosis ofcaffeine he provided me on a regular basis.

Martijn van Denderen for allowing me to use his excellent workplace and 21” TFT monitor duringhis absence.

And finally my parents for encouraging me to pursue my own interests, and providing me with bothmental and financial support.

5

6

Preface

Project M3X - the mobile memex will be conducted within the Freeband framework. Freeband is ajoint initiative by the Dutch government, leading companies and knowledge institutes with the col-lective aim to create a knowledge base for the construction and application of the upcoming ambientintelligent environment of 2010. In the ambient intelligent environment communication infrastruc-ture will become transparent and abundant in all layers with the individual as a new paradigm forcommunication.

The idea for this thesis (a system that uses common personal information devices like the watch,mobile phone, digital camera and the PDA to capture the context of the user by using the avail-able sensors, and make this information available to user by a google-like search engine that can beused to retrieve meaningful personal context and content) came to me after I temporarily lost myown short-term memory due to a skiing accident. This made me realize the importance of mem-ory to properly function in daily life. Human memory is not without it’s drawback however; oftenmemories are forgotten or become distorted. The mobile memex is an individuals autobiographicalcontext system that records and stores the contextual facts that can serve as retrieval cues for anindividual and in turn augment his own memory so experiences can be retrieved with a higher degreeof accuracy. The expression ”Sometimes an idea just hits you” can be taken very litteraly in this case.

7

8

Contents

1 Introduction 131.1 The Memex; Vision and Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.2 Trends in Information Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.3 The Digitalization of Devices and Media . . . . . . . . . . . . . . . . . . . . . . . . . 161.4 The Need for Augmentation and Integration . . . . . . . . . . . . . . . . . . . . . . . 161.5 The Body Area Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171.6 The Mobile Memex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.6.1 Scenario 1: Concrete-jungle Jane . . . . . . . . . . . . . . . . . . . . . . . . . 181.6.2 Scenario 2: Health-care Karen . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.7 Research Goals and Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.7.1 Research Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.7.2 Overview of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2 A Conceptual Framework 232.1 The Physical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.1.1 Personal Information Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.1.2 Sensors, Actuators and Resources . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.2 The Communication Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.1 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.2 Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.3 The Information Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3.1 Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3.2 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.3.3 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.3.4 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.3.5 Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.4 The Application Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.4.1 Application Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.4.2 Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.5 The Augmentation Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.5.1 Licklider’s Man-Computer Symbiosis . . . . . . . . . . . . . . . . . . . . . . . 422.5.2 Augmentation Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432.5.3 Privacy, ethics and other social issues . . . . . . . . . . . . . . . . . . . . . . 45

3 Requirements and Design 473.1 M3X Software Requirements Specification . . . . . . . . . . . . . . . . . . . . . . . . 47

3.1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.1.3 Definitions , acronyms and abbreviations . . . . . . . . . . . . . . . . . . . . 473.1.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483.1.5 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.2 Overall Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

9

3.2.1 Product Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.2.2 Product Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2.3 User Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.2.4 User Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.2.5 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.2.6 Assumptions and Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . 613.2.7 Apportioning of Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.3 Specific Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613.3.1 External Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

3.4 General Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.4.1 Performance Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633.4.2 Logical Database Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 633.4.3 Design Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633.4.4 Software System Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.4.5 Other Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4 A Proof of Concept application 714.1 The Personal Contextual Search Engine . . . . . . . . . . . . . . . . . . . . . . . . . 714.2 A base-line architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.3 User Interface Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.4 Prototype Walkthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5 Conclusions and Future Work 875.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 875.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

A Figures 93

B Use-Case-Model Survey for M3X UML diagrams 101B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102B.2 Actors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102B.3 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

C Mobile Memex Sourcecode 109C.1 Subscriber.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109C.2 Publisher.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112C.3 M3xJXTAPeer.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114C.4 mdidgen.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

10

List of Figures

1-1 Mobile memex: capture personal context information and make it available to appli-cations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1-2 From Timesharing Systems to Mobile Computing . . . . . . . . . . . . . . . . . . . . 16

1-3 This Figure contains two panels: (a) The location of the PAN and BAN in the networkhierarchy, and (b) Some common devices encountered in BANs and PANs . . . . . . 18

2-1 The Mobile Memex Conceptual Framework . . . . . . . . . . . . . . . . . . . . . . . 23

2-2 Functional overlap between personal information devices . . . . . . . . . . . . . . . . 24

2-3 Some characteristics of the 3 distinct device classes . . . . . . . . . . . . . . . . . . . 26

2-4 Sensors, Actuators and System resources in personal information devices . . . . . . . 27

2-5 Overview of sensor characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2-6 Network types and their corresponding technologies . . . . . . . . . . . . . . . . . . 28

2-7 Data processed into context and content in the M3X framework . . . . . . . . . . . . 30

2-8 Profiles in the mobile memex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2-9 Example XML profiles and relations . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2-10 An entity influences it’s environment and vice-versa . . . . . . . . . . . . . . . . . . 33

2-11 Events, states, activities and actions . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2-12 Instances of entities, states, activities and events . . . . . . . . . . . . . . . . . . . . 34

2-13 The five context dimensions and their ID encodings . . . . . . . . . . . . . . . . . . . 35

2-14 Types and instances of the content classes . . . . . . . . . . . . . . . . . . . . . . . . 36

2-15 Context and content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2-16 An approximation of daily storage requirements for a typcial mobile memex system. 37

2-17 Automatic annotation, and related content query suggestions . . . . . . . . . . . . . 41

2-18 Application characteristics in the mobile memex . . . . . . . . . . . . . . . . . . . . 42

2-19 This Figure contains two panels labeled (a) showing augmented surveillance, and (b)showing augmented coaching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2-20 This Figure contains two panels labeled (a) showing augmented medication, and (b)showing augmented navigation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2-21 Augmented memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3-1 Overview of the M3X system architecture . . . . . . . . . . . . . . . . . . . . . . . . 49

3-2 Diagram view of functional interaction between devices . . . . . . . . . . . . . . . . 53

3-3 A typical device architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3-4 Networks, bandwidth and power consumption . . . . . . . . . . . . . . . . . . . . . . 58

3-5 The XML structure of an example context history . . . . . . . . . . . . . . . . . . . 64

3-6 Mobile memex administration UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3-7 I/O functions for publishing, subscribing and storage devices . . . . . . . . . . . . . 69

4-1 Some sample questions a user might ask the information system . . . . . . . . . . . . 72

4-2 Overview and location of software components in the prototype . . . . . . . . . . . . 72

4-3 Overview of the base-line architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4-4 Software components in the proof of concept. . . . . . . . . . . . . . . . . . . . . . . 74

11

4-5 This Figure contains two panels: (a) showing the search window, and (b) showing theresult browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4-6 UML class diagram showing the JAVA classes for the M3X subscriber and publisherdevices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4-7 Initializing the Xindice NXD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764-8 Publish: The publisher console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824-9 Subscribe: The subscriber console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834-10 Store: View of nodes in the Xindice M3Xdb . . . . . . . . . . . . . . . . . . . . . . . 844-11 Initialization of Apache/Tomcat and the XINCON servlet . . . . . . . . . . . . . . . 854-12 Access: View of nodes in the remote search application . . . . . . . . . . . . . . . . . 85

A-1 List of used acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93A-2 An UML expanded class diagram showing the JXTA devices classes in the Mobile

Memex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94A-3 An UML activity diagram showing interdevice interaction . . . . . . . . . . . . . . . 95A-4 An XML example of a context history containing two timestamps and content references 96A-5 An UML sequence diagram for the personal search engine application . . . . . . . . 97A-6 UML Use case diagrams representing the publishing, subscribing and storage device 98A-7 UML Use Cases representing the differerent applications and devices . . . . . . . . . 99A-8 An UML Use case diagram showing the actors in the M3X . . . . . . . . . . . . . . . 100

12

Chapter 1

Introduction

”A device in which an individual stores all his books, records, and communications, andwhich is mechanized so that it may be consulted with exceeding speed and flexibility. Itis an enlarged intimate supplement to his memory.” - Vannevar Bush

This is how Vannevar Bush described an imaginary system he called the ”Memex” or short formemory extender in his 1945 essay ”As we may think” [1]. The Memex was primarily designed todeal with the information overload scientists in his day had to cope with. It allowed an individualto store all his documents, and create associations between them to capture what he called ”trailsof thought”. These trails would cover a specific topic and could be traversed at a later date. Dueto technological limitations of the time the system never materialized. The idea of ”associativeindexing of information” on which the system was based has survived and has found a rudementaryimplementation on the internet in what is known as ”hyperlinks”. The internet however deals withcommon information. When it comes to managing personal information the individual is still largelyat a loss. Today we deal with vastly increased amounts of personal information and the need toorganize this information is all the greater.

This thesis proposes a mobile memex: A portable system composed of common information devices,that is highly personal in nature and can be customized to fit the users functional needs. The goalof the mobile memex is to capture the context of the user, including the content that was created inthis context, and make it available to applications that can augment the user. A personal contextualsearch engine is one such application. Instead of following trails of thought, the user will be able tofollow trails of context in order to retrieve relevant and meaningful information.

Common information devices can contain a number of sensors that can measure were we are, whatwe see, what we hear and more. Figure 1-1 shows the basic idea behind the mobile memex; capturingthe context and content of it’s user.

In this chapter we discuss some of the ideas beyond the original memex and how they can beimplemented in a mobile memex composed of personal and portable information devices to augmentthe user in different ways.

13

1.1 The Memex; Vision and Function

The original memex was designed to provide three main functions:

1. Add documents: The user could add new documents in microfiche format to the system.

2. Annotate documents: The documents could then be annotated by the user, effectively addingmeta-data; information about the information.

3. Associate documents: If the user found documents logically related to each other in he couldcreate an association between these documents. Eventually this could lead to entire chains oflinked documents that capture a certain idea or a chain of reasoning.

To manage this mechanical system would probably be very user intensive, documents had to bescanned and transferred to microfiche, manually placed in the document stack, and finally anno-tated and associated by hand. Now it is possible to use computational systems to automize anumber of tasks, and digitally store all documents. The mobile memex aims to automize this sys-tem of adding, annotating and associating documents by developing a generic sensor plaform able toaggregate sensor data from distributed devices in a Body Area Network (BAN). Sensor data will besplit up into context data and content data, where context data will serve as an associative ”glue”to related content data. The system strives to minimize user input and intervention, while stillproviding the possibility to manually add annotations and associations where necessary.

Mon

5JAN

12

6

39

Bla bla

bar

Mobile MemexTemp

Air pressure

images

Sound, voice, chatPulse

Altitude

Location

Direction

Date & Agenda

Time

Mail

Figure 1-1: Mobile memex: capture personal context information and make it available to applica-tions

1.2 Trends in Information Technology

Developments in information technology open up new possibilities in how we organize and deal withpersonal information. The realization of the mobile memex is dependent on the development of anumber of key trends:

14

1. Information technology becomes affordable:Advances in lithographic techniques combined with economies of scale lead to smaller inte-grated circuits (IC) and generally mean cheaper IC’s that consume less power.

2. Information technology becomes personal:Affordable computing power initiated a move from time sharing systems such as mainframesto the personal computer many people have on their desks right now. The PC will not be theonly personal computing device anymore; An increasing number of people will own and use arange of information devices such as camera’s, phones, laptops and PDAs. The way we usecomputers to communicate and organize information is constantly changing, and according tothe Freeband initiative we are currently on the edge of a ’paradigm change’, which will movethe center of information control to the individual. Right under his personal guidance willbe a ’Wearable Digital Assistant’ that will serve as a direct extension of his intelligence [41].Figure 1-2 show the move from time sharing systems with multiple users to mobile computingwhere a user shares his time with multiple devices.

3. Information technology becomes portable:Currently we can identify a shift away from desktop computing to mobile computing. Minia-turization and integration of electronic components allow for smaller computing devices, suchas the notebook computer, the personal digital assistant and the so called ”smart-phones”that are becoming more mainstream. An advantage of portable computing devices is thatthe system always roughly shares the same context as it’s user. Standardization of softwarecomponents such as the TCP/IP network protocol, the XML markup language and and theJAVA platform independent programming language also increase portability and compatibilityon a software level.

4. Information technology becomes connected:After wiring workstations and personal computers to form the internet, it is now the turn ofwireless personal information devices and even domestic electronics. There are many differentwireless network system from the very long range for mobile phones to the very close range likeinfrared remote controls. The recent years have shown a widespread acceptance of standardizednetwork technologies, making it easy and cost effective to integrate them into a wide range ofdevices to extend their functionality. The result is an abundance of networked devices. Wirelesscommunication initiates an information shift away from the desktop and into a connectedmobile world.

5. Information technology becomes ubiquitous:The ongoing miniaturization of electronic components will lead to Ultra Large Scale Integrated(ULSI) circuits such as the system on a chip (SoC). Assuming continued progress in batterytechnology, minute but computationally powerful SoCs will find important uses in many areas.These kind of small and affordable integrated circuits will be embedded in the fabric of theworld around us. We see this happening with personal devices but also in every day productssuch as home equipment, kitchen utilities, cars, doors and even products in the supermarketthat contain embedded Radio Frequency ID (RFID) chips. Eventually this development willlead to the realization of the ubiquitous computing vision [3] where the physical world isaugmented by a digital overlay of embedded computing devices.As our environment becomes more and more ubiquitous we find that the channels throughwhich we conduct daily activities also become increasingly digital. For example; instead ofwriting a letter on paper and mailing by post, we write an email and send it over the network.We buy products on the internet and manage our finances online. Phone calls are made usingdigital telephony systems or by using voice over IP (VoIP). Products and services are payedfor by using a cash card that automatically deducts the amount from your bank-account. Weaccess to cars and buildings using digital key-cards and the list goes on. Every interactionmade through a digital channel generates information that can be potentially be captured toserve as contextual cues in the mobile memex.

15

1970:Timeshare Computingone system, many users

1990:Personal Computingone system, one user

2010:Portable Computingmany systems, one user

Figure 1-2: From Timesharing Systems to Mobile Computing

Essentially all of the above mentioned trends are a direct result of what is known as Moore’s Law 1

combined with economies of scale2. In the near future, the convergence of these trends will provide uswith an abundance of multi-functional devices that are affordable, portable, personal and connected.It is these devices the mobile memex will focus on.

1.3 The Digitalization of Devices and Media

Another direct result of the trends mentioned in the previous section is that digital devices can takeover more and more functions that were traditionally performed by mechanical devices. Take forexample the photo camera. A traditional camera would store the image on a light-sensitive stripof film, where a digital camera can now store a digital bitmap of the same image on flash memory.The same is true for the music player where the cassette walkman is being replaced by digital musicplayers that operate on the digital equivalent of the analog cassette be it a compact disk, minidiscor memory card. A mechanical watch with delicate radar works can be replaced by a digital wrist-watch. The agenda and address book have found a digital equivalent in the personal digtial assistant.

One thing all these digital devices have in common is that they can store their content on a com-mon medium such as random access memory (RAM) or magnetic hard disk drives (HDD), wheretraditional content each had their own specific medium be it paper, tape, vinyl, film or other mate-rials. This development offers a number of important benefits for portable information devices andconsequently the mobile memex:

1. A shared storage medium decrease cost and weight and can provide a central access point forthe user to access all content.

2. Digital content can easily be manipulated, copied and communicated to others.

1.4 The Need for Augmentation and Integration

People naturally augment their own abilities with technology that enables us to cope with thecomplex world we find ourselves in. Be it a car for transportation, spectacles to improve our vision,or a post-it note to remember an appointment. Some technologies address basic and fundamentalneeds so we carry them around with us on a daily basis and can hardly live without them anymore.Some examples of traditional technologies that people historically use to augment themselves are inchronological order:

1The observation made in 1965 by Gordon Moore, co-founder of Intel, that the number of transistors per squareinch on integrated circuits had doubled every year since the integrated circuit was invented. Moore predicted that thistrend would continue for the foreseeable future. In subsequent years, the pace slowed down a bit, but data densityhas doubled approximately every 18 months, and this is the current definition of Moore’s Law, which Moore himselfhas blessed. Most experts, including Moore himself, expect Moore’s Law to hold for at least another two decades.

2Economies of scale occur when mass producing a good results in lower average cost. Economies of scale occurwithin an firm (internal) or within an industry (external).

16

1. Clothes and jewelry, used to protect our body from the natural elements, but also used tobroadcast social information such as marital status, occupation and wealth.

2. Money, used to facilitate the exchange of physical goods and services.

3. Keys, used to secure private property and define rules of restriction and access to entities.

4. A watch, used to synchronize our activities with the world around us.

5. And recently the mobile phone, used to address the need for unicast communication withothers.

The mobile phone is the most recent addition but has the potential to become the encompassingdevice because of the possibility to incorporate the digital equivalents for most of the above listedentities with the notable exception of clothes. Money for example has a digital equivalent in theform of the cash-card, Keys in the form of key-cards. Integrating this functionality into the mobilephone will lessen the burden to carry around heavy keys and coins.

Besides keys and coins, a typical business man in Amsterdam might carry around a number of dig-ital information devices to augment his abilities and address his functional needs. It would not beunusual to find one person carrying around a notebook computer, a mobile phone, a PDA, a digitalmusic player, a digital camera and a digital wrist watch all at the same time. All these devices havetheir own user interface, functionality and power-source. This means that there is a lot of functionaloverlap that might be partially eliminated by functional integration or by networking devices in or-der to share resources with others. We will examine functional integration in more detail in chapter 2.

If current developments continue, people will have access to a multitude of wearable devices withdifferent functionalities within a body area network. These devices will be able to communicateto each other and allow the sharing and storage of all input and output information. Because thedevices in a body are network are very personal in nature so is the information they deal with.

1.5 The Body Area Network

”As electronic devices become smaller, lower in power requirements, and less expensive,we have begun to adorn our bodies with personal information and communication ap-pliances. Such devices include cellular phones, PDAs, pocket video games, and pagers.Networking these devices can reduce functional I/O redundancies and allow new conve-niences and services.” - Thomas Zimmerman.

The Personal Area Network (PAN) and it’s derative the Body Area Network (BAN) allow devices inclose proximity to connect and share information. Today many devices contain network interfaces,and some network technologies like Bluetooth are specifically targeted to operate in a PAN or BANenvironment.

Figure 1-3 shows a hierarchy of the most common network types and the type of devices one canexpect to find in the PAN/BAN domain. In a PAN, a user will typically use a wireless solution sincethe objects and devices will be scattered around in his vicinity. for a BAN however, the user mightprefer a wired solution because the information contained in the BAN might be highly personal innature and therefore demand a high level of privacy and security a wireless solution cannot offer.

Ultimately, The PAN and BAN network environments will be a rich source for the collection ofpersonal contextual information and files.

17

Internet

(W)WANATM

(W)MANCable�/�(A)DSL802.16a

(W)LAN802.3�/�802.11

(W)PANIrDA (3m)Bluetooth (10m)Wifi�(100m)802.15Bodylan�(2m)

PAN�devices

BAN�devicesWatch

Phone�/�PDAAgendaCameraWalletKeys

AVTV

DVDRadioPhone

IntercomBeamer

KitchenFridge

MicrowaveCoffee�machine

OvenToaster

BuildingDoorDeskLight

Candy�machineWindowComputer

PCPrinter

ScannerCamera

StreetStreetlight

ATMBillboard

Drink�AutomatParking

(a) (b)

Figure 1-3: This Figure contains two panels: (a) The location of the PAN and BAN in the networkhierarchy, and (b) Some common devices encountered in BANs and PANs

1.6 The Mobile Memex

Networking personal devices can eliminate functional overlap, and the combination of resources andfunctions into a distributed system will also allow for new applications and services that make useof this synergy between devices. As mentioned before, the aim of the mobile memex is to developa common framework on which these kind of applications and services can be based. The followingtwo scenarios will highlight some of it’s potential uses.

1.6.1 Scenario 1: Concrete-jungle Jane

Jane, a real-estate saleswoman in Amsterdam, recently bought a new communicator that allows herto optimize her lifestyle. Her communicator has a large color touch screen and a range on function-alities: It can take pictures and record movies with the on-board digital camera, she can use it as avoice phone, check her on-line electronic agenda, manage her emails, use it as a personal navigationsystem, play her favorite music on it and much more. Nevertheless the entire package easily fits inher pocket. Besides having a lot of built in functionality it is also able to communicate with otherelectronic devices and jewelry she is currently wearing (her watch, digital ID card in her wallet, andJava-enabled ring on her finger) and other devices around her (like the ATM on the corner) shecomes in contact with during the day.Now her communicator has become a very personal part of her live. Jane doesn’t have to worrythat the content she receives or creates will be lost because of the automated context annotationinherent in the system. If she can remember some of the contextual clues in which the content wascreated, she will be able to retrieve it.

During lunch with a friend at a local restaurant Janes communicator rings. A guy called Bob isinterested in a mansion under her care and wants to make an appointment at two o’clock. Janeagrees and promises him to send him the details. Jane has been to the house before and uses hercommunicator to look up the address, and images she took in the house. She selects the nicestfive and attaches a map with the address to and email she sends to Bob right after the phone con-versation. Because she has to put some things in order at the house before the customer arrivesshe has to leave right after dinner. She quickly pays for dinner by pressing the ”OK” button at

18

the counter. Because her communicator is connected to the cash-card in her wallet, and both areconnected by her body which functions as a ”wet-wire” the cash is automatically deducted fromher account simply by touching the button. The OK button lights up to indicate the transfer wassuccessful. Her communicator automatically registers the activity and the time and location were ittook place and updates her personal bank account.

Her car also opens up for her when she touches the handle of the front door. Janes places thecommunicator in her car cradle at the dashboard and uses it to calculate the fastest route from herpresent location to the house. The communicator knows her current location and contacts externaldatabases on the internet to get a recent map of the area and also traffic information. In the car shereplays the previous phone-call with Bob from her communicator because in the restaurant it wasvery noisy and she didn’t quite catch every thing Bob told her.

By the time she arrives at the front gate she has a good idea what Bob is looking for. Bob likesa house with a lot of sunlight. Fortunately its a sunny day and according to her communicator(that is in contact with the barometer in her wristwatch and the weather server on the internet)will stay like this for the remainder of the day. The house is pretty messy and there are someovergrown plants outside hanging in front of the windows. She has to get rid of them and hercommunicator cannot help her with this. Luckily she pulls it off within 15 minutes and the windowsare now clear allowing the sunshine to stream into the house making it look its best. Now she hasto finish the inside of the house and after 30 minutes of running around her communicator noticesJanes increase of pulse and body heat as she works. With a slight buzz the communicator notifiesJane to take it easy. Jane agrees and decides to make a cup of tea to relax and wait for Bob to arrive.

When Bob finally arrives she walks out to meet him and they shake hands. Because Bob also iscarrying a communicator they exchange electronic business cards in the handshake. Over some teathey discuss the house and after that Bob get the full tour through the house and seems to be ratherimpressed when they part again. Jane feels confident she can seal this deal before the week is over.

1.6.2 Scenario 2: Health-care Karen

Karen has her day off and has decided to spent her time well with some exercise in the park. Whileshe is jogging her sports watch keeps track of her current speed, her temperature and heartbeatand how much calories she is burning. The communicator in her pocket is keeping track of hercurrent location and direction. She doesn’t really need most of this information at the moment butnevertheless it is recorded and stored on the disk inside her communicator.

However, because Karen has got diabetes she is very interested in the information the bio-graphingsensor in her watch produces. The bio-graphing sensor in her watch takes non invasive glucose read-ings through the skin at regular intervals every 20 minutes. This information is then sent over herbody to her communicator in her pocket where a special program can analyze the data in real-timeand recognize tell-tale patterns that warn of excessive glucose in her blood.

When the level of glucose rises above a certain level the communicator automatically activatesanother device on her body that contains an insulin pump to administer the required dose based onthe calculations of the system. If on the other hand the level drops below the acceptable level hercommunicator will notify her so she can take appropriate counter-measures by eating glucose richfood. In the rare occasion when the pump fails or is empty and her levels exceed a critical pointthe system will automatically notify an alarm service by use of the built in wireless WAN capabilityin her communicator and transfer her current location and medical data so aid can be sent to herstraight away. In the past Karen had to test her blood manually every day, and inject the insulinherself. Now Karen doesn’t have to worry about her glucose level all the time and can enjoy herwalk in the park.

19

1.7 Research Goals and Thesis Overview

The scenario’s in the previous section illustrate the flexibility of the mobile memex.In summary: The mobile memex is a collection of networked personal information devices andaugmentation applications. The devices are enhanced to provide the functionality required for theacquisition, aggregation and archiving of contextual data. Applications operate on this contextualdata in order to augment the user in daily tasks. This means that there is no ”one” mobile memex:The composistion of the system is dependent of the users functional requirements in variable cir-cumstances and situations. It also means that the mobile memex should provide a generic platformbased on open standards that can be easiliy implemented in a wide range of personal informationdevices and support the development of compatible applications designed to augment the user.Apart from the two scenarios mentioned above, there can exist many different types of applicationsthat are able to augment the user in many diverse fields such as memory, security, military, medical,financial, navigation and even entertainment.

1.7.1 Research Goals

The aim of this project is to develop a conceptual framework and component architecture for thedevelopment of a personal information system that based on a variety of information devices in aBAN to record the context of the user, to structure and store it in a centralized location, and finallyto make it accessible at the application and service level. Although there are many applications andservices that might operate on subsets of contextual data we will primarily focus on the personalcontextual search engine. The search engine can be queried by the user to produce meaningfulsubsets of contextual data and plays an essential part in fulfilling the mobile memex vision.

This thesis seeks to indentify what a modernized implementation of the 1945 Memex could look likein the upcoming ambient environment of 2010, in regard to a number of characteristics:

1. Composition: What hard- and software is the mobile memex composed of?

2. Function: How will the mobile function, and what is the role of content and context in thesystem? We will focus on solving the practical problems regarding core functionality:

(a) Assemble: How can a distributed sensor platform be constructed that meets the usersfunctional requirements?

(b) Acquire: How can raw sensor data be obtained from sensors in personal informationdevices, and how can this raw sensor data be processed into either content or context.

(c) Aggregate: How personal information devices can be efficiently and securely networkedin a BAN that enables the sharing of information and resources without compromisingthe user’s privacy.

(d) Archive: How context documents and content files can be effectively structured andstored.

(e) Access: How this content and context can be easily retrieved and operated upon byapplications and services in the mobile memex.

(f) Annotate: How can the user annotate retrieved content and context?

(g) Associate: How can the user associate context and content to form meaningful sets ofinformation?

(h) Administrate: How can the system and its applications be monitored, configured andadministrated by the user?

3. Augmentation: How can the system be used to support applications that augment the user,and what kind of augmentation applications are possible?

4. Implication: What kind of social and ethical implications does a system such as the mobilememex have in terms of privacy and security?

20

1.7.2 Overview of the thesis

1. Chapter one explained the vision of the mobile memex and the current trends and developmentsin information technology that might lead to it’s realization by the year 2010.

2. Chapter two will identify, define and classify the key concepts in the mobile memex in order toconstruct a conceptual framework of the system. We identify possible applications and uses ofthe mobile memex and examine how these applications can operate on the aggregated contentand context in order to augment the user in its task.

3. Chapter three will deal with the requirements of the system. User requirements and Sys-tem requirements are used to create a component architecture that meets these requirements.Chapter 3 follows the structure recommended by the IEEE 830-1998 software requirementsstandard (SRS).

4. Chapter four will implement the minimal set of requirements as defined in the base-line ar-chitecture to implement a working proof of concept applications that demonstrates the func-tionality and flexibility of the system. On top of the base-line implementation will be a userapplication in the form of a personal contextual search engine.

5. Chapter five presents the results and conclusions of this thesis, Possible future expansions tothe mobile memex are also discussed.

21

22

Chapter 2

A Conceptual Framework

This chapter will present the mobile memex framework, as shown in Figure 2-1. The framework con-sists of five distinct layers, each with their own functionality and capable of obtaining and processingsensor data into two different repositories containing either content or context. The repositories arecentrally available to applications and services that operate on the information they contain. Thisframework will serve as a roadmap for this chapter: Each section will discuss one of the five layersin detail.

Information StoreInformation Operations

Sensors Actuators Resources

NetworkPlatform Middleware

Acquisition

Interpretation

Aggregation

Application Services Application Interfaces

Normalization

Abstraction

Extraction

Annotation

Association

AdministrationRemote

ApplicationInterface

Content Repository

Interface

Context History Interface

Query Interface

Context History

Content Repository

Physical Layer

Communication Layer

Information Layer

Application Layer

Memory Medication

Entertainment Navigation

Communication

CoachingAugmentation Layer

Figure 2-1: The Mobile Memex Conceptual Framework

Storage capacity will be abundant and consequentially the challenge is not to store informationanymore, but to retrieve it. This is were context will play an increasingly important role. Asstated in chapter one; The aim of the mobile memex is to store content and context obtained fromdistributed sources and to use this context information as an index of retrieval-cues that enable theretrieval of related context and content. The mobile memex framework is designed to support thisgoal.

23

2.1 The Physical Layer

The physical layer contains all hardware components required by the mobile memex. The maincomponents are sensors, actuators and system resources. These components will be embedded in thepersonal information devices that compose the user’s body area network. The following subsectionswill define the concept ”personal information device”, discuss the functional overlap between thesedevices and propose three classes of hypothetical devices best suited for the mobile memex. Theconcepts sensor, actuator and system resource are also defined and classified. Finally we will explaintheir purpose and function in the mobile memex.

2.1.1 Personal Information Devices

A personal information device is a device small and light enough to be carried around by the userand be used while on the move. It provides a certain functionality required by the user, be itthe capability to take digital pictures, listen to digital music, sending and receiving email, or anyother functional need that can be provided by these devices. In order to provide it’s functionality apersonal information device will contain sensors, actuators and system resources (such as processing,storage, memory and battery capacity). The system resources will determine the role of the devicein the mobile memex. As it stands, all personal information devices currently on the market eachhave their own interface often consisting of key based input and display based output, and their ownresources such an individual power source.

Functional Overlap

For a user to fulfill his or her information requirements, he or she might have to carry around anumber of these information devices. It is not uncommon to find an individual with a notebook,personal digital assistant, mobile phone and digital camera. Most of these device cannot communi-cate with each other in order to share sensor, actuators or system resources. Consequently there iscurrently a high degree of functional overlap between personal information devices as is illustradedby Figure 2-2.

occasionalYESYESoccasionaloccasionaloccasionaloccasionalYESCamera

YESNONOoccasionalYESNONOoccasionalFM radio

occasionalYESYESoccasionalYESoccasionalYESYESNotebook

occasionalYESYESoccasionaloccasionalYESoccasionalYESPDA

YESYESYESYESYESoccasionalYESYESCellphone

YESNOYESNONONONOYESPager

NOoccasionaloccasionalNONONOoccasionalYESWatch

XcvrCPUMemMicSpkrPenKeypadDisplay

Figure 2-2: Functional overlap between personal information devices

For the user to carry around many different personal information devices means carrying around amultitude of batteries, displays, keypads, displays and much more. This will increase the burdenand decrease the overall usefulness of the configuration. Every device is an island with a differentinterface which the user has to learn before it can be used. Also each device will have its ownand storage medium that is often incompatible with other devices (for example each device has it’sown type memorycard), therefore the user has to manually manage and transport it’s data betweendevices which can prove to be a time consuming and cubersome task.

There are two solutions to solving the problem of functional overlap. The first is functional inte-gration of devices, the second networking the devices a user carries around to allow the sharing ofsensors, actuators and certain system resources. A combination of both solutions will result in anoptimal configuration. The mobile memex is be based on this configuration as a starting point.

24

Functional Integration

While Moore’s law is simply an observation that states that transistor density will double roughlyevery 18 month and so far has held true, deratives for magnetic storage density and energy densityfor power sources also seem to hold true:

1. Increase in storage density for magnetic, flash and optical storage media. Magnetic density iscurrently doubling every 12 month. The projected areal density for Magnetic storage in 2010will be around 1 Terabit per square inch. Storage capacity will be cheap and abundant. Thechallenge is no longer whether you can store everything, it is whether you’ll be able to find itlater when you need it.

2. Increase in power-source capacity. Power sources are an important factor in the usability ofwearable devices because in the mobile memex most devices in the body area network will beactive at all times. Batteries are the most common power sources and are relatively heavycomponents that determine the portability and durability of a device. Development in thisarea is not as fast paced compared to transistor and magnetic densities and are only doublingroughly every 10 years with a predicted density of 600Whr/kg for fuel cells in 2010.

These developments together with an increase in transistor density will mean higher integratedcircuit complexity and processing speed with an equal or smaller surface area and reduced powerconsumption, abundant storage capablities and sufficient power for prolonged portable operation.If Moore and it’s deratives hold true it will be possible to produce portable information devices in2010 that have a very high degree of functional integration.

Functional integration can lead to three main classes of portable information devices by the year2010:

1. Small devices useful enough for the user to carry around at all times. These devices can beworn and will not obstruct the user at all. Also they offer a part of the core functionality. Awristwatch is a good example. But one can also think of implants such as a pacemaker, or ahearing-aid.

2. Medium devices can be worn by the user in a pocket. Although less portable that smalldevices they offer some essential functionality so the user will usually carry these devicesaround. Examples of medium devices are a mobile phone, personal digital assistant.

3. Large devices. These devices are cumbersome to carry around but offer a high system resourcecapacities in terms of processing, memory and storage.

Figure 2-3 shows some characteristics of the different device classes in the mobile memex.

2.1.2 Sensors, Actuators and Resources

Personal information devices such as those used in the mobile memex contain sensors, actuators andsystem resources. Figure 2-4 lists sensors, actuators and system resources commonly found in thesedevices.

Sensors

The word ”sensor” is derived from the word ”sentire” meaning ”to perceive”. A dictionary definitionof ’sensor’ is:

”A device that detects a change in a physical stimulus and turns it into a signal whichcan be measured or recorded.”

In general a sensor is a component that takes an external physical phenomena as input and producescorresponding system data representing this physical phenomena as output. The mobile memexassumes a number of common sensors will be available to the system most of the time:

25

Low (Bag, suitcase)

Good (Pocket, belt)

Excellent (Wrist, neck, finger)

Portablity & Location

Laptop, Tablet, PCCommunicator, PDA, Phone, Camera.

Watch, Digital Jewelry, Chipcards, Implants.

Typical devices

Device example

~1 Kilogram~150 Grams~25 GramsTypical Weight

Wifi, Ethernet, ADSL.Bluetooth, UWB, USB, GSM, WCDMA.

Magnetic inductionBody conductivity

Typical network interface

Speaker, XGA+ displayVibration unit, Speaker, VGA Display.

Status LED, Beeper, QCIF display

Typical actuators

Network, Keyboard, Touchpanel, Mouse

Phone, Schedule, Camera, Audio, Video, Messaging, GPS, Keypad. Stylus.

Time, Date, Compass Temperature,Pressure, Altitude, Salinity, Pulse, Buttons, Switch.

Typical sensors

~1~2~4Typical #devices

~1 Day~1 Week~1 YearPowersource capacity

High (1 – 10 TB)Med (1-10 GB)Low (1-512 MB)Storage capacity

High (5 – 20 Ghz)Med (1-5 Ghz)Low (1-100 Mhz)Processing capacity

Occasional (weekly)Regular (daily)High (hourly)Freq. of use

LargeMediumSmallDevice class

Figure 2-3: Some characteristics of the 3 distinct device classes

1. Time sensor: Most clocks are not real sensors and produce output based upon a single mea-surement (a user setting the time) and calculate the sequental time values after this which canresult in low precicion and reliability. Some sensors can accurately measure time and these arediscussed in section 2.3.4.

2. Location sensor: A GPS sensor able to provide the system with the users current longitudeand latitude coordinates and the altitude above sea level.

3. Network sensor: A network interface is actually a combination of an actuator and a sensor, alsoknown as a transmitter and receiver, or modulator and demodulator. The network interface isimportant because it can sense the interaction between devices in a BAN,PAN,LAN or WANand consequently identify other computational entities.

4. Sound sensor: A microphone able to record aural information.

5. Image sensor: A CMOS or CCD camera able to capture visual information

6. User input sensor: A keyboard, keypad or touchpad able to capture user input.

Figure 2-5 shows a classification of sensor characteristics according to different criteria.

Actuators

Actuators are components that take system data as input and generate physical phenomena asoutput. They are the complement of sensors. Some examples of actuators are the speaker, displayand vibration units in a mobile phone.

System Resources

System resources define the capacity of the system in terms of storage, battery, memory and pro-cessing capacity. The device capacity will in turn define its function in the mobile memex. Devices

26

SpeakerStatus LEDVibration UnitFlash unitDisplayMotorRadiographic TransmitterIR Transmitter

Actuator Instances

CPU Memory Storage Battery Bandwidth

Microphone Keypad / KeyboardAltimeterBarometerThermometerAccelerometerCompassInclinometer GyroscopeGPS Radio & TV receiverPiezoelectric sensorTime & Date sensorCMOS / CCD sensorGeomagnetic sensorConductivity sensorElectro Cardiac sensorInduction sensorInfrared sensorCO gas sensor

Resource InstancesSensor Instances

Figure 2-4: Sensors, Actuators and System resources in personal information devices

that meet the storage and processing requirements might be suitable to store all sensor data andperform processor intensive operations upon it such as datamining.

2.2 The Communication Layer

The communication layer contains all software components needed to network personal informationdevices in an ad-hoc body area network. The main software components are: 1) The system platformand it’s application programming interfaces, 2) The network technology, topology, protocols andinterfaces best suited for the mobile memex, and 3) The Middleware such as the message format anddocument structures that can facilitate efficient communication and information exchange betweenthe different devices and applications of the mobile memex.

2.2.1 Platform

A common platform is neccesary to standardize device functionality in the mobile memex. Infor-mation device are very heterogenous in nature with system resources at both end of the spectrum.A platform idependent language such as Sun’s JAVA is a good solution, especially since JAVA iscurrently widely supported on many mobile device ranging from PDAs, Cellphones to smartcards.Individual devices can have their own Operating System but have to provide JAVA support byincluding a Java Virtual Engine (JVM), on small-class devices this can also be a slimmed downversion of the JVM known as a KVM. Also the manufacturer of the personal information device hasto provide application programming interfaces (APIs) to sensors, actuators and resources in JAVA.This way developers can realize mobile memex support in a wide range of devices.

Network

The networking of information devices is another way of eliminating functional overlap and can leadto increased functionality as resources from individual devices can now be combined to provide newservices and applications not possible before.

27

CostSizeWeightOperating LifePower consumption

LocationTimeInteractionIdentificationAttribute

AccuracyResolutionRangeSaturationPrecisionGranularitySensitivityStabilityResponseOverloadHysteresisExitationSelectivity

AcousticBiologicalChemicalElectricMagneticOpticalMechanicalRadiationThermal

Economic characteristics

Domaincharacteristics

Qualitycharacteristics

Physical characteristics

Figure 2-5: Overview of sensor characteristics

Types and Technologies

Because the mobile memex is composed of personal information devices situated in, on or near theuser’s body the mobile memex wiil function within a body area network. As figure 2-6 shows, thereare many technologies that support on- or near body communication. Some of these technologieswill be better suited to the mobile memex than others based on a number of factors such as powerconsumption, cost, size, weight, bandwidth and others. If we take these factors into account thereare two primary candidates; a wired BAN solution that is based on using the natural conductivity ofthe human body to connect devices and a wireless BAN solution based on magnetic induction. Un-fortunately both of these technologies are currently in it’s research phase and are not commerciallyavailable, by 2010 the situation might be different however). Another suitable wireless technologythat can be put to use in a relatively short time is ultra-wideband communication also known asUWB.Personal information devices will usually contain more than one network interface. Some of thesedevice will be able to interact with wide area networks (WAN) and some with personal area networks(PAN). In the PAN, communication is heavily dependent on wireless network technologies such asWiFi, Bluetooth, Zigbee and UWB.

figure 2-6 shows a classification of network types and their corresponding technologies.

802.15BluetoothZigBeeIrDAWirelessUSBUltraWideBandRFID (1-way)Magnetic Induction

802.11a/b/g(WiFi)

802.16a(WiMAX)

GSMGPRS(W)CDMAUMTS

Wireless

USBFirewireBodybus

802.3(ethernet)

CABLEDSLISDNATM

Fixed

PAN / BANLANMANWAN

Figure 2-6: Network types and their corresponding technologies

28

A Peer-to-Peer topology

Basically the mobile memex is an ad-hoc sensor network situated on the body of the user. The BodyArea Network will not be a static network, instead information devices will continually be removedand added depending on the users functional requirements of that moment. This means that nodeswill come online and go offline at regular intervals. The mobile memex will have to continue tofunction properly under these circumstances. This calls for peer-to-peer (P2P) style networking ofinformation devices. Whenever a devices comes online, it will join the mobile memex peergroup andtry to discover other nodes in this peergroup. The peergroup itself will offer a number of servicessuch as discovery, resource sharing, security and more. Chapter 3, section 12 will deal with thetechnical aspects of P2P networking in the mobile memex in more detail.

2.2.2 Middleware

The mobile memex is a very generic contextual information system that has to support a wide rangeof hetergenous devices and applications. In order for the many diverse devices and applications tocommunicate with each other we need middleware. Middleware is software that mediates betweeentwo otherwise separate applications. It manages the interaction between disparate applicationsacross the heterogeneous computing platforms. Ultimately, middleware is all about integration.

Standards

Furthermore the information system should function over long durations, possibly during an entirelife time. In order to ensure compatablitity of information it is essential to user widely acceptedcommon standards and formats as middleware components.

1. All encoding of textual information should be done in the extendible markup language (XML)using the Unicode UTF-8 Characterset.

2. All encoding of visual information such as bitmap images and vector graphics should be doneusing the portable network graphics (PNG) standard in case of bitmaps, and using the scaleablevector image (SVG) standard for vectors.

3. All encoding of streaming material such as video and audiofiles should be done in Mpeg4(ISO-mp4) and OGG-vorbis respectively.

4. All encoding of executable material should be done in the JAVA programming language.

Network communication should use the IPv6 addressing standard, IP/TCP transmission protocoland the JXTA P2P networking protocol. Chapter 3 will give a more detailed technical descriptionof the above mentioned standards.

2.3 The Information Layer

The information layer deals with the acquisition of distributed sensor data by information devicesin the BAN, including the interpretation, encapsulation and aggregation of this data into a contexthistory and a content repository. Figure 2-7 shows the processing path of turning sensor-data intoa normalized context history and a normalized content repository.

2.3.1 Acquisition

The acquistion process involves information devices that have a publishing profile assigned to it.These devices will at reguralar intervals poll the sensors available in that specific device using theAPIs provided by the manufacturer, and according to the device preference profile. This results inraw sensor-data. Sensor-data has to be processed in order to be useful.

29

RAW SENSOR DATA

RAW CONTEXT DATA

RAW CONTENT DATA

RAW CONTEXT INFORMATION PARTIAL CONTEXT HISTORY RAW CONTEXT HISTORY

NORMALIZED CONTEXT HISTORY

PARTIAL CONTENT REPOSITORY RAW CONTENT REPOSITORY

NORMALIZED CONTENT REPOSITORY

RAW CONTENT FILE

QUERY RESULTS

Publisher Subscriber Storage

interpretation

interpretation

acquisition

encapsulation

encapsulation

aggregation

aggregation update

update

normalization

normalization

retrrieve

retrieve

Extraction Abstraction

abstraction

abstraction

Extraction

Extraction

Extraction

Extraction

Figure 2-7: Data processed into context and content in the M3X framework

Device Roles

Because there are different devices with different sensors, actuators and resources not every deviceis suited for every job. In the mobile memex we can identify 4 device roles that are defined in M3Xprofiles:

1. Publisher device: all wearable and portable devices containing sensors and a network interface.

2. Subscriber device : all wearable and portable devices with adequate memory and processingresources, allowing for caching of context data and the operation of M3X-enabled applicationsthat operate on recent to semi-realtime contextual data.

3. Storage device: all devices with abundant memory and processing resources, allowing for thestorage of the entire context history and the media created in that context. The storage deviceis also home to M3X-enabled applications that operate on the entire context history such asthe personal contextual search engine (PCSE). Most of these applications will be reactive.Datamining and pattern matching can also be done by resource intensive applications.

4. Access device: Some applications and services can be made remotely available over the internetif the user chooses to. These applications are known as ”remote applications” and can beaccessed by any device with an internet connection and a display. Other applications haveto be accessed locally on the host device and therefore known as ”local applications”. Asubscribing device will be the primary access device for applications that operate on recentand semi-realtime data. Local applications are usally pro-active applications that operate on(semi) real-time data, while remote applications are often re-active applications that operateon archived data.

Device Profiles

Information such as the role of the device, the sensor polling interval and many others are definedin profiles. Profiles play a central role in the mobile memex as they describe the nature of devices,sensors, actuators, system resources, applications and preferences. There are two master profiles:1) the system specification profile, and 2) the system preference profile. The master profiles areconstructed of sub-profiles. The system specification profile is the sum of all device specification

30

profiles. The system specification profile describes in detail every device, sensor, actuator and systemresource available in the system. Based on this profile the mobile memex can make an inventoryof applications that can operate on the system. Based on the system specification profile and theapplication inventory the user can create a system preference profile.

The following sequence illustrates the processing of profiles from the device specification profile tothe device preference profile:

1. Devices send their device specification profile to system (a device specification profile is acompilation of sensor specification profiles, actuator specification profiles and system resourceprofiles, provided by the manufacturer).

2. The system compiles the device specification profiles into the sytem specification profile.

3. The system specification profiles is used as a basis for the generic system preference profile inwich the user defines general system settings such as the level of security and privacy.

4. The system specifiation profile and the generic system preference profile define the functionalsystem specification profile.

5. The functional specification profile is matched against the list of all available M3X applica-tion specification profiles to deduct and compile a list of potential candidates that meet therequirements imposed by the system.

6. Based on the list of candidate applications the user will select the ones he or she deemsuseful and installs the applications. For every installed applications the user will create theapplication preference profile.

7. The system will now use the system specification profile in combination with the genericuser preference profile and the application preference profiles to deduct the system preferenceprofile.

8. The system preference profiles is split up into device preference profiles.

9. The devices will use the device preference profile to deduct their role in the system and theindividual sensor, actuator and system resource preference profiles.

Figure 2-8 shows the different types of profiles in the mobile memex and how they relate to eachother. Figure 2-9 shows the system specification profiles and it’s references to individual deviceprofiles which in turn refer to sensor, acturator and system resource profiles.

2.3.2 Interpretation

In general raw sensordata produced by the sensors in the physical layer has to be interpretated andprocessed into either context or content depending on the nature en encoding of the data. Sensordata is turned into contextual information by encapsulating the measured data in meta-data addingdescriptive information.

The primary function of the mobile memex is to capture the user’s context, including the contentobtained in this context, and make it available to applications in a centralized way. Thus contextand content can be considered the key concepts in the mobile memex. Sensor data can be interpretedto be either context or content depending on the type and encoding of the data.

31

System specification profile

Device specification profile

Device specification profile

Device specification profile

Sensor profiles

Actuator profiles

Resource profiles

Sensor profilesSensor

specificationprofiles

Actuator profilesActuator

specificatonprofiles

Resource profilesResource

specificationprofiles

System preference profile

Device preference profile

Device preference profile

Device preference profile

Sensor profiles

Actuator profiles

Resource profiles

Sensor profilesSensor

preferenceprofiles

Actuator profilesActuator

preferenceprofiles

Resource profilesResource Preference

profiles

1

2

3 4

5

6

UserManufacturer

Figure 2-8: Profiles in the mobile memex

2.3.3 Context

A deeper understanding of context will allow it to be structured in a more effectively, and increasethe useability of the system. Context has been defined many times depending on its application orperspective. In this thesis we will use the following definition of context:

”Any information that can be used to characterize the situation of an entity [2]”

This definition introduces two important new concepts, entity and situation. An entity can be aperson, place, object or phenomena. Entities are either physical (i.e. egg salad) or logical (i.e.email). Context is always associated with an entity. In the mobile memex there are three mainentities; (1)the user, (2) the environment and (3) the system. The situation of an entity refers toidentifying and describing the who, what, when, where and why context relevant to the entity. Eachof these contexts is represented in one of five context dimensions:

1. Existential context dimensions: Deals with the identification of physical and logical entities,where entities can be objects, people, programs, components, attributes, groups or naturalphenomena (tsunami, el-nino). Basically this dimension identifies the ”who” in a situation.

2. Functional context: Deals with the identification of events, states, activities and actions.

(a) Event: Events occur when the environment changes the state of the subject entity. Anevent is generated when an entity in the environment reaches a certain condition thatinfluences the subject entity. An event has no time span, it is either true or false. Anevent triggers a state change of an entity. Example: The sun has disappeared behind thehorizon and it is now dark.

(b) State: Every entity has a number of possible states that can be activated by events, Anentity can be in one or more active states simultaneously. The state of an entity definesa timespan during which a set of one or more meaningful or logical activities might takeplace. Example: I am blinded by the darkness.

(c) Activity: An activity is initiated by the entity and defines a sequence of actions requiredto complete the activity. Example: go turn on the light.

(d) Action: An action is an atomic unit of activity. The outcome of an action is often anevent for entities in the environment. Example: Switch the light switch on.

32

� � � � � � ��� � � � � ��� � � � � � � � � � � � � � � � � �� � � � � �

� � � � � � � � � ��� � � � ��� � � � � � � � � � � � � �� � � ��� �

� � � � � � � ��� � � � � � � � � � � � � � ��� �� � � � � � � ��� � � � � � � � � � � � � ��� �

� � � � �!� �� � � � � � �� " � � � # � � � "$" � � � # � � � " � � � ��� � � � ��� ��� � � � �

� � � ��� � ��� % " � � � # � � � " � � � � ��� �� � " � � � # � � � " �� � & � ' � � �

� � & � ' � � � � � � � � � � ��� � � � ( � � � � & � ' � � �� � & � ' � � � � � � � � � � ��� � � � ) � � � � & � ' � � �� � & � ' � � � � � � � � � � ��� � � � * � � � � & � ' � � �

� � � & � ' � � �� � � � � � � �$�

� � � & � ' �$ � & � ' � � � � � ��� � � � ��� � � � ) � � �� ��� � � � ' � � � � � �,+-� � � �,� � ��� � � � ' � � � � � �� � � " � �$ � # � � ��� ' � ��� � �,� � � � " ���� ��� � � �.+-* � � / 01� � ��� � � �� � " � ' � � � �

� � � " � � � 23242!5 ' � � � ��5 ' � �!� " � � � ' � � +-* � � 0�� " � ' 5 � � �$�� � � " � ' � � � �� � � � � � � � �

� � � � � � � � � � � ��� � � � ��� � � � ) � � � � � � � � � � ��� � � � � � � � � � � ��� � � � ��� � � � ) � � � � � � � � � � ��� � � � � � � � � � � ��� � � � ��� � � � ) � � � � � � � � � � ��

� � � ��� � � � � �� � ' � � � � � � � �

� � ' � � � � � � � � � � ��� ��� � � � � � � ) � � � � ' � � � � � � � �� � ' � � � � � � � � � � ��� ��� � � � � � � ) � � � � ' � � � � � � � �� � ' � � � � � � � � � � ��� ��� � � � � � � ) � � � � ' � � � � � � � �

� � � ' � � � � � � � �� � � � � � � ' � � �

� � � � � � � ' � � � � � ��� � � � ��� � � � ) � � � � � � � � � ' � � �� � � � � � � ' � � � � � ��� � � � ��� � � � ) � � � � � � � � � ' � � �� � � � � � � ' � � � � � ��� � � � ��� � � � ) � � � � � � � � � ' � � �

� � � � � � � � ' � � �� � � & � ' � �

� � ' � � � � � � � ' � � � � � � � � � � � � � � � � � � � � ) � � �� ' � � � � ��& � � � � � � � ' � � � � �� ��� � � � ' � � � � � �,67� � 8!� � ��� � � � ' � � � � � �� � � ��� �� � � � �$� � � � ��� �� � � " � �,6-9 �,� � � � " � �� � " � ' � � � �

� � � " � � � 23242!5 8 � � 8 5 ' � �:� � � � � � � " � ' � 8 � � � � ��5 � � �$�� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � # �,� � � � � � � � � ��� � � � � �,)$� � � � � # � �� � ' � � � � ' � � � � � � � ' � �

� � � ' � � � � � � �

� � � � � � � ' �,� � � � � � ' � � � � � � � ��� � ��� � � � ) � � �� ' � � � � �.; � � ' � � � � ��� � ' � � � � �� ��� � � � ' � � � � � �.< � � � � � � ��� � � � ' � � � � � �� � � " � ��=3� ' � � �.; � ) )$� � � � " � �� � � ' � � � � ' � � � � ��>4?-@A� � � � ' � � � � ' � � � � �� � � & � � � � � �,B75 � �$� � � � & � � � � � �� ' � " � ' � � �$� � � � � � � � � @!� C � ��(�� �$� � ' � " � ' � � � �� � " � ' � � � �

� � � " � � � 23242!5 � � � � � 5 ' � �!� " � � � ' � � � � � =7� ' � � � � ; � ) ) � " � ' 5 � � �$�� � � " � ' � � � �

� � � � � � � � ' � �

� � � & � ' �$ � & � ' � � � � � � � � � � � � � � � ) � � ������ � � � ' � � � � � �,+3� � � �,� � ��� � � � ' � � � � � �� � � " � �$ � # � � � � ' � ��� � �,� � � � " � ������ � � �,+-* � � / 01� � ��� � � �� � " � ' � � � �

� � � " � � � 2-2-2!5 ' � � � ��5 ' � �!� " � � � ' � � +3* � � 0 � " � ' 5 � � �$�� � � " � ' � � � �� � � � � � � � �

� � � � � � � � � � � � � � � � � � � � � ) � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � ) � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � ) � � � � � � � � � � �

� � � � � � � � � �� � ' � � � � � � � �

� � ' � � � � � � � �� � � ��� � � � ��� � � � ) � � � � ' � � � � � � � �� � ' � � � � � � � �� � � ��� � � � ��� � � � ) � � � � ' � � � � � � � �� � ' � � � � � � � �� � � ��� � � � ��� � � � ) � � � � ' � � � � � � � �

� � � ' � � � � � � � ���� � � � � � ' � � �

� � � � � � � ' � � � � � � � � � � � � � � � ) � � � � � � � � � ' � � �� � � � � � � ' � � � � � � � � � � � � � � � ) � � � � � � � � � ' � � �� � � � � � � ' � � � � � � � � � � � � � � � ) � � � � � � � � � ' � � �

� � � � � � � � ' � � �� � � & � ' � �

� � � & � ' �$ � & � ' � � � � � ��� � � � ��� � � � ( � � �� ��� � � � ' � � � ��� �.+3� � � �,� � �!� � � � ' � � � � � �� � � " � �.�-� # � � � � +3� ��� � �,� � � � " � �� ��� � � �.+-* � � / 01� � ��� � � �� � " � ' � � � �

� � � " � � � 2-2-2!5 ' � � � ��5 ' � �!� " � � � ' � � +-* � � 0�� " � ' 5 � � �$�� � � " � ' � � � �� � � � � � � � �

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � D � � � � � � � � � � �

� � � � � � � � � �� � ' � � � � � � � �

� � ' � � � � � � � � � � ��� � � � � � � � � � � � � � � ' � � � � � � � �� � ' � � � � � � � � � � ��� � � � � � � � � � � � � � � ' � � � � � � � �� � ' � � � � � � � � � � ��� � � � � � � � � � � � � � � ' � � � � � � � �

� � � ' � � � � � � � �� � � � � � � ' � � �

� � � � � � � ' � � � � � ��� � � � ��� � � � � � � � � � � � � � ' � � ��� � � � � � � ' � � � � � ��� � � � ��� � � � E � � � � � � � � � ' � � ��� � � � � � � ' � � � � � ��� � � � ��� � � � ' � � � � � � � � � ' � � ��

� � � � � � � � ' � � �� � � & � ' � �

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� ' � � � � � � � ' � � � � � � ' � � � � �� � � ��� � # � � � � � ',� (,E � � � � �$8 � � " � ,� � � � ������ � � " � �,8 � � " � ,� � � � " � �� ��� � � � ' � � � � � ��� � � � � E �,� � ��� � � � ' � � � ��� �� � � & � � � � � ��>-5 � �!� � � � & � � � � � �� � " � ' � � � �

� � � " � � � 23242!5 � � � � � E ��5 ' � �!� " � � � ' � � � � " " � � � � � " � ' � 8 � � " � � #�� (�E � ��� 5 � � �$�� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � " � � � � � � ���,� � � � � � � � � �!� � �F� � �$� � � � � " � � � � �� � ' � � � � ' � � � � � � � ' � �

� � � � � � � � �

� � ' � � � � � � � ' � � � � � � � � � � ��� � � � ��� � � � ) � � �� ' � � � � �$& � � � � � � � ' � � � � ������ � � � ' � � � � � �,6 � � 8!� � ��� � � � ' � � � ��� ���� � ��� �� � � � ��� � � � ��� �� � � " � �,639 �,� � � � " � �� � " � ' � � � �

� � � " � � � 232-2!5 8 � � 8 5 ' � �:� � � � � � � " � ' � 8 � � � � ��5 � � �$�� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � # �,� � � � � � � � � ��� � � � � �,)$� � � � � # � �� � ' � � � � ' � � � � � � � ' � �

� � � ' � � � � � � �

� � ' � � � � � ��� ' � � � � ��� � � � � � � � � � � � � � � � � � � �� ' � � � � ��G-� � � ��� � � ' � � � � �� ��� � � � ' � � � ��� �.6 � � 8!� � ��� � � � ' � � � � � �� � � ��� �.9 � � � �$� � � � ��� �� � � " � �.639 �,� � � � " � �� � " � ' � � � �

� � � "�� � � 2-2-2!5 8 � � 8 5 ' � �:� � � � � � � " � ' � 8 � � � � ��5 � � ���� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � # �,� � � � � � � � � ��� � � � � �,)$� � � � � # � �� � ' � � � � ' � � � � � � � ' � �

� � � ' � � � � � � �

� � � � ��� � ' �,� � � � � � ' � � � � � ��� � � � ��� ��� � ) � � �� ' � � � � �F;�� � ' � � � � ��� � ' � � � � ������ � � � ' � � � � � �.< � � � � � � ��� � � � ' � � � � � �� � � " � ��=7� ' � � �.;�� ) )$� � � � " � �� � � ' � � � � ' � � � � ��>-?4@A� � � � ' � � � � ' � � � � ���� � & � � � � � �,B35 � �$� � � � & � � � � � �� ' � " � ' � � �$� � � � � � � � � @�� C � ��( � �$� � ' � " � ' � � � �� � " � ' � � � �

� � � " � � � 232-2!5 � � � � � 5 ' � �!� " � � � ' � � � � � =7� ' � � � � ;�� ) ) � " � ' 5 � � �$�� � � " � ' � � � �

� � � � � � � � ' � �

� � � � � � � ' �,� � � � � � ' � � � � � ��� ��� � � � � � � � � � �� ' � � � � �.; � � ' � � � � � � � ' � � � � �� ��� � � � ' � � � ��� �F< � � � � � � ��� � � � ' � � � � � �� � � " � �$=7� ' � � �.; � ) )$� � � � " � �� � � ' � � � � ' � � � � ��>4?-@A� � � � ' � � � � ' � � � � �� � � & � � � � � �,B75 � �$� � � � & � � � � � �� ' � " � ' � � �$� � � � � � � � � @�� C � �$( � ��� � ' � " � ' � � � �� � " � ' � � � �

� � � "�� � � 2-2-2!5 � � � � � 5 ' � �!� " � � � ' � � � � � =7� ' ��� � � ;�� ) ) � " � ' 5 � � �$�� � � " � ' � � � �

� � � � � � � � ' � �

� � � � � � � � � � � � � � � � � ��� � � � ��� � � � � � � �� ' � � � � � � � ' � � � � � � ' � � � � ���� � ��� � # � � � � � ',� (.E � � � � �$8 � � " � ,� � � � ��� �� � � " � �,8 � � " � ,� � � � " � ������ � � � ' � � � � � ��� � � � � E �,� � ��� � � � ' � � � � � ���� � & � � � � � ��>45 ���!� � � � & � � � � � �� � " � ' � � � �

� � � " � � � 232-2!5 � � � � � E ��5 ' � �!� " � � � ' � � � � " " � � � � � " � ' � 8 � � " � � # � ( E � ��� 5 � � �$�� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � " � � � � � � ���$� � � � � � � � � �!� � �F� � �$� � � � � " � � � � �� � ' � � � � ' � � � � � � � ' � �

� � � � � � � � �

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� ' � � � � ��H � ' � � � � � � ' � � � � �� � � ��� �$# � � � � � ',� (.E � � � � �$8 � � " � ,� � � � ��� �� � � " � �.8 � � " � $� � � � " � �� ��� � � � ' � � � ��� �$H � � � � E �,� � �!��� � � ' � � � � � �� � � & � � � � � ��>-5 � �!� � � � & � � � � � �� � " � ' � � � �

� � � "�� � � 2-2-2!5 � � � � � E ��5 ' � �!� " � � � ' � � � � " " � � � � � " � ' � 8 � � " � � #�� (�E ��� � 5 � � �$�� � � " � ' � � � �� ' � � � � ' � � � � � � � ' � �

� � � � " � � � � � � ���,� � � � � � � � � �!� � �1� � �$� � � � � " � � � � �� � ' � � � � ' � � � � � � � ' � �

� � � � � � � � �

I.J:K3L M�NOIQP:M!R4S!TFU V4W X Y MI.J:K3L M�NOIQP4M!R:S!T.U V4W X Y MI.J:K3L M�NOIQP:M!R4S!TFU V4W X Y MI.J:K3L M�NOIQP4M!R:S!T.U V4W X Y M

Z M4[�X R-M\IQP4M!R:S!TFU V4W X Y MZ M4[$X R-M]I^P:M!R:S!TFU V4W X Y MZ M4[�X R-M\IQP4M!R:S!TFU V4W X Y MZ M4[$X R-M]I^P:M!R:S!TFU V4W X Y M

II II^MM MM.__ __:KK KK:VV VV$UU UU.II II�PP PP:MM MM�RR RR�SS SS�TT TT`UU UU VV VV:WW WW XX XX YY YY MM MM

aa aa RR RR-LL LL bb bb!cc cc:LL LL VV VV$UU UU,II IIdPP PP!MM MM�RR RR�SS SS�TT TT1UU UU VV VV:WW WW XX XX YY YY MM MM

ee ee MM MM�KK KK:VV VV.bb bb!UU UU RR RR4MM MMfII IIdPP PP!MM MM�RR RR�SS SS�TT TT`UU UU VV VV:WW WW XX XX YY YY MM MM

Figure 2-9: Example XML profiles and relations

Figure 2-10 and figure 2-11 show the relations between events, states, activities and actions.Basically this dimension identifies the ”what” in a situation.

environment entity

event action

state

activity

action

Figure 2-10: An entity influences it’s environment and vice-versa

3. Temporal context dimension: Deals with the identification of date and time such as year,month, day, hour, minutes, seconds, seasons etc. Basically this dimension describes the ”when”in a situation.

4. Spatial context dimension: Deals with the identification of location, direction, orientation,elevation and proximity. Basically this dimension identifies the ”where” in a situation.

5. Relational context dimension: Deals with the identification of reasons, relations and associa-tions between instances in the context dimensions based on user interpretation. The relational

33

Bob’s Birthday

Eat birthday cake

Stick fork in cake Stick fork in mouth

Open presents

Event:5 May 00:00Start Bob’s Birthday

State:5 May 00:00 – 23:59 Bob’s Birthday

Activity:5 May 16:00 – 16:30Eat birthday cake

Action:Stick fork in cake

Event:5 May 23:59End Bob’s Birthday

20030504120000 20030505233000

Figure 2-11: Events, states, activities and actions

Open, CloseEnter, LeavePull, PushTake, DropReceive, SendStart, Stop.Give, Take…

SingDanceExplainDrivePlayStudyCookInnovateOperateConstructEntertainRelaxLunch…

Alive, DeadAsleep, AwakeSick, HealthyHappy, DepressedYoung, OldRich, PoorSlow, FastGrey, WhiteBeautiful, UglyDirty, Clean.Empty, Full.Drunk, SoberModern, AntiqueHigh, Low…

BobArtComputer WindowsEl-ninoDeskForkBookPhoneSockAppleMountainSnowflakeCar…

Action and EventActivityStateEntity

Figure 2-12: Instances of entities, states, activities and events

dimension is dependent on the interpretation and perspective of the user, which is in turndependent on typical human attributes such as experience, intelligence, education and emo-tional state. Consequently, the first four context dimensions can be captured by sensors in themobile memex. The fifth has to be manually added by the user (a person is a kind of socialand relational sensor) by use of annotation and association. Basically this dimension identifiesthe ”why” in a situation. Where the first four context dimensions work bottom-up (system ¿context history), the last context dimension works top-down (user ¿ context history).

The relevance of contextual information to the user is dependent on a number of factors:

1. Perception and situation: based upon user perception we can divide context into two categoriesof relevance:

(a) Primary context. This is the context we naturally relate to, usually people, things,activities, places, time and reasons. The Who(entity), What(activity), When(time),Where(place) and Why(reason) of a situation

(b) Secondary context. context with low relevance such as light intensity, temperature andnoise level. Of course context relevance can depend on the user, situation and application.What is primary context to one, is secondary context to another.

2. Accuracy: Sensor-fusion techniques can increase the level of accuracy. Sensor fusion is basedon combining sensors that provide independend measurements of the same information toproduce a more accurate reading.

34

CONTEXT

Temporal DimensionPrimary Context: Time (When?)Notation: YR(####)MTH(##)DY(##)HR(##)MIN(##)SEC(##)ID Format: UCT-24

Spatial DimensionPrimary context: Location (Where?)Notation: LON(###.###)LAN(###.###)ALT(###.###). ID Format: WGS 84

Functional DimensionPrimary context: Activity (What?)Notation: Natural LanguageID Format: OED verbs in UTF-8

Relational DimensionPrimary Context: Reason (Why?)Notation: URL(localhost/file.ext) ID Format: Annotated URLs in UTF-8

Existential DimensionPrimary context: Entity (Who?)Notation: H(4):H(4):H(4):H(4):H(4):H(4):H(4):H(4) ID Format: IPv6

Figure 2-13: The five context dimensions and their ID encodings

3. Completeness: In general, more context information can lead to a more extensive assessmentof a situation. covering all five sensor dimensions will provide a relatively complete descriptionof a situation.

4. Abstraction: Different situations require different representations, therefore context must bedescribed at the appropriate level of abstraction to be useful. Based upon abstraction orrepresentation we can divide context into two separate categories:

(a) High level: context is presented in a format that makes sense to the user. An example isthe representation of location coordinates as a visual map of the area. High level contextis suited for the end user of the system.

(b) Low level: context is represented in a format that does not make sense to the user.Example; location as a string of coordinates. Low level context is well suited to thesystem and applications.

During the representation transformation of low-level context into high-level context, contextis often converted into content of a textual, arual or visual nature.

Figure 2-14 shows the context layer including examples of primary and secondary context, and thecontent layer including content types.To summarize; The mobile memex defines context as all sensor data that can be used to charac-terize the situation of an entity and be meaningfully represented as human-readable plain textualinformation with no special formatting requirements in the Unicode UTF-8 Characterset.

2.3.4 Content

Content is all data that cannot be losslesly represented as context, and usually requires externalprograms to decode the data in order to be meaningful. Typical content will be digital pictures,sound files, programs, spreadsheets and more. Content has to be decoded or interpreted by externalprograms (algorithms) before they can be presented to the user. Every piece of content has amatching identity in the existential context dimension.

Type, Format and Size

Most content will belong to one of four classes: Visual, Aural, Textual and Executable. Figure 2-14shows the types and instances of content that belong to these classes.

35

pdf, doc, rdf, ps, xls, sql, java, xml, html, ppt, etc.

DocumentDatabaseSpreadsheetSource

Textual

exe, com, bat, jar, php, asp, js, etc.

mp3, midi, wav, ogg, mod, wma, etc.

bmp, jpg, tiff, svg, png, ico, gif, mp3, avi, mp4, mov, raw, psd, etc.

Instances(Format)

ProgramAppletScript

Audio: WaveAudio: Melody

Bitmap pictureVector imageBitmap videoVector animation

Types

ExecutableAuralVisualClass

Figure 2-14: Types and instances of the content classes

Some content can be a hybrid form and contain textual, aural, visual and executable elements allat the same time.

Sizes of the content are relative to it’s class and type. Usually visual content takes up the mostspace especially streams of video data. Aural data based on pulse code modulation (PCM) are alsostreams of data and take up a lot of space also. Executable content can vary in size dependingon the application complexity. Textual content is usually relatively small but can grow in size ifthe document is a database. In the mobile memex textual content is always preferred above theother classes. This is because textual content can usually be processed to extract it’s content in acontextual format (for example a list of keywords) so it can be searched and indexed.

2.3.5 Aggregation

Aggregation is the process of combining the distributed context documents into one master docu-ment, and all content into a single repository. The advantage of aggregation is that all informationis centrally available at a singe device using a single interface, this allows applications and servicesto operate on the data more quickly and efficiently.

Secondary Context

Primary Context

ContentTextual Aural Visual

Activity Time Location

Direction Temperature Atmospheric Pressure Light Intensity

Entity

Executable

Reason

Battery levelDirection Temperature Atmospheric Pressure Light Intensity Battery level

Figure 2-15: Context and content

A Context History

The mobile memex will structure the acquired context based on temporality into what is called a”Context History”. It is also possible to structure the aggregated context based on the other dimen-sion, but since the aggregation process itself is defined by time it is the most logical structure. Inorder to create a context history all context must have a unique timestamp.

Fortunately time context is a relatively easy to obtain, as a built in clock is common in many digitaldevices. However to create a context history that is accurate it becomes necessary that all clocks inthe system are perfectly synchronized. In short; there is a need for a global timeclock mechanism to

36

ensure all clocks use the same system to represent the same time at the same moment. Furthermorethis context should preferably be easily obtained through multiple sources.

Every device in the M3X can add context data of any format and structure. To avoid a garbage in,garbage out situation there have to be strict rules as how context is identified and represented. sothere is minimal ambiguity and overlap between items.

A Content Repository

The content repository is based on the filesystem of the storage device. It is a directory that is able tocontain all files downloaded from the subscribing devices. All files will have a timestamp as a filenameand a file extension that displays the file type and format. This method ensures that the content canbe related to the context bearing the same timestamp. Over time the content repository will growvery large because content requires more storage space than contextual information. Figure 2-14shows an approximation for the daily storage requirements in a typical M3X system. It is obviousthat content information takes up the bulk of the storage requirements. A storage requirement ofaround 7 to 10 GB demonstrates it is economically feasable to record all context and content duringdaily usage.

19.5 MB/day( 7.1 GB/year )

TOTAL storage space required:

2 KBUTF-8 (IPv6)Context: Encountered computational entities23

10 KBUTF-8 (NWSE)Context: Directional coordinates1440

10 KBUTF-8 (dB)Context: Ambient noise level1440

Context: Other messages (SMS, CallerID)

Content: Digital Business Card (size: 200KB)

Context: Ambient temperature

Context: GPS coordinates

Content: Digital Movie (res: CIF, size:

Content: Digital Images (res: VGA, size: 64KB)

Context: IM Chatlog (avg. size: 8KB)

Content: Word Documents (avg. size: 200KB)

Content: Email (avg. size: 3KB)

Content: Voicecall (avg. duration: 7 minutes)

Type

400 KBHTML + JPG2

10 KBUTF-8 (Celcius)1440

10 KBUTF-8 (WGS84)1440

4.4 MBMPEG4 @ 200Kb/s1

384 KBJPG6

8 KBUTF-81

1 MBDOC5

60 KBHTML20

13.2 MBOGG @ 32Kb/s8

5 KBUTF-8 (TXT)5

Encoding Size# ofItems

Figure 2-16: An approximation of daily storage requirements for a typcial mobile memex system.

2.4 The Application Layer

The application layer supports applications and services that make use of the context history andcontent repository. It is possible to make a distinction between application interfaces and services.

2.4.1 Application Interfaces

Application interfaces provide access to the context history and content repository. The contexthistory has a direct interface, targeted at the management of collections and individual nodes, anda query interface that acts as a filter to select specific nodes.

2.4.2 Application Services

There are a number of services that provide essential fucntionality to the mobile memex. Mostservices deal with the adaptation of the context history in order to make it more complete and

37

uniform.

Normalization

Normalization of context: After the context history is updated with the latest PCH the normaliza-tion engine will perform a check to see if context types belonging to the same class have a commonnotation format. If this is not the case, nodes that deviate from the standard notation are converted.

Normalization of content: The same goes for files in the content repository, The system will check tosee if content types belonging to the same class have a common encoding format. Files that deviatefrom the common format for those respected types are converted.

Normalization will increase the uniformity of the context and content, which makes it easer forapplicastions and services to operate on.

In order to ensure portability and compatibility of context and content, the normalization processfavors the adoption of open standards (and if possible open implementations). And the lossless con-version to these standards. The context history itself is an XML document in the Unicode UTF-8format.

The five context dimensions are used by the system to construct a relatively complete descriptionof the situation of an entity. Each of the five context dimensions also has a primary context type(Who (entity), What (activity), When (time), Where (location), Why (relation)). For every primarycontext dimension type, we try to identify a notation that uniquely identifies that instance, and insuch a way that related types and instances can be derived from this supertype.

Context normalization and aggregation according to supertypes:

1. A normalized entity identification system:In order to uniquely identify entities, whether physical or logical it is possible to use the IPv6addressing system. IPv6 offers several improvements over IPv4. Most importantly, with 128-bit Internet addresses instead of the 32-bit addresses of IPv4, IPv6 vastly increases the numberof addresses available from about 4 billion to about 340 trillion trillion trillion, enough to giveevery person in the world a personal address space of 55.151.112.953.150.480.302.005.608.984IPv6 addresses. With 21̂28 addresses, it is theoretically possible to give every unique entity inthe world it’s own unique IP address. IPv6 also offers better security. Together with abundantInternet addresses, these improvements will pave the way for a huge range of new applications.

IPv6 is an ideal candidate for a global entity identifier. If every device in the mobile memex hasit’s own IPv6 address, and every sensor and actuator contained by the device also as an IPv6address, these addresses can now function as a uniform identifier (URI) for computationalentities. The IP address can be used to retrieve the specifications of the device, sensor oractuator for full identification of the entity. This way all the entities in the mobile memex canbe uniquely identified. If entities outside of the mobile memex also have and IPv6 address,interactions with entities in the environment can also be properly identified. IPv6 address canonly be distributed to computational entites. For common physical entities and objects suchas a chair or a door it is possible to use radio frequency ID (RFID) tags 1 that support IPv6.The combination of IPv6 and RFID tagging allows for the indentification of most relevantentities. With IPv6 any identity can be written as a 128 bit address in the following format:

HHHH:HHHH:HHHH:HHHH:HHHH:HHHH:HHHH:HHHH

1RFID systems are frequently used for a variety of applications. These include transportation, distribution, indus-trial uses, and animal identification. Tagging of products such as clothing and supermarket items.

38

Where ”H” is a hexidecimal number. For example a microphone sensor might be identifiedwith the abbreviated address:3ffe:80e8:d8::1

2. A normalized location coordinate system:For globally unique spatial coordinates it is possible to use the World Geodetic System of 1984(WGS 84). WGS84 is the default datum for most GPS receivers. Besides being a map/chartdatum WGS 84 also defines the shape and size of the ellipsoid of revolution (an oblate spheroid)that is considered to be the best representation of the Earth. The mobile memex will use theWGS84 notation of longitude, latitude and altitude in decimal degree format. Based on theseconventions a M3X locationstamp will have the following format:

LON###.###LAN###.###ALT###.###.

Where ”#” is a decimal number. For example a location with longitude coordinates 37.399,latitude coordinates 122.018, and altitude coordinates 14.021 will be written as:LON037.399LAT122.018ALT014.021.

3. A normalized time system:Coordinated Universal Time or UTC, also known as civil time, is the reference time zone fromwhich all other time zones around the world are calculated. It is the successor of GreenwichMean Time (GMT), This time scale is kept by time laboratories around the world, and isdetermined using highly precise atomic clocks. UTC is the time distributed by radio stationsthat broadcast time using the Radio Data System (RDS), and can also be obtained from theGlobal Positioning System (GPS) satellites. UTC includes the date in the Gregorian calendarsystem and the time according to Universal Time (UT) in 24 hour notation (Military Time).Based on these conventions a M3X timestamp will have the following format:

YYYYMMDDHHMMSS

The part ”YYYYMMDD” represents the date represented as decimals and according to theJulian calendar. Where ”YYYY” is the year, ”MM” is the month and DD the day.The part ”HHMMSS” represents the time represented as decimals and according to the uni-versal time (UT) system in 24 hour military notation. ”HH” is the hour, ”MM” are minutesand ”SS” are the seconds.The date and time at a certain moment, for example ”25 September 2003, 15:37:10 UT” iswritten as:20030925153710.

4. A normalized activity identification system:To uniquely identify activities, it is possible to extract a list of all verbs from the OxfordEnglish Dictionary (OED). This is of course not a fool-proof method as the system has to dealwith all the pitfalls a specific language might offer. At the moment however, it seems like themost suitable solution. English verbs might be mapped to verbs of other languages by the useof online translation services or vice versa.

For the system to recognize activities it has to use inference techniques in combination with AIpatterns and algorithms. For example to reconginze a user is performing the activity ”sleeping”it has to combine the pulse reading for the entity ”user” in combination with the reading fortime (night), the reading for light intensity (dark), and perhaps the reading for movementwhich is in turn a combination of locations at different times (not moving). Even after all thisinference the conclusion of the system might be wrong. Maybe the user is sleeping in a movingtrain for example.

5. A normalized relation identification system:To uniquely identify user annotations and associations that provide relational information such

39

as the reason or social status of a situation.

Annotations are unicode textfields that are filled out by the user in natural language to describethe social or relational situation of an entity. If the syntax / semantics of a user annotationcan be extracted, proposition logic might be used to formally describe annotations where thenouns point to entities and verbs to activities in the context history.

An association is a set containing one or more XPATH queries (a query might refer to onlyone node, in this case it could be seen as a kind of URL) that when combined define a set ofcontext and content that has meaning to the user.

To infer new data types in the relational dimension from user annotations and associationsis extremely difficult. relational information in the form of unicode annotations and XPATHassociations are therefore simply added to it’s corresponding timestamp in the context history.

From the supertypes it is possible to infer or refer subtypes that can then be abstracted into theirrequired representation and format.

Abstraction

It is also possible to make a distinction between context based on their representation:

1. Low level context is context that is not very meaningful to the user. When location is presentedas a string of coordinates the user cannot relate to, it is next to useless.

2. High level context is context that is meaningful to the user. If the low-level string of coordinatesis represented as a placename or a map it does become meaningful.

System applications operate on low-level representations of data (arrays, binary, etc.), while end-user will expect high-level representation of the same data (images, descriptions, etc.). Differentsituations require different representations. The abstraction layer can make use of XSLT templatesto transform context into meaningful representations. The abstraction process can be both lossless aswell as lossy. For example, an abstraction of spatial coordinates into a ZIP code is a lossy operation.Some abstractions can combine data from seperate sources into one single representation. Otherabstractions might be text to voice-XML or a VRML representation of proximity data.

Extraction

When context data is aggregated, it is possible to construct logical-sensors. A logical sensor is anapplication that takes the output from one or more sensors as it’s input to produce new context orcontent as it’s output.

there are two types of logical sensors:

1. Inference sensor: Existing sensor information is combined with knowledge represented as al-gorithms to infer new sensor information. For example if the system knows the positioninformation for two separate entities at the same instant in time it can calculate their proxim-ity. It is also possible to extract context from content; a recorded voicecall is abstracted by aspeech-to-text application into a text file. The text file serves as input to a keyword extractionsensor to produces context in the form of a list of keywords that can be searched so the contentcan be more easily retrieved.

2. Reference sensor: A reference sensor uses context or content in combination with an externalsource to obtain new data. Many external sources are available on the internet such as locationservers, route planners, weather servers, news server, movie databases, cd databases etc. Forexample if a time sensor is combined with a RSS news server it is possible to obtain the news

40

or a weather report at that time. An mp3 music file can be used in combination with CDDBto extract the song title and author and mp4 video files can be used with IMDB to extractit’s video cover and related contextual material. GPS coordinates can be used with locationserver to retrieve maps and directions, and so on. Many reference sensors can also be used inthe abstraction process.

Annotation

The original memex allowed the user to annotate it’s contents. In the mobile memex the systemcan do automatic annotation of content by using its context. However the system is not able toautomatically add the relational context dimension. Therefore the mobile memex also provides theuser the possibility to annotate documents and files and thus add the relational context dimensionto the context history. Figure 2-14 shows an example of content with automatic context annotationand the option to add additional user annotations.

Item : Digital PhotoDate : WED, 12 FEB 2003Time : 11:34 GMT+1Loc. : Switzerland, VillarsAct. : Skiing holidayAnn. : Annotate this item

Related content:View other images (23) taken in this context:index, previous image, next imageView other content (55) related to this context.Index, voicecall, email, newsView other content (71) related to activityIndex, skiing, holidayView other content (245) related to location.index, europe, switserland, villars

Figure 2-17: Automatic annotation, and related content query suggestions

Association

The original memex also provided a means for the user to associate documents he or she thoughtwere related in some meaningful way. The mobile memex achieves this by allowing the annotationof sets of queries that in turn define sets of related context and content. It’s a bit like bookmakinga particular query the user found useful so the information it refers to can be easily retrieved later.

Administration

The user controls the system and not the other way around. The administration engine allows theuser to define preference profiles and constraints that define how the system will function. theseprofile can contain anything from sensor polling intervals, preferred network interfaces, security set-tings to sharing constraints and more.

The user can select, install and uninstall applications that can function on the available sensordataand within the constraints set by the user. The administration engine also allows the user to monitorthe internal state of the system (capacity of system resources; battery capacity, network bandwidth,used storage capacity etc) as well as the ability to manually add, modify and delete (contextual)nodes and files.

41

Application Classification

There can exist many different types of applications each with their own characteristics and func-tionality. Figure 2-18 shows a number of applications characteristics that determine their functionand location in the mobile memex.

Archived data

User

Remote

Re-active

(semi) Real-time dataOperation

SystemTarget

LocalLocation

Pro-activeBehaviour

Figure 2-18: Application characteristics in the mobile memex

1. Re-active vs Pro active: It is possible to make a distinction between Re-active applicationsand Pro-active applications. Re-active applications react to user input with output. This is apull-based application model. A pro-active application uses a push-based model and if certainconditions are met, will automatically try to notify the user by using actuators available inthe system.

2. Realtime vs Non-Realtime data: Another distinction between applications can be made de-pending on the timeliness of the data they operate upon. Applications that operate on real-timedata, and applications that operate on historical data (or a combination of the two).

3. Local vs. Remote Applications: Applications that depend on realtime information will haveto be available to the user where this data is generated.

4. System vs. User Applications: System applications are often logical sensors, these run contin-uously in the background and do not have a user interface or require user interaction. Userapplications are designed to augment the user in some way and consequentially do have a userinterface.

2.5 The Augmentation Layer

3rd party applications provide the functionality to augment the user in different tasks. Applica-tions can range from augmented memory, augmented navigation, augmented engineering, augmentedtraining to augmented medication and others.

The secondary-task nature, proactivity, and sensors of wearables make them a natural match forsoftware agent technology. Software agents are programs that continuously run in the background,sensing their environment and occasionally proactively acting on behalf of their user [Maes, 1994].

2.5.1 Licklider’s Man-Computer Symbiosis

The Augmentation supports to a certain extend what was described in 1960 by J.C.R Licklider inwhat is known as the Man-Computer Symbiosis:The Man-computer symbiosis is an expected development in cooperative interaction between menand electronic computers. It will involve very close coupling between the human and the electronicmembers of the partnership. The main aims are 1) to let computers facilitate formulative thinkingas they now facilitate the solution of formulated problems, and 2) to enable men and computers tocooperate in making decisions and controlling complex situations without inflexible dependence onpredetermined programs.

The definition of symbiosis is:

42

”Living together in intimate association, or even close union, of two dissimilar organ-isms.”

The Licklider symbiosis suggest that Human brains and computing devices will be coupled verytightly. There exists a synergy between the two entities. For example computing machines can doreadily, well, and rapidly things that are difficult or impossible for man, and man can do readilyand well, though not rapidly, many things that are difficult or impossible for computers. They arecomplementary. The information system will do as much diagnosis, pattern-matching, and relevance-recognizing as it profitably can, but it will accept a clearly secondary status.The secondary-task nature, proactivity, and sensors of portable information devices in the mobilememex make them a natural match for software agent technology. Software agents are programsthat continuously run in the background, sensing their environment and occasionally proactivelyacting on behalf of their user [Maes, 1994].

2.5.2 Augmentation Applications

Some examples of possible user applications in different fields of augmentation are:

1. Augmented Security: Remote surveillance. Figure 2-19 shows the use case for a remote surveil-lance application. Remote surveillance is an example of a user allowing certain parts of thecontext history to be shared. A child might agree to allow her parents to monitor her locationduring schooltime. A parents will receive a message when she leaves the school compound, sothey can call her to check if everything is all right.

2. Augmented Entertainment: Smart toys. Smart toys can operate on sets of context data to makethem act more realistically and lifelike. One can imagine a system not unlike ”Teddy” [10].The teddy might recognize an activity such as school and react semi-intelligent with a quentionsuch as: ”how was school this moring?”. Yes; Eliza the teddy is your best friend. Toys thatcombine context information from multiple related users can awnser questions such as: ”Whereis Bob now?”. Of course; besides context it needs plenty of AI and and knowledge patterns tofunction realistically.

Monitoring Conditions

(f rom use ca ses)

Remote Surveillance Application Subjec t

(from actors)

Monitor Subject

(f rom use ca ses)

<<depends on>>

Surveil lant

(f rom acto rs)

Set Conditions

(from use cases)

Active Coaching Application User

(from actors) Instruct / Advise

(from use cases)

(a) (b)

Figure 2-19: This Figure contains two panels labeled (a) showing augmented surveillance, and (b)showing augmented coaching.

3. Augmented Finance: Finance planning, shopping. Finance planning and shopping applicationsdeal with money. Every time the user uses a digital cashcard or an internet banking applicationto make transactions context is created. If the user buys stocks it can be tied to an external

43

source on the internet that provides the most recent price and can proactively notify the user ifcertain levels are reached. Augmented shopping can be an application that uses a refridgeratorthat ”content-aware” in the sense that it can identify the types of products it contains if theseproducts are tagged with RFID tags. This refriderator also has a network connection such asa PAN,LAN or WAN interface and can pro-actively notify the user if his or her location is inclose proximity of a super-market and he or she is low or out of some stock products such asmilk or eggs.

4. Augmented Coaching: Active Sports Coaching. Figure ?? shows the use case for a remotesurveillance application

5. Augmented Medication: Healthcare monitoring Augmented Medication can help people bymonitoring vital signs such as pulse, salinity-levels, glucose levels, body temperature and othersto identify certain physical state that require medical support. A pro-active system mightnotify the user and contact the appropriate emergency services (these external sources have tobe identified in the application preference profile first) if nescessary. Figure ?? shows the usecase for an augmented mediacation application

Notify 3rd Party

(from use cases)

Emergency Service

(f ro m acto rs)

Set Application Preferences

(from use cases)

Notify User

(from use cases)

Inject Insuline

(from use cases)

Monitor Physical State (User)

(from use cases)

Medical Monitoring Application User

(from actors)

Provide additional information

(from use cases)

External Service

(from actors)Navigate

(from use cases)

Instruct / Advise

(from use cases)

Specify external services

(from use cases)

Travel Navigation Application User

(from actors)

(a) (b)

Figure 2-20: This Figure contains two panels labeled (a) showing augmented medication, and (b)showing augmented navigation.

6. Augmented Navigation: Navigation and Guidance

An augmented navigation application will only work if there are sensors instances available inthe Spatial dimension. GPS is one of the most common sensors and can provide the globalposition of the user. This information can be used with external sources (using the referencelayer) to provide real time information on the current location of the user and the best path tohis or her destination. Additional contextual information can be taken into account to opti-mize navigation between point, such as traffic and weather conditions, the amount of gasolinein the car, and so on.

If users share their positional data with others it is possible to calculate the shortest pathbetween two dynamic points. There are many applications possible that aid navigation in

44

different ways. These can be based on proximity, speed, accelleration, location, altitude,direction and other sensor instances. Figure ?? shows the use case for an augmented navigationapplication

7. Augmented Memory: Personal Search Engine.In the field of augmented memory there can be a variety of applications. The personal contex-tual search engine discussed in Chapter 4 is one of them. Another application is the Reminderservice. A reminder service can pro-actively notify the user if certain conditions are met. If aperson walks past the supermarket the applciation will check the position and time of the userand if both location and time are within set boundaries (i.e. the user is near the shop and theshop is open) the user will receive a message reminding him/her of a certain task (i.e. buymilk). This example also shows that Applications in different fields can overlap (augmentedshopping and augmented memory applications both augment the user in getting the neededproducts at the appropriate time).

Augmented memory applications can help people with memory disorders such as Alzheimer toimprove their quality of life. But it can also be benificial to most people that need to retrieveinformation related to memory and experience at certain times.

Figure 2-21 shows the use case for an augmented memory application

Browse Results

(from use cases)

Perform Search

(from use cases)

Personal & Contextual Search Engine Applicati ...

(f rom actors)

TimeOut

(from use cases)

<<extend>>

Figure 2-21: Augmented memory

2.5.3 Privacy, ethics and other social issues

In every system that deals with personal information privacy is a major isssue. The mobile memexis no exception and should support sufficient levels of privacy and security to the user.

There are at least two weak points in the system; 1) The body area network where the personal datais collected, and 2) The storage device where the entire context history is stored. wireless networktechnologies broadcast the information in a certain radius. Wired network technologies only sendthe information over a wire and are therfore inherently more secure. There are a number of ways tomake wireless technologies more secure: 1) use a smaller transimssion radius (upt to 2 meter radiusfor the BAN, a 10 meter radius for a PAN, a 100 meter radius for a LAN and a 1500m radius for aWAN), and 2) use strong encryption on the connections.

Other ethical issues are our increased dependency on information systems. Some people argue thatthe pocket calculator made people forget how to calculate. If one continues this line of though itcan also be argued that augmented memory might lead to increased forgetfullness, and augmentednavigation could result in people not being able to find their way around by themselves anymore.Of course this all remains to be seen. We assume augmentation is a good thing as long as there isa choice, so the user can decide when to use, or when not to use certain functionalities. The caseof augmented medication is another issue as these kind of systems can automatically (or possiblyeven remotely) influence the physical and mental condition of the user. Furthermore an augmented

45

medication system that can automatically apply a dosage of a certain medicine to it’s user must becompletely accurate and reliable. One could argue if an ad-hoc system based on personal informa-tion devices with a range of common sensors meets the requirements for this kind of augmentationapplication.

Chapter 3 will examine network technologies and encryption techniques that meet the security andprivacy requirements for general use.

46

Chapter 3

Requirements and Design

3.1 M3X Software Requirements Specification

This chapter describes the Software Requirements Specifications for the mobile memex (M3X) ac-cording to the guidelines of the IEEE 830-1998 standard for creating SRS documents [40].

3.1.1 Purpose

The purpose of the SRS is to investigate the possibility of using common mobile information devicesas a basis for an automated autobiographical context aggregation and archiving system. The M3Xsystem is a framework that supports context aware applications. It strives to be fully customizableto the users needs on both the hardware (devices) and software (applications) side of things. Thesespecifications will serve as a guideline for future M3X work.

3.1.2 Scope

This section will identify the software product to be produced by name and explain what the soft-ware product(s) will do.

In this thesis there exist two products:

• Name: The Mobile Memex (henceforth M3X)Objective: : Distributed sensor data aggregation and archiving in the body area network.Purpose: Create an autobiographic context history of a user.Product : Document. The product of this project will be a detailed overview of functionalrequirements that need to be addressed in order to construct the mobile memex system.

• Name: The personal conceptual search engine (M3X augmentation application)Objective : Local and remote retrieval of personal contextual information and related contentPurpose: Augment the user’s memory by offering retrieval cues through queries.Product: Application. A proof of concept application - the personal search engine, will bedeveloped to demonstrate the viability of the system. The M3X personal search engine willbe useful for retrieving contextual information and the content created in the context by usinga web based interface much like Google that allows the user to produce a query returningrelevant information. Please refer to Chapter 4 for a more detailed description of the proof ofconcept application.

3.1.3 Definitions , acronyms and abbreviations

This section will provide the definitions of all terms, acronyms and abbreviations required to properlyinterpret the SRS. Figure A-1 in Appendix B shows the acronyms used in the system requirements

47

specification.

3.1.4 References

This section provides a list of all documents referenced to elsewhere in the SRS. Please refer to thebibliography appendix in the back of this thesis.

3.1.5 Overview

This subsection will describe how the SRS is organized:

1. Section 1 will describe what the rest of the SRS contains and explain how the SRS is organized.

2. Section 2 will give a general overview of the functionality of the project (C-requirements). Thisfunctional overview is used to establish a context for both the informal requirements definitionass well as the technical requirement specifications. Section 2 consists of six subsections, asfollows:

• Product perspective. This section puts the product into perspective with other relatedproducts, and relates the requirements of the larger system to the functionality of theindividual software components. Also identified are the interfaces between the systemand these software components.

• Product functions. This section will provide a summary of the major functions that thesoftware will perform.

• User characteristics. This sections describes the general characteristics of the intendedusers of the M3X system, including educational level, experience, and technical experi-ence.

• Constraints. This section provides a general description of any other items that willlimit the developer’s options. These can include constraints such as hardware limitations,interface constraints and security constraints.

• Assumptions and dependencies. This section lists each of the factors that affect therequirements stated in the SRS. These factors are not design constraints on the softwarebut are, rather, any changes to them that can affect the requirements in the SRS.

3. Section 3 we will give a detailed overview of the functional and non-functional requirementsof the project (D-requirements). Section 3 consists of six subsections, as follows:

• External interfaces. This section contains a detailed description of all inputs into andoutputs from the M3X software system. It complements the interface descriptions insection 2 but does not repeat information there.

• Functions. This section discusses the functional requirements that define the fundamentalactions that must take place in the software in accepting and processing the inputs andin processing and generating the outputs.

• Performance requirements: This section specifies both the static and dynamic numericalrequirements placed on the software or on human interaction with the software, as awhole.

• Logical database requirements: This section specifies the logical requirements for anycontextual information that is to be placed into the native XML database. This mayinclude the context dimension including unique instance IDs, Content entities and theirrelationships and types of information used by various functions.

• Design constraints. This section specifies design constraints that can be imposed by otherstandards, hardware limitations etc.

• Software system attributes. This section presents a number of attributes of softwarethat can serve as requirements. Some examples are reliability, stability, portability andmaintainability of the system.

48

3.2 Overall Description

This section specifies the role the M3X plays in its environment and how it interacts with itsenvironment to fulfil that role. It specifies all interfaces between the software and its environmentincluding hardware, software and human systems. It also specifies all constraints imposed on thesoftware design and implementation.

3.2.1 Product Perspective

The product perspective puts the product into perspective with other related products, and relatesthe requirements of the larger system to the functionality of the individual software components.Also identified are the interfaces between the system and recommended software components.Figure 3-1 presents a block diagram that shows the major components of the larger system.

Context History NXD

XPATH query interface

Content Repository FS

Application Repository

REMOTE / LOCALSYSTEM / USER

XSLT normalization templates

XSLT abstraction templates

WEBDAVremote application interface

update

Local NormalizationApplication

store

update

Aggregation Engine

Access Device

XULinterface templates

Storage Device

Subscriber DevicePublishing Device

Publishing Device

XMLparser

Acquisition & InterpretationEngine

Acquisition & InterpretationEngine

Environment

User

System

XML:DB interface

M3X profilesSYSTEM / APP / DEVICE

XML SignatureXML Encryption

XML SignatureXML Encryption

Figure 3-1: Overview of the M3X system architecture

The mobile memex is dependent on a number of standardized software components. These compo-nents provide a standardized platform as well as standardized middleware on which the system canbe built. Below we propose a number of software components recommended for use in the mobilememex:

• Application Platform:

1. JAVA J2SE, a JAVA virtual machine (JVM) is the programming platform for subscribingand storage applications.

2. JAVA J2ME, a kilo virtual machine (KVM) for the publishing applications. Small devicescan user a connected limited device configuration (CLDC) profile, which is part of theMobile Information Device Profile (MIDP). CLDC-based profiles are targeted at very

49

constrained devices with some kind of network connectivity: Cell phones, interactivepagers, low-end PDAs. Medium device can use the a connected device configuration(CDC) profile, which is part of the Personal Profile. CDC-based profiles are target atrelatively constrained connected devices such as High-end PDA’s, Set-top boxes and thelike.

• P2P device networking toolkit:

1. JXTA (http://www.jxta.org/) JXTA technology is a set of open protocols that allow anyconnected device on the network ranging from cell phones and wireless PDAs to PCs andservers to communicate and collaborate in a P2P manner. The JXTA reference imple-mentation used in the mobile memex is written in JAVA.

2. JXME (http://jxme.jxta.org/) The purpose of project JXTA for J2ME is to provideJXTA compatible functionalities on constrained devices using the Connected LimitedDevice Configuration (CLDC) and the Mobile Information Device Profile 2.0 (MIDP).The range of devices include the smart phones to PDAs. Using JXTA for J2ME, anyMIDP device is able to participate in P2P activities with other MIDP devices. At thesame time, a MIDP device is able to participate, with some restrictions, in P2P activitieswith JXTA peers running on desktops/workstations/servers.

• Remote application interface:

1. HTTP Server (http://httpd.apache.org/) a webserver such as Apache, used for hostingremote M3X applications.

2. TOMCAT (http://jakarta.apache.org/tomcat/ Tomcat is the servlet container that isused in the official Reference Implementation for the Java Servlet and JavaServer Pagestechnologies.

• Remote application user interface:

1. XUL (http://www.mozilla.org/projects/xul/xul.html) XUL (XML User-interface Lan-guage) is a cross-platform language for describing user interfaces of applications.

2. SVG (http://www.w3.org/TR/SVG/) Graphical User Interface Elements such as iconscan be described using the Scalable Vector Graphics specification.

• Remote application authentication and encryption:

Applications that will be remotely accessible will have to perform authentication, so onlyauthorized user can access the application and its content. An authenticated user will thencommunicate with the application over a secure connection:

1. XML Signature (http://www.w3.org/Signature/) XML compliant syntax used for repre-senting the signature of Web resources and portions of protocol messages (anything thatcan be referred to by a URI) and procedures for computing and verifying such signatures.XML Signature recommends a standard means for specifying information content to bedigitally signed and for representing the resulting digital signatures in XML. Some appli-cations require the ability to specify a subset of a given XML document as the informationcontent to be signed. The XML Signature specification meets this requirement with theXPath transform.

2. XML Encryption (http://www.w3.org/Encryption/) A process for encrypting/decryptingdigital content (including XML documents and portions thereof) and an XML syntaxused to represent the (1) encrypted content and (2) information that enables an intendedrecipient to decrypt it.

50

• Remote Administration Application:

1. XINCON (http://metzner.org/projects/XINCON/) a prototype of a web and webdavadministration interface for the Xindice NXD, XINCON is a servlet that provides afrontend to XPath as a remote M3X application.

• Content Repository FS:

1. REISERFS (http://www.namesys.com/) the ReiserFS is a general-purpose file systemwhich uses journalled metadata.

ReiserFS is a suitable filesystem for storing content. Performance is greatly improved bythe use of dancing trees, wandering logs, allocate on flush, a repacker, and encryption oncommit. ReiserFS features a semantic layer, which means amongst other things that allattributes of a file are also stored in a file. Attributes could be: owner, ACLs, last timeof change, name of file. Within ReiserFS there could be unlimited number of attributesattached to one file. These attributes can be used by the mobile memex to specifycontent class, type and instance and attach some contextdimensionIDs such as timestamp,locationID, annotation etc. In Reiser4 an object can be both a file and a directory at thesame time. If you access it as a file, you obtain the named sequence of bytes. If you useit as a directory you can obtain files within it, directory listings, etc.Reiser was contracted by Darpa to realize security features not found in other Filesystems.For example, when someone reads a certain file, a email is sent to the administrator. Thismeans that in the mobile memex, all interactions with it’s content can also generate newcontext (when the file was opened, how many times it was opened, by who it was openedetc.) this context can then be stored as contextual attributes in the file object itself.

• Context History NXD:

1. Xindice (http://xml.apache.org/Xindice/) a native XML database (NXD) system. Analternative is eXist.Apache Xindice is a database designed from the ground up to store XML data or what ismore commonly referred to as a native XML database. The benefit of a native solutionis that you don’t have to worry about mapping your XML to some other data structure.You just insert the data as XML and retrieve it as XML. You also gain a lot of flexibilitythrough the semi-structured nature of XML and the schema independent model used byXindice. This is especially valuable when you have very complex XML structures thatwould be difficult or impossible to map to a more structured database.

• Context history query language:

1. XPATH (http://www.w3.org/TR/xpath) XPath is a language for addressing parts of anXML document, and can function as a query language for NXD’s. An alternative toXPath is XQuery (http://www.w3.org/TR/xquery/).

• Service application templates:

1. XSLT This is the transformation language, which lets you define a transformation fromXML into some other format. For example, you might use XSLT to produce HTML, or adifferent XML structure. You could even use it to produce plain text or to put the infor-mation in some other document format. An XPATH expression specifies a pattern thatselects a set of XML nodes. An XSLT template is a set of transformation instructionsthat apply to the nodes selected by the XPATH expression.

2. XSL-FO, The ”flow object” standard. This standard gives mechanisms for describingfont sizes, page layouts, and how information ”flows” from one page to another. It isused by the mobile memex for situation, preference and device specific presentation ofinformation.

51

• Advanced encryption standard:

1. AES Rijndael (http://csrc.nist.gov/CryptoToolkit/aes/rijndael/Rijndael-ammended.pdf)Rijndael’s combination of security, performance, efficiency, ease of implementation andflexibility make it an appropriate selection for the Mobile Memex. For example Rijndaelcan be implemented on devices with very low processing and memory capacities such asa javacard. Rijndael AES has the potential to remain secure well beyond twenty years.

• Vocabulary Description Language:

1. RDF (http://www.w3.org/TR/rdf-schema/) The Resource Description Framework (RDF)is a general-purpose language for representing information in the Web. Information onthe Semantic web, rather than being in natural language text, is maintained in a struc-tured form which is fairly easy for both computers and people to work with.The structuring is simple: knowledge is expressed as descriptive statements, saying somerelationship exists between one thing and another. ”Jane has a mother, Susan” or ”Susanis a mother of Jane”. An enormous amount of people’s knowledge can be expressed insentences like these. ”Part #4521 has a price, $19.95.” ”George has a city of residence,Washington D.C.” ”The United States has a president, George W. Bush.” This kind of in-formation structuring was standardized as RDF. Semantics of context can be representedin RDF documents relevant to the constructions of onthologies essential to applicationsin the augmentation layer. RDF could be used to describe content associations.

The M3X software components will rely on the availability of hardware devices that can providesensor information, a BAN connecting them and a P2P protocol that allows for automatic discoveryof devices and services in an ad-hoc network. The M3X also relies on a native XML database anda web server supporting servlets that can remotely be accessed from any web browser.

For a device to qualify as an M3X device it has to support:

• JXTA: A peer to peer protocol that allows for ad-hoc networks of different devices.

• Sensors (0..n): Including an API that allows the JXTA based application to gain access to thesensors

• Actuators (0..n): Including an API that allows the JXTA based application to gain access tothe actuators

• a network interface that provides access to the M3X peergroup

In the M3X there can be 4 different devices based on their role and function in the mobile memexas defined in the device preference profiles:

• A publisher device: This is a portable M3X device containing one or more sensors and zero ormore actuators. A publishing device is responsible for the acquiring, packaging, caching andpublishing of sensor data.

• A subscribing device: A portable M3X device with zero or more sensors and actuators anda fair amount of CPU and Memory resources. A subscribing device is responsible for theaggregation of published sensor data into a partial context history.

• A storage device: A portable or stationary M3X device with abundant CPU and Memoryresources. A storage device is responsible for the storing of the complete context history andthe querying hereof.

• An access device: A device with access to M3X enabled applications that can operate onthe partial context-history contained in a subscribing device or the complete context-historycontained in the storage device. An access device is responsible for allowing the user to interactwith the system to retrieve contextual information and related content from any device capableof browsing the internet.

52

Publishing Device Subscribing Device Storage Device

Access Device

publish store

retrievequery

Figure 3-2: Diagram view of functional interaction between devices

Figure 3-2 gives a schematic overview of the main interaction between types of devices in the M3X.It is possible for a device to perform multiple roles at once. A subscribing device can also be apublishing device, a storage device and an access device. If one device is able to perform all roles,the minimum number of devices required for a working M3X system is 1. For a fully functioningM3X there needs to be at least one of every kind. A fully functioning M3X is a system that bothacquires new context-data and content, and allows the user to access these context and content byuse of M3X enabled applications. In the M3X Peergroup there can be 0..n publishing device, 0..nsubscribing devices and 0..1 storage devices. Typically there will be 3 to 4 publishing devices, 1subscribing device and 1 storage device.

Interfaces

Here we list each system interface and identify the functionality of the software to accomplish thesystem requirement and the interface description to match the system:

• Remote M3X application: accessible from any device with a web browser able to parse XMLand a network connection to the server running the remote application.

• Local M3X application: accessible from the host device running the application (subscribingdevice or storage device)

• Native XML Database: hosted by the storage device and accessible to subscribing devices andapplications. Subscribing devices can only add to the NXD whereas applications will also beable to modify/delete entries.

• Query interface: XPATH query interface capable of sending and retrieving (sub-)sets of contextfrom the XND to the application.

• Interdevice communication: devices can discover other nodes and services by joining a peer-group. Communication with other devices is done by using the JXTA protocol.

• Network interface: devices can use different network interfaces and protocols depending ontheir capabilities and on the suitableness for a certain task. Protocols can be everything fromTCP/IP, PacketRadio, HTTP and others. Every device should have at least one networkinterface that is able to contact the M3X peergroup.

User interfaces:

• M3X application UI: interface dependent on the type of application and the device and platformit is running on. Preferable the interface would be UIML or XUL based.

53

• M3X administration UI: interface running on a subscriber device. It is a local M3X application.Although there can be multiple subscribing devices in a M3X system, there can be only oneM3X profile. The administration UI allows the user to define constraints (set privacy and secu-rity constraints) and application specific settings. It is here that the user can install/uninstallM3X applications and monitor the internal state of the system (such a network performance,memory usage and battery levels).

• Personal Search Engine UI: This is a remote M3X application (although it is also possible tocreate a local version) and allows the user to formulate a search query in order to obtain resultsfrom the NXD.

Hardware sensor (input) interfaces:

Sensors in the mobile memex provide the input for the system. Below we describe a number ofsensor interfaces commonly found in personal information devices:

• GPS: The Global Positioning System can provide the system with detailed positional coordi-nates of the user.

• CMOS,CCD: CMOS sensors can be used to create digital images of the environment.

• Light intensity: CMOS sensor often come combined with light intensity sensors.

• Microphone: Microphones are integrated into the communicator and can measure voice andambient sounds.

• Keypad: A keypad can be found on devices that fit into a users pocket. These are discretesensors to measure user input.

• Keyboard: A keyboard can be found on large class devices such as laptop’s and workstations.These are discrete sensors to measure user input.

• Mouse, Trackball, Pointer: A sensor that is able to record the hand/finger movement of a user

• Touch screen: A sensor that is able to sense user interaction with the display.

• Altimeter: Altimeters can be found in sports watches and measure the current altitude.

• Barometer: Barometers are also commonly found in sports watches and measure atmosphericpressure.

• Thermometer: Thermometers can measure the temperature in degrees ◦C.

• Accelerometer: Accelerometers are an example of mechanical sensors and can measure theacceleration of an entity.

• Geomagnetic: Geomagnetic sensors can be found in watches, phones or GPS devices and canmeasure the direction.

• Conductivity: Conductivity can measure the ability to conduct electricity between differentpoints. These can be used to sense the physical or mental state such as stress by conductingelectricity between points on the users body. Because sweat contains more salt it will conductbetter.

• Electro Cardiac: Electro cardiac sensors can be found in medical equipment.

• Infrared: Infrared is commonly found on remote controls and can be used as a network interface.

• Network sensor. A network interface can be regarded as a network sensor. It can sense theconnection quality and bandwidth but also all information received over this connection.

54

• Compass. A compass gives directional information based on the earth magnetic field

• Inclinometer. An inclinometer can measure the angle of an inclination.

• Gyroscope. One prerequisite is that the sensor has to be portable and operate within areasonable power-range that does not exceed system specifications.

There exist many more sensors (Gas sensors, Biological sensors, Piezoelectric sensors, Motion sensors,Proximity sensors, Wind speed sensors, etc), It is however not our intention to give an exhaustivelist, and therefore only listed the most common sensors.

Software sensor (input) interfaces:

• SMS,MMS,IM: All messages that enter and leave the system through its network interfaces.

• Email,agenda,notes. Textual representation of information sensed by applications running onthe system (an applications functions as a software sensor).

• Application logs (key logger, browsing history)

• Newsfeeds (RDF, RSS), Blogs. Information that can periodically be polled by internet con-nected applications.

Hardware actuator (output) interfaces:

• Screen: A display or screen is able to visually present information to the user.

• Speaker: A speaker is able to aurally present information to the user.

• Vibrator: A Vibrator is able to tactually present information to the user.

• LED indicator: A LED indicator is able to visually indicate the state of the device or systemto the user.

Software actuator (output) interfaces

• Application: An application is able to present information to the user by also making use of aphysical (hardware) actuator available in the system.

Figure 2-14 shows the architecture of a typical personal information device.

Manufacturer Operating System(Embedded Linux, QNX, windows etc…)

VibrationunitLCD DisplayKeypad Microphone Speaker IrDA Battery Status

LED GSM

Native device application 1.. Native device application ..N

JAVA VM & I/O API(J2ME CLDC KVM / J2SE)

JXTA

M3X App

Drivers

OS

Apps

GPS CMOS / CCD Touch Panel CF Card Geomagneticsensor Bluetooth USB Flash ROM Light intensity

sensor

Other JAVA apps

Caller ID SMS / MMS Email Agenda Contacts Todo

Figure 3-3: A typical device architecture

55

Software interfaces

This section specifies the required software products and interfaces with other applications systems:

• JXTA (http://www.jxta.org/):JXTA technology is a set of open protocols that allow any connected device on the networkranging from cell phones and wireless PDAs to PCs and servers to communicate and collaboratein a P2P manner.

• NXD (http://xml.apache.org/Xindice/):a native XML database system such as Xindice or eXist.

• XPATH (http://www.w3.org/TR/xpath):XPath is a language for addressing parts of an XML document, can function as a querylanguage for NXD’s. An alternative to XPath is XQuery (http://www.w3.org/TR/xquery/).

• HTTP Server (http://httpd.apache.org/):a webserver such as Apache, used for hosting remote M3X applications.

• TOMCAT (http://jakarta.apache.org/tomcat/):Tomcat is the servlet container that is used in the official Reference Implementation for theJava Servlet and JavaServer Pages technologies.

• XINCON (http://metzner.org/projects/XINCON/):a prototype of a web and webdav administration interface for the Xindice NXD, XINCON isa servlet that provides a frontend to XPath as a remote M3X application.

Communications interfaces

This section specifies the various communication interfaces available in the M3X. In the M3X everycommunication interface available is also a sensor and actuator.

Hardware network communication interfaces:

On the physical level there can be a multitude of different network technologies. Interdevice com-munication in the users BAN has to be achieved at a minimum power consumption levels, maximumprivacy and security, and preferably reasonable speeds and cost.

Because many users might not be willing to connect every device on their body with wires, themobile memex will use wireless network technology to connect the devices in a body area network.This wireless BAN network technology can be radio magnetic in origin such as UWB or based onmagnetic induction. It is possible that there is a suitable wired solution, this is a network technologythat uses the human body as ”wet-wire”. Experiments by NTT Docomo and IBM have shown itis possible to reach speeds of 10Mb/s at a power consumption level measured in picoamps. Thebody is also a contained entity so information can only leave the host in case of direct contact withother entities. This provides a fast, low power network that even without using encryption offers areasonable level of privacy.

• Induction, Magnetic:This technique uses magnetic induction rather than radio for close-range communications. Inradio, both electric and magnetic fields make up the signal, while in induction wireless, onlythe magnetic field is transmitted.

• Conduction, Electric:Communication between devices in the BAN by means of dermal conductivity provides a lowcost, low power and high speed network that provides a high level of privacy.

56

• Infrared, Optical:This was probably the first PAN technology for use over short distances. The IrDA infrared(IR) standard appeared during the early 1990s. IrDA initially offered a 115.2-kbit/s datarate over a range of up to 1 m. A 4-Mbit/s version was soon developed and has been widelyincorporated in laptops and PDAs for printer connections and short-range PANs. A 16-Mbit/sversion is available too. The problem with IrDA isn’t just its very short range, but also itsneed for a line-of-sight (LOS) connection.

• Radiation, Electromagnetic:Also know as radio waves. There are a number of wireless solutions based on radio suitablefor on-body communication (BAN) such as Utrawideband and ZigBee, near-body communi-cation (PAN/LAN) such as Bluetooth and WiFi, and off body communication (WAN) such asWCDMA. Here we list the characteristics for the respective network technologies:

– Bluetooth: Bluetooth operates in the 2.4-GHz band and with a spread spectrum, fre-quency hopping, full-duplex signal at up to 1600 hops/sec. The basic data rate is Thesignal hops among 79 frequencies at 1 MHz intervals to give a high degree of interfer-ence immunity. Up to seven simultaneous connections can established and maintained.Transmission power can range from 1mW for a 10m radius, to 30mW for a 100m radius.Bluetooth can achieve a maximum data-rate of 11Mb/s.

– ZigBee: A simpler, slower, lower-power, lower-cost cousin of Bluetooth,. ZigBee is likeBluetooth because it uses the 2.4-GHz band with frequency-hopping spread-spectrumwith 25 hops spaced every 4 MHz. The basic data rate is 250 kbits/s, but a slower 28-kbit rate is useful for extended range and greater reliability. With a 100mW power level,ZigBee can achieve a range of up to 134 meters at 28 kbits/s. It additionally lets younetwork up to 254 nodes.

– Ultra Wideband: UWB transmits data by way of baseband pulses applied directly tothe antenna. The narrow pulses (less than 1 ns) create an extremely broad bandwidthsignal. The pulses are modulated by pulse position modulation (PPM) or binary phase-shift keying (BPSK). UWB operates in 3.1- to 10.6-GHz band. UWB technology is usefulfor short-range LANs or PANs that require very high data rates (over 100 Mb/s). UWBis still new in the commercial and consumer world, but it’s well known in military andgovernment applications.

Considering factors such as range, speed, power consumption, cost, size, saleability, interfer-ence robustness and security it becomes possible to make a number of recommendations fornetwork technologies in their respective categories:

Power consumption becomes critical in wireless BANs because each node must have its ownenergy source. This may be a small battery, a generator or a solar cell. To run the networknodes with such stringent power constraints it is necessary that the nodes work extremelyenergy efficient. This can be achieved with low power circuits and very simple and energyefficient communication protocols. When it comes to Mb/s per mW, 802.15.3a (UWB) has a10x better ratio that 802.11a and a 100x better ratio than Bluetooth. UWB has an appreciableadvantage over 802.11 inside of 10 meters. As a wireless communication technology UWB isvery suitable for use in the WBAN (up to 2 meters) and the close proximity WPAN (up to 10meters). For WLANs (up to 100 meters), 802.11g is best suited and for WWAN (up to 1500meter) WCDMA based network technology can be used.

Figure 3-4 gives an overview of wireless network technologies and their characteristcs relevantto the mobile memex.

Software network communication interface:

57

Basically all network communication interfaces are supported. In JXTA HTTP is at the same levelas TCP/IP as an Endpoint Protocol.

3.0 – 10.0

2.45

2.4 – 5.0

0.8 – 1.9

Typical Frequency band (in GHz)

1 – 10m

10 – 100m

100 - 1000m

1 – 5Km

Typical Range

30 mW720 Kb/s - 1Mb/s BluetoothWPAN

100 mW10 – 54 Mb/sWiFi 802.11xWLAN

160 mW9.6 – 384 Kb/sGSM/GPRSWWAN

15 mW20 Mb/s – 1 Gb/sUWBWBAN

Typical Power-Consumption (Rx)

Typical Bandwidth

InstanceType

Figure 3-4: Networks, bandwidth and power consumption

Memory Constraints

Different devices mean different memory constraints.

• Devices in the category ”Small” such as a digital watch, or JAVA card/ring will usually onlyhave RAM type memory with sizes ranging from 1 to 64 MB

• Devices in the category ”Medium” such as a digital camera, phone or PDA will have RAMmemories in the range of 64MB to 5GB.

• Devices in the category ”Large” such as laptop’s, PC’s and workstations will have both RAMmemories ranging from 512MB to 5GB and magnetic memories such as hard disk drives in therange of 250GB-5TB.

Small and medium devices can only cache their sensor data and content in local memory until it iseither full or a large (storage) device is available in the network for offloading.

Operations

This section specifies the normal and special operations required by the user of the M3X system.

1. M3X system general configuration and constraints:

• define privacy settings (what will be measured and what not / what data will be sharedand what not)

• define security setting (encryption)

• define memory use (how much memory will the system require/consume?)

• define polling interval (what sensors will be polled in what intervals?)

2. M3X application management:

• application installation:

– application selection: based on available sensors and actuators in the system, whatapplications are available for use?

– where will the application be installed

• application configuration:

– define constraints (access: remote, local)

– define patterns (application dependent)

– define external sources (application dependent)

58

3. M3X device management:

• application preference profile:

– define specific device constraints

– define allocation and use of resources for system purposes(battery, memory, CPU,network use)

Figure A-6 shows use case diagrams of the publishing, subscribing and storage devices:

3.2.2 Product Functions

This section will provide a summary of the major functions the M3X software will perform.

1. domain JXTA device:

• Peergroup Registration

• Peer discovery

• Pipe creation

• Advertisement creation

2. domain M3X device:

• Profile processing

3. domain Publishing device:

• Acquisition: acquire data from sensors available to the device.

• Interpretation: interpret sensor data and produce a M3XML document.

• Notification: publish the context information to a subscribing device in the peergroup

4. domain Subscribing device:

• Aggregate: context information from multiple publishers is aggregated (based on times-tamp) into a partial context history (PCH).

• Store remote: the PCH is sent to a storage device if available.

• Host: store M3X applications that can operate on a PCH, usually applications thatoperate on (semi)-realtime information, and Pro-active applications.

5. domain Storage device:

• Update: the PCH is used to update the context history contained in the NXD.

• Query : Provide an interface that allows external applications to retrieve context infor-mation from the context history.

• Host : store M3X application that operate on the complete context history, Usuallyreactive applications that operate on historical information such as the personal searchengine.

6. domain Access device:

• Access: Access the local applications on the device, or the remote applications by meansof the web.

• Retrieve : Use of the application to retrieve the requested information.

• Process : Process the retrieved information, this is done by the application, the user, orboth.

• Annotate : The user is able to annotate content to make the context more complete andaccessible in the future.

59

3.2.3 User Characteristics

This sections describes the general characteristics of the intended users of the M3X system, includingeducational level, experience, and technical experience.

Users of the M3X system will vary wildly. This is because the M3X system is fully customizableto user needs. In general the system will be used by people that find themselves with a number ofelectronic devices that they carry around daily, and wish to extend their functionality by openingup synergy effects, resulting from the networking of these devices in a BAN.

The basis of the M3X system is nothing more than a decentralized system of devices sharing sensorand content data, and making this data available from a central location to external applications.It is the external applications that provide the functionality that the user seeks.

Applications can be found in the fields of information services, medical monitoring, sports coaching,remote surveillance, toys etc.

A user can be a diabetes patient, that uses the system to monitor insulin levels and blood pressureto a photographer that uses the system to automatically annotate and store all his photos in thecontext they were made in, to a child with a toy that uses the available context to interact in ameaningful and entertaining way.

3.2.4 User Requirements

This sections describes the basic user requirements of the intended users of the M3X system.In general a user of the M3X will expect the following system characteristics:

• Existing hardware: the user would like to use the system without having to purchase additionalhardware.

• Minimal user intervention: the system should operate with minimal user intervention.

• Privacy: The user has to be able to decide what sensors should be monitored and define sharingrestrictions.

• Security: the information recorded should by default not be accessible by anyone but the user.Encrypted context history.

• Personalized: The user would like to choose the applications and devices for the system he/shehas a need for.

3.2.5 Constraints

This section will provide a general description of items that will limit the developer’s options.

• Political issues: individual information recorded can be misused by others.

• Social issues: the user doesn’t want to record or share information available in the system andrequired by certain applications to function properly.

• Hardware limitations: The system is always as efficient as the weakest link. Some devices willhave limited battery power. There might not be enough hardware resources such as memoryand processing power to run the required application. Sensors have their own limitations suchas resolution, reliability, accuracy and range that can cause applications to function outside oftheir specifications and produce incorrect data.

60

• Interfaces to other applications: the interface to other applications is by means of the XPathquery language, imposing all the limitation of that language upon the applications. There canexist sensor information of the same type (for example ambient temperature) but in differentnotations (Celsius, Fahrenheit and Kelvin). While applications can only deal with data in onenotation, implying the need for a context translation service.

• Reliability requirements: Users might wear different devices with different functionalities atdifferent times resulting in a context history that contains gaps in continuity of sensor data.Also batteries might run out or local memories might fill up resulting in data loss. The networkconnection used by devices to share data might be unreliable in nature.

• Integrity: Information in the NXD can be modified and deleted.

• Criticality of the application: If used in critical situations, such as medical monitoring thereshould be strict requirements as to the Reliability and Stability of the system. Only devicesand network interfaces that fit the requirements can be used in this case.

• Safety and security considerations: all system settings should default to the highest securitysetting: no sensors will be monitored and all sharing of information is disabled. It is up to theuser to set the security settings and enable sensor monitoring. Pro-active applications mighttrigger actuators that carry a high safety risk (for example an insulin injection implant for thediabetes patient - wrong triggers and wrong dosage will put the health of the user at risk).These actuators should only be triggered after first notifying the user of the system intent andreceiving confirmation from the user.

3.2.6 Assumptions and Dependencies

M3X devices are dependent on JXTA. Currently there is only a reference implementation of thistechnology in JAVA. Therefore the devices currently have to support at least the J2ME profile tofunction in the system. It is possible to write a custom implementation of JXTA on the embeddedplatform of the device.

3.2.7 Apportioning of Requirements

This section will identity requirements that may be delayed until future versions of the system.

3.3 Specific Requirements

This section contains all the software requirements at a level of detail sufficient to enable designersto design a M3X system that satisfies those requirements.

Class Diagram

This section includes the Class Diagrams for the M3X domain.

Class diagrams represent the structure of the system and are used during: 1) Requirements analysisto model the problem domain, 2) System design to model subsystems and interfaces, and 3) Objectdesign to model classes. The Object Constraint Language (OCL) can be used to specify additionalknowledge of the domain.

Figure A-2 in Appendix A shows a fully expanded class diagram that represents the different classesin the package com.pink.jxta.m3x:

61

Use Case Diagrams

This section provides a range of different use cases that take place in the M3X system.

Use case modeling is used during the mobile memex requirements elicitation to represent externalbehavior.The M3X use case model is the set of all use cases. It is a complete description of thefunctionality of the system and its environment, but relatively vague.Actors present roles (types of user of the system). An actor models an external entity which com-municates with the system. Common actors in the mobile memex are users, external systems andthe physical environment.

Figure A-8 in Appendix A shows all actors in the mobile memex.Figure A-7 shows use cases for different applications and devices in the M3X system.

Please refer to Appendix B for a detailed description of the actors and use cases found in the moiblememex.

Functions

Functional requirements define the functional actions that must take place in the software in accept-ing and processing the inputs and in processing and generating the outputs. These include validitychecks on inputs, responses to abnormal situations and error handling and recovery.

Sequence Diagrams

This section describes appropriate sequence diagrams between entities.Sequence diagrams emphasize the sequence of the messages send between components in the system.Figure A-5 in Appendix A shows how the personal search engine application operates. Sequencediagrams for the core M3X system are omitted since they are not within the scope of this thesis dueto time constraints.

Activity Diagrams

This section discusses activity diagrams describing processes in the system. Figure A-3 gives anoverview of the activity in and between devices in the M3X system.

3.3.1 External Interfaces

This section contains a detailed description of all inputs into and outputs from the M3X softwaresystem.

• XML:DB interface: External applications can access and control the context history directly,by using the Xindice syntax from within the application itself to add, modify and deletedocuments and collections.

• XPATH query interface: External applications can query the context history to retrieve rele-vant subsets of contextual information

• FTP: application can upload content to the M3X storage device

• User input: the user can use the input methods available to send information and commandsto the M3X system.

3.4 General Requirements

This section separates general requirements from the specific requirements.

62

3.4.1 Performance Requirements

This section specifies both the static and dynamic numerical requirements placed on the software oron human interaction with the software, as a whole.

Statistical numeral requirements for the M3X system include: The maximum number of devicesupported (scalability of the system), the number of users supported, and the amount and type ofall data/information handled.

3.4.2 Logical Database Requirements

This section specifies the logical requirements for the information that is to be placed into the NativeXML Database.

• context document format: XML. An XML document is a tree-structured (hierarchical) collec-tion of nodes. (Context History Tree)

• context document type: M3XML (see M3Xml schema)

• context document encoding: Unicode. The Unicode Standard defines the universal characterset. It is a means to assign a unique integer for all characters used by humans in writtenlanguage. Its primary goal is to provide an unambiguous encoding of the content of plaintext, ultimately covering all languages in the world. UTF-8 is an encoding of Unicode (ISO10646) into 8-bit characters where the first 128 are the same as ASCII. For most texts that userelatively few non-ASCII characters (texts in most Western languages), the encoding is veryspace-efficient because it will require only slightly more than 8 bits per character.

The context history XML structure will use timestamps as keys to identify specific nodes in thedatabase. A timestamped node will contain all the context and references to content obtained inthis time instance. There are two main categories; a context category and a content category. Thecontext category is now built up out of the primary context instances in the remaining four contextdimensions. Under these context instances are the related, inferred and referred context entries.The content category is split up into the main content classes; textual, aural, visual and executablecontent. Each class can have several types on content. Using this structure will allow the orderingof all context and content information into a logical structure that can be easily extended andsearched. The context dimensions are all identified by unique identifiers allow content to be orderedand traversed in different directions. The same goes for content where the content classes can beused as directions of navigation.Figure 3-5 shows the XML structure of an example (partial) context history.

3.4.3 Design Constraints

This section specifies design constraints that can be imposed by other standards, hardware limita-tions etc.Battery constraints. The system should function at all times. This will require a lot of batterypower. This might not be feasible for some classes of portable devices until further improvementsin the field of battery technology is made.

Standards Compliance

This section specifies the requirements derived from existing standard or regulations. The mobilememex depends of the following standards:

• JAVA: The Java language is an object-oriented programming language created by Sun Mi-crosystems. JAVA is object-oriented, platform independent, contains language facilities andlibraries for networking and is designed to execute code from remote sources securely.

63

C:\m3x\contexthistory.xml 10/20/03 14:37:52

©1998-2002 Altova GmbH http://www.xmlspy.com Page 1Registered to martijn (pink)

XMLversion 1.0

encoding UTF-8

Comment created by R.M.Vaandrager for demonstration purpose only.Comment M3XML context history

DOCTYPE contexthistorySYSTEM "m3xml.dtd"

contexthistoryuserid 3ffe:80e8:d8::1

timestamp (2)timeid sensorid context content

1 200309251537... 3ffe:80e8:d8::2 context contentspatial aural

location voicecallsensorid 3ffe:80e8:d7::2 sensorid 3ffe:81e5:c2::2Text LON037.400LAT122.018ALT014.0... callerid

orientation name Jungle Jane

sensorid 3ffe:80e8:d7::2 phonenumber +312055501312

Text 181 file 20030925153710.voicecall.ogg

existential keywordstemperature sensorid 3ffe:80e8:c3::2

sensorid 3ffe:80e8:d8:12a2:7 Text

vacation,plane,money,week,appartment,beach,baggage

Text 27

functionalactivity

sensorid 3ffe:80e8:d7::5Text running

2 200309251537... 3ffe:80e8:d8::2 context contentspatial visual

location imagesensorid 3ffe:80e8:d7::2 sensorid 3ffe:80e8:c2::2

Text LON037.399LAT122.018ALT014.0... file 20030925153710.image.jpg

temporal annotationseason sensorid 3ffe:80e8:c1::2

sensorid 3ffe:80e8:d7::13 Text My not so new car

Text Fall textualemail

sensorid 3ffe:80e8:12::1

sendername John Doe

emailaddress jdoe@email.me

subject RE:RE: Here's Johnny!

file 20030925153710.email.utf8

keywordssensorid 3ffe:80e8:c1::2Text

John,work,call,email,open,late,tomorrow

Figure 3-5: The XML structure of an example context history

• XML: XML (Extensible Markup Language) is a W3C Recommendation for creating special-purpose markup languages. It is a simplified subset of SGML, capable of describing manydifferent kinds of data. Its primary purpose is to facilitate the sharing of structured textand information across the Internet. Languages based on XML (for example, RDF, SMIL,MathML, and SVG) are themselves described in a formal way, allowing programs to modifyand validate documents in these languages without prior knowledge of their form. XML iscompatible with web and internet protocols, simultaneously human- and machine-readable,supports Unicode, able to represent general data structures (records, lists and trees), and isself-documenting in that it describes the structure and field names as well as specific values.

• XML-RPC is a remote procedure call protocol encoded in XML. It is a very simple protocol,defining only a handful of data types and commands. The XML-RPC protocol can be usedfor simple web services.

• XSLT is a XML transformation language, which transforms documents in XML format. Totransform in this context means to take all data or part of it (Query of a selection with

64

XPath) and create another XML document or a document in a format which can directlybe used for displaying or printing (e.g. an HTML, RTF or TeX document). In particularthe transformations involve: adding constant text like HTML document type and headerinformation, moving text and sorting text.

• Web Service: A Web Service is a string-based service (i.e. functionality) provided over theInternet (Port 80), usually in a data language, meant as communication between computersoftware.

• RSS: Really Simple Syndication or, more commonly, RSS 0.9x and 2.x (the latter representsan attempt to stabilize the standard), is a Web syndication protocol used primarily by newssites and Weblogs. RSS is essentially a web service used for syndicating articles on Websitesand Weblogs in a ”latest news” manner.

• Unicode: The Unicode Standard defines the universal character set. It is a means to assigna unique integer for all characters used by humans in written language. It’s primary goalis to provide an unambiguous encoding of the content of plain text, ultimately covering alllanguages in the world. UTF-8 is an encoding of Unicode (ISO 10646) into 8-bit characterswhere the first 128 are the same as ASCII. For most texts that use relatively few non-ASCIIcharacters (texts in most Western languages), the encoding is very space-efficient because itwill require only slightly more than 8 bits per character.

• PNG (RFC 2083): PNG stands for Portable Network Graphics. PNG is a bitmap image formatthat uses a non-patented lossless data compression method known as deflation.

• OGG (RFC 3533): The Ogg bitstream format has been created as the framework of a largerinitiative aimed at developing a set of components for the coding and decoding of multime-dia content which are both freely available and freely re-implementable in software. OGGaudiocodecs feature both lossless and lossy compression of audiostreams and are optimizedfor encoding and archiving of voice data, general audio data and high-fidelity audio data.The OGG audiocodecs are free from the licensing or patent issues raised by other proprietaryformats such as MP3.

• SVG: Scalable Vector Graphics (SVG) is an open standard and a language for describing two-dimensional static and animated vector graphics in XML. SVG allows three types of graphicobjects: vector graphic shapes, images and text. The standard also allows for vector graphicsanimation via scripting.

3.4.4 Software System Attributes

There are a number of attributes of software that can serve as requirements.

Reliability

This section specifies the factors required to establish the required reliability of the system.

Reliability depends on simplicity, uniformity and separation. Simplicity is achieved by maximizinguniformity by using only open standards:

1. A standard platform: JAVA which is well tested and proven to work on a multitude of com-putational devices with a widely different system resources.

2. A standardized messaging and document format: XML. XML is also used for authentication,encryption, normalization and abstraction if possible.

65

Separation: A peer to peer network architecture: If one node disappears or goes down it will notaffect the system as a whole. If there are not enough device in the mobile memex to function prop-erly devices will cache the acquired information until it can be published.

The mobile memex can have many uses depending on the applications. Some of these may be morecritical than others when it comes to reliability. Augmented medication for instance will require avery high level of system reliability. In these cases the application should dictate the devices neededfor this application instead of the other way around.

Availability

This section specifies the factors required to guarantee a defined availability level for the entire sys-tem.

Availability of devices depend on the functionality the user decides to carry around at a specifictime. M3X devices that are added to the users body are network automatically register in the ad-hoc network and become available to the mobile memex peergroup.

The context history and it’s content repository should be available over a WAN network interface.Either by VPN connection or encrypted internet connection. An access device is a device that canparse XML code and display the result in a browser application.

Security

This section specifies the factors that would protect the M3X software from accidental or maliciousaccess, use, modification, destruction or disclosure.

• Cryptographic techniques:

Use the Rijndael Advanced Encryption Standard (AES) (128 bit keys) and Secure SocketLayers (SSL) for secure communication both between M3X devices on the body and betweenaccess devices and the M3X system.Rijndael’s combination of security, performance, efficiency, ease of implementation and flexi-bility make it an appropriate selection for the Mobile Memex. For example Rijndael can beimplemented on devices with very low processing and memory capacities such as a javacard.Rijndael AES has the potential to remain secure well beyond twenty years.

• Restricted communication areas:

The user can define the general security and privacy settings in the generic system preferenceprofile. This profile is used to infer restricted communication areas because they do not meetthe requirements of the active security level.

• Data integrity checks:

Data integrity checks can be performed using cycle redundancy checks (CRC).

Maintainability

This section specifies attributes of software that relate to the ease of maintenance of the M3X soft-ware itself.

66

Define M3X User Profile Administrator

Available DevicesBody (BAN)

Watch <operation: control, monitor> <profile: spec, pref>CommunicatorDigital Camera

Proximity (PAN)DVDHDTV

Local (LAN)Remote (WAN/Internet)

Available Sensors

Available Actuators

Available ApplicationsPersonal Search EngineActive CoachingNavigation

Figure 3-6: Mobile memex administration UI

Figure 2-14 shows a concept administration interface of the mobile memex.Maintainability of the system is achieved by the use of profiles:

• There are two types of profiles: 1) Specification profiles and 2) Preference profiles. In thesystem there are two entities that contain profiles: 1) Devices, 2) Applications.

• Device specification profile = Sensor specification profiles + Actuator specification profiles +Resource specification profiles.

• System specification profile = Device specification profiles + Application specification profiles.

• System preference profile = Device preference profiles + Application preference profiles

The user can set a general system preference profile that contains settings that apply to all devicesand applications in the system. Settings that can be found in this profile are:

• security level: highest - high - normal - low - no security (default: high)

• external sharing level: share all - share specific - don’t share (default: don’t share)

• peergroup connection: allow all - allow specific - don’t allow (default: allow all)

• sensor aggregation: all - specific - none (default: aggregate all)

• actuator aggregation: all - specific - none (default: aggregate all)

Portability

This section specifies attributes of software that relate to the ease of porting the M3X software toother devices and/or operating systems.

Portability is a very important requirement because it is paramount the system has to be able torun on a myriad of personal devices. Portability is achieved by using common and open standardswhere possible:

• JAVA - Sun JAVA is a proven portable language. Platform independent code called bytecodecan be executed by a JAVA VM. M3X device should support at least the JAVA 2 Micro Edition(J2ME) for the Connected Limited Device Configuration (CLDC) Profile.

67

• JXTA - JXTA is a standardized and open set of peer to peer communication protocols with aJAVA reference implementation. Messages are encoded in Unicode XML to ensure maximumcompatibility / portability.

• XML - The eXtendible Markup Language (XML) is used for interdevice messaging. A specialXML format for sensors in the M3X is used called M3XML. The M3XML files are encoded inthe Unicode character set to ensure maximum compatibility and international standardization.

3.4.5 Other Requirements

Other relevant requirements characteristics not discussed in this chapter are:

• Reusability

• Testability

• Usability

• Availability

Functional Hierarchy

The overall functionality can be organized into a hierarchy of functions organized by either commoninputs, common outputs or common internal data access.

Hierarchy of functions organized by inputs:

1. M3X publisher

(a) input: Raw sensor data (can be divided into raw content and raw context).

(b) output: Context information (M3XML document), timestamped content

2. M3X subscriber

(a) input: Context information (raw context data embedded in a self-describing XML format)and content files (timestamped content).

(b) output: Partial context history and aggregated content files

3. M3X storage

(a) NXD input: Partial context history (aggregated context information based on timestamp)

(b) FS input: Aggregated content files (filename:timestamp extension:content type)

(c) output: XML context history nodes including content references matching XPATH query.

4. M3X access

(a) input: User input

(b) output: NXD XPATH query

Figure 2-14 shows I/O functions per M3X device.

68

Physical Sensor

Context Sensor

Content Sensor

Interpreter

Publisher

Raw context data

Raw content file

M3Xencapsulation

TimestampTimestamped content

Context informationRaw sensor data

Aggregator

Context aggregator

Content aggregator

Local User Applications

Subscriber

Partial context history

Aggregated content Aggregated Content

Partial context history

Storage

NXD

Filesystem

Services

Normalize, AbstractExtract, Annotate,

Associate, Administrate

User Applications

Remote

Local

USER

Query Context

Content

Content

Context

SENSOR

Req.

Pro-active

Realtime

Context information

Timestamped content

Aggregated Content

Partial context history

Figure 3-7: I/O functions for publishing, subscribing and storage devices

69

70

Chapter 4

A Proof of Concept application

To demonstrate the mobile memex framework and it’s architecture a proof of concept applicationwas developed. This application is a base-line architecture implementation supporting the personalcontextual search engine. The search engine is a remote-application that allows the user to formulatea search query in order to retrieve matching contextual information.

4.1 The Personal Contextual Search Engine

The success of search engines like Google have demonstrated that users have become accustomed toretrieving information by formulating a search query. Modern desktop operating systems still seemto rely on hierarchical file systems where the user manually creates a directory with a logical nameand places his or her information there. Some applications have their own methods of storing theinformation on the files system and the result is that the process of storing and retrieving content isa very time consuming and confusing process. As the years pass users will have to cope with moreand more information that needs to be ordered and stored in a logical structure. It makes sense todevelop a system that provides the user with a Google like search interface to find his/her personalinformation with a simple query instead of relying navigation through numerous trees of folders andfiles.

To create a functional search engine that can be used by an individual to retrieve content is becomesnecessary to index the content based on different contextual characteristics. On desktop computingsystem the only contextual characteristics available in the file system are attributes such as the dateand time of creation, owner, filename and file extension. As computing moves away from the desktopand into the ambient environment suddenly a lot more context becomes available that can aid theuser in retrieving the relevant information. Context such as location, activity, identity, orientation,elevation and many more will all increase the probability that an accurate and effective search querycan be constructed by the user. Figure 4-1 lists some sample questions that could be formulated asqueries.

Notes on the proof of concept implementation:

• The prototype implements the physical layer by simulating sensor devices, the communicationlayer is based on JAVA as a programming platform, JXTA will function as a peer to peernetwork platform, and XML as the context document and message format.

• In the prototype there are a number of simulated devices: two publishing devices, one sub-scribing device, one storage device containing the Native XML Database and one access device.

• The publishing and subscribing devices have got a preference profile containing device specificsettings such as the polling interval. This profile is read and processed.

71

- Give me a chronological overview of all images I took during my last stay in Japan.

- Give me a list of all names including latest email address and telephone number of the people I met this year.

- What were the 5 most exiting events in my lifetime?

- What is the total time I spent in France?

- Who has called me most often last two weeks?

- What and where is the furthest place I’ve been from location x? the highest place?

- Show me all the messages I received last Monday

- Give me a list of e-mails related to “Jane’s wedding”

- Show me the location and time when I received the last call from bob.

- How many times have I been to café x? are there any images I took there in the past?

Figure 4-1: Some sample questions a user might ask the information system

• The publishing devices support the acquisition of (virtual) sensor data, and the interpretationand encapsulation of this data into an XML context document.

• The subscribing device receives the context documents from the different publishing devicesand updates the native XML database.

• The native XML database is remotely accessible by the access device using a web-browser. Thestorage devices contains the NXD, a web server supporting JAVA servlets and a WEBDAV asthe remote application user interface that supports XPATH queries.

Storage

Subscriber

Publisher SensorJXTA

JXTA

Xindice XPath Xincon

Access Browser

Figure 4-2: Overview and location of software components in the prototype

4.2 A base-line architecture

Figure 4-3 shows the base-line architecture of the mobile memex. It is a stripped down version ofthe full architecture described in the chapter 3. The base-line architecture was derived from the fullarchitecture by applying the MoSCoW method. It only incorporates the most essential requirementsthat need to be fulfilled in order to construct a distributed sensor aggregation network such as themobile memex.

The list below shows the MoSCoW evaluation used in the construction of the base-line architecture:

• Must have:

– in the physical layer: Sensors, Actuators and System resources

72

NXDContext History

Query interfaceWEBDAVremote application interface

update

update

Aggregation Engine

Access Device Storage Device

Subscriber Device

Publishing Device

Publishing Device

Acquisition & InterpretationEngine

Acquisition & InterpretationEngine

Environment

User

XML:DB interface

M3X device profiles

Webbrowser

XINCON,TOMCAT,APACHE

XPATH

JAVA,JXTA

XINDICE

Figure 4-3: Overview of the base-line architecture

– in the communication layer: Network interface supporting TCP/IP, a P2P networkingtoolkit

– in the information layer: Device profiles, XML document structure, File System, NativeXML Database.

– in the application layer: FS API, NXD API, Context History Query interface, Remoteapplication support.

– in the augmentation layer: the personal contextual search engine.

• Should have:

– In the physical- and network layer: Network technology based on Dermal conduction.

– in the information layer: Support for context super types: IPv6, UTC24, WGS84, OEDVerbs, OED Nouns, support for common context encoding: UTF-8

– in the application layer:

∗ Application repository containing:

· applications

· application specification profiles

· application preference profiles.

∗ Template repository containing XSLT transformation profiles for:

· normalization

· abstraction.

∗ In the augmentation layer: System management application to manage: applications,profiles, preferences, monitoring and external resources.

• Could have:

– Template repository containing:

∗ XUL or templates for device specific application user interface support

∗ XSLFO formatting profiles for device specific presentation of information.

• Won’t have:

73

– Knowledge base required by inference sensors. XSLT abstraction templates for individualcontext type transformations. Search engine query language based on natural language.

Only the ”Must-haves” are implemented in the proof of concept, here we provide some implemen-tation details according to the individual layers of the mobile memex framework:

• In the physical layer: Hardware devices and sensors are simulated by programs that mimictheir behavior. Implemented are the publishing, subscribing, storage and access device.

• In the communication layer: The platform used in this proof of concept application is JAVA.On top of the platform is JXTA; a collection of peer 2 peer network protocols and services.XML is used as the inter-device messaging language.

• In the information layer: Implemented are simple acquisition of sensor data generated by avirtual sensor, encapsulation of the sensor data into a context document containing both sensordata and meta information about the sensor encoded in XML, and the aggregation of thesedocuments into the context history NXD.

• In the application layer: Implemented are the interfaces to the NXD including the XML:DBand the XPATH query language. Remote applications are supported through servlets inApache/Tomcat. Services are not implemented, although they will be crucial to ensure thata wide range of applications can be supported. As we will only focus on one application - thepersonal contextual search engine, services need not to be supported in this case.

• In the augmentation layer: Implemented is a simple personal contextual search engine. Thisapplication is an instance of the augmented memory class of applications. The search engineis a remote application that can be accessed by any access device with a web browser.

Figure 4-4 shows the basic software component structure and processing sequence in the proof onconcept application.

Ne NetPeerGroupM3Xtest

P1

P2

S1

1

2

34

5

NXD

6

Figure 4-4: Software components in the proof of concept.

4.3 User Interface Design

The personal contextual search engine is a reactive application that operates on the entire archivedcontext history. The user will have to pull the required information out of the system by means ofsearch queries. The personal contextual search engine is well suited for use as a remote application.

74

The user interface of the search engine application was designed to be simple and scalable. Simplicitywill improve useability and also speed, as the size of the interface can be small which in turn reducesbandwidth requirement. Scalability will allow the interface to be displayed on devices with a rangeof different display resolutions. Google was chosen as a reference as many users are familiar with itand because of its minimal but very functional design.The user is first presented with a query window. After the user has sent the query the retrievedXML nodes will be presented in the result window. In this window the user can browse the resultsand formulate a new query. Figure ?? (in appendix A) shows the UML sequence diagram for thepersonal conceptual search engine.

Figure 4-5 shows both the search and result windows of the search engine application.

-Type(All)

-Location(Japan) –Date(2001)

Search

Media

Context

Advanced search options

Searched the M3X for all media created in Japan in 2001 Showing 12 out of 26 results

type g date g location g

SearchXPath query:

Digital movie: Okinawa beach 21:30, Trip to Okinawa, 20 May 2001

Phone call: Jungle Jane, Okinawa beach 21:30, 20 May 2001

Received email: Re: barbecue From: Peter Pan, 21:30, 20 May 2001

MSN Chat log: Willy Wonka, 11:37, 21 May 2001

Digital image: Etc.

(a) (b)

Figure 4-5: This Figure contains two panels: (a) showing the search window, and (b) showing theresult browser

4.4 Prototype Walkthrough

A test run according to the setup mentioned in the first section, yields a NXD with a sequence ofsimple context entries. The context entries are not ordered in a context history yet. Instead everynode is identified with a unique key. The database can be accessed by the access device that is ableto run the remote application based on a JAVA servlet and with a graphical user interface modeledloosely after the concepts mentioned in the above section. This section will walk through the devices,their software components and tasks in the order they are executed. This sequence of tasks will leadto the availability of the contextual information to any access device capable of browsing the web.Figure 4-6 shows the UML class diagram for the M3X devices publisher and subscriber. The storageand access devices can operate outside of the M3X peergroup. For a more detailed overview of theclass diagram please refer to Figure A-2 in Appendix A.

1. Storage start.First the storage device is started. This will trigger the execution of the database and thewebserver and components. In this prototype the storage is not a part of the peergroupbecause it is not a body area network device. The storage device is accessible by TCP/IP.After initialization the Xindice native XML database is now accessible through the XML:DBinterface at: XMLdb:XINDICE:////db/m3x on port 4080, Where ”db” is the name of thedatabase and ”m3x” the name of the collection. Also, the personal conceptual search engine is

75

checkAdv()

«interface»FindValidate

Publisher() init() process()

Publisher

M3xJXTAPeer() joinGroupIfExists()

M3xJXTAPeer

Subscriber() aquireData() createInputPipeAdvIfNotExist() init()

PipeAdvValidator

Subscriber

mdidgen() printIt() startJXTA()

mdidgen

Figure 4-6: UML class diagram showing the JAVA classes for the M3X subscriber and publisherdevices

available on: http://localhost:8080/m3x. Figure 4-7 shows the output created by the Xindicedatabase after initialization.

Figure 4-7: Initializing the Xindice NXD

2. Subscriber start.Next the subscribing device is activated. This device is responsible for creating the JXTAmobile memex peergroup and the aggregation of any published data. To achieve this, a numberof tasks will be executed in the following sequence:

(a) Initialize NXD (XINDICE).First the subscribing device will try to initialize the Xindice native XML database runningon the storage device by using the XML:DB API as shown in the code listing below:

// create and register the Xindice database driver

System.out.println("Initializing Xindice NXD...");

try {

Class c = Class.forName("org.apache.xindice.client.xmldb.DatabaseImpl");

Database database = (Database) c.newInstance();

DatabaseManager.registerDatabase(database);

col = DatabaseManager.getCollection("xmldb:XINDICE:///db/m3xdb");

} catch (XMLDBException e) {

76

System.err.println("XML:DB Exception occurred " + e.errorCode);

}

If successful the subscribing device is now able to add XML documents to the collection.

(b) Read profile (subscriberprofile).The device will read the device specification and preference profile. In this case it willsimply list the name of the peergroup to be created: the M3Xtest peergroup. Thispeergroup will contain all mobile information devices in the users body area network.

// read and process profile

System.out.println(

"Reading profile: [" + ConfigFileName + "], and processing...");

try {

inFileStream = new FileInputStream(new File(ConfigFileName));

myProp = new Properties();

myProp.load(inFileStream);

} catch (Exception ex) {

System.out.println("Profile access problem - file not found.");

}

if (myProp != null) {

groupName = myProp.getProperty("group", "");

System.out.println(" * Group to join will be: " + groupName);

}

(c) Initialize Peergroup (JXTA peergroup: M3Xtest).Once the profile has been processed the device can now join a peergroup. Peers selforganize into groups called a peergroup. Peergroups serve to:

i. Define a set of services and resources

ii. Provide a secure region

iii. Create scoping

iv. Create a monitoring environment.

There are two kinds of peergroups: 1) NetPeerGroup and 2) User peergroups. Users (andapplications) can create new peergroups with their own set of customized services andmembership policies. The M3Xpeegroup belongs to this category. All user peergroupsare subsets of the NetPeerGroup. All peers are automatically members of the NetPeer-Group. In the case that the peergroup does not exist the subscribing device will createone including it’s corresponding peergroup advertisement informing other devices of thepeergroup existence.

// start JXTA and join or create the M3X peergroup.

PeerGroup tpPG = null;

System.out.println("Joining default NetPeerGroup...");

try {

// default JXTA peergroup: NetPeerGroup

group = PeerGroupFactory.newNetPeerGroup();

// mobile memex peergroup: M3Xtest

System.out.println("Attempting to join group: " + groupName);

if (groupName.length() != 0) {

try {

tpPG = joinGroupIfExists(groupName);

77

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

if (tpPG != null) {

group = tpPG;

System.out.println(

"group " + groupName + " joined successfully");

}

}

} catch (Exception e) {

System.out.println("Err : group creation failure");

}

The NetPeerGroup defines the initial scope of operations of a peer and the default servicesit provides. There are seven core peergroup services: Discovery, Membership, Access,Pipe, Monitoring, Resolver and Rendezvous. If the M3Xtest peergroup has successfullybeen joined, the subscriber will request it’s services such as the discovery service and thepipe service:

// get the discovery and pipe services from the peergroup.

if (group != null) {

discovery = group.getDiscoveryService();

pipes = group.getPipeService();

}

After obtaining access to the discovery and pipe service, the subscribing device will seeif there are any existing pipe advertisements. If there aren’t any existing advertisementsit will create one.

In JXTA there are six different types of advertisements: Peer advertisements, peergroupadvertisements, pipe advertisements, service advertisements, content advertisements andendpoint advertisements.

// check for existing pipe existing advertisements

Advertisement padv = null;

if ((padv = findPipeAdv()) != null) {

System.out.println("Found previously published pipe advertisement");

return (PipeAdvertisement) padv;

}

System.out.println("Previously published pipe advertisement not found");

If a previously published pipe advertisement is not found, a new one has to be created.Next the subscriber device will try to find a previously published module specificationadvertisement. The ModuleSpecAdvertisement class is used for specifying a module’sexpected on-wire behavior and protocol implementation. The MSA associates a uniqueModuleSpecID to each module specification.

JXTA code uses the module specification ID to identify a particular network-compatiblefamily of implementations of a given module. The MSA has two purposes; 1) to publishthe URI (universal resource identifier) of its formal specs for developers, and 2) to publish

78

the means of remote access to the module’s services if that is appropriate. First thesubscriber will try to find the MSA locally, if this fails it will search remotely, and if thisalso fails a new MSA will be created. The basic code listing is shown below. For detailedinformation on findModuleSpecAdv() and publishModuleSpecAdv() please refer to thesource code file ”M3xJXTAPeer.java”.

// check for the module specification advertisement (MSA)

System.out.println("Checking to see if MSA previously published...");

if (findModuleSpecAdv() == null) {

System.out.println(".. MSA not found, need to create it");

publishModuleSpecAdv(ServicePipeAdv);

} else

System.out.println(".. found previously published MSA");

(d) Create input pipe (JXTA pipe: inputpipe).

Once the M3Xpeergroup is created and joined by the subscriber device it will create aninput pipe to which publishing devices can publish data.

A pipe is used to connect the output from one command to the input of another, similarto the Unix pipe system. JXTA Pipes are defined in terms of the endpoints available toa peer. A peer endpoint is a logical abstraction of an address on a network transportthat is capable of sending and receiving network messages. Pipe endpoints are dynam-ically bounded to a peer endpoint at runtime via the Pipe Binding Protocol (PBP).Pipes are uniquely identified by a pipe advertisement which contains a unique pipe IDand an optional pipe name. Pipe advertisements are a resource within the JXTA network.

There are two types of pipes: 1) Point-to-point and 2) Propagate.A propagate pipe connects one output pipe to multiple input pipes. A message sent overa propagate pipe is sent to all listening input pipes.

The code listing below shows the creation of a subscribing pipe:

// create subscribing pipe

try {

System.out.println(

"Creating subscribing pipe for data collection...");

ipipe = pipes.createInputPipe(ServicePipeAdv);

Thread.sleep(5000);

} catch (Exception ex) {

System.out.println("Subscribing pipe creation failed.");

return;

}

System.out.println("Subscribing pipe created successfully");

If a previously published pipe advertisement is found the pipe already exists and this stepcan be skipped. Else a pipe advertisement is also created to inform other devices in thepeergroup of it’s existence. The corresponding code listing is shown below:

79

// create a new input pipe advertisement

PipeAdvertisement pipeAdv;

pipeAdv =

(PipeAdvertisement) AdvertisementFactory.newAdvertisement(

PipeAdvertisement.getAdvertisementType());

pipeAdv.setName(servicePipeName); // M3XSubscriberPipe

pipeAdv.setType(servicePipeType); // PipeService.PropagateType

PipeID tpID;

try {

tpID = (PipeID) IDFactory.fromURL(new URL(servicePipeID));

} catch (Exception ex) {

tpID = IDFactory.newPipeID(group.getPeerGroupID());

}

pipeAdv.setPipeID(tpID);

System.out.println("Locally caching the new pipe advertisement...");

// publish the pipe advertisement

try {

discovery.publish(pipeAdv, DiscoveryService.ADV);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

return pipeAdv;

Now the subscribing device is ready to receive any published data.

3. Publisher startIf the storage device and the subscribing device are up and running it is time to add one ormore publishing devices. A publishing device has three main tasks:

(a) Acquire sensor data.

(b) Encapsulate the sensor data into an XML document.

(c) Publish the document.

The following sequence demonstrates how a publishing devices performs the above mentionedtasks.

(a) Read profile (publisherprofile)The publisher will read in it’s corresponding device specification and preference profile. Inthis case a file called ”publisherprofile”. This profile contains a number of fields includingthe peergroup to join, the name of the device, the type of sensor and it’s polling interval.An example profile is shown below:

group=m3xtest

device=Casio Sportswatch Z2

type=Ambient Temperature

interval=10000

The code for reading the profile is shown below and is similar to the correspondingsubscribing device code.

// read publisher profile

System.out.println(

"Reading profile: [" + ConfigFileName + "], and initializing...");

try {

inFileStream = new FileInputStream(new File(ConfigFileName));

myProp = new Properties();

80

myProp.load(inFileStream);

} catch (Exception ex) {

System.out.println("Profile access problem - file not found.");

}

if (myProp != null) {

groupName = myProp.getProperty("group", "default group");

System.out.println("Group to join will be: " + groupName);

pubDevice = myProp.getProperty("device", "default device");

System.out.println("Publishing device is : " + pubDevice);

sensorType = myProp.getProperty("type", "default type");

System.out.println("Sensor type is : " + sensorType);

sensorInterval = myProp.getProperty("interval", "default interval");

System.out.println("Measurement interval : " + sensorInterval + "ms");

}

(b) Register in peergroup (JXTA M3Xtest)When the profile is processed the device will know the peergroup to join. In this casethe mobile memex peergroup is called ”m3xtest”. The joining process is identical to thesubscriber device joining process described earlier.

(c) Acquire data (Virtual Sensor)Now the publishing device is part of the M3X test peergroup it can start the acquisition ofsensor data. It does so at regular time intervals as defined in the publisher profile. In thecase of this proof of concept application the sensor is simulated by a function that simplygenerates random integers in the range between 0 and 38. These values will represent theambient temperature in degrees Celsius. The code is shown below:

// acquire sensor data (random integer in range 0-38)

Random rn = new Random();

int n = 38;

int sensorData = rn.nextInt(n + 1);

(d) Encapsulate data (M3XML)Every time the publishing device acquires new sensor data it will encapsulate this datainto an XML document containing some meta-information about this data. The XMLdocument structure used in the proof of concept is an oversimplification of the recom-mended structure as described in Chapter 3. The code below shows the encapsulation ofsensor data into an XML document.

// create and return the collected data as an XML string

return "<sensor><device>"

+ pubDevice

+ "</device><type>"

+ sensorType

+ "</type><data>"

+ sensorData

+ "</data></sensor>";

}

(e) Publish data (JXTA output pipe (propagation pipe))The XML sensor document is stored in the variable collectedData. This data can now

81

be sent over the output pipe to any subscribing device available in the M3X peergroup.The XML document containing the sensor data will now be encapsulated into a JXTAmessage. The code listing below shows this process:

// send collected data to output pipe

if (collectedData != null) {

System.out.println("... publish sensor data ....");

sendMsg = pipes.createMessage();

sendMsg.setString(PUBLISHER_DEVICE, pubDevice);

sendMsg.setString(DATA_COLLECTED, collectedData);

try {

opipe.send(sendMsg);

} catch (Exception ex) {

System.out.println("Err: " + ex);

break;

}

}

Figure 4-8 shows the publisher console and the output produced by the application:

Figure 4-8: Publish: The publisher console

4. SubscriberNow it is the subscribers turn again. After it’s initialization at the beginning of this walk-through, the subscriber waits for any published messages to arrive on it’s input pipe. Becausea publishing device has just published an XML document containing sensor data it can nowstart the process of receiving and aggregating this data.

(a) Receive data (JXTA input pipe) Once a message has been received from a publisher it isanalyzed and processed. The following code shows the listening and receiving process ofthe subscribing device:

// Acquire the sensor data and send it over for processing

if (ipipe != null) {

System.out.println("Waiting for data from publisher...");

while (true) {

try {

imessage = ipipe.waitForMessage();

} catch (Exception ex) {

System.out.println("Err: " + ex);

82

break;

}

if ((imessage.hasElement(PUBLISHER_DEVICE)

&& imessage.hasElement(DATA_COLLECTED))) {

sensorData = imessage.getString(DATA_COLLECTED);

System.out.println("Received data from publisher: "

+ imessage.getString(PUBLISHER_DEVICE));

System.out.println("Data received is: " + sensorData);

processData();

}

}

}

(b) process data (XML:DB)If the subscriber succeeds in extracting the sensor data XML document from the JXTAmessage, it can finally be added to the mobile memex collection of the native XMLdatabase. The document collection is not a real context history as there are no timestampsin our case. Instead every XML document will be assigned a unique ID (UID). The codelisting shows how the subscribing device stores the XML documents into the NXD of thestorage device.

// Store the M3XML document in Xindice NXD

try {

XMLResource document =

(XMLResource) col.createResource(null,"XMLResource");

document.setContent(sensorData);

col.storeResource(document);

System.out.println ("processing data in Xindice NXD...");

System.out.println ("M3XML document inserted in Xindice as: "

+ document.getId());

}

catch (XMLDBException e) {

System.err.println("XML:DB Exception occurred "

+ e.errorCode + " " + e.getMessage());

}

Figure 4-9 shows the subscriber console and the output produced by the application:

Figure 4-9: Subscribe: The subscriber console

83

5. StorageThe storage device is the device with the most abundant system resources. This device willtherefore host the native XML database, the content repository and most applications. In theproof of concept applications we only implement the NXD. Content can easily be added to adirectory on the file system that serves as the content repository but as there is no contentdata generated by the virtual sensors this possibility will not be used. The storage device isresponsible for the storage of data and the interfaces to access it:

(a) Store data (NXD)Contextual data is stored in the Xindice NXD. Content data is stored on the local filesystem (FS) of the storage device.

(b) Query data (XPATH)The primary application interface is the XPATH query language that can be used to selectnodes in the M3X context collection. An XPATH query that selects all sensor nodes in thecontext collection for example will look like this: XMLdb:XINDICE:////db/m3xdb//sensor.The XMLdb statement tells the application to use the XML:DB interface, Xindice is theXML DB instance. The Xindice NXD contains a database called ”db” that containsthe context collection ”m3x”. From this collection we will now select all nodes with the”sensor” tag. Figure 4-10 shows the content of the Xindice NXD on an XPATH query:

Figure 4-10: Store: View of nodes in the Xindice M3Xdb

(c) Host remote application (APACHE/TOMCAT/XINCON)Not all applications are local applications. Because remote applications will be an impor-tant part of the mobile memex, the proof of concept application will itself be a remoteapplication. Consequently the proof of concept has to support remote applications. Thissupport comes is provided by the Apache HTTP Server combined with the Tomcat servletengine. Every remote application is a servlet. In this case it is a WebDAV called ”XIN-CON”. WebDAV stands for ”Web-based Distributed Authoring and Versioning”. It isa set of extensions to the HTTP protocol which allows users to collaboratively edit andmanage files on remote web servers. XINCON offers direct access to the context historyon the storage device. In the prototype XINCON is partly modified to function as themobile memex search engine frontend. Figure 4-11 shows the initialization of Apache,Tomcat and the XINCON servlet:

6. Access deviceThe access device can be any device that supports a webbrowser and an active internet con-nection.

(a) access dataIn the proof of concept all programs run locally so in this case the personal contextual

84

Figure 4-11: Initialization of Apache/Tomcat and the XINCON servlet

search engine can be accessed by HTTP at the URL: http://localhost:8080/m3x. In thecase of remote access ”localhost” should be replaced with the IP of the host computer.Figure 4-12 shows the results of the XPATH query in the remote search application:

Figure 4-12: Access: View of nodes in the remote search application

The user can enter any XPATH query to retrieve subsets of context from the database.For the sake of this proof of concept, using XPATH as a query language will suffice. Inreality however, most user would like to use a query language that more closely resemblesspoken language. There should be an additional translation layer in the contextual searchengine that converts high-level user queries into XPATH queries.

4.5 Results

The result of this base-line implementation is a system based on virtual devices with specific roles ina peer-to-peer style network, capable of acquiring and aggregating contextual data into a structuredformat that can be accessed and searched remotely by the user or other applications. It is based onopen standards and protocols, such as the JXTA P2P toolkit that can be used to implement P2Pnetworking capabilities even on devices with minimal system resources, to XML that can be usedto embed the context data in a container of descriptive meta information.

The proof of concept application shows it is feasible to implement a peer to peer style of networking

85

in personal portable devices with limited resources. The publishing and subscribing of contextualinformation all functioned as expected. Note that this is a very simple application and it has beentested in optimal conditions. The storage device was always available and consequently there wasno need to implement context caching mechanisms to avoid data loss. The removal of publishingnodes did not affect system stability or functionality. The aggregation process only took place atthe storage device, so the step of partial aggregation by subscribing devices was skipped. Partialaggregation is only useful for local application running on the subscribing device and in case oftemporary caching of context. Neither was relevant for this proof of concept application.

Ideally the search engine should define a more user friendly query language that is able to map toa corresponding XPATH query. Currently the user has to understand the structure of the XPATHquery language to formulate meaningful queries. The search engine only supports retrieval of contextinformation, there is no content and it is not possible to annotate context or create associationsbetween context nodes. The virtual sensors in the prototype do not generate content and theprototype thus only supports context.

86

Chapter 5

Conclusions and Future Work

In this chapter we present conclusions that can be drawn from this thesis in retrospect and reccomenddirections in which future work might take place.

5.1 Conclusions

Chapter 1 described the original memex and it’s intended function. Taking into account currentdevelopments in information technology such as functional integration and digitalization of deviceswe posed the main research question which served as the motivation behind this thesis:

How can the ideas behind the original mechanical memex of 1945 be implemented in amodern and mobile computational information system based on the upcoming ambientenvironment of 2010?

This research question is split up into four sub-questions, regarding:

1. Composition: What hard- and software is the mobile memex composed of?

2. Function: How will the mobile function, and what is the role of content and context in thesystem?

3. Augmentation: How can the system be used to support applications that augment the user,and what kind of augmentation applications are possible?

4. Implication: What kind of social and ethical implications does a system such as the mobilememex have in terms of privacy and security?

Chapter 2 proposed a 5 layer conceptual framework which answers these research questions:

• In the physical layer we defined and classified the types of devices that compose a mobilememex. Also defined and classified are the sensors, actuators and system resources availableby devices and how they can be used by the system.

• In the communication layer we proposed a standardized platform, networking protocols, mes-saging format and middleware essential to ensure the inter-device and inter-component com-patability in a system as dynamic and heterogenous as the mobile memex.

• In the information layer we discussed how raw-sensor data can be processed into either contextor content. The concepts of context and content are defined and classified into dimensions,classes, types and instances. Also we illustrate how context can be structured into a contexthistory. Finally, we discuss the entire processing path from the acquisition by publishingdevices, the aggregation by subscribing devices, the archiving into a context history and contentrepository on the storage device, to the retrieving of it’s information by the access device.

87

• In the application and service level we discussed the types of applications that can operate onthe context and content, and the interfaces required. A number of core services were discussedthat further extend the completeness and correctness of the context history, and the uniformityof both the context history as well as the content repository. Services include normalizationdesigned to synchronize and translate context and content instances into a common notationand format.

• In the augmentation layer we discussed a number of possible context-aware applications thatwould operate on the context and content available in the mobile memex, in order to augmentthe user. These examples are illustrated with their respective use-cases.

Chapter 3 described a component architecture based on the conceptual framework. A system suchas the mobile memex calls for the use of open standards since the system is extremely heterogenouson both the level of hardware possible as well as software. Personal information devices can widelydiffer in terms of available sensors, actuators and system resources. Also there can exist many kindsof context and content. The system should be flexible enough to allow it to be extended with newclasses and types of context and content. The use of open standards in the mobile memex increasescompatability and expendability of the components in the system. We recommended a numberof standards for the mobile memex: 1) Programming platform based on a JAVA Virtual Machine(JVM), 2) A Peer-to-Peer (P2P) Networking toolkit based on JXTA, 3) Contextual documents arebased on XML encoded in Unicode UTF-8. 4) Contextual documents are stored in their native XMLformat by using a native XML database (NXD). 5) Content is normalized to open standards suchas PNG, OGG, ISO-MP4, SVG etc.

Chapter 4 demonstrated a proof of concept application; The personal contextual search engine. Theproof of concept implements a base-line architecture for the mobile memex derived from the compo-nent architecture in the previous chapter. The proof of concept application serves to demonstratethe feasibility of the system. The personal contextual search engine is a Google-like search enginethat enables the user to retrieve and navigate through annotated content by formulating queriesbased on context. Retrieved context and it’s related content can in turn serve as retrieval cues thatenable a user to easily retrieve past experiences from memory with a higher accuracy.

Throughout this thesis we have made a number of observations and assumptions that can now beused to summarize the ideas behind mobile memex:

• Converging trends in information technology will lead to information devices that are portable,personal, affordable and highly integrated.

• Portable information devices such as camera’s, mobile phones, watches, PDAs and smartcardscan contain sensors.

• Sensors can be used to obtain context and content, dependent of the encoding of the data.

• Portable information devices can be connected in a body area network that uses peer-to-peertechnology to automatically discover new nodes and share resources.

• The combination of the body area network and software that enables the acquisition, aggre-gation and archiving of context and content forms the core of the mobile memex system.

• Because the system is both personal and portable it shares the same context as it’s user.

• Context can serve as meta information to content.

• Meta information can be used to retrieve relevant content.

• Content can be navigated through by following ”trails of context”.

88

• The context classification used in the mobile memex proposes 5 context dimensions that whencombined will accurately describe a situation or state of an entity. Consequently these di-mensions are used for automated content annotation and also serve as the main directions fornavigating through content.

• Every context dimension contains a primary contextual supertype that can be uniquely iden-tified and serve to infer or refer new classes, types and instances of related context belongingto the same dimension.

• Context aware services can extend and enhance the completeness, correctness, uniformity andthus usefulness of both the context history as well as the content repository.

• Context aware applications can augment the user in various daily tasks such as augmentingmemory, navigation, communication and medication.

The original 1945 memex was a stationary and mechanical device designed to store, annotate, as-sociate and retrieve an individuals collection of documents. The mobile memex is a mobile andcomputational system that provides the same functionality as the original memex but extends it’suse to incorporate context as the associative ”glue” between the archived content. Applicationscan operate on the context and content to augment the user in performing daily tasks, and as theenvironment of the user becomes more and more ubiquitous it is possible to construct increasinglycomplete records of context and content which will improve the quality of augmentation that canbe provided.

5.2 Future work

This thesis presented only a rough sketch of what an information like this could look like and functionin a 2010 time frame. There remain many open questions that need to be answered and bottlenecksto be solved before the mobile memex can be realized in full. Some tasks that remain are:

1. Create a flexible open XML format for the description on sensor data, the XML contextenvelope should be flexible and allow for the introduction of new classes and instances ofsensors and actuators.

2. Improve BAN network technology, especially those based on magnetic induction and bodyconductivity as they fit the requirements of the mobile memex.

3. Standardized device platform. Devices will have to support a common platform includingmiddleware such as IPv6, JAVA (at least the J2ME CLDC profile) and JXTA .

4. Manufacturers should ship their devices with embedded device profiles (containing specificationURL) that can be used to describe context-data on the detail level required by the mobilememex.

5. The system requires lots of low-level and high-level knowledge. In order to identify activitiesand states there need to be a large knowledge base consisting of patterns and algorithms thatmap patterns to meaningful behavior.

6. As the context history grows, performance of the system might decrease. Examine how thesystem might be optimized to improve perfomance.

7. Examine how the system might be misused. What are the security pitfalls?

8. Examine how the relational context dimension can be formalized to represent annotations andassocations in a logical form that applications and services can operate upon.

89

90

Bibliography

[1] As We May Think - Vannevar Bush (1945).

[2] The Context Toolkit: Aiding the development of context-enabled applications - Salber, D., Dey,AK, and Abowd (1999).

[3] The Computer for the 21st Century - Mark Weiser (1991).

[4] A survey of context-aware mobile computing research - Guanling Chen and David Kotz (2001).

[5] Supporting Context-Aware Computing in Ad Hoc Mobile Environments - Qingfeng Huang.

[6] Context-awareness in wearable and ubiquitous computing - Gregory D. Abowd, Anind K. Dey,Gregory Abowd, Robert Orr & Jason Brotherton (2001).

[7] A Context-based Document System for Wearable Computers - Kent Lyons, Thad Starner,Lonnie Harvel.

[8] Book of Visions 2001 - Wireless World Research Forum (2001).

[9] Man-Computer Symbiosis - J.C.R. Licklider (1960).

[10] Turn Signals Are the Facial Expressions of Automobiles - Donald A. Norman (1992)

[11] The wearable remembrance agent: a system for augmented memory - Rhodes (1997)

[12] ”Forget-me-not” Intimate Computing in Support of Human Memory - Mik Lamming, MikeFlynn (1994).

[13] ”Memories for life” Managing information over a human lifetime - Andrew Fitzgibbon, EhudReiter (2003)

[14] The What, Who, Where, When, and How of Context Awareness - Morse, Armstrong, Dey(2000).

[15] Augmenting Human Intellect: A Conceptual Framework - D. C. Engelbart.

[16] ”Mnemonet” An Augmented Memory System - Nigel Shadbolt (2002).

[17] Wearable Computing and Contextual Awareness - Thad Eugene Starner.

[18] Lifestreams: An Alternative to the Desktop Metaphor - Scott Fertig, Eric Freeman.

[19] Affective Computing - Picard (1995).

[20] Time-machine Computing: A Time-centric Approach for the Information Environment - Reki-moto, J (1999).

[21] Seeking a Foundation for Context-Aware Computing - Paul Dorish.

[22] There is more to Context than Location - Albrecht Schmidt, Michael Beigl, and Hans-W.Gellersen.

91

[23] Situated Computing: Bridging the Gap between Intention and Action - Anatole Gerhman,Joseph McCharthy, Andrew Fano (1999).

[24] Context-awareness: some compelling applications - Peter Brown, Winslow Burleson, Mik Lam-ming, Odd-Wiking Rahlff, Guy Romano, Jean Scholz, and Dave Snowdon (2000).

[25] The Conference Assistant: Combining Context-Awareness with Wearable Computing - AnindK. Dey, Daniel Salber, Gregory D. Abowd, Masayasu Futakawa (1999).

[26] Context-Aware Computing - Thomas P. Moran, Paul Dourish.

[27] CybreMinder: A Context-Aware System for Supporting Reminders - Anind K. Dey and GregoryD. Abowd.

[28] Distributed and Disappearing User Interfaces in Ubiquitous Computing - Anind K. Dey, PeterLjungstrand, Albrecht Schmidt.

[29] Computers that Recognise and Respond to User Emotion: Theoretical and Practical Implica-tions - Rosalind W. Picard, Jonathan Klein.

[30] Personal Area Networks: Near-field intrabody communication - T. G. Zimmerman (1996).

[31] The Galvactivator: A glove that senses and communicates skin conductivity - Rosalind W.Picard and Jocelyn Scheirer.

[32] Some Computer Science Issues in Ubiquitous Computing - Weiser (1993).

[33] Middleware for Mobile Applications Beyond 3G - Raatikainen.

[34] Perceptual Components for Context Aware Computing - James L. Crowley, Jolle Coutaz,Gaeten Rey, Patrick Reignier (2002).

[35] Adding Generic Contextual Capabilities to Wearable Computers, - Jason Pascoe (1998).

[36] The Memory Glasses: Towards a Wearable, Context Aware, Situation-appropriate ReminderSystem - Richard W. DeVaul, Brian Clarkson, Alex Sandy Pentland (2000).

[37] Modeling Context Information in Pervasive Computing Systems - Karen Henricksen, JadwigaIndulska, Andry Rakotonirainy

[38] The Personal Server: The Center of Your Ubiquitous World - Roy Want, Trevor Pering, JamesKardach, and Graham Kirby (2001).

[39] Ultra-Wideband Technology for Short- or Medium-Range Wireless Communications - JeffForester, Evan Green, Shrinivasa Somayazulu, Dave Leeper (2001).

[40] IEEE 830-1998 standard for creating SRS documents - IEEE-SA Computer Society (1998).

[41] Freeband Kennisimpuls - http://www.freeband.nl/

92

Appendix A

Figures

Scaleable Vector ImageSVG

Unicode Transformation FormatUTF

Uniform Resource LocatorURL

Global Positioning SystemGPS

Rich Site SummaryRSS

Graphical User InterfaceGUI

Java Virtual MachineJVM

Hypertext Transport ProtocolHTTP

Document Object ModelDOM

Document Type DefinitionDTD

Connected Limited Device ConfigurationCLDC

Mobile Information Device ProfileMIDP

A platform independent programming language from SUN Microsystems.JAVA

JAVA 2 MicroEditionJ2ME

Juxtapose; a P2P networking toolkitJXTA

System Requirements SpecificationSRS

Body Area NetworkBAN

Personal Area NetworkPAN

Peer to PeerP2P

Internet Protocol version 6IPv6

Advanced Encryption StandardAES

XML Stylesheet Flow-ObjectXSL-FO

XML Stylesheet TransformationXSLT

FilesystemFS

Native XML DatabaseNXD

XML User-interface LanguageXUL

Unified Modeling LanguageUML

Extensible Markup LanguageXML

The Mobile MemexM3X

Object Constraint LanguageOCL

DescriptionAcronym

Figure A-1: List of used acronyms

93

checkAdv()

«interface»FindValidate

ConfigFileName: StringgroupName: StringmsAdv: ModuleSpecAdvertisementopipe: OutputPipeopipeAdv: PipeAdvertisementpubLocation: String

Publisher() collectData() init() jxtaInit() main() process()

Publisher

ConfigFileName: StringgroupName: String

main() mdidgen() printIt() startJXTA()

mdidgen

ClassID: StringDATA_COLLECTED: StringGroupDiscoveryDelay: longMAXRETRIES: intModuleSpecName: StringPUBLISHER_LOCATION: StringSpecID: StringWaitingTime: intdiscovery: DiscoveryServicegroup: PeerGrouppipes: PipeService

M3xJXTAPeer() findAdv() findModuleSpecAdv() joinGroupIfExists() publishModuleSpecAdv()

ModuleSpecAdvValidator() checkAdv()

ModuleSpecAdvValidator

M3xJXTAPeer

ConfigFileName: StringServicePipeAdv: PipeAdvertisementcol: CollectiongroupName: Stringimessage: Messageipipe: InputPipesensorData: StringservicePipeID: StringservicePipeName: StringservicePipeType: String

Subscriber() aquireData() createInputPipeAdvIfNotExist() findPipeAdv() init() jxtaInit() main() processData() xindiceInit()

PipeAdvValidator

Subscriber

Figure A-2: An UML expanded class diagram showing the JXTA devices classes in the MobileMemex

94

Subscriber StoragePublisher

Aquire sensor data *

Package sensor data

Publish context data

Aggregate context data

Delete context data

Store context data

Check for subscriber

[unavailable]

[success]

store PCH local

store PCH remote

Check for storage device

[unavailable]

[available]

Notify Publishing device

Update NXD

[available]

Notify subscriber

[fail]

Delete PCH

Notify User

[fail]

Wait for notification

[fail]

Wait for notification

[success]

[fail]

[fail]

[success][success]

[success]

Figure A-3: An UML activity diagram showing interdevice interaction

95

<?xml version="1.0" encoding="UTF-8" ?><!-- M3XML context history --><!DOCTYPE contexthistory (View Source for full doctype...)><contexthistory userid="3ffe:80e8:d8::1">

<timestamp timeid="20030925153710" sensorid="3ffe:80e8:d8::2"><context>

<spatial><location sensorid="3ffe:80e8:d7::2">LON037.400LAT122.018ALT014.019</location><orientation sensorid="3ffe:80e8:d7::2">181</orientation>

</spatial><existential>

<temperature sensorid="3ffe:80e8:d8:12a2:7">27</temperature></existential><functional>

<activity sensorid="3ffe:80e8:d7::5">running</activity></functional>

</context><content>

<aural><voicecall sensorid="3ffe:81e5:c2::2">

<callerid><name>Jungle Jane</name><phonenumber>+312055501312</phonenumber>

</callerid><file>20030925153710.voicecall.ogg</file><keywords sensorid="3ffe:80e8:c3::2">vacation,plane,money,week,appartment,beach,baggage</keywords>

</voicecall></aural>

</content></timestamp><timestamp timeid="20030925153710" sensorid="3ffe:80e8:d8::2">

<context><spatial>

<location sensorid="3ffe:80e8:d7::2">LON037.399LAT122.018ALT014.021</location></spatial><temporal>

<season sensorid="3ffe:80e8:d7::13">Fall</season></temporal>

</context><content>

<visual><image sensorid="3ffe:80e8:c2::2">

<file> 20030925153710.photo.jpg</file><annotation sensorid="3ffe:80e8:c1::2">My not so new car</annotation>

</image></visual><textual>

<email sensorid="3ffe:80e8:12::1"><sender>

<name>John Doe</name><emailaddress>jdoe@email.me</emailaddress>

</sender><subject>RE:RE: Here's Johnny!</subject><file>20030925153710.email.utf8</file><keywords sensorid="3ffe:80e8:c1::2">John,work,call,email,open,late,tomorrow</keywords>

</email></textual>

</content></timestamp>

</contexthistory>

Figure A-4: An XML example of a context history containing two timestamps and content references

96

: M3X User Main Frame

text entry field

XML Native DB

result window

push button search

create

enter query

send query

send query

send results

close

create

send results

At this moment the user is able to view the results of the search query. Depending on the results the user may perform additional operations.

close

Figure A-5: An UML sequence diagram for the personal search engine application

97

Acquire data

(from use cases)

Publishing device

(f rom acto rs)

Publish data

(from use cases)

TimeOut

(from use cases)

Delegate task

(from use cases)

Leave

(from use cases)

Register

(from use cases)

<<extend>>

M3X BAN Device

(from actors)

Index aggregated data

(f ro m use cases)

Register

(f ro m use cases)

Delegate task

(f ro m use cases)

Leave

(f ro m use cases)

M3X BAN Device

(f rom acto rs)

Storage device

(from actors)

Store aggregated data

(f ro m use cases)<<include>>

M3X Device

(from actors)

Device

(f ro m a ctors)

TimeOut

(from use cases)

<<extend>>

Register

(from use cases)

Delegate task

(from use cases)

M3X BAN Device

(from actors)

Leave

(from use cases)

Aggregate published data

(from use cases)

Upload partial context history

(from use cases)

Subscribing device

(f rom ac to rs)

TimeOut

(from use cases)

<<extend>>

Figure A-6: UML Use case diagrams representing the publishing, subscribing and storage device

98

Monitoring Conditions

(from use cases)

Remote Surveillance Application Subject

(from actors)

Monitor Subject

(from use cases)

Surveillant

(from actors)

<<depends on>>

Select Application

(from use cases)Application Browsing

Application User(from actors)

Set Conditions

(from use cases)

Active Coaching Application User

(from actors)Instruct / Advise

(from use cases)

Emergency Service

(from actors)

Set Conditions

(from use cases)

Reminder Service User

(from actors) Remind user

(from use cases)

Notify 3rd Party

(from use cases)

Monitor Physical State (User)

(from use cases)

Emergency Service

(f ro m ac tors)

Set Application Preferences

(from use cases)

Notify User

(from use cases)Medical Monitoring

Application User(from actors)

Inject Insuline

(from use cases)

Provide additional information

(from use cases)

External Service

(from actors)Navigate

(from use cases)

Instruct / Advise

(from use cases)

Specify external services

(from use cases)

Travel Navigation Application User

(f rom actors)

Publish data

(from use cases)

Acquire data

(from use cases)

Cache data

(from use cases)

Publishing device

(from actors)

Acquire aggregated data

(from use cases)

Store aggregated data

(from use cases)

Index aggregated data

(from use cases)

Storage device

(from actors)

Register

(from use cases)

Advertise

(from use cases)

Delegate task

(from use cases)

M3X BAN Device

(from actors)

M3X Device

(from actors)

Cache data

(from use cases)

Aggregate published data

(from use cases)

Subscribing device

(from actors)

Device

(from actors)

Retrieve data

(from use cases)Access device

(from actors)

Browse Results

(from use cases)

Perform Search

(from use cases)

Personal & Contextual Search Engine Applicati...

(from actors)

TimeOut

(from use cases)

<<extend>>

Define M3X Profile

(from use cases)

Monitor System State

(from use cases)

Install M3X Application

(from use cases)

M3X Administrator

(f rom actors)

Define Application Profile

(f ro m use cases)

<<include>>

Figure A-7: UML Use Cases representing the differerent applications and devices

99

Emergency Service

(from actors)

External Service

(f ro m a ctors)

M3X Administrator

(f rom a ctors)

Remote Surveillance Application Subject

(from actors)

Surveillant

(from actors)

Medical Monitoring Application User

(from actors)

Personal & Contex tual Search Engine Applicati...

(from actors)

Reminder Service User

(f rom actors)

Travel Navigat ion Application User

(from actors)

User

(from actors)

Applicat ion Browsing Applicat ion User

(from actors)

Active Coaching Application User

(from actors)

Location Service

(from actors)

Weather Service

(from actors)

News Service

(from actors)

Traffic informat ion service

(from actors)

Figure A-8: An UML Use case diagram showing the actors in the M3X

100

Appendix B

Use-Case-Model Survey for M3X

UML diagrams

Issue <1.0>

Revision History

Date Issue Description Author<dd/mmm/yy> <x.x> <details> <name>

101

B.1 Introduction

B.2 Actors

Personal & Contextual Search Engine Application User

Documentation: The user of a personal contextual search engine is interested in facts and mediacreated by the user in the past. The search engine is a reactive application that allows the user toformulate a query that allows for the retrieval of the required information.

Active Coaching Application User

Documentation: The user of the active coaching application. This user is interested in activecoaching while sporting.

Application Browsing Application User

Documentation: The user has to install and select an M3X supported application.

Emergency Service

Documentation: An emergency service is provided by a medical institution. and depends on themedical needs of the user.

External Service

Documentation: An external service is an entity that provides a sevices related to context or mediacontained in the M3X system. Examples are location, news and weather servers on the internet.These services may or may not be used by the search engine application in the search process. Suc-cessful usage of external services may result in a more complete, useful and meaningful representationof information to the user.

Location Service

Documentation:

M3X Administrator

Documentation: The M3X administrator is the owner of the system. He has the rights to define therules on how the system should function.

Medical Monitoring Application User

Documentation: A medical monitoring application user has a need for automatic and pro-activemonitoring of the users physical state.

News Service

Documentation:

Reminder Service User

Documentation: The User of the reminder service application uses this application to be automat-ically reminded of things and tasks relevant to the context (such as location and time) of the taksand the user.An example of a reminder service user can be a senior citizen that would like to be remindedautomatically when he or she should take his/her medication.

102

Remote Surveillance Application Subject

Documentation:

Surveillant

Documentation: The surveillant is a person or system that is interested in monitoring certain aspectsof another person who is also the owner of an M3X system and has agreed the surveillant to monitorthese aspects (defined in conditions)

Travel Navigation Application User

Documentation: A travel navigation application user is interested in navigating from one locationto another in an efficient and fast way.

User

Documentation: Type: PersonRole: Owner of the M3X systemPurpose: Retrieve personal media and/or contextual information.The user of the M3X is often also the owner of the system.It is a person with a need for personal and/or contextual information contained in the M3X system.In this case the user will perform a search operation.

Weather Service

Documentation:

Publishing device

Documentation: A publishing device is a device that contains one or more sensors. The functionalityof the publishing device consists of acquireing the sensor data, adding device specific meta-data. andthe notification of subcribing devices by publishing this information if possible.

Subscribing device

Documentation: A subscribing device is a device that has a different functionality from a publishingdevice. Subscribing devices aggregate the contextal information and media created by publishingdevices into a context history stored in a native XML database.A subscribing device can also be a publishing device.The communicator would be the primary subscribing device

Storage device

Documentation: A Storage device can be a M3X device (portable laptop or tablet), but also astationary device such as a PC, Workstation or webserver.The storage device is a node in the M3X peergroup but is not nescessarily part of the body areanetwork.A storage device has to provide enough space to store the context and media created by a userwithin the M3X framework for several decades (preferrably an entire human life).Furthermore this device has to be accessible by both the users M3X body area network devices(So the on-body subscibing devices can upload cached data) and applications that make use of thissystem. either locally or remote.Eventually a storage device might not be nescessary anymore as technological improvements allowfor enough computational and storage capability in wearable M3X devices such as the communicator.

103

M3X BAN Device

Documentation: An M3X device is a device capable of functioning in the M3X system.

To be able to function in a M3X system the device has to support the P2P protocols (JXTA) used,and understand (parse) the M3XML language.

External User

Documentation: The external user is a person or system that is aware of the m3x user using areminder application and has the rights (as defined in the preferences) to notify the user. (Think ofa persons wife, friends or collegues).

M3X Device

Documentation:

Traffic information service

Documentation:

Device

Documentation:

Access device

Documentation:

B.3 Use Cases

Acquire aggregated data

Documentation:

Acquire data

Documentation: The main task a publishing device performs for the system is to aquire data usingsensor devices embedded in this device and publish this data to the system for further processing.

Advertise

Documentation: Once the device is registered to the M3X peergroup it will advertise it’s role (pub-lishing/subscribing/access) and functionality (available sensors, actuators, interfaces, applications)and status (battery life, memory) to the other members of the peergroup.

Aggregate published data

Documentation: The system uses the subscibing devices to aggregate the contextual data and mediapublished by the publishing devices.

The subscibing devices receive sensor data and media in the M3XML format describing the dataand adding a timestamp.

because all data is timestamped the data can be encapsulated in a partial context history.

104

Browse Results

Documentation: Results can be a list of related media types (textual, visual or aural) and theircorresponding (relevant) contexts.The user can browse these results to retrieve the nescessary media. or can use these results to narrowthe search action.The results dont always have to produce media.it could also be a calculated answer to a well formulated question. (i.e. how many hours have ispent in the office since august 12 1999?)* Browse Media* Browse Context

Cache data

Documentation:

Define Application Profile

Documentation: Most pro-active application allow the user to set preferences as to how the appli-cation should behave. Settings can range from constrainst/triggers that define when to measure aphenomena or contact an external service, to appropriate responses to context patterns observed inthe context history. Also the user can define what actuators the pro-active application should useto notify the user under different circumstances.

Define M3X Profile

Documentation: Because the owner has got different interests and concerns it is possible to de-fine a personal profile that contains the preferences and constraints as to how the system shouldcollect, share and store information. Every device will advertise a Module Specification Adver-tisement (MSA) to advertise its capablities. The administration application can collect all theseadvertisements from the peergroup to create a list of devices including functions, services, sensorsand actuators. The user can use this list to make a selection, defining what sensordata is to bestored in the system and at what intervals.(Applications also should define what sensordata they operate on)

Delegate task

Documentation: The M3X device will check to see if the user defined preference profile containssettings relevant to the device.

Index aggregated data

Documentation: Besides storing the aggregated data the storage device also indexes the informationfor future querying by supported applications.

Inject Insuline

Documentation: In the case of diabetes (see scenario). the user can carry an insulin pump thatis also part of the M3X system. If the subscribing device reaches a certain state (defined by theuser/emergency service) it can automatically inject the user with the required amount of insulin

Install M3X Application

Documentation: There may exist a number of applications that operate on the m3x context his-tory stored in the native XML database. The user has to choose the application that provide thefunctionality required by the M3X user.Usually the application will be installed on the storage device of the user.

105

It might be possible to install M3X supported applications on subscibing device such as the com-municator so semi-realtime applications can be used that operate on partial context history.

Instruct / Advise

Documentation: In case of a pro-active navigation application:

If certain conditions are met, the active coaching application reaches a state where it can advise theuser to adapt a matching behaviour.

Leave

Documentation: The M3X device can leave the M3X peergroup at any time and without notice ifthe user removes the device from the body area network.

It is up to the system to notice the device cannot be reached anymore and conclude that the devicehas left the peergroup.

Monitor Physical State (User)

Documentation: This usecase describes how the physical state as registerd by the sensors accordingto the set application preferences can be monitored by both the user of the system as well as externalemergency services selected by the user.

Monitor Subject

Documentation: This use case descibes how the subject can be monitored by the surveillant basedon the monitoring conditions set by the subject.

Monitor System State

Documentation: The administrator should be able to monitor the system state of the M3X. Systemstate include available memory and remaining power per device. and can be extended with otherrelevant information related to the proper fucntioning of the M3X

Monitoring Conditions

Documentation: The remote surveillance application user can set the conditions under which he/sheagrees to be monitored by external entities.

These can be based on location (outside the house). or other criteria.

Navigate

Documentation: Based on the information recorded by the system and related to external servicesthe user is able to navigate more efficiently to his/her destination.

Notify 3rd Party

Documentation: If the user is notified and the contition continues to deteriorate, eventually exceedinga set alarm-level, the sytem will automatically notify the emergency service.

Notify User

Documentation: This use case describes how the pro-active monitoring application can notify theuser in case of abnormal patterns and deviations that might threaten the health of the user.

106

Perform Search

Documentation: This use case prescribes the way in which a user performs a search operation in theM3X system.

It shows how user input can generate the system output required by the user.

Provide additional information

Documentation: External services can provide relevant additional information related to the in-formation gathered by the M3X system. Additional information can range from traffic conditionsrelative to the road the user is on, to route planning information and providing maps.

Publish data

Documentation: Data the publishing device has aquired should be published when possible. To dothis the device first has to check if any subscribing devices are available within the M3X peergroup.If there are the device publishes the data to the subscibing device(s). If none are available the devicecaches the measured data in local memory until a subscibing device becomes available.

In the case a subscribing device does not become available and the internal memory of a publishingdevice is full. data will be lost.

(data aquired by a publishing device can be context data, but also media.

the difference between media is that these files cannot be searched or indexed on their contents.these files can be aural, visual or textual in nature. To open these files a special application isusually required).

Register

Documentation: When a device is added to the collection of devices that constitutes a wearersM3X system. the new device will try to discover the m3x peergroup and register the device to thispeergroup.

A registered device can advertise its capabilities in terms of available actuators and sensors.

This advertisement is used by the system to compile a list on which the user can define a preferenceprofile.

(This profile will contain the sensors the user decides to monitor and their constrainst such asmeasuring interval, but also privacy and security settings concerning storage and the sharing ofcollected information)

Remind user

Documentation: When the application notices that the conditions are met it will automaticallynotify the user with the set message.

Retrieve data

Documentation:

Select Application

Documentation: The user can have a multitude of M3X enabled applications installed on the accessdevice.

This use case shows how the user can active the application of his/her choice.

107

Set Application Preferences

Documentation: This use case shows how the user can define a user profile for the application. Thisuser profile contains all preferences relevant to the application.For the medical monitoring application this can be setting the level at wich a certain dosage shouldbe injected, or the address of the emergency service.

Set Conditions

Documentation: This usecase shows how the user can set application specific conditions that deter-mine the applications behaviour.If different sensors all fullfil a set of pre-defined conditions the system will recognize this patternand initiate the corresponding correct behaviour.In case of Remider service: Choose preferred actuator and Display reminder message on preferreddevice.

Specify external services

Documentation: The user chooses external services that can help during navigation such as a onlinemapping service and actual traffic information.

Store aggregated data

Documentation: The system uses the storage device to store the aggregated data (context historyand related media). When a storage peer enters the peergroup the subscibing devices will uploadtheir partial context history wich is then merged with the existing context history in the native XMLdatabase residing on the storage device.

TimeOut

Documentation:

Upload partial context history

Documentation: The aggregation proceess leads to a partial context history. when this locally cachescontext history reaches a certain size or covers a timespan defined in the user profile, the subscribingdevices will try to locate a storage server to upload the context history.

108

Appendix C

Mobile Memex Sourcecode

C.1 Subscriber.java

package com.pink.m3x.jxta;

/*

M3X - the mobile memex prototype

based on "IBM DeveloperWorks JXTA Sample" by S. Li and

the Xindice Developers Guide "Simple XML:DB Example".

Extended to provide distributed context aggregation

and storage in the Xindice Natvive XML Database by

R.M.Vaandrager, rmvaandr@cs.vu.nl

*/

import java.io.File;

import java.io.FileInputStream;

import java.net.URL;

import java.util.Properties;

import net.jxta.discovery.DiscoveryService;

import net.jxta.document.Advertisement;

import net.jxta.document.AdvertisementFactory;

import net.jxta.endpoint.Message;

import net.jxta.id.IDFactory;

import net.jxta.peergroup.PeerGroup;

import net.jxta.peergroup.PeerGroupFactory;

import net.jxta.pipe.InputPipe;

import net.jxta.pipe.PipeID;

import net.jxta.pipe.PipeService;

import net.jxta.protocol.PipeAdvertisement;

import org.xmldb.api.base.*;

import org.xmldb.api.modules.*;

import org.xmldb.api.*;

public class Subscriber extends M3xJXTAPeer {

private String groupName = "m3xtest";

private String sensorData = "";

private static String ConfigFileName = "subscriberProfile";

private static String servicePipeName = "m3xSubscriberPipe";

private static String servicePipeType = PipeService.PropagateType;

public static String servicePipeID =

"urn:jxta:uuid-969610EE8945417CA56F9771197EE3965207E4C303154E7EB5878751AE22761804";

private PipeAdvertisement ServicePipeAdv = null;

private InputPipe ipipe = null;

private Message imessage = null;

private Collection col = null; // this is a JXTA Collection

public Subscriber() {

}

public void init() throws Exception {

XindiceInit(); // initialize the Xindice Native XML Database

jxtaInit(); // initialize the JXTA P2P Network

// create subscribing pipe

try {

System.out.println(

"Creating subscribing pipe for data collection...");

ipipe = pipes.createInputPipe(ServicePipeAdv);

Thread.sleep(5000);

} catch (Exception ex) {

System.out.println("Subscribing pipe creation failed.");

return;

}

System.out.println("Subscribing pipe created successfully");

}

public void aquireData() throws Exception {

// here we will aquire the sensordata and send it over for processing

if (ipipe != null) {

System.out.println("Waiting for data from publisher...");

while (true) {

try {

109

imessage = ipipe.waitForMessage();

} catch (Exception ex) {

System.out.println("Err: " + ex);

break;

}

if ((imessage.hasElement(PUBLISHER_DEVICE) && imessage.hasElement(DATA_COLLECTED))) {

sensorData = imessage.getString(DATA_COLLECTED);

System.out.println("Received data from publisher: " + imessage.getString(PUBLISHER_DEVICE));

//System.out.println("Data received is: " + imessage.getString(DATA_COLLECTED));

//System.out.println("Sending to internal network for processing.");

processData();

}

}

}

}

private void processData() throws Exception{

// this is where we would send data to the RDBMS for analysis

// Simple XML:DB API Example to insert a new document into the database.

try {

XMLResource document = (XMLResource) col.createResource(null,"XMLResource");

document.setContent(sensorData);

col.storeResource(document);

System.out.println("processing data in Xindice NXD...");

System.out.println("M3XML document inserted in Xindice as: " + document.getId());

}

catch (XMLDBException e) {

System.err.println("XML:DB Exception occured " + e.errorCode + " " + e.getMessage());

}

}

private void XindiceInit() throws Exception {

// create and register the Xindice database driver

System.out.println("Initializing Xindice NXD...");

try {

Class c =

Class.forName("org.apache.xindice.client.xmldb.DatabaseImpl");

Database database = (Database) c.newInstance();

DatabaseManager.registerDatabase(database);

col =

DatabaseManager.getCollection("xmldb:xindice:///db/m3xdb");

} catch (XMLDBException e) {

System.err.println("XML:DB Exception occured " + e.errorCode);

}

}

private void jxtaInit() {

Properties myProp = null;

FileInputStream inFileStream = null;

// read and process profile

System.out.println(

"Reading profile: [" + ConfigFileName + "], and processing...");

try {

inFileStream = new FileInputStream(new File(ConfigFileName));

myProp = new Properties();

myProp.load(inFileStream);

} catch (Exception ex) {

System.out.println("Profile access problem - file not found.");

}

if (myProp != null) {

groupName = myProp.getProperty("group", "");

System.out.println(" * Group to join will be: " + groupName);

}

// start JXTA and join group

PeerGroup tpPG = null;

System.out.println("Starting M3X prototype in JXTA...");

System.out.println("Joining default NetPeerGroup...");

try {

// create, and Start the default jxta NetPeerGroup

group = PeerGroupFactory.newNetPeerGroup();

System.out.println("Attempting to join group: " + groupName);

if (groupName.length() != 0) {

try {

tpPG = joinGroupIfExists(groupName);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

if (tpPG != null) {

group = tpPG;

System.out.println(

"group " + groupName + " joined successfully");

}

}

} catch (Exception e) {

System.out.println("Err : group creation failure");

}

if (group != null) {

discovery = group.getDiscoveryService();

pipes = group.getPipeService();

}

System.out.println(

"Look for previously created Pipe Adv, create one if not exist");

ServicePipeAdv = createInputPipeAdvIfNotExist();

// check for ModuleSpecAdv

// The ModuleSpecAdvertisement has two purposes,

// 1) to publish the uri of its formal specs for developers and

// 2) to publish the means of remote access to the module’s

// services if that is appropriate.

System.out.println("Checking to see if MSA previously published...");

110

if (findModuleSpecAdv() == null) {

System.out.println(".. MSA not found, need to create it");

publishModuleSpecAdv(ServicePipeAdv);

} else

System.out.println(".. found previously published MSA");

}

protected PipeAdvertisement findPipeAdv() {

return (

PipeAdvertisement) findAdv(

"PipeAdvertisement",

"Name",

servicePipeName,

new PipeAdvValidator(),

// local only

true);

}

class PipeAdvValidator implements FindValidate {

public PipeAdvValidator() {

}

public boolean checkAdv(Advertisement adv) {

// we can compare pipe ID as well if so desired

return true;

}

}

public PipeAdvertisement createInputPipeAdvIfNotExist() {

Advertisement padv = null;

if ((padv = findPipeAdv()) != null) {

// found it

return (PipeAdvertisement) padv;

}

System.out.println(

"Previously published pipe advertisment not found, creating a new one.");// create a new one

PipeAdvertisement pipeAdv;

pipeAdv =

(PipeAdvertisement) AdvertisementFactory.newAdvertisement(

PipeAdvertisement.getAdvertisementType());

pipeAdv.setName(servicePipeName);

pipeAdv.setType(servicePipeType);

PipeID tpID;

try {

tpID = (PipeID) IDFactory.fromURL(new URL(servicePipeID));

} catch (Exception ex) {

tpID = IDFactory.newPipeID(group.getPeerGroupID());

}

pipeAdv.setPipeID(tpID);

System.out.println("Locally caching the new pipe advertisement...");

try {

discovery.publish(pipeAdv, DiscoveryService.ADV);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

return pipeAdv;

}

public static void main(String[] args) throws Exception {

Subscriber subscr = new Subscriber();

subscr.init();

subscr.aquireData();

}

}

111

C.2 Publisher.java

package com.pink.m3x.jxta;

/*

M3X - the mobile memex prototype

based on "IBM DeveloperWorks JXTA Sample" by S. Li and

the Xindice Developers Guide "Simple XML:DB Example".

Extended to provide distributed context aggregation

and storage in the Xindice Natvive XML Database by

R.M.Vaandrager, rmvaandr@cs.vu.nl

*/

import java.io.File;

import java.io.FileInputStream;

import java.util.*;

import net.jxta.endpoint.Message;

import net.jxta.peergroup.PeerGroup;

import net.jxta.peergroup.PeerGroupFactory;

import net.jxta.pipe.OutputPipe;

import net.jxta.protocol.ModuleSpecAdvertisement;

import net.jxta.protocol.PipeAdvertisement;

public class Publisher extends M3xJXTAPeer {

private OutputPipe opipe = null;

private PipeAdvertisement opipeAdv = null;

private ModuleSpecAdvertisement msAdv = null;

private static String ConfigFileName = "publisherProfile";

private String groupName = null;

private String pubDevice = null;

private String sensorType = null;

private String sensorInterval = null;

public Publisher() {

}

public static void main(String[] args) {

Publisher pub = new Publisher();

pub.init();

pub.process();

}

public void process() {

String collectedData = null;

Message sendMsg = null;

while (true) {

System.out.println("... acquire sensor data ....");

collectedData = collectData();

if (collectedData != null) {

System.out.println("... publish sensor data ....");

sendMsg = pipes.createMessage();

sendMsg.setString(PUBLISHER_DEVICE, pubDevice);

sendMsg.setString(DATA_COLLECTED, collectedData);

try {

opipe.send(sendMsg);

} catch (Exception ex) {

System.out.println("Err: " + ex);

break;

}

}

System.out.println("standby for set interval...");

long interval = Long.parseLong(sensorInterval);

try {

Thread.sleep(interval); //in ms

} catch (Exception ex) {

}

}

}

private String collectData() {

// this is where data collection is triggered and a corresponding M3Xml document is created

Random rn = new Random();

int n = 38;

int sensorData = rn.nextInt(n + 1); //create a random integer in range 0-38

return "<sensor><device>"

+ pubDevice

+ "</device><type>"

+ sensorType

+ "</type><data>"

+ sensorData

+ "</data></sensor>"; // returns the M3Xml document as a string

}

public void init() {

jxtaInit();

try {

System.out.println("connecting to M3X network...");

opipe = pipes.createOutputPipe(opipeAdv, 10000l);

} catch (Exception ex) {

System.out.println("publish pipe creation failed.");

return;

}

if (opipe != null)

System.out.println("publish pipe created successfully");

else {

System.out.println("Cannot create output pipe.");

System.exit(1);

}

}

private void jxtaInit() {

Properties myProp = null;

FileInputStream inFileStream = null;

System.out.println(

112

"Reading profile: [" + ConfigFileName + "], and initializing...");

try {

inFileStream = new FileInputStream(new File(ConfigFileName));

myProp = new Properties();

myProp.load(inFileStream);

} catch (Exception ex) {

System.out.println("Profile access problem - file not found.");

}

if (myProp != null) {

groupName = myProp.getProperty("group", "");

System.out.println(" * Group to join will be : " + groupName);

pubDevice = myProp.getProperty("device", "default device");

System.out.println(" * Publishing device is : " + pubDevice);

sensorType = myProp.getProperty("type", "default device");

System.out.println(" * Sensor type is : " + sensorType);

sensorInterval = myProp.getProperty("interval", "default device");

System.out.println(" * Measurement interval : " + sensorInterval + "ms");

}

// Start JXTA and join the M3X peergroup

PeerGroup tpPG = null;

System.out.println("Starting M3X prototype in JXTA...");

System.out.println("Joining default NetPeerGroup...");

try {

// create, and Start the default jxta NetPeerGroup

group = PeerGroupFactory.newNetPeerGroup();

System.out.println("Attempting to join group: " + groupName);

if (groupName.length() != 0) {

try {

tpPG = joinGroupIfExists(groupName);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

if (tpPG != null) {

group = tpPG;

System.out.println(

"group " + groupName + " joined successfully");

}

}

} catch (Exception e) {

System.out.println("Err : group creation failure");

}

if (group != null) {

discovery = group.getDiscoveryService();

pipes = group.getPipeService();

}

// check for ModuleSpecAdv

System.out.println(

"Seaching for publisher’s Module Spec Advertisement (MSA)...");

if ((msAdv = findModuleSpecAdv()) == null) {

System.out.println(

".. MSA not found, please check connection - terminating.");

System.exit(1);

} else

System.out.println(".. found subscribers’s MSA");

System.out.println("Extracting pipe adv from MSA...");

opipeAdv = msAdv.getPipeAdvertisement();

}

}

113

C.3 M3xJXTAPeer.java

package com.pink.m3x.jxta;

/*

M3X - the mobile memex prototype

based on "IBM DeveloperWorks JXTA Sample" by S. Li and

the Xindice Developers Guide "Simple XML:DB Example".

Extended to provide distributed context aggregation

and storage in the Xindice Natvive XML Database by

R.M.Vaandrager, rmvaandr@cs.vu.nl

*/

import java.util.Enumeration;

import net.jxta.discovery.DiscoveryService;

import net.jxta.document.Advertisement;

import net.jxta.impl.wire.AdvCooker;

import net.jxta.peergroup.PeerGroup;

import net.jxta.pipe.PipeService;

import net.jxta.platform.ModuleSpecID;

import net.jxta.protocol.ModuleSpecAdvertisement;

import net.jxta.protocol.PeerGroupAdvertisement;

import net.jxta.protocol.PipeAdvertisement;

public class M3xJXTAPeer {

public static final String ClassID =

"urn:jxta:uuid-EE99266A1DE84E3DB34D9CC842EC889105";

public static final String SpecID =

"urn:jxta:uuid-EE99266A1DE84E3DB34D9CC842EC8891B9EB13ECA6FE44DDA112B5F5E357763006";

public static final String PUBLISHER_DEVICE = "PubDevice";

public static final String DATA_COLLECTED = "ColData";

public static final String ModuleSpecName = "JXTASPEC:m3xPublisher-SPEC";

public static final int MAXRETRIES = 5;

public static final int WaitingTime = 1000;

public static final long GroupDiscoveryDelay = 25000l;

protected PeerGroup group = null;

protected DiscoveryService discovery = null;

protected PipeService pipes = null;

public M3xJXTAPeer() {

}

protected ModuleSpecAdvertisement findModuleSpecAdv() {

return (ModuleSpecAdvertisement) findAdv(

"ModuleSpecAdvertisement",

"Name",

ModuleSpecName,

new ModuleSpecAdvValidator(),

false);

}

class ModuleSpecAdvValidator implements FindValidate {

public ModuleSpecAdvValidator() {

}

public boolean checkAdv(Advertisement adv) {

// perform further validation here if necessary

return true;

}

}

protected Advertisement findAdv(

String advType, String attr, String val, FindValidate validator, boolean localOnly) {

Enumeration enum = null;

System.out.println("Looking for " + advType + ", please wait...");

// First look in the local storage

try {

enum =

discovery.getLocalAdvertisements(

DiscoveryService.ADV,

attr,

val);

} catch (Exception e) {

}

if ((enum != null) && (enum.hasMoreElements())) {

Advertisement adv = null;

while (enum.hasMoreElements()) {

try {

adv = (Advertisement) enum.nextElement();

if (validator.checkAdv(adv))

return adv;

} catch (Exception e) {

continue;

}

}

}

if (localOnly)

return null;

System.out.println(" cannot find it locally, trying remote");

// Now, search remote

discovery.getRemoteAdvertisements(

null, DiscoveryService.ADV, attr, val, 2, null);

// Wait a bit in order to get an answer.

int i = 0;

while (true) {

try {

if (i > MAXRETRIES) {

System.out.print(".");

break;

}

Thread.sleep(WaitingTime);

i++;

} catch (Exception e) {

114

}

System.out.println("");

// Look in the local storage again

try {

enum =

discovery.getLocalAdvertisements(

DiscoveryService.ADV,

attr,

val);

if ((enum != null) && (enum.hasMoreElements())) {

Advertisement adv = null;

while (enum.hasMoreElements()) {

try {

adv = (Advertisement) enum.nextElement();

if (validator.checkAdv(adv))

return adv;

} catch (Exception e) {

continue;

}

}

}

} catch (Exception e) {

}

}

return null;

}

public PeerGroup joinGroupIfExists(String ingname) throws Exception {

String attr = "Name";

int threshold = 1;

PeerGroup tpGroup = null;

if ((ingname == null) || (ingname.length() == 0))

return null;

discovery = group.getDiscoveryService();

// sent remote group discovery message

discovery.getRemoteAdvertisements(

null, DiscoveryService.GROUP, attr, ingname, threshold, null);

// wait 5 sec

System.out.println("remote group discovery message sent");

Thread.sleep(GroupDiscoveryDelay);

// check local cache

Enumeration res;

res =

discovery.getLocalAdvertisements(

DiscoveryService.GROUP,attr,ingname);

PeerGroupAdvertisement groupAdv = null;

if (res.hasMoreElements()) {

groupAdv = (PeerGroupAdvertisement) res.nextElement();

// create group with advertisement

tpGroup = group.newGroup(groupAdv);

System.out.println("group joined successfully");

}

return (tpGroup);

}

protected void publishModuleSpecAdv(PipeAdvertisement ServicePipeAdv) {

ModuleSpecAdvertisement msAdv = null;

System.out.println("Creating new MSA");

ModuleSpecID tpID = null;

try {

tpID = AdvCooker.buildModuleSpecID(SpecID);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

if (tpID != null) {

msAdv =

AdvCooker.buildModuleSpecAdvertisement(tpID, //ModuleSpecID msid,

ModuleSpecName, // String moduleSpecName,

"m3x aggregation engine", // String moduleSpecDescription,

"m3x JXTA", // String creator,

"Version 1.0", // String version,

"http://www.cs.vu.nl/~rmvaandr/m3x.html", //String specURI,

ServicePipeAdv, // PipeAdvertisement pipeAdv,

null, // ModuleSpecID proxySpecID,

null, //ModuleSpecID authorizationSpecID,

null // StructuredDocument param

);

System.out.println("Locally and remotely publish the MSA");

try {

discovery.publish(msAdv, DiscoveryService.ADV);

discovery.remotePublish(msAdv, DiscoveryService.ADV);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

}

}

}

115

C.4 mdidgen.javapackage com.pink.m3x.jxta;

/*

M3X - the mobile memex prototype

based on "IBM DeveloperWorks JXTA Sample" by S. Li and

the Xindice Developers Guide "Simple XML:DB Example".

Extended to provide distributed context aggregation

and storage in the Xindice Natvive XML Database by

R.M.Vaandrager, rmvaandr@cs.vu.nl

*/

import java.io.File;

import java.io.FileInputStream;

import java.util.Properties;

import net.jxta.id.IDFactory;

import net.jxta.impl.wire.AdvCooker;

import net.jxta.peergroup.PeerGroup;

import net.jxta.peergroup.PeerGroupFactory;

import net.jxta.pipe.PipeID;

public class mdidgen extends M3xJXTAPeer {

protected static final String ConfigFileName = "profile";

protected String groupName = "";

public mdidgen() {

}

public void printIt() {

AdvCooker.printNewClassAndModuleID();

PipeID pid = IDFactory.newPipeID(group.getPeerGroupID());

System.out.println(

"public static final String PipeID = \"" + pid + "\";");

}

public static void main(String[] args) {

mdidgen mgen = new mdidgen();

mgen.startJXTA();

mgen.printIt();

}

public void startJXTA() {

Properties myProp = null;

FileInputStream inFileStream = null;

try {

inFileStream = new FileInputStream(new File(ConfigFileName));

myProp = new Properties();

myProp.load(inFileStream);

} catch (Exception ex) {

System.out.println("profile access problem");

}

if (myProp != null) {

groupName = myProp.getProperty("group", "");

}

/***

* Start JXTA and join the group

*/

PeerGroup tpPG = null;

try {

// create, and Start the default jxta NetPeerGroup

group = PeerGroupFactory.newNetPeerGroup();

if (groupName.length() != 0) {

try {

tpPG = joinGroupIfExists(groupName);

} catch (Exception ex) {

System.out.println("Err: " + ex);

}

if (tpPG != null)

group = tpPG;

}

} catch (Exception e) {

System.out.println("ERROR : group creation failure");

}

if (group != null) {

discovery = group.getDiscoveryService();

pipes = group.getPipeService();

}

}

}

116