Collaborative Virtual Environmentspetriu/ELG5124-NDG-CollabVirtEnv-2004.pdf · Nicolas D. Georganas DISCOVER Laboratory @ SITE: Distributed and Collaborative Virtual Environments

Nicolas D. Georganas

Collaborative Virtual Environments

Nicolas D. Georganas, FRSC, FIEEECanada Research Chair in Information Technology

Distributed and Collaborative Virtual Environments Research Laboratory (DISCOVER)

School of Information Technology and EngineeringUniversity of Ottawa

[email protected]


Presentation

• Virtual Reality & Collaborative Virtual Environments

• Applications• Standards• Performance Issues


Virtual Reality• Virtual Reality or Real Virtuality?• Originally, the term referred to 'Immersive

Virtual Reality.' – the user becomes fully immersed in an

artificial, 3-D world that is completely generated by a computer.

• The head-mounted display (HMD) was the first device providing its wearer with an immersive experience (1965)


Virtual Reality

• Desktop VR– Two dimensional medium , used for 3-D

• Augmented Reality– the ability to see the real world and the virtual

world at the same time. • Immersive VR

– the user becomes fully immersed in an artificial, three-dimensional world that is completely generated by a computer


CAVE


DISCOVER Laboratory @ SITE: Distributed and Collaborative Virtual

Environments ResearchDISCOVER Laboratory ($4 million):

ONYX 3400 with 3 graphic pipes and 12 processorsMechdyne FLEX 3-panel screen$1 million of other VR and Haptic equipment

Professors:N.D.Georganas (Distributed Virtual Environments,

Augmented Reality, Tele-Haptics, Real-Time HLA)E.M.Petriu (Virtual Prototyping, Virtual Theater,

Tele-Haptics, Intelligent Sensors)E.Dubois (Virtualized Environments)A.El Saddik (Tele-Collaboration, Web Engineering)W. Lee (Human Animation)S. Shirmohammadi (Tele-Collaboration)


DISCOVER Flex CAVE


Zaxel Virtual Viewpoint

-12 camera capture and avatar/object production


DISCOVER CAVE Stations


DISCOVER CAVE Stations


Haptics

• Hapicts: derived from the Greek verb “haptesthai” meaning “to touch”

• Haptics introduces the sense of force and touch in human-computer interaction

• Tele-haptics can be defined as the use of haptics in a distributed network context


DISCOVER Haptic Devices

• SensAble Phantom Desktop

6 DOF Input. 3 DOF output.High-Fidelity force feedbackDirect parallel port connectionVery impressive


DISCOVER Haptic Devices (cont.)


DISCOVER Haptic Devices (cont.)• Four MPB Freedom

6S6 DOFHigh-Fidelity force feedbackConnects through HRTCC box to your computer


DISCOVER Haptic Devices (cont.)

Immersion CyberForce/ CyberGrasp/ CyberGlove


DISCOVER Haptic Devices (cont.)HAPTIC MASTER


• a simulated world runs not on one computer system, but on several.

• The computers are connected over a network (possibly the global Internet) and people using those computers are able to interact in real time, sharing the same virtual world.

• On each host there are a number of "entities"(things in the virtual environment) that communicate their changing state by sending "update messages".

Distributed Virtual Environments


The Origin of Distributed Virtual Environments

• Department of Defense Networked Virtual Environments – SIMNET – Distributed Interactive

SimulationPromote IEEE standard

• Networked Games and Demos – SGI Flight and Dogfight – Doom – Other GamesInteractive graphics applications


Distributed Virtual Environment

---------------------------------------------

--------------------------------------

-------

CVE

Real-time Interaction-Between the customers-Between customers and salesmen-Between customers and goods

Shared Environment-Global coordinate system-Static background-Dynamic entities

Mutual awareness-Customers can “see” and “talk” to each other in the CVE

--------------------------------------

-------

Server


Collaborative Virtual Environments (CVE)

• Distributed Virtual Worlds• Users represented there by their “avatars”

(=incarnations)• Collaborative work performed in CVE• A Collaborative Virtual Environment

requires an unconventionally broad range of networking, database and graphics capabilities.


CVE Data Types• real-time audio and video data: data carrying the

video or audio streams in the environment.• object/scene description data: data describing an

object and its attributes; e.g., description of an avatar.• CSCW data: this represents traditional 2D

collaborative data such as whiteboards.• control data: used by the system to perform tasks

such as consistency control, management, and so on.

• update messages: such as avatar/vehicle motions.• synchronous collaborative update messages:

update messages representing a synchronous collaborative task.


CVE Issues

• human-factors research in VR has traditionally focused on the development of natural interfaces for manipulating virtual objects and traversing virtual landscapes.

• CVEs require the consideration of – how participants should interact with each

other in a shared space, in addition to how co-manipulated objects should behave.


CVE issues• Other issues include:

– how participants should be represented in the collaborative environment;

– how to effectively transmit non-verbal cues that real-world collaborators so casually and effectively use;

– how to best transmit video and audio via a channel that allows both public addressing as well as private conversations to occur; and

– how to sustain a virtual environment even when all its participants have left.


Causality in CVE

Time

Simulator A(tank)

Simulator B(target)

“fire”

“targetdestroyed”

Simulator C (observer)

eventmessage

real world simulated world

the observer should see the tank fire before the target is destroyed.

• “Things” happen in the real world in a certain order (e.g., cause & effect).• It should appear that events in the simulated world happen in the same order as the real world actions that they represent.• If the message for the “fire” event is delayed in the network, the observer will “see” the target is destroyed before the tank fired upon it!


Presentation




Are needed in order to better train medical students and doctors alike;

Allow errors to be made on virtual rather than real patients;Allow rare cases to be modeled and studied;Alleviate the animal rights controversy and the shortage of

cadavers problem;Permit more enjoyable and efficacious treatment for patients

(example rehab);Allow more realistic examination (board exams);Allow reduction in medical costs (less liability suits, cheaper

training – emergency response).

MEDICAL APPLICATIONS OF VRMEDICAL APPLICATIONS OF VR


DISCOVER Tele-Surgery (Trachea) Training








Virtual Environment Technology Laboratory, Univ. of Houston and

NASA Johnson Space Centre, US

• Shared Virtual Environments for Training http://www.vetl.uh.edu/overview/overview2.html


Human Interface Technology Laboratory (HITL), Univ. of Washington, US

• Shared Space: Collaborative Augmented Reality• http://www.hitl.washington.edu/projects/shared_space/


Human Interface Technology Laboratory (HITL), Univ. of Washington, US

•collaborative Web browser in which users can see each other and virtual Web pages simultaneously. Using a see-through display, participants are naturally aware of each other's facial expressions, body language, gestures and gaze.


Sony Computer Science Lab., Sony Co. , Japan

TransVision: A Collaborative Augmented Reality Testbed

http://www.csl.sony.co.jp/person/rekimoto/transvision.html

•Augmented reality for collaborative designing

•The system uses the palmtop video-see-through display instead of bulky head-mounted displays. The user can see a computer-generated 3D model superimposed on the real world view


SICS - Swedish Institute of Computer Science, Sweden

Multi-user Virtual Workplace Conferencing


Virtual Theater projectDISCOVER Lab, UofO

VisualFeedback(s)

Machine/ Human Interface

TactileFeedback(s)

ForceFeedback(s)

VideoSensor(s)

TactileSensor(s)

ForceSensor(s)

StructuredLight

Human/Machine Interface

Object Recognition

Motion TrackingVirtual Object Manipulation

Object Shape &Behavior ModelsObject Interaction Models

Computer Generated Models Animation Script

Pseudo-Random Encoding

VIRTUAL SCENE / THEATER (where virtual objects interact)

Head-Tracking system

Body Tracking System


VIRTUAL THEATER for Industrial Training: A Collaborative Virtual

EnvironmentDISCOVER Lab , Univ. of Ottawa


CVE for Industrial Training• Reduces Training Expenses

– Waste of equipment which likely gets damaged– Maintenance of Training Facilities– Training of users in remote locations

• Very little cost associated with creating virtual objects– It can be replicated as many times as necessary

• Any damage done to the object does not incur in financial loss

• Maintenance fee is very low• Actual objects may be used for its intended use


Industrial Training Prototype


Head Tracking

• Head tracking allows natural way of controlling the avatar’s head.

• Users have a better awareness of others while listening to their voice and watching their avatar’s animation

• Extension to body motion detection allows Virtual Theater like collaboration

• 3D Scene is projected in a wall for stereoscopic view through 3D shutter glasses and more natural experience for the users.


V-COM: CVE in e-CommerceDISCOVER Lab, UofO

User ProfileManager

User

VirtualWorld

RTI VirtualWorld

VirtualShopping

Mall

RTI VirtualShopping Mall

Security Manager

QoS Broker

Sales Agent

CGIRegistration/Login

Negotiation

Search Agent

CGIQuery

Transactions

Multimedia catalogs

HTTP Server HTTP Server

Vendors


User Interface


Collaborative e-Commerce

federate Afederate B

HLA MiddlewareTCP/UDP/IP Network

federate C

federate A


Mobile Collaborative Augmented Reality

DISCOVER Lab, UofO

• Computer Supported Collaborative Work(CSCW) – Allows a computer to be used as a medium

for human communication • Mobile Collaborative Augmented Reality

– The combination of augmented reality, mobile computing and CSCW promises exciting applications for industrial training


Augmented Reality (AR) Prototype

A trainee works with a wearable computer

Where and which virtual objects should be superimposed on the real world?

A marker with a square region and some pattern inside of the square region is used

Supports multiple trainees and trainers distributed in different locations


AR Prototype (cont’)

(a) A physical board with virtual chips around

(c) A virtual chip is laid on the board

(d) The virtual chip is rotated by remote trainee B

(b) Local trainee A is moving a virtual chip


Presentation




VR Standards: Rendering• OpenGL

– Introduced by SGI in 1992– Multiplatform– Hardware acceleration

• DirectX– Created by Microsoft– Supports Win9x, WinNT (DirectX 3) and Win2000– Hardware acceleration

• VRML– VRML 1.0 based on SGI’s Open Inventor– VRML 2.0 in 1996– ISO standard in 1997– Browsers based on OpenGL or DirectX

• Java3D– Introduced by Javasoft in 1998– Multiplatform– Build on top of OpenGL and ActiveX


VRML BrowsersVRML-Browsers are software tools that generate views from a VRML file.


•VRML is a file format •It enables authors to control shapes and some animation and interaction •For more control, authors may write short programs in Java or JavaScript •To gain greater control, something more is needed...

•Java3D is a Java Application Programmer's Interface (API) •Java3D is a set of standardized classes for use by Java programmers •It enables authors to control shapes, animation, and interaction, PLUS rendering,

input devices, and lots more •It is (roughly) a Java-based superset of VRML


CVE Standards: Middleware• ALSP and DIS• Living Worlds (LW)

– Designed by the VRML Consortium– Aims at providing collaboration features for VRML– VRML 3.0 = VRML 2.0 + Living Worlds

• Open Community (OC)– Proposed by Mitsubishi– SPLINE (Scalable Platform for Large Interactive Networked

Environments) is a OC compliant implementation– SPLINE uses ISTP (Interactive Sharing Transfer Protocol)

• High Level Architecture (HLA)– Developed by the U.S. Defence Modeling Simulation Office (DMSO)– RTI (Run Time Infrastructure) is an HLA compliant implementation– HLA is now an IEEE standard (IEEE 1516)

• Java Shared Data Toolkit (JSDT)– Part of Java Media API developed by Javasoft– Java provision for real-time sharing of applets and/or applications


A History of DVEs1980

1985

1990

1995

SIMNET Start (83)

SGI Flight (84)SGI Dogfight (85)

Amaze (84)

SIMNET First Demo (86)NPS FOG-M (86)NPS VEH (87)NPS MPS-1 (88)

SIMNET to Army (90)NPSNET-1 (90)

BrickNet (91)

DIS (93)

DIVE (92)NPS-Stealth (93)

NPSNET-IV (93)Doom (93)Paradise (93)

HLA (97)


SIMNET

• SIMNET (simulator networking) is a distributed military virtual environment.

• SIMNET was begun in 1983 and delivered to the U.S. Army at the end of March 1990.


Distributed Interactive Simulation (DIS) protocol

• DIS (IEEE 1278), and its predecessor SIMNET, arose from real-time interactive computing.

• DIS is a government/industry initiative to define an infrastructure for linking simulations of various types at multiple locations to create realistic, complex, virtual worlds for the simulation of highly interactive activities.

• DIS is a virtual environment within which humans may interact through simulation at multiple sites that are networked using compliant architecture, modeling, protocols, standards and databases.

• DIS exercises involve the interconnection of a number of simulators in which simulated entities are able to interact within a computer generated environment.

• The simulators may be present in one location or be distributed geographically and the communications are conducted over the network.


Distributed Interactive Simulation (DIS)

The DIS (IEEE 1278) standard.Defines 27 different PDUs, only 4 of which (Entity State, Fire, Detonation, and Collision) are used by nodes to interact with the virtual environment.

A fully distributed, heterogeneous network software architecture.


LiveParticipants

SupportUtilities

Interface

Interfaces to Live Players

Simulations

Runtime InfrastructureFederation Management Declaration ManagementObject Management Ownership ManagementTime Management Data Distribution Management

High Level Architecture (HLA)

• OMG, IEEE, DoD standard


Federate FederateAmbassador

RTI Ambassador


RTI Ambassador

Components in a Federation Execution

RTI Executive( rtiexec.exe )

Federation Executive( fedexec.exe )

FEDFileFEDFile

Network

Inter-Process Communications


RTI Ambassador

Network

Inter-Process Communications

RTI Executive Federation Executive FEDFileFEDFile


HLA Services

Federation Management

Declaration Management

Object Management

Ownership Management

Time Management

Data Distribution Management


Example of Distributed Virtual Environment over HLA

federate Afederate B

HLA MiddlewareTCP/UDP/IP Network

federate C

federate A


Open CommunitySPLINE


Open Community, SPLINE

• SPLINE (Scalable Platform for Large Interactive Networked Environments) is a software library compliant with O.C. which provides an API for C (and promises of Java soon).

• ISTP (Interactive Sharing Transfer Protocol) is a hybrid protocol used by SPLINE which supports the sharing of information about a Virtual World among a group of user processes.

• OC (Open Community) is a proposed standard which provides many essential services necessary to make real-time multi-user cooperative environments possible.


OC (SPLINE) Functionality• Network Communication issues

– ISTP• Real-time audio transport ( no video)• Application neutral transport of large or

complicated objects– VRML models, sound files, etc.

• Filtering of Regions of Interest• Scalability to large Virtual World and

large number of users allowing users in not so powerful stations to participate.


OC World Model - High Level Overview


SPLINE: Scalability• World is partitioned in regions called locales.

– Each process is supposed to render a reduced number of locales (1- It’s own locale, 2- Neighbour locales)

• Each locale has its own multicast addresses– Each process only deals with relevant messages/audio

• Each locale has its own coordinate system– Different parts of the VW can be designed

independently


SPLINE: Locales• Each locale has a list of adjacent locales

– This list includes coordinate transformations as well

– Portals (non-Euclidean motion)• Locales may overlap, be enclosed or contain

empty spaces


Presentation




SPLINE - Performance

44.4688.8

170.65239.23

300.3

0

50

100

150

200

250

300

350

1 25 50 75 100

spWMUpdate Refresh Interval

End-

to-E

nd D

elay

Ownership transfer between 2 clients connected in confined 10Base2 Network (VRML browser access via JNI and DLL)


Performance for Various CVE Middleware Standards

RTI with VRML

RTI with Java3D

JSDT SPLINE with

VRMLCCD (Client-to-Client Delay)

132 112 - 129

CommunicationDelay *

89 89 60 85

OTD (Ownership Transfer Delay)

225 192 5 217

* This is CCD minus the delays between the middleware and the rendering part (VRML/Java3D).


The Updates Distribution Problem• When the number of participants in a CVE grows

sufficiently, the amount of updates each participant has to deal with becomes overwhelming.

• Some architectures proposed allow a large number of users to collaborate; however they all fail under certain conditions.

• It is usually very hard to balance the load in the various systems participating in a VE simulation, yet worst if a great range of systems with different processing power is employed (heterogeneous systems):

– Everything is usually handled based on the slowest system; or

– Any system which does not provides a minimum capacity is simply denied access to the session


Limitation of Existing Models

• Space based solutions assume that users are uniformly spread in the CVE.

– Reducing the area you see must reduce what you see...



• It fails to filter incoming dataif a sufficiently large amountof users is packed togetherin a single “locale”.

• Space based solutions assumethat users are uniformly spreadin the CVE.

– Reducing the area you see must reduce what you see… Not necessarily...



• All users in the same area receive the same load, which overloads and/or under uses heterogeneous systems.


VELVET: An Architecture for Very Large Virtual Environments

(DISCOVER Lab, UofO)

• Adaptive Architecture– Allows the system to adapt itself according to the load.– Allows diverse CPU powered stations to participate in

the simulation.– Best effort approach (a user will “see” as much as

his/her station allows)

• Hybrid Architecture– Builds on top of space based (i.e., at light load it may

work just like a space-based solution, such as SPLINE)


VELVET - Best Effort Adaptation










Check-in/Check-out• When an object crosses the edge of the Area of Interest

it checks-in/out.• There is unnecessary processing performed if a given

object stays at the border for a long time with light motion back and forth, which generates of large amount of check-in/out operations

• VELVET defines two Areas of Interest aiming at avoiding the problem described above

– AI-Check-In (AICI)– AI-Check-Out (AICO)


Check-in/Check-out

AICOAICI


Check-in/Check-out

AICOAICI


Check-in/Check-out

AICOAICI


Check-in/Check-out

AICOAICI


Check-in/Check-out

AICOAICI


Check-in/Check-out

AICOAICI


Check-in/Check-out

• The distance between AICI and AICO may be controlled in such a way to increase if a given number of Check-In/Out operations are performed aiming at reducing them in the future and vice-versa.


Parallel Virtual World• A way to see the world

based on what matters the most, rather than what is geometrically closer (Euclidean space).

– World is organized according to a pre-defined metric.

– Area of Interest is actually defined in the PVW, rather than the CVE itself.


Simulation Results


Conclusion• Distributed and Collaborative Virtual

Environments open a new avenue for multimedia human interaction, collaboration and learning

• Many Applications:– Defense Training– Medical training – Industrial Training, – Telecollaboration, – E-commerce,….

• Phychological and social issues: an open area for study.

Nicolas D. Georganasمتشکرم

Documents

Collaborative Virtual Environmentspetriu/ELG5124-NDG-CollabVirtEnv-2004.pdf · Nicolas D. Georganas DISCOVER Laboratory @ SITE: Distributed and Collaborative Virtual Environments