1 5/4/99ISORC ‘99 BBN Technologies An Object-level Gateway Supporting Integrated Property Quality of Service Rick Schantz John Zinky, David Karr, Dave

1 5/4/99 ISORC ‘99 BBN Technologies

An Object-level Gateway SupportingIntegrated Property Quality of Service

Rick SchantzJohn Zinky, David Karr, Dave Bakken,

James Megquier, Joe Loyall, Partha PalBBN Technologies

ISORC ‘99Saint Malo, France

May 4, 1999

“Adapt or perish, now as ever, is nature's inexorable imperative.” – H. G. Wells

QuOQuO

DARPA

Air Force Research Lab/Rome Laboratory


Outline

Motivation & Context

Background & Overview of QuO technology

The Object Gateway ComponentExamples of UseDesign Considerations

Experience to Date & Current Status

QuOQuO


Many Distributed Systems have Critical QoS Requirements, e.g., Performance, Dependability, and Survivability

CINCJ2

CINC OperationsPlanning Group

OPS/ INTELWORKSTATION

TARGET

CJTFPlanners

TARGET

MissionsCenters of GravityTask refinementForces refinementPhases refinementCOA evaluationCOA selection

FLTCINC MaritimePlanning Center

TMS CASES & HPC

Maritime & AirCampaign Assessment

JOINT FORCE MARITIMECOMPONENT CMDR

CASES TARGET

Task refinementIntegrated Target PrioritiesForces refinementSchedule refinementCOA evaluation

JOINT FORCE AIRCOMPONENT CMDR

TARGET ACPT

IDB

MissionsCOAsStrategy

Crisisassessment

COA evalCOA eval AirMar

XIDB

Target Nomination ListMaster Attack PlanAttack Plan Status

JFACCCombat Ops

Target Nomination ListWeaponeering

XIDB

APS/FLEX

TAMPS/COMPASS

JMCISIDB

Air Tasking Order (ATO)

NATOOPS/INTELWORKSTATION

Rear-echelonSim Center

toSHAPE

MasterTargetList

APS/FLEX RAAP

Refinement& eval

collaborative development

rehearse

Refinement& eval

ATO

Target data / Weaponeering


Chronology

1975 1980 1985 1990 19951970

ARPANET EthernetIP/TCP

OSF

-DC

E

OMG/CORBA

PC

JAV

A

OL

E/D

CO

M

SBSRSEXEC

National SoftwareWorks (NSW)

CRONUS/CORBUS QualityObjects (QuO)

NG

I?

WW

W[Simnet]

C+

+

TE

NE

X/

Multics

UN

IX

MS-D

OS

200?

Distrib O

bjects

MessageP

assing

RP

C

Distrib SQ

L


State of the Art Today

• No effective way to control behavior of critical applications in today’s highly networked environment

• Gap between– Emerging low-level mechanisms to control some types of resources– High-level strategies appropriate for adaptable, dependable systems

• Researchers generally working on one piece of the solution e.g.– Focus on one critical QoS property: real time behavior, fault tolerance ...– Focus on implications of one time epoch; (boundaries are fuzzy and getting

fuzzier with improved dynamic binding)– Functional integration has taken precedence over system properties

• End goal is support building integrated QoS, adaptive applications (with varying requirements on granularity of changing behavior):

– adaptable: change at runtime while application/service running• reconfigurable: change at runtime while application/service halted• evolvable: change at development time


The Quality Objects (QuO) framework supports development of distributed applications with QoS

QuOQuO

The QuO framework provides• Separation of concerns between software functional properties and QoS needs

• Consistent interfaces for QoS measurement and resource management control

• Standard middleware interfaces between application and QoS-provider layers

• Facilities to enable application- and system-level adaptation

QuO is being developed as a common approach to adaptable QoS for a number of projects focusing on different QoS dimensions:

– Dependability, replication, group communication (U.Illinois, Cornell, UCSB)– Managed bandwidth, resource reservation (CMU, Columbia Univ.)– Real-time behavior (Washington Univ in St. Louis)– Security, access control, survivability, intrusion detection (TIS)

– Integrating different dimensions in an application, e.g., real-time and dependability– Suitability for supporting different approaches to a dimension


Outline





QuOQuO


QuO’s emphasis is on specification, measurement, control, integration, and adaptation, not enforcement

• QuO is based upon distributed object computing– Prototype system is based on CORBA, but the concepts are equally applicable to DCOM, RMI

• QuO provides a combination of high-level languages for specifying aspects of handling QoS– Aspect Oriented Programming – Code generators generate executable code from the high-level descriptions

• QuO enables the application to control and measure QoS and adapt to changes in the level of QoS provided• If a manager or mechanism(s) exists that enforces QoS, QuO provides the means to interface to it


System condition objects monitor QoS in the system

• system condition objects recognize changes in the system and notify the contracts that observe them

• QuO contracts notify client programs, users, managers, and other system condition objects through transition behavior

SystemCondition Objects

QuO applications specify, control, monitor, and adapt to QoS in the system

Application

Alternate Implementations

Contract (operating regions)

Servers

Network

ORB

Replication Mgr

Resource ReservationManager

IDS

Specification of operating regions, alternate implementations, and adaptation strategies using QuO’s QDL

Multiple layers of adaptation• managers and mechanisms can

adapt to changes in the system• QuO contracts provide another

layer of adaptation• Client and user can also adapt

Mechanisms and managers control QoS in the system

• a layer below QuO that provides ORB-level services, such as managed communi-cation, replication, or security

• contracts and delegates interface to these services through system condition objects


Simplified DOC (CORBA) Runtime Components

Client Network Server

ApplicationDeveloper

MiddlewareDeveloper

Logical Method Call Client

ORB Proxy

COTS ORB

Object

ORB Proxy

COTS ORB

Network


A QuO application contains additional components (from traditional DOC applications)

• Contracts summarize the possible states of QoS in the system and behavior to trigger when QoS changes– Regions can be nested, representing different epochs at which QoS information becomes available, e.g., negotiated regions represent the levels of service a client expects to receive and a server expects to

provide, while reality regions represent observed levels of service– Regions are defined by predicates over system condition objects– Transitions specify behavior to trigger when the active regions change

• System condition objects are used to measure and control QoS– Provide interfaces to system resources, client and object expectations, mechanisms, managers, and specialized ORB functions– Changes in system condition objects observed by contracts can cause region transitions– Methods on system condition objects can be used to access QoS controls provided by resources, mechanisms, managers, and ORBs

• Delegates implement the QoS specific adaptivity behavior– Upon method call/return, delegate can check the current contract state and choose behavior based upon the current state of QoS– For example, delegate can choose between alternate methods, alternate remote object bindings, perform local processing of data, or simply pass the method call or return through


Measured capacity >= 10

As_expected:

Insufficient_resources:Measured capacity < 10

Contracts summarize system conditions into regions and define transitions between them

• The contract defines nested regions representing possible states based upon measured system conditions• Predicates using system condition objects determine which regions are valid• Transitions occur when a region becomes invalid and another becomes valid• Transitions might trigger adaptation by the client, object, ORB, or system

Normal:Expected capacity >= 10

Degraded:Expected capacity < 10Expected capacity >= 2

As_expected:

Extra_resources:

Measured capacity < 10Measured capacity >= 2

Measured capacity < 2Insufficient_resources:


Unusable:Expected capacity < 2

As_expected:

Extra_resources:

Measured capacity < 2


= Expected Region

= Reality Region


System Conditions Project a Value to the Application, But also Must Maintain the Value

SimpleValue Measured

Value(Sensor)

ComposedValue


QoSDeveloper

Mechanism Developer

QuO Kernel

ControlValue

Specialized ORBs or Services

RSVPController

ControlValue

StatusValue

CORBA Object

DeviceStatus

Service


Delegates change their behavior based on their contract’s current regions

Dispatch

ContractPa

ss T

hrou

gh

Blo

ck

Shap

e

PreMethodMeasurements

Method Call

Lower Method Call

Current Regions

Result

Lower Result

SysCond

SysCond

Current Regions

PostMethodMeasurements

Delegate

AlternativeBehaviors


The QuO Toolkit provides tools for building QuO applications

CORBA IDL

CodeGenerators

CodeGenerators

Contract DescriptionLanguage (CDL)

QuO RuntimeQuO RuntimeStructure DescriptionLanguage (SDL)

Delegates Contracts

• Quality Description Languages (QDL)

– Support the specification of QoS contracts (CDL), delegates and their adaptive behaviors (SDL), connection, creation, and initialization of QuO application components (TBD)

– QuO includes code generators that parse QDL descriptions and generates Java and C++ code for contracts, delegates, creation, and initialization

• System Condition Objects, implemented as CORBA objects

• QuO Runtime Kernel– Contract evaluator– Factory object which instantiates

contract and system condition objects


QuO adds QoS control and measurement into the DOC remote method call

Client Network Server


Qosketeer

MechanismDeveloper


Delegate

ORB Proxy

Specialized ORB

ContractSysCond

SysCond

SysCond SysCond

Object

Delegate

ORB Proxy

Specialized ORB

Contract

Network

Mechanism/PropertyManager

SysCond

SysCond

SysCond


Outline





QuOQuO


QuO GatewayQuO Gateway

IIOPGlue

Control

QuO Gateways Support Specialized Communication Protocols and Below the ORB Mechanisms

Cli

ent-

Side

OR

B

IIOP Group Replication (AQuA)

WAN

Bandwidth Reservation (DIRM)

IIOP over TCP/IP (default)

IIOPGlue

Control

IIOP

Serv

er-S

ide

OR

B

• The QuO gateway enables insertion of below-the-ORB mechanisms and specialized network controls

• The gateway translates IIOP messages into specialized communication protocols or network level controls

• To the client-side, the QuO gateway looks like the remote ORB

• To the object-side, the QuO gateway looks like the client’s ORB

• The two ends of the gate-way are on the same LAN as the client/object

• Currently, we have gate-ways that support Ensemble group communication, RSVP resource reservation, and IIOP over TCP/IP


QuO Based Bandwidth Management Measures and Controls Network Resources

ORB Proxy

Commercial ORB

contract

MeasuredPerformance

Skeleton

Contract

BandwidthControl

BandwidthControl

UserExpectation

Commercial ORB

Logical Method Call

ExpectedPerformance

RightsPolicies

MeasuredPerformance

ApplicationManager

StatusCollection

NetworkManagement

RightsPolicies

Server Host

Skeleton

Client Object

Delegate Delegate

Sensors

QuO Gateway

SNMP SNMPcustom

Corba CORBA

ORB Proxy

CORBACORBA

With Trace RecordWith Trace Record

FunctionalFunctional

SignalSignal

Client Host Network

IIO

PG

lue

IIO

PG

lue

IIOPResource

AwareTransport

ResourceAware

Transport

IIOP

RSVP bandwidth reservation

Darwin bandwidth reservation HostStatusSensor

HostStatusSensor

Functional

Sensors

Functional


An RSVP Aware Gateway can Set up and Tear Down RSVP Sessions Between Hosts.

QoSME from Columbia University will be used to control RSVP for the <X>-ORB

QuO-GTWY

QoSME

RT. RSVP

QoSME

QuO-GTWY

Client

BandwidthReservationAPI

RT. RSVP

RSVP Reservation

Object

ORB Proxy

SmartsInCharge

CORBA QoSAPI

QoS SocketAPI IIOP IIOP

QuO D & CQuO Delegate and Contract

ORB Proxy SysCond

QoSMib

Application ManagementAPI

SNMP


A simplified example: Improved Dependability through Replication

Contract

Low Cost:Requested = 1

Too Low: Measured < 1

Normal: Measured > 1

Available:Requested >= 2

Too Low: Measured < 2

Normal: Measured > 2

Object Replica Group

Client application

CodeRemote method call

Object Delegate

Region?

Normal Multicast


Replication Groups: composed of oneor more identical objects

Connection Groups: composed of themembers of two replication groupsthat wish to communicate

• • • • • • • • •

• • • •• • •

•

•

•

Connection group 1

Connection group 3

Connection group 2

Replicationgroup 1

Replicationgroup 4

Replicationgroup 2

Replicationgroup 3

Proteus manager 1

Proteus manager 2

Proteus manager 3

PCS group

•

•

•

•

•

Object Factory 1

Object Factory 2

Object Factory 3

Object Factory 4

Object Factory 5

Point-to-Pointgroups

PCS Group: composed of the Proteusreplicas and objects that desire tocommunicate with the Proteus manager

Point-to-point Groups: composed of onedependability manager replica and one object factory

Groups in AQuA


CORBAORB

Client IIOPGlue Dispatcher

Handler #1

VoterGroupMember Buffers

Maestro/Ensemble

Object Gateway-Half ProcessApplication Process


Network

DTEL++ and OO DTE add access control into the DOC remote method call

Network Server


DTEL++SecurityDeveloper

MechanismDeveloper


ORB Proxy

ORB

Object

ORB Proxy

ORB

DTE metadata

OO domain definitions

type bindings for interfaces, and operations;policy distribution manager

ORB GatewayOO DTE

Client


The QuO Gateway Manages IIOP Connections and Interfaces to Protocols which Manage QoS

• Shell is a Base to build Specialized QoS Aware Gateways

• To the “Client” ORB, the QuO Gateway looks like the object

• To the “Server” ORB, the QuO Gateway looks like a client

• The ends of the gateway are on the same LAN as the Client/Object and may be on the same host

• IIOP Glue will multiplex and demultiplex GIOP messages

IIOPGlue

Specialized Protocol IIOPGlue

Control

Client Server

ControlWAN

IIOP IIOP

Client-Side Gateway half Server-Side Gateway half


Client-Side Interface Presents a Partially Parsed GIOP Request and Call Block for the Reply

• The DSI interface returns a Server_Request structure.

• Additional information is derived from call context

• The suspended thread is remembered in Call block

• This info is bundled into a Call block and puts it on the Request_Queue for the specialized transport

• When the specialized transport attaches the reply and puts it on the Reply_Queue for the DSI wrapper.

MechanismSpecificHandler

IIOPGlue

ClientIIOP

DSI

Control

Server Request

Request Queue

Reply Queue

ContextCall

Block

Thread Suspend

CallBlock


Server-Side Interface Accepts a Call Blockand Creates a DII Request.

• The Call Block is removed from the Request_Queue

• The DII interface sends a Request structure to an object reference

• The return values is bundled into the Call block and put on the Reply_Queue for the specialized transport

IIOPGlue

ServerIIOP

DII

Request

MechanismSpecificHandler

Control

Request Queue

Reply Queue

Obj RefCallBlock

Thread Suspend

CallBlock


IIOPGlue

IIOPGlue

RSVP Network

RSVPd

Bandwidth Aware

Transport

Instrumentation

Bandw - Control

SNMP

CONVERT

QoSMEBandwidth Reservations

RSVPd

Bandwidth Aware

Transport

Instrumentation

Bandw - Control

SNMP

QoSME

CONVERT

Specialized Embedded Protocols:Network Resource Reservations


(Leader)

C-Rep1ORB

GW

Group Comm

GW

ORB S-Rep1

C-Rep2ORB

GW

GW

ORB S-Rep2

C-RepNORB

GW

GW

ORB S-RepM

...

...(Leader)

12

4

5

6

7

12

5

6

7

12

5

7

5

333

Specialized Embedded Protocols:Group Communications


Outline





QuOQuO


Status

• QuO version 1.0 released September 1998, including the QuO runtime kernel, CDL and SDL and their code generators, system condition object library, concept demonstration gateway, example applications, and documentation

• Version 2.0 scheduled for end of May 1999– improved language and contract support– a revised general gateway mechanism supporting RSVP– instrumentation support– Quorum Distributed Object individual QoS property incremental results:

DIRM bandwidth management using RSVP & CMU Beagle

AQuA dependability using Proteus and Ensemble

TAO real time ORB and schedulableevent channels

TIS ORB based DTE access control with dynamic security policy (tentative)


QuO Architecture

Delegate

ORB Proxy

Specialized ORB

SysCondDelegate

ORB Proxy

Network

Mechanism/PropertyManager

SysCond

SysCondSysCondSysCond

SysCond

SysCond

Specialized ORB

WAN

CORBA IDL

CodeGenerators

CodeGenerators

Contract DescriptionLanguage (CDL)

QuO RuntimeQuO RuntimeStructure Description

Language (SDL)

Delegates

Client Object

QuO GatewayQuO Gateway

Control

Cli

ent-

Sid

e O

RB

Group Replication (AQuA)

Bandwidth Reserve (DIRM)

IIOP over TCP/IP (default)

Control

Obj

ect-

Sid

e O

RB

IIO

PG

lueIIOP

Logical Method Call

Contracts

IIO

PG

lue IIOP

• Quality description languages– currently CDL (QoS contracts), SDL

(adaptation strategies), simple connection language

– soon to come: capabilities to describe real-time behavior (TAO’s RIDL) and security (TIS’s DTEL++), plus improved SDL and connection

• QuO runtime kernel• System condition libraries, example

applications, documentation• QuO gateway and instantiations for

group communication, resource reservation, TCP/IP, and (soon to come) security

• QuO instrumentation


QuOIN - Distributed Object IntegrationDARPA

SM

OO domain definitions

type bindings for modules, interfaces, and operations

ORB Gateway

OO DTE

SSIGMAIGMATrusted Trusted Information Information SystemsSystems

DarwinControl Obj

RemosCollector

Network Management

SNMP SNMPBeagle

Network

Darwin Bandwidth Reservation Remos

HostLoad

RSVP Network

RSVPd

Bandwidth Aware

Transport

Instrumentation

Bandw - Control

SNMP

RSVPBandwidth Reservations RSVPd

Bandwidth Aware

Transport

Instrumentation

Bandw - Control

SNMP

QoSME QoSME

RemosHostLoad

QuO Based Bandwidth Mgmt

QuO Based Dependability Mgmt

Caching and Consistency

Georgia Tech

Common IDL Compiler

Specialized Realtime ORB

OO Security

Realtime Control Language

Cached Data & Impl

Consistency Protocols

Expectations Callbacks

ContractObject

Delegate

IIOP / Proteus / Ensemble Interface

ApplicationLogic

SystemConditions

ProteusDependability

Manager

Application

QuO

AQuAGateway

Managed Network with Ensemble

ObjectReplica

ObjectReplica

...

ObjectReplica

ObjectReplica

ObjectReplica

ObjectFactory

ObjectFactory

ObjectFactory

ProtocolCoordinator

Advisor

AQuAGateway

Documents

1 5/4/99ISORC ‘99 BBN Technologies An Object-level Gateway Supporting Integrated Property Quality of Service Rick Schantz John Zinky, David Karr, Dave