Interactive Simulations and advanced Visualization

with Modelica

Tobias BellmannInstitute of Robotics and Mechatronics, German Aerospace Center (DLR)

Munchner Straße 20, 82234 Weßling, Telefon: +49 8153 28-0, Fax: +49 8153 28-2243

Abstract

In this paper a Modelica library for interactivesimulation and advanced visualization called Ex-ternalDevices is introduced and presented. Pro-viding support for standard input devices like key-board and joystick as well as for communicationvia UDP and shared memory, this library allowsthe user to interact with a running simulationand process the output data of the simulation inother processes capable of UDP connections. Anadvanced visualization system replaces the stan-dard Dymola visualization and offers additionalfeatures like full-screen viewing, transparency andsupport for flexible bodies.

Keywords: interactive simulation; visualization;simulation; network; flexible bodies

Introduction

Simulations with Modelica normally are not de-signed for interactive control. In the standardModelica 3.1 library, no blocks for input devicesor other control possibilities are existent. Never-theless it can be help- and useful to interact witha running multi-physics simulation, either to re-duce the effort needed for the generation of inputdata for the simulation, or to react directly to theresults of a running simulation.The integrated visualization of the ModelicaMultiBody Library is vendor-specific. It is there-fore limited to the specified visualization methodsprovided by the simulation tool. The visualizationdefinitions of Modelica 3.1 are limited to some ba-sic features like some elementary shapes and un-textured .dxf CAD files.To overcome the missing interaction and visual-ization possibilities, the ExternalDevices Libraryprovides a set of blocks and techniques to allowinteractive simulations, as well as an advanced

real-time visualization of the running simulation,considerably extending the scope of operation es-pecially for multi-body simulations. The Exter-nalDevices library is structured in the followingfunctional packages:

∙ Input devices: Blocks for the direct control ofsimulation states by the user

∙ Communication devices: Blocks allowing thesimulation to communicate with other pro-cesses via network or shared memory

∙ External visualization: Blocks and models re-placing vendor-specific visualization systemsand adding additional visualization possibili-ties.

The ExternalDevices library links either tostatic or dynamic C++ libraries to provide thisadditional functionalities. It is available for Dy-mola 7.x, in a version for Windows, a version forUnix/Linux is planned. Every Modelica imple-mentation able to link external C libraries can usethe ExternalDevices library and the visualizationsystem.

1 Input devices

Input devices are needed for interactive control ofthe simulation states, for example to trigger eventsor to control actors of a multi-body simulation.For this purpose the ExternalDevices library pro-vides blocks for three common PC input devices:

Keyboard: This block allows the monitoring ofsingle keyboard keys, and has a boolean outputfor the chosen key state. Several blocks can beused parallel, each with a selectable key ID likeVK Return for the return key.

Figure 1: Input devices blocks

Joystick: This block includes support for threeaxis, eight buttons joysticks for Modelica. Thejoysticks must be configured and calibrated cor-rectly in the Windows system control panel. Sev-eral blocks at the same time are usable with sep-arate joystick IDs, allowing the use of more thanone joystick attached to the PC. The three out-puts are from the type Real and are normed from-1 to 1 for each joystick axis.

SpaceMouse: The SpaceMouse block includessupport for the 3dConnexion Spacemouse, a sixDOF input device originally developed at the In-stitute of Robotics and Mechatronics of the ger-man aerospace center (DLR). This devices con-sists of a pressure-sensible handle, which canbe pushed and rotated to manipulate objects inthree-dimensional space. The output connectorsof this block also are normed from -1 to 1 and allbuttons of the Spacemouse are retrievable via aboolean output vector.

2 Communication devices

It is often useful to control a simulation via net-work or to process the simulation output datain another program or simulation. The Exter-nalDevices library supplies the user with blocksto communicate with external processes via UDPor Shared Memory. As an example, one simulationcan provide input data for another simulation viaUDP.

Figure 2: Network devices blocks

UDPRecieve block: This block introduces anUDP client communication input for Modelica.The incoming data must be binary coded doublevectors, in the form [double 1,.....,double n] withthe selectable length n. The listening port can beselected and must be unique on the system. Morethan one block in a model may be used under thispremise.

UDPSend block: This block allows the send-ing of Real vectors with selectable length n. Pa-rameters are the target port and IP address. Morethan one block may be used in a model.

UDPServer block: Combination of UDPSendand UDPRecieve functionality.

Shared memory Block: A shared memoryblock, using the QT framework from QT Soft-ware [1] allows the communication between twoprocesses on the same computer system. Support-ing Real, Integer and Boolean vectors it can beused as an interface to other parallel running sim-ulations or processes with the same QT sharedmemory interface. Several blocks can be usedwithin the same model by defining different mem-ory storage IDs.

3 A model-based approach forvisualization

Including the complete Modelica 3.0 standard forvisualization of multi-body models, this libraryfurthermore allows the user to build more complexvisual environments within the Modelica model,to be simulated on an external visualization tool,DLR SimVis. The visualization package uses net-work communication to transmit the visualizationdata from the simulation to the external visual-ization tool. This tool, DLR SimVis, provided to-gether with this library is based on OpenGL [4]and the OpenSceneGraph [2] 3D scenegraph.The main purpose is the visualization ofmulti-body simulations, replacing vendor-specifiggraphic engines with a presentable graphic engine,supporting full-screen viewing and a large varietyof textured 3D CAD file formats.Every object visualized in the external viewer isprovided by a visualization component in Model-ica, containing all necessary data for correct visualrepresentation.

3.1 Object-oriented approach for visu-alization components

The object-oriented approach of the ExternalDe-vices visualization system easily allows to inte-grate the visualization components in to physicalcomponents. The complete informations neces-sary for the visualization are already existent inthe physical component and can be used in thevisualization block. Figure 3 shows the integra-tion of a visualization block in the model of aspring. The information about the spring’s po-sition, orientation and length is available via theframe connectors, other parameters like windingnumber, wire diameter e.g. can be chosen via pa-rameter dialog.

Figure 3: Integration of the visualization compo-nent in a physical model

3.2 Comparison with existing physicalvisualization systems

Most existing visualization systems rely on a cen-tral configuration file defining the scene and theinput channels for a specific visualization task.The MathWorks VR Toolbox [5] for example usesVRML Files, containing the complete kinematicand degrees of freedom for scene manipulationsas well as the CAD Data. In the correspond-ing Simulink model, the VR Block provides theinputs for the desired degrees of freedom in thescene. These inputs are now connected with theaccording signals of the Simulink physics model.This leads to increased configuration efforts if themodel has to be changed, because both the VRMLConfiguration and the signals have to be adapted.Another example for the separation of model andvisualization system is the VisEngine of AerolabsGmbH [6]. The VisEngine uses a configuration filedefining the used CAD data for the scene and theinput channels (e.g. UDP, tables from the file sys-tem, etc.) for moving the CAD objects.

In both cases, a change in the model requires amodification in the visualization. With the object-oriented approach of ExternalDevices combinedwith Modelica, this is not necessary, because ofthe complete integration of the visualization intothe model components.Because of an additional software layer, hiddenfrom the user, there is no need for complicatedadditional signal connections. The visualizationdata is collected in a data core controlled by an Ex-ternalObject construction and transmitted to theexternal visualization viewer (see Figure 4).

Model layer

Softwarelayer Data core

Visualization

Figure 4: Model layer and software layer with dataadministration (hidden)

3.3 Modified Modelica 3.0 standard li-brary

One strength of Dymola’s visualization systemis the automatic generation of the scene viathe visualization properties of every multi-body system. This is done via implementinga visualization definition in every part of theMulti-body library. These definitions all inheritthe Modelica.Mechanics.MultiBody.Visualizers-.Advanced.Shape block, which is the connectionto the Dymola visualization. By replacing thisblock with a modified variant, the complete modelvisualization is redirected to the external visu-alization. This modification allows to use everyexisting model with the external Visualization.Since Modelica 3.1 the vendor-specific libraryelements like the Shape block are grouped inan additional service library ModelicaServices.Analogue to the modified Modelica library, it ispossible by providing a customized ModelicaSer-vices implementation to redirect the visualizationdata to the external viewer DLR SimVis.

3.4 External viewer software DLRSimVis

The visualization data is sent from the simulationprocess via network to the external viewer soft-ware SimVis. This software is responsible for theinterpretation of the visualization data and for therendering of the scene. Because of the use of anetwork communication, the Viewer software doesnot have to run on the same system as the simula-tion, and therefore more computing power for thesimulation can be provided. Based on the opensource scene graph OpenSceneGraph [2] a widerange of formats for CAD data is supported. Thecontinuing development of OpenSceneGraph pro-vides the base for highly detailed visualizations.Utilizing the ffmpeg video en/decoding library [3],several video codecs for video grabbing are sup-ported. The following features already are imple-mented:

∙ Fullscreen mode

∙ Support for multiple cameras

∙ Multi-monitor support

∙ Textured CAD files (.dxf, .stl, .3ds, .obj, ...)

∙ Video grabbing, formats: (MPEG4, MSMPEG4 2/3 (.avi), Flash video (.flv), Huff-man Encoding (lossless), Windows MediaVideo (.wmv))

∙ Video grabbing with free configurable bit rateand replay speed

∙ Wireframe mode, Stereo mode (anaglyph andindirectly by OpenGL graphics card drivers)

∙ Precise replay controls including a jog-dial

3.5 Object-oriented network protocol

In order to reduce the amount of visualization datato be transported over the network connection, anoptimized protocol is necessary. In the implemen-tation of the ExternalDevices library, an object-oriented approach with data and packet objectshas been chosen. While the data objects handlethe data storage and the tasks of serialization /deserialization, the packet objects are responsiblefor data object administration. This includes anincremental packaging of data, where only dataobjects, which changed in the last time step are

included in the transmitted data. A lossless trans-port protocol provided, like TCP/IP, this methodreduces the needed communication bandwidth sig-nificantly, because static properties of visualiza-tion elements must not be communicated everytime step.Every packet is identified and assigned to a visual-ization element by an unique, automatically gen-erated ID. Figure 5 provides an overview of thenetwork communication architecture.

4 Visualization package content

The visualization package is structured in sub-packages for Shape blocks, Camera blocks, Lightblocks, Energy Flow blocks and Effect blocks.

4.1 The UpdateVisualization block

The UpdateVisualization block is responsiblefor the integration of the visualization blocks inthe simulation process. Similar to the Multi-Body.World block, it will be automaticallyinserted as an inner component if a visualizationblock is used in the model. The UpdateVisu-alization block controls parameters like the IPaddress and port for the communication with thevisualization software SimVis and can be used todisable the complete visualization.

During a simulation run, the block triggers atime event every time visualization data shall besent to SimVis. This update interval time can bevaried via a parameter and should be < 0.04 s for25 frames per second during real-time simulations.If the update time event is triggered, every blocksends its data to the data core triggered by theBoolean variable UpdateVisualization.send.

4.2 Shape blocks

Every Shape block has a frame connector as input.The resulting forces and torques of such a block iszero, so the block only has visual and no dynamicor kinematic effects. This blocks can be used as

... ...

DLR SimVisSimulation process

Packet 1

Packet nData 1. . .

Data k

Data 1. . .

Data k

incrementalcomparison serialization

deseriali-zationEthernet

rootnode

Object 1 Object n

Box CAD file

Figure 5: Basic principle of visualization network architecture

integrated components in a Modelica multi-bodymodel:

Figure 6: Shape blocks

Elementary Shape: This block represents thesupported basic shapes, such as boxes, spheres etc.The available primitives are:

Basic shape Existent in Existenttype Modelica 3.0 in library

Box Yes YesSphere Yes YesCone Yes Yes + FeaturesSpring Yes YesCylinder Yes YesPipe Yes YesBeam Yes Yes

Gearwheel Yes Yes + FeaturesCoordinateSystem

Yes Yes

Grid No Yes

Every ElementaryShape can be parametrized in

size, color, transparency and reflection behavior(for specular highlights). For the more complexshapes like spring, gearwheel and cone, additionalparameters can be set. Beyond the standard pa-rameters, required by the Modelica 3.1 standard,some additional parametrization is possible, forexample the operating angle of gearwheels allowsthe construction of bevel gears.

FileShape: The FileShape block allows to use3D CAD models, supporting numerous file for-mats like .obj, .dxf, .3ds, .stl, and every otherfile formats supported by the OpenSceneGraphplug-in system. The key features of this blockare support for textured 3D models and addi-tional parameters like transparency and a wire-frame modus. The loaded 3D model can be scaledin x,y,z directions.

Line The Line block implements linear or Bezierinterpolated lines, with n control points relative tothe frame connector of the block.

Text Shape: This block allows to place 2D textsin the scene, aligned to a specified direction orto the screen. Font, character size and color areparameterizable.

Text Shape with Value: In addition to theTextShape, this block has a Real input connector.The text in the visualization is followed by the

input value of this connector, allowing the displayof simulation data in the visualization.

4.3 Visualization of flexible bodies

Especially for the use in the DLR FlexibleBodieslibrary, a visualization module for flexible bodiesis available in the library. For topologically sim-ple objects (e.g. beams or tori) a parameterizablesurface can be used, visualizing an array of points(see Figure 7)For more complex models, an spatial interpolat-

Figure 7: quadratic, parameterizable surface

ing algorithm is implemented. This algorithm al-

Figure 8: Deformed(wireframe) component of anindustrial robot

lows to deform CAD models visually according toa displacement set of n control points (n << nCAD)and interpolates the CAD data points spatiallybetween the control points. With this algorithmthe number of points to be communicated to thevisualization system can be reduced drastically, asonly the displacement set has to be calculated andsubmitted to the visualization (see Figure 8).

4.4 Energy flow visualization

The visualization package supports the visual-ization of energy flows with several blocks (seeFigure 9). The energy flow is represented visually

Figure 9: Energy flow blocks

by a transparent pipe (can be deactivated) withmoving arrows inside. The spatial configurationof the pipe is specified within the model byparameterizing the single components of thepipe (MultiBody Library compatible) with infor-mations like length, diameter, radius of curvedsegments etc. The basic flow elements availableare StraightPipe, CurvedPipe and FlexiblePipe(flexible interpolated). For visualization purposes,the color, size and speed of the arrows can bedynamically changed during the simulation.

Every energy flow pipe has to begin with aFlowBegin block and ends with an FlowEndblock. The start position of a pipe is defined bya MultiBody frame connector, whereas the flowspeed of the pipe indicators can be set by anReal input of the FlowBegin block. The inter-connection between the pipe segments is handledby a special connector containing the connectingframe of the segment, the flow through the pipeand an ID of both connected pipe segments. TheIDs are necessary to provide information aboutthe assembly of the energy flow system for thevisualization software as a double-linked list.In order to avoid kinematic loops, the frameinformation of the pipe segments is encapsulated

Figure 10: Car Radiator with coolant flow

in the connector and should not be accesseddirectly. If a connection between two definedpoints is desired, the FlexiblePipe block can beused to exactly construct a pipe connecting thesetwo points. This is done by a Bezier interpolationalgorithm and can be updated during the simu-lation to visualize a flexible connection betweentwo moving points.

4.5 HUD Elements:

This objects support the generation of simplehead-up-displays, allowing the placement of text,bar graphs and bitmaps. This elements can becombined to form e.g. analogue instruments (seeFigure 11) or digital gauges. The displayed val-ues are updated during the simulation and allowa direct insight into the connected states of thesimulation. The head-up-display is placed as a 2Doverlay over the 3D scene.

Figure 11: Several bitmaps combined to an ana-logue tachometer and rpm gauge

4.6 Cameras

The visualization system supports multiple cam-era views. If there is no dedicated camera in themodel, a standard view will be used. For everynew camera block in the model, an additional sub-window is available in the visualization viewer,showing the scene from the cameras perspective.Every camera can have its own background color,viewing distance and field of view. A full screenmode allows to display the camera perspective onthe complete display, multiple displays are sup-ported. The following camera blocks are available:

FreeCamera: A free movable camera, initial-ized at the camera-frame connectors start posi-tion. The cameras position and perspective can beadjusted in the viewer with the computer mouse.

FixedCamera: The perspective and position ofthis camera is fixed. The position of the cameraonly can be changed by the simulation itself, andno user interaction is possible. The direction ofthe camera view can be parametrized.

FollowCamera: This camera is centered on theposition of the camera’s reference-frame connec-tor. As the reference connector moves, the camerakeeps focused on this position. The position of thecamera is defined via the camera frame connector.

DynamicFollowCamera: The position of thiscamera is defined, like the FollowCamera via thecamera frame connector position. But unlike theFollow Camera it is no direct coupling but a de-layed following behavior characterized by a PT 1system. The time constant of this following be-havior can be parametrized.

AttachedCamera: A camera with free ad-justable perspective, but with a position definedby the camera’s reference-frame connector. Thiscamera is useful for the observation of a dedicatedobject from different perspectives.

4.7 Lights

To create a well lightened scene, this sub-packageprovides Light blocks. Without a dedicated light-ing block in the model, a standard light will becreated and placed at the position of the camera.

Figure 12: Lighting blocks

Light block: This is the most flexible lightingblock, including the complete OpenGL definitionfor lights. Ambient, specular and diffuse light col-ors as well as directional lighting and attenuationof light are parameterizable.

Spotlight block: To reduce the configurationeffort, this block provides a preconfigured spot-light with a selectable color and a spot angle anddirection. The specular and diffuse color are setto the same value as the light color.

Diffuse Light block: This block provides a pre-configured directional light, with selectable colorand light direction. This light can be used forenvironment lighting, as it produces parallel lightrays like sunlight.

4.8 Effects

This sub-package contains blocks for additional vi-sual effects. In the current release version of theExternalDevices library, the available effects areweather and particle effects.

Weather effect block: Especially for environ-ment visualization, this block provides the threeeffects fog, rain and snow for more realism. Thefog effect, combined with a reduced viewing dis-tance of the camera can be used to reduce theviewing distance in the scene, in order to increasethe frame rate of the visualization.The Rain and Snow effect are done with a parti-cle system. A wind strength can be parametrized.All weather effects can be triggered with a booleaninput by the simulation. Weather effects haveno distinct position but are located around everycamera.

Particle effect block: With this block simu-lation of smoke or fire with variable intensity ispossible. A wind strength can be parametrized as

well as particle size and -lifetime. The ParticleEf-fect block can be connected to MultiBody systemswith a frame connector defining the position of theparticle origin.

5 Application examples

The following samples are generated with the Ex-ternalDevices library and are demonstrating se-lected models with external visualization.

Example 1: Electric motor Figure 13 showsa electrical engine propelling a rotor. In Figure14 the associated Modelica model is shown, witha magnification of the motor. File Shape blocksare used to represent the CAD Data of the motorand the rotor, a Flow Shape block visualizes theenergy flow between motor and rotor.

Figure 13: Electrical motor example

Example 2: Hybrid vehicle Figure 15 showsa screen shot of a visualization of a hybrid vehi-cle, rendered with SimVis. Advanced renderingeffects like the transparency of the chassis allowan insight into the car’s components. The sim-ulation is controlled with a SensoDrive steeringwheel via CAN bus, and a Logitech pedal systemvia USB joystick input. In Figure 16 the com-plete driving simulation, as shown on the FISITAworld automotive conference 2008 in Munich canbe seen. There are actually two simulations run-ning, a driving simulation and a simulation of arobot based motion simulation (see also Example3). The driving simulation renders two cameraperspectives on the left and upper display, whilethe motion simulator is shown on the right dis-play. The motion simulator receives acceleration

Figure 14: Model of the example

and angular velocity data from the driving sim-ulation and creates a trajectory simulating thesemotions.

Example 3: Robot visualization Figure 17shows a visualization of a KUKA KR500/1 robot(Source CAD Data: kuka.com) used as motionsimulator. The effects of specular highlights arevisible, generating a more plastic look of the robot.In this simulation, the robot joint angles are re-ceived via UDP from the real KUKA KR500/1’scontrol computer, the path-planning of the mo-tion simulator is done in a Modelica model andtransmitted back to the robot control.

5.1 General performance

During internal tests and projects, scenes with400 dynamically moved objects have been created,reaching frame rates >25 fps. Models with a mem-ory sizes up to 600 MB (uncompressed) have beenloaded and used as scenery. With the increas-ing rendering power of actual graphic processingunits, CAD models with numbers of vertices > 106

can be displayed with acceptable frame rates.

Figure 15: Visualisation of a hybrid vehicle

Figure 16: Driving simulator shown at FISITA2008

6 Conclusion and Outlook

The ExternalDevices library has shown its useful-ness during DLR internal tests, especially for cre-ating dynamic, interactive simulations. The pos-sibility to interact with the simulation reduces theeffort to generate input trajectories and allows theuser to determine the progress of the simulation.

Figure 17: Visualisation of KUKA KR500/1 in-dustrial robot

With UDP and shared memory interfaces, the co-operation of several simulations or reading andwriting to other external sources is feasible.The complete replacement of vendor-specific visu-alizations and the shift to a platform independentset of Modelica blocks allows much more flexibil-ity in the development of visualization solutions.The further development of the library will nowfocus to extend the support of input instruments(e.g. six-axis, force-feedback Joysticks) and theimprovement of the visualization, with a new,modular HUD system and full integration of par-ticle systems. Another very focused project aimsto integrate plug-ins for loading large terraindatabases into the SimVis framework.

References

[1] QT Software: http://www.qtsoftware.com/

products/

[2] OpenSceneGraph: http://www.

openscenegraph.org/

[3] FFMEPG: http://ffmpeg.org/

[4] OpenGL - The Industry Standard for HighPerformance Graphics: http://www.opengl.

org/

[5] Simulink 3D Animation 5 - User’s Guide,MathWorks Inc, http://www.mathworks.

de/access/helpdesk/help/pdf_doc/sl3d/

sl3d.pdf

[6] Aerolabs VisEngine: http://www.aerolabs.

net

[7] Vires Simulationstechnik GmbH: http://

www.vires.com/