Upload
safe-software
View
173
Download
2
Embed Size (px)
Citation preview
FME Server Workspace Patterns - ContinuedReed WhittingtonBurns & McDonnell
About Us
FME Certified Pro ServicesBurns & McDonnell, Dallas, TX
Reed Whittington
Burns & McDonnell is a full-service engineering, architecture, construction, environmental and consulting solutions firm.
Our multidiscipline staff of more than 5,000 employee-owners includes engineers, architects, construction experts, planners, estimators, economists, technicians and scientists, representing virtually all design disciplines.
Business Technology Services Burns and McDonnell Technology Consulting Services is a virtual team of technical experts around the globe offering a wide array of services and deliverables including:• Web Sites/Information Management Portals/Doc Management solutions• 2d/Web Maps/3D/4D Globe Applications• Field Data Collection/Inspections/Real-Time Monitoring• CAD/GIS/Database Integration• Automated Report Generation and Distribution• Project Metrics/Analytics/KPI Dashboards
Integrated, field tested, project proven solutions
Many of these services are bundled under OneTouchPM which is a hybrid cloud based, web enabled, managed service allowing project managers to engage their customers and share information, summary progressing with aggregated executive views from all disciplines.
What is a pattern?
In software engineering, a software design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. It is a description or template for how to solve a problem that can be used in many different situations. Not a specific implementation; design patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system.https://en.wikipedia.org/wiki/Software_design_pattern
4 FME Server Workspace Patterns• Workflow Submitter Iterator (or Work Status)
• Strategy for Asynchronous processing of large datasets• Runner / Controller Workspace (or Tool Chain)
• General approach using FMEServerJobSubmitter for workflow control
• Dataset Tool Switch (or Multi-Pass)• Strategy for Synchronous Sequential processing• Operates on same dataset with different modes of operation
• Flatten, Explode, Export (or Import/Merge/Publish)• 3 stages of processing typically used for master data management
Submitter Iterator with Status• Uses a selection strategy to select mutually exclusive
parts of a source data set so that the parts can be operated on independently and run on different fme engines.
• Used to process files or db operations so that sql select statements can be used.
• Needs a better name• Work Status, Delegator, …
• This was the subject of a previous presentation.• http://cdn.safe.com/events/FME2011WorldTour/Austin_Reed_Whittington.pdf
Example using Path reader to Iterate and track status.can also use url as path or list of SQL queries
Submitter Iterator with StatusWait=No
Runner / Controller Workspace• General strategy for Job Control and Tool Chain• Lets you run the controller on desktop• FME 2015 improves FMEServerJobsubmitter:
• In Engine Sequential Run with Wait=Yes• Output datasets to temp files or as specified
Excellent Article by DeWet@Safe:https://knowledge.safe.com/articles/1439/fme-server-workflow-management-1.html
Dataset Tool Switch (aka multi-pass)• The “Multi-pass" pattern presents a common interface
to iterate over datasets using successive refinement. • Meant to be run multiple times with different RUN_MODEs• RUN_MODE Choice defines the Switch behavior• TestFilter on RUN_MODE controls Feature RoutingUseful for
processing directory/tree of files and/or database exports or publishing.
• Interface includes:• SourceDir, StagingDir, Run Modes, DB connection, Publish
location
Use the Path reader to Iterate and Route based on Run Mode
Flatten, Explode, Export• 3 separate workspaces that act together but run asynchronous• Each may use “multi-pass”• Flatten
• Denormalization into one big table• Pack attributes into dictionaries• Think NoSQL and Operational Data Store
• Explode• Unpacks big table• Data model mapping and normalization happens here• May use templates for standard data models• Might also use Dynamic Schema
Flatten, Explode, Export (cont)• Export• Database to files (.kmz, .csv, .json, .gdb, …)• Heavy use of Fanout (ds and ft)• May use Tool Switch RUN_MODE to define
export type or staging/publish processing steps
More possibilities with FME 2016…What do you use?
Tool Chain Workspace• FeatureReader
– Selective reading/caching– Error Checking
• FeatureWriter– Write then read back in
same .fmw– In workspace tool chain– Removes the need for temp
output files of the FMEServerJobSubmitter
Before FME 2016• Required chaining
FMEJobSubmitters together
• Next in chain reads output of Previous
Main point: FME Server Patterns help you develop and organize your data management tools.
Thank you!Reed WhittingtonFME Certified Pro ServicesBurns & McDonnellDallas, [email protected]