Embed Size (px)
Citation preview
Sascha WenningerIntegration Architect, Blue T
The CSV File Strikes Back
• Technology Architect• Integration Focus• Co-Founder, Blue T• @sufw
About Me
• RESTful APIs• Modern Message Queues• Microservices• Event Stream Processing• Polyglot Programming
Integration Trends
Diversity
Image by Alexandre Dulaunoy
Use the Right Tool for the Job!
Image by Bruce Murray
Interaction?
A
B
B
A
A pushes to B
B fetches from A
Data Volume?
File Transfer
Messaging
SOA
REST
100B 1kB 10kB 100kB 1MB 10MB 100MB 1GB
Frequency?
File Transfer
Messaging
SOA
REST
week day hour minute second millisecond
Focus on what’s importantCoreContext
Generic
Defines your company, makes moneyNever “done”, always being refined
Directly supports the CoreTends to be bespoke to some extent
Necessary to stay in business. Reduce costs, drive to commodity
See Subdomains from Domain-Driven Design for more…
Focus on what’s importantCoreContext
GenericSee Subdomains from Domain-Driven Design for more…
If applied to one system:
5% of the code base
20% of value
80% of the effort
Focus on what’s importantCoreContext
Generic
Best of Breed
Just Do It
Example: Successfactors Employee Master Data
SAP ECC Successfactors
Environmental Factors
SaaS app managed by HR ➧ Design for ObservabilityNon-technical support
Used for Performance & Goals ➧ Inconsistent, non-constant usageTolerates latency
In 2012, expected other modules to be added “later” ➧ YAGNI
Don’t predict the future
Looks like ‘Generic’ Subdomain…CoreContext
Generic
Best of Breed
Just Do It
• SAP-delivered ABAP code for ERP data extraction• Creates a CSV file• File must be uploaded to SFTP server
Option 1: SAP-Standard File-Based Integration
Pros Cons• Unfashionable• SFSF import job run exactly once per day
• Easy to inspect• Easy to post-process• Efficient• Full data set every day• Undemanding on middleware
• SAP-delivered ABAP code for ERP data extraction• Creates XML messages for sending via SOAP• Requires middleware to invoke SFSF APIs
Option 2: SAP-Standard Web Services Integration
Pros Cons• More complex delta processing• Requires IT support to manage sequence• Requires XML display/editing tools• Not easy to inspect in Excel
• APIs! Yay!• Faster updates due to delta processing
Different Data Models
OU
Position
Person
1
1
n
n
Person
OU
Manager
Position
1
1
1
SAP HCM Org Structure Successfactors Reporting Relationships
🤔
• Fit for purpose• Great usability and observability• Outsource testing to HR• Empower HR to service support calls without IT help• One file in Excel = whole data set
CSV Files FTW!
Our SolutionSAP ECC Successfactors
SAP Code
Z Code CSV file CSV file
Process Orchestration
✉️
Post-processing: remove contractors unless they manage permanent staff
Send to SF support inbox for inspection if required
Load via standard import job
Guaranteed Delivery via SFTP
Do: Files as MessagesSAP Business SuiteProcess Orchestration
ABAP Proxy
Load Program
store
invoke
read
CSV file
CSV file
XI msg
CSV file
• SAP PO’s adapters for FTP(S) and SFTP, large file support
• Guaranteed Delivery• Failure detection and alerting• Retry handling• Many standard-SAP import interface still require files…
Do: Files as Messages
• FTP Scripts– Difficult to inventory– Difficult to monitor– Password management
• Filesystem mounts everywhere– The “shared database” of integration
Don’t: Muddling through
Do: Atomic File Handling
~file.tmp
openwritewrite
close
…
file.csvrename
file.csv
check size
check size
read contentsmove to archive
…
Writing Reading
• Trigger files• Start/End files• Trailers and checksums
Don’t: Work-arounds
• Commas or tabs as separators• Meaningful headers• 1 line == 1 record
Do: Simple Data Structures
• Different Record types• Fixed-width files• Application Logic
– “01” means insert, “02” delete, etc.• Trailers, Checksums
Don’t: Anything else
Image by Alexandre Dulaunoy
</presentation>
