Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
NJIN 2.0Kate Silhol, Lead Software Engineer
NJIN 2.0 Design
Background
■ XML Message Router (XMR) installed 12
years ago
– Advanced Technology Systems product
– First foray into XML, web services
■ XMR converted to Nlets Justice Information
Network (NJIN)
– Majority of code re-written, now considered in-
house product
Overview
■ Current NJIN is approaching capacity
– Scalability limitations inherent to the 12 year old design
– Volume has increased by 209% in last 10 years
– Manageable but recurring system issues (slowness)
– Low risk mitigating changes made to sustain current NJIN
■ Design work for new system commenced end of 2015
Design Overview and Principles
■ Distributed architecture
■ Three primary components which guide
each transaction through the system
■ Specific processes located in distinct
modules – true SOA
High Level Design
Input/Output Layer
Validation and ESB Initiation Layer (VEIL)
Enterprise Service Bus
Modules
SBG Customized
PMB
Completely
separate code;
“black boxes”
mostly based on
existing code
Initial validation,
broker to the ESB
Input-Output Layer
■ Provides communication between the NJIN
2.0 and the users
■ Leverages SBG’s Plexus Message Broker
Code
■ Communication between Input-Output Layer
and the VEIL is via queues (one queue for
each user/line)
Enterprise Service Bus
■ Backbone of the NJIN 2.0
■ Provides communication between the
system components
■ Dynamic and static metadata will dictate
the actions taken for each transaction
ESB Selection ProcessProduct Open Source/COTS Pros Cons
WSO2
Open SourceCostGreat reputation
Java-based open source –significant ramp-up for Nlets staff
NServiceBus
Open Source .NET based platform
Performance may not be adequateFunctionality specifics limit outside contractor helpUnclear roadmap
Neuron ESBCOTS .NET based, local
Prior experience with toolPoor reputation
MuleSoftCOTS (Open Source also available but inadequate)
Good reputationFunctionality appears to be adequate
Company is focusing on cloud-oriented architecture.Requires on-going support costs
TIBCO
COTS
Reputation/functionality/performance – best in classSupport availabilityAbility to involve contractors
Cost
Home Grown N/A Ability to control functionality Time to implement
Processing Queues
■ Between the ESB implementations and
connected layers and modules,
communication will be via queueing
■ Message durability, prioritization,
guaranteed delivery, etc.
Pipelines
■ The VEIL will apply a Pipeline to the transaction before sending to the ESB
– Set of instructions based upon configurable rules
■ Different Pipelines based on message type, destination, etc.
■ Pipeline may be updated along the way – error conditions, etc.
■ Provide for better configurability and enhanced diagnostics
Sample Catalog of Pipeline Components
Rules ValidationRules Validation
Parsing
Logging
Conversion
Sub Route
Message Header Change
Alerting (PAL/ Graham Scrape )
Federation
Image Resizing
Image Stripping
Special Instructions
Send Message
Caveating
Sample Pipeline Configuration
Rules Validation
Parsing
Conversion
Image Resizing
Send Message
Transactional Objects
■ Each message traversing the NJIN 2.0 will
be wrapped in a Transactional Object
■ Metadata about the transaction and its
processing that will follow the message
through the process
■ Pipeline will be maintained within the
Transactional Object
<NletsTransactionalWrapper>
<NletsTransactionalObject>
<OriginalUniqueID>111111111111</OriginalUniqueID>
<UniqueID>111111111120</UniqueID>
<IPAddress>10.10.10.10</IPAddress>
<InboundHeaderData>
<MessageKey>DR</MessageKey>
<OriginatingORI>AZMVD0000</OriginatingORI>
<DestinationORI>AZNLETS23</DestinationORI>
<ControlField>CONTROLFLD</ControlField>
</InboundHeaderData>
<OutboundHeaderData/>
<MessagePriority>3</MessagePriority>
<InputHeaderFormat>Text</InputHeaderFormat>
<InputBodyFormat>Text</InputBodyFormat>
<OriginalMessage>DR.AZMVD0000.AZNLETS23.CONTROLFLD.TXTDRIVER STATUS
DATA</OriginalMessage>
<ErrorCondition/>
<PrimaryIndicator>true</PrimaryIndicator>
</NletsTransactionalObject>
<TransactionalPipeline>
<PipelineStep>
<StartDateTime/>
<EndDateTime/>
<Service>GetDestinationInfo</Service>
</PipelineStep>
<PipelineStep>
<StartDateTime/>
<EndDateTime/>
<Service>Validation</Service>
</PipelineStep>
</TransactionalPipeline>
<OutboundMessage/>
</NletsTransactionalWrapper>
Enhancements to NJIN
■ Better performance
■ Scalability
■ More detailed and usable logging
■ Improved diagnostics
■ Easier configurability
■ Correlation of inbound message with outbound
■ Spill queues
– Side by side queues allowing new traffic to flow alongside queued traffic following a state outage
■ Schema validation as an option (test system)
■ Statistics
■ Dashboard/console improvements
Monitoring Improvements
Monitoring Possibilities
0
200
400
600
800
1000
1200
1400
1600
0:0
0
1:0
0
2:0
0
3:0
0
4:0
0
5:0
0
6:0
0
7:0
0
8:0
0
9:0
0
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Overall Message Volume
SUM
0
100
200
300
400
500
600
700
800
900
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Volume by Message Key
Message Key AM CR
DQ DR FQ
FR RQ RR
0
500
1000
1500
2000
2500
0:0
0
1:0
0
2:0
0
3:0
0
4:0
0
5:0
0
6:0
0
7:0
0
8:0
0
9:0
0
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
0
200
400
600
800
1000
1200
1400
1600
0:0
0
1:0
0
2:0
0
3:0
0
4:0
0
5:0
0
6:0
0
7:0
0
8:0
0
9:0
0
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Overall Message Volume
Total Traffic Volume (Past Hour)
Monitoring Possibilities
Future Possibilities
■ Analytics
■ Proactive scaling and load balancing
■ Proactive alerting functionality
■ Reuse of modules
■ Executive Dashboard
NJIN 2.0 Implementation Plan
Addition of QA/Dev System
■ Current test system doing double duty – we
need a true test system!
■ First implementation of NJIN 2.0 will be in
new QA/Development system
■ No external users – sandbox
■ Then followed by implementations in test
and production
Production
NJIN
Test
NJIN
Production
NJIN 2.0
Test
NJIN 2.0
Development
NJIN 2.0
Phased Approach
■ Same phased approach will be used on the
Test System and Production System
■ Incremental process to minimize risk
■ All steps may be iterated for each individual
state/user in a category
■ All phases have clear roll-back plans
Nlets Users
NJIN 1.0PMB
Nlets Users
Current NJIN
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Current NJIN
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Native MQ (PMB)
users sending via
new Input Layer
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Native MQ (PMB)
users sending and
receiving via new
Input and Output
Layers
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
All users sending and
receiving via new
Input and Output
Layers
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
NJIN 2.0 processes in
parallel (no messages
outbound) to allow
testing and review of
logs
Logging
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Parallel processing
turned off to prepare
for cutover
Logging
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Logging
NJIN 2.0 now
performs all
processing
Status
■ Finalizing ESB selection and procurement
process
■ Development work to begin immediately
– Input-Output layers and modules first
■ Timeline: Plan to implement in
approximately one year