- 1. FROM EVENTING TO JOB PROCESSING Carsten Ziegeler | Adobe
Research Switzerland 1
2. 2 RnD Team at Adobe Research Switzerland OSGi Core Platform
and Enterprise Expert Groups Member of the ASF Current PMC Chair of
Apache Sling Apache Sling, Felix, ACE Conference Speaker Technical
Reviewer Article/Book Author ABOUT 3. 3 OVERVIEW Where to fit in
the stack Discovery Offloading Workflow Distribution DAM Ingestion
Sling Granite AEM Job Distribution Felix OSGi Event Admin
FromEventingtoJobProcessing 4. 4 OSGI EVENT ADMIN Publish /
Subscribe Model OSGi Event Admin Component A publish deliver
Component X Component Y 5. 5 OSGi event is a data object with Topic
(hierarchical namespace) Properties (key-value-pairs) Page Event
Topic: com/day/cq/wcm/core/page Properties: path, change type
(add/remove/edit) etc. OSGI EVENT ADMIN Publish / Subscribe Model
6. 6 Publisher creates event object Sends event through EventAdmin
service Either sync or async delivery Subscriber is an OSGi service
Service registration properties Interested topic(s) Additional
filters (optional) OSGI EVENT ADMIN Publish / Subscribe Model 7. 7
Immediate delivery to available subscribers No guarantee of
delivery No distributed delivery OSGI EVENT ADMIN Publish /
Subscribe Model 8. 8 Immediate delivery to available subscribers No
guarantee of delivery No distributed delivery OSGI EVENT ADMIN
Publish / Subscribe Model DiscoverySling Job Distribution 9. 9
OVERVIEW Where to fit in the stack Discovery Offloading Workflow
Distribution DAM Ingestion Sling Granite AEM Job Distribution 10.
10 OVERVIEW Where to fit in the stack Discovery Offloading Workflow
Distribution DAM Ingestion Sling Granite AEM Job Distribution 11.
11 INSTALLATION SCENARIOS Clustered CRX Single Instance CRX
Instance 1 Instance 2 Instance 3 12. 12 Instance: Unique Id (Sling
ID) TOPOLOGIES I Apache Sling Discovery Clustered CRXCRX ID : A ID
: X ID : 42ID : 1 Single Instance Instance 1 Instance 2 Instance 3
13. 13 Instance: Unique Id (Sling ID) Cluster: Unique Id and leader
TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35 Clustered
CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance Instance 1
Instance 2 Instance 3 Leader Leader 14. 14 Topology: Set of
clusters TOPOLOGIES I Apache Sling Discovery Cluster 99Cluster 35
Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance
Instance 1 Instance 2 Instance 3 Leader Leader TopologyTopology
Instance: Unique Id (Sling ID) Cluster: Unique Id and leader 15. 15
TOPOLOGIES I Apache Sling Discovery Instance: Unique Id (Sling ID)
Cluster: Unique Id and leader Topology: Set of clusters Cluster 99
Clustered CRXCRX ID : A ID : X ID : 42ID : 1 Single Instance
Instance 1 Instance 2 Instance 3 Leader Topology Cluster 35 Leader
16. Cluster 99 16 Instance Sling ID Optional: Name and description
Belongs to a cluster Might be the cluster leader Additional
properties which are distributed Extensible through own services
(PropertyProvider) E.g. data center, region or enabled job topics
Cluster Elects (stable) leader Stable instance ordering
TopologyEventListener: receives events on topology changes
TOPOLOGIES II Apache Sling Discovery ID : 42 Instance 3 Topology
Leader 17. 17 OVERVIEW Where to fit in the stack Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution 18. 18 Job : Guaranteed processing, exactly once
Exactly one job consumer Started by client code, e.g. for
replication, workflow... Job topic Payload is a serializable map
Sling Job Manager handles and distributes them Sends out payload
events to a consumer and waits for response Retry and failover
Notification listeners (fail, retry, success) JOB HANDLING I Apache
Sling Job Handling 19. 19 STARTING / PROCESSING A JOB Apache Sling
Job Handling public interface JobConsumer { String PROPERTY_TOPICS
= "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC }
JobResult process(Job job); } public interface JobManager { Job
addJob(String topic, String optionalName, Map properties); }
Starting a job Processing a job Note: Starting/processing of jobs
through Event Admin is deprecated but still supported New in 5.6.1
20. 20 New jobs are immediately persisted (resource tree /
repository) Jobs are pushed to the processing instance Processing
instances use different queues Associated with job topic(s) Main
queue 0..n custom queues For example: replication agent queue or
workflow queue JOB HANDLING II Apache Sling Job Handling 21. 21
Queue is configurable Queue is started on demand in own thread And
stopped if unused for some time Types Ordered queue (eg
replication) Parallel queue: Topic Round Robin (eg workflow) Limit
for parallel threads per queue Number of retries (-1 = endless)
Retry delay Thread priority JOB QUEUE Apache Sling Job Handling 22.
22 Job Manager Configuration = Main Queue Configuration Maximum
parallel jobs (15) Retries (10) Retry Delay Eventing Thread Pool
Configuration Used by all queues Pool size (35) = Maximum parallel
jobs for a single instance ADDITIONAL CONFIGURATIONS Apache Sling
Job Handling 23. 23 MONITORING WEB CONSOLE Apache Sling Job
Handling 24. 24 Each instance determines enabled job topics Derived
from Job Consumers (new API required) Can be whitelist/blacklisted
(in Job Consumer Manager) Announced through Topology (used e.g. by
Offloading) Job Distribution depends on enabled job topics and
queue type Potential set of instances derived from topology
(enabled job topics) Ordered : processing on leader only, one job
after the other Parallel: Round robin distribution on all potential
instances Local cluster instances have preference Failover Instance
crash: leader redistributes jobs to available instances Leader
change taken into account On enabled job topics changes: potential
redistribution JOB DISTRIBUTION Apache Sling Job Handling 25. 25
Scalability in AEM: DAM Ingestion Non-Clustered installation
requirement The term Offloading: In AEM used for all things job
distribution and topology in clustered and non- clustered
installations, e.g. Offloading Browser More technically its only a
little add-on in Granite to Sling Job Distribution for handling
non-clustered installations WHY OFFLOADING 26. 26 OVERVIEW Where to
fit in the stack Discovery Offloading Workflow Distribution DAM
Ingestion Sling Granite AEM Job Distribution 27. 27 OVERVIEW Where
to fit in the stack Discovery Offloading Workflow Distribution DAM
Ingestion Sling Granite AEM Job Distribution 28. 28 SLING JOB
DISTRIBUTION Topology Instance Sling ID: 1 Job Manager Instance
Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance
Sling ID: 4 Job Manager Job Consumer Topic: A Job Consumer Topic: B
Job Consumer Topic: C 29. 29 SLING JOB DISTRIBUTION Instance Sling
ID: 1 Job Manager Instance Sling ID: 2 Job Manager Instance Sling
ID: 3 Job Manager Instance Sling ID: 4 Job ManagerA Job Consumer
Topic: A Job Consumer Topic: B A:2 Job Consumer Topic: CJob
Topology 30. 30 SLING JOB DISTRIBUTION Instance Sling ID: 1 Job
Manager Instance Sling ID: 2 Job Manager Instance Sling ID: 3 Job
Manager Instance Sling ID: 4 Job ManagerB Job Consumer Topic: A Job
Consumer Topic: B B:3 Job Consumer Topic: CJob Topology 31. 31
SLING JOB DISTRIBUTION Instance Sling ID: 1 Job Manager Instance
Sling ID: 2 Job Manager Instance Sling ID: 3 Job Manager Instance
Sling ID: 4 Job ManagerC Job Consumer Topic: A Job Consumer Topic:
B C:4 Job Consumer Topic: CJob ? Topology 32. 32 SLING JOB
DISTRIBUTION Instance Sling ID: 1 Job Manager Instance Sling ID: 2
Job Manager Instance Sling ID: 3 Job Manager Instance Sling ID: 4
Job ManagerC Job Consumer Topic: A Job Consumer Topic: B C:4 Job
Consumer Topic: CJob Offloading Framework Topology 33. 33 Detects
offloading jobs Transport of job and job payload between origin and
target instance Uses replication for the transport No distribution
of jobs No execution of jobs OFFLOADING FRAMEWORK Overview Instance
Sling ID: 1 Job Manager Job Consumer Topic: A Offloading Job
Manager C:2 Instance Sling ID: 2 Job Manager Job Consumer Topic: C
Offloading Job Manager C:2 C Job Replication 1 2 3 5 4 6 7 Topology
34. 34 Replication agents are created automatically Uses naming
convention Needs manual adjustments (replication user) OFFLOADING
FRAMEWORK Replication Worker Sling ID: 2 Author Sling ID: 1
offloading_2 (outgoing agent) offloading_reverse_2 (reverse agent)
offloading_outbox (outbox agent) 35. 35 Defines the content to be
transported between the instances Properties on the job payload
Takes comma separated list of paths Used to build the replication
package The job itself is implicitly added by the offloading
framework Offloading job input Property: OffloadingJobProperties.
INPUT_PAYLOAD (offloading.input.payload) Offloading job output
Property: OffloadingJobProperties. OUTPUT_PAYLOAD
(offloading.output.payload) OFFLOADING FRAMEWORK Payload 36. 36
Configures Job Consumers Configures the topic white/black listing
properties of each instance What jobs to execute on what instance
Configures the distribution Configuration applies for both,
clustered and non-clustered installations OFFLOADING BROWSER UI 37.
37 OFFLOADING BROWSER UI 38. 38 OFFLOADING BROWSER UI 39. 39
OFFLOADING BROWSER UI 40. 40 OVERVIEW Where to fit in the stack
Discovery Offloading Workflow Distribution DAM Ingestion Sling
Granite AEM Job Distribution 41. 41 New JobConsumer Class:
WorkflowOffloadingJobConsumer Topic:
com/adobe/granite/workflow/offloading Can launch new workflows
Expects the workflow model on the job payload Expects the workflow
payload on the job payload For use with clustered and non-clustered
installations WORKFLOW DISTRIBUTION 42. 42 @Component @Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) 43. 43 @Component @Service @Property(name =
JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) Create service component Must register with topic
Implement new JobConsumer interface 44. 44 @Component @Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) Define the job topic 45. 45 @Component @Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) Access job properties (payload) Read workflow model
and payload from job properties 46. 46 @Component @Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) Workflow specific Use workflow API to start workflow
for the given model and payload 47. 47 @Component @Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value =
WorkflowOffloadingJobConsumer.TOPIC) public class
WorkflowOffloadingJobConsumer implements JobConsumer { public
static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public static final String
WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public
static final String WORKFLOW_OFFLOADING_PAYLOAD =
"offloading.workflow.payload; public JobResult process(Job job) {
// read workflow model and payload from job payload String
modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String
payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); //
get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..; WorkflowModel wfModel = ..;
WorkflowData wfData = ..; // start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData); // all good
return JobResutl.OK; } } WORKFLOW DISTRIBUTION Job Consumer
(Simplified) Use JobResult enumeration to report back the job
status 48. 48 OVERVIEW Where to fit in the stack Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution 49. 49 Default ingestion workflow: DAM Update
Asset Load is put on the instance where the workflow is started,
usually the author New ingestion workflow: DAM Update Asset
Offloading Needs to be manually enabled by changing the workflow
launcher New workflow model with a single step:
AssetOffloadingProcess Uses WorkflowExternalProcess API Creates a
new job on topic: com/adobe/granite/workflow/offloading Allows
distributing the default ingestion workflow Load is put on the
instance where the job is distributed to Can be used to distribute
in clustered and non-clustered installations DAM INGESTION 50. 50
@Component @Service public class AssetOffloadingProcess implements
WorkflowExternalProcess { @Reference private JobManager jobManager;
private static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public Serializable
execute(WorkItem workItem, WorkflowSession workflowSession,
MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) 51. 51 @Component @Service public
class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference private JobManager jobManager; private static final
String TOPIC = "com/adobe/granite/workflow/offloading"; public
Serializable execute(WorkItem workItem, WorkflowSession
workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String
workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) Create service component Implement
WorkflowExternalProcess interface Reference JobManager service 52.
52 @Component @Service public class AssetOffloadingProcess
implements WorkflowExternalProcess { @Reference private JobManager
jobManager; private static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public Serializable
execute(WorkItem workItem, WorkflowSession workflowSession,
MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) DAM and Workflow specific Resolve
to Asset Read model from meta data Read workflow payload from Asset
path 53. 53 public Serializable execute(WorkItem workItem,
WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset
asset = ..; String workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) ValueMap for job properties Put
model and payload on job properties Used by the JobConsumer 54. 54
public Serializable execute(WorkItem workItem, WorkflowSession
workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String
workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) Build offloading payload properties
Comma separated list of paths Put them on the job payload as well
Only used for non-clustered distribution 55. 55 public Serializable
execute(WorkItem workItem, WorkflowSession workflowSession,
MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel =
/etc/workflow/models/dam/update_asset/jcr:content/model; String
workflowPayload =
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg;
ValueMap jobProperties = new ValueMapDecorator(new HashMap());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
; String offloadingOutput =
/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg
;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(),
offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(),
offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC,
null, jobProperties); return offloadingJob.getId(); } DAM INGESTION
Create Job (from workflow step) Create job using JobManager service
Use topic from job consumer Put job payload properties Return the
jobId as the workflow process id (workflow specific) 56. 56
@Component @Service public class AssetOffloadingProcess implements
WorkflowExternalProcess { @Reference private JobManager jobManager;
private static final String TOPIC =
"com/adobe/granite/workflow/offloading"; public Serializable
execute(WorkItem workItem, WorkflowSession workflowSession,
MetaDataMap metaDataMap){ . } public boolean
hasFinished(Serializable externalProcessId, ..){ // returns null,
if job is finished Job offloadingJob =
jobManager.getJobById((String) externalProcessId); return
offloadingJob == null; } } DAM INGESTION Create Job (from workflow
step) Workflow API specific callback Process id = jobId, from
execute() Query job by jobId Workflow step finished when job is
finished 57. 57 Choose a job topic Create JobConsumer component and
register with topic chosen To create a new job use new
JobManager.addJob() API with the topic chosen and the job payload
Add offloading payload to job payload Bundle and deploy JobConsumer
on topology instances Enable/Disable the new topic on the
instances, using Offloading Browser DEVELOPMENT - RECIPE 58. 58
TAKE AWAY Discovery Offloading Workflow Distribution DAM Ingestion
Sling Granite AEM Job Distribution 59. 59 TAKE AWAY Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution org.apache.sling.discovery.api Discovers topology
Cross-Cluster detection Foundation for job distribution 60. 60 TAKE
AWAY Discovery Offloading Workflow Distribution DAM Ingestion Sling
Granite AEM Job Distribution org.apache.sling.event Uses Sling
Discovery New JobConsumer API and job topics New JobManager API for
creating new distributed jobs Distributes jobs based on available
job topics and job queue configuration. Distributes in the whole
topology, including clustered and non- clustered instances Can
execute cluster local jobs only 61. 61 TAKE AWAY Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution com.adobe.granite.offloading.core Builds on top of
Sling Distributed Jobs Does not perform distribution Detects jobs
distributed to non-clustered instances Transports the jobs and
payload to non-clustered instances Uses replication for transport
Does not execute jobs 62. 62 TAKE AWAY Discovery Offloading
Workflow Distribution DAM Ingestion Sling Granite AEM Job
Distribution com.adobe.granite.workflow.core Defines a job consumer
to distribute the execution of whole workflows Defines topic
com/adobe/granite/workflow/ offloading Implements
WorkflowOffloadingJobConsumer component Supports clustered and non
clustered installations 63. 63 TAKE AWAY Discovery Offloading
Workflow Distribution DAM Ingestion Sling Granite AEM Job
Distribution cq-dam-core Makes use of com/adobe/granite/workflow/
offloading topic from Workflow Distribution New workflow step
(external step) that creates a new job on topic
com/adobe/granite/workflow/ offloading New DAM Update Asset
Offloading workflow Supports clustered and non clustered
configurations 64. 64 TAKE AWAY Potential Future Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
65. 65 TAKE AWAY Potential Future Discovery Offloading Workflow
Distribution DAM Ingestion Sling Granite AEM Job Distribution Felix
OSGi Event Admin FromEventingtoJobProcessing New OSGi Specification
Distributed Eventing Cloud Computing 66. 66 TAKE AWAY Potential
Future Discovery Offloading Workflow Distribution DAM Ingestion
Sling Granite AEM Job Distribution Felix OSGi Event Admin
FromEventingtoJobProcessing Job Distribution Improved load
balancing Pull based distribution 67. 67 TAKE AWAY Potential Future
Discovery Offloading Workflow Distribution DAM Ingestion Sling
Granite AEM Job Distribution Felix OSGi Event Admin
FromEventingtoJobProcessing Offloading As part of job distribution
Even simpler setup 68. 68 TAKE AWAY Potential Future Discovery
Offloading Workflow Distribution DAM Ingestion Sling Granite AEM
Job Distribution Felix OSGi Event Admin FromEventingtoJobProcessing
69. 69
