A Rule-based Framework for Enhancing Architectural Decision Guidance
Doctoral DefenceMohsen AnvaariMay 27, 2016
1
In a Nutshell
Ultra-Large-ScaleSystems
SoftwareEcosystems
CognitiveBiases
ResearchMethods
ArchitecturalDebt
ArchitecturalDecision-Making
ArchitecturalDecisionGuidance
SoftwareQuality
Agility
InformationExtractionMethods
Rule-basedFrameworkforEnhancingArchitectural
DecisionGuidance
havemorepotentialtocreate
haveimpacton
developedwithin
causedbypoor
diminishes
negativelyaffect
Improvesqualityandspeedof
enhances
develop andevaluate
compromises
accumulates
increases
increase
employs
requiremoresystematic
3
Architectural Decision Making Process
4
ArchitecturalIssues
AlternativeSolutions
DecisionDrivers
QualityAttributes BusinessFactors
DecisionOutcome(SelectedAlternative)
Rationale
have
relate
has
isatrade-offbetween
extends extends
affects(positively/negatively)
Identifying
ATAM,CBAM,SAAM,etc.
DecisionVP,SEURAT,etc.
Intuitiveandexperiencebased
CognitiveBiases
Cognitive Biases
Cognitive bias refers to a systematic pattern of deviation fromrationality in judgment.
The availability heuristic– Only 46% of architects search for alternative options ifthey already have a solution in mind (van Heesch andAvgeriou, 2011)
The affect heuristic– 68% of architects prefer well-known solutions in favor ofunknownalternatives (van Heesch and Avgeriou, 2011)
5
An Example
ArchitecturalIssue1 Whatshouldthe integrationstylebe? ESB
ArchitecturalIssue2 Whatshouldthe integrationproduct be? BizTalk
ArchitecturalIssue3 Whatshouldthedeploymentstrategyofintegration solutionbe?
Private Cloud
…
Project:IntegratingApplications inEnterpriseX
6
Which critical architectural issue is missing?• What should the ESB implementation strategy or pattern be? Global ESB or
ESB Gateway or Federated ESB?
The default alternative (global ESB) has been implemented (subconscious decision)• Scalability is compromised
Three years later• Global ESB has become the single point of failure = Architectural Debt
Architectural Debt
• Technical Debt (TD) refers to technical compromises that canprovide short-term benefit for project stakeholders but mayhurt the long-term health of a software system
• Architectural debt (AD) is a type of TD which is caused byarchitecture decisions that consciously or subconsciouslycompromise software quality attributes
7
Research Goals and Questions
Goal1. This thesis explores the current state of architecturaldecision making process: in companies that develop large anddistributed software and in the literature that propose tools forsupporting practitioners
RQ1. What is the attitude of large-scale enterprises in makingand reusing architectural decisions and how do available toolsand research prototypes support them?
8
Research Goals and Questions (cont.)
Goal2. By grounding on the advances in information extractionand natural language processing domains, it develops a rule-based framework called SADGE that enhances architecturalguidance from architecture-related documents in a rapid way.
RQ2. How can a framework be established to developarchitectural decision guidance from architecture-relateddocuments in a rapid way?
9
Research Goals and Questions (cont.)
Goal3. The thesis evaluates the efficiency and effectiveness ofthe framework by conducting a case study on documentsreceived from a telecommunication company, and by performingexperiments on both IT students and expert IT/softwarearchitects.
RQ3. How efficient and effective will such a framework be indeveloping architectural decision guidance?
10
Research Methods
11
ResearchMethod
Setting ResearchQuestions
Papers
QualitativeInterviews
Interviewing8informants(ITmanagersandarchitects)from6organizationsintheNorwegianelectricityindustry
RQ1 Paper1
Online Survey Aninter-rateragreementstudyon37ITexperts RQ1 Paper5
LiteratureReview
Analyzing5toolsandresearchprototypes RQ1 Paper2
DesignScience Developingandtestinganinformation extractionframework
RQ2 Paper3,4
CaseStudy Using3architecturaldocuments fromatelecommunication companytotestandevaluatetheframework
RQ2,3 Paper4
Experiment1 Evaluatingtheframeworkwith19ITstudents RQ3 Paper3
Experiment2 Evaluatingtheframeworkwith21expertarchitectsfrom5Norwegianorganizations
RQ3 Paper4
Research Scope
Solution Problem
Theory Practice
RQ1 P2
RQ1 P1
RQ1 P5
RQ2 P3,P4
RQ3 P3,P4
Inception
Development Evaluation
Inception
12
Research Contributions
C1. An overview of the state-of-the-art and state-of-the-practicein making and reusing architectural decisions.
C2. SADGE, a rule-based framework for enhancing architecturaldecision guidance from architectural knowledge in projectdocuments and in the domain literature.
C3. Results of empirical evaluation of developing architecturaldecision guidance by employing a rule-based framework.
13
Contribution 1 State-of-Practice
• Most of the companies are not using well-known academicapproaches such as ATAM, they are rather using their ownprocedures
• There is high potential and need for reusing the architectural issuesand decisions across various projects inside an enterprise or acrosssimilar enterprises in a domain– In 63% of cases, architects stated that “no previous similar decisions in
their projects to compare the current decision making against” was achallenge for their decision-making (Tofan et. al, 2013).
• The relationships among the actors of a software ecosystem (SECO)could significantly affect the architectural-decision making processin each of the actors
14
Contribution 1Expert Agreement on Associating Architectural
Issues with Quality Attributes
ICC:Intra-ClassCorrelationsKalpha:Krippendorff's alpha
15
Category of subjects ICC Kalpha All scenarios together 0.12 0.09
Scenarios
Scenario1 0.12 0.09 Scenario2 0.15 0.10 Scenario3 0.04 0.01 Scenario4 0.06 0.03 Scenario5 0.25 0.09
Quality Attributes
Functionality 0.14 0.09 Security 0.15 0.07
Reliability 0.13 0.09 Efficiency 0.10 0.05 Usability 0.11 0.04
Maintainability 0.08 0.05 Portability 0.16 0.11
0ICC,Kalpha 1
NoagreementFullagreement
Contribution 1State-of-Art
Available tools and research prototypes havemade significant contributions in the area ofarchitectural knowledge capturing, but stillrequire a number of extensions so that thecaptured knowledge can serve as architecturaldecision guidance
16
Contribution 2Semi-automated Design Guidance Enhancer (SADGE)
17
ArchitecturalIssuesIdentifying
Intuitiveandexperiencebased
CognitiveBiases
Company documents
Software Architecture
Domain literature
Electronic Documents (PDF, DOC, TXT, etc.)
Architectural Decisions
Step1
Step2
O3
Catalogue Construction
Catalogue of Terms (CoT)
High Priority
LowPriority
Annotation Rules
Automatic Annotator
(AA)
Guidance Generator
Annotated SentencesStep3
to step1, for annotating the next document
to step4, when all documentsare annotated
Contribution 2Semi-automated Design Guidance Enhancer (SADGE)
«AnnotationRule»
Conditionif
(asentencecontainsatleastoneofthetermsfromthecatalogueofhighpriorityterms)
or(containsatleasttwotermsfromthecatalogueoflowpriorityterms)
18
Action
annotateitasanarchitecturalissue
Contribution 2Semi-automated Design Guidance Enhancer (SADGE)
«Catalogue of Terms (Default Version)»
19
Contribution 2Semi-automated Design Guidance Enhancer (SADGE)
«Catalogue Construction Procedure»
20
Target Documents
New Clue Terms
1- Manually find clue terms from
10% of documents
Task Work Product Input/Output
Default Catalog of
Terms 2- Find
synonym of each term
in WordNet
Ad-hoc Catalog of Terms +
Add new terms
Contribution 3Evaluation of SADGE: Metrics
21
Relevant Sentences
Annotated Sentences
Non-relevant Sentences
Recall = Effort Reduction=
Contribution 3Evaluation of SADGE
22
Document Effort Reduction (%)
Recall (%)
Doc1 66 89 Doc2 51 72 Doc3 63 73 Mean 60 78
SD 7.93 9.53 ResultsofCaseStudy
Contribution 3Evaluation of SADGE
“Several studies have shown that human decision makers are inferior to a predictionformula even when they are given the score suggested by the formula. They feelthat they can overrule the formula because they have additional information aboutthe case, but they are wrong more often than not” (Kahneman, 2011)
24
Inlinewiththeempiricalresultsinthefieldofpsychologyofdecision-making:
Approach Processing Time (min) Recall (%)
Manual 14 34 Automatic 0.03 53.5
Semi-automatic 7 32
Comparison p-value
Document1 Automatic vs. Manual 0.000225 Automatic vs. Semi-automatic 0.0000908
Document2 Automatic vs. Manual 0.003323 Automatic vs. Semi-automatic 0.0002254
Approach Processing Time (min)
Recall (%)
Manual 9 38 Automatic 0.03 86
Semi-automatic 3 55 ResultsofexperimentonITstudents Resultsofexperimentonexpertarchitects
ResultsofK-Stest
Strengths and Weaknesses of Research Outcome
Strengths• Balancing agility and architecture-awaredevelopment• Ad-hoc construction method for developing the Catalogue of
Terms• Informing the practitioners that the quality attributes they
consider irrelevant or less relevant for an architectural issue,might be very relevant in the opinion of other practitioners
Weaknesses• Eliminating the context• Treatingall architectural issueswith a general attitude
25
Limitations of Research Studies
• Generalization of results of exploratory study and casestudy
• Just one of the authors has annotated the referencedocuments in the case study
• Participants might interpret an "architectural issue"differently from researchers
• Inconsistence interpretations of "architectural issue" and"quality attribute" between researchers and participants
26
Directions for Future Study
• Supporting more natural languages
• Developing domain-specific Catalogues of Terms
• Creating data sets for training and testinginformation extraction framework
• More in-depth understanding of architects'rationale for making architectural decision andthe role of cognitivebiases in their decisions
27
PapersP0 Smart Grid SoftwareApplications as an Ultra-Large-Scale System - Challenges for Evolution.Mohsen Anvaari, Daniela S. Cruzes, Reidar Conradi.In Proc. Innovative Smart Grid Technologies (ISGT), 2012 IEEE PES.P1 Architectural Decision-Making in Enterprises - Preliminary Findings From an Exploratory Study inNorwegian Electricity Industry.Mohsen Anvaari, Reidar Conradi, Letizia Jaccheri.In Proc. The 7th European Conference on Software Architecture, ECSA 2013.P2 Towards Reusing Architectural Knowledge as Design Guides - Functional Requirements, Tool Analysisand Research Roadmap.Mohsen Anvaari, Olaf Zimmermann.In Proc. The 26th International Conference on Software Engineering and Knowledge Engineering, SEKE2014.P3 Semi-automated Design Guidance Enhancer (SADGE): A Framework for Architectural Guidance
Development.Mohsen Anvaari, Olaf Zimmermann.In Proc. The 8th European Conference on Software Architecture, ECSA 2014.P4 Rule-based Extraction of Architectural Issues fromSoftware Architecture Documents.Mohsen Anvaari, Olaf Zimmermann, Carl-Fredrik Sørensen.Submitted forpublication.P5 Associating Architectural Issues with Quality Attributes: A Survey on Expert Agreement.Mohsen Anvaari, Carl-Fredrik Sørensen, Olaf Zimmermann.Submitted forpublication.
28