Upload
andrewhilts
View
515
Download
0
Tags:
Embed Size (px)
Citation preview
Design and Evaluation of the Goal-oriented Design Knowledge Library
Framework
iConference 2012
Andrew HiltsEric Yu
Faculty of Information, University of Toronto
February 9, 2012
iConference 2012Hilts & Yu 2
The GO-DKL Framework
•A model and method for extracting, codifying and storing relational excerpts of design knowledge from scholarly publications.
•A method for analyzing such a knowledge base to support information systems design.
iConference 2012Hilts & Yu 3
Siloed e-Democracy Design
•Emerging scholarly community
•“Design knowledge” expressed textually in journal articles or white papers.
•A shared knowledge base could accelerate and strengthen advances in the domain.
iConference 2012Hilts & Yu 4
Study objectives
•Unified model of scholarly findings for accessible and efficient retrieval.
•Support designers / analysts to leverage past work to apply to current problem contexts.
•Highlight related areas of application domain that may be relevant to a designer’s project.
iConference 2012Hilts & Yu 5
Design Knowledge Management
•How to effectively manage information systems design knowledge?
•No one solution -- domain specific.
iConference 2012Hilts & Yu 6
Design Knowledge: a metamodel
iConference 2012Hilts & Yu 7
Design Knowledge: a metamodel
•Conceptualize
•What is design knowledge?
•What is not?
Fludd, R. (1619). Description of perception.
iConference 2012Hilts & Yu 8
Design Knowledge: a metamodel
•Codify
•How do we obtain and store this knowledge?
Stohlman, T. (2008). Touching Rosetta. http://www.flickr.com/photos/namlhots/227505138/
iConference 2012Hilts & Yu 9
Design Knowledge: a metamodel
•Retrieve
•How can this knowledge be accessed?
http://www.examiner.com/sites/default/files/styles/image_full_width/hash/StanfordLibrary-JustinSullivan_Getty%20Images-12-17-2004.jpg
iConference 2012Hilts & Yu 10
Design Knowledge: a metamodel
•Represent
•How is retrieved knowledge depicted?
Magritte, R. (1935). The Human Condition.
iConference 2012Hilts & Yu 11
Design Knowledge: a metamodel
•Situate
•How to contextualize?
iConference 2012Hilts & Yu 12
Design Knowledge: a metamodel
•Evaluate
•How to determine if appropriate for usage context?
iConference 2012Hilts & Yu 13
Methodology
iConference 2012Hilts & Yu 14
Hevner et al’s model of I.S. research
iConference 2012Hilts & Yu 15
A goal-oriented view of design knowledge
•Design knowledge as the relationship between I.S. design features and stakeholder objectives.
•Features may help or hurt goals.
•Assess potential designs based on potential goal satisfaction.
iConference 2012Hilts & Yu 16
Building a design knowledge base
•How to codify design knowledge expressed in narrative textual form? Our approach:
•Qualitative, inductive coding scheme based on our notion of goal-oriented design knowledge.
•Coding scheme applied to academic papers; instances of scheme concepts tagged and converted into database records and relationships.
•29 codified publications.
iConference 2012Hilts & Yu 17
Library and Project Data
iConference 2012Hilts & Yu 18
Codification and Analysis
iConference 2012Hilts & Yu 19
Retrieval
•How to effectively navigate these relationships?
•Goal-oriented browsing - start from goals, end up with design feature recommendations
•Directly browsing long lists of goals is not scalable.
•Solution - aggregate goal classifications
•User builds a custom project model, a collection of relevant goals from the retrieved information.
iConference 2012Hilts & Yu 20
Application Example
•Facebook DSS•Design goals:
•Participation be incentivized
•External information be considered
•Degree of Consensus be obtained
•Social Pressures be reduced
iConference 2012Hilts & Yu 21
Goal association
Users’ project goal
Users’ project goal
Aggregate goals
associated with
project goals
Aggregate goals
associated with
project goals
iConference 2012Hilts & Yu 22
Goal browsing
Description of a library goalDescription of a library goal
iConference 2012Hilts & Yu 23
Relationally guided retrieval
Impacted & Related library goalsImpacted & Related library goals
iConference 2012Hilts & Yu 24
Tree-list representation and contextualization of project model
Alter contributionsto parent
Alter contributionsto parent
Deselect nodes to negatetheir contributions
Deselect nodes to negatetheir contributions
Export “model slice”Export “model slice”
Label colour calculatedbased on
received contributions
Label colour calculatedbased on
received contributions
Drill down to reveal high-level goals’ contributions
Drill down to reveal high-level goals’ contributions
iConference 2012Hilts & Yu 25
A generated goal model slice
iConference 2012Hilts & Yu 26
A closer look
iConference 2012Hilts & Yu 27
Generated Design Feature Report
iConference 2012Hilts & Yu 28
Generated Publication Report
iConference 2012Hilts & Yu 29
e-Democracy design practitioner work practices: methods
•Semi-structured interviews with 6 individuals.
•Interviews transcribed and qualitatively analyzed.
iConference 2012Hilts & Yu 30
e-Democracy design practitioner work practices: findings
•Primarily rely on tacit experience; little on codified guidelines.
•Inspired by popular websites; tacit evaluations thereof.
•Design goals very similar to those codified in the GO-DKL knowledge base.
“The things that inform me are not documentation about projects, but the
projects themselves.”
iConference 2012Hilts & Yu 31
Practitioner Evaluation of GO-DKL
• Relevance
• Useful as a heuristic - “did I miss anything?”
• Anticipate problems in advance
• Suggestions
• Link to “off-the-shelf” solutions
• Goal Prioritization
• Designers can contribute
• Critiques
• Trustworthiness
• Vocabulary
• Usability
GO-DKL: A source of inspiration for
potential solutions to design problems that
must be primarily considered in their own
context.
iConference 2012Hilts & Yu 32
Future Work
•Develop web-based community platform for goal-oriented design knowledge sharing.
•User-contributed design knowledge; less focussed on scholarly publications.
•Peer assessments of design relationship trustworthiness and contextual dependence.
•Extensive user testing and interface redesign.
iConference 2012Hilts & Yu 34
References•Ahmed, S. (2005). Encouraging reuse of design knowledge: a method to index knowledge. Design Studies, 26(6):565–592.
•Ahmed, S. and Wallace, K. M. (2004). Understanding the knowledge needs of novice designers in the aerospace industry. Design Studies, 25(2):155–173.
•Alexander, I. and Kiedaisch, F. (2002). Towards recyclable system requirements. In Proceedings Ninth Annual IEEE International Conference and Workshop on the Engineering of Computer-Based Systems, pages 9–16. IEEE Comput. Soc.
•Babar, M., Gorton, I., and Kitchenham, B. (2006). A framework for supporting architecture knowledge and rationale management. Rationale Management in Software.
•Davies, T. (2009). The Blossoming Field of Online Deliberation. In Davies, T. R. and Pena Gangadharan, S., editors, Online Deliberation: Design, Research, and Practice, pages 1–20, San Francisco, CA. CSLI Publications.
•Eilouti, B. H. (2009). Design knowledge recycling using precedent-based analysis and synthesis models. Design Studies, 30(4):340–368.
•Esfahani, H. and Yu, E. (2010). A Repository of Agile Method Fragments. In Munch, J., Yang, Y., and Schafer, W., editors, LNCS 6195: New Modeling Concepts for Todays Software Processes, pages 163–174, Berlin / Heidelberg. Springer.
•Muller, W. (1996). Typology and the organization of design knowledge. Design Studies, 17(2):111–130.
•Ormerod, T., Mariani, J., Ball, L., and Lambell, N. (1999). Desperado: Three-in-one indexing for innovative design. In Sasse, A. and Johnson, C., editors, 13th International Conference on Human-Computer Interaction, page 336, Edinburgh, UK. Ios PR Inc.
•Oxman, R. E. and Planning, T. (1994). Precedents in design: a computational model for the organization of precedent knowledge. Design Studies, 15(2):141–157.
•Wahid, S., Mccrickard, D. S., Chewar, C. M., and Chong, J. (2006). Entangled Design Knowledge : Relationships as an Approach to Claims Reuse.