Applying the Stratified Model of Relevance Interactions to Music Information Retrieval

David M. Weigl & Catherine Guastavino School of Information Studies, McGill University

Centre for Interdisciplinary Research in Music, Media and Technology 3661 Peel Street, Montreal, Quebec H3A 1X1

david.weigl@mail.mcgill.ca catherine.guastavino@mcgill.ca ABSTRACT While research on the notion of relevance has a long and rich history in information retrieval for textual documents, formal considerations of relevance concepts in Music Information Retrieval (MIR) remain scarce. We discuss the application of Saracevic’s stratified model of relevance interactions to the music information domain. This model offers a tool for deliberation on the development of user-oriented MIR systems, and a framework for the aggregation of findings on the music information needs and behaviours of potential users.

Keywords Relevance, music information retrieval, music information users, interaction, conceptual framework

INTRODUCTION Relevance, as a concept or set of concepts, has received prolific research attention in the field of information science. Discussions on the nature of relevance and its place at the heart of Information Retrieval (IR) have been on-going for decades, and have been traced to the very beginnings of organized academic research into IR (Saracevic, 1975). A large majority of IR systems organize information around textual entities, such as words and phrases (Saracevic, 2007, p. 1931), and so it is not surprising that relevance research has focused overwhelmingly on textual information domains; indeed, so ingrained is the notion that the relevance concepts under discussion operate on textual data, that this assumption is rarely stated explicitly.

This poster presents the initial findings of a larger project to clarify relevance considerations in another information domain, that of music. By proposing the application of the stratified model of relevance interactions (Saracevic, 2007)

in the context of Music Information Retrieval (MIR), we hope to inform future approaches toward the important task of operationalizing relevance concepts for this area.

STRATIFIED MODEL OF RELEVANCE INTERACTIONS Many competing models of relevance have been considered in the textual IR literature. Relevance is a notion that can be decomposed into a complex variety of concepts, and this decomposition can be conducted in different ways. Several authors have enumerated different relevance criteria for IR, resulting in large numbers of distinct components—as many as 80 (Schamber, 1994). In selecting a relevance framework to extend to applications in MIR, it is desirable to aim for an established model that is sufficiently complex to distinguish among significant relevance concepts, yet abstract and flexible enough to be applicable across information domains.

We believe that Saracevic’s stratified model of relevance interactions fulfils these criteria. The stratified model casts the act or process of information retrieval as a set of interactions between users and systems, through an interface at a surface level. Both user and system are represented by a “set of interdependent, interacting layers” (Saracevic, 2007, p.1927) that characterize this dialogue: the system by content, processing, and engineering layers; and the user by cognitive, affective, and situational layers. There is an implicit assumption that this process is “connected with cognition and then situational application” of the retrieved information (Saracevic, 1997, p. 315). A contextual component characterizes the influence of social and cultural factors that may influence or trigger adaptations in various layers.

Saracevic acknowledges a limitation of the stratified model, in that “it has not yet enough details for experimentation and verification” (p. 317). Nevertheless, we feel that the model can serve a useful purpose; user-centric MIR research in its current state is largely exploratory, lacking established conceptual frameworks (Weigl & Guastavino, 2011); the number of studies, although growing in recent years, remains small compared to systems-centric MIR research (Lee & Cunningham, 2012); and “comprehensive user models remain rare in MIR” (Schedl, Flexer, & Urbano, 2013, p. 4). Given the near-universal human propensity toward music consumption across individuals

This is the space reserved for copyright notices. ASIST 2013, November 1-6, 2013, Montreal, Quebec, Canada. Copyright notice continues right here.

and cultures, defining and representatively sampling from the population of potential MIR users is far from trivial. Current approaches frequently side step this issue by relying on convenience sampling of small groups of participants selected from highly homogenous sources (e.g. university students), limiting the generalizability of findings (Lee & Cunningham, 2013, p.18). However, we may still assess the transferability of findings by observing common themes that emerge among multiple studies, and unique themes that remain applicable only in the limited context of a particular study. In considering the application of the stratified model of relevance interactions to the music information domain, we propose a conceptual framework for the aggregated assessment of the findings of individual user studies. Although future research may suggest the need for modification or even the replacement of this model, a shared framework must be established if MIR user research is to proceed beyond the exploration of music information needs and behaviours in narrowly defined user group and use case contexts.

We will proceed by examining each layer of the stratified model in terms of its role in MIR relevance interactions.

The System

Content Musical content may be stored symbolically or as audio recordings, on a variety of media and in various encoding formats, resulting in numerous representational complexities. To the listener, music exists as the holistic combination of many individual facets. Perceptual qualities (e.g., pitch, rhythm, timbre, tempo, harmony, loudness) arise from the complex interaction of physical factors (e.g., frequency, amplitude, spectrum, temporal evolution). Musical content may also include textual information (e.g., lyrics, performance instructions), and textual metadata (e.g. annotations on mood or genre). A host of bibliographic metadata is applicable (e.g., composer, performer, publisher, song title, album title). Finally, associated with any representation of music is a complicated set of copyrights tied to various stakeholders that require careful consideration even when providing access to works ostensibly in the public domain. Downie (2003) refers to these content-specific considerations as the “multirepresentational” and “multifaceted” challenges to MIR research.

In textual IR, characteristics such as informativeness and reliability are associated with this layer. While difficult to quantify in terms of musical content, there is a role for these notions in association with music reviews and recommendations encountered in the context of music information seeking behaviour (Laplante, 2010a). The perceptual qualities arising from individual musical facets have strong implications for notions of similarity and topicality in the absence of textual information, providing relevance interactions with the cognitive, affective, and situational layers.

Processing The processing layer concerns the implementation of IR algorithms and software. Since the late 1990’s, remarkable progress has been made in the creation and optimization of algorithms for a large variety of MIR-related tasks. In contrast, the number of user-oriented software packages offering full-featured, robust access to such algorithms has remained small (Downie, Byrd, & Crawford, 2009; Wiering, 2005). The number of such systems to have undergone rigorous usability evaluations is smaller still. Furthermore, evaluations of MIR systems often “completely ignore user context and user properties, even though they clearly influence the result” (Schedl & Flexer, 2012, p.388).

Engineering Analysis on this layer is focused on hardware characteristics. Several concerns on this layer (e.g. storage capacity, processing power) are shared between MIR and textual IR contexts. The popularity of streaming audio playback, and of music listening on mobile devices, suggest additional MIR-specific concerns. The impact of listening device selection on various factors relating to everyday music consumption is subject to recent research in the Music Perception and Cognition (MPC) field (Krause, North, & Hewitt, 2013), but has not received much explicit consideration in MIR user studies.

The User

Cognitive Music listening is an experiential activity governed by cognitive processes in the mind as much as by grooves on the vinyl record. The process by which physical sound waves give rise to auditory experience is complex and varies among individuals; this is one part of the “multiexperiential challenge” to MIR (Downie, 2003, p. 304). Notions of similarity and relevance must therefore take insights from MPC into account in order to reflect the listener’s experience. While the exchange of ideas between the fields of MIR and MPC is gradually becoming more common, communication challenges remain (Aucouturier & Bigand, 2012).

In the absence of “aboutness” or well-defined meaning in instrumental music, topicality can be defined in terms of song identification or classification, an interaction of the cognitive and content layers; if the listener recognizes a musical query as representative of a particular song, result sets that include (versions of) this song may be deemed topically relevant.

Further considerations on the cognitive layer include user-properties such as listening history and taste profile. In the music information domain, individual tastes and beliefs play a greater role than in other settings (Laplante, 2010a). As in other recreational information seeking contexts, novelty prominently influences relevance judgements (Laplante, 2010a; Xu, 2006).

Affective The other part of the “multiexperiential challenge” concerns the variability of musical experience within the same individual under different affective conditions. A listener’s mood influences their appreciation of different pieces of music, with effects on relevance judgements. Conversely, music is capable of eliciting emotions in the mind of the listener, and may be applied purposefully for mood management (Laplante, 2010a, p.605). The identification of cognitive mechanisms and neural correlates underlying music emotion elicitation is a topic of active research in the MPC field. At present, MIR algorithms to address such issues are designed and evaluated in terms of listener-generated mood annotation datasets (Hu et al., 2008), and typically do not integrate insights from MPC.

Situational Musical relevance is highly situational in nature; beyond the sheer joy of listening, music selection may be highly purposeful, as when selecting a soundtrack to accompany a TV show or advertisement (Inskip, MacFarlane, & Rafferty, 2010), or when deciding on the first dance of a wedding. More generally, music is commonly selected for particular use cases or activities (Cunningham, Jones, & Jones, 2004), such as relaxation, socializing, driving, housework, or exercise. Finally, users may seek music information in order to acquire or expand musical knowledge, mirroring “traditional” information behaviours; for instance, a user seeking out new releases by favourite artists is engaging in ongoing search behaviour as per Wilson (1997).

Interface The interface is the site of explicit interaction between the user and the system as the information retrieval process unfolds. Saracevic (1997) terms this “interaction on the surface level.” While the design of the interface may facilitate or impede the dialogue between user and system, interaction in its full extent is “a sequence of processes occurring in several connected levels or strata” (p. 316). In the user-centric MIR literature, rigorous usability evaluations of interfaces remain rare; instead, the focus has been on the exploration of user requirements and music information needs—involving concerns on deeper layers of the stratified model—in order to derive implications for future interface design (Weigl & Guastavino, 2011).

Query Numerous query mechanisms have been considered and implemented in the MIR field. These include variations on the idea of query-by-performance (e.g. query-by-humming or query-by-tapping); query-by-example, where audio fingerprinting is used to determine the identity of an audio recording submitted by the user; symbolic queries, representing note or note-interval sequences, or pitch contours; and textual queries, providing bibliographic information (e.g. song title, artist name), lyric fragments, or metadata (e.g. genre; mood). It is not apparent that all of these query mechanisms are equally accessible by all

potential users; for instance, symbolic queries encoding pitch contour, envisioned by Parsons and Levin (1975) as a music retrieval mechanism for non-musicians, have been shown to be effectively unusable by participants lacking musical expertise (Uitdenbogerd & Yap, 2003).

Context The listener is situated within a social and cultural context. Social connections carry great influence on the evolution of musical tastes (Laplante, 2010b). Recommendations by friends and family are appraised as more trustworthy than expert reviews in publications. Musical tastes and knowledge can be worn as a “social badge” to portray information about one’s character to one’s peers (Laplante, 2010a). In certain cases, it is desirable to obscure aspects of one’s music consumption in order to avoid embarrassment (Cunningham, Jones, & Jones, 2004). In addition to their influence on relevance judgements, such social considerations can be applied directly to the task of music recommendation, in the context of collaborative filtering.

In terms of cultural context, MIR research has predominantly considered Western popular music of the 20th and 21st centuries, introducing biases that present a challenge both to MIR systems performance and to the transferability of user study findings (Downie, Byrd, & Crawford, 2009; Serra, 2011). Beyond these “macro-cultural” concerns, sub-cultural affiliations must be considered. For instance, a genre taxonomy of metal music, derived from a large dataset of user-contributed annotations, produces 35 distinct sub-genres (Lamere, 2008); while a heavy metal aficionado may distinguish between brutal death metal, technical death metal, and progressive death metal, another listener may lump the entire taxonomy into one larger “rock” category. Genre labels may thus be simultaneously too broad and too narrow, to the point of irrelevance, depending on the individual listener’s self-identification within a sub-culture and corresponding familiarity with the music (Laplante, 2010a).

CONCLUSION While MIR algorithms for a variety of tasks are becoming more and more refined, user-centric MIR research remains largely exploratory. Lacking concrete meaning or “aboutness” beyond the relatively superficial level of lyrics and song or album titles, it is particularly difficult to define relevance concepts in terms of traditional, topical notions such as precision and recall. While there is a significant overlap with textual IR in terms of the criteria used in relevance judgements (Inskip, MacFarlane, & Rafferty, 2010; Laplante, 2010a), relevance considerations particular to the music information domain have as yet received only limited attention in the literature. To extend our investigation of such considerations beyond the scope of narrowly defined user groups and use cases, a shared conceptual framework is required. We have demonstrated that the stratified model of relevance interactions’ broad

conceptualization of the system, user, interface, and context, along with its substrata representing finer-grained layers within these concepts, affords a fairly comprehensive framework for the contextualization of the findings of individual studies. We thus postulate that the stratified model, at this time, provides a viable platform for such aggregated assessment.

REFERENCES Aucouturier, J. J., & Bigand, E. (2012). Mel Cepstrum &

Ann Ova: The difficult dialog between MIR and music cognition. In Proceedings of the 13th International Society for Music Information Retrieval conference (ISMIR 2012), 397-402.

Cunningham, S. J., Jones, M., & Jones, S. (2004). Organizing digital music for use: an examination of personal music collections. In Proceedings of the 5th International Symposium on Music Information Retrieval (ISMIR 2004), 447-454.

Downie, J. S. (2003). Music information retrieval. Annual review of information science and technology, 37 (1), 295-340.

Downie, J. S., Byrd, D., & Crawford, T. (2009). Ten years of ISMIR: Reflections on challenges and opportunities. In Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR 2009), 13-18.

Hu, X., Downie, J. S., Laurier, C., Bay, M., & Ehmann, A. F. (2008). The 2007 MIREX audio mood classification task: Lessons learned. In Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2007), 462-467.

Inskip, C., MacFarlane, A., & Raffery, P. (2010). Creative professional users’ musical relevance criteria. Journal of Information Science 36 (4), 517-529.

Lamere, P. (2008). Social tagging and music information retrieval. Journal of New Music Research 37 (2), 101-114.

Laplante, A. (2010a). Users’ relevance criteria in music retrieval in everyday life: an exploratory study. In Proceedings of the 11th International Conference on Music Information Retrieval (ISMIR 2010), 601-606.

Laplante, A. (2010b). The role people play in adolescents’ music information acquisition. In Workshop on Music Recommendation and Discovery, 29-32.

Lee, J. H., & Cunningham, S. J. (2012). The impact (or non-impact) of user studies in music information retrieval. In Proceedings of the 13th International Society for Music Information Retrieval conference (ISMIR 2012), 391-396.

Lee, J. H., & Cunningham, S. J. (2013). Toward an understanding of the history and impact of user studies in

music information retrieval. Journal of Intelligent Information Systems, 1-23.

Parsons, D., & Levin, B. (1975). The directory of tunes and musical themes. Cambridge, England: S. Brown.

Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science 26, 321-343.

Saracevic, T. (1997). The stratified model of information retrieval interaction: Extension and applications. In Proceedings of the American Society for Information Science 34, 313-327.

Saracevic, T. (2007). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology 58 (13), 1915-1933.

Schedl, M., & Flexer, A. (2012). Putting the user in the center of music information retrieval. In Proceedings of the 13th International Society for Music Information Retrieval Conference (ISMIR 2012), 385-390.

Schedl, M., Flexer, A., & Urbano, J. (2013). The neglected user in music information retrieval research. Journal of Intelligent Information Systems. In press, retrieved June 18, 2013 from http://www.cp.jku.at/research/papers/ schedl_jiis_2013.pdf

Serra, X. (2011). A multicultural approach in music information research. In Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), 151-156.

Uitdenbogerd, A. L., & Yap, Y. W. (2003). Was Parsons right? An experiment in usability of music representations for melody-based music retrieval. In Proceedings of the 4th International Conference on Music Information Retrieval (ISMIR 2003), 75-79.

Weigl, D. M., & Guastavino, C. (2011). User studies in the music information retrieval literature. In Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), 335-340.

Wiering, F. (2007). Can humans benefit from music information retrieval? In Adaptive Multimedia Retrieval: User, Context, and Feedback (pp. 82-94). Springer Berlin Heidelberg.

Wilson, T. (1997). Information behaviour: An interdisciplinary perspective*. Information Processing & Management, 33(4), 551-571.

Xu, Y. (2007). Relevance judgment in epistemic and hedonic information searches. Journal of the American Society for Information Science and Technology, 58(2), 179-189.