The Representation, Indexing and Retrieval of Music Data at NTHUArbee L.P. ChenNational Tsing Hua UniversityTaiwan, R.O.C.http://www.cs.nthu.edu.tw/~alpchen

OutlineContent-based media data retrievalMusic data retrievalFeatures of music dataFeature indexing and matchingPrototypesReference

Content-based Media Data RetrievalRepresentation of media contentsfeaturesFeature extraction from media dataFeature indexingQuery interface

Content-based Media Data RetrievalMatching query features against the feature indexapproximate/partial matchingsimilarity measureprecision: how many of the answers are in fact correctrecall: how many of the correct answers are in fact retrievedrelevance feedback

Music Data Retrieval: System Architecture

Features of Music Data

Features of Music DataStatic music informationThe intrinsic music characteristics of music objectsKey, beat, and tempoE.g., the Beethoven Symphony No. 5, Op. 67, C minor, 4/4, Allegro con brioAcoustical featuresLoudness, pitch, duration, bandwidth and brightnessCan be computed and represented as numerical values

Features of Music DataThematic featuresThemes, melodies, rhythms, and chordsCan be derived from the staff information of a music objectMelodyThe melody of a song is the sequence of the pitches of all notes in the songsE.g., the melody of the theme of the Beethovens Symphony No.5 is sol sol sol mi fa fa fa - re

Features of Music DataRhythmThe rhythm of a song is the sequence of the durations of all notes in the songsE.g., the rhythm of the theme of the Beethovens Symphony No.5 is 1/2-1/2-1/2-2-1/2-1/2-1/2-4ChordA chord consists of three (root, third, and fifth) or more notes which sound together in harmony

Features of Music DataCoding scheme: a music object a sequence of music segments music segment = (segment type, segment duration, segment pitch)four segment types: (type A), (type B), (type C), and (type D)

Features of Music DataFor example,the sequence of music segments: (B,3,-3) (A,1,+1) (D,3,-3) (B,1,-2) (C,1,+2) (C,1,+2) (C,1,+1)

music segment = (type, duration, pitch)

Features of Music DataRepeating PatternA sequence of notes appearing more than once in the music objectEfficient content-based retrievalSemantics-rich representationExtracting repeating patternsTree-based approachMatrix-based approach

Features of Music DataExperiment 1

Features of Music DataDissimilarity of melody strings

Features of Music DataDissimilarity of repeating patterns

Features of Music DataExperiment 2

Features of Music DataValidity of classes

Finding Repeating Patterns: Tree-based ApproachConstruct an RP-tree for RPs with lengths 2n, n 0, 1, ... S = ABCDEFGHABCDEFGHIJABC

Finding Repeating Patterns: Tree-based ApproachLength 1{A, 3, (1, 9, 19)}{B, 3, (2, 10, 20)}{C, 3, (3, 11, 21)}{D, 2, (4, 12)}{E, 2, (5, 13)}{F, 2, (6, 14)}{G, 2, (7, 15)}{H, 2, (8, 16)}

Finding Repeating Patterns: Tree-based ApproachLength 2{AB, 3, (1, 9, 19)}= {A, 3, (1, 9, 19)} 0 {B, 3, (2, 10, 20)}{BC, 3, (2, 10, 20)}= {B, 3, (2, 10, 20)} 0 {C, 3, (3, 11, 21)}{CD, 2, (3, 11)}= {C, 3, (3, 11, 21)} 0 {D, 2, (4, 12)}

Finding Repeating Patterns: Tree-based ApproachLength 4{ABCD, 2, (1, 9)}= {AB, 3, (1, 9, 19)} 0 {CD, 2, (3, 11)}{BCDE, 2, (2, 10)}= {BC, 2, (2, 10, 20)} 0 {DE, 2, (4, 12)}Length 8{ABCDEFGH, 2, (1, 9)} = {ABCD, 2, (1, 9)} 0 {EFGH, 2, (5, 13)}

Finding Repeating Patterns: Tree-based Approach

Finding Repeating Patterns: Tree-based ApproachPrune trivial patterns of length 2n, n = 0, 1, Let X be an RP of S, Y a substring of X, and Z a substring of YIf freq(X) = freq(Z), Y is trivial

Finding Repeating Patterns: Tree-based ApproachLength 1

Finding Repeating Patterns: Tree-based ApproachLength 2

Finding Repeating Patterns: Tree-based ApproachLength 4

Finding Repeating Patterns: Tree-based ApproachGenerate all patterns of lengths 2n, n 0, 1, ...order-1 string-join AB 1 BC = ABC

Finding Repeating Patterns: Tree-based ApproachPrune all trivial patterns{ABCDEFGH, 2, (1, 9)}{ABC, 3, (1, 9, 19)}

Feature Indexing and Matching1D-ListPAT-TreeL-TreeAugmented Suffix TreeGrid-Twin Suffix Tree

Feature Indexing and Matching: 1D-ListThere are two music objects M1 and M2M1: sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-solM2: do-mi-sol-sol-re-mi-fa-fa-do-re-re-miThe melody string of the music queryQ: do-re-miProblem: to find whether M1 and M2 contain the melody string Q

exact

link

dropout

link

transportation

link

insertion

link

Feature Indexing and Matching: PAT-TreeExample, songs in chord stringsSong1 : Am F2 Dm AmSong2 : C C F CSong3 : G E1 C DSong4 : E1 G Am Bm

Feature Indexing and Matching: PAT-Tree

Prototype 1

Prototype 1

Prototype 2

Prototype 2

Prototype 2

References (http://db.nthu.edu.tw)Chen, A.L.P., M. Chang, J. Chen, J.L. Hsu, C.H. Hsu, and S.Y.S. Hua, Query by Music Segments:An Efficient Approach for Song Retrieval, in Proc. of IEEE Intl. Conference on Multimedia and Expo, 2000.Chen, J.C.C. and A.L.P. Chen, Query by Rhythm:An Approach for Song Retrieval in Music Database, in Proc. of IEEE Intl. Workshop on Research Issues in Data Engineering, 1998.Chou, T.C., A.L.P. Chen, and C.C. Liu, Music Databases: Indexing Techniques and Implementation, in Proc. of IEEE Intl. Workshop on Multimedia Data Base Management System, 1996.Hsu, J.L., C.C. Liu, and A.L.P. Chen, Efficient Repeating Pattern Finding in Music Databases, in Proc. of ACM Intl. Conference on Information and Knowledge Management, 1998.

References (http://db.nthu.edu.tw)Lee, W and A.L.P. Chen, Efficient Multi-Feature Index Structure for Music Data Retrieval, in Proc. of SPIE Conference on Storage and Retrieval for Image and Video Databases, 2000.Liu, C.C., J.L. Hsu, and A.L.P. Chen, An Approximate String Matching Algorithm for Content-Based Music Data Retrieval, in Proc. of IEEE Intl.Conference on Multimedia Computing and Systems, 1999.Liu, C.C., J.L. Hsu, and A.L.P. Chen, Efficient Theme and Non-Trivial Repeating Pattern Discovering in Music Databases, in Proc. of IEEE Intl. Conference on Data Engineering, 1999.