Journal of Intelligent Information Systems

Content-based and Collaborative Techniques for Tag Recommendation: an EmpiricalEvaluation

--Manuscript Draft--

Manuscript Number: JIIS-817R1

Full Title: Content-based and Collaborative Techniques for Tag Recommendation: an EmpiricalEvaluation

Article Type: Manuscript

Keywords: Recommender systems; Web 2.0; Collaborative Tagging; Folksonomies; Tagrecommendation

Corresponding Author: Pasquale Lops, Ph.D.University of Bari Aldo MoroItalia, ITALY

Corresponding Author SecondaryInformation:

Corresponding Author's Institution: University of Bari Aldo Moro

Corresponding Author's SecondaryInstitution:

First Author: Pasquale Lops, Ph.D.

First Author Secondary Information:

Order of Authors: Pasquale Lops, Ph.D.

Marco de Gemmis, Ph.D.

Giovanni Semeraro

Cataldo Musto

Fedelucio Narducci

Order of Authors Secondary Information:

Powered by Editorial Manager® and Preprint Manager® from Aries Systems Corporation

Noname manuscript No.(will be inserted by the editor)

Content-based and Collaborative Techniques for Tag

Recommendation: an Empirical Evaluation

Received: date / Accepted: date

Abstract The rapid growth of the so-called Web 2.0 has changed the surfers’behavior. A new democratic vision emerged, in which users can actively con-tribute to the evolution of the Web by producing new content or enrichingthe existing one with user generated metadata. In this context the use of tags,keywords freely chosen by users for describing and organizing resources, spreadas a model for browsing and retrieving web contents. The success of that col-laborative model is justified by two factors: firstly, information is organized ina way that closely reflects the users’ mental model; secondly, the absence of acontrolled vocabulary reduces the users’ learning curve and allows the use ofevolving vocabularies.

Since tags are handled in a purely syntactical way, annotations provided byusers generate a very sparse and noisy tag space that limits the effectiveness forcomplex tasks. Consequently, tag recommenders, with their ability of providingusers with the most suitable tags for the resources to be annotated, recentlyemerged as a way of speeding up the process of tag convergence.

The contribution of this work is a tag recommender system implement-ing both a collaborative and a content-based recommendation technique. Theformer exploits the user and community tagging behavior for producing recom-mendations, while the latter exploits some heuristics to extract tags directlyfrom the textual content of resources. Results of experiments carried out on adataset gathered from Bibsonomy show that hybrid recommendation strate-gies can outperform single ones and the way of combining them matters forobtaining more accurate results.

1 Introduction

The recent success of collaborative platforms has radically changed the roleof Internet users and the nature of services offered on the Web: the strong

Address(es) of author(s) should be given

ManuscriptClick here to download Manuscript: star_rebuttal.tex Click here to view linked References

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

2

dichotomy between web users and web administrators has been replaced bya new and more democratic vision where users can actively contribute to theevolution of the World Wide Web. This happens:

1. by producing new contents, the so-called User Generated Content (UGC);2. by enriching already available contents with novel metadata.

In platforms like YouTube1, Flickr2 or Wikipedia3 the contributions of thecommunity fall in the first category, because they share resources (new videos,photos, . . . ) that other users will enjoy. In collaborative tagging systems (likeBibsonomy4 or Delicious5) users contribute in the second way, by annotatingalready available resources through tags. That activity is generally known astagging.

Thanks to its immediacy and simplicity, tagging is one of the forms ofUGC that gained more attention from both users and research communities.The idea behind tagging is to identify those terms that better represent theinformation conveyed by a specific resource in order to simplify data organi-zation, description, browsing and retrieval. For example, a photo taken fromour holidays album could be annotated with the tags holidays, sardinia, beach,sea and so on. The set of tags used to annotate a specific resource is chosen ina totally subjective way: a photo could be annotated by a specific user withthe tag holidays, by another user with the tag summer and by another onewith both of them. The choice of tags is absolutely free and strictly dependenton the user mental model. Furthermore, if the same resource is annotated byseveral users in different times, some of them may reuse tags already assignedby other users, while some others may introduce new tags. The process of col-laborative annotation of a resource allows for the construction of a bottom-upclassification schema, called folksonomy [17]. We can define a folksonomy asthe set of tags used by a community of users to annotate a specific resource.The inceptive idea behind this lexical structure is that the more a tag is usedby the community to annotate a resource, the more is the likelihood the tagcorrectly describes that content.

As underlined by Mathes [23], thanks to folksonomies users can freelymodel the information without the constraints of a predefined lexicon or hier-archy. Moreover, collaborative tagging systems gained more and more atten-tion because they take into account the way all users conceive the informationcontained in a resource, in contrast to the classical paradigm where the infor-mation about resources is provided by a small subset of people, such as website administrators [37].

However, the simplicity of the approach has also some important draw-backs. As stated in [14], the information in folksonomies is modeled in a simplelexical way, thus the classic problems of Information Retrieval (IR) systems

1 http://www.youtube.com/2 http://www.flickr.com/3 http://en.wikipedia.org/4 http://www.bibsonomy.org/5 http://delicious.com/

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

3

occur, such as polysemy, synonymy and level variation. Polysemy concernsthe situation of tags having multiple meanings. We can think of a resourcetagged with the term turkey: it might indicate a news in an online newspaperabout politics in the country Turkey or a recipe for the Thanksgiving Dayor a post/comment on a film. Synonymy indicates multiple tags that share acommon meaning. In collaborative tagging synonymy appears through simplemorphological variations, such as the tags holiday, holidays, 2010 holidays toidentify the photos taken from our 2010 holidays album, but also through tagssemantically similar, such as resources tagged with holidays versus vacation.The level variation problem refers to the phenomenon of tagging at differentlevels of abstraction: some people can annotate a photo with a beach landscapewith the tag beach, but also with a more generic terms such as sea. Finally, theset of tags is modeled in a flat way, without any kind of relation connectingthem.

All these drawbacks undoubtedly hinder the use of folksonomies for com-plex tasks. For example in a simple scenario, the retrieval of resources anno-tated with just the tag holidays will exclude all the resources annotated withthe tag vacation although there is a strong and obvious correlation betweenthese concepts. In the same way, by retrieving resources annotated with thetag turkey a collaborative system will return heterogeneous entries, includingresources not related to the user needs.

In order to reduce the impact of these drawbacks and to emphasize thequality of collaborative tagging systems by speeding up the so-called tag con-vergence [9], many tools were recently developed to facilitate the user in thetask of tagging. Those systems, usually known as tag recommenders, are ableto provide users with suggestions about tags to be used.

In general, tag recommenders work in a very simple way:

1. a user posts a resource;2. the tag recommender analyzes some information related to that resource

(usually metadata) depending on the approach;3. the tag recommender processes that information and produces a list of

recommended tags;4. the user freely chooses the most appropriate tags to annotate that resource.

This paper presents STaR, a tag recommender system designed by takinginto account very general principles which allow to make the system usablewith different resources in different scenarios. More specifically, the systemimplements both a social/collaborative and a content-based approach thatcan be effectively combined in different ways depending on the specific task(Section 2.4). STaR merges a collaborative technique already presented in[1], with a content-based recommendation algorithm which extracts tags bydirectly analyzing the content of the resource.

The solution implemented in STaR combines three different approachesbased on the following main assumptions:

– textual metadata describing resources to be annotated is the primary sourcefor recommending tags (content-based approach);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

4

– previous tagging activity of users is taken into account, in order to assign ahigher weight to tags already used to annotate similar resources (user-basedapproach);

– different users tend to annotate similar contents with similar tags (social-based approach).

The last assumption was used for the definition of a collaborative recom-mendation model, while the first two assumptions were exploited for devel-oping a content-based recommendation model, which plays a key role since itallows to face the typical cold start problem of collaborative strategies, i.e. thenew item and the new user problem. The new item problem refers to items notyet tagged by other users, while the new user problem refers to users not yetactive in tagging resources. In both these situations collaborative approachesusually fail.

The paper is organized as follows. Section 2 depicts the general systemarchitecture and the recommendation approaches implemented in STaR. Sec-tion 3 describes the experimental evaluation carried out in the context of theECML-PKDD 2009 Discovery Challenge6, using a state of the art datasetgathered from Bibsonomy containing bookmarks and BibTeX entries. Section4 analyzes some related works, while conclusions and future works are drawnin the last section.

2 Architecture of the STaR Tag Recommender System

2.1 Tag Recommendation Scenario

The tag recommendation task for a given user and a specific resource can bedescribed as the generation of a set of tags according to some criterion.

Following the definition introduced in [17], a collaborative tagging systemcan be formally described using the following finite sets:

– U: a set of users ;– R: a set of resources ;– T: a set of tags.

Each user u ∈ U annotates a resource r ∈ R by using a tag t ∈ T (moretags can be used to annotate a resource). Each annotation provided by the useru on the resource r can be described using a ternary relation between users,resources and tags called Tag Assignment function (TAS). More formally:

TAS ⊆ U ×R× T (1)

The tag recommendation task for a given user u ∈ U and a resource r ∈ R

consists in generating a set of tags T ⊆ T according to some criterion. Theapproach builds a preliminary set of candidate tags. Each tag is then assigned

6 http://www.kde.cs.uni-kassel.de/ws/dc09/

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

5

Fig. 1 Architecture of STaR

with a score and the tags are finally ranked. The top-k tags are returned tothe user.

The inceptive idea behind STaR is to exploit both a content-based anda collaborative technique for providing users with a set of tags able to effec-tively describe the information conveyed by the resource. For the content-basedcomponent of the algorithm, the recommendation is based on the analysis ofinformation coming from the content of the resources, while the collaborativecomponent is based on the analysis of similar resources that users have alreadytagged. The insight behind the collaborative approach is that two resourcesthat convey a similar information (e.g. two papers referring to the same topic)can be annotated with the same, or at least a similar set of tags.

Figure 1 shows the functional architecture of STaR. The collaborative partis based on the analysis of similar resources, while the content-based partexploits the content of resources in order to directly extract tags. Both theapproaches produce a set of candidate tags, named Collaborative CandidateTags and Content-based Candidate Tags. These sets are finally merged in orderto obtain a final set of recommendations. Specifically, the relevance of each tagis obtained by a linear combination of the partial relevance scores returned bythe collaborative and content-based part of the recommendation algorithm.

In the next sections the recommendation models implemented in STaRare thoroughly analyzed, and different combinations of the approaches aredescribed and experimentally evaluated.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

6

2.2 Collaborative approach

The collaborative recommendation approach relies on the analysis of similarresources already tagged by the user (tagging behavior) or by the entire com-munity of users.

The approach exploits two indexes built by the Indexer module containing1) the set of resources tagged by a specific user – Personal Index, 2) theset of resources previously tagged by the entire community – Social Index.More details about the indexing of resources are in Section 2.2.1. When a newresource is posted on the platform, the Processor component extracts relevantinformation about the target user as well as the metadata about the resourceto be annotated and submits a query against both the Personal Index andthe Social Index in order to retrieve the most similar resources (to that tobe annotated) and collect the most relevant tags used by the community toannotate them.

More specifically, given a user u and a resource r to be annotated, STaRreturns the resources whose similarity with r is greater or equal than a certainthreshold β (tuned in the experimental session). Similarity between resourcesis computed by using text similarity measures. These techniques can be appliedto textual documents (such as web sites, scientific papers and so on) as wellas multimedia content, such as videos and images, provided that some textualfeatures (e.g. a summary of the video or a caption of the image) is available.More details are provided in Section 2.2.2.

The Collaborative Tags Extractor builds a set of Collaborative CandidateTags by extracting tags assigned to the most similar resources retrieved fromthe indexes. More specifically, Collaborative Candidate Tags contains two dif-ferent sets of tags, named Personal Tags and Social Tags. The Personal Tagsset contains tags extracted from resources retrieved from the Personal Index,while the Social Tags set contains tags extracted from resources retrieved fromthe Social Index. For each tag, a score is computed by weighing the similarityscore returned by the query, with the normalized occurrence of the tag. Finally,the Collaborative Candidate Tags list will contain the top n tags, previouslyordered according to their descending relevance score.

2.2.1 Indexing of Resources

Given a collection of resources (corpus), the Indexer module exploits theApache Lucene7 APIs in order to store all the information about the resourcesalready annotated by a specific user and the community as well. In the sce-nario presented and evaluated in this paper the corpus contains bookmarksand BibTeX entries. Each bookmark consists of the title of the Web page,its URL, and a description provided by users (Figure 2), while each BibTeXentry contains the title, the authors, the abstract and the journal/book name(Figure 3).

7 http://lucene.apache.org

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

7

id = 135005

title = ’AdamNation.org’

url = ’http://adam.easyjournal.com/entry.aspx?eid=2632426’

Descripion = ’where is tagging going?’

Fig. 2 Example of a Bookmark

id = 688159,

Journal = ’IEEE Intelligent Systems’

Number = ’18’

Date = ’March/April’

Title = ’Identifying Communities of Practice through Ontology Network Analysis’,

Authors = ’Harith Alani and Srinandan Dasmahapatra and Kieron O’Hara and

Nigel Shadbolt’

Abstract = ’This article describes Ontocopi, a tool for identifying communities

of practice by analyzing ontologies of relevant working domains.

Ontocopi spots patterns in ontological formal relations, traversing

the ontology from instance to instance via selected relations.’

Year = ’2003’

Fig. 3 Example of a BibTeX entry

Let U be the set of users and N be the cardinality of this set, the indexingprocedure is repeated N +1 times: an index for each user (Personal Index ) isbuilt, storing the information on her previously tagged resources; an index forthe whole community (Social Index ) is also built by storing the informationabout all the resources previously tagged by the community.

By exploiting the definitions presented in Section 2.1, given a user u ∈ U ,PersonalIndex(u) can be defined as:

PersonalIndex(u) = r ∈ R|∃t ∈ T : (u, r, t) ⊆ TAS (2)

where TAS is the tag assignment function (see Equation 1).SocialIndex represents the union of all the user personal indexes:

SocialIndex =

N⋃

i=1

PersonalIndex(ui) (3)

2.2.2 Retrieval of Similar Resources

When both the Personal Index and the Social Index are available, STaR isable to extract the most similar resources (to that to be tagged) already taggedby the target user (PersonalRes) or by the other users in the community(SocialRes). The title of the resource is used as query against the indexesboth for bookmark and BibTeX entries.

More formally, given a user u ∈ U and a resource r to be tagged, Lucene re-turns the resources whose similarity with r is greater or equal than a thresholdβ:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

8

Fig. 4 Retrieval of Similar Resources

PersonalRes(u, r) = r′ ∈ PersonalIndex(u)|sim(r, r′) ≥ β (4)

SocialRes(r) = r′′ ∈ SocialIndex|sim(r, r′′) ≥ β (5)

In order to improve the performances of the Lucene Querying Engine theoriginal Lucene Scoring function has been replaced with an Okapi BM25 im-plementation8. BM25 is nowadays considered as one of the state-of-the artretrieval models by the IR community [27]. In a previous study [2], we provedthat the BM25 integration increases the overall accuracy of STaR. The simi-larity score returned by Lucene has been normalized to obtain values fallingin the interval [0, 1].

Figure 4 depicts an example of the described process. The target resourceis represented by The New York Times, one of the most famous Americannewspaper. Proper queries allows to access the Personal Index and get theThe New York Post as the most similar resource to The New York Times.The Social Index, instead, returns the CNN and the Daily News.

2.2.3 Extraction of Collaborative Candidate Tags

The Collaborative Tags Extractor gets the most similar resources returned byApache Lucene and builds a set of Collaborative Candidate Tags by extracting

8 http://nlp.uned.es/ jperezi/Lucene-BM25/

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

9

tags assigned to these resources. For each tag, a score is computed by weighingthe similarity score returned by Lucene with the normalized occurrence of thetag. Tags coming from the Personal Index or the Social Index can be weigheddifferently, for example in order to boost users’ previously used tags.

Formally, for each query q (namely, the title of the resource to be tagged),we can define a set of tags to recommend by building two sets: PersonalTags(u, q)and SocialTags(q). These sets are defined as follows:

PersonalTags(u, q) = t ∈ T |r ∈ PersonalRes(u, q) ∧ (u, r, t) ⊆ TAS (6)

SocialTags(q) = t ∈ T |r ∈ SocialRes(q) ∧ (u, r, t) ⊆ TAS (7)

The set of Collaborative Candidate Tags is defined as:

CollCandTags(u, q) = PersonalTags(u, q)⋃

SocialTags(q) (8)

For each personal or social tag, a relevance with respect to the query q iscomputed:

relpersonal(t, u, q) =

∑r∈PersonalRes(u,q) n

tr ∗ sim(r, q)

nt(9)

relsocial(t, q) =

∑r∈SocialRes(q) n

tr ∗ sim(r, q)

nt(10)

where ntr is the number of occurrences of the tag t in the annotation for the

resource r and nt is the sum of the occurrences of tag t among all similarresources. Finally, the global relevance assigned to each tag t is defined as:

rel(t, q) = α ∗ relpersonal(t, u, q) + (1− α) ∗ relsocial(t, q) (11)

where α is a factor to assign differents weights to personal or social tags. α iscalled PersonalTagWeight, while (1 − α) is called SocialTagWeight.

Figure 5 depicts the process performed by the Collaborative Tags Extractor :in this case we have a set of three social tags (News, Stories, and Events) andthree personal tags (Paper, Blog and News). These sets are then merged,building the set of Collaborative Candidate Tags. This set contains five tagssince the tag “News” occurs both in social and personal tags. To each tag thesystem associates a score that measures its effectiveness for the target resource.Besides, the scores for the Collaborative Candidate Tags are weighed againaccording to PersonalTagWeight and SocialTagWeight values (in the example,0.7 and 0.3 respectively), in order to boost the tags already exploited by theuser in the final tag rank. At the end of the process the system will suggestthose tags whose score is greater than a certain threshold. For example, settinga threshold γ = 0.40, the system would suggest the tags news, paper and blog.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

10

Fig. 5 Description of the process performed by the Collaborative Tags Extractor

2.3 Content-based approach

The content-based recommendation strategy might help to tackle the coldstart problem of the collaborative strategy mentioned in Section 1.

Given a resource r to be annotated and S = s1, ..., sn the set of slots,i.e. a set of fields describing the content of the resource, we can define the setof Content-based Candidate Tags as follows:

ContentCandTags(r) =⋃

si∈S

ContentCandTagssi(r) (12)

where contentCandTagssi(r) is the set of candidate tags coming from the slotsi. Different slots were adopted for bookmarks and BibTeX entries.

As regards the bookmarks, the main idea is to extract tags by analyzingthe HTML source of the Web page according to some heuristics. Starting fromthe URL of the bookmark, the Metadata Analyzer module (Figure 1) gets theHTML source of the Web page. Next, the Content Tags Extractor extracts thestring representing the domain name directly from the URL (for example, thetag cnn is extracted from www.cnn.com). This represents the first element ofthe Content-based Candidate Tags set. The other candidate tags are extractedby analyzing the meta-tags stored in the HTML source, namely 〈title〉 and〈meta〉. As regards the tag 〈meta〉, the value of the attributes keywords anddescription are extracted, since in our opinion they are the most informativeand significant. The final list of Content-based Candidate Tags is filtered byremoving stopwords and verbs.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

11

When the set of Content-based Candidate Tags is built, the algorithm as-signs a score to each tag, that is the weighed sum of the normalized occurrenceof the tag in each slot multiplied for the weight of that slot. More formally,the score of the tag t is:

relcontent(t, r) =∑

si∈S

wsi · nt,r (13)

where si identifies the slot of the candidate tag, wsi is the weight of the slotsi, and nt,r is the normalized occurrence of the tag t in the HTML source ofthe resource r (nt,r = 0 if the tag t does not occur in the slot si). Severalcombinations of weights assigned to each slot (url, title, meta keywords,meta description) were evaluated in order to define the most appropriateones.

In a similar way, a Content-based Candidate Tags set is built for eachBibTeX entry. We considered title, journal/book name, and abstract as themost significant slots to extract a set of candidate tags. In order to build theset of candidate tags, the text associated to each slot is processed in order toremove stopwords. Given a candidate tag t and a resource r, we compute:

– nsi : the number of occurrences of t in the slot si of resource r;– N : the sum of occurrences of t in the different slots for all the BibTeX

entries in the whole dataset;– Ntag: the number of times t is used as a tag in the whole dataset.

All the values are normalized in a range between 0 and 1. The score of acandidate tag t for the resource r is computed by the following formula:

relcontent(t, r) = ǫ · (∑

si∈S

nsi · wsi) + (1 − ǫ) · tagf (14)

where wsi is the weight of the slot si, S is the set of slots, tagf is the frequencyof using t as a tag in the whole dataset:

tagf =Ntag

N(15)

The parameter ǫ weighs differently the significance of the tag t for the resourcewith respect to its frequency of use in the whole dataset. In our experiment ǫis set to 0.5 to give the same importance to both the components.

2.4 Tag Recommendation step

After the tag extraction step, the Filter component removes from Content-based Candidate Tags and Collaborative Candidate Tags those tags whose scoreis under a certain threshold, and decides which set(s) to exploit to providerecommendations. STaR implements two different hybrid strategies:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

12

– Cascade: a unique set of tags (among Social, Personal, or Content-based)is selected. If that set is empty the others are used. The first non-emptyset is returned as recommendations for the target user. According to anexperimental evaluation performed in [3], the highest accuracy is obtainedby recommending first content-based tags, then personal tags, and finallysocial tags;

– Fusion: a weight is assigned to each set of tags (for example: 0.5 to theContent-based tags, 0.3 to the Personal tags and 0.2 to the Social tags).Those sets are merged by weighing each tag in a proper way.

In [3] we demonstrated that the Fusion approach outperforms the Cascadeone, thus in this work we adopted the former strategy.

Formally, given a user u, a tag t and a resource q to be annotated, let α, βand γ the weights for the Personal, Social and Content-based tags respectively,the relevance of the tag t for the resource q can be defined as:

score(t, q) = α ∗ relpersonal(t, u, q) + β ∗ relsocial(t, q) + γ ∗ relcontent(t, q)(16)

3 Experimental Evaluation

The goal of the experimental evaluation is threefold:

1. to evaluate the accuracy of the content-based approach when different com-binations of weights for the slots are adopted;

2. to evaluate the accuracy of the collaborative approach, by distinguishingthe accuracy of the single sets of tags, namely Personal Tags, Social Tags ;

3. to evaluate the accuracy of the Fusion approach, by weighing differentlythe previous sets of tags.

3.1 Description of the dataset

In this experiment we adopted the dataset used for the content-based recom-mendation task at the ECML-PKDD 2009 Discovery Challenge. The main goalis to exploit content-based recommendation approaches in order to provide arelevant set of tags to the user when she submits a new item, bookmark orBibTeX entry, into Bibsonomy.

The organizers made available a training set with some examples of tagassignment containing 263,004 bookmark posts and 158,924 BibTeX entriessubmitted by 3,617 different users. For each of the 235,328 different URLs, thetitle of the resource and the description were provided as textual metadata. Forthe 143,050 different BibTeX entries, the title of the publication, the abstractand the journal/book title were provided.

We evaluated STaR by comparing the real tags (namely, the tags a useradopts to annotate a new unseen resource) with the suggested ones. The accu-racy was finally computed using classical Information Retrieval metrics, suchas Precision (Pr), Recall (Re) and F1-Measure (F1) [30].

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

13

3.2 Experiment 1: Evaluation of the Content-based Approach

Table 1 presents the results obtained by the content-based approach on book-marks with different combinations of weights for the slots title, description,and keywords.

Table 1 Results of the Content-based Approach for Bookmarks

Configuration Precision Recall F1title (0.8), description (0.1), keywords (0.1) 20.85 18.33 19.51title (0.1), description (0.1), keywords (0.8) 23.40 14.93 18.23title (0.1), description (0.8), keywords (0.1) 22.17 14.33 17.41title (0.5), description (0.0), keywords (0.5) 21.25 18.21 19.61

title (0.6), description (0.1), keywords (0.3) 21.27 18.43 19.70title (0.3), description (0.1), keywords (0.6) 22.31 17.51 19.63

The best result in terms of F1 measure (highlighted in boldface) is achievedby assigning 0.6, 0.1 and 0.3 to the slot title, description and keywords, re-spectively. Tags extracted from the keywords slot are more accurate in termof precision with respect to tags extracted from the other slots. This resultwas expected, since keywords are generally provided by humans in order toidentify a Web page content. In order to improve the performance in termsof recall, we increased the weight of the slot title, that generally well sum-marizes the content of a Web page. The slot description is quite accurate interms of precision as well, even though it might contain quite long and noisydescriptions.

Table 2 presents the results obtained by the content-based approach onthe BibTeX entries with different combinations of weights for the slots title,abstract, and journal/book name.

Table 2 Results of the Content-based Approach for BibTeX entries

Configuration Precision Recall F1title (0.8), abstract (0.1), journal/book (0.1) 14.55 16.55 15.49title (0.8), abstract (0.1), journal/book (0.1) 15.32 16.03 15.67title (0.1), abstract (0.1), journal/book (0.8) 14.60 16.48 15.48title (0.1), abstract (0.8), journal/book (0.1) 14.04 15.83 14.89title (0.6), abstract (0.1), journal/book (0.3) 14.78 16.74 15.70title (0.6), abstract (0.1), journal/book (0.3) 15.70 16.19 15.93title (0.5), abstract (0.3), journal/book (0.2) 14.77 16.70 15.68

title (0.5), abstract (0.3), journal/book (0.2) 16.10 16.29 16.19

The best combination of weights in terms of F1 measure (again in boldface)is 0.5, 0.3 and 0.2 for the slot title, abstract and journal or book name, re-spectively. As for bookmarks, the title is the most significant slot to take intoaccount for obtaining accurate results, since it generally contains keywordsstrictly related to the topic of the BibTeX entry, while the abstract must becarefully used, since it might be too noisy due to its length. The highest value

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

14

of recall is obtained by weighing more the journal/book name, since that fieldcontains very generic keywords.

3.3 Experiment 2: Evaluation of the Collaborative Approach

Table 3 presents the accuracy of the sets of tags produced by the Collabora-tive Tags Extractor, namely Personal Tags and Social Tags. The threshold β

for retrieving the most similar resources from which to extract collaborativecandidate tags is experimentally tuned to 0.4. Experiments are not reportedfor the sake of brevity.

The effectiveness of the collaborative approach is strictly dependent on theamount of the available resources already annotated. Since the approach basedon Personal Tags builds the set of candidates tags on the ground of the re-sources the user annotated in the past, we observed a significant loss in termsof F1 measure with respect to the content-based approach (-10.12% for book-marks, -6.32% for BibTeX entries). In order to overcome this problem, the rec-ommendation model based on Social Tags analyzes the set of resources alreadytagged by all the users in the community, and this reduces the loss in terms ofF1 with respect to the content-based approach (-7.29% for bookmarks, -4.17%for BibTeX entries). The better performance of the content-based method isdue to the fact that it is totally independent from the number of the resourcesalready annotated in the corpus, and also because it can effectively tackle thecold start problem.

Table 3 Results of the Collaborative Approach

Resource Recommendation set Pr Re F1Bookmark Personal Tags 11.20 8.34 9.58Bookmark Social Tags 13.80 11.28 12.41BibTex Personal Tags 10.72 9.15 9.87BibTex Social Tags 12.35 11.71 12.02

3.4 Experiment 3: Evaluation of the Hybrid/Fusion Approach

In this session we investigated whether the fusion approach for combiningdifferent sets of tags is useful to improve the predictive accuracy of STaR.

Results for bookmarks and BibTeX entries are reported in Table 4 andTable 5, respectively. As regard the bookmarks, we can observe that all theconfigurations integrating the content-based approach outperform that notintegrating it. The best configuration in terms of F1 measure (highlighted inboldface) is obtained by weighing in the same way Personal Tags and Content-based Tags, while the worst result is obtained by combining Personal Tags (0.5)and Social Tags (0.5). In general, the higher the weight of the content-based

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

15

Table 4 Results of the Fusion Strategy for Bookmarks

Configuration Pr Re F1

Personal (0.5)+Social (0.5) 13.35 18.11 15.37Personal (0.5)+Content-based (0.5) 18.59 22.06 20.18

Personal (0.7)+Content-based (0.3) 17.49 21.22 19.18Personal (0.5)+Content-based (0.3)+Social (0.2) 17.07 20.19 18.47Personal (0.3)+Content-based (0.2)+Social (0.5) 15.34 19.17 17.04

Personal (0.3) + Content-based (0.6) + Social (0.1) 22.48 17.06 19.40

Table 5 Results of the Fusion Strategy for BibTeX entries

Configuration Pr Re F1

Personal (0.5)+Social (0.5) 12.83 15.36 13.98Personal (0.5) + Content-based (0.5) 16.39 15.32 15.84

Personal (0.7) + Content-based (0.3) 15.51 17.15 16.29Personal (0.3) + Content-based (0.7) 16.96 14.17 15.75

Personal (0.5) + Content-based (0.3) + Social (0.2) 13.30 16.30 14.65Personal (0.4) + Content-based (0.3) + Social (0.3) 15.08 15.71 15.39

approach, the better the precision, while the introduction of Social Tags hurtsthe precision. It seems that Social Tags introduce some noise when the systemhas difficulties to retrieve similar resources previously tagged by the user.Finally, Personal Tags play a prominent role for obtaining high performancein terms of recall.

Similar outcomes were obtained for BibTeX entries. The best result isobtained by combining Personal Tags (0.7) and Content-based Tags (0.3).Higher weights for the content-based approach hurt the performance in termsof recall, but the precision is improved. The reason might be that severalBibTeX entries have an empty abstract slot, thus the set of Content-basedTags is not huge, but those few elements are quite accurate. This is the reasonwhy we weighed more the Personal Tags with respect to the Content-basedones. Also for BibTeX entries we observe a loss of performance by introducingSocial Tags.

Since the experiments were performed on the dataset used for the ECML-PKDD 2009 Discovery Challenge, we compared the best configuration of STaRwith the top five systems participating to the competition. The challenge con-sisted of two different tasks for content-based and graph-based methods. Wecompared our work with the systems participating to the content-based task.We downloaded the result sets submitted by each participant from the ECML-PKDD 2009 Discovery Challenge home page and we calculated precision andrecall by evaluating the accuracy of their predictions on bookmarks and Bib-TeX entries. Results are presented in Table 6 and Table 7.

We can observe that STaR is ranked second in recommending tags for Bib-TeX entries and fourth in recommending tags for bookmarks. The Lipzack’ssystem described in the related work (Section 4) slightly outperforms STaR inrecommending BibTeX entries (F1 +1.04%), but it does not adopt a pure con-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

16

Table 6 STaR vs top five systems for BibTeX entries

System F1

Lipczak et al. [21] 17.33STaR 16.29

Ju and Hwang [18] 15.89Mrosek et al. [25] 12.50Zhang et al. [38] 12.03Wang et al. [35] 11.47

Table 7 STaR vs top five systems for Bookmarks

System F1

Mrosek et al. [25] 26.47Ju and Hwang [18] 21.11Lipczak et al. [21] 20.85

STaR 20.18Wang et al. [35] 17.94Zhang et al. [38] 17.35

tent based approach. Indeed, it is a hybrid recommender combining content-based and graph-based techniques, while STaR can be seen as a pure content-based system, since the best configuration does not exploit social tags. Thisallows to claim that STaR is the best pure content-based method for recom-mending BibTeX entries among those participating to the challenge. Anotheradvantage is that STaR avoids any operation related to the construction andnavigation of graphs, and moreover it can provide recommendations even incase of a very few number of users and resources.

As regards bookmarks, the best system proposed by Mrosek et al. [25]considerably outperforms STaR (F1 +6.29%), but it exploits additional infor-mation coming from external data sources such as Del.icio.us. This means thatthe comparison with STaR is not fair, since almost 90% of resources occur onlyonce in the dataset, thus external information are very useful in that situation.The results obtained by the other two systems are comparable to STaR. Morespecifically, for the system proposed by Lipzack et al. [21] (F1 +0.67%) wehave the same previous observation related to the use of graph-based meth-ods, while for the system proposed by Ju and Hwang [18] (F1 +0.93%) theslight improvement is due to a smarter weighting method of keywords ex-tracted from resource descriptions, even though the approach is very similarto that implemented in STaR.

4 Related Work

Related work in the area of tag recommendation can be broadly divided intothree classes: collaborative, graph-based and content-based approaches.

The collaborative approach for tag recommendation presents some analo-gies with the collaborative filtering methods [7]. These systems compute the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

17

relevance of a tag to be suggested by exploiting the community behavior. Inthe model proposed by Mishne and implemented in AutoTag [24], the systemsuggests tags based on the other tags associated with similar posts in a givencollection. The recommendation process is performed in three steps: first, thetool finds similar posts and extracts their tags. All the tags are then merged,building a general folksonomy that is filtered and re-ranked. The top-rankedtags are suggested to the user, who selects the most appropriate ones to be at-tached to the post. TagAssist [32] improves the AutoTag approach performinga lossless compression over existing sets of tags. It finds similar blog posts andsuggests a subset of the associated tags through a Tag Suggestion Engine whichleverages previously tagged posts providing appropriate suggestions for newcontent. In [22] the tag recommendation task is performed through a user-based collaborative filtering approach. The method seems to produce goodresults when applied to the user-tag matrix, so the authors show that userswith similar tag vocabularies tend to tag alike. In [13] the authors proposean adaptation of the classical K-nearest neighbor approach to create a set ofrecommended tags. The neighbors are defined by looking for users tagging thesame resources with the same tags. In this way tags used by similar users areboosted in the set of recommended tags. The approach adopted by our tagrecommender is inspired to AutoTag and TagAssist which have the advantageof using similarity between resources rather than similarity between users, thatmay be not significant, since generally each user assigns tags to resources in away that closely reflects her own mental model.

The problem of tag recommendation through graph-based approaches hasbeen firstly addressed by Jaschke et al. [17]. They compared some recommen-dation techniques including collaborative filtering, PageRank and FolkRank.The key idea behind FolkRank algorithm is that a resource which is taggedby important tags from important users becomes important itself. The sameconcept holds for tags and users, thus the approach uses a graph whose ver-texes mutually reinforce themselves by spreading their weights. The evaluationshowed that FolkRank outperforms other approaches. Schmitz et al. [29] pro-posed association rule mining as a technique that might be useful in the tagrecommendation process. In order to model the ternary relationship amongusers, resources and tags, in [33] the authors proposed to use a 3-order tensor(a tensor is used to define relationships among vectors). To capture the latentassociation among user-resource-tag a Latent Semantic Analysis algorithm isexecuted on tensors. In [15], a novel method for key term extraction fromtext documents is presented. Firstly, each document is modeled as a graphin which nodes are terms and edges represent semantic relationships betweenthem. These graphs are then partitioned using communities detection tech-niques and weighed exploiting information extracted fromWikipedia. The tagscomposing the most relevant communities (a set of terms related with the topicof the resource) are then suggested to the user. In [31], the authors perform acomparison of a graph-based method, which extracts the topic of the resourceto be tagged by exploiting graph partitioning algorithms over the resource/tagand resource/term graphs, with a classification algorithm based on Gaussian

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

18

process, which provides recommendations by selecting the most relevant tagscoming from the class the resource belongs to. The experiments showed thatthe latter approach is able to outperform the graph-based method as well asthree simple baselines based on machine learning techniques (Support VectorMachine, Latent Dirichlet Allocation and Vector Similarity).

Content-based tag recommenders are based on the assumption that tagsfor a specific resource can be extracted by processing the textual informa-tion about the resource to be annotated. Usually these approaches adapt In-formation Retrieval-related techniques [4] in order to extract relevant terms(unigrams or bigrams) to label a resource. Obviously resources have to beequipped by textual descriptions. Brooks and Montanez [8] developed a tagrecommender system that automatically suggests tags for a blog post by ex-tracting the top three terms exploiting the TF-IDF scoring [28]. Another workexploiting the TF-IDF measure is KEA [36]. This system applies a supervisedBayesian classifier able to extract relevant phrases from the textual content.The system presented by Lee and Chun [19] recommends tags retrieved fromthe content of a blog using artificial neural networks. The network is trainedbased on statistical information about word frequencies and lexical informa-tion about word semantics extracted from WordNet. Other systems exploitontologies for recommending tags: in [5], terms are extracted from the doc-ument and subsequently, surfing the ontology, more abstract and conceptualtags are suggested. Similarly to the content-based approaches previously de-scribed, we have extracted tags from textual description of resources (book-marks and BibTeX entries). We have designed a simple extraction pipelinewhich has two main peculiarities:

1. it is able to exploit the structure of resources, such as that of bookmarks(Fig. 2) and BibTeX entries (Fig. 3) to weigh differently specific parts ofresources according to their importance (e.g. title in a BibTeX entry couldbe more informative than journal name)

2. it is able to automatically extract metadata as an additional source ofinformation, by analyzing the HTML source code starting from the URLof bookmarks.

It is worth to note that most of the approaches proposed in literature aresyntactic, and do not use semantic techniques. We considered the opportunityto adopt semantic approaches, but we finally decided to avoid their adoptionfor two main reasons:

1. In the specific tag recommendation task, the main goal was to exactlypredict tags adopted by users. This means that, even though semanticapproaches allow to understand the correct meaning of tags, this could benot useful for the tag recommendation task. Indeed, even if the conceptassociated with a tag is correctly identified, we need a way to choose theright word form to explicit that concept, and this can be even more difficultdue to the inherent personal traits involved in linking tags to resources

2. the participation to the ECML-PKDD 2009 Discovery Challenge provideda strong evidence that semantic approaches are not useful in that specific

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

19

context. Indeed, the best performing algorithms to tag recommendationwere mainly based on collaborative or content-based strategies, possiblywith the integration of external resources (Del.icio.us for example) [21,25].An attempt to adopt a semantic approach was proposed by Murfi in [26],in which Nonnegative Matrix Factorization (NMF) is applied to extractsemantic concepts from document collections. The authors motivated theadoption of NMF instead of other semantic techniques such as LSA mainlyfor its computational efficiency. However, results obtained by applying theNMF model are worse than those obtained using other state-of-the-artmethodologies.

Our idea of merging in a different way different recommendation methodsis inspired by several works. Indeed, in the literature there are some hybridmethods integrating two or more approaches in order to reduce their typicaldrawbacks and point out their qualities. Lipczak et al. [21] present a tag rec-ommender that combines a content-based approach with a graph-based one.The recommender suggests tags for bookmarks and BibTeX by exploiting theresource content, the resource related tags and the user profile tags. The con-tent used by the recommender is the title of the resource and its URL (only forbookmark). Two graphs are built as well: a TitleToTag graph and a TagToTaggraph. The former discovers relations between words from resource title andtags used to label that resource; the latter captures relations among tags byexploiting their co-occurrences. A similar approach is presented in [18] wherethe authors exploit three kinds of information sources: the description of theresources, their folksonomies, and the tags previously used by the same per-son. In order to remove inappropriate candidate tags, a filtering method anda weighting scheme for assigning different importance to sources are defined.Heymann et. al [16] present a tag recommender that exploits at the same timesocial knowledge and textual sources. They suggest tags based on page text,anchor text, surrounding hosts, adding tags introduced by other users to labelthe URL. The effectiveness of this approach is also confirmed by the use ofa large dataset crawled from Delicious for the experimental evaluation. A hy-brid approach is also proposed in [20]. Firstly, the system extracts tags fromthe title of the resource. Afterwards, based on an analysis of co-occurrences,the set of candidate tags is expanded adding also tags that usually co-occurwith terms in the title. Finally, tags are filtered and re-ranked by exploitingthe information stored in a so-called “personomy”, the set of the tags of theuser. In [34] the authors proposed a model based on both textual content andtags associated with the resource. They introduce the concept of conflatedtags to indicate a set of related tags (like “blog”, “blogs”, ecc.) used to anno-tate a resource. Modeling in that way the existing tag space they are able tosuggest various tags for a given bookmark by exploiting both user and docu-ment representations. However, the use of more complex techniques to mergesets of tags coming from different information sources is proposed in [11],where the authors propose a model that exploits Bayesian networks to com-bine content-based and collaborative features. Thanks to Bayesian networks

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

20

the relationships between users and items as well as their strenght can be ex-plicitly encoded. Next, recommendations are generated by using probabilisticreasoning and the probability distribution over the expected rating is finallycomputed. The effectiveness of the approach, evaluated with respect two state-of-the-art datasets, confirmed the validity of the insights behind the model,and represents a promising research direction for future research. Finally, adifferent point of view of the tag recommendation task emerged from [10],where the authors addressed the problem of tag recommendation for Flickrusers, that is to say, the problem of suggesting tags that match the interestsof the users instead of the information conveyed by the resource itself. As inour model, the authors propose the adoption of techniques based on MachineLearning in order to merge and combine both textual and social features (thepopularity of the tag for the user and its representativeness for Flickr commu-nity, respectively) when tags describing users have to be recommended. Theexperimental results, performed on a Flickr dataset, confirmed the goodnessof the insight behind the model.

5 Conclusions and Future Work

Collaborative Tagging Systems are powerful tools, since they let users to orga-nize the information in a way that perfectly fits their mental model. However,typical drawbacks of collaborative tagging systems represent an hindrance,since the complete tag space is too noisy to be exploited for retrieval and fil-tering tasks. In this scenario, systems that assist users in the task of taggingare more and more required since they speed up the tag convergence.

In this paper we presented the STaR tag recommender system. The ideabehind our work is to discover similarities among resources in order to ex-ploit communities and user tagging behavior. Moreover, a content-based ap-proach able to extract tags by analyzing the textual content of resources to beannotated has been integrated: we showed that this improves the predictiveaccuracy of the whole recommendation model.

We exploited three different tag sources and we used a simple linear com-bination of tag scores as a merging strategy. We observed that the communitybehavior, while is a precious support when the user has not a previous tag-ging history, could introduce some noise in the recommendation process. Thecontent-based approach plays a prominent role to achieve a good predictiveaccuracy, therefore we are planning to integrate a tool able to process text ofPDF files for BibTeX entries. In this way, we will be able to extract additionalcontent related to a specific publication.

Another issue to investigate is the application of our methodology in differ-ent domains such as multimedia environment. In this field discovering similar-ities among items just on the ground of textual content could be not sufficient.

As discussed in Section 4, several results in literature proved that the taskof tag recommendation is not suitable to be faced with semantic approaches.In particular, linguistic resources such as WordNet might help to address the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

21

problem of Word Sense Disambiguation (WSD)[6], but the problem of taginterpretation is very different and even more complicated than WSD dueto the lack of context. We plan to perform a deeper investigation aiming atdeciding whether other semantic approaches than those based on linguisticanalysis might improve the overall performance of tag recommender systems.A semantic indexing strategy which can be exploited to associate words (tags)to Wikipedia concepts is Explicit Semantic Analysis [12]. The main advantageof ESA are:

– it allows the interpretation of keywords by means of world knowledge,rather than linguistic knowledge, therefore a deeper understanding of themeaning of tags is hopefully performed;

– it does not need to define the context of words to be indexed.

References

1. ****. ****. In ****, editor, ****, page ****, ****. ****.2. ****. ****. In ****, page ****, ****.3. ****. ****. In ****, editor, ****, volume **** of ****, page ****, ****, ****. ****.

****.4. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley,

1999.5. A. Baruzzo, A. Dattolo, N. Pudota, and C. Tasso. Recommending new tags using

domain-ontologies. In Proceedings of the Web Intelligence/IAT Workshops, pages 409–412, 2009.

6. P. Basile, M. Degemmis, A.L. Gentile, P. Lops, and G. Semeraro. UNIBA: JIGSAWalgorithm for Word Sense Disambiguation. In Proceedings of the 4th ACL 2007 Inter-national Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic,pages 398–401. Association for Computational Linguistics, June 23-24, 2007.

7. D. Billsus and M. J. Pazzani. Learning collaborative information filters. In Proceed-ing of the 15th International Conference on Machine Learning, pages 46–54. MorganKaufmann, San Francisco, CA, 1998.

8. C. H. Brooks and N. Montanez. Improved annotation of the blogosphere via autotag-ging and hierarchical clustering. In WWW ’06: Proceedings of the 15th internationalconference on World Wide Web, pages 625–632, New York, NY, USA, 2006. ACM Press.

9. C. Cattuto, C. Schmitz, A. Baldassarri, V. D. P. Servedio, V. Loreto, A. Hotho,M. Grahl, and G. Stumme. Network properties of folksonomies. AI Communications,20(4):245–262, 2007.

10. X. Chen and H. Shin. Tag recommendation by machine learning with textual and socialfeatures. Journal of Intelligent information Systems (JIIS), 2012. To appear.

11. Luis M. de Campos, Juan M. Fernndez-Luna, Juan F. Huete, and Miguel A. Rueda-Morales. Combining content-based and collaborative recommendations: A hybrid ap-proach based on bayesian networks. Int. J. Approx. Reasoning, 51(7):785–799, 2010.

12. Evgeniy Gabrilovich and Shaul Markovitch. Wikipedia-based semantic interpretationfor natural language processing. J. Artif. Intell. Res. (JAIR), 34:443–498, 2009.

13. J. Gemmell, T. Schimoler, M. Ramezani, and B. Mobasher. Adapting k-nearest neighborfor tag recommendation in folksonomies. In 7th Workshop on Intelligent Techniquesfor Web Personalization and Recommender Systems, held in conjunction with The 21stInternational Joint Conference on Artificial Intelligence (IJCAI-09), 2009.

14. S. Golder and B. A. Huberman. The structure of collaborative tagging systems. Journalof Information Science, 32(2):198–208, 2006.

15. M. P. Grineva, M. N. Grinev, and D. Lizorkin. Extracting key terms from noisy andmultitheme documents. In J. Quemada, G. Leon, Y. S. Maarek, and W. Nejdl, editors,Proceedings of the 18th International Conference on World Wide Web, WWW 2009,pages 661–670. ACM, 2009.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

22

16. P. Heymann, D. Ramage, and H. Garcia-Molina. Social tag prediction. In S. Myaeng,D. W. Oard, F. Sebastiani, T. Chua, and M. Leong, editors, SIGIR ’08: Proceedings ofthe 31st annual international ACM SIGIR conference on Research and development ininformation retrieval, pages 531–538, New York, NY, USA, 2008. ACM.

17. R. Jaschke, L. Balby Marinho, A. Hotho, L. Schmidt-Thieme, and G. Stumme. Tagrecommendations in folksonomies. In J. N. Kok, J. Koronacki, R. Lopez de Mantaras,S. Matwin, D. Mladenic, and A. Skowron, editors, Knowledge Discovery in Databases:PKDD 2007, 11th European Conference on Principles and Practice of Knowledge Dis-covery in Databases, volume 4702 of Lecture Notes in Computer Science, pages 506–514.Springer, 2007.

18. S. Ju and K. Hwang. A weighting scheme for tag recommendation in social bookmarkingsystems. In F. Eisterlehner, A. Hotho, and R. Jaschke, editors, ECML PKDD DiscoveryChallenge 2009 (DC09), volume 497, pages 109–118. CEUR Workshop Proceedings,2009.

19. S. On Kee Lee and A. Hon Wai Chun. Automatic tag recommendation for the web2.0 blogosphere using collaborative tagging and hybrid ANN semantic structures. InACOS’07: Proceedings of the 6th Conference on WSEAS International Conference onApplied Computer Science, pages 88–93. World Scientific and Engineering Academyand Society, 2007.

20. M. Lipczak. Tag recommendation for folksonomies oriented towards individual users.In Proceedings of ECML PKDD Discovery Challenge (DC08), pages 84–95, 2008.

21. M. Lipczak, Y. Hu, Y. Kollet, and E. Milios. Tag sources for recommendation incollaborative tagging systems. In F. Eisterlehner, A. Hotho, and R. Jaschke, editors,ECML PKDD Discovery Challenge 2009 (DC09), volume 497, pages 157–172. CEURWorkshop Proceedings, 2009.

22. L. Balby Marinho and L. Schmidt-Thieme. Collaborative tag recommendations. InC. Preisach, H. Burkhardt, L. Schmidt-Thieme, and R. Decker, editors, Data Analysis,Machine Learning and Applications - Proceedings of the 31st Annual Conference ofthe Gesellschaft fur Klassifikation e.V., Albert-Ludwigs-Universitat Freiburg, Studiesin Classification, Data Analysis, and Knowledge Organization, pages 533–540. Springer,2008.

23. A. Mathes. Folksonomies - cooperative classification and communication through sharedmetadata, December 2004.

24. G. Mishne. Autotag: a collaborative approach to automated tag assignment for weblogposts. In WWW ’06: Proceedings of the 15th international conference on World WideWeb, pages 953–954, New York, NY, USA, 2006. ACM Press.

25. J. Mrosek, S. Bussmann, H. Albers, K. Posdziech, B. Hengefeld, N. Opperman,S. Robert, and G. Spira. Content- and graph-based tag recommendation: Two vari-ations. In F. Eisterlehner, A. Hotho, and R. Jaschke, editors, ECML PKDD DiscoveryChallenge 2009 (DC09), volume 497, pages 189–199. CEUR Workshop Proceedings,2009.

26. H. Murfi and K. Obermayer. A two-level learning hierarchy of concept based keywordextraction for tag recommendations. In F. Eisterlehner, A. Hotho, and R. Jaschke,editors, ECML PKDD Discovery Challenge 2009 (DC09), volume 497, pages 201–214.CEUR Workshop Proceedings, 2009.

27. S. E. Robertson, S. Walker, M. H. Beaulieu, A. Gull, and M. Lau. Okapi at TREC. InText REtrieval Conference, pages 21–30, 1992.

28. G. Salton. Automatic Text Processing. Addison-Wesley, 1989.29. C. Schmitz, A. Hotho, R. Jaschke, and G. Stumme. Mining association rules in

folksonomies. In Data Science and Classification (Proc. IFCS 2006 Conference),Studies in Classification, Data Analysis, and Knowledge Organization, pages 261–270,Berlin/Heidelberg, July 2006. Springer. Ljubljana.

30. F. Sebastiani. Machine Learning in Automated Text Categorization. ACM ComputingSurveys, 34(1):1–47, 2002.

31. Yang Song, Lu Zhang, and C. Lee Giles. Automatic tag recommendation algorithmsfor social recommender systems. Transactions on the Web, 5(1):1–31, 2011.

32. S. Sood, S. Owsley, K. Hammond, and L. Birnbaum. TagAssist: Automatic Tag Sug-gestion for Blog Posts. In Proceedings of the International Conference on Weblogs andSocial Media (ICWSM 2007), 2007.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

23

33. P. Symeonidis. User recommendations based on tensor dimensionality reduction. InL. S. Iliadis, I. Maglogiannis, G. Tsoumakas, I. P. Vlahavas, and Max Bramer, edi-tors, Artificial Intelligence Applications and Innovations III, Proceedings of the 5thIFIP Conference on Artificial Intelligence Applications and Innovations (AIAI’2009),volume 296 of IFIP, pages 331–340. Springer, 2009.

34. M. Tatu, M. Srikanth, and T. D’Silva. RSDC’08: Tag Recommendations using Book-mark Content. In Proceedings of ECML PKDD Discovery Challenge (DC08), pages96–107, 2008.

35. J. Wang, L. Hong, and B. D. Davison. RSDC09: Tag Recommendation Using Keywordsand Association Rules. In F. Eisterlehner, A. Hotho, and R. Jaschke, editors, ECMLPKDD Discovery Challenge 2009 (DC09), volume 497, pages 261–274. CEURWorkshopProceedings, 2009.

36. I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, and C. G. Nevill-Manning. Kea:Practical automatic keyphrase extraction. In Proceedings of the Fourth ACM conferenceon Digital Libraries, pages 254–255. ACM, 1999.

37. H. Wu, M. Zubair, and K. Maly. Harvesting social knowledge from folksonomies. InHYPERTEXT ’06: Proceedings of the seventeenth conference on Hypertext and hyper-media, pages 111–114, New York, NY, USA, 2006. ACM Press.

38. Y. Zhang, N. Zhang, and J. Tang. A collaborative filtering tag recommendation systembased on graph. In F. Eisterlehner, A. Hotho, and R. Jaschke, editors, ECML PKDDDiscovery Challenge 2009 (DC09), volume 497, pages 297–306. CEUR Workshop Pro-ceedings, 2009.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

SUMMARY OF CHANGES

TITLE:

“Content-based and Collaborative Techniques for Tag Recommendation: an Empirical Evaluation”

AUTHORS:

Blind submission We would like to thank the editor and the reviewers for their constructive comments and suggestions, that we have carefully considered. In the following we report comments by reviewers and the corresponding reply by the authors. When possible, we provide the reference to the specific section of the paper that has been modified. The structure of the paper has been changed by moving the related work section at the end of the manuscript. This makes the paper more readable allowing a simpler comparison of our work with those cited in literature. We slightly changed the title of the manuscript by removing the first word “Integrating”. Reviewer #1 Comment 1 First, it seems clear from the authors' experiments that social/collaborative tagging does not help, and so including it in the preliminary sections is odd. In essence, the authors are extracting tags from article metadata, which is not unique. It's also very dependent on the particular data being used: META tags and keywords in bibtex entries are particularly designed to essentially act as tags already, so little extra work is needed. Authors Reply/Reaction We have designed the tag recommender system by taking into account very general principles which allow to make the system usable with different resources, besides BibTeX entries and bookmarks. For example, the system can use metadata assigned to pictures in a photo sharing system, or hashtags in twitter posts, etc. However, we tailored the system in order to participate to the ECML/PKDD Discovery Challenge 2009. In this particular scenario the adoption of strategies which exploit the previous tagging behavior of the user and the whole community can hopefully lead to better predictions than those computed by content-based techniques alone, which can give poor results because of limited availability and quality of content. In particular, the approach which exploits tags provided by the whole community can help to address the ‘new user’ problem typical of content-based recommendation algorithms. To sum up, we have defined both a social/collaborative and a content-based approach that can be effectively combined in different ways for different tasks and datasets.

*Response to Reviewer Comments

Comment 2 Despite this, the precision and accuracy are quite low (to be fair, so are the other systems). This might indicate that the problem is very challenging, or that a IR, syntactic approach that relies solely on extracting strings without analyzing their semantics is not going to be effective for this task. Authors Reply/Reaction We finally decided to avoid the adoption of semantic approaches, even though we previously evaluated the use of Word Sense Disambiguation (WSD), in order to deal with the problem of natural language ambiguity. WSD can help to assign the proper meaning to words according to the context in which they are used. In the specific tag recommendation task, WSD might be useful in order to deal with the problems of synonymy and polysemy of tags, or in general ambiguity of natural language. We identified several difficulties in the adoption of WSD for tag disambiguation: - WSD is generally computationally expensive and it is quite difficult to perform an on-the-fly

disambiguation for each new resource introduced by users - The definition of context for tags is not a trivial task. While for an ambiguous keyword

occurring in a document the context can be easily defined as a set of words surrounding it, in our specific scenario it is not simple to associate the ambiguous tag with a specific “piece of content” which can help for its disambiguation. We already experienced this problem in a cultural heritage scenario, and discussed it in some publications which we cannot cite here due to the blind submission required by the journal.

Another possible semantic alternative to WSD has been taken into account after getting suggestions/comments by the reviewers (of this paper submitted to JIIS) . More specifically, we have evaluated semantic approaches based on distributional models, such as Latent Semantic Analysis (LSA), nonnegative matrix factorization (NMF), etc., mainly based on statistics rather than linguistics. That kind of approaches are not suitable for the tag recommendation task. Indeed, a paper presented at ECML/PKDD discovery challenge titled: “A Two-Level Learning Hierarchy of Concept Based Keyword Extraction for Tag Recommendations” (Hendri Murfi and Klaus Obermayer) applied NMF to extract the semantic concepts from document collections and has obtained not accurate results. The authors also motivated the adoption of NMF instead of other semantic techniques such as LSA. We discussed those aspects in the ‘Related Work’ section of the paper. Comment 3 The approach is very much tuned to these two datasets, which limits its applicability. If one wants to do an IR=based content extraction task, I would've preferred to see the authors work with the text, which is guaranteed to be present, as opposed to metadata. While I found the paper interesting and the question very much one worth approaching, these issues limit both the impact of the results and also their generality. Authors Reply/Reaction As mentioned in the reply to Comment 1, we tried to design a very general system by implementing different approaches (content, collaborative, personal), which can be effectively combined in order to adapt to the specific scenario. As regards the exploitation of the content associated to items rather metadata for tag recommendation, there are not specific problems in implementing such an approach. In our work we decided to analyze metadata instead of text after a preliminary analysis performed to identify the best source of information for extracting tags.

Reviewer #2 Comment 1 Page 3. "a novel content-based approach". The author should briefly mention what is the novelty. Is it completely a new idea. I don't think so. At least two lines should be here to show the novelty. Authors Reply/Reaction The novelty of this submission with respect to our previous publications relies in the adoption of content-based techniques for the BibTeX entries and in a different way of combining recommendations produced by the different proposed strategies. In our first paper about tag recommendation we only adopted collaborative and personal strategies. Subsequently, we exploited content associated to items for recommending tags for bookmark resources. Given the good performance of content-based approach in terms of accuracy (also compared with other works participating to the discovery challenge) we decided to investigate the adoption of this model to BiBteX entries as well. Furthermore, we also investigated: 1. the impact of different strategies to weigh the different content sources 2. the accuracy of different ways of combining tags extracted by the different proposed strategies

(content, collaborative, personal)

Comment 2 Page 3. Ref. 1. I don't find the reference details. So it is not possible to find what is considered in this paper. Authors Reply/Reaction References 1,2,3 have been anonymized since it is a blind submission. In the final version of the paper the complete references will be provided. Comment 3 Page 3. Author should mention briefly what does merging of two approaches give new result. Is it just merging or gives better result. So readers will be motivated to read the paper. Authors Reply/Reaction In the last part of the Abstract a specific sentence already provides an overview of the outcomes of the experiments. There is also a reference in the Introduction. Comment 4 Why the system name is given STar? Authors Reply/Reaction It is the first name we used for our tag recommender system. We have continued to use that name. If reviewers think it is misleading we can modify it. Comment 5 Page 2. "elitist systems". What is this? Not clear. Authors Reply/Reaction The adjective ‘elitist’ refers to the fact that in classical web sites the information could be provided only by web site administrators. Users were confined to a passive role. Nowadays with collaborative platforms they can ‘produce’ information, as well. However, the adjective ‘elitist’ is now replaced with ‘classical’. Comment 6 Page 3. Line 2. "polysemy, synonymy?." The terms should be italic. Authors Reply/Reaction The terms are in italic, now. Fixed.

Comment 7 Page 3. Line 4. "think at a resource" . Should be "think of a?" Authors Reply/Reaction Fixed. Comment 8 Related work section. The author should clearly mention the difference with the existing approaches. The authors claim a "novel content-based approach". What is the difference with the existing approaches. What is the advantage of combining the "existing collaborative" and "novel content-based" compared to "existing collaborative" and "existing content-based" approach. Authors Reply/Reaction We moved the related work section at the end of the paper, in order to provide a comparison of our approach with respect to those presented in the related work section. We also discussed the opportunity to use semantic strategies to tag recommendations, and the main motivations for not implementing them. Comment 9 Page 7. "Tag ASsignment function". Should be "Tag Assignment function". Looks better to me. Authors Reply/Reaction Fixed. Comment 10 Page 7.What is relevance model. Need Ref. or define. Authors Reply/Reaction The term ‘relevance model’ has been replaced with ‘criterion’. Comment 11 Page 7. "ranked set of candidate tags". Need Ref. or define. Authors Reply/Reaction The sentence has been re-formulated as follows: “The approach builds a preliminary set of candidate tags. Each tag is then assigned with a score and the tags are finally ranked. The top-k tags are returned to the user.” Comment 12 Page 7. ``Suitable tags``. Need to be defined. Authors Reply/Reaction The adjective ‘suitable’ has been deleted. The sentence has been clarified as follows: “a set of tags able to effectively describe the information conveyed by the resource”. Comment 13 Page 7. Paragraph ``For the content-based ??for similar resources`` is not clear. Pls rewrite. The assumptions are not clear. Authors Reply/Reaction The sentence has been re-formulated as follows: “For the content-based component of the algorithm, the recommendation is based on the analysis of information coming from the content of the resources, while the collaborative component is based on the analysis of similar resources that users have already tagged.”

Comment 14 Page 7. ``merged with different ways``. What are the ways? Pls mention clearly where they are proposed or mentioned. Authors Reply/Reaction The sentence has been re-formulated as follows: “…These sets are finally merged in order to obtain a final set of recommendations. Specifically, the relevance of each tag is obtained by a linear combination of the partial relevance scores returned by the collaborative and content-based part of the recommendation algorithm.” Comment 15 Page 7, section 7. ``Similar resources``. When is two resources similar. Need clarification. Authors Reply/Reaction Two resources can be considered as ‘similar’ if the convey a similar informative content. For example two news referring to the same event or two scientific papers referring to the same research area. The sentence has been extended by introducing this clarification: (“The insight behind the collaborative approach is that two resources that convey a similar information (e.g. two papers referring to the same topic) can be annotated with the same (or at least, a similar) set of tags.”) Comment 16 Page 8. ``It explorers ?``. What does it refer? Authors Reply/Reaction ‘It’ refers to the approach. Now in the sentence the subject has been made explicit (“The approach exploits”). Comment 17 Page 8. ``Similarity between resources is computed? ``. What about image, video, ect. Authors Reply/Reaction The recommendation model implemented in STaR follows a content-based (CB) approach. A typical assumption of CB filtering is that each resource to be recommended needs to be described by means of (textual) features. The implemented approach is able to provide suggestions for image and video resources as well, provided that some ‘content’ describing those resources is available (e.g. a plot or summary of the video, a description of the image, and so on). The sentence has been extended by describing how image and video content are managed in our framework (“These techniques can be applied to textual documents (such as web sites, scientific papers and so on) as well as multimedia content, such as videos and images, provided that some textual features (e.g. a summary of the video or a caption of the image) is available”) Comment 18 Page 8. This list is finally ranked and stored in Collaborative Tags list. How the ranking is done and what is the storage structure. Authors Reply/Reaction The list is ranked according to the relevance function proposed in Equation 9. The semantics of the ranking function is explained in this sentence (“a score is computed by weighing the similarity score returned by the query, with the normalized occurrence of the tag “). The storage structure refers to technical details that are not described in the paper. Thus, the last part of the sentence has been re-formulated as follows: “Finally, the Collaborative Candidate Tags list will contain the top-n tags, previously ordered according to their descending relevance score.”

Comment 19 Page 10. Equation 3. Does it includes index of user Authors Reply/Reaction In SocialIndex we want to merge the information about all the annotations provided by all the users in the community. It is not necessary to include the index of the user. Comment 20 Page 10. That user. Which user Authors Reply/Reaction ‘That user’ refers to the target user, i.e. the user who is going to receive recommendations. ‘That user’ has been replaced with ‘the target user’. Comment 21 Page 10, Section 3.2.2. Lucene ? Need ref. Authors Reply/Reaction Apache Lucene is a search engine library written in Java. In STaR, it is used to index the information about tags and documents and to retrieve the resources similar to the one to be annotated. A footnote with the URL of the Apace Lucene project is provided Comment 22 Page 10. Equation 4, 5. What are the purposes of these. Need clarification. Authors Reply/Reaction In the collaborative part of the framework the algorithm first looks for resources similar to the one that is going to be annotated. This is done to understand how similar resources were tagged in the past by similar users (or by the user itself). This is formalized in Equation 4 and 5, were we model the set of the resources similar to those tagged in the past by the user and by the community, respectively. Comment 23 Page 11. Equation 6. Here personalRes has parameters user and query but in equation 4, user and resource. Same for SocialRes. Authors Reply/Reaction The textual description of the resource to be annotated is exploited as query to get the most similar resources. Consequently, both terms can be used in a interchangeable way. Comment 24 Equation 9, 10. Need clarification. Authors Reply/Reaction Once the set of candidate tags is built, it needs to be ranked. Through Equation 9 and 10 each tag is assigned with a relevance score. The score is affected by the occurrences of the tag and the similarity of the resource the tag comes from. The insight is that tags that have been used many times to annotate very similar resource are the most relevant ones. The semantics of the formula is strictly related with the nature of the task, where a test set of already tagged resources was adopted. Consequently, it makes sense to exploit a strategy that takes into account the previous tagging activity of both the user and the community.

Comment 25 How is collaborative and content-based approach merged. Authors Reply/Reaction The collaborative and content-based approaches have been merged by adopting a technique called ‘Fusion’. Given two separate sets of candidate tags, they are merged into a single set: the final weight of each tag is obtained as a (linear) combination of the original weight of the tag with the weight of the source it comes from. Possible ways to merge the recommendation approaches are presented in Section 2.4 Comment 26 References are old with respect to the advancement of the topics of the paper. Authors should consider recent papers in the year of 2010, 2011. Authors Reply/Reaction References have been extended by introducing the following related work:

- Xian Chen and Hyoseop Shin: Tag recommendation by machine learning with textual and social features. Journal of Intelligent Information Systems (JIIS), 2012.

Comment 27 Authors should cite the following papers and should mention the difference with the proposed approach. a. Automatic Tag Recommendation Algorithms for Social Recommender Systems b. Combining content-based and collaborative recommendations: A hybrid approach based on Bayesian networks Authors Reply/Reaction Both papers have been added to the references and a description of the approach has been included in the ‘Related Work’ section. Comment 28 The paper presented a combined approach for tag recommender considering collaborative and content-based approach. However, most of the contents, equations, figures in Section 3.1 is presented is mentioned in the following paper. This considers major part of the paper. The main contribution is mentioned is section 3.3 which seems not sufficient for a journal like JIIS. **"A Tag Recommender System Exploiting User and Community Behavior" **"STaR: a Social Tag Recommender System" Authors also published another paper titled "Combining Collaborative and Content-Based Techniques for Tag Recommendation" The title of the current paper resembles the above mentioned paper. However, the authors did cite or mention the paper. Therefore, this is difficult to find the real contribution of the current paper. Authors Reply/Reaction The aforementioned papers are cited in the paper (references [1], [2], [3]) in order to motivate some choices and to point out differences with respect to our previous work. Because of blind submission required by the journal, we did not provide the details of the publications.