International Journal of Technology Assessment in Health Care 305 (2014) 478ndash487ccopy Cambridge University Press 2015

doi101017S0266462314000646 Theme SubmissionsDEVELOPING THE HTA CORE MODEL FORTHE ONLINE ENVIRONMENTKristian LampeFinnish Office for Health Techology Assessment (FINOHTA) National Institute for Health andWelfare (THL)

Iris PasternackFinnish Office for Health Techology Assessment (FINOHTA) National Institute for Health andWelfare (THL) and Summaryx Ltd

Oskari SaarekasFinnish Office for Health Techology Assessment (FINOHTA) National Institute for Health andWelfare (THL)

Leena RaustiaFinnish Office for Health Techology Assessment (FINOHTA) National Institute for Health andWelfare (THL)

Irina CleemputBelgian Health Care Knowledge Centre (KCE) and Hasselt University

Mirella CorioInnovazione sperimentazione e sviluppo Agenzia Nazionale per i Servizi Sanitari Regionali(Agenas)

Gottfried EndelTeam HTA Department for Evidence Based Economic Health Care Main association of AustrianSocial Insurance Institutions (HVB)

Katrine FroslashnsdalNorwegian Knowledge Centre for Health Services (NOKC)

Inaki ImazAgencia de Evaluacion de Tecnologıas Sanitarias (AETS) Instituto de Salud ldquoCarlos IIIrdquo

Sarah KleijnenDutch National Health Care Institute (ZIN)

Finn KristensenEUnetHTA Secretariat Danish Health and Medicines Authority (DHMA) and University of SouthernDenmark

Alric RutherDepartment of Health Care Quality Institute for Quality and Efficiency in Health Care (IQWiG)

Sophie WerkoSwedish Council on Health Technology Assessment (SBU) and Department of Learning InformaticsManagement and Ethics (LIME) Karolinska Institutet

Marina CerboInnovazione sperimentazione e sviluppo Agenzia nazionale per i servizi sanitari regionali (Agenas)

for the EUnetHTA HTA Core Model Online Working Group

Background A framework for collaborative production and sharing of HTA information the HTA Core Model was originally developed within EUnetHTA in 2006ndash08 In this paperwe describe the further development of the Model to allow implementation and utilization of the Model online The aim was to capture a generic HTA process that would alloweffective use of the HTA Core Model and resulting HTA information while at the same time not interfering with HTA agenciesrsquo internal processesMethods The work was coordinated by a development team in Finland supported by an international expert group Two pilot testing rounds were organized among EUnetHTAagencies and two extensive core HTA projects tested the tool in a real setting The final work was also formally validated by a group of HTA agenciesResults The HTA Core Model Onlinemdashavailable at httpwwwcorehtainfomdashis a web site hosting a) a tool to allow electronic utilization of the HTA Core Model and b) adatabase of produced HTA information While access to the HTA information is free to all the production features are currently available to EUnetHTA member agencies only A policywas crafted to steer the use of the Model and produced informationConclusions We have successfully enabled electronic use of the HTA Core Model and agreed on a policy for its utilization The system is already being used in subsequent HTAprojects within EUnetHTA Joint Action 2 Identified shortcomings and further needs will be addressed in subsequent development

Keywords health technology assessment models medical informatics applications classification research design

A new framework for collaborative production and sharing ofHTA information the HTA Core Model

Rcopy hereafter referred to

The HTA Core Model Online Working Group consists of (in addition to the authors) L Akiola(THLFINOHTA Finland) C De Laet (KCE Belgium) C Filippi (Regione del Veneto Italy) TGasparetto (Regione del Veneto Italy) W Goettsch (CVZ the Netherlands) R Grilli (RegioneEmilia Romagna Italy) G Jamtvedt (NOKC Norway) T Jefferson (AGENAS Italy) M Perrini(AGENAS Italy) M Rosen (SBU Sweden) I Saeligterdal (NOKC Norway) N Vicari (AGENASItaly) The views expressed herein are the responsibility of the authors and do not necessarilyreflect the views of EUnetHTA EUnetHTA assumes no liability for the contents or use thereofEUnetHTA Joint Action was supported by a grant from the European Commission Agreementnumber 2009 23 02 The sole responsibility of this article lies with the author(s) and neither the

as the ldquoModelrdquo was developed within the EUnetHTA projectin 2006ndash08 (1) The aim was to reduce redundant duplicationof assessments in different countries through enabling the pro-duction of robust evidence base of high quality that could beeasily used and tailored for local needs and updated when-ever needed This goal was sought primarily by standardiz-ing the information contents of HTA and summarizing keyresearch methodologies A comprehensive set of HTA infor-mation was defined by analyzing relevant literature and using

Commission nor EUnetHTA is responsible for any use that may be made of the informationcontained therein

478

HTA Core Model for the Online Environment

practical experience from HTA projects This large set was di-vided into standardized units which were designated as assess-ment elements indicating their nature as fundamental parts ofan HTA The elements were divided across the following ninedomains of HTA identified earlier in the EUR-ASSESS project(2)

bull Health problem and current use of technology

bull Description and technical characteristics of technology

bull Safety

bull Clinical effectiveness

bull Costs and economic evaluation

bull Ethical analysis

bull Organizational aspects

bull Social aspects

bull Legal aspects

The original mostly paper-based development and test-ing of the Model have been reported elsewhere (13ndash8)In short the Model was originally constructed by sev-eral international expert groups each focusing either on theoverall design or one of the nine domains Several work-shops piloting and formal validation were used in the pro-cess From the beginning however future employment ofthe Model was meant to take place through computerizednetworks because using these for the production and publica-tion of information would be needed for effective internationalcollaboration

We are not aware of similar frameworks in the field ofhealth research Most relevant systems take a quite different ap-proach focusing primarily on research reporting or appraisalof information (9ndash13) They do not define the contents ofthe research in similar detail as the HTA Core Model doesThe MAST (Model for Assessment of Telemedicine) is toour knowledge the only more closely related work applyingsome of the basic ideas of the HTA Core Model in the field oftelemedicine (14)

The HTA Core Model Online available at wwwcorehtainfo referred to also as ldquothe online toolrdquo provides aweb-based interface for using the Model for HTA informationproduction According to the Policy for the HTA Core Modeland core HTA information EUnetHTA Partners and Associateorganizations listed at wwweunethtaeu can use the site forproducing and publishing HTA information (15) Informationproduced and published within the HTA Core Model Onlinewas designated in the policy as core HTA information TheModel can assist also other parties (beyond EUnetHTA mem-bers) in designing or reporting health-related research Anynoncommercial party can use the online tool for producinginformation but organizations that are not EUnetHTA mem-bers must publish their products elsewhere for example onthe userrsquos own web site Commercial organizations can accessthe core HTA informationmdashlike anyone elsemdashfree of charge

but they are not allowed to access the information productionfunctions

In this study we describe the further development withinEUnetHTA Joint Action (JA) in 2010ndash12 of moving the Modelfrom paper to the online environment The intention was tocapture a generic project flow that would allow effective us-age of the Model online and sharing of structured HTA in-formation amongst agencies while at the same time allow-ing for local research-related internal processes within HTAagencies

METHODSThe HTA Core Model Online was developed in EUnetHTAJoint Action Work Package 4 Strand A (WP4A) through aniterative process on the basis of the paper-based version of theModel

The work was led by a team within the Finnish NationalInstitute for Health and Welfare supported by an internationalexpert group consisting of twenty-three individuals represent-ing HTA agencies in nine countries Challenges options andsolutions were discussed and developed through email com-munication e-meetings and a total of ten face-to-face meet-ings Decisions were mostly made by consensus or if neededby majority The broad experience of these agencies in HTAallowed us to better understand the needs of HTA professionalsand consequently balance between the theoretical ideal and thepractically feasible

Within the technical development we aimed at ensuringgood usability sustainable database structures features forgroup work and work flow management effective search func-tions as well as standardized interfaces reusability and pub-lishing The online service was developed using the MicrosoftASPNET environment and C programming language Thedatabase is managed by Microsoft SQL Server Jointly agreed-on guidelines for EUnetHTA toolsrsquo design were applied when-ever feasible (16)

When implementing the Model in the online environmentsome concepts behind the original ideas required further clar-ification reconsideration and consolidation so that the re-sulting information system would be based on a clear con-ceptual basis This formed a large part of the developmentwork

The practical development of the HTA Core Model Onlinewas divided into two phases first focusing on the basic func-tionalities needed to support core HTA production and thenconsidering advanced functionalities such as publishing infor-mation search and retrieval and adaptation of information (17)Development was supported by continuously collecting usersrsquofeedback and carrying out two specific pilot testing phases dur-ing autumn 2010 and spring 2011 Most of the feedback camefrom WP4 member agencies participating in the pilots Im-provements to the system were made after both pilot tests The

479 INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014

Lampe et al

Figure 1 The three components of the HTA Core Model

piloting focused on the toolrsquos ability to support users in devel-oping a project protocol for assessments Subsequently WP4Strand B (WP4B) piloted the complete production processby producing two core HTAs between 2011 and 2012 usingthe online tool This experience of WP4B is reported else-where in this issue (18) and was used for further developing thetool

WP4 also developed a policy (15) to guide use of the HTACore Model and information produced by using the Model Thiswas done by surveying the views of EUnetHTA member agen-cies Preferences of the majority were typically chosen to be in-cluded in the policy but feedback from stakeholders and publicconsultation was taken into account as well The policy craftingprocess also required us to further clarify some concepts Thepolicy was used when preparing a more concise document theTerms of Use for the Model (available at wwwcorehtainfo)

The EUnetHTArsquos Stakeholder Forum contains a broadrange of stakeholders who participate in the activities ofthe network (19) The Forum nominated representatives tothe WP4 Stakeholder Advisory Group (SAG) The SAG wasconsulted during various development processes The HTACore Model Online was also subject to public consulta-tion and formal validation by HTA agencies during 2012(17)

RESULTS

Basic ConceptsThe Model and its Use Three separate but strongly inter-connectedcomponents of the Model were identified First the assessmentelements delineate the contents of an HTA and hence consti-tute an ontology of HTA that is a formal representation of

knowledge within HTA The second component of the Modelconsists of all the methodological guidance included in variousparts of the content Finally the Model also contains a com-mon reporting structure that aims at a standardized flexible andinformative presentation of information

The three components all aim at producing and reportinga set of structured HTA information The ontology defines thepotentially relevant research questions for an HTA the method-ological guidance assists in finding answers to the questions andthe reporting structure defines how to present the answers Userscan employ either some or all of the three components in theirHTA work as long as the Terms of Use are followed The termsallow a versatile use of the Model but certain requirements areset both for noncommercial and commercial use For examplewhen using the noncommercial license the produced informa-tion must be made publicly available The commercial licensein turn sets for instance some conditions for advertising nearbymaterials produced using the Model (Figure 1)

The online environment was taken into account for the firsttime while preparing an application of the Model for screeningtechnologies (20) The interest for such an application had beenidentified in discussions when setting up the JA as a project Thetexts were shortened and structured under consistent subtitlessimilar for all domains relations between assessment elementswere identified and coded to enable cross-referencing in theonline tool Furthermore methodological guidance applicableto several domains was placed in appendices and marked withhyperlinks in the domain chapters (Table 1)

HTA Ontology The HTA ontology defines a generic set of ques-tions about health technologies It is a formal representation ofknowledge within the field of HTA providing a standardizedstructure for any set of HTA information

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 480

HTA Core Model for the Online Environment

Table 1 Terms Used in the HTA Core Model

HTA Core ModelThe HTA Core Model is a methodological framework for production and sharing of HTA informationApplication of the HTA Core ModelA ldquoversionrdquo of the Model tailored for a specific type of technologyCore HTA informationAny information on health technologies that has been produced using the HTA Core Model and made available through the HTA Core Model OnlineAssessment elementA piece of information that describes the technology or the consequences or implications of its use or any other implication that is relevant for the assessmentElement cardAn assessment element described in greater detailResult cardContains answer(s) to a question or a set of questions defined by one assessment elementCollectionA set of core HTA information consisting of (a) a set of result cards in which each research question is answered in a concise manner (b) general content that combines the cards

into a coherent information package and (c) optional appendices that enable inclusion of additional information to eg the result cards without crowding the cardsrsquo content

The original list of assessment elements was created whendeveloping the HTA Core Model for medical and surgical in-terventions Another list was subsequently created during thedevelopment of the Model for diagnostic technologies Furthermodifications to the elements were made after completion of theModel application for screening technologies and rapid relativeeffectiveness assessment of pharmaceuticals It was decided thatthese application-specific lists would draw from a common poolof assessment elements In other words the HTA Core Modelcontains an extensive number of assessment elements flexiblefor future developments Each model application uses only partof this whole pool The most extensive application (for diag-nostic technologies) makes use of 157 elements while the mostconcise application (for medical and surgical interventions) usesonly 135 elements

The generic questions referred to as ldquoissuesrdquo in the Modelare organized within the nine domains using an intermediatelayer of ldquotopicsrdquo Hence each element is defined by a domaintopic and issue

The following are examples of assessment elements

bull Safety (domain) Occupational Safety (topic) What kind of occupationalharms can occur when using the technology (issue)

bull Clinical Effectiveness Function How does use of the technology affectactivities of daily living

bull Ethical analysis Autonomy Is the technology used for patientspeoplethat are especially vulnerable

The assessment elements each define a ldquogeneric questionrdquothat could be answered in an HTA Each element is definedin further detail in an element card that is application-specificFor example a question that considers ldquomortality related tousing the technologyrdquo may be defined differently depending

on whether it is applied to a therapeutic intervention or to ascreening test

We maintained the earlier division of elements into coreelements and non-core elements based on their importance andtransferability (1) The core elements are those whose impor-tance and transferability are defined as higher than those of non-core elements Consequently they are likely to be more usefulin the international context The importance and transferabilityof each element can be different across model applications

Methodological Guidance Different levels of methodological guidanceare a major constituent of the Model International workinggroups consisting primarily of HTA experts have produced theguidance using relevant literature and their own expertise Theguidance is provided on the level of individual elements do-mains and on whole HTA projects Assessment element-levelguidance provides practical assistance to researchers for tack-ling a specific research question Such guidance is available inthe element cards Domain-level guidance is provided for eachof the nine domains consisting of an overview of recommendedscientific methodologies within different domains The project-level guidance is currently still scarce but for example specificethical requirements have been set for projects using the Model

We maintained the original (156) principle that allguidancemdashacross the above mentioned three levelsmdashcan beeither in the form of hintstips somewhat stronger recommen-dations or strict requirements or standards It has been the taskof the expert groups and WP4 to decide on the level of guidance

Common Reporting Structure The Model provides a standardized struc-ture for presenting and reporting the HTA information producedusing the Model The information is organized into collections

481 INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014

Lampe et al

Table 2 From Generic Question to Result Card

of result cards The fundamental difference in the reportingstructure compared with the reporting of traditional HTA reportsis that the results are presented as standardized question-answerpairs organized under the nine domains The information islocated in a standard easily identifiable and searchable locationin the reporting instead of being embedded within the narrativetext flow of traditional reports

According to the experience gained in the two pilot coreHTA projects (18) most assessment elements translate into asingle research question It is however possible to divide thegeneric question in an assessment element into more than onequestion The answer(s) are presented in result cards each con-taining the answer(s) to question(s) defined by one assessmentelement The result cards are then organized into collectionseach consisting of a set of result cards and collection-level con-tent for example an introduction and a summary Each resultcard should contain the answer in a relatively concise formatAlthough no definite character limit was set at this point anoverall recommendation of 1ndash2 pages per answer was given topilot projects This length was considered appropriate when de-signing the Model within the EUnetHTA Project 2006ndash08 as itwould allow the text fit on a single page and hence resemble aldquocardrdquo More extensive materials such as large evidence tablesor detailed results can be added as appendices to the collection

The earlier paper-based reporting has evolved to a struc-tured set of information that can be used shared and updatedin computerized environments The standardized informationstructure allows flexible searching and viewing of informationOnly relevant parts of the information can be used for each pur-pose From the technical viewpoint the predetermined structure

also enables interoperability with other information systemssuch as the EUnetHTA POP Database or any systems used forexample for guideline development or decision-support

A practical example of how a generic question in the ontol-ogy is translated into a project-specific research question andanswered within a core HTA project is available in Table 2

The Core HTA Structure The interrelation between core HTA infor-mation and local HTA information was originally suggested inLampe et al (1) and designated as the core HTA structure Itconsisted of a pool of structured HTA information a core HTAas an intermediary information product and local HTAs as endproducts The current new structure illustrated in Figure 2 al-lows different types of core HTA information collections Thefeatures of EUnetHTA Collections such as ldquocore HTAsrdquo aredefined by EUnetHTA while Other Collections may be formedbased on usersrsquo own preferences It was again agreed onmdashnowin the formal policymdashthat core HTA information is publishedin English language The process between the collections andunderlying information pool (containing result cards) is auto-mated and hence the user interacts mostly with collections Theutilization process from core HTA information to local prod-ucts is voluntary and the format language and content of endproducts are defined mostly by local users Local reports canalso form a collection in the database

The HTA Core Model OnlineA service for using the HTA Core Model in the online environ-ment was delivered at wwwcorehtainfo It allows production

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 482

Lampe et al

Table 2 From Generic Question to Result Card

of result cards The fundamental difference in the reportingstructure compared with the reporting of traditional HTA reportsis that the results are presented as standardized question-answerpairs organized under the nine domains The information islocated in a standard easily identifiable and searchable locationin the reporting instead of being embedded within the narrativetext flow of traditional reports

According to the experience gained in the two pilot coreHTA projects (18) most assessment elements translate into asingle research question It is however possible to divide thegeneric question in an assessment element into more than onequestion The answer(s) are presented in result cards each con-taining the answer(s) to question(s) defined by one assessmentelement The result cards are then organized into collectionseach consisting of a set of result cards and collection-level con-tent for example an introduction and a summary Each resultcard should contain the answer in a relatively concise formatAlthough no definite character limit was set at this point anoverall recommendation of 1ndash2 pages per answer was given topilot projects This length was considered appropriate when de-signing the Model within the EUnetHTA Project 2006ndash08 as itwould allow the text fit on a single page and hence resemble aldquocardrdquo More extensive materials such as large evidence tablesor detailed results can be added as appendices to the collection

The earlier paper-based reporting has evolved to a struc-tured set of information that can be used shared and updatedin computerized environments The standardized informationstructure allows flexible searching and viewing of informationOnly relevant parts of the information can be used for each pur-pose From the technical viewpoint the predetermined structure

also enables interoperability with other information systemssuch as the EUnetHTA POP Database or any systems used forexample for guideline development or decision-support

A practical example of how a generic question in the ontol-ogy is translated into a project-specific research question andanswered within a core HTA project is available in Table 2

The Core HTA Structure The interrelation between core HTA infor-mation and local HTA information was originally suggested inLampe et al (1) and designated as the core HTA structure Itconsisted of a pool of structured HTA information a core HTAas an intermediary information product and local HTAs as endproducts The current new structure illustrated in Figure 2 al-lows different types of core HTA information collections Thefeatures of EUnetHTA Collections such as ldquocore HTAsrdquo aredefined by EUnetHTA while Other Collections may be formedbased on usersrsquo own preferences It was again agreed onmdashnowin the formal policymdashthat core HTA information is publishedin English language The process between the collections andunderlying information pool (containing result cards) is auto-mated and hence the user interacts mostly with collections Theutilization process from core HTA information to local prod-ucts is voluntary and the format language and content of endproducts are defined mostly by local users Local reports canalso form a collection in the database

The HTA Core Model OnlineA service for using the HTA Core Model in the online environ-ment was delivered at wwwcorehtainfo It allows production

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 482

HTA Core Model for the Online Environment

Figure 2 The core HTA structure defined in EUnetHTA Joint Action (L HTA = Local HTA R HTA = Rapid HTA)

and publication of core HTA information according to conceptsoutlined above Some key functions are discussed below

Phases of Information Production The online tool organises the workflowof information production into five phases Proceeding from onephase to the next requires locking of the phase after which nofurther changes are allowed in the preceding phases Unlockinga phase is possible but as any changes to the earlier phasesmay have an impact on the subsequent phases it should be doneconsciously and after consideration

1 Project definition and scopeIn the first phase the project is created as a new entity within

the database including its name scope and metadataThe project scope is organized in a manner that (a) allows

a robust scope for each project from the research perspectiveas well as (b) enables automated functionalities within the sub-sequent phases The authors are requested to name the technol-ogy the indications and comparators they want to assess Thelsquoindicationrsquo includes information of the disease in question in-cluding its level of severity the patients with possible age andgender limits and risk level and the intended use of the tech-nology for example for screening or diagnosing the disease orfor first or second line treatment The most important outcomesassessed in the project are outlined in the scoping phase butthey are further expanded later during the project within eachdomain separately

2 Protocol designDuring the second phase a set of generic questions defined

by the ontology and divided within the nine domains of theModel are presented to the user who must consider their rele-

vance in the context of the technology under assessment If aquestion is relevant it should be translated into a topic-specificresearch question that should be answered in the subsequentphases The system assists the researcher in this phase by sug-gesting questions based on the structured scope

Each domain uses by default the project scope as its researchframe The user has the option of applying a wider frame if thatsuits the analysis better For example the project scope maydefine the technology as ldquomulti-slice Computed Tomography(CT)rdquo but in the safety domain it may be more useful to considerthe safety of ldquoCTrdquo in general The project scope should alwaysbe embedded within the frame keeping the overall analysisfocused on the same technology while at the same allowingflexibility in various domainsrsquo analysis

The project protocol defines the research questions thatshould be answered in the project and should be used to formu-late a further detailed research protocol including plans aboutmethodology used The research protocol is not yet included inthe tool allowing variation in local practices

3 Research (that is finding the answers)In this phase the answers to the questions included in the

protocol are sought and formulated In addition to the method-ological guidance provided by the Model the researchers canuse any tools and practices they normally apply in their work tofind answers The nature of the questions and the answers variesto some extent across domains In some cases the answer maybe a compilation of scientific evidence while in other cases theanswer may consist of data from a single database or reportingof moral deliberations Independent of these differences theresults of this phase are always reported in a standardized

483 INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014

Lampe et al

format using an MS Word document template available foreach domain The template is based on the common reportingstructure and contains all topic-specific research questions ofa domain selected in the previous phase and all other requireddomain-specific fields

The only mandatory information is included in the ldquoRe-sultsrdquo field which should contain the answer in a concise for-mat Optionally researchers can add a description of question-specific methodology (if it deviates from domain methodology)and a comment text The methods should be disclosed as de-tailed as in any scientific work The comment can address forexample research methods used reliability of results or needsfor further research

On the domain-level researchers should include text chap-ters providing an introduction and a description of methodologyused in the analysis as well as a general discussion related to thefindings for the specific domain The discussion section allowsfor bringing together and interpreting the findings that are pre-sented in the result cards of a domain as well as for a discussionon methodological challenges indications for further researchand any other domain-level considerations

4 ResultsIn this phase all results are uploaded to the system and

consequently another entity the collection is created in thedatabase The HTA Core Model Online presents the user witha standard collection structure that contains all relevant partsThe researchers should upload all the domain-specific results(including the result cards) and add the rest of the collection-level information such as a summary

The summary provides an overview of the facts found inthe assessment but it must not contain any recommendationsfor or against the use of the assessed technology Control of thisprinciple takes currently place at the JA work packages produc-ing EUnetHTA collections but more detailed processes will bedeveloped in the future Such recommendations can be added tothe local reports based on the collection Collection-level infor-mation also contains an introduction and a methodology chapter

Currently the upload process from the MS Word templatesto the HTA Core Model Online is a manual process

5 View and SubmitThe final phase allows viewing and submitting the collec-

tion for publication Viewing is available at any point of timebut (before publication) restricted to those who participate inthe project If the project is owned by one of the EUnetHTAPartners or Associates it can be published within the pool ofcore HTA information If the project is owned by others publi-cation within the system is currently not allowed In such casesthe researchers can savedownload the collection and publishit elsewhere Allowing a more free publication platform wouldhave required additional editorial control mechanisms as well asadministrative effort which were not prioritized at this phase

Presentation of Collections Each collection contains a cover page thatdisplays key information the collectionrsquos name producers pub-lishing agency and various data on the Model version used thescope and summary All other contents of the collection areavailable through selecting the desired parts (or all) of the con-tent to be viewed This allows flexible viewing that easily adaptsto various usersrsquo needs

The key contents of each result card are included in the col-lection view as flowing text one after another It is also possibleto open each card in a separate window to access all detailedcontents of each card

Some key features of the Adaptation Toolkit developedby WP5 of the EUnetHTA project 2006ndash08 were included inthe HTA Core Model Online to assist local use of core HTAinformation (21)

DISCUSSIONWe succeeded in clarifying the conceptual basis of the HTA CoreModel and a pragmatic information production process neededfor the online implementation of the Model Some themes thatrequired more attention during the development process or thatperhaps necessitate further development in the future are dis-cussed next

Compared with formal ontology development and someexisting ontologies (2223) the HTA ontology is still in a crudeformat and lacks certain features (for example identificationof some important relations between the content) This turnsinto some practical disadvantages for example in the form ofoverlapping content within different assessment elements andeven across domains Further development work is needed toaddress these issues by identifying relevant relations and byremoving redundant overlaps

The aim of the HTA Core Model has been fromthe beginning to provide overviews of worthwhile researchmethods within various HTA domains as well as practi-cal advice on different assessment elements A considerableamount of guidance is embedded in the Model itself but anequally important aspect is the possibility to link to exist-ing guidance elsewheremdashwhether produced by the EUnetHTAor other parties Practical examples include the SureINFOservice (httpvortalhtaiorgq=sure-info) and EUnetHTAguidelines

After the Joint Action the Model supports the assessmentof medical and surgical interventions diagnostic and screen-ing technologies and rapid relative effectiveness assessment ofpharmaceuticals Due to the inherently extensible design fur-ther categories of technologies can be included in the Modelthrough amending the ontology and methodological guidance

The rationale for organizing information as collections ofresult cards is twofold First organizing information in resultcards available through an electronic database allows for theirusage at an elementary level not embedded within targeted

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 484

Lampe et al

format using an MS Word document template available foreach domain The template is based on the common reportingstructure and contains all topic-specific research questions ofa domain selected in the previous phase and all other requireddomain-specific fields

The only mandatory information is included in the ldquoRe-sultsrdquo field which should contain the answer in a concise for-mat Optionally researchers can add a description of question-specific methodology (if it deviates from domain methodology)and a comment text The methods should be disclosed as de-tailed as in any scientific work The comment can address forexample research methods used reliability of results or needsfor further research

On the domain-level researchers should include text chap-ters providing an introduction and a description of methodologyused in the analysis as well as a general discussion related to thefindings for the specific domain The discussion section allowsfor bringing together and interpreting the findings that are pre-sented in the result cards of a domain as well as for a discussionon methodological challenges indications for further researchand any other domain-level considerations

4 ResultsIn this phase all results are uploaded to the system and

consequently another entity the collection is created in thedatabase The HTA Core Model Online presents the user witha standard collection structure that contains all relevant partsThe researchers should upload all the domain-specific results(including the result cards) and add the rest of the collection-level information such as a summary

The summary provides an overview of the facts found inthe assessment but it must not contain any recommendationsfor or against the use of the assessed technology Control of thisprinciple takes currently place at the JA work packages produc-ing EUnetHTA collections but more detailed processes will bedeveloped in the future Such recommendations can be added tothe local reports based on the collection Collection-level infor-mation also contains an introduction and a methodology chapter

Currently the upload process from the MS Word templatesto the HTA Core Model Online is a manual process

5 View and SubmitThe final phase allows viewing and submitting the collec-

tion for publication Viewing is available at any point of timebut (before publication) restricted to those who participate inthe project If the project is owned by one of the EUnetHTAPartners or Associates it can be published within the pool ofcore HTA information If the project is owned by others publi-cation within the system is currently not allowed In such casesthe researchers can savedownload the collection and publishit elsewhere Allowing a more free publication platform wouldhave required additional editorial control mechanisms as well asadministrative effort which were not prioritized at this phase

Presentation of Collections Each collection contains a cover page thatdisplays key information the collectionrsquos name producers pub-lishing agency and various data on the Model version used thescope and summary All other contents of the collection areavailable through selecting the desired parts (or all) of the con-tent to be viewed This allows flexible viewing that easily adaptsto various usersrsquo needs

The key contents of each result card are included in the col-lection view as flowing text one after another It is also possibleto open each card in a separate window to access all detailedcontents of each card

Some key features of the Adaptation Toolkit developedby WP5 of the EUnetHTA project 2006ndash08 were included inthe HTA Core Model Online to assist local use of core HTAinformation (21)

DISCUSSIONWe succeeded in clarifying the conceptual basis of the HTA CoreModel and a pragmatic information production process neededfor the online implementation of the Model Some themes thatrequired more attention during the development process or thatperhaps necessitate further development in the future are dis-cussed next

Compared with formal ontology development and someexisting ontologies (2223) the HTA ontology is still in a crudeformat and lacks certain features (for example identificationof some important relations between the content) This turnsinto some practical disadvantages for example in the form ofoverlapping content within different assessment elements andeven across domains Further development work is needed toaddress these issues by identifying relevant relations and byremoving redundant overlaps

The aim of the HTA Core Model has been fromthe beginning to provide overviews of worthwhile researchmethods within various HTA domains as well as practi-cal advice on different assessment elements A considerableamount of guidance is embedded in the Model itself but anequally important aspect is the possibility to link to exist-ing guidance elsewheremdashwhether produced by the EUnetHTAor other parties Practical examples include the SureINFOservice (httpvortalhtaiorgq=sure-info) and EUnetHTAguidelines

After the Joint Action the Model supports the assessmentof medical and surgical interventions diagnostic and screen-ing technologies and rapid relative effectiveness assessment ofpharmaceuticals Due to the inherently extensible design fur-ther categories of technologies can be included in the Modelthrough amending the ontology and methodological guidance

The rationale for organizing information as collections ofresult cards is twofold First organizing information in resultcards available through an electronic database allows for theirusage at an elementary level not embedded within targeted

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 484

HTA Core Model for the Online Environment

reporting for a particular context such as country-specific set-ting Second representatives of some countries wished or re-quired that the end result should not be viewed as a ldquoEuro-pean HTA reportrdquo because the local information needs andrequirements may be different and such a report might bearconsequencesmdashbeyond those intendedmdashto national health sys-tems Consequently the developers aimed at a system that pro-vides a robust evidence base that should be interpreted by localHTA operators How this process will take place is beyond theinfluence of EUnetHTA but it could be done for example byproducing a local HTA report that is partly largely or only basedon a collection of core HTA information (24)

The choice of presenting information as ldquocollections ofcardsrdquo is likely to challenge researchers who are more usedto reading reports and articles We assume however that afterreaching a specific point on the learning curve the standardizedstructure will facilitate easy access to and sharing of relevantinformation This will be further tested in EUnetHTA JointAction 2 (2012ndash15)

Currently the pool of core HTA information is limited tothe HTA Core Model Online This allows concentration of stillrelatively scarce pilot materials into a single knowledge basethat can be further developed piloted and used In the future itmay be useful to expand the concept of core HTA information tocover also information published in other sources It could meaninclusion of any information produced using the Model ormdashforexamplemdasha select assortment of web sites of trusted partnersThe common information structure would allow shared produc-tion publication and usage of structured HTA information evenif it would be distributed to more than one information reposi-tories The political and practical implications of such decisionsneed to be carefully considered in the future

The relation between the project protocol and a (much-needed) more detailed research protocol requires further con-sideration The current solution allows many different and lo-cally familiar practices in developing research protocols On theother hand requests for better functionalities for this purposehave also been expressed during the piloting Incorporating intothe HTA Core Model Online an optional feature of designing adetailed research protocol could be considered in the future

The manual work required within the research phase at themoment has raised criticism by several participants of the pilotprojects The choices are based on relevant decisions withinthe original EUnetHTA project by 2006ndash08 where the primaryfocus was standardization of the contents and methodology ofHTA to allow easier sharing of information Automated upload-ing of results is now on the subsequent development list Theoverall process of using external text editing program versusbuilding web-based text tools should be further considered

There were repeated requests from researchers to includeevidence table templates in the Model The Model does notyet provide such templates A slightly differentmdashand more

advancedmdashapproach has been considered for the Model It in-volves the idea of extracting information from original studiesinto study cards each containing information from one studySuch cards could then be used in a flexible way to produce dif-ferent kinds of evidence tables reducing double work withinsingle projects and across projects A pilot version of such astudy card was drafted to encourage piloting but the work wasnot fully completed within EUnetHTA JA It forms a basis forfurther development taking also into account the work of theEvidence Tables Working Group of the Guidelines InternationalNetwork (httpwwwg-i-nnetactivitiesetwg)

The possibility of allowing users to add comments to exist-ing result cards was considered but not yet implemented

The policy (15) requires that all those who use core HTAinformation must provide an English language summary of thefinal conclusions of their local report This summary will beincluded in the HTA Core Model Online This feature has notbeen implemented yet but it will be included in the future

An important challenge in the production and usage of coreHTA information has been to address all the different require-ments and expectations of various countriesrsquo HTA agenciesThis has been addressed by promoting the idea of core HTAinformation as a robust information source that is used andpossibly further processed locally When users of core HTAinformation start adding summaries of their conclusions intothe database a new important information base is formed Byscrutinizing these local reports and associated comments onthe core HTA information users can possibly gain importantinsights into how specific collections should or could be inter-preted in the subsequent usersrsquo own context

Overall we managed to reach good consensus in the keyaspects of the development and proceeded implementing themonline for further piloting The pilot projects of WP4B provideda valuable testing experience through which some importantbenefits and hurdles of the system were identified (18) Themore challenging themes included for example the appropriategranularity of the ontology (so that each assessment elementwould contain a proper amount of information) the overlapsbetween assessment elements and the method we used to divideassessment elements into core elements and non-core elementsThese themes require further consideration and development

The HTA Core Model is a registered trademark but it shouldnot be seen as reserved for a limited group of HTA agenciesonly The information production features of the HTA CoreModel Online are available to any noncommercial users TheModel in its PDF format can be used by anyone Licensesexist for both noncommercial and commercial use (availableat wwwcorehtainfo)

It remains to be seen how useful the Model is in prac-tice Therefore it is important that HTA researchers start orcontinue using the online tool to build up experience and iden-tify strengths and remaining weaknesses which can then be

485 INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014

Lampe et al

remedied to improve the practicality and added-value of theModel and the online tool The development of the HTA CoreModel Online continues within EUnetHTA Joint Action 2 Mainchallenges include updating and harmonizing the contents of theModel and adding features to enable production of rapid HTAsand local reports

In conclusion we have developed through an internationalcollaborative network an online tool for facilitating the usage ofthe HTA Core Model by many researchers and organizationsThe resulting pool of core HTA information is likely to developinto an important information source that can be used as aservice for local HTA information production Links to resultinglocal reports are also likely to be very useful for users

Existing core HTA information has an important potentialin reducing overlapping work within HTA agencies and promot-ing efficient use of HTA resources By using special expertiseavailable through international collaboration and use of jointlydeveloped or agreed-on scientific methodologies the quality ofHTA information is likely to increase as well

CONTACT INFORMATIONKristian Lampe MD Senior Medical Officer Finnish Officefor Health Techology Assessment (FINOHTA) National Insti-tute for Health and Welfare (THL) POB 30 00271 HelsinkiFinlandIris Pasternack MD HTA professional Summaryx Ltd Un-tuvaisentie 2 A 19 00820 Helsinki FinlandOskari Saarekas MSc Software Developer Finnish Office forHealth Techology Assessment (FINOHTA) National Institutefor Health and Welfare (THL) POB 30 00271 Helsinki FinlandLeena Raustia MSc Senior Planning Officer Finnish Officefor Health Techology Assessment (FINOHTA) National Insti-tute for Health and Welfare (THL) POB 30 00271 HelsinkiFinlandIrina Cleemput Professor Health Economics Hasselt Univer-sity Senior Health Economist Belgian Health Care KnowledgeCentre (KCE) Kruidtuinlaan 55 1000 Brussels BelgiumMirella Corio PhD MSc Health economist - researcherInnovation Research and Development Department NationalAgency for Regional Healthcare Services (Agenas) Via Puglie23 00187 Rome ItalyGottfried Endel MD Team HTA Department for EvidenceBased Economic Health Care Main association of Austrian So-cial Insurance Institutions (HVB) Kundmanngasse 21 ViennaAustriaKatrine Froslashnsdal BASc MSc PhD Senior Researcher Nor-wegian Knowledge Centre for Health Services (NOKC) POBox 7004 St Olavs plass 0130 Oslo NorwayInaki Imaz MD PhD MPH Senior Researcher Agenciade Evaluacion de Tecnologıas Sanitarias (AETS) Instituto deSalud ldquoCarlos IIIrdquo 5 Monforte de Lemos 28029 Madrid Spain

Sarah Kleijnen MSc Advisor Dutch National Health CareInstitute (ZIN) Postbus 320 1110 AH Diemen The Nether-landsFinn Kristensen MD PhD Adjunct Professor University ofSouthern Denmark Campusvej 55 DK 5230 Odense M Di-rector EUnetHTA Secretariat Danish Health and MedicinesAuthority (DHMA) Axel Heides gade 1 2300 Copenhagen SDenmarkAlric Ruther MD PhD Head of Department Departmentof Health Care Quality Institute for Quality and Efficiencyin Health Care (IQWiG) Im Mediapark 8 50670 CologneGermanySophie Werko PhD International Relations Manager ProjectDirector Swedish Council on Health Technology AssessmentSBU Box 3657 103 59 Stockholm SwedenMarina Cerbo DSc Director Innovation Research and Devel-opment Department National Agency for Regional HealthcareServices (Agenas) Via Puglie 23 00187 Rome Italy

CONFLICTS OF INTERESTKristian Lampe reports grants to his institution from the Euro-pean Commission he presented results of this work in a work-shop organised by CIRS (Centre for Innovation in RegulatoryScience) which covered his travel costs Iris Pasternack has re-ceived consultancy money from Summaryx Ltd a company sheowns has been employed by THL (National Institute for Healthand Welfare) her institution has received co-funding throughthe grant for the work described in this paper and has had travelexpenses covered by ISPOR The other authors report grants totheir institutions from the European Commission

REFERENCES

1 Lampe K Makela M Garrido MV et al The HTA core model A novelmethod for producing and reporting health technology assessments IntJ Technol Assess Health Care 200925(Suppl 2)9ndash20

2 Liberati A Sheldon TA Banta HD EUR-ASSESS Project Subgroup re-port on methodology Methodological guidance for the conduct of healthtechnology assessment Int J Technol Assess Health Care 199713186ndash219

3 Pasternack I Anttila H Makela M Ikonen T Rasanen P Lampe K et alTesting the HTA core model Experiences from two pilot projects Int JTechnol Assess Health Care 200925(Suppl 2)21ndash27

4 Kristensen FB Lampe K Chase DL et al Practical tools and methodsfor health technology assessment in Europe Structures methodologiesand tools developed by the European Network for Health Technology As-sessment EUnetHTA Int J Technol Assess Health Care 200925(Suppl2)1ndash8

5 EUnetHTA Work Package 4 HTA core model for medical and surgicalinterventions 10r httpwwwcorehtainfomodelHTA20Core20Model20for20Medical20and20Surgical20Interventions2010rpdf (accessed November 11 2013)

6 EUnetHTA Work Package 4 HTA core model for diagnostic technolo-gies 10r httpwwwcorehtainfomodelHTA20Core20Model20for20Diagnostic20Technologies2010rpdf (accessed November11 2013)

INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014 486

HTA Core Model for the Online Environment

7 EUnetHTA Work Package 4 Core HTA on drug eluting stents httpwwweunethtaeuoutputscore-hta-drug-eluting-stents (accessedNovember 11 2013)

8 EUnetHTA Work Package 4 Core HTA on MSCT Angiography httpwwweunethtaeuoutputscore-hta-msct-angiography (accessed Novem-ber 11 2013)

9 Liberati A Altman DG Tetzlaff J et al The PRISMA statement forreporting systematic reviews and meta-analyses of studies that evaluatehealth care interventions Explanation and elaboration Ann Intern Med2009151W65ndashW94

10 Moher D Hopewell S Schulz KF et al CONSORT 2010 ex-planation and elaboration Updated guidelines for reporting parallelgroup randomised trials Int J Surg 20121028ndash55 doi 101016jijsu201110001

11 von Elm E Altman DG Egger M et al STROBE Initiative TheStrengthening the Reporting of Observational Studies in Epidemiology(STROBE) statement Guidelines for reporting observational studies JClin Epidemiol 200861344ndash349

12 Brouwers M Kho ME Browman GP et al AGREE II Advancing guide-line development reporting and evaluation in healthcare Can Med AssocJ 2010182E839ndashE842

13 Review Manager (RevMan) [Computer program] Version 52 Copen-hagen The Nordic Cochrane Centre The Cochrane Collaboration2012

14 Kidholm K Ekeland AG Jensen LK et al A model for assessmentof telemedicine applications Mast Int J Technol Assess Health Care20122844ndash51

15 EUnetHTA Joint Action WP4 Policy for the HTA Core Model andcore HTA information Version 10 httpwwweunethtaeuoutputspolicy-hta-core-model-and-core-hta-information (accessed November11 2013)

16 Chalon PX Kraemer P The EUnetHTA Information Management Sys-tem Development and lessons learned Int J Technol Assess Health Care2014514ndash520

17 EUnetHTA Joint Action Final Technical Report httpwwweunethtaeuoutputseunethta-ja1-final-technical-report (accessed November 112013)

18 Lo Scalzo A Vicari N Corio M et al Collaborative models for thejoint production of Core Health Technology Assessments Negative andpositive aspects for the joint work of different European agencies Int JTechnol Assess Health Care 2014536ndash543

19 European Network for Health Technology Assessment EUnetHTAStakeholder Forum EUnetHTA httpwwweunethtaeueunethta-stakeholder-forum (accessed November 11 2013)

20 EUnetHTA Work Package 4 HTA Core Model for Screening Technolo-gies 10 httpwwwcorehtainfomodelHTA20Core20Model20for20Screening20Technologies2010pdf (accessed November11 2013)

21 Chase D Rosten C Turner S Hicks N Milne R Development ofa toolkit and glossary to aid in the adaptation of health technologyassessment (HTA) reports for use in different contexts Health TechnolAssess 2009131ndash142 iii

22 Vickery BC Ontologies J Inform Sci 199723277ndash28623 Lumsden J Hall H Cruickshank P Ontology definition and construction

and epistemological adequacy for systems interoperability A practitioneranalysis J Inform Sci 200137248ndash255

24 Hauptverband der osterreichischen Sozialversicherungstrager Abdom-inal aorta aneurysm ndash Cost-effectiveness analysis on introduction oforganized screening in comparison to current practice in Austria 2013httpwwwhauptverbandatportal27portalhvbportalchannel_contentcmsWindowaction=2ampp_menuid=75602ampp_tabid=5 (accessedNovember 11 2013)

487 INTL J OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 305 2014