Panorama Special Issue 1/99 - PortalPanorama Special Issue 1/99 Evaluating Transdisciplinary Research. 3. Editorial / Introduction. Editorial. Research support by the Swiss Na-tional

SCHWERPUNKTPROGRAMMUMWELT SCHWEIZSchweiz. Nationalfonds zur Förderungder wissenschaftlichen ForschungInformationsheft

PROGRAMME PRIORITAIREENVIRONNEMENT SUISSEFonds National Suissede la Recherche ScientifiqueBulletin

SWISS PRIORITY PROGRAMMEENVIRONMENTSwiss National Science FoundationNewsletter

Panorama

Evaluation Criteria forInter and Trans-

disciplinary Research:

Project Report

Instrument

Evaluating Transdisciplinary ResearchInterdisciplinary Centerfor General Ecology (IKAÖ),

University of Berne,Switzerland

Silvia Odermatt

http://www.ikaoe.unibe.ch/forschung/ip/Specialissue.Pano.1.99.pdf

Silvia Odermatt

2 Panorama Special Issue 1/99 Evaluating Transdisciplinary Research

Cont

ents Editorial ............................................................................................................... ................................................. Page 3

Introduction ............................................................................................... ........................................................... Page 3

Please share your experiences with us! ................................................................................................................ Page 4

Project Report ...................................................................................................................................................... Page 5The Task – Our Approach – The Procedure – The Results – Interesting Ideas and Notes from theConsultation of Experts – Outlook: The Dissemination of the Proposal – Addresses of the Project Teamand of the Monitoring Group

Bibliography, Other Documents .............................................................................................................................. Page 12

Instrument – Evaluation Criteria for Inter and Transdisciplinary ResearchExplanations ............................................................................................................................................................ Page 13

Evaluation Sequence (Table) .................................................................................................................................. Page 17

Catalogs of Criteria:Evaluation of Research Proposal (ex ante) – Overarching Project ........................................................................ Page 18Evaluation of Research Proposal (ex ante) – Sub-Project ...................................................................................... Page 20Notes on Intermediary Evaluations ........................................................................................................................ Page 22Final Evaluation (ex post) – Overarching Project ................................................................................................... Page 24Final Evaluation (ex post) – Sub-Project ................................................................................................................. Page 26

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch

3Panorama Special Issue 1/99 Evaluating Transdisciplinary Research

Edi

toria

l / In

trodu

ctio

n Editorial

Research support by the Swiss Na-tional Science Foundation (SNSF) hasalways been competitive and publiclyaccountable. Similarly there is an in-ternational consensus that competi-tive and peer-reviewed research sup-port is the most effective available.Appropriate procedures for the evalu-ation of proposals and the productivityand effectiveness of researchers in-clude the traditional quantitativemeasures of numbers of peer-re-viewed article publications and thecitation index.

For the evaluation of project proposalsin oriented research, however, an ex-panded set of criteria, which go be-yond narrow disciplinary qualifica-tions, are needed. In the case of theSwiss Priority Programme Environment(SPPE) several additional criteria wererequired. First, the proposals have tobe examined with respect to their co-herence with the original goals de-fined in the request for proposals. Anew elements is the evaluation of re-quirements for interdisciplinarity (nat-ural, social, engineering, medical sci-ences and humanities) and fortransdisciplinarity (participation ofaddressees and users of expected re-search results).

Experience in our SPPE with externalpeer-reviews and evaluation proce-dures within the expert group hasshown that such expanded evaluation

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch

Introduction

The SPP Environment Group of Ex-perts asked us to elaborate a proposalon how, and according to what crite-ria, interdisciplinary and transdiscipli-nary research projects may be suita-bly evaluated. We were happy to ac-cept, and carried out the task in closecooperation with the SPP EnvironmentProgram Management and the SNSFsecretariat responsible for the SwissPriority Programs.

In our view it is impossible to laydown once and for all how interdisci-plinary and transdisciplinary researchshould be evaluated; as with research

criteria, which were not readily availa-ble, first had to be developed and im-plemented. Accordingly, criticism inthe initial phase of the SPPE wasmainly levelled against recommenda-tions involving such not strictly disci-plinary criteria.

The expert group in its deliberationshas continuously worked on the devel-opment of these additional evaluationcriteria. Starting from a wide array ofopinions and possible approaches,through intense debate and continu-ous discussions, a group consensushas gradually developed over severalyears. The recognition that this oc-curred in an empirical way and thatthe catalog of questions to be ad-dressed expanded continuously, led tothe proposal of a systematic surveyand treatment of the problems andmethods of evaluation of transdiscipli-nary research proposals and projects.Interest in such a project also wasexpressed by the administrators at theSNSF, who wanted to learn and profitfrom the experience gained within theSPPE in view of the upcoming evalua-tions for the planned new initiative forNational Centres of Competence inResearch.

The authors of the commissionedstudy, Rico Defila and Antonietta DiGiulio, present their results in the cur-rent special issue of «Panorama» to awider audience. The goal of publicis-ing the study now is to promote a wid-er discussion of the issues involved.

Towards this goal, the following per-sonal questions and remarks relatingto my own experience with the SPPEare added. Why did the authors limittheir evaluation catalogues to qualita-tive criteria? Would it not be useful tohave quantitative criteria for compari-sons between competing project pro-posals or for the evaluation of an en-tire program such as the SPPE in com-parison to other oriented research pro-grams or to the traditional basic re-search funding programs of the SNSF?

Could such a comparison not providethe evidence that the efforts and ex-penditures incurred within SPPE forplanning, coordinating and managingresearch, and for synthesizing andimplementing the results have indeedbeen worthwhile and have not onlylead to a lot of «frictional heat» but toquantifiable surplus value results notobtained and obtainable with the tra-ditional research programs? The re-sults of the study presented in thisspecial issue of «Panorama» are afirst and important step towards thatgoal. ◆

in general, any such evaluation mustbe adapted to basic conditions, objec-tives and needs relevant in particularcases. We therefore decided to assigna modular conception to our proposal,with various units designed for adap-tation to the specific situation andfeatures of, for instance, a researchprogram. The modular structure isalso intended to serve as the basisfor discussing and establishing themethods and criteria to be applied inpractice.

The result of our work is printed in thisspecial issue of Panorama, in whichwe present a work report describingprocedures and our approach, and the

instrument for evaluating interdiscipli-nary and transdisciplinary researchitself, in the form of various comple-mentary documents. Certain parts ofthe reportand proposal are redundant:the instrument is intended for use onits own, without further reference tothe report. Thus the proposal itselfrepeats a certain amount already con-tained within the report.

We hope that this instrument provesto be a useful «evaluative tool», andwould welcome any feedback on yourexperience of working with it. ◆

Rico Defila and Antonietta Di Giulio


Plea

se s

hare

you

r exp

erie

nces

with

us!

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch Send this coupon to IKAÖ / Project Evaluation Criteriathe following address: Falkenplatz 16Fax ++41 (0)31 631 87 33 3012 Bern, Switzerland

1. In what context and for what purpose did you use the Instrument?

2. How useful to your purpose (s. question no. 1) did you find ...

... the Instrument overall?very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

... the «Explanations»?very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

... the «Evaluation Sequence (Table)»:very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

... the Catalogs of Criteria, «Evaluation of Research Proposal»:very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

... the «Notes on Intermediary Evaluation»:very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

... the Catalogs of Criteria, «Final Evaluation»:very useful ❐ ❐ ❐ ❐ ❐ ❐ not useful at all

3. Please use this space for any explanations of your assessments in question no. 2 and any further notesand comments you wish to make:

Name/First Name/Institution:

Street/Postcode/City:

Phone/Fax/e-mail:

If you wish to receive further copies of this Newsletter, please note the required number here:

Thank you for your feedback!


Proj

ect R

epor

t

The Task

The issue of adequate evaluation ofinter and transdisciplinary researchhas gained in significance in recentyears, in Switzerland particularly asregards the Swiss Priority Programs(SPP) and the National Centers ofCompetence in Research (NCCR).Since the earliest days of activity, theExpert Group of the Swiss Priority Pro-gramme Environment (SPP Environ-ment) of the Swiss National ScienceFoundation (SNSF) has been con-cerned with this issue. In June 1998,the Group asked Rico Defila andAntonietta Di Giulio from the Interdis-ciplinary Center for General Ecology(IKAÖ) of the University of Berne, todraw up a Catalog of Criteria for theEvaluation of Inter and Transdiscipli-nary Projects – in close cooperationwith the Secretariat of the SPP at theSNSF, and with the Program Manage-ment of SPP Environment. While thisproject is a part of the final work ofSPP Environment, it also served as aninstrument during the preliminarywork of the National Centers of Com-petence in Research (NCCR) in 1999.Rather than being a theoretical discus-sion of issues regarding the evaluationof inter and transdisciplinary research,the aim of this project has been tosuggest procedures for actual, practi-cal use. The present report will there-fore outline the theoretical basis forand development of the Proposal,which is presented in brief, as well assome interesting ideas which resultedfrom a consultation of experts butwere not integrated in the Proposal.Finally, the section Outlook focuses onthe dissemination of the Proposal.

meaningful; there can and should al-ways be an element of self-evaluationby the scientists involved, be that fortheir own benefit or for a third party.A research project invariably encom-passes various aspects whose evalua-tion can be entrusted to various peo-ple.

For ethical reasons it is imperative forthe process to be transparent. Theevaluees must be informed from theoutset of the criteria and consequenc-es, of the procedure, and of who is incharge of which part of the evaluation.

The following must, therefore, be de-termined:• how the evaluation is to proceed;• the aim of any evaluation;• the respective target, and the evalu-

ation criteria;• the consequences if the target has

or has not been reached;• the identity of those who carry out

the target performance comparison.

In research, planning down to theleast detail is impossible. Any re-search evaluation therefore needs tohave a certain degree of flexibility.Moreover, when defining the target, itmakes sense from the outset to en-sure adequate participation of those

concerned in the course of the evalua-tion.

Inter and TransdisciplinaryResearchThe evaluation of inter and transdisci-plinary research requires adequateconsideration of the characteristics of

Our Approach

EvaluationIn research as elsewhere, evaluationgenerally means target performancecomparison. The evaluation of re-search encompasses various criteriaranging from quality control (e.g., indeciding which projects to support) toquality enhancement, i.e., by coach-ing. Any evaluation therefore has acertain aim and there must be conse-quences, both if the target has or hasnot been reached. In quality enhance-ment, any evaluation needs to containsuggestions on how to reach this tar-get.

Research evaluation can be carriedout by various kinds of people, such asexperts in various disciplines, secre-tariats of institutions promoting re-search (or of research programs), prac-titioners, or experts in organisationaldevelopment. However, external eval-uations are not always necessary nor

Project TeamRico DefilaAntonietta Di GiulioInterdisciplinary Center forGeneral Ecology (IKAÖ),University of Berne

Monitoring GroupDr. Rudolf HäberliWalter Grossenbacher-MansuyProgram Management SPPEnvironment

Dr. Urs ChristDr. Stefan BachmannSecretariat, Division IV, SNSF

Participants in theConsultation of ExpertsExperts, both Swiss and foreign,from the realms of science,research promotion and researchmanagement

Evaluation

Target

Quality Control

Target

Quality Enhancement

Target

Per-formance

Per-formance

Per-formance

Evaluation means the comparison of a target with actual performance. While QualityControl compares performance and target, the aim of Quality Enhancement is to mini-mize the difference between the two states.

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch

Rico

Def

ila /

Anto

niet

ta D

i Giu

lio


this kind of research in the evaluation;these characteristics, therefore, needto be recorded in a requirement profileand to be operationalised for evalua-tion.

«Interdisciplinary research» here de-notes the integration-oriented cooper-ation of scientists from at least twodisciplines with the aim to reach com-mon objectives, thereby merging thesundry disciplinary viewpoints into agreater, more complete view. The dis-ciplines involved are those likely tomake a useful contribution to thetreatment of a theme. «Transdiscipli-nary research», in turn, denotes a spe-cial form of interdisciplinary researchinvolving practitioners from beyondthe realm of science. The premise,moreover, is problem-oriented re-search, i.e., research intended to makea contribution towards the solution(and prevention) of socially relevantissues.

The following are some of the specialcharacteristics of inter and transdisci-plinary research in the sense of a re-quirement profile:• Inter and transdisciplinary research

is usually carried out in projectgroups uniting various sub-projects,or research groups in an overarch-ing project. Individual contributionstoward the overarching project needto be identified.

• Consensus: The participants need toagree upon common objectives andquestions, and upon a shared ap-proach to dealing with them; theyneed to arrive at both a shared viewof the problem, and a common lan-

guage. Therefore, consensus is herenot intended to mean agreement orauthorisation, as in everyday lan-guage, but denotes the integrationof various disciplinary viewpoints toachieve a common ground. For ex-

ample, a description of the researchsubject that is equally valid for allwill have to be found. Appropriateprocedures and methods need to beused to achieve consensus.

• Integration: It is necessary from theoutset to combine, by adequate

means and methods, the results ofthe individual sub-projects (or re-search groups) to form a whole thatgoes beyond a simple addition ofthe individual results, and one thataims at answering the questions

held in common. The aim, therefore,is to achieve shared results andproducts.

• Diffusion: The results require ade-quate publication, and their recep-tion with the target audience needsto be promoted. Usually, this audi-ence will be neither disciplinary norpurely scientific, just as the chan-nels of dissemination will often benon-disciplinary. The knowledgegained will need to be useful to thetarget audience and their actions.

• It will be necessary, especially intransdisciplinary research, to ensurefrom the outset the adequate partic-ipation of users in the project andresearch work.

• In problem-oriented research, boththe problem needing to be resolvedand the contribution expected fromthe research work has to be pre-sented.

• Cooperation in such a project groupneeds to be structured, and theprocesses of consensus buildingand integration also need to bestimulated, moderated and moni-tored. Special management is there-fore required.

• There is also the problem of the so-called «surplus value» of inter andtransdisciplinary research: this val-

ue can actually be assessed only ina direct comparison of disciplinaryand inter/transdisciplinary researchprojects dealing with the samequestion – which, for obvious rea-sons, is not feasible. The task,

Dissemination of results – not only among scientists, but also among the general public

«In any case, care should be taken

to avoid the hampering of research

activities by any evaluation,

especially in the course of the

project.»␣ *)

*) Voices from the Consulta-tion of Experts

Proj

ect R

epor

t

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch


therefore, is to ascertain the extentto which an inter or transdiscipli-nary approach to the research sub-ject is justified or even necessary,by demonstrating, for example, thata disciplinary treatment of the sub-ject has not been successful.

• Of course, inter and transdiscipli-nary research must be scientific. Itshould be noted, however, that thesub-projects (or research groups) ofsuch a project group may not al-ways be at the cutting edge of dis-ciplinary and specialised research.

The Procedure

On the basis of the theory outlinedabove, the Team consulted selectedliterature on research evaluation, andexisting or proposed procedures andcriteria for the evaluation of research

– both disciplinary and inter/transdis-ciplinary (see p. 12), always keeping inmind the pragmatic objective of thisproject, i.e. the drafting of a proposalfor the actual evaluation of inter andtransdisciplinary research. The point,then, has not been to present a scien-tific discussion of issues of researchevaluation in general, or inter andtransdisciplinary research in particular.

The present Proposal, «Evaluation Cri-teria for Inter and TransdisciplinaryResearch», was drawn up by the au-thors (Project Team) in a continuousdialogue with a Monitoring Group con-sisting of the following representa-tives (see also box p. 11):• Dr. Rudolf Häberli and Walter Gros-

senbacher-Mansuy, Program Man-

agement, SPP Environment, SwissNational Science Foundation (SNSF);

• Dr. Stefan Bachmann, SPP Environ-ment, and Dr. Urs Christ, SPP De-

main la Suisse/Switzerland: To-wards the Future, Secretariat, Divi-sion IV, NSF.

The Monitoring Group ensured theinclusion in this Proposal of practicalexperience from evaluating inter andtransdisciplinary research, therebyensuring that the result would consid-er the needs of those ultimately in-tended to use the instrument. Theyhave also been essential in the adop-tion of our Proposal in NSF proce-dures. We would like to take this op-portunity to express our sincerethanks to the members of the Monitor-ing Group for their invaluable coopera-tion on this project.

In order to integrate as much experi-ence from inter and transdisciplinaryresearch and its evaluation, the firstdraft of this Proposal was submittedfor criticism to a panel of experts inSwitzerland and abroad. Approx. 100

experts from the realms of science,research promotion and research man-agement were invited to comment.They were

• scientists whose work will be evalu-ated, and those acting as reviewers;

• managers of project groups or re-search programs;

• scientists concerned with the theoryof inter and transdisciplinary re-search, and those doing accompany-ing research;

• people actively involved in tertiaryeducation and science policy, andthose working in interdisciplinaryinstitutions in the tertiary educa-tional sector.

There were 40 responses to our con-sultation of experts, some of themwith extensive comments. The resultsof this consultation as well as a list ofthe respondents can be found in the

German version of this Special Issue(http://ikaoewww.unibe.ch/dokus/Sondernummer_Pano_1=99.pdf).Some particularly interesting sugges-tions which, for methodological rea-sons, could not be integrated into our

«Usually, the demands placed on

leadership in an interdisciplinary

research project are completely

underestimated.»

«How should

transdisciplinary ‹surplus

value› be assessed?

This issue remains

unresolved.»

Proj

ect R

epor

t

«The timing and number of

evaluations need to be handled

with flexibility and regard to each

specific project.»


Proposal will be presented brieflyfurther on. The consultation had asignificant impact on the outcome ofour Proposal, and we would like toextend our sincere thanks to everyonewho contributed to our project in thisway.

The Results

The following is a brief overview ofthe Proposal, «Evaluation Criteria forInter and Transdisciplinary Research».It consists of a set of documentswhich, with a few adaptations, can beused for the evaluation of varioustypes of inter and transdisciplinaryresearch. The set consists of the fol-lowing documents:1. Explanations2. Evaluation Sequence (Table)3. Catalogs of Criteria

ExplanationsThe explanations outline the theoreti-cal basis of the Proposal; moreover,some of the terms used in the cata-logs of criteria are defined. In particu-lar, it is explained what will need tobe defined in the event of an evalua-tion (e.g., for a specific research pro-gram), and how to adapt and usethese catalogs.

Evaluation Sequence (Table)This table gives an overview of theEvaluation Sequence in its variousstages, assuming a research program

running approximately four years. Theprocess involves an evaluation of theresearch proposal, followed by threeintermediary and one final evaluation,

as well as an impact evaluation (theimpact of research on science andsociety). Specific objectives set for

each evaluation, and possible conse-quences have been outlined (i.e.,sanctions as an example for possibleconsequences). It is also suggestedthat not all aspects of a researchproject should be assessed in eachevaluation; the table shows whichaspects should be focused upon atwhich stage.

The number of evaluations to be car-ried out in any specific research pro-gram, and the time at which theyshould take place will need to be de-termined for each program, takinginto account its particular parametersand requirements. Likewise, possibleconsequences to the evaluation willneed to be stated at the outset of anyevaluation. Although peer reviewinghas been assumed to be the mostlikely procedure, the question of whois in charge of which aspect has notbeen resolved; this, too, will need tobe determined for each individualresearch program.

No suggestions have been made forthe basis of the evaluation, i.e.,whether it should be based uponreports, interviews, site visits, etc.:this will need to be determined foreach specific research program,taking into account its particularparameters.

Evaluating the impact of a programhas been adopted for the sake of com-pleteness without making any further

suggestions as regards the actual pro-cedure or criteria, etc. It should also

be pointed out that, for the vast major-ity of the projects, impact assessment

Intermediary evaluations – assessment should focus upon selected aspects

«In view of the

various

intermediary

evaluations it is

important to

avoid a mentality

of scrambling to

‹fulfill the plan›

in the projects.»

Proj

ect R

epor

t

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch

«I think training of the

evaluators by the

research promoting

organisation will be

required.»


at three to five years after completionwill probably come rather too earlysince the impact of research work of-ten manifests itself only after tenyears or more.

Catalogs of CriteriaThe criteria to be used at the differentstages of the evaluation have beenformulated as questions. An attempthas been made to draw up a compre-hensive questionnaire to encompassthe characteristics of inter andtransdisciplinary research that readilymanifest themselves to the outsideobserver. The questions can be usedboth in self-evaluation and in externalevaluations, in coaching as well as inassessments. The vast majority ofthese questions are neither quantified

nor quantifiable; the reviewers’ judg-ment is required.

Catalogs of criteria were developedboth for the evaluation of a projectgroup as well as for that of sub-projects, or research groups within

such a project group. These catalogshave been harmonized and are com-plementary. (For example, it is possi-

ble to assess whether the objectivesof the project group are «shared» or«common» ones, i.e., whether every-one involved is willing and able to

reach them: if so, this should becomeevident both from the design of thesub-projects and the structure of theoverarching project.) Only the catalogsfor the evaluation of the proposal andfor the final evaluation have beendrawn up in detail. As regards inter-

mediary evaluations, some sugges-tions have been made on how to de-velop catalogs of criteria on the basis

of the evaluation of the research pro-posal and/or the final evaluation.

In the event of an evaluation, thequestions suitable for the evaluationof the respective research programwill have to be selected from thesecatalogs, and suitable catalogs willhave to be drawn up for the intermedi-ary evaluations. The flexibility re-quired in evaluating research shouldbecome particularly evident in the in-termediary evaluations; for example, itshould be possible to adapt the re-search objectives and questions aswell as the desired products to chang-ing parameters. The quantifiable crite-ria have been kept intentionally openin the Proposal («a sufficientnumber»); they will have to be deter-mined in the actual event. The scien-tists will also need to participate inthe selection and definition of thecriteria, one possible result of thisactivity being an agreement definingthe respective obligations of thescientists and the research promotinginstitution. In the event of an actualevaluation, the criteria also need tobe identified for which the evaluatorswill have to justify their assessments.

Finally, it must be pointed out that thisProposal is based upon the assump-tion that the individual sub-projects orresearch groups within a project groupmay again be inter or transdiscipli-nary. However, this will not always bethe case and the questions have beenformulated in such a way that theycan also apply to disciplinary sub-projects or research groups.

To be determined for each program: how much is enough?

«I don’t think there is a general rule

as regards ‹a sufficient number of›

publications, etc.»

Proj

ect R

epor

t


Interesting Ideas and Notesfrom the Consultation ofExperts

The following is a selection of sugges-tions and notes taken from the consul-tation of experts, which were notadopted in this Proposal.

On high demands, and their fund-ing: The demands placed on scientistswishing to draw up a research propos-

al for an inter or transdisciplinaryproject are very high, not only as re-gards quality, but especially as re-gards the resources required in theearly stages of designing the proposal.Such high demands can only be madeif the required time and funding havebeen budgeted. Sufficient funds alsohave to be earmarked for the specialdemands of inter and transdisciplinaryresearch projects (e.g., achieving syn-thesis).

On the selection of evaluators:Potential evaluators should be recruit-ed not just from among the traditionalactors. Especially in inter andtransdisciplinary research it is essen-tial for the evaluation of specific as-pects to be entrusted to individualscompetent in the relevant field, e.g.,issues of project management shouldbe evaluated by experts in organisa-tional development, issues of scientif-ic quality by scientists, issues of so-cial relevance and the description ofsocietal problems by practitioners.

On self-evaluation and evalueeparticipation: Adequate relevancemust be accorded to self-evaluation by

the scientists themselves. One possi-bility for the scientists’ participationwould be the evaluation of the sub-projects by the management of aproject group prior to submission ofthe research proposal. Another possi-bility would be a joint internal evalua-tion by the scientists on the sub-projects (or research groups) with themanagement, which, however, placesparticularly high demands on the man-agement’s social competence. A fur-

ther possibility would be for each sub-project participating in the proposal toidentify three criteria on which as-sessment should be carried out.

On taking risks vs. securing suc-cess: To make predictions of successis highly problematic as this might

lead to risky projects being systemati-cally excluded. This would be regretta-ble since risky projects often have ahigh innovative potential and shouldtherefore be admitted to researchfunding.

On the unpredictability of externalparticipants: It must be rememberedthat, while external participants canbe contracted to make a specific con-tribution to a research project, ulti-mately these outsiders cannot beforced to make that contribution.Moreover, while it may be possible toschedule this contribution, it may beimpossible to manage it at all times,which may lead to the required contri-bution not being made.

On the unpredictability of productorientation: As regards product ori-entation of research, a certain degreeof caution needs to be exercised:while it is possible to conceptuallydevelop and target products, their ac-tual realisation and market accept-ance often depends upon parametersbeyond the scientists’ sphere of influ-ence.

On the problem of overall assess-ments and of quantitative criteria:While overall assessments are mean-ingful and necessary, there is a risk ofthe assessment of the individual crite-ria being swept under the carpet inthe final analysis. This needs to beavoided. As regards quantitative crite-ria («a sufficient number», etc.) thefact needs to be remembered thatsuch criteria may result in quantitybeing given primacy over quality.Moreover, it is usually impossible todetermine the desirable number ofpapers, lectures and seminars, etc., inadvance; this is more appropriately

done in the course of negotiations,once the project is underway. Finally,the fact needs to be borne in mindthat the number of publications tendsto depend upon parameters beyondthe scientists’ control.

«Obviously, the energy required

for this type of evaluation has

increased over the past years –

this must be made clear when

recruiting experts.»

«Depending on the desired product,

the researchers may find it to be

incomparably less manageable and

controllable than publications.»

Proj

ect R

epor

t

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch


Addresses of the Project Team and of the Monitoring Group

Project TeamRico Defila / Antonietta Di GiulioUniversity of BerneInterdisciplinary Center for General Ecology (IKAÖ)Falkenplatz 163012 Bern, [email protected]@ikaoe.unibe.ch

Monitoring GroupDr. Rudolf Häberli / Walter Grossenbacher-MansuySPP EnvironmentLänggassstr. 233012 Bern, Switzerland

Dr. Urs Christ / Dr. Stefan BachmannSwiss National Science FoundationSPP SecretariatWildhainweg 203001 Bern, Switzerland

On the consequences of the finalevaluation: While it is usually quitesimple to determine the consequencesof evaluations of proposals and inter-mediary evaluations, it is more diffi-cult to do so as regards the final eval-uation. One possibility would be toauthorise the payment of a final, addi-tional sum, subject to certain condi-

tions, in order to ensure, for example,the completion of certain implementa-tion work.

On accompanying research vs.evaluation: Research on the programitself is called for if the aim is to ob-serve the functioning of research proc-esses in a research program, and toidentify suitable methods of integra-tion, or cooperation with users, etc.Evaluation of research cannot replaceaccompanying research because anevaluation always considers only aslice of the research process, that is,process results rather than the proc-esses as such.

On program evaluation: It remainsunclear, how and by whom an entireresearch program would best be eval-uated. One possibility could be to askquestions during the final evaluation(or even when evaluating impact) thatwould be aimed at a program evalua-tion. One such question could bewhether the research results mightjeopardise the objectives of the re-search program.

Outlook – the Dissemination ofthe Proposal

Now that the Project Report has beenhanded to the Swiss National ScienceFoundation, for the attention of theGroup of Experts of SPP Environment,

and of those in charge of the NationalCenters of Competence in Research(NCCR), work on the Report has cometo a formal conclusion. Apart from theSNSF, the primary audience of the Pro-posal, «Evaluation Criteria for Interand Transdisciplinary Research», areother institutions of research promo-tion, both public and private, both in

Switzerland and abroad. Moreover,especially those who have alreadybeen involved in the consultation ofexperts, as well as the scientiststhemselves are also interested in theresults. The Project Team picked upsignals to this effect, both when con-ducting the consultation and whenpresenting their project. A desire has

«The criteria read like guidelines

for the design of inter and

transdisciplinary research

projects.»

been expressed for the results to bepublished so as to be available to any-one interested and for this Proposal tobe tested, discussed and developedfurther subsequent to practical imple-mentation.

The Project Team is therefore veryhappy with the decision taken by themanagement of SPP Environment topublish the present Report as a Spe-cial Issue of Panorama, not only inGerman, but in English as well. TheTeam will also ensure the dissemina-tion of the Report on other occasionsfor discussion by a larger audiencefrom the realms of science and sci-ence policy. Among other things, theReport is on the WWW (http://ikaoewww.unibe.ch/top_forsch.html)– also enabling the publication on theWeb of feedback on implementation.The Team hopes to have made a smallcontribution towards the discussion ofa meaningful evaluation of inter andtransdisciplinary research. ◆

Proj

ect R

epor

t


Selected Bibliography

Aebi F. (1987): Technisch und wirtschaftlich orientierteNationale Forschungsprogramme (NFP). Gedanken undErfahrungen. Wissenschaftspolitik 16 (1–2): S. 11–18.Aebi F. (1985): Was, wie und wem nützen die NationalenForschungsprogramme? Technische Rundschau (40).Amann K., Knorr-Cetina K. (1991): Qualitative Wissen-schaftssoziologie. In: Flick U., Kardorff E. von, Keupp H.,Rosenstiel L. von, Wolff S. (Hg.): Handbuch qualitativeSozialforschung. Grundlagen, Konzepte, Methoden undAnwendungen. München: Psychologie Verlags Union: S.419–423.Arbeitsgruppe «Interdisziplinarität – Evaluationskri-terien» der Kommission für Umweltwissenschaftenund der Hochschulplanungskommission (1998):Vorschläge für Evaluationskriterien inter- und trans-disziplinärer Projekte. In: Schweizerische Hochschulkonfe-renz (Hg.): Interdisziplinarität im Zusammenwirken vonUniversität und Öffentlichkeit. Kolloquium vom 25./26.September 1997 in Neuenburg. Bern.Arbeitsgruppe Praxisbegleitende UmweltforschungSchweiz (PUSCH) (1996): Forschen für eine nachhaltigeSchweiz. Ein Diskussionsbeitrag der SchweizerischenAkademischen Gesellschaft für Umweltforschung undÖkologie (SAGUF) für Planung, Durchführung und Umset-zung von Forschungsprojekten. Zürich.Balsiger Ph., Defila R., Di Giulio A. (Hg.) (1996): Ökolo-gie und Interdisziplinarität – eine Beziehung mit Zukunft?Wissenschaftsforschung zur Verbesserung der fachüber-greifenden Zusammenarbeit. Basel, Boston, Berlin: Birk-häuser.Berk R. A., Rossi P. H. (1990): Thinking About ProgramEvaluation. Newbury Park, London, New Delhi: SAGEPublications.Blaschke D. (1986): Zur Beurteilung interdisziplinärersozialwissenschaftlicher Forschung. In: Fisch R., DanielH.-D. (Hg.): Messung und Förderung von Forschungslei-stung. Person – Team – Institution. Konstanz: Universi-tätsverlag (Konstanzer Beiträge zur sozialwissenschaftli-chen Forschung; Bd. 2): S. 167–190.Bortz J., Döring N. (1995): Forschungsmethoden undEvaluation. Berlin: Springer.Büchi H. (1996/97): Inter-, Trans- und Multidisziplinarität– Bemerkungen zu einer Begriffsverwirrung. Probleme inder realen Welt wahrnehmen. ETH-intern 15: S. 8–9.Buschor E. (1991): Ansätze für die Universitäts- undWissenschafts-Evaluation. Schweizerische Hochschulkon-ferenz. Hochschulplanungskommission. Hochschule St.Gallen: Typoskript.CASS –Konferenz der Schweizerischen Wissen-schaftlichen Akademien, Forum für Klima und Glo-bal Change (ProClim) (1997): Visionen der Forschenden.Forschung zu Nachhaltigkeit und Globalem Wandel –Wissenschaftspolitische Visionen der Schweizer For-schenden. Bern.Chen H.-T. (1990): Theory-Driven Evaluations. NewburyPark, London, New Delhi: SAGE Publications.Chubin D. E., Porter A. L., Rossini F. A., Conolly T.(Hg.) (1986): Interdisciplinary Analysis and Research.Theory and Practice of Problem-Focused Research andDevelopment. Maryland: Lomond Publications.Da Pozzo F. (1995): Forschungsevaluationen des Schwei-zerischen Wissenschaftsrates. FUTURA 1 (2): S. 19–34.Daele W. van den, Krohn W., Weingart P. (Hg.) (1979):Geplante Forschung. Vergleichende Studien über denEinfluss politischer Programme auf die Wissenschaftsent-wicklung. Frankfurt a. M.: Suhrkamp.Defila R., Di Giulio A. (1996): Interdisziplinäre For-schungsprozesse: Erwartungen und Realisierungsmöglich-keiten in einem Forschungsprogramm – das Schwerpunkt-zentrum «Umweltverantwortliches Handeln» in seinemUmfeld. In: Kaufmann-Hayoz R., Di Giulio, A. (Hg.): Um-weltproblem Mensch. Humanwissenschaftliche Zugängezu umweltverantwortlichem Handeln. Bern, Stuttgart,Wien: Haupt: S. 79–129.Effenberger F. (1992): Leistungsbewertung in der For-schung aus der Sicht der Hochschulen. In: Bundesministe-rium für Wissenschaft und Forschung (Hg.): Die Bewer-tung von Leistungen im Bereich von Lehre und Forschung.L&R Sozialforschung. Wien: Bundesamt für Wissenschaftund Forschung: S. 125–128.Eliassen A., Malcorps H., Bengtsson L., Fijnaut H. M.(1997): New Opportunities and Challenges. InternationalEvaluation of the Finnish Meteorological Institute. Evalua-tion Report of the Ministry of Transport and Communicati-ons Finland. Helsinki: Edita Ltd.Gibbons M., Limoges C., Nowotny H., SchwartzmanS., Scott P., Trow M. (1994): The New Production ofKnowledge. The Dynamics of Science and Research inContemporary Societies. Newbury Park, London, NewDelhi: SAGE Publications.Häberli R., Grossenbacher-Mansuy W. (1998): Trans-disziplinarität zwischen Förderung und Überforderung.Erkenntnisse aus dem SPP Umwelt. GAIA 7 (3): S. 196–213.Hellstern G.-M., Wollmann H. (1984): Handbuch zurEvaluationsforschung. Bd. 1. Opladen: WestdeutscherVerlag.Hill W., Rieser I. (1983): Die Förderungspolitik desNationalfonds im Kontext der schweizerischen For-schungspolitik. Bern, Stuttgart, Wien: Haupt.Hirsch G. (1995): Beziehungen zwischen Umweltforschungund disziplinärer Forschung. GAIA 4 (5/6): S. 302–314.Hirsch G., Hungerbühler K., Koller T. (1996/97): Dis-kussionsbeitrag «Transdisziplinäre Forschung – was istdamit gemeint?» ETH-intern 9: S. 6–7.Hirsch G., Hungerbühler K., Koller T. (1996/97): Rück-

meldungen zum Diskussionsbeitrag «TransdisziplinäreForschung – was ist damit gemeint?». VerschiedeneDefinitionen. ETH-intern 15: S. 8.Hordijk L., Kroeze C., Alcamo J., Davies T., DrewryD. J., Ford D., Mearns L., Wright R. F. (1996): Evaluati-on of the Finnish Research Programme on Climate ChangeSILMU. Publications of the Academy of Finland 5/95.Helsinki: Edita Ltd.Hornbostel S. (1997): Wissenschaftsindikatoren. Bewer-tungen in der Wissenschaft. Opladen: WestdeutscherVerlag.Klein Thompson J. (1990): Interdisziplinarity. History,Theory & Practice. Detroit: Wayne State University Press.Kocka J. (Hg.) (1987): Interdisziplinarität. Praxis – Her-ausforderung – Ideologie. Frankfurt a. M.: Suhrkamp.Krott M. (1997): Evaluierung als Instrument der For-schungspolitik. Alternative Verfahren am Beispiel inter-disziplinärer Projekte. In: Sozialwissenschaftliche Studi-engesellschaft – Rundschau (SWS)37 (3): S. 269–283.Krull W., Sensi D., Sotiriou D. (1991): Evaluation ofResearch & Development. Current Practise and Guideli-nes. Synthesis Report. Brüssel: Kommission der Europäi-schen Gemeinschaften.Kuhlmann S., Holland D. (1995): Evaluation von Techno-logiepolitik in Deuschland. Konzepte, Anwendung, Per-spektiven. Heidelberg: Physica-Verlag.Mayntz R. (1985): Forschungsmanagement – Steuerungs-versuche zwischen Scylla und Charybdis. Probleme derOrganisation und Leitung von hochschulfreien, öffentlichfinanzierten Forschungsinstituten. Opladen: Westdeut-scher Verlag.Meadows J. (1976): Diffusion of Information Across theSciences. Interdisciplinary Science Reviews 1 (3): S. 259–267.Mittelstrass J. (1989): Wohin geht die Wissenschaft?Über Disziplinarität, Transdisziplinarität und das Wissenin einer Leibniz-Welt. In: Mittelstrass J. (Hg.): Der Flugder Eule. Von der Vernunft der Wissenschaft und derAufgabe der Philosophie. Frankfurt a. M.: Suhrkamp: S.60–88.Mittelstrass J. (1995): Transdisziplinarität. Panorama (5):S. 45–53.Mogalle M. (1999): Transdisziplinäre Nachhaltigkeitsfor-schung – vom Schlagwort zu Forschungskonzeptionen. St.Gallen: IWÖ-HSG.Müller B. (1992): Leistungsbewertung in der Forschung,insbesondere aus der Sicht der Deutschen Forschungsge-meinschaft. In: Bundesministerium für Wissenschaft undForschung (Hg.): Die Bewertung von Leistungen im Be-reich von Lehre und Forschung. L&R Sozialforschung.Wien: Bundesamt für Wissenschaft und Forschung: S.106–112.Neidhardt F. (1988): Selbststeuerung in der Forschungs-förderung – das Gutachterwesen der DFG. Opladen:Westdeutscher Verlag.OECD – Organisation for Economic Cooperation andDevelopment (Hg.) (1987): Evaluation of Research. ASelection of Current Practices. Paris.Raan A. F. J. van (1995): Bewertung von Forschungslei-stungen: Fortgeschrittene bibliometrische Methoden alsquantitativer Kern von Peer Review basierten Evaluatio-nen. Vortrag zum Thema «Qualitätssicherung in der For-schung», Eröffnungsveranstaltung des Centrums fürHochschulentwicklung (CHE) am 25./26. Januar 1995:Typoskript.Rauch H. (1992): Evaluation von Forschungsprojekten. In:Bundesministerium für Wissenschaft und Forschung (Hg.):Die Bewertung von Leistungen im Bereich von Lehre undForschung. L&R Sozialforschung. Wien: Bundesamt fürWissenschaft und Forschung: S. 113–124.Rossi P. H., Freeman H. E., Hofmann G. (1988): Pro-gramm-Evaluation. Einführung in die Methoden ange-wandter Sozialforschung. Stuttgart: Enke.Rossini F. A., Porter A. L. (1978): The Management ofInterdisciplinary, Policyrelated Research. In: Sutherland J.W., Legasto A. Jr. (Hg.): Management Handbook forPublic Administrators. New York, Cincinatti, Atlanta,Dallas, San Francisco: Van Nostrand Reinhold Company:S. 302–333.Schenker-Wicki A. (1996): Evaluation von Hochschullei-stungen: Leistungs-Indikatoren und Performance Measu-rements. Wiesbaden.Scheuermann M. (1998): Kooperation durch Koordinati-on – Wissenschaftsmanagement in der sozialwissen-schaftlichen Umweltforschung. Dissertation UniversitätFreiburg (im Druck).Schwarz R. (Hg.) (1974/75): Wissenschaft als interdiszi-plinäres Problem. Internationales Jahrbuch für interdiszi-plinäre Forschung. 2 Bde. Berlin, New York: de Gruyter.Schweizerischer Nationalfonds (1998): Mehrjahres-programm des SNF für die Beitragsperiode 2000–2003.Bern.Schweizerischer Wissenschaftsrat (Hg.) (1994):Evaluation der Schwerpunktprogramme des Bundes. Bern.Steck R., Dupree B. (1976): Interne Kontrolle in Sonder-forschungsbereichen. Ist die Forderung nach Leistungs-überwachung in interdisziplinären Forschungsorganisatio-nen der Hochschulen realisierbar? Deutsche Universitäts-zeitung (8): S. 232–234.Stiftung Mensch-Gesellschaft-Umwelt MGU (1998):Ausführungsplan zur Ausschreibung für interdisziplinäreForschungsprojekte. Forschungsperiode 2000-2002. Uni-versität Basel.Tempus P. (1990): Bericht über die in den Jahren 1989und 1990 an den ETH durchgeführten Evaluationen überdie Forschungstätigkeit von Departementen. Zürich:Typoskript.Trier U. P. (1997): Ertragsbilanz (Editorial). Nationales

Forschungsprogramm 33 – Die Wirksamkeit unsererBildungssysteme. Bulletin 5. Bern: S. 11.Weingart P. (1995): Forschungsindikatoren: Instrumentepolitischer Legitimierung oder organisatorischen Lernens. In:Müller-Böling D. (Hg.): Qualitätssicherung in Hochschulen.Forschung – Lehre – Management. Eröffnungsveranstaltungdes Centrums für Hochschulentwicklung (CHE) am 25./26.Januar 1995. Gütersloh: Verlag Bertelsmann: S. 73–84.Weingart P. (1997): Interdisziplinarität – der paradoxeDiskurs. Ethik und Sozialwissenschaften (EuS). Streitfo-rum für Erwägungskultur 8 (4): S. 521–529.Weingart P., Winterhager M. (1984): Die Vermessungder Forschung. Theorie und Praxis der Wissenschaftsindi-katoren. Frankfurt: Campus Verlag.Winter H. (1992): Evaluation der physikalischen For-schung in Österreich – Erfahrungen und Empfehlungen. In:Bundesministerium für Wissenschaft und Forschung (Hg.):Die Bewertung von Leistungen im Bereich von Lehre undForschung. L&R Sozialforschung. Wien: Bundesamt fürWissenschaft und Forschung: S. 129–134.

Other Documents

Deutsche Forschungsgemeinschaft (1972): Leitsätzeder Deutschen Forschungsgemeinschaft für die Förderungvon Sonderforschungsbereichen.Deutsche Forschungsgemeinschaft (o. J.): Hinweisefür die Begutachtung von Anträgen auf Einrichtung undFörderung von Graduiertenkollegs. Bewertungskriterien.Deutsche Forschungsgemeinschaft (o. J.): Hinweisefür die Begutachtung von Anträgen auf Weiterführung vonGraduiertenkollegs.Di Giulio A., Künzli Ch. (1998): Interdisziplinarität.Methodenseminar WS 97/98. Universität Bern.Gebert Rüf Stiftung (1999): Kriterien der Projektförde-rung. Zürich.Management and Coordination Office Biodiversity(1998): Integriertes Projekt Biodiversität SPPU, 96–99.Fragenkatalog. Workshop Diskussionsforum Transdiszipli-närer Forschungsprozess 24./25.3.98.National Academy of Sciences (1998): Science andTechnology Centers – Principles and Guidelines. A Reportby the Panel on Science and Technology Centers.Schweizerische Kommission für Forschungspart-nerschaften mit Entwicklungsländern (1997): 11Prinzipien für Forschungspartnerschaften mit Entwick-lungsländern. Bern.Schweizerischer Nationalfonds (1985): NFP 7 – Roh-stoff- und Materialprobleme. Das NFP 7 aus der Sicht derBeteiligten. Ergebnisse einer Umfrage bei den Projektneh-mern und Experten, zusammengestellt durch den Vorsit-zenden der Expertengruppe, Prof. Dr. Franz Aebi. Bern.Schweizerischer Nationalfonds (1992): Ausführungs-plan zum Schwerpunktprogramm Umwelt. Bern.Schweizerischer Nationalfonds (1994/98): Evaluationdes Demandes de Subsides de Recherche par la Division2: un Apercu. Bern.Schweizerischer Nationalfonds (1995): Ausführungs-plan zum Schwerpunktprogramm Umwelt. Bern.Schweizerischer Nationalfonds (1995): Leitsätze fürdie Schwerpunktprogramme des Bundes (SPP). Bern.Schweizerischer Nationalfonds (1995): SPP Umwelt.Beurteilungsblatt für Gesuche. Bern.Schweizerischer Nationalfonds (1997): NationaleForschungsschwerpunkte. Eine Initiative des Schwei-zerischen Nationalfonds zur Reform der Schwerpunktpro-gramme. Bern.Schweizerischer Nationalfonds (1998): SPP Umwelt.Formular: Progress Report 1998. Bern.Schweizerischer Nationalfonds (o. J.): Gesuchseva-luation in der Abteilung III, Biologie und Medizin. Bern.Schweizerischer Nationalfonds (o. J.): NationaleForschungsprogramme. Beurteilungsblatt für Skizzen undGesuche. Bern.Schweizerischer Nationalfonds (o. J.): NationaleForschungsprogramme. Gutachten zum wissenschaftli-chen Schlussbericht. Bern.Schweizerischer Nationalfonds (o. J.): NationaleForschungsprogramme. Gutachten zum wissenschaftli-chen Zwischenbericht. Bern.Schweizerischer Nationalfonds (o. J.): NFP 33 – DieWirksamkeit unserer Bildungssysteme. Beurteilungsblattfür Schlussberichte.Schweizerischer Nationalfonds (o. J.): Recueil desdécisions de principe de la division I. Bern.Schweizerischer Nationalfonds (o. J.): Schwerpunkt-programm Zukunft Schweiz. Beurteilungsblatt für Gradu-iertenprogramme. Bern.Schweizerischer Nationalfonds (o. J.): SPP Informati-ons- und Kommunikationsstrukturen. Beurteilungsblatt fürGesuche. Bern.Schweizerischer Nationalfonds (o. J.): SPP Umwelt.Beurteilungsblatt für die Zwischenevaluation. Bern.Schweizerischer Nationalfonds (o. J.): SPP ZukunftSchweiz. Beurteilungsblatt für Skizzen und Gesuche. Bern.Schweizerischer Wissenschaftsrat (1995): CodexEthik Wissenschaftsevaluation. Beilage zur Grundsatzer-klärung des SWR vom 14. Dezember 1995. Bern.Schweizerischer Wissenschaftsrat (1995): Prinzipienzur Durchführung von Evaluationen im Bereich der Wis-senschaft. Grundsatzerklärung des SWR vom 14. Dezem-ber 1995. Bern.U.S. Department of Energy (1991): Office of EnergyResearch, Office of Program Analysis. Procedures for PeerReview Assessments. Executive Summary. o. O.

Bibl

iogr

aphy

and

Oth

er D

ocum

ents

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch


Inst

rum

ent tended to make a contribution towards

their solution. It comprises both fun-damental and applied research.

If the evaluation criteria are to be ap-plied to free interdisciplinary re-search, i.e., interdisciplinary researchnot carried out in the context of ori-ented research, and therefore not nec-essarily informed by societal issues,the criteria need to be adapted ac-cordingly. This is especially true forthe evaluation of objectives and ques-tions, or results and products, as wellas transfer of knowledge and technol-ogy (for example, «Presentation of thesocietal problem» could be rewordedas «Presentation of the scientific/technical problem»; also, in this case,the section «Involvement of users andexternal participants» might be omit-ted.

The basic premise is for the researchunder scrutiny to be structured as fol-lows: an overarching project unitingvarious sub-projects (or similar, e.g.,research groups). Sub-projects may bedisciplinary, inter or transdisciplinary.The object of evaluation is alwaysboth the overarching project and theindividual sub-projects. Catalogs ofcriteria, which have been harmonizedand are complementary, are thereforepresented for both categories. In par-ticular, it is possible to assess wheth-er the justification for sub-projects (orresearch groups) results from the over-arching project, and whether they areoriented towards it. In order to exam-ine «minimal disciplinary require-ments» in the case of interdisciplinarysub-projects, these projects will haveto be divided up and ascribed to theirrespective disciplines. Whether cer-tain disciplines are missing from anoverarching project will be determinedby examining whether any sub-projects required to reach the objec-tives of the overarching project aremissing.

The evaluation criteria have been for-mulated as questions. The suggest-ed catalogs of criteria are an attemptat presenting a comprehensive ques-tionnaire to evaluate those character-istics of inter and transdisciplinaryresearch that readily manifest them-selves to the outside viewer.

One unresolved issue is that of as-sessing the so-called «surplus

value» of an inter or transdisciplinaryproject. Strictly speaking, this can onlybe done in direct comparison with adisciplinary project focused on thesame subject. One possible solutioncould be to assess the justification foran inter or transdisciplinary study atthe beginning of the project (ex ante),and to examine at its conclusion (expost) whether this approach has beena success.

The present proposal is essentiallybased upon the peer review princi-ple, which is considered appropriatealso for the evaluation of inter andtransdisciplinary research. Most is-sues to be assessed are unquantifia-ble and will therefore require the re-viewers’ judgment. Accordingly, amechanistic view of evaluation, in thesense that the assessment of eachcriterion should always be the same,irrespective of the person carrying outthe evaluation, has been rejected asbeing unrealistic.

Documents:• Explanations• Evaluation Se-

quence (Table)• Catalogs of

criteria

Inter and TransdisciplinaryResearch:

Interdisciplinary research heredenotes the integration-orientedcooperation of scientists from atleast two disciplines with the aimto work on common questions andthe achievement of shared results.

Transdisciplinary research, inturn, here denotes interdiscipli-nary cooperation, involving notonly scientists but also practition-ers from beyond the realm of sci-ence (e.g., the users) in the re-search work.

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch Explanations

In the following, the parameters guid-ing the project will be outlined, aswell as the procedure to follow in anyevaluation, and explanations are givenas to how the catalogs of criteriashould be used and adapted. Moreo-ver, some terms, such as «interdiscipli-nary research», «transdisciplinary re-search», «external participants», «con-sensus» or «product», will be defined(see boxes).

Principles

Evaluation is used here in a verywide sense encompassing assessmentand controlling as well as monitoringin the sense of coaching; it comprisesboth self-evaluation and external eval-uation. These various forms of evalua-tion are not differentiated further; thiswill have to be done for each specificresearch program.

Transparency: Both the participantsin an evaluation and the evalueesthemselves must from the outset beinformed of the evaluation criteria (forexample, the evaluation criteria couldbe integrated into the program call). Ifcriteria were not pre-defined but areto be defined together with the scien-tists, it must be clear from the startwhich ones they are and how they willbe defined.

The premise in the development ofevaluation criteria was that of orient-ed inter and transdisciplinary re-search, i.e., research informed bypre-existing societal issues, and in-

What is the philosophy under-lying this Proposal?

The philosophy underlying thisProposal on the evaluation of interand transdisciplinary research isthat of classic LEGO: we wouldlike to put at your disposal thegreatest possible number of buil-ding blocks to «construct» a me-aningful evaluation of your rese-arch program.


What will be evaluated atwhat stage?

The following sequence is sug-gested:The evaluation of the research pro-posal should soon be followed by thefirst intermediary evaluation; abouthalf-way through the project, the sec-ond intermediary evaluation shouldtake place, and the third intermediaryevaluation towards the completion ofthe project. After completion, the finalevaluation should be carried out, andimpact should be assessed severalyears later.

Each one of these evaluations hasa specific objective:Evaluation of research proposal

(ex ante): evaluate the selection ofprojects to be granted.

First intermediary evaluation: as-sess whether the projects are oper-ative, i.e., whether work has begun,and whether the planned steps arebeing taken.

Second intermediary evaluation:assess whether the projects areproductive, i.e., wether the re-search schedules can be kept, andwhat changes might result as re-gards their objectives and ques-tions.

Third intermediary evaluation:asses whether there will be an out-put from the projects, i.e., whethertheir objectives will be reached,and the diffusion of results is wellon the way.

Final evaluation (ex post): assesswhether the projects have beensuccessful and their objectiveshave been reached.

Impact: Assess whether the researchwork has produced any/the desiredeffects.

Depending on the actual objective ofthe evaluation the focus is on spe-cific aspects while other aspectsare excluded from evaluation. Basical-ly, any specific aspect should only beassessed in depth once in the courseof a project. This is especially true forthe three intermediary evaluations,which are complementary. The table«Evaluation Sequence» gives an over-view of the various evaluation stagesand their focal points.

In the course of the evaluation a cer-tain shift of emphasis will occur.

The aspect «Scientific quality», forexample, will be virtually irrelevant inthe final evaluation, because this as-pect will now express itself chiefly inresults, products, publications, etc.,where it can be evaluated. Likewisethe aspect of «Integration/Synthesis»:if these processes were successful,joint results and products can be pre-sented, if not, they will be absent.Hence, it is unnecessary to evaluatethis aspect specially.

The present study is based upon theassumption that a project will run fora total of four years. If the duration islonger or shorter, the Evaluation Se-quence will have to be adapted asregards both the number and the tim-ing of evaluations, especially concern-ing the suggested intermediary evalu-ations (e.g., by combining the secondand third ones). The number ofevaluations and their timing will

generally have to be adapted to therequirements and parameters of aspecific research program. To improvethe overall quality of proposals, forexample, it might be useful to insertan evaluation prior to the actualevaluation of the research proposal(evaluation of project outlines), or toevaluate the research proposal intwo stages. For a particular programit may also make sense to staggerthe various parts of the final evalua-tion.

Ideally, the impact of research – i.e.,whether the work carried out is havingany/the desired effect on science andsociety, and whether the results andproducts have been accepted andadopted – should be studied approxi-mately no earlier than three to fiveyears after conclusion of the researchproject. No criteria for impact evalua-tion were formulated for the presentcatalog; impact has been added to thetable for the sake of completeness.Caution is required in assessing im-pact both as regards timing, makingsure the evaluation does not occur toosoon, and the fact that impact rarelydepends only on criteria that the sci-entists can influence.

Any evaluation needs to have conse-quences (e.g., rewards, incentives,recommendations, conditions, sanc-tions). The primary objective of anyevaluation is to ensure or enhancequality, hence the consequences needto be designed accordingly, especiallyin the intermediary evaluations. Con-sequences can only be more specifi-cally defined and described in the con-text of actual research programs, in-cluding their requirements and param-eters, and they need to be defined andcommunicated at the outset of everyresearch program and every evalua-tion. If a given consequence is formu-lated in terms of conditions, the sub-sequent evaluation will also need toexamine whether they have been met.The table «Evaluation Sequence» onlylists possible sanctions.

How Should the Catalogs ofCriteria Be Used?

The catalogs of criteria are to be con-sidered as a «pool» since not everyresearch program necessarily needs totake all questions into account. Con-text-related adaptations, deletionsand additions will have to be made,thereby creating specific catalogs ofcriteria for each specific research pro-gram. At the outset of such a program,then, the relevant questions to be ap-plied to a research program will haveto be selected from the «pool», andthe required adaptations will have tobe made.

No catalogs of criteria for interme-diary evaluations have been drawnup, since such catalogs (both for over-

Results, Products:

The term results denotes the out-come of the research process.

The term product denotes allthose things specially developedand designed for users (such asmanuals, apparatuses, courses,programs, itinerary exhibitions,analytical kits, etc.); their foremostcharacteristic being their practicalusability. Papers and the like, in-tended exclusively for the dissemi-nation of results, are not part ofthe term «product» but evaluatedseparately.

Inst

rum

ent

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch


arching and sub-projects) to be ap-plied to a specific research programcan be derived from the catalogs ofcriteria of this program. Some noteson intermediary evaluations have beenadded in lieu of catalogs of criteria.

It is essential to define certain crite-ria more precisely prior to applyingthem to any specific research pro-gram. This is especially true for crite-ria using elastic quantitative formula-tions, such as «sufficient», «a suffi-cient number», etc. It is necessary todefine early on the quantity that willbe considered «sufficient». Essentially,such a definition can be made in twoways, either in (a) plans of execution,etc., or (b) project-specifically, andinvolving the participating scientists(to be recorded in correspondingagreements). Which one of these pro-cedures (or which combination) is tobe chosen needs to be determined inthe context of the respective researchprogram.

The possible answers to the ques-tions in the catalogs of criteria areusually a simple «yes» or «no». Whenapplying the catalogs to a specificresearch program, however, this willoften prove to be inadequate since amore refined scale may be required fora more differentiated assessment(e.g., adding the category «partly»).Here, too, the most sensible range ofanswers will have to be determinedfor each specific research program.Similarly, additional questions such as

«If so, in what way?» have only beeninserted in the catalogs if they wereconsidered to be absolutely essential.The need for additional questions ofthis type to be asked of the evaluatorswill have to be determined for each

program. Where desirable, it may alsobe useful to reserve space for «Re-marks» so that assessments may bejustified, or to enable the free expres-sion of opinion on a project.

Because it wouldn’t make much senseto do so in a general manner, the cri-teria have not been weighted.Weighting the questions will have tobe decided for each specific researchprogram, i.e., which criteria are abso-lutely essential, and for which onesfulfilment will be rewarded while non-fulfilment will not be sanctioned, etc.Based upon the evaluators’ judgementof the weighted criteria, a final as-sessment (a «mark») will be given inevery assessment, both on the over-arching project and on all sub-projects.

Questionnaires to be answered bythe evaluators will have to be de-signed once the criteria have beenselected, defined more precisely andweighted, and once the types of an-swers have been determined.

At the outset, the basis of evalua-tion should be defined for each re-search program. It is therefore neces-sary to define the type of reporting

and documentation by the scientists(e.g., what the forms for progress re-ports, contracts with external partici-pants, etc., should look like) so thatthe evaluators have the required infor-mation to answer the relevant ques-tions. Which aspects shall be evaluat-ed based upon written documents,oral presentations, or site visits, etc.,will also need to be determined at thispoint.

Who carries out theevaluation?

Again at the outset of each researchprogram, the question of who willparticipate in the evaluation willneed to be settled. Participants maybe a group of experts, a steering com-mittee, external scientific reviewers,the management of a researchprogram, the management of theoverarching project, experts on issuesof organisational development, users,or the scientists involved in theprogram.

The question of who is in charge ofevaluating which aspects will alsohave to be settled for each specificresearch program. It is neither possi-ble nor necessary for everyone toevaluate everything. Rather, it will beuseful to define precisely who shouldassess which aspects at which stageof evaluation. For example, the «for-mal requirements» might most useful-ly be evaluated by the secretariat ofthe research promoting institution, thefirst intermediary evaluation by themanagement of the research program,and «project organisation/projectmanagement» by an expert in organi-sational development. It might also beuseful to entrust the management ofthe overarching project with the evalu-ation of certain aspects of the sub-projects, while users might more prof-itably assess the «presentation of thesocietal problem», or the «practicalimplementation». Last but not least, itmust be pointed out that any evalua-tion should always contain an elementof self-evaluation by the scientistsinvolved. The question of who evalu-ates what may be decided either inplans of execution, etc. or project-specifically, involving the participatingscientists (and recorded in agree-ments, e.g., between the group of ex-perts and the project management).

Consensus, Integration:

Consensus denotes both the process and the result of arriving at a sharedview of the problem, common objectives and questions, and a common ap-proach to dealing with them, as well as the development of a shared lan-guage. Therefore, consensus is here not intended to mean agreement orauthorisation, as in everyday language, but denotes the development ofshared methods, models and theories, as well as of a description of thesubject being studied that is equally valid for all, common definitions oftechnical terms, etc.

The term integration denotes the ongoing combination of the results ofindividual sub-projects to form a whole (a synonym used here is «synthe-sis»). Methods (a term used in a wide sense encompassing techniques,heuristic solutions, procedures, etc.) for consensus building and for integra-tion therefore serve to guarantee inter and transdisciplinary quality.

Inst

rum

ent


On the various groups of people involved:

Management of the overarching project: This is the team in charge ofmanaging the overarching project. It may consist of scientists as well aspractitioners – especially in transdisciplinary research.

Sub-project team: This is the team doing research on a sub-project(project-internal group). It may consist of scientists as well as practitioners– especially in transdisciplinary research.

External participants: This term denotes all the people who are neithermembers of the management of the overarching project nor of a team on asub-project, but who are nevertheless substantially involved in a project,e.g., in a monitoring group (project-external group). They may be scientistsas well as practitioners (e.g., citizens or neighbourhood residents, or repre-sentatives of organisations, associations, institutions or companies). Theterm does not apply to individuals subject to studies (in the context of sur-veys, observations, etc.).

Users: These are the groups of people towards whom the products andresearch results are targeted, or who shall use products and results in theirprofessional or everyday lives. While the users may be part of the manage-ment of the overarching project, or of a sub-project team, they may alsoparticipate as external participants.

Results and products at the level of the overarching project:

If the catalogs of criteria for the overarching project contain the question of whether a sufficient number of disserta-tions or theses has been written (transfer of knowledge and technology), this does not refer to the sum total of writ-ings at the level of the individual sub-projects. What it always does refer to is academic qualifications, events, publi-cations, etc., which were not done within individual sub-projects but resulted from the cooperation of the sub-projects or from the activities of the management of the overarching project. For example, this might be a disserta-tion written in the context of managing the overarching project, or resulting from a «Graduiertenkolleg» (GraduateProgram) on the subject of the overarching project, a paper by the management of the overarching project, or a jointpaper by some or all sub-projects. The same is true for the results and products at the level of the overarchingproject: far from being simply the sum total of the results and products of the sub-projects, they have been devel-oped in processes of consensus building and integration.

Inst

rum

ent

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch This may be particularly useful whenassessing the applicability of prod-ucts, etc. It will only rarely be possiblein each individual case to define theevaluator both in advance and at theprogram level.

Finally, it is essential to ensure thequality of evaluation. The demandsplaced on the assessors should not beunderestimated, both as regards theirexpertise and the input expected ofthem. When recruiting evaluators,therefore, these aspects will need tobe taken into serious consideration.The quality of the evaluation may alsobe enhanced by discussing with theevaluators, for example, the criteriathat should be brought to bear on aspecific program, or by an ongoingexchange on the evaluators’ experi-ences, or even by offering specialtraining to evaluators.


Eval

uatio

n of

rese

arch

prop

osal

(ex

ante

)Im

pact

Asse

ss w

heth

er th

ere

will

be a

n ou

tput

from

pro

ject

s

appr

ox. t

hree

to fi

ve ye

ars

afte

r com

plet

ion

of th

epr

ojec

tap

prox

. six

mon

ths

afte

r com

plet

ion

at le

ast s

ix m

onth

s prio

rto

com

plet

ion

appr

ox. h

alf-w

ay th

roug

hth

e pr

ojec

taf

ter s

ix m

onth

sup

on su

bmiss

ion

of th

ere

sear

ch p

ropo

sal

Whe

n do

es e

valu

atio

n ta

kepl

ace?

Cons

eque

nces

for d

esig

nof

rese

arch

pro

mot

ion

Prov

iso fo

r fut

ure

proj

ects

Cond

ition

s for

com

plet

ion

and/

or p

rovis

o fo

r fut

ure

proj

ects

Abor

tion

or co

ntin

uatio

nw

ith co

nditi

ons

Cont

inua

tion

with

cond

ition

sRe

fusa

l or r

etur

nfo

r rev

ision

Exam

ples

for s

anct

ions

Com

pete

nce

Qual

ifica

tion

Proj

ecto

rgan

isat

ion/

Proj

ect m

anag

emen

tIn

tern

al a

nd e

xter

nal

orga

nisa

tion

Tran

sfer

of k

now

ledg

ean

d te

chno

logy

Tran

sfer

of k

now

ledg

ean

d te

chno

logy

Scie

ntifi

c qu

ality

Inte

grat

ion/

Synt

hesi

sIn

tegr

atio

n/Sy

nthe

sis

Scie

ntifi

c qu

ality

Tran

sfer

of k

now

ledg

ean

d te

chno

logy

Tran

sfer

of k

now

ledg

ean

d te

chno

logy

Impa

ct o

n sc

ienc

e an

dso

ciet

y (e

.g.,

popu

latio

n,ec

onom

y, po

litic

s)

Cont

ents

/Obj

ectiv

esCo

nten

ts/O

bjec

tives

Resu

lts/P

rodu

cts

Resu

lts/P

rodu

cts

Form

al re

quire

men

tsW

hat a

re th

e fo

cal p

oint

s?

Wha

t is t

he o

bjec

tive

of th

eev

alua

tion?

Sele

ctio

n of

the

proj

ects

tobe

gra

nted

Asse

ss w

heth

er p

roje

cts

are

oper

ative

Asse

ss w

heth

er p

roje

cts

are

prod

uctiv

eAs

sess

whe

ther

pro

ject

sw

ere

succ

essf

ul a

nd h

ave

reac

hed

thei

r obj

ectiv

es

Asse

ss w

heth

er re

sear

chw

ork h

as p

rodu

ced

any/

the

desir

ed e

ffect

s

Inte

rmed

iary

eva

luat

ions

(est

imat

ed d

urat

ion:

4 y

ears

)Fi

nal e

valu

atio

n(e

x po

st)

Eval

uatio

n Se

quen

ce

Inst

rum

ent


A. Formal requirements

1. Requirements according to program call achieved not achieved

B. Contents/Objectives

Presentation of the societal problem

1. Has the problem been sufficiently described? yes no2. Has the relevance of the problem been convincingly presented? yes no3. Has a convincing case been made for the fact that only inter or transdisciplinary research can make

the promised contribution towards resolving the problem? yes no

Objectives and questions

4. Does it become sufficiently clear what the contribution towards resolving the problem is to be? yes no5. Do the objectives of the overarching project correspond with the objectives of the research program? yes no6. Do the scientific objectives follow from the problem-related objectives? yes no7. Do the questions serve the purpose of reaching the objectives and do they follow from those? yes no8. Are any essential sub-projects missing to reach the objective, and/or to answer the questions?

If so, which ones? yes no9. Do all the intended sub-projects follow from the objectives and questions of the overarching project?

If not, which ones do not? yes no10. Do the objectives of the overarching project appear to be attainable? yes no

Originality

11. Is the overarching project original?If so, in what way? If no, why not? yes no

Involvement of users and external participants

12. Were the users sufficiently involved in the wording of the objectives and questions? yes no13. Do the objectives and questions justify the participation of external participants? yes no14. Has the contribution to be made by external participants been presented with sufficient clarity? yes no

Results/Products

15. Have the results to be expected been presented clearly? yes no16. Do the results to be expected appear to be achievable? yes no17. What ideas are there on how products should be developed, and are they realistic? yes no18. Do the expected results and product ideas meet the objectives of the overarching project convincingly? yes no

C. Integration/Synthesis

1. Have the methods intended for consensus building and integration been presented clearly? yes no2. Do the methods intended for consensus building and integration appear to be suitable to achieve

the intended results and products? yes no

D. Scientific quality

1. Are the objectives of the overarching project based upon the current state of knowledge? yes no2. Does the research activity have internal logic? Does each step follow from the preceding one? yes no3. Has work in the sub-projects been harmonized? yes no4. Does the schedule appear to be realistic? yes no

Evaluation of Research Proposal (ex ante) – Overarching Project


E. Transfer of knowledge and technology (concept of implementation)

1. Has it been made sufficiently clear how the practical implementation of the results and productsshall be assured? yes no

2. Has it been made sufficiently clear how the results shall be transferred into (continuing) education? yes no3. Has it been made sufficiently clear how the results shall be disseminated? yes no4. Can the planned activities be realised with the available resources? yes no

F. Project organisation/Project management

Internal organisation

1. Are the tasks and competences clearly distributed (organisation chart, task specifications)? yes no2. Does the project structure (organisation chart, task distribution) appear to be suitable for consensus

building, integration and networking between the sub-projects? yes no

External organisation

3. Does the manner in which external participants are to participate appear to be appropriate tothe objectives of the overarching project? yes no

4. Are the contracts with external participants regarding the contribution expected from themsufficiently clear and binding? yes no

5. Is the procedure for establishing missing contacts clear? yes no

Infrastructure

6. Is the infrastructure required for the overarching project available? yes no

Costs/Benefits

7. What is the correlation between input and expected results and products? good fair poor8. Have the means applied for been sufficiently justified? yes no9. Do the means applied for appear to be sufficient? yes no10. Does the correlation of means applied for, own means, and means from third parties appear to

be adequate? yes no

G. Competence of the management of the overarching project

1. What is the extent of previous input (previous achievements) of the management of theoverarching project to the contents of the overarching project, and how does itscompetence rate? excellent sufficient insufficient

2. How does the management of the overarching project rate as regards theimplementation of the intended methods for consensus building and integration? excellent sufficient insufficient

3. How does the management of the overarching project rate as regards projectmanagement (previous input, education and training)? excellent sufficient insufficient

H. Overall assessment

1. What is the relevance of the overarching project? high medium low2. What is the quality of the overarching project? high medium low3. What is the likely success rate of the overarching project? high medium low


A. Formal requirements

1. Requirements according to program call achieved not achieved



1. Do the (problem-related and scientific) objectives follow from the objectives of the overarching project? yes no2. Do the questions and hypotheses serve the purpose of reaching the objectives? yes no3. Does it become sufficiently clear what the contribution of the sub-project towards the overarching

project is to be? yes no4. Has the relevance of the sub-project to reaching the objectives or to answering the questions of the

overarching project been presented convincingly? yes no5. Do the objectives of the sub-project appear to be attainable? yes no


6. Were the users sufficiently involved in the wording of the objectives and questions? yes no7. Do the objectives and questions justify the participation of external participants? yes no8. Has the contribution to be made by external participants been presented with sufficient clarity? yes no

Results/Products

9. Have the results to be expected been presented clearly? yes no10. Do the results to be expected appear to be achievable? yes no11. What ideas are there on how products should be developed, and are they realistic? yes no12. Do the results and product ideas to be expected meet the objectives of the overarching project

convincingly? yes no13. Do the expected results and product ideas meet the objectives of the sub-projects convincingly? yes no

C. Integration/Synthesis

1. Have sufficient resources been budgeted for the promised participation in the processes of theoverarching project (particularly consensus building and integration)? yes no


1. Are the objectives of the sub-project based upon the current state of knowledge? yes no2. Have the minimal disciplinary requirements (standards) been met? yes no3. Do the intended methods appear to be appropriate to reach the objectives of the sub-project, and to

answer its questions? yes no4. Does the research schedule appear to be consistent, and does each step follow from the preceding one? yes no5. Has the work been harmonized with that of the other sub-projects? yes no6. Does the research schedule appear realistic? yes no

Evaluation of Research Proposal (ex ante) – Sub-project


E. Transfer of knowledge and technology (concept of implementation)

1. Has it been made sufficiently clear how the practical implementation of results and products is to beassured? yes no

2. Has it been made sufficiently clear how the results shall be transferred into (continuing) education? yes no3. Has it been made sufficiently clear how the results shall be disseminated? yes no4. Can the planned activities be realised with the available resources? yes no


Internal organisation

1. Does the distribution of tasks in the project team appear to be suitable as regards participation inthe processes of the overarching project (especially consensus building and integration)? yes no

External organisation

2. Does the manner in which external participants are to participate appear to be appropriate to theobjectives of the sub-project? yes no

3. Are the contracts with external participants regarding the contribution expected from them sufficientlyclear and binding? yes no

4. Is the procedure for establishing missing contacts clear? yes no

Infrastructure

5. Is the infrastructure required for the sub-project available? yes no

Costs/Benefits

6. What is the correlation between input and expected results and products? good fair poor7. Have the means applied for been sufficiently justified? yes no8. Do the means applied for appear to be sufficient? yes no9. Does the correlation of means applied for, own means, and means from third parties appear to be

adequate? yes no

G. Competence of the sub-project team

1. What is the extent of previous input (previous achievements) of the sub-project teamto the contents of the sub-project, and how does their competence rate? excellent sufficient insufficient

2. How does the sub-project team rate as regards experience in cooperation in aproject group? excellent sufficient insufficient


1. What is the relevance of the sub-project to the overarching project? high medium low2. What is the quality of the sup-project? high medium low3. What is the likely success rate of the sub-project? high medium low


No catalogs of criteria for intermedi-ary evaluations have been drawn up,since such catalogs (both for over-arching and sub-projects) can be de-rived from the specific catalogs ofcriteria for the evaluation of the re-search proposal and/or the final eval-uation of a specific program.

Some indications are given below onhow to develop catalogs of criteriafor intermediary evaluation, and whatthe focus of each evaluation shouldbe.

appeared to be suitable (overarchingproject), and whether sufficient re-sources were budgeted for participa-tion in the processes of the overarch-ing project (sub-project), the pointnow is to find out whether implemen-tation of the respective methods hasactually begun, and whether they ap-pear to be successful.

The aspect «Internal and externalorganisation» has an analogous aim.An example will show how criteria forintermediary evaluations can be derivedfrom the evaluation catalogs referringto the research proposal (see box).

Example for the Derivation of Questions (First Intermediary Evaluation):

The aspect «Internal and external organisation» examines, on the one hand, whether the internal structure of boththe overarching project and the sub-projects is conducive to good cooperation. On the other hand, it examineswhether cooperation with external individuals, organisations, etc., is structured and maintained in such a way thatthe objectives can be reached.

At the level of the overarching project, the aspect «Project organisation/Project management» in the evaluation ofthe research proposal examines the following points, among others:

Internal organisation1. Are the tasks and competences clearly distributed (organisation chart, task specifications)?2. Does the project structure (organisation chart, task distribution) appear to be suitable for consensus building, in-

tegration and networking between the sub-projects?

External organisation3. Does the manner in which external participants are to participate appear to be appropriate to the objectives of

the overarching project?4. Are the contracts with external participants regarding the contribution expected from them sufficiently clear and

binding?5. Is the procedure for establishing missing contacts clear?

Questions corresponding to the objective of the first intermediary evaluation might now be:

1. Has the project structure been realised according to the research proposal and/or according to the conditions inthe grant?If not, have the changes been convincingly justified?

2. Are the external participants being involved as planned?3. Are any missing contacts being established as planned, or have they been established in the meantime?4. If the answer is no for items 2 and/or 3:

Was the reaction adequate (adapting the research schedule, the objectives and questions, the expectedproducts, etc.)?

Notes on Intermediary Evaluations

First Intermediary Evaluation

• Objective:To assess whether the projects areoperative, i.e., whether work hasbegun, and the planned steps arebeing taken.

• Focal points:Integration/Synthesis;Internal and external organisation.

While the aspect «Integration/Syn-thesis» in the evaluation of the re-search proposal inquired whether themethods for consensus building andintegration were presented clearly and

Inst

rum

ent

Eval

uatio

n Cr

iteria

for I

nter

and

Tra

nsdi

scip

linar

y Re

sear

ch


Second IntermediaryEvaluation

• Objective:To assess whether the projects areproductive, i.e., whether the re-search schedules can be kept, andwhat changes might result as re-gards their objectives and questions.

• Focal points:Contents/Objectives;Scientific quality;Transfer of knowledge andtechnology.

As regards «Contents/Objectives»,it is now possible, based upon inter-mediary results, to see whether anychanges in objectives and questionsare required for the remainder of theproject, be that for the overarchingproject or for individual sub-projects.Questions such as the following needto be asked: Are the intermediary re-sultats convincing as regards objec-tives and questions? Does it appearlikely that the contribution towardsresolving the problem will be made?Have the ideas on product develop-ment been specified further? Are theexternal participants making their con-tribution? It should also be possible toassess whether it might be useful tocomplement the overarching projectby additional sub-projects in order toreach the objectives or answer thequestions, e.g., to take into accountchanges in the parameters or newinsights.

The aspect «Scientific quality» askswhether procedures are «state of theart», whether the planned steps havebeen taken, and whether any changesto the research schedule are convinc-ing, as well as whether the methodsof consensus building and integrationare being implemented appropriately,and whether intermediary resultsmeet the minimal disciplinary require-ments (standards).

The aspect «Transfer of knowledgeand technology» in the evaluation ofthe research proposal inquires into theconcept of implementation (as regardspractical implementation, educationand continuing education, publication,etc.). However, the question now iswhether this concept has been ade-quately adapted and specified, e.g.,whether steps are being taken to pro-mote the reception of the results.

Third IntermediaryEvaluation

• Objective:To assess whether there will be anoutput from the projects, i.e.,whether their objectives will bereached, and the diffusion of resultsis well on the way.

• Focal points:Results/Products;Transfer of knowledge andtechnology.

«Results/Products» and «Transferof knowledge and technology» arenow at the center of attention: thequestion here is whether the availableresults and products, both preliminaryand to be expected, correspond to the(possibly updated) research schedule,whether the steps outlined in the(possibly updated and further speci-fied) concept of implementation arebeing taken and appear to be success-ful, and whether the project is likely tobe completed in time. One possiblequestion might be whether the trans-fer of results into (continuing) educa-tion has been assured, or whether thetreatment of the results for a wideraudience has begun, and whethersteps have been taken to promote theuse of the products by the target audi-ence. As regards the adaptation of thethird intermediary evaluation, the cri-teria for the final evaluation are morerelevant than those for the evaluationof the research proposal. ◆

Inst

rum

ent



Results/Products

1. Do the products meet the objectives of the overarching project convincingly? yes no2. Do the results meet the objectives of the overarching project convincingly? yes no3. Has more been achieved than would have been the case by autonomous (sub-) projects?

If so, the surplus value consists in: yes no


4. Has the promised contribution been made to the resolution of the societal problem? Have theproblem-related objectives been met? yes no

5. Has an additional contribution been made beyond that which was promised? yes no6. Have the scientific objectives been met, the questions answered? yes no7. Have the problem-related societal changes been considered adequately? yes no8. Has the assumption been confirmed that the contribution could be made only by means of inter and/or

transdisciplinary research?If so, in what way? If not, why not? yes no


9. Were users sufficiently involved in the development of the products? yes no10. Did the external participants make the expected contributions? yes no

E. Transfer of knowledge and technology

Practical implementation

1. Have the products been designed in such a way that the users can actually use them? yes no2. Have adequate steps been taken to promote the implementation of the products? yes no3. Are the products already being used? yes no4. Have the results been treated in such a way that reception by the users is likely? yes no5. Have adequate steps been taken to promote the reception of the results? yes no6. Are the results already being received by their audience? yes no

Education, continuing education

7. Are the results being transferred to a sufficient number of courses for graduate and diploma studies, etc.? yes no8. Is the teaching of the results in courses for graduate and diploma studies, etc., assured for a sufficient

time period? yes no9. Are the results being sufficiently transferred in PhD programs, and the like? yes no10. Has a sufficient number of diploma and graduate papers been written? yes no11. Has a sufficient number of PhD theses been written? yes no12. Has a sufficient number of habilitation theses been written? yes no13. Are the results being transferred to a sufficient number of courses in continuing education? yes no14. Is the teaching of the results in continuing education programs assured for a sufficient time period? yes no

Final Evaluation (ex post) – Overarching Project


Dissemination

15. Is there a sufficient number of scientific publications (e.g., specialised and interdisciplinary journals,books, electronic media)? yes no

16. Has the overarching project been presented at a sufficient number of scientific meetings? yes no17. Is there a sufficient number of publications for the target audience (users) of the overarching project

(e.g., specialised journals, newsletters of associations, books, electronic media, brochures)? yes no18. Is there a sufficient number of publications for a wider audience (e.g., newspapers, journals, magazines,

radio and TV, books, electronic media)? yes no19. Has the overarching project been presented at a sufficient number of symposia, public events, etc.,

for the target audience of the overarching project, and/or for a wider audience? yes no


Costs/Benefits

1. What is the correlation between input and present results and products? good fair poor

G. Qualification of the management of the overarching project

1. Was the research schedule adhered to, and/or was it adapted with the required flexibility? yes no2. Did the processes of consensus building and integration prove to be successful? yes no3. Was the project management successful overall? yes no4. Did the internal organisation prove to be successful? yes no5. Did the external organisation prove to be successful? yes no6. Did the involvement of external participants succeed? yes no


1. What is the relevance of the contribution of the overarching project? high medium low2. What is the quality of the results and products of the overarching project? high medium low3. What is the success of the overarching project? great medium slight



Results/Products

1. Do the products meet the objectives of the overarching project convincingly? yes no2. Do the products meet the objectives of the sub-project convincingly? yes no3. Do the results meet the objectives of the overarching project convincingly? yes no4. Do the results meet the objectives of the sub-project convincingly? yes no


5. Has the promised contribution been made to the overarching project? yes no6. Has an additional contribution been made beyond that which was promised? yes no7. Have the (problem-related and scientific) objectives been met? yes no8. Have the questions been answered, the hypotheses tested? yes no9. Have problem-related societal changes been considered adequately? yes no


10. Were users sufficiently involved in the development of the products? yes no11. Did the external participants make the expected contributions? yes no


1. Have the minimal disciplinary requirements (standards) been met? yes no

E. Transfer of knowledge and technology

Practical implementation

1. Have the products been designed in such a way that the users can actually use them? yes no2. Have adequate steps been taken to promote the implementation of the products? yes no3. Are the products already being used? yes no4. Have the results been treated in such a way that reception by the users is likely? yes no5. Have adequate steps been taken to promote the reception of the results? yes no6. Are the results already being received by their audience? yes no

Education, continuing education

7. Are the results being transferred to a sufficient number of courses for graduate and diploma studies, etc.? yes no8. Is the teaching of the results in courses for graduate and diploma studies, etc., assured for a sufficient

time period? yes no9. Are the results being sufficiently transferred in PhD programs, and the like? yes no10. Has a sufficient number of diploma and graduate papers been written? yes no11. Has a sufficient number of PhD theses been written? yes no12. Has a sufficient number of habilitation theses been written? yes no13. Are the results being transferred to a sufficient number of courses in continuing education? yes no14. Is the teaching of the results in continuing education programs assured for a sufficient time period? yes no

Final Evaluation (ex post) – Sub-project


Dissemination

15. Is there a sufficient number of scientific publications (e.g., specialised and interdisciplinary journals,books, electronic media)? yes no

16. Has the sub-project been presented at a sufficient number of scientific meetings? yes no17. Is there a sufficient number of publications for the target audience (users) of the sub-project

(e.g., specialised journals, newsletters of associations, books, electronic media, brochures)? yes no18. Is there a sufficient number of publications for a wider audience (e.g., newspapers, journals, magazines,

radio and TV, books, electronic media)? yes no19. Has the sub-project been presented at a sufficient number of symposia, public events, etc., for the

target audience of the sub-project, and/or for a wider audience? yes no


Costs/Benefits

1. What is the correlation between input and present results and products? good fair poor

G. Qualification of the sub-project team

1. Was the research schedule adhered to, and/or was it adapted with the required flexibility? yes no2. Did the sub-project participate adequately in the processes of the overarching project (esp. consensus

building, integration)? yes no3. Did the external organisation prove to be successful? yes no4. Did the involvement of external participants succeed? yes no


1. What is the relevance of the contribution to the overarching project? high medium low2. What is the quality of the results and products of the sub-project? high medium low3. What is the success of the sub-project? great medium slight


Inte

rnat

iona

l Tra

nsdi

scip

linar

ity C

onfe

renc

e

❑ The programme management periodically issues the bulletin «Panorama» toinform on the progress and results of the SPP Environment. There is no sub-scription charge.

Name: _______________________________

Address: _______________________________

_______________________________

_______________________________

Please send to:Schwerpunktprogramm UmweltProgrammleitungLänggassstrasse 23CH-3012 Bern, Fax +41 31 307 25 26

ImpressumNewsletter of the Swiss PriorityProgramme (SPP) «Environment»,Swiss National Science Foundation:Special issue 1/99 (July 1999)

Editor:Programme MangementSPP «Environment»Dr. sc.techn. Rudolf HäberliLänggassstrasse 23CH-3012 BernTel. +41 31 307 25 25Fax + 41 31 307 25 26E-Mail: [email protected]: http://www.snf.ch/spp_umwelt/overview.html

Editing/Layout:mediatec Landolt, Bernard Landolt,Worb; Walter Grossenbacher-Mansuy,Bern; Rico Defila, Antonietta Di Giulio,Bern

Cartoon front page:Sylvia Vananderoye, Bern

Translation:Margret Powell-Joss, Bern

Printing and dispatching:Graf-Lehmann AG, Bern

Transdisciplinarity:Joint problem-solvingamong Science, Technologyand Society

February 27, 28, 29 andMarch 1, 2000

Swiss Federal Instituteof Technology, Zurich,Switzerland

TransdisciplinarityConference

Goals• Developing transdisciplinary practice• Promoting transdisciplinary research• Creating favourable institutional structures and powerful incentives for

transdisciplinarity

Invited to contribute and to participate are:• Scientists and Researchers• Politicians and public administrators, university managers• Opinion-formers• Media representatives

Swiss National Science Foundation, Berne – Swiss PriorityProgramme EnvironmentSwiss Federal Institut of Technology, Zurich – Natural and SocialScience Interface (ETH-UNS)Asea Brown Boveri – ABB Corporate Research Ltd,Baden-Dättwil

With the patronage of UNESCOIn cooperation with the Swiss Foundation Science et Cité

E-mail: [email protected]: http://www.snf.ch/transdisciplinarity/home.html

Man

agin

g co

mpl

exity

:

An e

ffect

ive

way

to m

eet t

he c

halle

nges

of t

he 2

1st c

entu

ry International