DOI: 10.4018/IJICTE.2021010101

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

Copyright©2021,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.

1

Powerpoint Presentation Evaluation Based on Aggregation of Quality CriteriaSergey Sakulin, Bauman Moscow State Technical University, Russia

https://orcid.org/0000-0001-9218-9725

Alexander Alfimtsev, Bauman Moscow State Technical University, Russia

https://orcid.org/0000-0002-3805-4499

Dmitry Sokolov, Bauman Moscow State Technical University, Russia

ABSTRACT

Today, there is no consensus about proper timing and conditions for integration of PowerPointpresentationsintotheeducationalprocess.Butthemodel-basedevaluationcanmakeadecision-makingprocesseasierwhenitcomestousingpresentations.Thepurposeofthisstudyistobuildaformalmodeltoevaluatepresentations.Inordertobuildaformalmodel,theauthorssuggestemployinghierarchicalstructureconsistingofaggregationoperators,suchastheweightedaveragingoperator,minimumoperator,andfuzzyChoquetintegral.Theproposedformalmodelshowsexperts’knowledgeoftheinterdependenciesbetweenthecriteria.Theexperimentdescribedinthepaperdemonstratestheeffectivenessofsuchamodelasitallowstoformalizeexpertpreferencesgradually,whichmayincludeinterdependenciesbetweenthequalitycriteriaofapresentation.Thus,thismodelwillallowtostore,analyze,andcomparepresentationspropertiesthatarenecessaryfortheirsuccessfulapplication.

KEywoRdSAggregation Operator, Choquet Integral, Partial Data, Preference Relations, Presentation Evaluation, Quality Criteria

1. INTRodUCTIoN

MSPowerPoint(PP)asaninstrumentwasinitiallydesignedforMacintoshin1984andthenitwaspurchasedbyMicrosoft(Gaskins,2012).Overtheyears,thenumberofeducatorswhousePowerPointpresentations(PPP)forteachinghasbecomeoverwhelmingbecausePPhasbeenthemostfamousmeansofmakinguppresentations.Indeed,PPisinstalledonmorethan1billioncomputers(Brock,2017).

However,noconsensushasbeenreachedifpresentationsareworthemployingineducationalprocess.

Thisarticle,publishedasanOpenAccessarticleonJanuary7,2021inthegoldOpenAccessjournal,InternationalJournalofInformationandCommunicationTechnologyEducation(convertedtogoldOpenAccessJanuary1,2021),isdistributedunderthetermsoftheCreativeCommonsAttributionLicense(http://creativecommons.org/licenses/by/4.0/)whichpermitsunrestricteduse,distribution,andproductionin

anymedium,providedtheauthoroftheoriginalworkandoriginalpublicationsourceareproperlycredited.

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

2

Ontheonehand,(Fedisson&Braidic,2007)describesanexperimentdedicatedtoevaluationofPPPeffectiveness.TheresultsofthisexperimentprovethattheuseofPPPincreasesthestudents’levelofcomfortand,subsequently,improvestheirperformance.

Ontheotherhand,(Kedareetal.,2019)describedanexperimentwherestudentswhocompletedthecourseusingthetraditionalteachingmethod(chalk,blackboard,lecture)showedbetterresultscomparedtotheircounterpartswhousedPowerPointpresentations.Thisstudyconfirmstheideathatconventionallecturecanbeevaluatedasmorecomprehensibleandinformative,sincesuchlecturesmaycreateinteractivelearningenvironmentsinwhichpiecesofslidesareexcluded(Cosgun,2017).

Recentmeta-analysis (Bakeretal.,2018)hasshownthat therewashardlyanydifference inlearningresultswhenemployingPPPornot.In(Kernbachetal.,2015)identifiessomedeterrentmeasuresforPPP.Inparticular,fulllengthsentencesarehardtobeputonthePPPslidestoholdthesence(Farkas,2009)andunderstanding(Yates&Orlikowski,2007).Popularbulletlistsfosteranillusionofclarityanddonotshowthewholepicture(Gabriel,2008).Meanwhile,(Jenkins,2012)studiesthedebatesabouttheneedtoapplyPPP.

Therefore, instead of contemplating on whether to use this tool, it appears that we shouldconcentrateonhowandwhenitshouldbeusedtohelpstudentsinthelearningprocess(Jourdan&Papp,2013).Besidesthesetwoimportantissues,theissueofwhatkindofpresentationitshouldbetohelpstudentsinlearningthebestwayisnottheleastimportant.

Forexample,PowerPointcanbeverybeneficial,but thematerial that isnotpertinenttothelecturesubjectisharmfultostudentslearning(Bartsch&Cobern,2003).Andtheriseofformovercontentspoilsthepresentation(Grech,2018).ThiscorrespondstoMayermultimedialearningtheory(Mayer,2002).In(Hallewell&Crook,2019)thelecturesstylesisanalyzedanddrawsaconclusionthatthepresentationofthematerialshouldbeconsistentandcoherentdespitetheindividualstylesoflecturers.AlsotheeducatorsmustusePowerPointpresentationsinsuchawaythatstudentsunderstandthatthepresentationissupplementarytoclassattendance,notareplacementforit(Crawley&Frey,2008).OnewaytotackletheissueistocreateinteractivePowerPointpresentations(Boyas,2008).

ThereisawiderangeofstudiesofPPPapplicationsinvariousareas:physics(Erdemir,2011),medicine(Grech,2018);(Bamne&Bamne,2016),accounting(Sugahara&Boland,2006),astronomy(Miller&James,2011),sociology(Hilletal.,2012),educationofforeignlanguagestudents(Gordani&Khajavi,2019).EachofthedisciplinesneedsPPPtomeetitsparticularrequirementstomakestudentslearninthemosteffectiveway(Garrett,2016).

TofindoutpossiblewaystoimprovePPP,thesoftwareengineerscanbeguidedbyscientificresearchcarriedout invarious fieldsof activity,on theonehand, andby the students feedback(Williamsetal.,2016)andstudentssurveys(Cullenetal.,2018),(Szabo&Hastings,2000)ontheotherhand.

SuchsurveysareusuallycarriedoutamongstudentstofindouttheiropinionaboutacertainPPP(Appersonetal.,2008).TheresultsofsuchsurveysarethePPPqualitycriteriavaluesaveragedoverthestudents.ThatcriteriacorrespondtovariousPPPqualities.In(Basturk,2008),theauthorliststhefollowinggroupsofqualitycriteriaforapresentation:“content”and“design”.Thesegroupscorrespondtothesamecompositecriteria,which,inturn,aretheresultsoftheconvolutionofcriteriaintherespectivegroups.TocomparePPPandgivepreferencetooneofthem,thelessformalizedexpertsevaluationisapplied(Appersonetal.,2008,Akella,2017,Bridges&Luks,2016).

However,suchanapproachisconstrainedbythefactthatitdoesnotletsomeoftheexpertsseeclearlyandinformaldetailsthereasoningoftheotherexperts.Moreover,theapplicationofsimpleaverageforaggregationofcertainPPPqualitycriteria,thereisnochancetotakeintoaccounttheirpossiblemutualinfluencethat,initsturn,makestheevaluationofPPPinexact.

ThisbringsustothecryingtaskofdevelopingaformalapproachtoevaluatePPPsthatwouldallowtheidentificationoftheexpertsreasoningandtakingintoaccountpossiblemutualinfluenceofPPPqualitycriteria.

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

3

To make an educated decision backed up by experts’ opinion, this paper suggests a formalevaluationofpresentationsthatreliesonqualitycriteria.Themainprincipalissimilartotheevaluationofwebinterfacesonthebasisofusercriteria(Alfimtsevetal.,2016).Todevelopsuchamodel,itisnecessarytosetthecriteriaofpresentationqualityfirstandthen,withthehelpofexperts,setthecriteriaaggregationoperator.Afterthemodelisdeveloped,itcanbeusedfortheevaluationofaparticularpresentation.

Thispaperisorganizedasfollows.Section2describestheproposedapproachtodevelopingamodelforevaluatingPPPquality.Inparticular,insubsection2.1,thecriteriaforthequalityofPPParestated;insubsection2.2,theproposedformalmodelforevaluatingthepresentationisshown;insubsection2.3,astep-by-stepprocedureforevaluatingthePPPisdescribed.Section3containstheexperimentsandtheirresults.Inparticular,insubsection3.1,presentationsselectedfortheexperimentsaredescribed;insubsection3.2,theproposedevaluationprocedureisimplemented;subsection3.3interpretstheresults.Section4presentsconclusionsdrawnfromtheexperimentresultsregardingtheapplicationoftheproposedapproachandsuggestssomedirectionsforfurtherresearchofthisproblem.

2. PRoPoSEd APPRoACH

2.1. Criteria of Presentation QualityCriteriaforpresentationqualitycanbesetoutindifferentways.In(Appersonetal.,2008,Akella,2017,Bridges&Luks,2016),anumberofsimilarcriteriaarediscussed.

Inparticular,in(Bridges&Luks,2016),possiblereasoningofanexpertregardingthecontentofthepresentationisdescribed.Formalizationofsuchreasoning,wedenote“qualityofcontent”bythecriterionG

1. In turn, it canbeobtainedas a resultof aggregationof the followingcriteria:

g11—“breadthofconsideredproblem”,g

21—“focusontheproblem”,andg

31 —“qualityofthelistof

references”.Criteriag

11–g

31 areevaluatedbyanexpertbya10-pointscale.Ifthereisnolistofreferencesor

tableofcontents,thecriteria g21 andg

31 areimpossibletoevaluate.Inthiscase,thecontentquality

ofsuchapresentation isalsonotevaluated.Thecriterion g41 denotes thepresenceof the listof

references,and g51—thepresenceofthetableofcontents.Thesecriteriaarebinaryandtakevalues

fromtheset0,1.Theintroductionofthecriteria g41 and g

51 impliesthenecessityofatableof

contentsandlistofreferencesinagoodpresentation.Thequalityofthepresentationalsodependsonitsdesign.Thereareseveralqualitycriteriafor

design.Oneofsuchcriteriais“thedegreeofstructureofindividualslidesandthepresentationingeneral”(Akella,2017,Bridges&Luks,2016).In(Appersonetal.,2008),thepresenceofsoundina presentation was determined by students to be significant (69% of respondents preferred apresentationwithsoundtothesameonewithoutit).Otherdesignqualitycriteriaarefontparameters(size,charactercolor,colorcombination(contrast;background;colorchoices);qualityofanimation,graphs,andtables.Theabovecriteriawillbemergedintotwoabstractcriteria: g

12 —“design”(font

parameters,colorcombination,animationquality)and g22 —“presentationandslidestructure.”The

criteriag12 andg

22 arealsoevaluatedbyanexpertbya10-pointscale.Asinthecaseofthecriterion

G1“qualityofcontent,”thelackofintroductioninthepresentationwillmakeitimpossibletoevaluate

thequalityofthedesignduetoitsimpactonthepresentationstructure.Therefore,thepresenceofthe “presentation and slide structure” criterion g

22 makes sense only if the binary “presence of

introduction”criterion g32 is1.LetG

2denotethecompositecriterion“qualityofdesign.”Inturn,

G2willberesultingfromtheaggregationcriteriag

12 –g

32 .

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

4

Nowweshallconsiderthemutualinfluenceofthedenotedcriteria.ThecompositecriteriaG1

“qualityofcontent”andG2“qualityofdesign”canbeevaluatedseparatelysincethedesignand

contentofthesamepresentationcanbedonebydifferentpeople,anditwillnotaffectthequalityofthepresentation.CriteriawithinrelevantindependentG

1andG

2groupscaninfluenceeachother.

Considertheappropriaterelation.Thecriteria g11 “breadthofconsideredproblem”, g

21 “focuson

topic”,andg31 “qualityofthelistofreferences”correlatepositivelywitheachother,asitisobvious

thatthebreadthofconsideredproblemdirectlydependsonbothlistofreferencesusedandnotstrayingfromthetopic.Inaddition,thecriteriag

11 “breadthofconsideredproblem”andg

21 “focusontopic”

areinterdependentand,inasense,interchangeable.Meetingthecriterion g21 canhavealmostthe

sameeffectontheevaluationresultasthemeetingofboththecriteriag11 andg

21 duetothefactthat

ifthecontentofthepresentationcorrespondstothetopic,evenifthedesignatedproblemisnotfullyconsidered,thepresentationismostlikelygoodfromtheinformationalstandpoint.Thecriteria g

12

“design”andg22 “presentationandslidestructure”correlatepositively˗high-qualitydesigncanbe

achievedonlyifthepresentationfeaturesahigh-qualitystructure.Inthiscase,itislogicaltogivepreferencetog

22 overg

12 becausethestructureismoreimportantintermsofinformationperception

thandesignwhenitcomestopresentations.Theinfluenceofthebinarycriteria g41 ,g

51 and g

32 on

theothercriteriaisdescribedearlierinthispaper.

2.2. Formal Model for Presentation EvaluationTheargumentsconcerningthepresentationqualitycriteriafromtheprevioussectioncanbeformalizedusingthehierarchyofthecorrespondingaggregationoperators.Anaggregationoperatorisafunctionthatdependsoninputcriteriaandfeaturesspecifiedproperties(Beliakovetal.,2016).Thecriteriathemselvesandtheresultoftheaggregationaredeterminedandtakevaluesfromtheunitinterval[0,1],andtherefore,inthemodelofaformalqualityevaluationforpresentations,allthecriteriashouldbereducedtotheinterval[0,1].Theycanbereducedbydividingthecorrespondingratingsona10-pointscaleby10.InordertoobtaintheoverallevaluationofthepresentationqualityΩfromthevaluesoftheinitialcriteriag

11 ,…,g

51 ,g

12 ,…,g

32 ,theaggregationoperatorsshouldbechosen.

Theselectionofsuchoperatorsisnotanobvioustask.Thedifferencesbetweentheoperatorsandthemethodsofsettingoutthecorrespondinghierarchiesareduetothespecificsofthetasksbeingsolved.Theexamplesofsuchapplicationareintegratedevaluationofstudentperformance(Magadum&Bapat,2018,Maneetal.,2014),engineeringriskassessment(Rozhok&Tatarinov,2019),evaluationofthetechnologicalprocessstate(Sakulinetal.,2017),estimationofweakclassifiersinpatternrecognition(Samorodov,2016)orevaluationinforensicmedicine(Andersonetal.,2010).

Themostcommonandrelativelysimpleoperatorsareminimum(min),maximum(max),andweightedaveragingoperators.However,usingonlythemmakesitimpossibletotakeintoaccountanyinteractionbetweenthecriteria,includingcorrelationandinterchangeabilitydiscussedearlier.Inordertoformalizeexpertknowledgeofdependentcriteria,discreteChoquetandSugenointegralsareused(Samorodov,2011).ThemaindifferencebetweenthetwoisthattheChoquetintegralisbasedonlinearoperators,whiletheSugenointegralisbasedonnonlinearoperators(minandmax).TheChoquetintegralissuitableforquantitativeaggregation,whiletheSugenointegralismoresuitableforordinalaggregation.Thequalitycriteriaofthepresentationarequantitative,andthechangeineachofthemshouldinfluencetheresultoftheevaluation.Therefore,theChoquetintegralwaschosenasanaggregationoperatorforthedependentcriteria.

InordertousetheChoquetintegral,theidentificationoffuzzymeasureisneeded.Fuzzymeasureexpressesthesubjectiveweightorimportanceofeachsubsetofcriteriaandcanbedefinedasfollows

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

5

(Grabischetal.,1998):Fuzzy(discrete)measureisafunctionµ : [ , ]2 0 1J → ,where2J isthesetofallsubsetsofthecriteriaindexsetJ H= ,..., 1 ,whichsatisfiesthefollowingconditions:

1) µ µ( ) , ( )∅ = =0 1J ;2) ∀ ⊆ ⊆ ⇒ ≤D B J D B D B, : ( ) ( )µ µ .

Thevalueoftheµ( )D measurecanbeinterpretedas“weight”orcombinedimpactofthecriteriaincludedintothesubsetD ofthecriteriaindexsetJ H= ,..., 1 .Intheconsideredproblem,thepresentationqualitycriteriaaretheChoquetintegralcriteria.

TheaboveconsideredinterdependenciesbetweenthecriteriacanbeformalizedbytheChoquetintegral(Marichal,2000).

Correlationisthebestknownofthedependenciesbetweencriteria.Twocriteria i j J, ∈ arepositively(negatively)correlatedifexpertscanobserveapositive(negative)correlationbetweenthecontributionsoftwocriteriatotheaggregationresult.Positivecorrelationwillbeexpressedbytheinequation I i j( , )< 0 ,butnegativeone–byI i j( , )> 0 .Incasethecriteriai and j donotcorrelate,wecometotheequation I i j( , )= 0 .

Substitutiveness(complementarity)isanothertypeofdependence.Theideaofformalizingthistype of dependencies using fuzzy measures was proposed by (Murofushi &Soneda, 1993). Forexample,ifwehavetwocriteria i j J, ∈ andcansupposethattheexpertbelievesthatmeetingonlyonecriterioncausesalmostthesameeffectasmeetingofboth.Heretheimportanceofapairofcriteriaisclosetotheimportanceofeachofthemindividually,evenifothercriteriaarepresent.Inthiscaseweseethatthecriteriai and j almostsubstitutiveorinterchangeable.Theinequationsliketheonesforexpressingpositiveandnegativecorrelationcriteriacanbeapplied toexpress thesedependenciesbyfuzzymeasuresandtheChoquetintegral.

Theidentificationiscomplicatedbythenecessitytosetameasurevalueforeachsubsetofthecriteriathatshowsexpertspreferences,whichisimpossibletoimplementinpractice(Grabisch,1997).Therefore,suchidentificationiscarriedoutusingvariousindirectmethods(Grabischetal.,2008).

TousetheChoquetintegral,wehavetoidentifyafuzzymeasureonthebasisofexpertknowledge.Thisidentificationiscomplicatedbyexponentialincreasingcomplexityinthesensethatitisnecessarytosetavalueoffuzzymeasureforeachsubsetofcriteria.Settingthevaluesofall2J coefficientsofthefuzzymeasureµ( ),D D J⊆ isverydifficultorevenimpossiblefortheexpert.Notethatevenincaseofthreecriteriatodeterminethefuzzymeasure,itisnecessarytoobtain2 83 = coefficients.DespitethiscomplexitytheChoquetintegralstillcanbeappliedinpractice.ForthisGrabischproposedtheconceptofκ -orderfuzzymeasureorκ -additivefuzzymeasure(Grabischetal.,2008).Thisorder κ canbelessthanthenumberofaggregatedcriteria, κ < =J H .Theessenceofthe κ -additivityconceptliesinsimplificationofthetaskoffuzzymeasuresdeterminationbyexcludingthedependenciesbetweenmorethan κ criteria.Accordingtothat,wechosethree-additiveChoquetintegral. To identify the fuzzy measure for the model, we chose the method of least variance(Kojadinovic,2007).Thismethodallowsustoobtainauniquefuzzymeasureornoneifitcontradictsexpert preferences; it also provides the objectivity as a result of aggregation in addition to thesubjectivityofanexpert’sopinion.

Let’sbuildupahierarchyintheformofatreeofaggregationoperatorstoevaluatethequalityofpresentations.Thechoiceofoperatorsthetreeconsistsofwillbebasedontheinfluenceofthecriteriaoneachotherandontheresultasawhole.

ThecompositecriteriaG1“qualityofcontent”andG

2“qualityofdesign”areindependentof

eachother.Forthatreason,weuseasimpleweightedaveragingoperatorAGG G G wG w G( , )1 2 1 1 2 2

= + ,

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

6

wherew1

andw2areweightsdeterminedbyanexpertdirectlyorindirectly.TheFishburn’sscheme

isapplicabletoobtainthevaluesw1

andw2(Fishburn,1970).

Duetothementionedinterchangeabilityofthecriteriag11 “breadthofconsideredproblem”and

g21 ”focusontopic,”wecanapplytheChoquetintegralfortheiraggregation.Atthesametime,itis

necessarythatwetakeintoaccounttheimpactofthecriteriag41 “presenceofthelistofreferences”

and g51 “presenceof introduction”on thecompositecriterionG

1 “qualityofcontent”described

earlier.Todothis,weusetheminimumoperator,whichwillreducethevalueofcriterionG1tozero

ifatleastoneofthecriteria g41 , g

51 takesazerovalue.Sincethecriteriafor g

12 “design”and g

22

“presentation and slide structure” are interdependent, the Choquet integral is applicable as anappropriateaggregationoperator.Sometimesexpertsstruggletoanswercertainquestionsfromthequestionnaire.Forthatreason,weintroducedtheoption“Ifinditdifficulttoanswerthisquestion”.Tocopewiththis,wewillapplytheKolmogorov-Nagumotheoreminourevaluationmodel(Nagumo,1930,Kolmogorov,1930).Accordingtothistheorem,ifthereisnovalueforanycriterion,itcanbereplacedbytheaveragevalueoftheremainingcriteria,whichwillslightlyaffecttheresult(Dujmović&Tré,2011).

ThewaywereceivedthecriterionG1,wewillobtainthecriterionG

2throughtheminimum

operator,whichwillnullify thecriterionvalueof criterionG2 if there isno introduction in the

presentation(nullificationoftheg32 “presenceofintroduction”criterion).Theresultinghierarchyof

theaggregationoperatorsisshowninFigure1.Inaccordancewiththehierarchyabove,theevaluationofthepresentationquality Ω canbe

expressedwiththeformula:

Ω = +w g g C g g g w C g g g1 4

151

11

21

31

2 12

22

32

1 2min[ , , ( , , )] min[ ( , ), ]µ µ (1)

Here, the Choquet integralsC g g gµ1 11

21

31( , , ) andC g gµ2 1

222( , ) together with the corresponding

minimumoperatorsareusedtocalculatethevaluesofthecompositecriteriaG1andG

2.

Figure 1. Hierarchy of aggregation operators

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

7

2.3. The Procedure for Assessing The Presentation QualityPresentationevaluationbasedontheproposedmodelconsistsofthefollowingsteps.

Step 1.Provideexpertswithselectedpresentationsandaskthemtorateeachpresentationbythescalefrom0to10accordingthequestionnaire:

1. Isthetopicofthepresentationfullyconsidered?2. Howrelevantisthecontenttothedesignatedtopic?3. Evaluatethereliabilityofthesources.4. Isthereanintroduction?(0—no,10—yes)5. Istherealistofreferences?(0—no,10—yes)6. Assessthedesign(graphics,soundelements,animation,colorcombination,fontreadability).7. Assessthestructureofthepresentationandindividualslides.8. Isthereatableofcontents?(0—no,10—yes)9. Whichismoreimportantforthepresentation,contentordesign?Orboth?10.Whichpresentationisbetterintermsofdesign?11.Whichpresentationisbetterintermsofcontent?

Ifanexpertfindsitdifficulttoanswerthequestion,theycanleavethefieldinthequestionnaireblank.

Step 2.Averagetheratingsobtainedintheprevioussteponquestions1–8byfindingthearithmeticmeanforeachcriterion.Iftherewereanyblanksinthequestionnaire,theaverageoftheexistingratingsfortherelevantcriteriashouldbeputthere.Theaveragedratingsaredividedby10forthemtofittheinterval[0,1].

Thestepshouldresultinobtainingthevaluesofthequalitycriteriag11 ,…,g

51 ,g

12 ,…,g

32 forthe

presentations.

Step 3.Averagetheanswerstoquestion9byselectingthemostcommonexpertresponse.Theresultofprocessingtheanswerstothisquestionwillbepresentedasaratiointhesetcontent,design.Forexample,content~designmeansthat,accordingtotheexperts,thepresentationcontentisjustasimportantasitsdesign.

Answerstoquestions10and11shouldbeaveragedbychoosingthebestpresentationbasedontherelevantexpertsratings.Theresultsofprocessingexperts’answerstothesequestionswillbepresentedasrelationshipsofpartialnon-rigorousorderbetweenpresentations.Forexample,g g

11

21 ,

where meansthat,accordingtotheexperts,thepresentationg2

isbetterthanthepresentationg1

intermsofcontent.

Step 4. Formalize the relationship between the criteria identified in Section 2.1 in the form ofcorrespondinginequalitiesusingtherepresentationoftheinteractionoftheChoquetintegral.

Step 5.Applytheentropymaximizationmethodtoidentifyfuzzymeasures µ µ1 2, basedonthe

resultsfromSteps3and4withtheKappalabpackage(Grabischetal.,2008).Obtainthevaluesoftheweightcoefficientsw

1andw

2withtheFishburnweightsscheme(Fishburn,1970).

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

8

Step 6.Applythevaluesfromtheprevioussteps g11 ,…,g

51 , g

12 ,…, g

32 andfuzzymeasuresµ µ

1 2,

toformula(1)toobtaintheresultintheformoftheoverallqualityevaluationsΩ foreachofthepresentationsusingtheKappalabpackage.

3. EXPERIMENTS ANd RESULTS

3.1. Presentations Used for The ExperimentsThe evaluation procedure was applied to two publicly available presentations. The first is anintroductorypresentationfromtheCSFcourseonbasicmachinelearningconcepts(Figure2)byStanfordUniversity(SU)(Re&Ma,2019).

Figure 2. SU presentation on machine learning

Figure 3. MIT presentation on computer thinking and data science

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

9

ThesecondpresentationisanintroductorylectureoncomputerthinkinganddatasciencebytheMassachusettsInstituteofTechnology(MIT),whichisdevotedtooptimizationproblems(Figure3)(Guttag,2019).

3.2. Implementation of The ProcedureWechose30first-yearpostgraduatestudentsfromtheUniversityasourexperts.

Step 1.Theexpertsreceivethequestionnaireandanswerthequestions.Step 2.Weaveragedtheobtainedanswerstoquestions1–8.Youmayfindtheaveragedresultsofthe

surveyintheformofnormalizedcriteriavaluesinTable1.

Bothpresentationshavecontent,introduction,andalistofreferences(unitvaluesinthetable).Thevalueofthecriterion g1

1 “breadthofconsideredproblem”isthesameforbothpresentations.Thetopicsareexploredtosomeextentinbothpresentations,buttherearesomeflawsaccordingtotheexperts.Themaindrawbackof theMITpresentation isexcessof information,while theSUpresentationsuffersfromthelackofit.Asforthecriterion g

2

1 “focusontopic,”theSUpresentationhasahigherratingastheinformationisveryconsistentwiththetopicthroughoutthepresentation.TheMITpresentation,ontheotherhand,hastheinformationthatisirrelevanttothemaintopic.ThelistofreferencesinMITpresentationcontainsalistofscientificandeducationalliterature.TheoneoftheSUpresentationincludesthelinkstoInternetresourcesthatmaynotbereliable.Therefore,thevalueofthecriteriong

3

1 “evaluationofthelistofreferences”inthepresentationofMITisgreater.FromTable1,itbecomesclearthatthecriterion g

1

2 “design”intheMITpresentationisofalesservaluethantheoneintheSUpresentationpossiblyduetoaninsufficientnumberofgraphicelements

Table 1. Averaged results of experts ratings

Criterion MIT (g1 ) SU (g2 )

g11 0.8 0.8

g21 0.8 0.9

g31 0.9 0.7

g41 1 1

g51 1 1

g12 0.7 0.8

g22 0.7 0.8

g32 1 1

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

10

(images,tables,graphs)intheMITpresentation.Thevalueofthecriterion g22 “presentationand

slidestructure”isequalforbothpresentations.

Step 3.Theanswerstoquestion9ofthequestionnairerevealedthatmostexpertsconsiderbothdesignandcontentequallyimportantforpresentations.Inourcase,whichisfairlysimple,theweightcoefficientsw1 andw2 canbeobtainedwiththeFishburnmethodasfollows:

w w1 2

0 5= = , (2)

Theanswerstoquestion10ofthequestionnaireshowthattheMITpresentationissuperiortotheSUoneintermsofthecriterionG1 “qualityofcontent”.Thisexpertspreferenceisexpressedasarelationbetweenthecorrespondingimplementations:

g g1

1

2

1 (3)

Here g11 denotestheimplementationofthecriteria g g

1

1

5

1,..., fortheMITpresentation,and

g2

1 —theimplementationofthesamecriteriafortheSUpresentation.Basedontheanswerstoquestion11ofthequestionnaire,wecanconcludethat,accordingtothemajorityofexperts,theSUpresentationhasbetterdesign(G2 )thantheMITpresentation.Thisexpertpreferenceisexpressedasarelationbetweenthecorrespondingimplementations:

g g1

2

2

2 (4)

Here g1

2 denotestheimplementationofthecriteria g g1

2

3

2,..., fortheMITpresentation,and

g2

1 —theimplementationofthesamecriteriafortheSUpresentation.

Step 4.TheinterdependenciesbetweenthecriteriafromSection2.1areformalized.Thepositivecorrelationofcriteria, g1

1 ,..., g3

1 ,canbeformalizedusingthesignoftheinteractionindexoftherelevantcriteria:

I1

1 2 3 0( , , ) (5)

Heretheindexµ1 indicatesthattheinteractionindex(5)referstotheintegralC g g gµ1

1

1

2

1

3

1( , , ) .

Thepositivecorrelationbetweenthe g1

2 “design”and g22 “presentationandslidestructure”criteria

isformalizedusingthesignoftheinteractionindexoftherelevantcriteria:

I2

1 2 0( , ) (6)

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

11

Heretheindex µ2 indicatesthattheinteractionindex(6)referstotheintegralC g gµ2

1

2

2

2( , ) .

Theexpertsreasoningstatesthat,whendeterminingthevalueofthecriterionG2 “qualityofdesign”,thecriteriong2

2 “presentationandslidestructure”ismoreimportantthanthecriteriong1

2 “design”.LetusformalizeitusingthecorrespondingShapleyindices:

2 2

1 2( ) ( ) (7)

Step 5.Basedon theresultsofSteps3and4, thefuzzymeasures µ1 and µ2 are identifiedasinequalities(2–7).Foridentification,weappliedamethodforminimizingthedispersionoffuzzymeasuresundergivenconstraints,whichisimplementedinaspecializedKappalabpackage.Inaccordancewiththismethod,preferencerelations(3,4)aretranslatedintoinequalities:

C C c

1 1

10 8 0 7 0 9 0 8 0 8 0 5( . , . , . ) ( . , . , . ) (8)

C C c

2 2

20 9 0 8 0 7 0 5( . , . ) ( . , . ) (9)

Here c

1 , c

2 areexpertsindifferencethresholds.Criteria g4

1 , g5

1 and g3

2 ,whenconvertingpreferencerelations(3,4)intoinequalities(8,9),arenotconsideredastheyhaveunitvaluesanddonotaffecttheresultsofaggregationintheformofvaluesofthecompositecriteriaG1 “qualityofcontent”andG2 “qualityofdesign”.

Inequalities(5–7)areconverted,respectively,intoinequalities:

1 1 2 3 1I I( , , ) (10)

1 1 2 2I I( , ) (11)

$ g $ g $( ) ( )2

2

1

22 (12)

Here I

1 , I

2 , $

2 , c

1 , c

2 areindifferencethresholdssetbytheexpertstoidentifyrelevantfuzzymeasures.Allinputcriteriaaredefinedwithin0,0.1,...,0.9,1,whiletherestrictionsonindifference thresholdvaluesareset topreventselectionofvaluesforwhich thefuzzymeasuresidentificationproblemdoesnothaveasolution.Basedonthis,thefollowingvaluesofthesethresholdswereset:

I1 0 005 , ,

I2 0 005 , ,

$2 0 02 , ,

c c1 2 0 055 , .

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

12

Theidentificationresultsintheformoftheparametersofthecorrespondingintegrals(interactionindicesandShapleyindices)areshowninTable2.Theproblemofweightcoefficientsw1 andw2 wasaddressedearlier(2).

InStep 6,basedonthevaluesofthecriteria g11 ,…, g

5

1 , g1

2 ,…, g3

2 andthefuzzymeasuresµ µ1 2, intheformofthecorrespondinginteractionindicesandShapleyindices,thevaluesofthe

ChoquetintegralsC g g gµ1

1

1

2

1

3

1( , , ) andC g gµ

21

2

2

2( , ) for g1 , g2 areobtained.Thesevaluesare

showninTable2alongwiththevaluesoftheoverallevaluationΩ forthereviewedpresentations.Letussupposethat,forsomereason,wecouldnotgetfromtheexpertsthevalueofthecriterion

g2

1 “focusontopic.”Inthiscase,inaccordancewiththeKolmogorov-Nagumotheorem,thevalueofthiscriterionisreplacedwiththeaveragevalueoftheremainingcriteriaforeachpresentation.TheresultsofaggregationforsuchareplacementareshowninTable3.

3.3. Interpretation of The Results

TheresultsobtainedduringtheexperimentsareshowninTable2.Theinteractionindex Iµ1

1 2( , )

takesthevalue−0.0050fromtheinterval[−1; I

1 ].Thisexpressestheinterdependenceofthecriteria g1

1 , g2

1 ,whichcorrespondstotheexpertsreasoning.TheremaininginteractionindicesforthecriteriarelatedtoC g g gµ

11

1

2

1

3

1( , , ) areequalto0.Asexpected,theresultsoftheaggregation

reflect the experts opinion formulated in (1–7). The obtained values of the Choquet integralsC g g gµ

11

1

2

1

3

1( , , ) andC g gµ

21

2

2

2( , ) ,giveninthetable,demonstratetheexpertspreferenceofthe

MITpresentationin termsof thecriterionG1 “qualityofcontent”andtheSUpresentationin

Table 2. The results of criteria aggregation

Φ( )1 Φ( )2 Φ( )3 Iµ1

1 2 3( , , ) Iµ2

1 2( , ) g1 g2

Cµ1 0.4975 0.4975 0.005 -0.005 – 0.799 0.7969997

Cµ2 0.5025 0.5025 – – -0.005 0.7 0.85025

Ω – – – – – 0.7495 0.8236248

Table 3. The results of criteria aggregation with partial absence of input data

g1 g2

Cµ10.8003502 0.7969998

Cµ20.7 0.85025

Ω 0.7501751 0.8236249

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

13

termsofthecriterionG2 “qualityofdesign.”Theresultsoftheaggregationalsoreflecttheexpertsopinionthatthehigherthequalityofthematerialpresented(criteria g1

1 and g2

1 ),thelowertheimportanceofthelistofreferencesevaluation(criterion g

3

1 ).Also,asexpected,theevaluationofthecompositecriterionG1 “qualityofcontent”ishigherfortheimplementationg

2.Overall,the

SUpresentationisevaluatedhigherthantheMITpresentation,whichisshowninthevaluesofthefinalevaluationΩ .

Theresultsintheabsenceofinformationfromtheexpertsona g2

1 criterionvalueareshowninTable3.TheoverallevaluationoftheSUpresentationremainedrelativelyhigherthanthatoftheMITpresentation.ThedifferenceinfinalevaluationΩ haschangedby4.9%.Thisconfirmsthattheproposedmodelremainssuccessfulandhealthyinthepartialabsenceofdatafromexperts.

Thus,alltheexpertsreasoningaboutthequalitycriteriaofpreferencesweretakencareoffortheformalmodelofpresentationevaluation.

4. dISCUSSIoN

Bothtechnologiesineducationalprocessandtechnologicalliteracyoflecturersontheirowndonotalwaysmean theireffectiveapplication.Their integrationand theeffectiveuse require lecturers’knowledgeandunderstandingofthetheoreticalbasisthatwouldjustifyandsupporttheapplicationoftechnologies(Yang&Skelcher,2019).

Notonly the theoretical basis for the applicationof technologies, but strongbeliefs andlong-heldstereotypesofacademicstaffinfluencetheapplicationofthetechnologies(Tondeureal.,2017).Moreover,itisnotonlyimportanttointroducethetechnologies,buttousetheminanintelligentway.Thereasonabledecisiontouseatechnologyintheeducationalprocess,informalorweaklyformalizedmethodsofanalysisareemployed.Forexample,(Page,2019)comparesandcontraststheuseoftactileanddigitalsketcheswhendesigning,(Costa,Teixeira&Alvelos,2018)analysesthecurrentapplicationofwholescalefreeonlinecoursesatuniversities,(Tafazoli,María&Abril,2019)analysestheconditionsofthesuccessfulintroductionofintelligentcomputer-assistedlanguagelearning,(Tuapawa,2017)bringsoutsomefeaturesofeducationalonlinetechnologiesapplication.

Arecenttendencytoconvergevariousfieldsofexpertisehasnotsparedsuchfieldsofscienceaspedagogyandeducation,ontheonehand,andformaltheoryofdecisionmaking,ontheotherhand.Therefore,thereareformalapproachestodecisionmakinginthefieldofeducation.Inparticular,(Debetal.,2019)discussestheapplicationoffuzzylogictofindpossiblegapsinteachingstudents;(Singh&Rawani,2019)discussesthefuzzyapproachtoratevariousexpectationsofstudentsoftechnicaluniversities;(Yassine,Battou&Omar,2019)describeshowtousefuzzyanalyticalhierarchyprocesstodefinetheweightsofopenonlinecoursestoincreasecompletionratesandtodecreasedropoutrates;(Bendjebar,Lafifi&Seridi,2016)employsfuzzylogicforaformalestimationoftutorprofilestoclassifyonlinetutors.Theresultsofthepracticalapplicationofsuchformalapproacheshaveshowntheeffectivenessandrepeatabilityofthedecisionsmade.

Atthesametime,theapplicationofsuchformalapproachesfacesthechallengesthatinvolvealackofmeanstoformalizepossibleinterconnectionsbetweenthecriteriatomakeadecisionaswellaspossiblelackofdata.Methodsofdecision-makingtheoryinpossibleaccountingforinterconnectionsbetweenthecriteriahasallowedustoformalizeexperts’preferenceswithhighaccuracyand‘finetuning’ofthemodel.OneofthemostpopularapplicationoftechnologiesineducationalprocessistheuseofPowerPoint.ThereforethemodelwasusedtoevaluatePPpresentations.Themodeltakesintoaccounttheinterconnectionsbetweenthecriteriaandnonsensitivetoalackofdata.

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

14

5. CoNCLUSIoN ANd RESEARCH PRoSPECTS

Recentachievementsinthetheoryofdecisionmakinghaveallowedustoformalizehumanreasoningwherethereisuncertaintyand(or)somedataismissed.TheformalevaluationofPPPisbridgedbysuchresearchareasaspedagogy,psychologyandcomputersciences.

ThepaperconsidersamodelofPPPevaluationonthebasisofhierarchyofaggregationoperatorsincludingtheChoquetintegrals.ThehierarchyistoleranttotheabsenceofinputdataduetotheapplicationoftheKolmogorov-Nagumoapproach.ThismodelallowsPPPevaluationevenifapartofexperts’evaluationsismissedandtakesintoaccounttheinterdependenciesbetweenthecriteriaaswell.TheresultsoftheexperimentsonPPPevaluationhaveshowedthatalltheexperts’preferencesweretakenintoaccountbythemodelbuiltbythesuggestedapproach.

Inaddition,suchanapproachfacesanumberofchallengessuchaschoiceofcertainqualitycriteriaofPPP,typeofhierarchyforaggregationoperators,choiceofaggregationoperatorsthemselves,parametersoftheaggregationoperators.Allthisdependsonthepersonalityoftheexpertinacertainappliedfieldofactivity,ontheirsubjectiveopinion.

Moreover,poorintuitiveunderstandingofsuchaggregationoperatorsastheChoquetintegralmakesexpertsemploysimplifiedmodelsthatdonotallowformalizationofinterdependenciesbetweenqualitycriteria.

Nevertheless,byincreasingrefinementitbecomespossibletodevelopformalmodelsforPPPevaluationinvariousfieldsofactivity.Iftheconstructionofthedescribedhierarchyisperformedbyseveralexpertswhomayhavedifferentopinionsaboutthesubject,theuseoftheconsideredapproachwillhelpthemfindacompromisesolution.Thisispossiblebecauseallconsideredtypesofexpertpreferencescanbepresentedexplicitlyandunambiguouslyintheformofthecorrespondingoperatorsincorporatedinthehierarchy.Theprocessofformalizingpresentationevaluationcanbeiterative,whichimpliesagradualrefinementofexpertspreferences.Aftersuchmodelsaredeveloped,itwillbecomepossibletostorethedataaboutthenecessarypropertiesofPPPinvariousfieldsofactivity.ThisinitsturnwillallowtoidentifythedifferencesmorecarefullybetweenPPPpropertiesapplyingthemtocertainareasofactivityandprovideessentialrecommendationsforPPPdesigners.

TheresearchplansinthisareaincludedevelopingsoftwarewithdeeplearningtoimprovetheexperienceofpresentationevaluationaswellastobuildingautomaticpatternrecognitionmodelstoevaluatePPP.Besides,toovercomethedifficultiestheexpertsfacewhenworkingwithaggregationoperators, additional research isneeded in the fieldofvisualizationof aggregationoperatorsbycognitivegraphicstohelptheexpertsinintuitiveunderstanding.

ACKNowLEdGMENT

ThisresearchwassupportedbyMinistryofScienceandHigherEducationoftheRussianFederation,R&DstateprojectNº2.5048.2017/8.9.

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

15

REFERENCES

Akella,D.(2017).Powerpoint vs. Recorded Lectures: An African American Perspective on Online Education.IGIGlobal.doi:10.4018/978-1-5225-0968-4.ch013

Alfimtsev,A.,Sakulin,S.,&Levanov,A.(2016).Formalizationofexpertknowledgeabouttheusabilityofwebpagesbasedonusercriteriaaggregation.International Journal of Software Innovation,4(3),38–50.doi:10.4018/IJSI.2016070103

Anderson,M.F.,Anderson,D.T.,&Wescott,D.J.(2010).EstimationofadultskeletalageatdeathusingtheSugenofuzzyintegral.American Journal of Physical Anthropology,142(1),30–41.PMID:19845027

Apperson,J.M.,Laws,E.L.,&Scepansky,J.A.(2008).AnassessmentofstudentpreferencesforPowerPointpresentation structure in undergraduate courses. Computers & Education, 50(1), 148–153. doi:10.1016/j.compedu.2006.04.003

Baker,J.P.,Goodboy,A.K.,Bowman,N.D.,&Wright,A.A.(2018).DoesteachingwithPowerPointincreasestudents’learning?Ameta-analysis.Computers & Education,126,376–387.doi:10.1016/j.compedu.2018.08.003

Bamne,S.N.,&Bamne,A.S.(2016).ComparativestudyofchalkboardteachingoverPowerPointteachingasateachingtoolinundergraduatemedicalteaching.International Journal of Medical Science and Public Health,5(12),2585–2587.doi:10.5455/ijmsph.2016.01072016532

Bartsch,R.A.,&Cobern,K.M.(2003).EffectivenessofPowerPointpresentationsinlectures.Computers & Education,41(1),77–86.doi:10.1016/S0360-1315(03)00027-7

Basturk,R.(2008).ApplyingthemanyfacetRaschmodeltoevaluatePowerPointpresentationperformanceinhighereducation.Assessment & Evaluation in Higher Education,33(4),431–444.doi:10.1080/02602930701562775

Beliakov, G., Sola, H. B., & Sánchez, T. C. (2016). A practical guide to averaging functions. Springer.doi:10.1007/978-3-319-24753-3

Bendjebar,S.,Lafifi,Y.,&Seridi,H.(2016).ModelingandEvaluatingTutors’FunctionusingDataMiningandFuzzyLogicTechniques.International Journal of Web-Based Learning and Teaching Technologies,11(2),39–60.doi:10.4018/IJWLTT.2016040103

Boyas,E.A.(2008).UsingPowerPointtoEncourageActiveLearning:ATooltoEnhanceStudentLearningintheFirstAccountingCourse.International Journal of Information and Communication Technology Education,4(2),14–25.doi:10.4018/jicte.2008040102

Bridges,T.M.,&Luks,A.M.(2016).How to Give a Great PowerPoint Presentation.Springer.doi:10.1007/978-3-319-33193-5_8

Brock, D. C. (2017). The improbable origins of Powerpoint. IEEE Spectrum, 54(11), 42–49. doi:10.1109/MSPEC.2017.8093800

CosgunÖgeyik,M.(2017).TheeffectivenessofPowerPointpresentationandconventionallectureonpedagogicalcontentknowledgeattainment.Innovations in Education and Teaching International,54(5),503–510.doi:10.1080/14703297.2016.1250663

Costa,C.,Teixeira,L.,&Alvelos,H.(2018).ExploringtheUsageofMOOCsinHigherEducationInstitutions:Characterization of the Most Used Platforms. International Journal of Information and Communication Technology Education,14(4),1–17.doi:10.4018/IJICTE.2018100101

Crawley, D. C., & Frey, B. A. (2008). Examining the relationship between course management systems,presentationsoftware,andstudentlearning:Anexploratoryfactoranalysis.International Journal of Information and Communication Technology Education,4(1),1–14.doi:10.4018/jicte.2008010101

Cullen,A.E.,Williams,J.L.,&McCarley,N.G.(2018).ConscientiousnessandLearningviaFeedbacktoIdentifyRelevantInformationonPowerPointSlides.North American Journal of Psychology,20(2),425–443.

Deb,K.,Banerjee,S.,Das,A.,&Bag,R.(2019).TutorialGapIdentificationTowardsStudentModelingUsingFuzzyLogic.International Journal of Information and Communication Technology Education,15(3),30–41.doi:10.4018/IJICTE.2019070103

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

16

Dujmović,J.,&Tré,G.D.(2011).Multicriteriamethodsandlogicaggregationinsuitabilitymaps.International Journal of Intelligent Systems,26(10),971–1001.doi:10.1002/int.20509

Erdemir,N. (2011).The effect ofPowerPoint and traditional lectureson students’ achievement inphysics.Journal of Turkish Science Education,8,176–189.

Farkas,D.K.(2009).ManagingthreemediationeffectsthatinfluencePowerPointdeckauthoring.Technical Communication (Washington),56,1–11.

Fedisson,M.,&Braidic,S.(2007).PowerPointpresentationsincreaseachievementandstudentattitudestowardstechnology. International Journal of Information and Communication Technology Education, 3(4), 64–75.doi:10.4018/jicte.2007100106

Fishburn,P.C.(1970).Utility theory for decision making.Wiley.doi:10.21236/AD0708563

Gabriel,Y.(2008).AgainstthetyrannyofPowerPoint:Technology-in-useandtechnologyabuse.Organization Studies,29(2),255–276.doi:10.1177/0170840607079536

Garrett, N. (2016). How Do Academic Disciplines Use PowerPoint? Innovative Higher Education, 41(5),365–380.doi:10.1007/s10755-016-9381-8

Gaskins,R.(2012).Sweating bullets: Notes about inventing PowerPoint.VinlandBooks.

Gordani,Y.,&Khajavi,Y.(2019).Theimpactsofmulti-modalPowerPointpresentationontheEFLstudents’contentknowledgeattainmentandretentionovertime.Education and Information Technologies,1–15.

Grabisch,M.(1997).K-orderadditivediscretefuzzymeasuresandtheirrepresentation.Fuzzy Sets and Systems,92(2),167–189.doi:10.1016/S0165-0114(97)00168-1

Grabisch,M.,Kojadinovic,I.,&Meyer,P.(2008).AreviewofmethodsforcapacityidentificationinChoquetintegralbasedmulti-attributeutility theory:Applicationsof theKappalabRpackage.European Journal of Operational Research,186(2),766–785.doi:10.1016/j.ejor.2007.02.025

Grabisch,M.,Orlovski,S.A.,&Yager,R.R.(1998).Fuzzy aggregation of numerical preferences.Springer.doi:10.1007/978-1-4615-5645-9_2

Grech,V.(2018).TheapplicationoftheMayermultimedialearningtheorytomedicalPowerPointslideshowpresentations.Journal of Visual Communication in Medicine,41(1),36–41.doi:10.1080/17453054.2017.1408400PMID:29381105

Grech,V.(2018).WASP(Writeascientificpaper):OptimisationofPowerPointpresentationsandskills.Early Human Development,125,53–56.doi:10.1016/j.earlhumdev.2018.06.006PMID:29929910

Guttag,J.(2019).Introduction and Optimization Problems.MassachusettsInstituteofTechnology.Retrievedfromhttps://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-0002-introduction-to-computational-thinking-and-data-science-fall-2016/lecture-slides-and-files/MIT6_0002F16_lec1.pdf

Hallewell, M. J., & Crook, C. (2019). Performing PowerPoint lectures: Examining the extent of slide-textintegrationintolecturers’spokenexpositions.Journal of Further and Higher Education,1–16.doi:10.1080/0309877X.2019.1579895

Hill, A., Arford, T., Lubitow, A., & Smollin, L. M. (2012). “I’m Ambivalent about It” The Dilemmas ofPowerPoint.Teaching Sociology,40(3),242–256.doi:10.1177/0092055X12444071

Jenkins, E. (2012). Death by PowerPoint? A Media Ecology Examination of the Debates Over Slideware.Explorations in Media Ecology,10(3-4),225–245.doi:10.1386/eme.10.3-4.225_1

Jourdan,L.,&Papp,R.(2013).PowerPoint:It’snot“Yes”or“No”–it’s“When”and“How”.Research in Higher Education,22,1–12.

Kedare, R. V., Kharat, R. D., & Wagh, R. J. (2019). Impact of PowerPoint and Chalkboard teaching inPhysiotherapy Undergraduates. International Journal of Clinical and Biomedical Research, 5(1), 9–11.doi:10.31878/ijcbr.2018.51.03

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

17

Kernbach, S., Bresciani, S., & Eppler, M. J. (2015). Slip-sliding-away: A review of the literature on theconstrainingqualitiesofPowerPoint.Business and Professional Communication Quarterly,78(3),292–313.doi:10.1177/2329490615595499

Kojadinovic,I.(2007).Minimumvariancecapacityidentification.European Journal of Operational Research,177(1),498–514.doi:10.1016/j.ejor.2005.10.059

Kolmogorov,A.N.(1930).Surlanotiondelamoyenne.Accad. Naz. Lincei Mem. Cl. Sci. Fis. Mat. Natur. Sez.,12,388–391.

Magadum,C.G.,&Bapat,M.S.(2018).RankingofStudentsforAdmissionProcessbyUsingChoquetIntegral.International Journal of Fuzzy Mathematical Archive,15(2),105–113.

Mane,A.M.,Dongale,T.D.,&Bapat,M.S.(2014).Applicationoffuzzymeasureandfuzzyintegralinstudentsfailuredecisionmaking.IOSR Journal of Mathematics,10(6),47–53.doi:10.9790/5728-10634753

Marichal,J.-L.(2000).AnaxiomaticapproachtothediscreteChoquetintegralasatooltoaggregateinteractingcriteria.IEEE Transactions on Fuzzy Systems,8(6),800–807.doi:10.1109/91.890347

Mayer,R.E.(2002).Multimedialearning.Psychology of Learning and Motivation,41,85–139.doi:10.1016/S0079-7421(02)80005-6

Miller,S.T.,&James,C.R.(2011).TheeffectofanimationswithinPowerPointpresentationsonlearningintroductoryastronomy.Astronomy Education Review,10(1),1–13.doi:10.3847/AER2010041

Murofushi,T.,&Soneda,S.(1993).Techniquesforreadingfuzzymeasures(III):interactionindex.9th Fuzzy System Symposium,693–696.

Nagumo,M.(1930).Übereineklassedermittelwerte.Japanese Journal of Mathematics: Transactions and Abstracts,7,71-79.

Page,T.(2019).AComparisonofHapticSketchingandDigitalSketching:ConsiderationsofFinalYearDesignStudents.International Journal of Information and Communication Technology Education,15(2),146–161.doi:10.4018/IJICTE.2019040109

Re,C.,&Ma,T. (2019).Machine Learning.StanfordUniversity.Retrievedfromhttp://cs229.stanford.edu/notes-spring2019/lecture1_slide.pdf

Rozhok,A.,&Tatarinov,V.(2019).Theapplicationofdynamicriskanalysismethodsforsafetyimprovementonhighwayslocatednearorontheterritoryofhydraulicengineeringdams.IOP Conference Series. Materials Science and Engineering,492(1),1–6.doi:10.1088/1757-899X/492/1/012005

Sakulin, S. A., & Alfimtsev, A. N. (2017). Data fusion based on the fuzzy integral: model, methods and applications.NovaSciencePublishers.

Samorodov,A.V.(2011).Applicationofafuzzyintegralforweakclassifiersboosting.Pattern Recognition and Image Analysis,21(2),206–210.doi:10.1134/S1054661811020945

Samorodov,A.V.(2016).Methodofweakclassifiersfuzzyboosting:Iterativelearningofquasi-linearalgorithmiccomposition.Pattern Recognition and Image Analysis,26(2),266–273.doi:10.1134/S105466181602019X

Singh, A. K., & Rawani, A. M. (2019). A Fuzzy Approach for Ranking of Student’s Expectation From aTechnicalInstitute.International Journal of Smart Education and Urban Society,10(2),41–52.doi:10.4018/IJSEUS.2019040103

Sugahara,S.,&Boland,G.(2006).TheeffectivenessofPowerPointpresentationsintheaccountingclassroom.Accounting Education,15(4),391–403.doi:10.1080/09639280601011099

Szabo,A.,&Hastings,N.(2000).UsingITintheundergraduateclassroom:ShouldwereplacetheblackboardwithPowerPoint?Computers & Education,35(3),175–187.doi:10.1016/S0360-1315(00)00030-0

Tafazoli,D.,María,E.G.,&Abril,C.A.H.(2019).IntelligentLanguageTutoringSystem:IntegratingIntelligentComputer-AssistedLanguageLearningIntoLanguageEducation.International Journal of Information and Communication Technology Education,15(3),60–74.doi:10.4018/IJICTE.2019070105

International Journal of Information and Communication Technology EducationVolume 17 • Issue 1 • January-March 2021

18

Sergey Sakulin graduated the Bauman Moscow State Technical University in 2001. He is a Ph.D. by BMSTU in 2009. Today he is assistant professor of Information systems and telecommunications department. He has thirty scientific papers and five educational books. Scientific interests lie in the fields of artificial intelligence methods and expert knowledge formalization and visualization.

Alexander Alfimtsev received the B. S. and M. S. degrees in computer science from the Bauman Moscow State Technical University (BMSTU), Russia, in 2003 and 2005 respectively, and the Ph. D. degree in 2008. He is currently a Professor of Information systems and telecommunications department at BMSTU. He is also a Member of the International Institute of Informatics and Systemics, USA. He has 70 scientific papers, including 5 patents for inventions. His main research interests include the use of computer vision and artificial intelligence methods for human-computer interaction, pattern recognition and multimodal interfaces.

Dmitry Sokolov is an MSc student at Bauman Moscow State Technical University. He has five scientific papers. Scientific interests lie in the fields of artificial intelligence methods and programming engineering.

Tondeur,J.,VanBraak,J.,Ertmer,P.A.,&Ottenbreit-Leftwich,A.(2017).Understandingtherelationshipbetween teachers’ pedagogical beliefs and technologyuse in education:A systematic reviewof qualitativeevidence.Educational Technology Research and Development,65(3),555–575.doi:10.1007/s11423-016-9481-2

Tuapawa,K.(2017).Educationalonlinetechnologiesinblendedtertiaryenvironments:Experts’perspectives.International Journal of Information and Communication Technology Education,13(3),1–14.doi:10.4018/IJICTE.2017070101

Williams,J.L.,McCarley,N.G.,Haynes,J.M.,Williams,E.H.,Whetzel,T.,Reilly,T.,&Bailey,L.et al.(2016).TheUseofFeedbacktoHelpCollegeStudentsIdentifyRelevantInformationonPowerPointSlides.North American Journal of Psychology,18(2),239–256.

Yang,D.,&Skelcher,S.(2019).ImprovingTeachers’UnderstandingofTheoreticalFoundationsofTechnologyUse:ConnectingTheoriesWithTechnology.International Journal of Information and Communication Technology Education,15(4),113–127.doi:10.4018/IJICTE.2019100108

Yassine,A.F.A.,Battou,A.,&Omar,B.A.Z. (2019).LearningThroughMassiveOpenOnlineCoursesPlatformsBasedonFuzzyAnalyticHierarchyProcess.International Journal of Smart Education and Urban Society,10(3),1–12.doi:10.4018/IJSEUS.2019070101

Yates, J., & Orlikowski, W. (2007). The PowerPoint presentation and its corollaries: How genres shapecommunicativeactioninorganizations.Communicative practices in workplaces and the professions: Cultural perspectives on the regulation of discourse and organizations,1,67-92.