MSR- Methodologies for Scientific Researchpaginas.fe.up.pt/~eol/PRODEI/mic1314_files_i/Methods.pdfSource: Eugénio Oliveira- 2013 MSR- Methodologies for Scientific Research Questioning

04-09-2013

1

Eugénio Oliveira- 2013 MSR- Methodologies for Scientific Research

MSR- Methodologies for Scientific Research

1. Methods and Techniques for scientific Research

A possible vision about Theories on the development and

characterization of Scientific Research

http://paginas.fe.up.pt/~eol/PRODEI/mic1314_i.htm

Eugénio Oliveira- 2013/2014




1. Abstract of my Lectures


Understanding about Science

Scientific Methods and Techniques

Writing scientific papers

Evaluating scientific papers

Consulting scientific Data Bases and searching for information ( Dr. Ana Azevedo)

To criticize a scientific paper

To organize a workshop

To apply scientific methods

To write your own scientific paper

To present a scientific paper

2. Abstract of your practical Work

http://paginas.fe.up.pt/~eol/PRODEI/mic1314_i.htm

04-09-2013

2

Eugénio Oliveira- 2013 MSR- Methodologies for Scientific Research MSR- Methodologies for Scientific Research

1.1 Basic Motivations: Issues on Scientific

Methods


Source: GETA and Graduate School of Electrical and Communications Engineering course:

Introduction to Research Methodology, Aarne Mammela, VTT-

Helsinki University of Technology,

04-09-2013

3


Source: http://www.experiment-resources.com/


Questioning Scientific Research- T.Kuhn

• Basic Characteristics:

• Cumulative versus reformulating always from the principles

• However:

•Thomas Kuhn

(Harvard, Berkley, Princeton, MIT until 91. +96)

“The Structure of Scientific Revolutions” 1962.

• He points out the role in scientific

research of the emergence of NEW

PARADIGMS.


http://des.emory.edu/mfp/kuhnquote.html

04-09-2013

4


Questioning Scientific Research- T.Kuhn

• He points out the role in scientific research of the emergence of

NEW PARADIGMS.

•A PARADIGM is defined as:

Scientific discoveries, which universally recognized during a

certain period of time, create a new model for both problems

and their respective solutions to be used by a community of

practicians ans experimentalists


What is your interpretation

Of the figure?

It depends on your mental,

A priori perspective….


• Paradigms support theories based on concepts, phenomena and

techniques for helping on explaining new facts or information .

• Science progresses through successive ruptures depending on

the current accepted paradigms.

Paradigms correspond to “different views” over the reality which

are influenced by aspects other than scientific ones.

Questioning Scientific Research- L. Wittgenstein

L.Wittgenstein “Tractatus Logico-Philosophicus Wttgeinstein’s Poker”, David Edmonds, J. Eidinow

MSR- Methodologies for Scientific Research Eugénio Oliveira- 2010

http://des.emory.edu/mfp/kuhnquote.html

04-09-2013

5


•

Wittgenstein and duck-??????…

Questioning Scientific Research- L. Wittgenstein


That’s the rabbit

Mouth!!

Eugénio Oliveira- 2010


• Paradigms correspond to “different views” over the reality which

are influenced by aspects other than scientific ones.

Questioning Scientific Research- L.Wittgenstein


“ Well, God has arrived. I met him on the 5:15 train. He has a

plan to stay in Cambridge permanently”.

John Maynard Keynes, after meeting with Wittgenstein at his

arrival in Cambridge, in a letter to his wife

http://fr.wikipedia.org/wiki/Image:Duck-Rabbit_illusion.jpg

http://en.wikiquote.org/wiki/John_Maynard_Keynes

04-09-2013

6


• Scientific interpretation (and perception) is mostly affected by

Social as well as Psychological factors.

• Does it lead to Irrationality dominating Science?

No !

• A good discussion on prevalence of science can be found in:

• Kuhn, T.S.

“Black-Body Theory and the Quantum Discontinuity”, 1987.

However: Paradigm change non-cumulative Science (sometimes)

Questioning Scientific Research- A. Einstein & C.



• Examples of important Paradigm changes:

• Solar System Ptolemy's vision changed by Copernicus

vision).

Questioning Scientific Research- A. Einstein & C.


• Going from Newtonian Physics to Quantic Physics (Uncertainty

Versus Determinism..) and Relativity (Space-Time relationship).

04-09-2013

7


• Karl Popper (London S. E., + 94)

• “The Logic of Scientific Discover”, Basic Books, 1959 (1ªEd Ing)

• Conjecture and Refutations, Harper & Row, 1968

Questioning Scientific Research- K.Popper


• Idealistic. He developed a Scientific Method as an Epistomology

(a way of “knowing”) independently of the intrinsic value.

• T. Khun and Feyerabend (1993) criticize this point of view showing how

science is guided by value and, therefore being non-idealistic.

• This is a debate on “Consciousness over Existence”


• Popper argues that scientific theory will always be conjectural and

provisional.

It is not possible to confirm veracity (the truth) of a theory by means

of simply recognizing that the results of a preview already done based

on that very same theory, hold.

That Theory should only be seen as a theory not yet contradicted by

facts.

• “Critical Rationalism” proposal defends “falsifiability”

as a criterion for demarcating (separating) science and non-science

Induction Baaaad!!! - Falsifiability gooood!!!

Questioning Scientific Research- K. Popper


04-09-2013

8


• Verificationism:

a sentence should be either verifiable or falsifiable

• Critics addressed to Verificationism: • Generalization: specific instances lead to universal

assumptions

Observation X, being an instance of Theory T

Increases the probability of Theory T being correct

The “Black Crow paradox”:

• The “Black Crow paradox”: by Hempel:

“All Crows are Black" is logically equivalent to

“Everything which is Non-Black is not a Crow“

x(Cx → Bx) ≡ x(¬Bx → ¬Cx)

Questioning Scientific Research


• The “Black Crow paradox”: by Hempel:

“All Crows are Black" is logically equivalent to

“Everything which is Non-Black is not a Crow".

x(Cx → Bx) ≡ x(¬Bx → ¬Cx)


• Thus, IF x(Cx Bx) reassures that x(Cx→Bx),

Then x(¬Bx ¬Cx) reassures that x(¬Bx→¬Cx)

which, being equivalent to x(Cx→Bx),

this last implication would also see its probability

of being True increased !!?

http://pt.wikipedia.org/wiki/Imagem:Corvus_corax_%28FWS%29.jpg


04-09-2013

9


• Crítics addressed to Verificationism: • Generalization: specific instances lead to

universal assumptions

• “There are Apples (non-Crows) not Black”

Increases the probability that the sentence

“All Crows are Black” is True !!?

• If verificationism (generalization) was

acceptable, any trivial evidence would support

a Theory


There are attempts to overcome this paradox.

“falsifiability” proposed by KP states that keeping

just the Existential assertion permit us to know

that if another existential proposition that contradicts it, arrises,

this falsifies the Theory T !!



Scientific Research



http://en.wikipedia.org/wiki/Image:Apples.jpg

04-09-2013

10


Scientific Research Exploration


Motivation: ambiguous…

Problem: a new way to “India”

Hypothesis: over the Atlantic and not around

south Africa

Experiment: Yes he have done it !!

Conclusions: ?????

An exploration Enterprise:

Christopher Columbus – the “Explorer”


Scientific Research Exploration


research: discover new knowledge

• basic research (no specific application in mind)

• applied research (ideas into operational form)

• development: systematic use of the existing

knowledge

04-09-2013

11


Scientific Research Object

•


Science, Technology and Engineering [Jain97]

• science: organized or systematic body of knowledge

• technology: application of scientific knowledge for

practical ends in engineering, medicine, agriculture, etc.

•natural sciences and engineering sciences differ in

the object of study:

• natural sciences: objects in the nature

• engineering sciences: objects (products,

services, methods) not found in the nature, using

results of mathematics and natural sciences


Scientific ResearchERS


Requirements for success

• You have to be:

• analytical and curious,

• autonomous and collaborative

• criticism and tolerant

Science, Technology and Engineering [Jain97]

Verification, confirmation, and justification are

synonymous terms in philosophy of science. The

opposite is falsification or refutation. R. K. Jain and H. C. Triandis, Management of Research and Development

Organizations: Managing the Unmanageable. John Wiley & Sons, 1997.

04-09-2013

12


Scientific ResearchERS


How does a scientific researcher work?

• make always notes in a “notebook”

• make summaries about what has been learned.

• make plans for the future all the time (outlines,

roadmaps)

• discuss, ask questions and argue (criticism)


Scientific Research and Engineering


Learning carefully by induction (bottom up, generalization

from examples to models) [Felder88]

• Defending theories by deduction (top down, from

models to results)

Break the problem down and then generalize the results

(divide and conquer)

R. M. Felder and L. K. Silverman, “Learning and teaching styles in

engineering education, ”Engineering Education, pp. 674681, April 1988.

04-09-2013

13


Scientific Research


1.2 Supporting Theories


Scientific Research

•B- Techniques refer to the means and specific Tools

enabling relevant information acquisition, the respective data

analysis, as well as the inferences that can be made to get

the results.

• Basic Characteristics

• Research involves Methodologie(s) and Techniques


• A- Methodology includes the study of several methods that are

applicable to a class of problems

• i.e., the set of processes to conduct each specific Research

Project

04-09-2013

14


Research Methods Dimensions

• ‘Instrumentalist Research’:

• It contributes for making human intervention in Real world

environments, more effective.

• A- Methodologies

• depending on the MOTIVATION:

• ‘Pure Research’:

• It contributes to a deep theoretical understanding and for a

more abstract formulation of the phenomena.

• Two major sub-categories are identified:

• ‘Applied Research’

• ‘Problem - Oriented Research’




• ‘Apllied Research’.

• Starting from a technology (devices, specific techniques, both)

use them for dealing with processes (physic, organizational, social,

individual,...)

• ‘Problem- Oriented Research’.

• Starting with the problem description and then looking for

the adequated techniques to solve it.


Who wants to classify the research (s)he is willing to pursue?

04-09-2013

15



• Depending on the Theory underlying the Research:

• ‘Descriptive’, of all significative aspects of the domain

• ‘Explicative’, of the behaviour of all the phenomena

• ‘Predictive’ of the future phenomena behaviour

• ‘Prescriptive’, beyond the prediction it should also prescribe and apply

norms and processes in identified specific circumstances

Research in Informatics,IE, IS, Electronics and Telecommunications

should be of this kind


To solve predicted problems in a flexible way !!

Or even to deal with some unexpected situations



• Depending on the tradition in that scientific area or Knowledge Domain:

i) ‘Traditional Scientific Research’ which is more quantitative

ii) ‘Interpretative Research’ which is more qualitative

iii) ‘Engineering-oriented Research’

• iii) most appropriated for Informatics, Electronics and Telecommunications

• But do not ignore the other approaches!


04-09-2013

16


Scientific Research Traditional Methods

1.3 Positivist (classical) Research method




• From observations we may build up Theories trying to

explain what has been observed

• Theories are expressed in Deductive form or using

Axioms and Postulates which are then operated through

Logics.

• Traditional Scientific Research is based on positivist rational

thinking:


Deduction Vs Induction Vs Abduction

04-09-2013

17




• Scientific Theories, following K. Popper, are capable of

generating Inferences which are, in principle, falsifiable

through references to the real world.

• Results coming from the process of testing hypothesis

give feedback to the Theory, enabling verification of its

trueness or detecting possible “ab-normalities” (T.Kuhn).

Do Not Forget :


• Whenever the Observation comes from the Real World it is called empirical

and it becomes relevant in two situations:



• during the formation of the theoretical knowledge through the so-called

passive observations

• during Hypothesis verification, gathering information in a more

active way, although guided by that same theory.

04-09-2013

18


• When we do not have a suitable Theory available, we have to pursue

‘exploratory research’ where experiments and interpretation are open,

not guided by a Theory but making available Empirical Knowledge

that may postulate other Theories.



• Informatics, IS and Telecommunications are disciplines (scientific areas)

without a given stable theory.

• They encompass applied subjects using a partial theory or co-opting a

theory from “reference disciplines” such as:

• Mathematics

• Information Theory (C.Shannon)

• Physics

• “Theory of Organizations”,

• “Management Sciences”,

• “Computer Science”.


RECAP

Methods and Techniques for guiding Research activities

• Cumulative versus reformulating always from the principles

• emergence of NEW PARADIGMS

• “Critical Rationalism” proposal defends “falsifiability”

as a criterion for demarcating science and non-science

• Critics addressed to Verificationism

• ‘Instrumentalist Research’:

• It contributes for making human intervention in Real world

environments, more effective.

• ‘Applied Research’

• ‘Problem - Oriented Research’

A- Methodology

B- Techniques refer to the means and specific Tools

‘Pure Research’ X

04-09-2013

19


•A- Methodology

•‘Descriptive’, of all significative aspects of the domain

• ‘Explicative’, of the behaviour of all the phenomena

• ‘Predictive’ of the future phenomena behaviour

• ‘Prescriptive’, apply norms and processes in pre-

identified specific circumstances

• Depending on the tradition in that scientific area or Knowledge Domain:

i) ‘Classical Scientific Research’ which is more quantitative

ii) ‘Interpretative Research’ which is more qualitative

iii) ‘Engineering-oriented Research’



Theory of

Disciplines

of Reference

Theory

Axioms

Postulates

Deductive Logic

Hypothesis

2

3

4

5

7

Real World 1

1- There exists a Real World that we can Perceive.

2- Knowledge is Organized in Theories making inferences possible

3- Applied Disciplines like IS or T rely on Theories of Reference of other Disciplines

4- A Research Project permits testing formal Hypothesis formulated through inferences expressed in the

Theory.

5- In Empirical Research, Hypothesis derived from the Theory are directly cheked in the Real World

6- Testing Hypothesis through Simulation

7- Results comming out of the Research give feedback to the Theory

Research

Project

Sim

ula

tio

n

6

04-09-2013

20


• Summarizing: In traditional (or conventional) Science, first new Hypothesis

are extracted from the existent Theory, they are tested and new results are

added to that Theory.


• Thus, this implies the pre-existance of:

• A theoretical Body of Knowledge

• An explicit theoretical Framework to guide the research

• Definition of what issues have to be investigated

• What are the explicit Hypotheses (and how could they be refuted (falsifiability))

• Technique for Applied Research and well defined techniques for

testing the Hypothesis.



1.4 Interpretative Research method

Scientific Research Methods


Is concerned with trying to understand lived

experience and with how participants themselves

make sense of their experiences.

IRM is phenomenological in that it wishes to explore

an individual’s personal perception or account of an

event or state as opposed to attempting to produce an

objective record of the event or state itself.

http://www.psyc.bbk.ac.uk/ipa/ (Birkbeck- U. London)

Therefore it is concerned with the meanings which

those experiences hold for the participants.

http://www.psyc.bbk.ac.uk/ipa/

04-09-2013

21


• Interpretative Method (or philosophy):

• post-positivist

• criticizes the (conventional) scientific “chimera”

• declares the impossibility of objective observation

• Observations and Interpretations depend on the Observer.

• Addresses critics to both “hard Sciences” and Social Sciences,

Interpretative Approach to Scientific Research


• Difficulties for Objectivity:


• Concepts, Factors, relationships that cannot be accessed

Interpretative Approach to Scientific Research

• Researchers involvement in the research domain

• Results depending on the researcher own perspective:

- On selecting and defining the research (sub-) domain

- On selecting a pre-existant theory

- On setting the issues to be investigated

- On setting the precise scope of the research

- On selecting and defining the Variables to be instantiated

- On measuring those variables possible values

• Different attitudes on all those issues may lead to many different

interpretations of the same observed phenomena.


04-09-2013

22


1.5 Research in Engineering disciplines


Engineering-oriented Research Approach


• For this approach to research, technology is very important!

(devices, artefacts, tools, practical techniques to be used)


• At least up to a certain extent is the most suitable for research in

Informatics, Information Systems, Electronics, Telecommunications.

• Research in Informatics, IS and Telecomm is applied and

problem-oriented.

• Implying:

- application of suitable technology,

- conceptualization: requirements, models

- prototyping,

- implementation

- demonstration of (new) technology (real world/sim)

- Evaluation phase


04-09-2013

23


Observation and Theory [Wohlin99]

• In engineering an hypothesis (also defined in system

specifications) is usually an idea of the relationship between the

cause and effect (defined in system requirements)

• Theoretical model is always only an approximation of observation

in real world

C. Wohlin et al., Experimentation in Software Engineering: An Introduction, Springer, 1999.



In engineering we are fundamentally interested in

how efficiently the basic resources are used for a

needed system performance.



Moreover

when finding solutions for a problem:

04-09-2013

24


Research Techniques


1.6 Techniques associated to Research


I. Non empírical Techniques

II. Scientific Research positivist Techniques

III. Scientific Research Interpretative Techniques

IV. Scientific Research Techniques at the positivist/interpretative boundery

V. Scientific Research Engineering-based Techniques

Research Techniques

B- Techniques


04-09-2013

25


I. Non empirical Techniques depend on:

• Artificially created data, or

• Conceptual thinking about abstractions

• Non empirical Techniques include:

• conceptual research based on opinion and speculation.

( argumentative and dialectic analysis)

• theorem proving: This applies to mathematical abstractions:

• formal methods application,

• Induction (in the sense that results are “abstractions”),

• mathematical abstractions

• Models verification

• Simulation:

• Mapping a complex environment in a simplified Model.


Research Techniques


• Other non empiric Techniques:

• Using hypothetic future scenarios

• Mapping a real scenario into Games

• ...

• Reviewing the State-of-the-art and doing Meta-analysis


Research Techniques

04-09-2013

26


FEUP-LIACC / Eugénio Oliveira

Using Games for Research

RoboSoccer Competition-middle size


Março 2007 FEUP-LIACC / Eugénio Oliveira

IA- Futebol Robótico- Sony legged

04-09-2013

27


II. Scientific Research positivist Techniques

(applicable to Informatics, IS and other Engineering disciplines):

• Techniques for forecasting applying

• regression algorithms (Technique for fitting a simple equation to real data points)

• time-series based algorithms to extrapolate on historic

data ; to extract meaningful statistics from data; • Field Experiments :

• Possibly isolating and controlling a set of phenomena

to be looked at

• Laboratory Experiments:

• Creating artificial setups in which phenomena, factors

and variables are isolated and controlled.


Research Techniques


• Research approach is Descriptive/Interpretative:

Going from empirical observations to a limited rigour analysis.

Need for controlling the researcher’s perspective:

- Critical self- Examination

- Revising all the postulates and biases

- Varying Observations

- Submission to peer reviewing

III. Scientific Research Interpretativist Techniques:

• Group Research:

• frequent discussion with a group of people including

who may be affected by the application (or the technology)

• Use Collaborative Work Tools.


Research Techniques

04-09-2013

28


• Action-based Research:

- Researcher is like an “Agent” changing existent

conditions to measure the respective reaction

• Other Methods for Social Sciences(Ethnographic..)

In http://www.qual.auckland.ac.nz/ Michael Myers Ed. Ass.Inf.Sys.

It is advocated qualitative research, for example, for IS.

We also can do (but it is not recommended) R&D just through

Case-study analysis


Research Techniques


IV. Scientific Research Techniques at the positivist/interpretativist

boundery

Including:

- Field Work : Data related to the object of study is

directly observed by the Researcher in the original

context

- Questionary-based : Structured Data collection comming

from interviewees

- Case Study: - Isolate and describe a relevant real-world

situation,

- Gather a collection of data from multiple

sources


Research Techniques

Eugénio Oliveira- 2010

http://www.qual.auckland.ac.nz/

04-09-2013

29


• Secondary Research:

• Analysis starting from pre-existent documents (texts,

papers, descriptions, memoirs).

• Analysis is done under new perspective and using

different techniques or procedures.


Research Techniques


v. Scientific Research Engineering-based Techniques

(Informatics Engineering, IS, Electronic Engineering, Telecomm...)

• Divided into two categories:

1- Construction Technique:

• Includes Design, Project, Implementation

(or prototyping) of a system

(ex: Computer program System, or physical device)

- objective:

-- explicitly testing an hypothesis or

-- to solve a class of problems.

- it implies:

-- Existence of either a theory or a model

for explaining the results

(testing the hypothesis)


Research Techniques

04-09-2013

30


• theoretical model is defined (preferably a mathematical model),

including boundary conditions (define the environment)

and initial conditions (define the initial values of the

parameters of the model) [Losee01]

J. Losee, A Historical Introduction to the Philosophy of Science,

4th ed. Oxford Univ. Press, 2001.


Research Techniques

Materials and Methods

• in exact sciences materials include definitions,

model, and related assumptions


• rules of analysis follow deductive reasoning,

statistical analysis and approximations (analysis

must often be replaced by simulations due to

mathematical tractability problems)

• the theoretical model is verified by comparing the results

with reality (measurements with a prototype, known as

hypothetical-deductive method)


Research Techniques

• methods include:

rules of analysis and

rules of verification and validation

04-09-2013

31


• rules of analysis follow deductive reasoning,

statistical analysis and approximations (analysis

must often be replaced by simulations due to

mathematical tractability problems)

• the theoretical model is verified by comparing the results

with reality (measurements with a prototype, known as

hypothetical-deductive method)


Research Techniques

• methods include:

rules of analysis and

rules of verification and validation


• use analytical results in simple limited cases to obtain reference

values

• the value of the results depend on how well they can be

generalized (bottom up approach will help you)


Research Techniques

• Results

• results can be analytical (deductive), from simulation, or

measurement results

• it is important to present numerical results that

verify (or not) your own thesis,

http://en.wikipedia.org/wiki/Verification

http://en.wikipedia.org/wiki/Quality_control

http://en.wikipedia.org/wiki/Specification

http://en.wikipedia.org/wiki/Validation

http://en.wikipedia.org/wiki/Quality_assurance

04-09-2013

32


Conclusions (“constructive “-based technique report example)

(1) “We have shown that [some] algorithms proved right…”

(2) “Our approach is simpler to implement than previous algorithms and is

practically feasible …. On the theoretical side, the greatest deficiency in our

results is …”

(3) “In a practical situation the advantages and limitations of our

approach are…”

(4) “These results can be extended in a number of directions…”


Research Techniques

1. What is shown by that work and its significance.

2. Limitations and advantages.

3. Applications of the results.

4. Recommendations for further work.


• Destructive Techniques:

• To analyse a system, or a set of methods to gather

new information about those technology procedures

under analysis.

- Recognized information and characteristics are

then reused may be in a different way

- Example: Reverse Engineering


Research Techniques

04-09-2013

33


• There are philosophies permitting the Classification of Research

Methods through Paradigms or basic Epistemologies:

I. Positivist

II. Interpretative (post-positivist)

III. Critical

IV. (Constructivism)

At Harvard Business School they use just the three categories

Phylosophy Underlying Research



• Positivist Method:

• Assumes that reality is objectively accessible, it can be

measured and described by the observer and measured by

the same observer through their instruments and devices.

Tests are made making predictions possible over future

phenomena

• IS research (also Telecomm research?) can be of the

positivist kind (for IS, Orlikowski and Baroudi,1995)

- Use math/logic formalisms

- use quantifiable and measurable variables

- Test an Hypothesis and make inferences from what is

known.


Philosophy Underlying Research

04-09-2013

34


• Interpretativist Method

• Assumes that accessing to reality is only possible through

social constructions like language, consciousness and

ontologies (sharing of concepts’ meaning).

• Hermeneutics-based philosophy (based on the human

understanding and text interpretation) as well as in

phenomenology

(describing, understanding and interpretation perceived

phenomena)

Proposes to abolish separation between "subject“ and

"object",

It opposes to XIX century positivist thinking.




• Can Informatics, IS and Telecomm studies influence or be

influenced by the context?

• Here, the focus is not as much on the variables,

measurements and processes as it is on interpretation of

the results of the test, the measurement or implementation



04-09-2013

35


• Research Criticism:

• It is a radical enterprise once it tries to identify what constraints

research:

• socially, culturally,

• Historically, politically,

• economically, physically

trying to emancipate science by eliminating causes for ignorance,

alienation etc...

Eg: Why is it not possible to do research on X?

Eg; Why cannot we implement systems of type Y?

Habermas from Frankfurt School following the way of Adorno,

Lukacs, Marcuse




• Junk Science (?) Stories of the Past Decade

Dial “F” for Fear.

Since the 1993 Larry King Live broadcast featuring a man suing a

cell phone maker claiming his wife died from a cell phone-

induced brain cancer, many cell phone users have worried

about phone safety.

But studies failed to identify any risk for many, many years …



04-09-2013

36


• Junk Science (?) Stories of the Past Decade



Powerline scare unplugged.

Fears that electric and magnetic fields (EMFs) created by power

lines and appliances caused cancer started in 1978. Parents

worried about power lines over schools.

Power companies worried about burying power lines. The National

Academy of Sciences finally unplugged the scare in October 1996,

concluding that no evidence showed EMFs presented a health

hazard


Quality of Research

1.7 Quality of Research


04-09-2013

37


Quality of Research

• Quality depends on the Objectives to reach

1. To increase available (and also general) Knowledge

• Objectives and Quality criteria are internal to the discipline

• RIGOUR is very critical

2. Developing Computer Systems– (Instrumentalisme)

• Objectives and Criteria are external (regarding R&D)

• Relevance can be measured by the impact and is of the

most importance

• Relevance Versus Rigour

• Not antagonic but taken with different emphasis



• Case-studies based research is mainly descriptive concerning

simple instances and then less scientific

• Empirical research is now dominated by the descriptive

approaches

• Specific Models have limited capabilities for explaining a

phenomen as well as little predictive power.

• Computer Science (and other Engineering-like areas) tendency:

• Until the 80ies conceptual (not empirical) papers

dominated

• After the 80 ies Less theoretical papers, more

instantiated, less general and Knowledge less cumulative


Quality and directions of Research

04-09-2013

38


• Include support to:

• Logistics, procurement, contracting, trading

• Reorganization and reconfiguration

• Informatics and Computer Science Research:

• Services, Platforms and Frameworks for Electronic

inter-operability between Organizations (enterprises,

schools, institutions…) inside themselves and interfacing

to society (markets)




• Supporting Sciences:

• Biotechnology

• Graphic Computing

• Medicine

• Law

• Astronomy, Phisics, History, Management….

• Informatics and Computer Science Research:

• Algorithms and Systems for Information management:

• Visual (image, graphic…)

• Textual (Text Corpora, data in DB)

• Signals (traffic, biologic, environmental….)



04-09-2013

39


Quality of Research

• Requirements for the Research in CS and TC

• Choose an appropriate Research Method:

• Make explicit a Body of Theory sometimes coming from

other adjacent disciplines (ex: Computer Science, Organization

Theory, Information Theory, Physics, Statistics…)

• Traditional/positivist, “interpretativiste” or

Engineering-oriented

• Extend the Theory and Infer predictions to be checked

(at least in the case of positivist/traditional research)



(following Björn Lisper, Datorteknik, Mälardalens högskola

Västerås, Sweeden)

http://www.idt.mdh.se/kurser/ct3340/ht03/

Theoretical methods:

Create formal models (mathematics, logic)

Define concepts within thesis

Prove properties of the concepts

Abstraction, hide details to make the whole more understandable

(and to make it possible to prove properties of it)

Proofs of properties by deductive methods

Scientific Methods, General

http://www.idt.mdh.se/kurser/ct3340/ht03/

04-09-2013

40


Empirical methods:

Perform experiments

See how it turned out

Draw conclusions

Simulation:

Start with a formal model at some "easy-to-understand" level

Make "artificial experiments" in your computer

Collect statistics and draw conclusions

Scientific Methods, General


Common pattern in Computing Science:

The system is constructed to behave according to some

theoretical model

Deviations are seen as construction errors rather than

deficiencies in the theory (hardware error, bug in OS, : : :)

In both cases: the theory helps us understand and predict, but in

different ways!

Scientific Methods

In Physics:

Make hypotheses about the surrounding world (theory),

observe it (experiment)

Relate the result of experiment to the theory

Adjust the theory if it doesn’t predict the reality well enough

Theory is used to predict the future (e.g., if a bridge will hold for a

certain load, or an asteroid fall down on our heads)

04-09-2013

41


Scientific Methods

Theoretical vs. Empirical Methods in Computing Science

Computing Science really has a “spectrum”, from “extreme

constructivism” to the use of theory close to the one in physics:

• “Extreme constructivism”: (ideal) programming language design:

Formal semantics for the language, pure construction of model

defining the mathematical meaning of each program

Abstraction of details to make the meaning of the language

simpler (for instance, assume that data structures can grow

arbitrarily big)

Implement the language according to the semantics


Scientific Methods


One can prove formally within the model that a

program is correct –valuable!

But the model does not cover all kinds of failures.

E.g., hardware errors, or stack overflow (or an

asteroid falling down on the computer)

04-09-2013

42


Scientific Methods


Extreme “physics” approach:

performance modeling of complex computer-

and communication systems

Extremely hard to make analytical calculations

Simplified performance models, tested against

experiments (e.g., long suites of benchmarks)

Discrepancy leads to a modified theory, as in physics

Often simulation (desire to evaluate systems before

building them)


Scientific Methods


In-between: algorithm analysis

Build on some form of formal model for how the

algorithm executes (meta-language with formal

semantics), and some performance model (how long

does a step in the algorithm take, how much memory

is needed to store an entity)

04-09-2013

43


Scientific Methods


In-between: algorithm analysis

Given that the performance model is correct, one

proves

mathematically that the algorithm needs certain

resources (time, memory) to be carried out

But the performance model is often very approximate.

Sometimes is possible to refine the performance model,

but this can make it impossible to calculate the resource

needs of the algorithm


Quality of Research

• Characteristics of the Research in CS

• Always Report the most recent S-o-A. (always moving...)

• Prove you made an advancement in Knowledge

• Combination of Techniques trying to compensate one’s

weaknesses with other one strengths

• Report about the object of study and the nature of the domain

• Validate Data

• Analysis using statistics

• Extract relationships between variables

• Praticable according to the resources

• Relevant results (at different levels: community,society...)

• Publish

• Ambitious


04-09-2013

44


• Characterístics of the research in CS:

• Research may have as an objective:

• Building a Theory

• Testing a Theory

• Extending a Theory


Quality of Research

• The practical research objectives may include:

• Exploring a specific domain

• Describing a domain

• Explaining a domain

• Forecasting over a domain

• (Testing formalized hipothesis)


• Other characteristics:

• Concerning Data:

• Extensive

• Representative

• What is Data validity (ex. temporal, precision…)

• Granularity:

• Simple or agregate

• Specific or large generic experiments

• Temporal Horizon:

• “Snapshot” of the domain at a specific point in time

• “Longitudinal cut section” (múltiple points in time)

• Methods for colecting and analising data; quantification...


Quality of Research

04-09-2013

45


1.8 Challenges


Quality of Research


Research in CS and TC Challenges

• Challenges:

• There exists shortage of theory. Mainly theories in related disciplines

• Phenomena are unstable:

• Data may get old quickly, usually a snapshot that may vary

- Pay attention to the validity period

• Organizations and Markets are always moving

• Significant cultural variations:

• Different national/regional cultures imply different speed/rhythm

• When you are doing research (requirements, interaction,

questionnaires, demos) you influence the domain

• Pressure for publishing during the research action


04-09-2013

46


• Pressure for technical instantiation:

• Prototyping without a theoretical ground or well defined

objective

• Economics and business models either not known or consolidate.

• Impact of the automation and rationalization:

• Depends on time dynamics concerns

• Depends on who is directly affected

• Depending on if researchers are also objects of the study

• Adopting a new technology may take long time

• Impact on other partners, markets, standards, tools…




• Conclusions on Research actions in CS and TC

• Consider tradoff Rigour Versus Relevance

• Identify challenges BEFORE starting doing research

• Be clear about conventions related with research

to be done

• Select a significative sub-domain

• Define questions to be answered through that research

• Select the apropriate Research Method and Techniques

(possibly a good combination of some) and justify the choice.

Whenever the selected aproach is instrumentalist, you need

to be sure about the external quality, data validation and rigour

as well as:

Relevance



04-09-2013

47


Other Bibliography


• D. Sternberg, How to Complete and Survive a Doctoral Thesis.

St. Martin’s Press, 1981.

Documents

MSR- Methodologies for Scientific Researchpaginas.fe.up.pt/~eol/PRODEI/mic1314_files_i/Methods.pdfSource: Eugénio Oliveira- 2013 MSR- Methodologies for Scientific Research Questioning