“Knowing Revisited”• And that’s how we can move toward really

knowing something:

Richard Feynman on the Scientific Method

Scientific Broadly Defined

• Science is an objective, logical, and systematic method of analyzing and explaining phenomena, devised to permit the accumulation of reliable knowledge.

• The product of science is knowledge

Scientific Broadly Defined• Premises of science are

– empiricism – objectivity– falsifiability– control

• What this means: – rationality and skepticism are how we learn about the universe and

shape new principles – If the arguments and experiments are sound, if the theory can

withstand skeptical scrutiny, if the work was undertaken within the framework of past research and provides a basis for further discovery, then it is science

The Process of Science• The process of science begins with speculation,

observation, and growing understanding of some idea or phenomenon. This understanding is used to shape research questions, which in turn are used to develop hypotheses that can be tested by proof or experimentation. The results are described in a paper, which is then submitted for independent review before (hopefully) being published; or the results are described in a thesis that us then submitted for examination.

--Justin Zobel

Elements of Good Scientific Research?

• Problem clearly specified and hypotheses formulated based on observations

• Objectives clearly defined - scope, limitations are fully specified

• Process detailed - can be repeated and except when secrecy is imposed reveal the sources of data and means by which they were obtained

• Design thoroughly planned - make as objective as possible

Elements of Good Scientific Research?

• High ethical standards applied• Limitations frankly revealed - there are very few

perfect designs• adequate analysis - data classified to clearly reveal

findings, probability of error should be estimated• Threats to validity (external, internal, construct)

clearly identified and addressed• findings presented unambiguously• conclusions justified

Research Project Phases• An individual research project (such as a Ph.D.

dissertation) follows a specific life cycle: – Choose research question/problem; formulate hypotheses– Determine current state of knowledge– Apply appropriate methods to produce research results

• To verify the hypotheses• To evaluate the proposed solutions

– Write up research results

• Research is not complete until it is written up (and published) – Peer Review is critical!

A Research Project Checklist (I)• Are ideas clear and consistent?• Is the problem clearly specified?• Is it worthy of investigation?• Are there proposed solutions to be explored?• Does the project have appropriate scope?• What are the specific hypotheses / research

questions• What would disprove the hypothesis?• What are the assumptions and are they “sensible”?

A Research Project Checklist (II)• Has the research plan/design been critically

assessed? Are you convinced that it is based on sound science?

• What forms of evidence are to be used?• How are outcomes to be evaluated? What are the

metrics? Why are the selected methods appropriate for verifying the hypothesis or evaluating the proposed solutions?

• What are the likely weaknesses of your solutions?

A Research Project Checklist (III)

• Is there a detailed written research plan?• Have milestones, timelines, and deadlines been

identified?• Do the deadlines leave enough time to receive

feedback on the drafts and to allow colleagues to contribute?

• Has the literature been adequately explored? Once the work is done – and your perspective has changed – does in need to be explored again?

Unstructured Environment• especially in PhD• need to develop a routine• make research a priority• regular meetings with your advisor• join or set up a research group• role of course work• keep a journal - get in habit of writing down ideas

Research Is About Reading• Foundation of good research: find and critically read

related scientific articles• Locate important research tools in your area

– journals and conferences– workshops– important or seminal papers in your area– good research sites and groups– other faculty web sites– develop your own Web repository (why?)

• How to read papers– first look at abstract, keywords, references– next: for some papers look at intro. and conclusions– next: for some, read in detail and write notes

What to consider when critically reading papers (write a summary for every paper you read)

• what is the main problem? why is it important?• what is the hypothesis? how precise are the claims?• what are the main results (in your own words)?• how does it relate to previous work? what are the most

important references?• what is the evidence and how was it gathered?• what are the metrics and how are measurements taken?• how carefully are the algorithms and experiments described?

what would the reproduction of the results involve?• why and how are the results significant?• ideas for future work?

Exercise

• Select one recent paper relevant to you area of interest

• Critically read the paper (using criteria in previous slide)

• Submit the paper and your notes in two weeks

• You may be asked to summarize in class

Research Is About Writing• writing is the most fundamental part of research• writing not just an end, but also a tool• writing <=> logical reasoning• take a writing course• when reading papers, mark those that are especially well-

written• keeping a journal, revisited• consider writing a survey paper• write one or more research proposals• Note: programming or math are not writing, by

themselves; programming is also not research