Documentation Documentation Generators: Generators:

Internals of DoxygenInternals of Doxygen

John TullyJohn Tully

Motivation: Why Motivation: Why document code?document code?

Required by company / administrators who Required by company / administrators who examine codeexamine code

Typically in software engineering, code Typically in software engineering, code isn’t consistently worked onisn’t consistently worked on

Co-workers (or “smart” users) look at Co-workers (or “smart” users) look at source code for a better understanding of source code for a better understanding of functionalityfunctionality

Code modified/built upon by othersCode modified/built upon by others

Motivation: Why Motivation: Why Automated?Automated?

Docs much more likely to be up to date - Docs much more likely to be up to date - developers only need to update comments in code developers only need to update comments in code (editing word docs/latex files isn’t enjoyable)(editing word docs/latex files isn’t enjoyable)

Reuse of comments = ½ the workReuse of comments = ½ the work

Auto formattingAuto formatting More importantly, More importantly, automatic cross-referencingautomatic cross-referencing if the if the

automated system is sophisticated automated system is sophisticated

HugeHuge advantage: what if recipients want different advantage: what if recipients want different formats?formats? Management team (or dumb readers)Management team (or dumb readers) End usersEnd users Software testers or other software developersSoftware testers or other software developers

Problem / GoalsProblem / Goals Create a simple, easy-to-use, portable, Create a simple, easy-to-use, portable, highly highly

configurableconfigurable automatic documentation generator for a automatic documentation generator for a variety of output formatsvariety of output formats

Real motivation: Creator (Dimitri Van Heesch) used Qt, Real motivation: Creator (Dimitri Van Heesch) used Qt, an application development framework (popular for an application development framework (popular for creating GUIs)creating GUIs)

Wrote docs by hand to Wrote docs by hand to look exactly like Qt look exactly like Qt documentation; tried doc++ documentation; tried doc++ to do this automatically (not to do this automatically (not configurable enough)configurable enough)

source: sourceforge

Other Doc GeneratorsOther Doc Generators Several generators support more languages than Several generators support more languages than

Doxygen *Doxygen * ROBODoc, TwinText, Natural DocsROBODoc, TwinText, Natural Docs VB, Pascal, .NET, Perl, JavaScript, SQLVB, Pascal, .NET, Perl, JavaScript, SQL Many (including Doxygen) allow addition of new Many (including Doxygen) allow addition of new

languageslanguages

Discussion – ease use of these tools?Discussion – ease use of these tools?

As far as output formats, Doxygen most versatileAs far as output formats, Doxygen most versatile

Also seems to be best for diagram generationAlso seems to be best for diagram generation Dependency graphs, inheritance / collaboration diagramsDependency graphs, inheritance / collaboration diagrams Certain tools may do some things that Doxygen can’t Certain tools may do some things that Doxygen can’t

(control flow, data flow); but they’re language-specific(control flow, data flow); but they’re language-specific* Source for info on other generators: Wikipedia

Doxygen Information Doxygen Information FlowFlow

Source Code

Custom Stuff:- Headers- Footers- Images

DoxywizardGUI

Doxygen

Doxyfile(Config file)

Tag file

DoxytagProgram

HTMLPages

Man Pages

RTF Files

Latex Files

XML

ps or pdf

MS-Wordformatted

dox xmlparser (for

custom output)

external docs

Source: Doxygen “Getting Started” Page

1. Documentation1. Documentation

Source Code

Custom Stuff:- Headers- Footers- Images

DoxywizardGUI

Doxygen

Doxyfile(Config file)

Tag file

DoxytagProgram

HTMLPages

Man Pages

RTF Files

Latex Files

XML

ps or pdf

MS-Wordformatted

dox xmlparser (for

custom output)

external docs

2. Configuration2. Configuration

Source Code

Custom Stuff:- Headers- Footers- Images

DoxywizardGUI

Doxygen

Doxyfile(Config file)

Tag file

DoxytagProgram

HTMLPages

Man Pages

RTF Files

Latex Files

XML

ps or pdf

MS-Wordformatted

dox xmlparser (for

custom output)

external docs

3. Parsing / Execution 3. Parsing / Execution

Source Code

Custom Stuff:- Headers- Footers- Images

DoxywizardGUI

Doxygen

Doxyfile(Config file)

Tag file

DoxytagProgram

HTMLPages

Man Pages

RTF Files

Latex Files

XML

ps or pdf

MS-Wordformatted

dox xmlparser (for

custom output)

external docs

4. Output Options4. Output Options

Source Code

Custom Stuff:- Headers- Footers- Images

DoxywizardGUI

Doxygen

Doxyfile(Config file)

Tag file

DoxytagProgram

HTMLPages

Man Pages

RTF Files

Latex Files

XML

ps or pdf

MS-Wordformatted

dox xmlparser (for

custom output)

external docs

Doxygen InternalsDoxygen Internals

Doxygen Information Flow: high-level Doxygen Information Flow: high-level view of steps taken by the user to view of steps taken by the user to document codedocument code

Doxygen Internals: Doxygen Internals: overview of data overview of data flow as source files are parsed by flow as source files are parsed by DoxygenDoxygen

Closer look at Parsing / Execution Closer look at Parsing / Execution Phase mentioned previouslyPhase mentioned previously

Doxygen Internals: Data Doxygen Internals: Data FlowFlow

Config parser: processes configuration fileConfig parser: processes configuration file Written in flexWritten in flex Simple (since config options are pretty simple – just 5 types)Simple (since config options are pretty simple – just 5 types)

C Preprocessor: fairly similar to standard preprocessorC Preprocessor: fairly similar to standard preprocessor Written in flex; uses yacc-based parser for expression evaluationWritten in flex; uses yacc-based parser for expression evaluation

Language parser: convert input buffer into a tree of Language parser: convert input buffer into a tree of entriesentries Basically breaks up code into smaller modules that will be Basically breaks up code into smaller modules that will be

reorganized laterreorganized later

Config Parser

C Preprocessor

Data Organizer

Output Generators

Source Parser

Language Parser

DocumentationParser

Source: Doxygen“internals” page

Doxygen Internals: Data Doxygen Internals: Data FlowFlow

Data organizer: Gather information from these entriesData organizer: Gather information from these entries Build dictionaries of classes, files, variables, functions, groups, etc.Build dictionaries of classes, files, variables, functions, groups, etc. Determine relationships between these entitiesDetermine relationships between these entities

Documentation parser: find comment blocks / tags in entities; feed results to Documentation parser: find comment blocks / tags in entities; feed results to output generatorsoutput generators

Source parser (if source is documented): cross-referencing, syntax Source parser (if source is documented): cross-referencing, syntax highlightinghighlighting

Output generators: take data, which was found by language parser and Output generators: take data, which was found by language parser and organized/cross-referenced by data organizer, and generate output in organized/cross-referenced by data organizer, and generate output in specified formatspecified format

Config Parser

C Preprocessor

Data Organizer

Output Generators

Source Parser

Language Parser

DocumentationParser

Debugging Doxygen Debugging Doxygen

For a thorough understanding of doxygen’s For a thorough understanding of doxygen’s source code, understanding of source code, understanding of flexflex important important Lexical analyzer: generates Lexical analyzer: generates scannersscanners (programs (programs

that recognize lexical patterns in text)that recognize lexical patterns in text) Executable produced from scanners, which Executable produced from scanners, which

execute C code when patterns are foundexecute C code when patterns are found

Flex has debugging option to output matched Flex has debugging option to output matched rules when they’re found – easy to follow the rules when they’re found – easy to follow the steps doxygen is takingsteps doxygen is taking

On internals page, a tool (script) is available On internals page, a tool (script) is available which turns on debugging for any flex filewhich turns on debugging for any flex file

Limitations / Future Limitations / Future Work Work

Limitations / “wish list” of doxygen Limitations / “wish list” of doxygen discussed during demodiscussed during demo More languages, more output formats, better More languages, more output formats, better

template filestemplate files

Several improvements to internals also Several improvements to internals also mentioned:mentioned: Language Parser:Language Parser:

One scanner / parser per language (currently one One scanner / parser per language (currently one huge combined scanner)huge combined scanner)

Modulate parsing (documentation blocks vs code)Modulate parsing (documentation blocks vs code) Parse preprocessor input for extended documentation Parse preprocessor input for extended documentation

(i.e. #defines)(i.e. #defines)

Limitations / Future Limitations / Future Work Work

Output generators – interesting future Output generators – interesting future workwork Instead of using data structures (generated Instead of using data structures (generated

from the data organizer) to produce XML, from the data organizer) to produce XML, use XML as an use XML as an intermediate languageintermediate language

More info could be extracted by various More info could be extracted by various output generators – with more output generators – with more understandable IL, easy to create better end understandable IL, easy to create better end toolstools Interactive source browsersInteractive source browsers Configurable class diagram generatorsConfigurable class diagram generators Computing code metricsComputing code metrics

Discussion Discussion