Embed Size (px)
Citation preview
A mathematical formula recognition method and its performance
evaluation
Masayuki OkamotoShinshu UniversityJAPAN
Goal of our study Character and symbol recognition Structure analysis and recognition Performance evaluation method Experimental results Future works
Overview of presentation
High performance formula recognition system for “Archiv der Mathematik”
Goal of our study
Overview of Recognition System
Labeling
Character or symbol recognition
Structure recognition
Touching character separation
Font type (1/2) Alphabet
1. Roman2. Italic3. Bold4. Calligraphy5. German
Greek
Font type (2/2) Digits Mathematical symbols
CharactersNormal size Small size
Number of characters or symbols: 650
Dictionary data Following three features are
calculated from each sample
featurecalculation
1. Mesh features2. Peripheral features3. PDC features
dictionaryfeaturesimage
Character recognition process
result
Givenimage
Featurecalculation
features
comparison
Dictionary data
Character recognition process
We classify the given image with each feature and we use the majority vote
Result from mesh features
Result from peripheral features
Result from PDC features
Majority vote
Touching characters We assume a character which has a
low score of similarity as a touching character
Result/Score ‘O’/0.847 ‘(’ /0.980 ‘y’ /0.990
Touching character segmentation(1)
Blurring the image
Calculate minimal points
Estimate cutting lines
Comparison
Classification
Touching character segmentation (2)
Make projection profile
Projectionprofile
Image
|hi – hi+1 | > θ
Recognize
Segmentation experiment 47 touching characters found in our
experimental data
Touching type samples
errors rate
Vertically 12 9 25%
Horizontally 25 9 64%
Fraction bar 6 2 67%
Three characters 4 4 0%
Correct result Correct examples
Touch withfraction bar
Errors Errors
Three touching characters
Other types
Recognition experiment Number of symbols : 12659 We excluded touching characters We distinguished following similar
shape characters
‘C’ upper case, ‘ｃ’ lower case
alphabet ‘l’, digit ‘1’,
alphabet ‘O’, digit ‘0’
x, chi
‘C’ upper case, ‘ｃ’ lower case
alphabet ‘l’, digit ‘1’,
alphabet ‘O’, digit ‘0’
x, chi
Recognition rate
Font type samples errors rate
Digit 1940 2 99.90%Alphabet 3811 29 99.24%
Greek 518 3 99.42%
Bold 171 0 100.0%Calligraphy 204 3 98.53%German 58 2 96.55%Symbol 5957 29 99.85%
Total 12659 68 99.95%
Recognition rate Similar shaped characters
Type Samples Errors Rate
C,c 98 6 93.88%
1,l 502 3 99.40%
O,0 368 12 96.74%
x,χ 452 0 100.00%
S,s 189 10 94.71%
Total 1609 31 98.07%
Examples of recognition errors
Most errors occurred at small characters such as scripts
Our previous methods （１）
Projection profile cutting
Our previous methods （２） Specific structure
processing(Bottom-up) Script Root Matrix
Fundamental structure processing(Top-down) Vertical division by
symbols Horizontal division
by symbols Horizontal division
by blank space
Core symbol in subexpression
Character recognition
Structure recognition *
Output
Outline of structure recognition
Target symbol
Horizontal connection
Top to bottom
Group A processing
Group B processing
[symbol = fraction,root,matrix]
[symbol = script,limit]
Recursion
Output in LaTeX/mathML
Image
Structure Recognition (1/2)
•Fractions•Roots•Matrices
Target symbolMatrix Recognition
Target symbol
Structure Recognition (2/2)
Scripts Limits
Adjacent symbolAdjacent symbol
Target symbol
Target symbol
Matrix Recognition
Vertical Overlap
Horizontal O
verlap
Case-distinction
Vertical Overlap
Horizontal O
verlap
Right EdgeLeft Parenthesis
<mrow> <msubsup rect="1,1,209,210"> <mrow> <mo>(</mo> <mfrac rect="43,11,87,187"> <mrow> <mi>β</mi> </mrow> <mrow> <mi>α</mi> </mrow> </mfrac rect="43,11,87,187"> <mo>)</mo> </mrow> <mrow> </mrow> <mrow> <mo>(</mo> <msubsup rect="152,24,189,56"> <mi>e</mi> <mrow> <mi>i</mi> </mrow> <mrow> </mrow> </msubsup rect="152,24,189,56"> <mo>)</mo> </mrow> </msubsup rect="1,1,209,210"> <mo>=</mo> . . .
Original expression
Answer Database Format
<msubsup rect="1,1,209,210">
Positional Information
<mrow> <mrow> <mo>(</mo> <mfrac rect="43,11,87,187"> <mrow> <mi>β</mi> </mrow> <mrow> <mi>α</mi> </mrow> </mfrac rect="43,11,87,187"> <mo>)</mo> </mrow> <mrow> <mo>(</mo> <msubsup rect="152,24,189,56"> <mi>e</mi> <mrow> <mi>i</mi> </mrow> <mrow> </mrow> </msubsup rect="152,24,189,56"> <mo>)</mo> </mrow> <mo>=</mo> . . .
<mrow> <msubsup rect="1,1,209,210"> <mrow> <mo>(</mo> <mfrac rect="43,11,87,187"> <mrow> <mi>β</mi> </mrow> <mrow> <mi>α</mi> </mrow> </mfrac rect="43,11,87,187"> <mo>)</mo> </mrow> <mrow> </mrow> <mrow> <mo>(</mo> <msubsup rect="152,24,189,56"> <mi>e</mi> <mrow> <mi>i</mi> </mrow> <mrow> </mrow> </msubsup rect="152,24,189,56"> <mo>)</mo> </mrow> </msubsup rect="1,1,209,210"> <mo>=</mo> . . .
not found
Comparison between Results and Answers
(a) Original expression (b) Recognition result
found
found
Correct Recognition Count
11Fractions
12Scripts
Number correctly recognized (C)
Number in original expression (N)
Recognition rate = C / N
Arch.Math., Page 44, Vol. 64
limit
Correct Results (1/4)
Arch.Math., Page 272, Vol. 65
Multi-fraction
Correct Results (2/4)
Arch.Math., Page 277, Vol. 64
Sparse Matrix
Correct Results (3/4)
Original expression
Recognition result
Correct Results (4/4)
Arch.Math., Page 108, Vol. 64
Nested case-distinction
Original expression
Recognition result
Errors (1/2)
Arch.Math., Page 65, Vol. 24
Matrix
Originalexpression
Recognitionresult
Errors (2/2)
Arch.Math., Page 104, Vol. 64
Case-distinction
Originalexpression
Recognitionresult
Inapplicable expressions (1)
Inapplicable expressions (2)
Inapplicable expressions (3)
Structure Recognition Rate
Structure Total Error Correct rate (%)
Scripts 3841 43 98.9
Limits 605 37 93.9
Fractions 119 3 97.5
Roots 70 1 98.6
Matrices(Case-distinctions)
66 8 87.9
Total 4701 92 98.0
Summary of structure recognition
Extension of recognition method Matrix and case-distinction
Performance evaluation Quantitative evaluation for a large
number of expressions Automatic calculation of recognition
rate for each typical structure
