doi.org/10.26434/chemrxiv.11542632.v1

Machine Learning-Based Multidomain Processing for Texture-BasedImage Segmentation and AnalysisNikolay Borodinov, Wan-Yu Tsai, Vladimir V. Korolkov, Nina Balke, Sergei Kalinin, Olga S. Ovchinnikova

Submitted date: 08/01/2020 • Posted date: 09/01/2020Licence: CC BY-NC-ND 4.0Citation information: Borodinov, Nikolay; Tsai, Wan-Yu; Korolkov, Vladimir V.; Balke, Nina; Kalinin, Sergei;Ovchinnikova, Olga S. (2020): Machine Learning-Based Multidomain Processing for Texture-Based ImageSegmentation and Analysis. ChemRxiv. Preprint. https://doi.org/10.26434/chemrxiv.11542632.v1

Atomic and molecular resolved atomic force microscopy (AFM) images offer unique insights into materialsproperties such as local ordering, molecular orientation and topological defects, which can be used to pinpointphysical and chemical interactions occurring at the surface. Utilizing machine learning for extractingunderlying physical parameters increases the throughput of AFM data processing and eliminatesinconsistencies intrinsic to manual image analysis thus enabling the creation of reliable frameworks forqualitative and quantitative evaluation of experimental data. Here, we present a robust and scalable approachto the segmentation of AFM images based on flexible pre-selected classification criteria. Usage of supervisedlearning and feature extraction allows to retain the consideration of specific problem-dependent features (suchas types of periodical structure observed in the images and the associated numerical parameters: spacing,orientation, etc.). We highlight the applicability of this approach for segmentation of molecular resolved AFMimages based on crystal orientation of observed domains, automated selection of boundaries and collection ofrelevant statistics. Overall, we outline a general strategy for machine learning-enabled analysis of nanoscalesystems exhibiting periodic order that could be applied to any analytical imaging technique.

File list (3)

download fileview on ChemRxivMachine learning-based multidomain processing for text... (906.36 KiB)

download fileview on ChemRxivESI Machine learning-based multidomain processing for te... (1.84 MiB)

download fileview on ChemRxivMachine_learning_based_multidomain_processing_for_t... (41.07 MiB)

1

Machine learning-based multidomain processing for texture-based image segmentation and

analysis

Nikolay Borodinov1, Wan-Yu Tsai1, Vladimir V. Korolkov2, Nina Balke1, Sergei V. Kalinin1,

Olga S. Ovchinnikova1*

1Center for Nanophase Materials Science, Oak Ridge National Laboratory, Oak Ridge,

Tennessee 37831, United States

2School of Chemistry, University of Nottingham, University Park, Nottingham, NG7 2RD,

UK.

* Author to whom correspondence should be addressed.

Olga S. Ovchinnikova

Center for Nanophase Materials Sciences

Oak Ridge National Laboratory

1 Bethel Valley Rd

Oak Ridge TN, 37831-6493

ovchinnikovo@ornl.gov

2

Abstract

Atomic and molecular resolved atomic force microscopy (AFM) images offer unique insights into

materials properties such as local ordering, molecular orientation and topological defects, which can be

used to pinpoint physical and chemical interactions occurring at the surface. Utilizing machine learning for

extracting underlying physical parameters increases the throughput of AFM data processing and eliminates

inconsistencies intrinsic to manual image analysis thus enabling the creation of reliable frameworks for

qualitative and quantitative evaluation of experimental data. Here, we present a robust and scalable

approach to the segmentation of AFM images based on flexible pre-selected classification criteria. Usage

of supervised learning and feature extraction allows to retain the consideration of specific problem-

dependent features (such as types of periodical structure observed in the images and the associated

numerical parameters: spacing, orientation, etc.). We highlight the applicability of this approach for

segmentation of molecular resolved AFM images based on crystal orientation of observed domains,

automated selection of boundaries and collection of relevant statistics. Overall, we outline a general strategy

for machine learning-enabled analysis of nanoscale systems exhibiting periodic order that could be applied

to any analytical imaging technique.

3

Advances in atomic force microscopy (AFM) instrumentation in the past several decades have

enabled measurements and experiments capable of probing structure and functionalities in exquisite details

with molecular and atomic resolutions.1 A complex combination of electromechanical, magnetic,

electrochemical and photothermal signals can now be probed and analyzed with nanoscale resolution.

Moreover, with the recent developments in video rate AFM2-3 direct observation of complex self-assembly

and migration phenomena can be seen in real time.4-5 Growing amounts of data collected from a single

experiment necessitate a paradigm shift in handling and processing of the information coming from the

instrument. While it is possible to manually analyze a single image, such approach becomes impractical for

streaming data due to scalability issues. While several fields have transitioned to automated analysis

workflows,6-7 this is still not the case for scanning probe microscopy (SPM). Another significant challenge

is the creation of consistent metrics that would prevent incorporating and propagating human error through

the analysis. Therefore, machine learning becomes a valuable tool for streamlining the processing of data

generated by AFM to yield useful insights into the materials structure and functionality.8-13

Spatial arrangement of the features observed by the microscope gives unique insights into the

structural organization of the material14 and distortions present in it.15-16 Specific types of ordering generate

a number of distinct patterns characteristic to the materials of interest, hence, the processing of atomic or

molecular resolved images can be viewed as the analysis of the textures appearing in the image. Analysis

of such data, especially when the dimensionality of the data exceeds classical 2D imaging (e.g.

hyperspectral imaging), represents a significant challenge as raw data is difficult to directly visualize in

human-friendly fashion. Meanwhile, the classification of such textures and subsequent segmentation can

provide a comprehensive description of atomic or molecular organization, i.e. ordered domains reflecting

crystal structure, charge density waves,17 and nematic superconductivity.11 To address this gap, image

processing and computer vision can be readily applied to streamline processing and feature extraction in

AFM analysis. As an example, image segmentation has been effectively applied for the diffraction study of

nanocrystalline structures in organic semiconductor molecular thin films.18

There are several methods that can be used to yield robust texture descriptors. A recent rise in the

use of deep neural networks (DNN) for structural analysis as well as increasing availability of programming

and hardware tools for designing and training DNNs, makes them a viable fit for these types of problems.

19-22 However, deep neural networks require comprehensive training sets which incorporate all possible

distortions that may be presented in the actual data. In cases when such dataset can be generated, DNN

performance in the classification and segmentation of the image is extremely efficient.23 However, when

such training set in unavailable, more traditional machine learning tools become necessary. Here, different

types of feature spatial organization within the image are captured by algorithms targeted at specific

4

structural descriptors such as symmetry, periodicity, directionality, etc. While spatial organization is non-

local (as more than one element is needed to comprise a distinct structure), the problem of image

segmentation requires a pixel-based assignment of the image. To resolve this conflict, a sliding window

approach has been used to extract structural descriptors within a given frame and then assign it to the central

pixel.14, 24-25 Alternatively, wavelet-based decomposition extracts details of varying periodicity as the

wavelet transforms preserve both frequency and location information and can be efficiently used to select

specific features on images.26-27

In this paper, we present a machine-learning based approach that we refer to as multidomain

processing which we apply for segmentation of atomical and molecular resolved AFM images. We use

AFM images of ionic liquid layers assembled on top of graphite and melem on boron nitride as benchmarks

for the algorithm. We designed a supervised three-step procedure which can be easily tuned to

accommodate varying nature of the datasets as well as high noise levels. First, structural descriptors are

extracted from an image. We characterize the performance of Fast-Fourier transform (FFT), Radon

transform and FFT-based cross-correlation (CC) and compare it to stationary wavelet decomposition

(SWD). On the second step, we apply dimensionality reduction algorithms such as principal component

analysis (PCA) for FFT, Radon and CC and non-negative matrix factorization (NMF) for SWD. Then, we

use resulting abundance maps to build a multidimensional representation of structural types in the dataset.

Finally, we perform assignment of pixels in an image and calculate relevant statistics. We highlight the

most efficient ways to tune the segmentation algorithm and provide direct recommendations on its

applications for image processing. We believe that this approach can be readily used for segmentation of

atomically and molecular resolved images due to its modularity on each of three steps.

From a fundamental standpoint, the analysis of local ordering in AFM represents a dual problem

of pattern recognition and physics extraction. As individual elements (atoms, molecules, etc..) exhibit

specific arrangements which are indicative of the sample structure, they form corresponding textures on the

image, which can be traced back to the fundamental physical processes in the material. A general workflow

which could incorporate a diverse set of prior knowledge, use it for pre-processing or dataset decomposition

and relate the results back to the expected model is hence instrumental for in-depth analysis of complex

systems with multimodal scanning probe microscopy-based techniques. The ability to extract specific

physical parameters and clearly indicate regions with similar properties must be effectively realized in the

workflow. For example, material may exhibit domains with different orientations which could indicate

apparent epitaxial effects. Therefore, machine-learning approaches for understanding of underlying physics

have to efficiently segment the datasets. The framework for data analysis should rely on expert insights into

5

materials organization hence narrowing the feature finding algorithm accordingly. In this light, a supervised

machine learning is a natural choice for processing of AFM structural data.

The particular mechanism of extracting order parameters plays pivotal role in the workflow. In this

paper, we provide systematic outlook on the structural descriptors and supply visualization tool necessary

to evaluate the output of the workflow. Here, we expand on the arsenal of implemented techniques for

processing of AFM images. This expands on our previous works in this field and allows to tailor the analysis

for a specific system. In Supplementary sections 1 we provide a detailed outlook on the structural

decriptors. Our approach takes full advantage of the unique advantages provided by open-source software.

It can further incorporate other methods as dictated by a specific analytical task. Furthermore we create

visualization tools which will accelerate the exploration and deployment of our approach.

Figure 1. An example of a multidomain AFM image (A). A EMI-TFSI ionic liquid displays self-assembling pattern

on a graphite surface preferential crystallographic orientation. (B, D) Zoom-ins show domains have a well-defined

local structure which changes across the image. (C, E) A structure descriptor (in this case, the absolute value of Fast

Fourier transform of the windowed image follows this change and serves as the basis for classifier. (F) The resulting

output of a local descriptor generator overlaid over the original AFM image. Red, green and blue colors designate the

orientation of the corresponding domain of the ionic liquid.

An example of a multidomain molecular resolved AFM image is shown in Figure 1A. The first

adsorbed ion layer of 1-ethyl-3-methylimidazolium bis(trifluoromethylsulfonyl)imide (EMI-TFSI) ionic

liquid forms domains with ordered structure with three preferred orientations directed at 120 relative to

each other. The domains are irregularly shaped and there is no contrast difference between them, which

renders standard particle analyzing software unsuitable for this case. We expect that the orientation of the

lines on the image is the factor that separates domains from each other (Figure 1B, D), hence, it can be

used to segment the AFM image. A well-designed structure descriptor can summarize the relevant

information in the vicinity of a given point and serve as the basis for subsequent analysis. Using Fast Fourier

6

transform on an image within a given window is a well-known example of such descriptor.17, 28 The

translational invariance of the absolute values of FFT is particularly useful here as its output is insensitive

to the choice of the starting point within a uniform pattern. Crystallographic domains have clearly different

FFTs (Figure 1C, E) which preserve information about ordering type, spacing and orientation. Absolute

values of FFT smoothly change as the window is sliding across the image, for example, a border between

two domains will display both orientations. It is worth noting that while FFT is a suitable choice in many

cases, other transforms can be chosen as alternatives – as long as they support the extraction of relevant

physical parameters. The resulting output of a local descriptor generator overlaid over the original AFM

image can be seen in Figure 1F.

Extraction of structure descriptors in each point inevitably leads to the redundancy of the analysis.

For example, calculating FFT in each sliding window transforms a 2D image into a 4D dataset. In order to

tackle this issue, a dimensionality reduction technique must be used to select the most informative subset

which is chosen according to existing prior knowledge of a system.29-31 This consideration heavily impacts

the following steps of the analysis. For instance, if If periodicity of a pattern is continuously changing

throughout a section of an image, it would be difficult to represent any given FFT as a linear mixture of a

limited set or archetypical FFTs. In a more common case of potentially overlapping structural types, the

linearity of the structural descriptors mixing should be directly utilized. Depending of the exact nature of a

chosen descriptor, a non-negativity or orthogonality constraints can be incorporated in the dimensionality

reduction. Thus, application of principal component analysis32 or non-negative matrix factorization is done

during the second step. The typical output expected here is a series of maps which display the intensity of

a given component in each point which is interpreted as a measure of the likelihood between the structural

archetypes and localized arrangements exhibited in the vicinity of a given pixel.

When these structure type maps are extracted, we classify the pixels in an image based on the values

found in those maps. A criterion for this classification is also system dependent. One can imagine an

example of mutually exclusive or, on contrary, co-existing structural types. At this point, the final

assignment of a pixel in the original AFM image is assigned to a domain belonging to a specific structure.

Overall, the logic of the algorithm presented in this paper has four steps; 1) selection of a structure

descriptor, 2) dimensionality reduction of the resulting dataset and extraction of abundance maps, 3)

construction of a feature space, 4) assignment of the image pixels to a relevant structure type (Figure S1).

Ideally, the approach should be able to support wide range of systems and provide the user with a significant

level of control over the algorithm but require limited supervision once the initialization is complete. More

details on dimensionality reduction techniques is available in Supplementary section 2.

7

One of the advantages of our framework is that the pattern abundances generated by any method

are processed within the same pipeline. It is evident that all four of these methods generate satisfactory

results (Supplementary section 2). They perform structure-based segmentation of the image once the

supervised selection of the patterns is complete. This offers users an ability to control pattern processing

without necessity to perform training of neural networks or constructing case-specific filter banks. At the

same time, the existing knowledge of the system in question is naturally incorporated into the image

segmentation. Considering the higher throughput of the machine learning-based approaches, our framework

is particularly suitable for batch processing of the AFM data. In this paper, we will highlight the methods

referred to as FFT-PCA where Fast Fourier transform is followed by the principal component analysis.

Once the pattern abundance maps are extracted, they need to be further processed to finalize the

segmentation. Gaussian blur can be used to denoise the abundance maps. Then, we can use the intensities

of these maps in specific points to build a feature space (Figure 2). Dense clusters found spanning along

the coordinate lines correspond to the points which have one pattern dominating their structural descriptor.

The lines connecting these dense clusters appear due the toe fact that a series of points located on the

boundary of two domains their structural descriptors are changing continuously. This is well illustrated on

Figure 1C,E where absolute values of the FFT of the sliding window on the boundary are linear mixtures

of absolute values of the FFT taken within neighboring domains. The fact that with the exception of the

boundary the majority of the points are aligned along certain axis indicates that in this case the patterns are

mutually exclusive. In principle, the presence of dense clusters with other directions (or even more complex

shapes) will indicate presence of image domains with combined types of periodicity. This feature space

abstraction allows to study the typological view of the initial dataset.

8

Figure 2. Processing of (A, B, C) the abundance maps (here, they correspond to three different orientations of the

domains of ionic liquid) and replotting the dataset in the (D) feature space. Here the normalized intensities of the

respective component in a given point are taken as coordinates of that point. Ultimately, each point of the (E) original

map is colored according the (F) segmentation of the feature space– in this case, it was done by the coordinate of the

point.

The consequent image segmentation is done in the feature space. The exact rule applied for this

task can be selected based on the layout of the dataset. For example, the points can be assigned based on

their coordinate (Figure 2 E,F). The position in the feature space is the measure of likelihood between the

pattern in the vicinity of the point and the archetypical pattern provided by structure descriptor extraction.

Alternatively, a clustering algorithm can be used to find the characteristic behaviors. For example, K-Means

clustering finds subdomains with different pattern intensities as well as highlights the domain boundaries

(Supplementary section 3). This way the thickness of a boundary can be used as a measure of how well

the structural patterns are pronounced. Finally, the clusters generated by this method can be further assigned

based on the position of their centroids to generate high-level maps (Supplementary section 3). The output

of such workflow is very similar to the segmentation using feature space coordinates.

Overall, automated extraction of features provides user with the ability to view the dataset in a

pattern-centered manner. Different parts of the feature space then correspond to different structural types

and hence the segmentation of this space serves as an efficient basis for the segmentation of the original

image. The supplementary notebook highlights the ability to perform extraction of the domain-specific

statistics such as area, border length and spacings for each domain of every orientation.

Flexibility of this framework can be illustrated by further expanding the feature space. For example,

a separate ‘null” pattern can be used to highlight the absence of patterns of interest. If well-defined domains

of ionic liquids located on a graphite substrate do not cover the surface completely, then feature space needs

to be expanded to incorporate the absence of a pattern. In can be done by taking first FFT-PCA component

or (in this case) a separate source of information such as deflection channel. Segmentation of the feature

space is done using the same approach as before – thresholding or clustering are both applicable. Note, that

small domains containing few features have smaller intensity than the large ones due, so there is an optimal

size of the sliding window that needs to be selected before the algorithm is run. Increasing the number of

repeating units within the window better extracts the characteristic patterns, at the same time, window that

is too big will affect small domains (Figure 3B). Second issue that may arise is related to the domains with

the same pattern located near each other. As shown on Figure 3B, domains in the bottom left highlighted

in purple were found to belong to the same segment.

9

Figure 3. (A, B) Image segmentation in case where pixels may belong to one of three patterns or to a “null” pattern

which is typically a background exemplified by BMI-TFSI (A) and PYR14-TFSI (B) ionic liquids on HOPG surface.

(C, D) Example of more complex texture segmentation: tapping mode AFM scans of molecular domains of melem

assembled on hexagonal boron nitride.

1-Butyl-3-methylimidazolium bis(trifluoromethylsulfonyl)imide (BMI-TFSI) (Figure 3A) forms

patterns covering about 90 % of highly oriented pyrolytic graphite (HOPG) surface, and 1-butyl-1-

methylpyrrolidinium bis-(trifluoromethanesulfonyl)imide (PYR14-TFSI) ionic liquids cover about 60 % of

10

the surface (Figure 3B) serving as examples of images requiring consideration of a background “null”

pattern. The segmentation can be further refined if specific rules about domain shapes are given.

Furthermore, the workflow presented here can be even expanded to the cases with arbitrary patterns

– for example, for analysis of melem which is thought to be one of the possible precursors for graphitic

carbon nitride (g-C3N4). This material has been proposed as a photocatalyst for splitting water into H2 and

O2.33 On surface thermal or photoinduced decomposition of melem is one of the possible synthetic routes

to g-C3N4 and similar 2D carbon nitrides. Here, the segmentation highlights ordered domains of melem on

hexagonal boron nitride (Figure 3C, D).

Figure 4. (A) Image segmentation based on the preferential orientation of FFT pattern, (B) segmentation based on the

algorithm reported here, (C) K-means clustering-based segmentation used for domain assignment. It is clearly seen

11

that while simple FFT orientation can be used for AFM processing, (D) it fails to capture the structural subtypes

observed in the material – in this case, the orientation of melem. (E) Our algorithm allows to isolate the domains and

(F) borders between domains.

While domain assignment can be done using other types of processing34, it sometimes may produce

segmentation that displays the clear signs of overfitting. For example, in Figure 4A the orientation of local

FFT pattern was used to highlight regions with the same pattern direction and type. The output of our

workflow is shown on panels B and C. While FFT orientation generates some reasonable segmentation, it

also features multiple cases of incorrect assignment (Figure 4D). At the same time, using K-means-based

clustering of the feature space it is possible to select both domains (E) and domain borders (F). In addition,

our workflow allows for the collection of domain statistics and is presented in Figure S12. A shape, area,

boundary length and the structure of each domain can be gathered and processed. Then, the individual

elements of a segmented image can be analyzed, in this case, we highlight the FFT of a single domain. The

spacing of the FFT pattern as well as its orientation can be directly accessed (Supplementary section 3).

By applying this procedure to each domain on the image, a comprehensive characterization of the AFM

image can be provided (Supplementary section 3).

Here we present a robust workflow for image analytics that allows incorporating the elements of

physical constraints in the system. By using the modularity of our approach, it is possible to combine

system-specific structural descriptors and dimensionality reduction techniques and relate the segmentation

of the feature space to the segmentation of the original image. It allows to thoroughly investigate samples

in the absence of large training datasets and gather relevant statistics on the identified domains.

Furthermore, it establishes a user-independent workflow that can be used to perform the quantitative

comparison within a series AFM images.

This processing tool can be further expanded to accommodate other special cases of the AFM image

processing. Due to the freedom of structural descriptor choice, any additional unmixing constraints (such

as statistical independence, non-negativity, orthogonality and/or regularization terms) with can be added

into the workflow. In principle, it supports import of known endmembers/eigenvectors and custom penalties

for the decomposition thus allowing to explicitly introduce the physical understanding of the system into

the image analysis. Similarly, a dimensionality reduction can be custom selected to consider the non-linear

cases. Kernel-based PCA or such manifold learning tools as t-SNE or UMAP can be used to generate the

feature space. Here, a choice of specific kernel/distance metric is dictated by the physics of a specific

sample. Finally, the shape of the dataset in the feature space (or several feature spaces generated by different

approaches) is a valuable exploration tool as it reveals the internal organization of the information in the

dataset. The co-existence of specific features in a given set of points across the image can be easily

12

highlighted. As a result, this processing workflow can be used for the hypothesis testing which combines

the machine learning throughput with the fundamentals of the samples being studied.

Supplementary Material

Online supplementary material which includes materials and methods, overview of structural descriptors

and dimensionality reduction techniques, as well as components loading maps for different methods is

available at XXX. The Jupyter notebook link is provided in the Code availability section.

Acknowledgements

Algorithm development was conducted at the Center for Nanophase Materials Sciences, which is a DOE

Office of Science User Facility, and using instrumentation within ORNL's Materials Characterization Core

provided by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of

Energy. AFM measurements were supported by the Fluid Interface Reactions, Structures and Transport

(FIRST) Center, an Energy Frontier Research Center (EFRC) funded by the U.S. Department of Energy,

Office of Science, Office of Basic Energy Sciences (W.-Y.T., N.B.). The experiments and sample

preparation in this work were performed and supported at the Center for Nanophase Materials Sciences in

Oak Ridge National Lab. Authors thank Peter H. Beton for providing melem samples.

Notice: This manuscript has been authored by UT-Battelle, LLC, under Contract No. DE-

AC0500OR22725 with the U.S. Department of Energy. The United States Government retains and

the publisher, by accepting the article for publication, acknowledges that the United States

Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or

reproduce the published form of this manuscript, or allow others to do so, for the United States

Government purposes. The Department of Energy will provide public access to these results of

federally sponsored research in accordance with the DOE Public Access Plan

(http://energy.gov/downloads/doe-public-access-plan).

Data Availability

Scanning probe microscopy data used for the analysis are available at:

github.com/nickborodinov/multidomainprocessing/

Code Availability

Python scripts used for the analysis are available at:

13

github.com/nickborodinov/multidomainprocessing/

https://colab.research.google.com/drive/1xIiXXIwHHY8DE-EzSL0idxthkYVNKqR-

Competing interests

The authors declare no competing interests.

Author contributions

N.B. wrote the manuscript, developed Python scripts and compiled Jupiter notebook deployable in Google

Colaboratory. S.V.K. proposed the concept of the linear physics-based workflow for image analytics and

co-wrote the manuscript. O.S.O. has proposed the usage of multicomponent analysis for soft matter. W.-Y.

T. and N. Balke designed and carried out the ionic liquid related AFM experiments. W.-Y.T. verified the

code with other datasets (not presented in this manuscript) and gave feedback to help developing the code.

V.V.K. and P.H.B. designed and carried out the melem related AFM experiments.

References

1. Kalinin, S. V.; Strelcov, E.; Belianinov, A.; Somnath, S.; Vasudevan, R. K.; Lingerfelt, E. J.;

Archibald, R. K.; Chen, C.; Proksch, R.; Laanait, N.; Jesse, S., ACS Nano 2016, 10 (10), 9068-9086. 2. Yong, Y. K.; Bazaei, A.; Moheimani, S. O. R., Ieee Transactions on Nanotechnology 2014, 13 (1),

85-93.

3. Bazaei, A.; Yong, Y. K.; Moheimani, S. O. R., Ieee-Asme Transactions on Mechatronics 2017, 22

(1), 371-380. 4. Qin, B.; Zhang, S.; Huang, Z. H.; Xu, J. F.; Zhang, X., Macromolecules 2018, 51 (5), 1620-1625.

5. Sigdel, K. P.; Wilt, L. A.; Marsh, B. P.; Roberts, A. G.; King, G. M., Biochem Pharmacol 2018,

156, 302-311. 6. Jesse, S.; Chi, M.; Belianinov, A.; Beekman, C.; Kalinin, S. V.; Borisevich, A. Y.; Lupini, A. R.,

Sci Rep 2016, 6, 26348.

7. Belianinov, A.; Vasudevan, R.; Strelcov, E.; Steed, C.; Yang, S. M.; Tselev, A.; Jesse, S.; Biegalski, M.; Shipman, G.; Symons, C.; Borisevich, A.; Archibald, R.; Kalinin, S., Adv Struct Chem Imaging 2015,

1 (1), 6.

8. Ziatdinov, M.; Maksov, A.; Li, L.; Sefat, A. S.; Maksymovych, P.; Kalinin, S. V., Nanotechnology

2016, 27 (47), 475706. 9. Ziatdinov, M.; Fujii, S.; Kiguchi, M.; Enoki, T.; Jesse, S.; Kalinin, S. V., Nanotechnology 2016, 27

(49), 495703.

10. Belianinov, A.; Ganesh, P.; Lin, W. Z.; Sales, B. C.; Sefat, A. S.; Jesse, S.; Pan, M. H.; Kalinin, S. V., Apl Materials 2014, 2 (12), 120701.

11. Lin, W.; Li, Q.; Sales, B. C.; Jesse, S.; Sefat, A. S.; Kalinin, S. V.; Pan, M., ACS Nano 2013, 7 (3),

2634-41. 12. Vasudevan, R. K.; Tselev, A.; Baddorf, A. P.; Kalinin, S. V., ACS Nano 2014, 8 (10), 10899-908.

13. Strelcov, E.; Belianinov, A.; Hsieh, Y. H.; Jesse, S.; Baddorf, A. P.; Chu, Y. H.; Kalinin, S. V.,

ACS Nano 2014, 8 (6), 6449-57.

14. Vasudevan, R. K.; Belianinov, A.; Gianfrancesco, A. G.; Baddorf, A. P.; Tselev, A.; Kalinin, S. V.; Jesse, S., Applied Physics Letters 2015, 106 (9), 091601.

14

15. Gai, Z.; Lin, W.; Burton, J. D.; Fuchigami, K.; Snijders, P. C.; Ward, T. Z.; Tsymbal, E. Y.; Shen,

J.; Jesse, S.; Kalinin, S. V.; Baddorf, A. P., Nature Communications 2014, 5, 4528. 16. Lin, W.; Li, Q.; Belianinov, A.; Sales, B. C.; Sefat, A.; Gai, Z.; Baddorf, A. P.; Pan, M.; Jesse, S.;

Kalinin, S. V., Nanotechnology 2013, 24 (41), 415707.

17. Li, Q.; Lin, W.; Yan, J.; Chen, X.; Gianfrancesco, A. G.; Singh, D. J.; Mandrus, D.; Kalinin, S. V.;

Pan, M., Nat Commun 2014, 5, 5358. 18. Panova, O.; Ophus, C.; Takacs, C. J.; Bustillo, K. C.; Balhorn, L.; Salleo, A.; Balsara, N.; Minor,

A. M., Nat Mater 2019.

19. Kumar, A.; Ovchinnikov, O.; Guo, S.; Griggio, F.; Jesse, S.; Trolier-McKinstry, S.; Kalinin, S. V., Physical Review B 2011, 84 (2), 024203.

20. Ovchinnikov, O. S.; Jesse, S.; Bintacchit, P.; Trolier-McKinstry, S.; Kalinin, S. V., Phys Rev Lett

2009, 103 (15), 157203. 21. Nikiforov, M. P.; Reukov, V. V.; Thompson, G. L.; Vertegel, A. A.; Guo, S.; Kalinin, S. V.; Jesse,

S., Nanotechnology 2009, 20 (40), 405708.

22. Tiryaki, V. M.; Adia-Nimuwa, U.; Ayres, V. M.; Ahmed, I.; Shreiber, D. I., Cytometry A 2015, 87

(12), 1090-100. 23. Borodinov, N.; Neumayer, S.; Kalinin, S. V.; Ovchinnikova, O. S.; Vasudevan, R. K.; Jesse, S.,

Npj Computational Materials 2019, 5 (1), 25.

24. Somnath, S.; Smith, C. R.; Kalinin, S. V.; Chi, M.; Borisevich, A.; Cross, N.; Duscher, G.; Jesse, S., Adv Struct Chem Imaging 2018, 4 (1), 3.

25. Vasudevan, R. K.; Ziatdinov, M.; Jesse, S.; Kalinin, S. V., Nano Lett 2016, 16 (9), 5574-81.

26. Essafi, S.; Langs, G.; Deux, J.; Rahmouni, A.; Bassez, G.; Paragios, N. In Wavelet-driven knowledge-based MRI calf muscle segmentation, 2009 IEEE International Symposium on Biomedical

Imaging: From Nano to Macro, 28 June-1 July 2009; 2009; pp 225-228.

27. Luciani, X.; Patrone, L.; Courmontagne, P., Journal De Physique Iv 2006, 132, 237-241.

28. Belianinov, A.; He, Q.; Kravchenko, M.; Jesse, S.; Borisevich, A.; Kalinin, S. V., Nat Commun 2015, 6, 7801.

29. Ievlev, A. V.; Susner, M. A.; McGuire, M. A.; Maksymovych, P.; Kalinin, S. V., ACS Nano 2015,

9 (12), 12442-50. 30. Kannan, R.; Ievlev, A. V.; Laanait, N.; Ziatdinov, M. A.; Vasudevan, R. K.; Jesse, S.; Kalinin, S.

V., Adv Struct Chem Imaging 2018, 4 (1), 6.

31. Strelcov, E.; Belianinov, A.; Hsieh, Y. H.; Chu, Y. H.; Kalinin, S. V., Nano Lett 2015, 15 (10),

6650-7. 32. Jesse, S.; Kalinin, S. V., Nanotechnology 2009, 20 (8), 085714.

33. Wang, X.; Maeda, K.; Thomas, A.; Takanabe, K.; Xin, G.; Carlsson, J. M.; Domen, K.; Antonietti,

M., Nat. Mater. 2009, 8 (1), 76-80. 34. Korolkov, V. V.; Summerfield, A.; Murphy, A.; Amabilino, D. B.; Watanabe, K.; Taniguchi, T.;

Beton, P. H., Nat. Commun. 2019, 10 (1), 1537.

download fileview on ChemRxivMachine learning-based multidomain processing for text... (906.36 KiB)

Supplementary information for

Machine learning-based multidomain processing for AFM texture-based image

segmentation and analysis

Nikolay Borodinov1, Wan-Yu Tsai1, Vladimir V. Korolkov2, Nina Balke1, Sergei V. Kalinin1,

Olga S. Ovchinnikova1*

1Center for Nanophase Materials Science, Oak Ridge National Laboratory, Oak Ridge,

Tennessee 37831, United States

2School of Chemistry, University of Nottingham, University Park, Nottingham, NG7 2RD,

UK.

* Author to whom correspondence should be addressed.

Olga S. Ovchinnikova

Center for Nanophase Materials Sciences

Oak Ridge National Laboratory

1 Bethel Valley Rd

Oak Ridge TN, 37831-6493

ovchinnikovo@ornl.gov

Notice: This manuscript has been authored by UT-Battelle, LLC, under Contract No. DE-

AC0500OR22725 with the U.S. Department of Energy. The United States Government retains and

the publisher, by accepting the article for publication, acknowledges that the United States

Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or

reproduce the published form of this manuscript, or allow others to do so, for the United States

Government purposes. The Department of Energy will provide public access to these results of

federally sponsored research in accordance with the DOE Public Access Plan

(http://energy.gov/downloads/doe-public-access-plan).

Materials and Methods

Data analysis

Data processing was done using Python 3.6 using scikit-learn 0.19.2 library. All calculations were

performed on a desktop computer with Intel Xeon CPU E-5-1650 v3 3.50 GHz processor and 40 GB of

RAM were used to perform the computations.

AFM dataset acquisition

The AFM experiments were conducted using 1-Ethyl-3-methylimidazolium

bis(trifluoromethylsulfonyl)imide (EMI-TFSI), 1-Butyl-3-methylimidazolium

bis(trifluoromethylsulfonyl)imide (BMI-TFSI) or 1-butyl-1-methylpyrrolidinium bis-

(trifluoromethanesulfonyl)imide (PYR14-TFSI) ionic liquid on an atomically flat freshly-cleaved highly

ordered pyrolytic graphite (HOPG). Grade ZYA highly ordered pyrolytic graphite (HOPG) with Mosaic

spread angle of 0.4 ± 0.1° was purchased from SPI Supplies. The AFM images were collected using a

commercial AFM (Cypher, Asylum Research an Oxford Instruments Company) with tapping mode. The

tip oscillation amplitude in bulk liquid was ~3 nm, and the setpoint oscillation amplitude was between 30%

and 50% of this value. The AFM tips used in this study are Au-coated silicon nitride tips with nominal

spring constant k of 0.6 N/m (calibrated using thermal noise method and Sader method).

Section 1. Capturing local structure using local descriptors

While the texture is an inherently non-local property as it is related to the mutual arrangement of

building blocks spanning over some space, the ultimate goal of the image domain finding is to assign each

point with the structural type and generate the summary of its physical properties. This constitutes an

example of uncertainty principle as we explore the simultaneous analysis in real and reciprocal space.

If the characteristic scale of the ordering is known, a series of filters can be readily applied to solve

the pattern recognition problem. Usage of Gabor filters has been considered as a reliable and efficient

solution for such problem.1 Here, we expand this approach and use stationary wavelet decomposition as a

tool for extracting regions where a given type of pattern is observed. Generally, wavelet decomposition has

a unique advantage of preserving information both in frequency and space.2-3 Stationary wavelet

decomposition (SWD), while being redundant, allows for straightforward interpretation of the components

as they have the same dimension as the original image. Thus, selecting an appropriate detail scale of the

SWD, it is possible to detect a given type and periodicity (Figure S2,3). For example, a series of horizontal

lines on the image separated by spacing close to the dimensions of a corresponding wavelet, will have a

strong signal in the component corresponding to the vertical details of the image. If the original image is

rotated, the domains of different orientations will become well-pronounced at different angles which will

correspond to the directionality of the pattern. Overall, to effectively use this approach one has to design a

filter bank that can segment image of a given pattern type considering scaling and rotations found in the

data. While SWD performs well in general case, more specific approaches can be used to select relevant

features if the pattern type belong to a specific category of interest.

Incorporating the knowledge about the periodicity type allows the usage of structure-oriented data

transforms. For example, Radon transform (RT) can be efficiently used to analyze spatially ordered data.1

Here, all angle-dependent features are presented on the same image, thus, it works well for capturing 1D

periodicity (Figure S4,5). Fast Fourier transform (FFT) extends the capability of ordering characterization

to arbitrarily complex patterns and hence is widely applied to the processing of microscopy data (Figure

S6-9).4 To use RT or FFT for local ordering analysis, the image is represented as a set of windowed sections

centered around the central point. This approach is commonly referred to as sliding window method. The

overall number of sections is proportional to the squared number of steps.

Finally, if the system to be characterized is well-understood, the information about its structure can

be explicitly used in the workflow. For example, generalized Hough transform can be used to find instances

of objects of a given shape.5 Another way to incorporate the ordering type into the analysis is to use a given

metric to measure the apparent local structural descriptor with the expected or pre-computed archetypes.

By taking absolute values of selected physically relevant FFT components and cross-correlating (CC) them

to the absolute values of sliding window FFT will generate continuous abundance map where 0 would

correspond to a pattern not observed in the point and 1 corresponding to the component being pronounced

to a maximum extent. This method readily generates maps that do not require post-processing and are easy

to interpret (supplementary notebook, Figure S10). Extraction of these descriptors with FFT can be done

with very sparse sampling.

Overall, the selection of a structural descriptor heavily depends on the system and the information

about ordering in that system. Here, we are using SWD, RT, FFT and FFT-guided CC approaches, but the

workflow can incorporate other methods as well as long as it generates information about local structural

type.

Section 2. Dimensionality reduction of the feature space

The multidimensional datasets generated by the transforms are difficult to directly interpret, and

often have highly redundant and physically-unconstrained components. Compressing these datasets into a

format that can be used for the examination of the system is hence critical for the workflow. The exact

choice of the dimensionality reduction techniques is driven by the type of data and physical constraints,

which again invokes the usage of our knowledge of the material and its structure. Principal component

analysis (PCA) is one of the most well-known methods which perfectly works for the output of FFT or RT

sliding window transform which are 4D datasets.6 Here, the components are constrained to be orthonormal

and are ranked according to their explained variance. It is worth noticing that the PCA ranks eigenvectors

based on the variance. First component is typically the averaged map, and pattern-specific components

follow together with others corresponding to borders and higher order features.

Note, that PCA result strongly depends on the variance of the corresponding data. For example,

absolute values of the FFT for windowed sections are much easier to interpret rather than real (or imaginary

values), and one could assume that using them for the construction of a 4D dataset would be physically

meaningful. However, PCA eigenvectors for such dataset will be a mixture of individual patterns. Taking

absolute values changes the direction of maximum variance for the dataset and hence affects the

decomposition. To tackle this issue, a real part of the FFT can be taken. Now, the oscillations of the real

part of the FFT will boost the variance along the directions corresponding to the patterns presented in the

image. This highlights that using PCA for decomposition of absolute value datasets does not result in

efficient decoupling of identified behaviors.

Incorporating non-negativity, a natural constraint for many signals, can be achieved using non-

negative matrix factorization (NMF). Here, a given number of endmembers are expected to produce any

spectrum in the dataset by linear mixing. Considering that such types of spectra (mass spectra, FTIR, NMR)

are found quite often in scientific analysis, NMF becomes a very popular tool. For example, for the case of

stationary wavelet decomposition using NMF makes perfect sense. It is possible to take the full 180°

rotation of the original image and then take the absolute values of the SWD components. The endmembers

of this NMF model will correspond to the angles of rotations that match the orientation of individual

textures. The example of SWD-NMF applied to the dataset shown in Figure 1 is presented in the

supplementary notebook. The areas corresponding to a specific type of pattern will exhibit strong

oscillations of the corresponding component which can be converted to absolute values and smoothed out.

These oscillations are the result of interference between wavelet function and periodic image which alters

between constructive and destructive type within the pattern.

Other types of dimensionality reduction can be used if warranted by the nature of the measured

data. Independent component analysis (ICA)7 separates dataset into non-Gaussian components that are

statistically independent of each other.

Using Bayesian linear unmixing (BLU)8 allows to provide a probabilistic outlook on the non-

negative data. A N-dimensional geometric unmixing methods can also be used. For example, for N-FINDR

the spectra in the dataset are expected to be linear combinations of N endmembers such that corresponding

weights are constrained to be summed up to 1. While considering the shape of the multidimensional

datasets, three convexity-based criteria, orthogonal projection (OP), convex cone/hull, and simplex are

generally used for this task.9 These criteria physically correspond to having no constraints on the

abundances (pixel purity index, PPI)10, imposing non-negativity only (vertex component analysis)11 or

imposing both sum-to-one and non-negativity (N-FINDR)12, respectively.

Finally, for the cases of non-linear mixing kernel-based approaches13-14 or manifold learning15-17

can be used if the similarity metric for a given pair of spectra is well-known. Overall, a wide selection of

dimensionality reduction techniques is available for a very diverse set of system. Here, we use PCA for

FFT, RT and CC and NMF for SWD, however, the overall workflow can incorporate any other suitable

method.

Figure S1. An overview of the workflow presented in this paper. Note that both the stages of structure descriptor

selection and dimensionality reduction allow for connection with the physics of the system. At the descriptor stage,

this is accomplished via a priori chosen descriptor shape (e.g. defining periodicity in 2D, 1D, or presence of specific

geometries) containing partial knowledge on system being explored. At the dimensionality reduction stage, this is

incorporated via hard or soft constraints on the end members (non-negative, sum to one, symmetry) and abundance

maps (sparsity). Note that availability of multiple possible feature selection and dimensionality reduction states in turn

enables hypothesis testing and uncertainty quantification as a part of the process.

Figure S2. Intensity of SWD vertical component of level 1 (as denoted in PyWavelets documentation) as the image

is rotated. Here we used biorthogonal 5.5 wavelet. The patterns with specific directionality are highlighted at certain

rotation angles.

Figure S3. SWD-NMF loading maps for three patterns with different orientation, their absolute values and the effect

of blurring.

Figure S4. RT-FFT loading maps and corresponding eigenvectors.

Figure S5. RT-PCA loading maps for three patterns with different orientation, their absolute values and the effect of

blurring.

Figure S6. FFT-PCA loading maps and corresponding eigenvectors.

Figure S7. FFT-PCA eigenvectors, eigenvector FFTs, loading maps and absolute values of loading maps when

absolute values of FFTs are considered.

Figure S8. FFT-PCA eigenvectors, eigenvector FFTs, loading maps and absolute values of loading maps when real

values of FFTs are considered.

Figure S9. FFT-PCA loading maps for three patterns with different orientation, their absolute values and the effect of

blurring.

Figure S10. FFT-PCA-CC loading maps for three patterns with different orientation, their absolute values and the

effect of blurring.

Section 3. Processing data in the feature space.

Figure S11. Pattern feature space for image segmentation. (A, B) The pixels of the original image are colored based

on (C) the coordinate in feature space, (D) K-Mean cluster assignment. (E, F).The centroids of K-Means clusters also

can be used for the pixel assignment.

Figure S12. The statistics gathered using the workflow presented in the paper: (A) the original segmented image, (B)

the selection of the domains with specific orientation, (C) selection of a selected single domain, (D) FFT of the selected

domain, which can be further processed using (E) radial and (F) angular integration. The overall spacing statistics is

collected and shown in panel G.

References

1. Orlov, N.; Shamir, L.; Macura, T.; Johnston, J.; Eckley, D. M.; Goldberg, I. G., WND-CHARM:

Multi-purpose image classification using compound image transforms. Pattern Recognition Letters 2008,

29 (11), 1684-1693.

2. Workman, M. J.; Serov, A.; Halevi, B.; Atanassov, P.; Artyushkova, K., Application of the Discrete Wavelet Transform to SEM and AFM Micrographs for Quantitative Analysis of Complex Surfaces.

Langmuir 2015, 31 (17), 4924-33.

3. Maksumov, A.; Vidu, R.; Palazoglu, A.; Stroeve, P., Enhanced feature analysis using wavelets for scanning probe microscopy images of surfaces. J Colloid Interface Sci 2004, 272 (2), 365-77.

4. Vasudevan, R. K.; Ziatdinov, M.; Jesse, S.; Kalinin, S. V., Phases and Interfaces from Real Space

Atomically Resolved Data: Physics-Based Deep Data Image Analysis. Nano Lett 2016, 16 (9), 5574-81. 5. Wang, Y.; Lu, T.; Li, X.; Ren, S.; Bi, S., Robust nanobubble and nanodroplet segmentation in

atomic force microscope images using the spherical Hough transform. Beilstein J Nanotechnol 2017, 8,

2572-2582.

6. Belianinov, A.; He, Q.; Kravchenko, M.; Jesse, S.; Borisevich, A.; Kalinin, S. V., Identification of phases, symmetries and defects through local crystallography. Nat Commun 2015, 6, 7801.

7. Beckmann, C. F.; Smith, S. M., Probabilistic independent component analysis for functional

magnetic resonance imaging. IEEE Trans Med Imaging 2004, 23 (2), 137-52. 8. Dobigeon, N.; Tourneret, J.; Hero, A. O. In Bayesian linear unmixing of hyperspectral images

corrupted by colored Gaussian noise with unknown covariance matrix, 2008 IEEE International

Conference on Acoustics, Speech and Signal Processing, 31 March-4 April 2008; 2008; pp 3433-3436. 9. Chang, C. I.; Chen, S. Y.; Li, H. C.; Chen, H. M.; Wen, C. H., Comparative Study and Analysis

Among ATGP, VCA, and SGA for Finding Endmembers in Hyperspectral Imagery. Ieee Journal of

Selected Topics in Applied Earth Observations and Remote Sensing 2016, 9 (9), 4280-4306.

10. Chang, C. I.; Plaza, A., A fast iterative algorithm for implementation of pixel purity index. Ieee Geoscience and Remote Sensing Letters 2006, 3 (1), 63-67.

11. Nascimento, J. M. P.; Dias, J. M. B., Vertex component analysis: A fast algorithm to unmix

hyperspectral data. Ieee Transactions on Geoscience and Remote Sensing 2005, 43 (4), 898-910. 12. Xiong, W.; Chang, C. I.; Wu, C. C.; Kalpakis, K.; Chen, H. M., Fast Algorithms to Implement N-

FINDR for Hyperspectral Endmember Extraction. Ieee Journal of Selected Topics in Applied Earth

Observations and Remote Sensing 2011, 4 (3), 545-564.

13. Kuo, B. C.; Ho, H. H.; Li, C. H.; Hung, C. C.; Taur, J. S., A Kernel-Based Feature Selection Method for SVM With RBF Kernel for Hyperspectral Image Classification. Ieee Journal of Selected Topics in

Applied Earth Observations and Remote Sensing 2014, 7 (1), 317-326.

14. Di, W.; Crawford, M. M., Active Learning via Multi-View and Local Proximity Co-Regularization for Hyperspectral Image Classification. Ieee Journal of Selected Topics in Signal Processing 2011, 5 (3),

618-628.

15. Ma, L.; Crawford, M. M.; Tian, J. W., Anomaly Detection for Hyperspectral Images Based on Robust Locally Linear Embedding. J. Infrared Millim. Terahertz Waves 2010, 31 (6), 753-762.

16. Li, X.; Dyck, O. E.; Oxley, M. P.; Lupini, A. R.; McInnes, L.; Healy, J.; Jesse, S.; Kalinin, S. V.,

Manifold learning of four-dimensional scanning transmission electron microscopy. Npj Computational

Materials 2019, 5 (1), 5. 17. Li, X.; Collins, L.; Miyazawa, K.; Fukuma, T.; Jesse, S.; Kalinin, S. V., High-veracity functional

imaging in scanning probe microscopy via Graph-Bootstrapping. Nat Commun 2018, 9 (1), 2428.

download fileview on ChemRxivESI Machine learning-based multidomain processing for te... (1.84 MiB)

Other files

download fileview on ChemRxivMachine_learning_based_multidomain_processing_for_t... (41.07 MiB)