PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing -

Machine Learning in Image Processing

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Feature Classes

Feature Classes

There are basically two different ML tasks:

1. Classification ⇒ Symbolic / categorical and discrete target feature variables
2. Regression ⇒ Numeric and continuous target feature variables

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Feature Classes

Feature Classes

Examples for categorical features:

• Damage (boolean decision, classification of damages like cracks, delaminations, and so on)
• Quality assessment (boolean decision, grade levels, class A/B/C, and so on)
• Geometrical objects (shapes, like lines, circles, ellipses)

Examples for numerical features:

• Material density, average pore size and/or density, crack length and/or density, and so on
• Mechanical properties (stiffness, homogeneity, and so on)
• Predictive lifetime
• Statistical aggregates (noise, average inhomogeneity, average size or density of impurities)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

1. Two-dimensional intensity (photography) images (intensity represents surface reflection or material transmission)
   • Gray-level or multi-channel color (RGB, red, green, and blue channels);
   • Typical dimension 1000 × 1000 pixels;
   • Typical intensity resolution 8 bit (256 levels, gray or RGB), high-quality 16 bit (65536 levels)
   • Volume dimension: height × width × channels
   • Common data file formats: PNG, BMP, TIFF (not JPEG: irreversible compression creating image artifacts)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

2. Three-dimensional tomography images (intensity represents material density)
   • Gray-level
   • Sliced image stack
   • Typical dimension 1000 × 1000 pixels × 1000 (100) pixels;
   • Volume dimension: height × width × depth
   • Common data file formats: numpy, ZIP of tiff files, Vol3, RAW, DICOM (medicine) ..

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

https://www.matsusada.com/column/ct-tech2.html DICOM CT scan data file format merging meta and raw image data

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Target output features can be predicted by the data-driven model basically in two ways:

1. Using raw image input data;
2. Using computed (intermediate) image features.

Examples of computed image features:

• Image intensity distribution, inhomogeneity
• Detection and characterisation of geometrical object shapes (e.g., circles)
• Image transformations, i.e.,
  • wave-number frequency transformation,
  • gradient amplification using convolutional filter operations,
  • binarisation using a threshold

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Example: Pore characterisation in Microsclices of AM parts

(Left) Microgrpah image of a material slice with pores/impurities (Middle) Edge detection using a Canny filter (Right) Shape characterisation by ellipse fitting

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Example: Pore characterisation in X-ray images from die casted plates

(left) Die casted aluminum plate with pores (Center) Single projection X-ray image of plate (Right) Pore marking by semantic pixel classifier (white=pore feature)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Workflow for Object Feature Detection

Workflow for Object Feature Detection

Data: Micrograph images of material slices from rectangular probes proced with Additive Manufacturing technologies (metal powder laser melting).

Object Features: Elliptical pores (impurities) characterised by varying size (axis lengths of ellipse), orientation, density, and spatial distribution (inhomogeneity)

Target Features: Statistical and geometrical characterisation of pores, material density, distribution of defects

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Workflow for Object Feature Detection

Workflow for Object Feature Detection

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Binarization

Image Binarization

$${{I}_{{b}}^{{1}}}{\left({x},{y}\right)}={\left\lbrace\matrix{\matrix{{1}&{I}{\left({x},{y}\right)}\ge{I}_{{{t}{h}{r}}}\\{0}&{I}{\left({x},{y}\right)}<{I}_{{{t}{h}{r}}}}}\right.}\\ {{I}_{{b}}^{{0}}}{\left({x},{y}\right)}={\left\lbrace\matrix{\matrix{{1}&{I}{\left({x},{y}\right)}\le{I}_{{{t}{h}{r}}}\\{0}&{I}{\left({x},{y}\right)}>{I}_{{{t}{h}{r}}}}}\right.}$$

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Binarization

Image Binarization

i1 = load.image('http://edu-9.de/uploads/assets/pores_microslice_ti_1.png',format='RGBA')
m1 = as.matrix(i1,mode='uint8')
print(minMax(m1[100:200,100:200]))
m1.binary = matrix(255,nrow(m1),ncol(m1),mode='uint8')
m1.binary[m1>100]=0
plot(m1.binary)

R+ microslice image pre-processing: Normalization and Binarization (0/255)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

DOI:10.1109/ICSMC.2009.5346873 Canny edge detection algorithm

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

use math,imager,plot
i1 = load.image('http://edu-9.de/uploads/assets/pores_microslice_ti_1.png',format='GRAY')
m1 = as.matrix(i1,mode='uint8')
m1.stats = minMax(m1)
m1.stats$mean = mean(m1)
m1 = (m1 - m1.stats$min)
m1[m1>100] = 255
print(m1.stats)
plot(m1)

R+ microslice image pre-processing: Normalization and Binarization (0/255)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

use math,imager,plot
m1.edges=cannyEdges(m1,t1=50)
plot(m1.edges)

Canny edge filter in R+

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

Point Clustering using DBSCAN

• Try to group points from Canny edges to define a ROI marking a pore candidate

https://medium.com/@agarwalvibhor84/lets-cluster-data-points-using-dbscan-278c5459bee5 Three point classes: Core, Border, Noise

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - ROI Bounding Box Approximation from point list

ROI Bounding Box Approximation from point list

• Output from DBSCAN: List of point groups from canny edge filter

• Calculate rectangular (not rotated) average bounding boxes (pore boundary point group)

An initial ROI (red) positioned at the center of mass of a point cluster is expanded iteratively by increasing one side and computing the point sum along this side. If the line is empty, the next side is expanded.

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - ROI Bounding Box Approximation from point list

ROI Bounding Box Approximation from point list

• Bounding boxes can be computed from a pixel set list, i.e., a list of coordinate vectors

use math, imager, plot
pxs = {
  [1,2],
  [3,9],
  [4,7]
}
pxs.bbox = bbox(pxs)
print(pxs.bbox)

Example of a bbox calculation from a pixel set list

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Ellipse Fitting from point list

Problem: Calculate the parameters of an ellipse equation for a set of boundary points.

Halı́ř, FLusser, Numerically stable direct least squares fitting of an ellipse

General Ellipse Equation

$${F}{\left({x},{y}\right)}={a}{x}^{{2}}+{b}{x}{y}+{c}{y}^{{2}}+{\left.{d}{x}\right.}+{e}{y}+{f}={0}\\ {b}^{{2}}-{4}{a}{c}<{0}$$

with a,b,c,d,e, and f coefficients.

$$\vec{{a}}={\left[{a},{b},{c},{d},{e},{f}\right]}^{{T}}\\ \vec{{x}}={\left[{x}^{{2}},{x}{y},{y}^{{2}},{x},{y},{1}\right]}$$

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Minimization Problem

The ellipse-specific fitting problem for a set of points p can be reformulated as:

$$\min_{{a}}{\left|{\left|\hat{{D}}\vec{{a}}\right|}\right|}^{{2}},\vec{{a}}^{{T}}\hat{{C}}\vec{{a}}={1}$$

with D as the design matrix containing expanded ellipse equation terms, one row for each point:

$$\hat{{D}}={\left(\matrix{{{x}_{{1}}^{{2}}}&{x}_{{1}}{y}_{{1}}&{{y}_{{1}}^{{2}}}&{x}_{{1}}&{y}_{{1}}&{1}\\..&..&..&..&..&..\\{{x}_{{n}}^{{2}}}&{x}_{{n}}{y}_{{n}}&{{y}_{{n}}^{{2}}}&{x}_{{n}}&{y}_{{n}}&{1}}\right)}$$

and C is a 6 × 6 constraint matrix (independent from the number of points).

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Solving an Eigenvalue Problem

• The minimization problem is ready to be solved by a quadratically constrained least squares minimization.

• We get a solution of the minimization problem by solving the Eigenvalue Problem, getting the Eigenvectors, and applying some filtering (only positive Eigenvalue are selected)

$$\min_{{a}}{\left|{\left|\hat{{D}}\vec{{a}}\right|}\right|}^{{2}},\vec{{a}}^{{T}}\hat{{D}}^{{T}}\vec{{a}}=\lambda\vec{{a}}^{{T}}\hat{{C}}\vec{{a}}=\lambda$$

%% Pseudo Code!
function fit_ellipse(x, y) {
 D = [x.*x x.*y y.*y x y ones(size(x))]; % design matrix
 S = D’ * D; % scatter matrix
 C(6, 6) = 0; C(1, 3) = 2; C(2, 2) = -1; C(3, 1) = 2; % constraint matrix
 [gevec, geval] = eig(inv(S) * C); % solve eigensystem
 [PosR, PosC] = find(geval > 0 &  ̃isinf(geval)); % find positive eigenvalue
 a = gevec(:, PosC); % corresponding eigenvector
 a
}

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Examples of results from the Canny-DBSCAN-ROI-ElliFit workflow for a ADM micrograph image

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Features

Ellipse Features

Solving the general ellipse equation delivers six polynomial parameters. But relevant for pore analysis are the following parameters:

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Features

Ellipse Features

These parameters can be derived from the general equation parameters:

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Statistical Analysis

Statistical Analysis

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Data-driven Modelling and Machine Learning

Data-driven Modelling and Machine Learning

Khan et al., 2018 (Top) Training of a data-driven machine model (Bottom) Application and inference

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Neural Networks

Convolutional Neural Networks

Convolutional Neural Networks combine typically:

1. Multi-dimensional matrix convolution with arbitrarily sized filter kernels

   • Mapping of matrix data on matrix data (commonly dimensionality expansion)

2. Fully connected neural node layers

   • Mapping of vectors on scalar valies (dimensionality reduction)

3. Pooling layers

   • Data reduction (fusion)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - CNN Architecture

CNN Architecture

Ragav Venkatesan and Baoxin Li,2018 Examples of CNN architecture consisting of interlacing different class layers

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Neural Networks

Convolutional Neural Networks

Khan et al., 2018 The relation between human vision, computer vision, machine learning, deep learning, and CNNs.

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Vector

A vector is commonly a linear list of values (real or complex type):

$$\vec{{x}}={\left[{v}_{{1}},{v}_{{2}},..,{v}_{{n}}\right]}\\ \vec{{a}}\odot\vec{{b}}={\left[{c}_{{1}},{c}_{{2}},{c}_{{i}},..,{c}_{{n}}\right]},{c}_{{i}}={a}_{{i}}\cdot{b}_{{i}},{n}={\left|{a}\right|}={\left|{b}\right|}\\ \vec{{x}}\cdot\vec{{w}}={\sum_{{{i}={1}}}^{{{n}}}}{x}_{{i}}{w}_{{i}},{n}={\left|{x}\right|}={\left|{w}\right|}$$

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Matrix

$$\hat{{m}}={\left[\matrix{{v}_{{{1},{1}}}&{v}_{{{1},{2}}}&..&{v}_{{{1},{i}}}\\..&..&..&..\\{v}_{{{j},{1}}}&{v}_{{{j},{2}}}&..&{v}_{{{j},{i}}}}\right]}\\ \hat{{a}}\otimes\hat{{b}}={\sum_{{{i}={1}}}^{{{n}}}}{\sum_{{{j}={1}}}^{{{m}}}}{a}_{{{i},{j}}}{b}_{{{j},{i}}},{n}=\text{rows}{\left({a}\right)},{m}=\text{cols}{\left({b}\right)}$$

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Tensor

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Volumes

• Volumes are (here) three-dimensional data structures representing vectors, 2D matrix, and 3D tensor objects.

• A volume is a packed linear array of values with a 321 Layout:

  • First order (most significant) index dimension is depth (sz)
  • Second order index dimension is width (sx)
  • Third order(least significant) index height (sy)

• Input, intermediate, output and kernel data can be represented by volumes

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Packed linear data model (memory layout) of 3D volumes

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Operations

• Addition (elementwise)
• Multiplication (elementwise)
• Multiplication and Addition (Dot Product)
• Convolution (Mapping)
• Transformation (including Fourier)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Layer

Convolutional Layer

• In contrast to kernel-based filtering operations using commonly 3 × 3 two-dimensional filters, convolution can be performed here with any kernel size and dimension.

• In contrast to kernel-based filtering operations, the kernel parameters (weights) are not pre-determined. They are evolved during the ML training process.

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Layer

Convolutional Layer

Convolution with N filters applied to one input image (stride: shift of filter position in each dimension)

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Example: Handwritten Digit Recognition

Example: Handwritten Digit Recognition

Ragav Venkatesan and Baoxin Li,2018 CNN layers and configuration for handwritten digit recognition (using the MNIST data set consisting of 28 × 28 × 1 images)