Automated Feature Extraction with Machine Learning and Image Processing

PD Stefan Bosse

University of Siegen - Dept. Maschinenbau
University of Bremen - Dept. Mathematics and Computer Science

1 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing -

Machine Learning in Image Processing

2 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Feature Classes

Feature Classes

Machine Learning is commonly a data-driven approximation of a functional model f(x):xy, with x as input and +y* as output (target) features.

There are basically two different ML tasks:

  1. Classification ⇒ Symbolic / categorical and discrete target feature variables
  2. Regression ⇒ Numeric and continuous target feature variables
3 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Feature Classes

Feature Classes

Examples for categorical features:

  • Damage (boolean decision, classification of damages like cracks, delaminations, and so on)
  • Quality assessment (boolean decision, grade levels, class A/B/C, and so on)
  • Geometrical objects (shapes, like lines, circles, ellipses)

Examples for numerical features:

  • Material density, average pore size and/or density, crack length and/or density, and so on
  • Mechanical properties (stiffness, homogeneity, and so on)
  • Predictive lifetime
  • Statistical aggregates (noise, average inhomogeneity, average size or density of impurities)
4 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

  1. Two-dimensional intensity (photography) images (intensity represents surface reflection or material transmission)
    • Gray-level or multi-channel color (RGB, red, green, and blue channels);
    • Typical dimension 1000 × 1000 pixels;
    • Typical intensity resolution 8 bit (256 levels, gray or RGB), high-quality 16 bit (65536 levels)
    • Volume dimension: height × width × channels
    • Common data file formats: PNG, BMP, TIFF (not JPEG: irreversible compression creating image artifacts)
5 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

  1. Three-dimensional tomography images (intensity represents material density)
    • Gray-level
    • Sliced image stack
    • Typical dimension 1000 × 1000 pixels × 1000 (100) pixels;
    • Volume dimension: height × width × depth
    • Common data file formats: numpy, ZIP of tiff files, Vol3, RAW, DICOM (medicine) ..
6 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Classes

Image Classes

https://www.matsusada.com/column/ct-tech2.html DICOM CT scan data file format merging meta and raw image data

7 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Target output features can be predicted by the data-driven model basically in two ways:

  1. Using raw image input data;
  2. Using computed (intermediate) image features.

Examples of computed image features:

  • Image intensity distribution, inhomogeneity
  • Detection and characterisation of geometrical object shapes (e.g., circles)
  • Image transformations, i.e.,
    • wave-number frequency transformation,
    • gradient amplification using convolutional filter operations,
    • binarisation using a threshold
8 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Example: Pore characterisation in Microsclices of AM parts

(Left) Microgrpah image of a material slice with pores/impurities (Middle) Edge detection using a Canny filter (Right) Shape characterisation by ellipse fitting

9 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Feature Extraction

Image Feature Extraction

Example: Pore characterisation in X-ray images from die casted plates

(left) Die casted aluminum plate with pores (Center) Single projection X-ray image of plate (Right) Pore marking by semantic pixel classifier (white=pore feature)

10 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Workflow for Object Feature Detection

Workflow for Object Feature Detection

Data: Micrograph images of material slices from rectangular probes proced with Additive Manufacturing technologies (metal powder laser melting).

Object Features: Elliptical pores (impurities) characterised by varying size (axis lengths of ellipse), orientation, density, and spatial distribution (inhomogeneity)

Target Features: Statistical and geometrical characterisation of pores, material density, distribution of defects

Feature extraction should be scale, intensity, and position invariant! I.e., object detection should be possible for objects of different sizes, orientation, and position within the images and different image exposures.

11 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Workflow for Object Feature Detection

Workflow for Object Feature Detection

  1. Threshold binarisation of micrograph images
  2. Application of Canny Filter to extract pore edges / boundaries (Parameter selection!)
  3. Creation of a linear point list (coordinates of marked boundary points of pores)
  4. Density-based Clustering (DBSCAN) to get groups of points belonging to one pore object
  5. ROI bounding box approximation for each point cluster group (iterative expansion and shrinking)
  6. Ellipse fitting (direct algebraic method), feature calculation (axis lengths, orientation, area)
  7. Statistical characterisation
12 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Binarization

Image Binarization

I1b(x,y)={1I(x,y)Ithr0I(x,y)<IthrI0b(x,y)={1I(x,y)Ithr0I(x,y)>Ithr

13 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Image Binarization

Image Binarization

i1 = load.image('http://edu-9.de/uploads/assets/pores_microslice_ti_1.png',format='RGBA')
m1 = as.matrix(i1,mode='uint8')
print(minMax(m1[100:200,100:200]))
m1.binary = matrix(255,nrow(m1),ncol(m1),mode='uint8')
m1.binary[m1>100]=0
plot(m1.binary)

R+ microslice image pre-processing: Normalization and Binarization (0/255)

14 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

DOI:10.1109/ICSMC.2009.5346873 Canny edge detection algorithm

15 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

use math,imager,plot
i1 = load.image('http://edu-9.de/uploads/assets/pores_microslice_ti_1.png',format='GRAY')
m1 = as.matrix(i1,mode='uint8')
m1.stats = minMax(m1)
m1.stats$mean = mean(m1)
m1 = (m1 - m1.stats$min)
m1[m1>100] = 255
print(m1.stats)
plot(m1)

R+ microslice image pre-processing: Normalization and Binarization (0/255)

16 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Edge Detection with Canny Filter

Edge Detection with Canny Filter

use math,imager,plot
m1.edges=cannyEdges(m1,t1=50)
plot(m1.edges)

Canny edge filter in R+

17 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

Point Clustering using DBSCAN

  • Try to group points from Canny edges to define a ROI marking a pore candidate

https://medium.com/@agarwalvibhor84/lets-cluster-data-points-using-dbscan-278c5459bee5 Three point classes: Core, Border, Noise

18 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

Core point
A selected point is a core point if it has at least minimum number of points (MinPts) including itself within its epsilon-neighborhood. In figure 1, red points are core points that have at least MinPts=4 in their neighborhood. If we’ve a core point, it means it is a dense region.
Border point
A selected point that is within a neighborhood of a core point but it itself cannot be a core point. In the figure 1, yellow points are identified as border points. If we’ve a border point, it means the point is in a vicinity or at the border of dense region.
Noise point
A selected point that is neither a core point nor a border point. It means these points are outliers that are not associated with any dense clusters. In the figure 1, blue point is identified as noise point.
19 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Point Clustering using DBSCAN

DBSCAN Algorithm

Initially, the algorithm begins by selecting a point randomly uniformly from the set of data points. Checks if the selected point is a core point. Again, a point is a core point if it contains at least MinPoints number of minimum points in its epsilon-neighborhood.

Then, finds the connected components of all the core points, ignoring non-core points.

Assign each non-core point to the nearest cluster if the cluster is its epsilon-neighbor. Otherwise, assign it to noise.

The algorithm stops when it explores all the points one by one and classifies them as either core, border or noise point.

DBSCAN
20 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - ROI Bounding Box Approximation from point list

ROI Bounding Box Approximation from point list

  • Output from DBSCAN: List of point groups from canny edge filter

  • Calculate rectangular (not rotated) average bounding boxes (pore boundary point group)

An initial ROI (red) positioned at the center of mass of a point cluster is expanded iteratively by increasing one side and computing the point sum along this side. If the line is empty, the next side is expanded.

21 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - ROI Bounding Box Approximation from point list

ROI Bounding Box Approximation from point list

  • Bounding boxes can be computed from a pixel set list, i.e., a list of coordinate vectors
use math, imager, plot
pxs = {
[1,2],
[3,9],
[4,7]
}
pxs.bbox = bbox(pxs)
print(pxs.bbox)

Example of a bbox calculation from a pixel set list

22 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Ellipse Fitting from point list

Problem: Calculate the parameters of an ellipse equation for a set of boundary points.

Halı́ř, FLusser, Numerically stable direct least squares fitting of an ellipse

General Ellipse Equation

F(x,y)=ax2+bxy+cy2+dx+ey+f=0b24ac<0

with a,b,c,d,e, and f coefficients.

a=[a,b,c,d,e,f]Tx=[x2,xy,y2,x,y,1]

23 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Minimization Problem

The ellipse-specific fitting problem for a set of points p can be reformulated as:

mina^Da2,aT^Ca=1

with D as the design matrix containing expanded ellipse equation terms, one row for each point:

^D=x21x1y1y21x1y11............x2nxnyny2nxnyn1

and C is a 6 × 6 constraint matrix (independent from the number of points).

24 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Solving an Eigenvalue Problem

  • The minimization problem is ready to be solved by a quadratically constrained least squares minimization.

  • We get a solution of the minimization problem by solving the Eigenvalue Problem, getting the Eigenvectors, and applying some filtering (only positive Eigenvalue are selected)

mina^Da2,aT^DTa=λaT^Ca=λ

%% Pseudo Code!
function fit_ellipse(x, y) {
D = [x.*x x.*y y.*y x y ones(size(x))]; % design matrix
S = D’ * D; % scatter matrix
C(6, 6) = 0; C(1, 3) = 2; C(2, 2) = -1; C(3, 1) = 2; % constraint matrix
[gevec, geval] = eig(inv(S) * C); % solve eigensystem
[PosR, PosC] = find(geval > 0 & ̃isinf(geval)); % find positive eigenvalue
a = gevec(:, PosC); % corresponding eigenvector
a
}
25 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Fitting from point list

Examples of results from the Canny-DBSCAN-ROI-ElliFit workflow for a ADM micrograph image

26 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Features

Ellipse Features

Solving the general ellipse equation delivers six polynomial parameters. But relevant for pore analysis are the following parameters:

  1. Major and minor axis lengths w and h

  2. Orientation angle of the major axis θ

  3. Area A of the ellipse

  4. Center coordinates

27 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Ellipse Features

Ellipse Features

These parameters can be derived from the general equation parameters:

cx=2cdbeb24accy=2aedbb24acw=4fac+cd2+ae24ac2h=4fac+cd2+ae24a2cθ=tan1(2bac)3604π

28 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Statistical Analysis

Statistical Analysis

  1. Average Density (from binarized image)

  2. Average pore size (from ellipse fitting), aspect ratio w/h, variance of these features

  3. Average pore orientation

  4. Spatial distribution of pores

29 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Data-driven Modelling and Machine Learning

Data-driven Modelling and Machine Learning

Khan et al., 2018 (Top) Training of a data-driven machine model (Bottom) Application and inference

30 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Neural Networks

Convolutional Neural Networks

Convolutional Neural Networks combine typically:

  1. Multi-dimensional matrix convolution with arbitrarily sized filter kernels

    • Mapping of matrix data on matrix data (commonly dimensionality expansion)
  2. Fully connected neural node layers

    • Mapping of vectors on scalar valies (dimensionality reduction)
  3. Pooling layers

    • Data reduction (fusion)
31 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - CNN Architecture

CNN Architecture

Ragav Venkatesan and Baoxin Li,2018 Examples of CNN architecture consisting of interlacing different class layers

32 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Neural Networks

Convolutional Neural Networks

Khan et al., 2018 The relation between human vision, computer vision, machine learning, deep learning, and CNNs.

33 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Vector

A vector is commonly a linear list of values (real or complex type):

x=[v1,v2,..,vn]ab=[c1,c2,ci,..,cn],ci=aibi,n=|a|=|b|xw=ni=1xiwi,n=|x|=|w|

34 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Matrix

^m=v1,1v1,2..v1,i........vj,1vj,2..vj,i^a^b=ni=1mj=1ai,jbj,i,n=rows(a),m=cols(b)

35 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Tensor

A scalar is a level zero tensor.

A vector is an array of numbers along an axis (level one tensor).

A matrix is an arrangement of numbers along two axes (level two tensor).

A tensor is an arrangement of numbers along n axes.

36 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Volumes

  • Volumes are (here) three-dimensional data structures representing vectors, 2D matrix, and 3D tensor objects.

  • A volume is a packed linear array of values with a 321 Layout:

    • First order (most significant) index dimension is depth (sz)
    • Second order index dimension is width (sx)
    • Third order(least significant) index height (sy)
  • Input, intermediate, output and kernel data can be represented by volumes

37 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Packed linear data model (memory layout) of 3D volumes

38 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Vector, Matrix, and Tensor Data

Vector, Matrix, and Tensor Data

Operations

  • Addition (elementwise)
  • Multiplication (elementwise)
  • Multiplication and Addition (Dot Product)
  • Convolution (Mapping)
  • Transformation (including Fourier)
39 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Layer

Convolutional Layer

  • In contrast to kernel-based filtering operations using commonly 3 × 3 two-dimensional filters, convolution can be performed here with any kernel size and dimension.

  • In contrast to kernel-based filtering operations, the kernel parameters (weights) are not pre-determined. They are evolved during the ML training process.

40 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Convolutional Layer

Convolutional Layer

Convolution with N filters applied to one input image (stride: shift of filter position in each dimension)

41 / 42

PD Stefan Bosse - AFEML - Module D: Machine Learning in Image Processing - Example: Handwritten Digit Recognition

Example: Handwritten Digit Recognition

Ragav Venkatesan and Baoxin Li,2018 CNN layers and configuration for handwritten digit recognition (using the MNIST data set consisting of 28 × 28 × 1 images)

42 / 42