In this tutorial, I will discuss about how to perform texture matching using Local Binary Patterns (LBP). Local Binary Patterns is an important feature descriptor that is used in computer vision for texture matching. It was first released in 1990 and subsequently various modified versions have been released.

# LBP Descriptor

Let’s first discuss how to calculate the LBP Descriptor. Firstly, we convert the input color image to grayscale, since LBP works on grayscale images. For each pixel in the grayscale image, a neighbourhood is selected around the current pixel and then we calculate the LBP value for the pixel using the neighbourhood. After calculating the LBP value of the current pixel, we update the corresponding pixel location in the LBP mask (It is of same height and width as the input image.) with the LBP value calculated as shown below. In the image, we have 8 neighbouring pixels.

But how is the LBP values calculated?

To calculate the LBP value for a pixel in the grayscale image, we compare the central pixel value with the neighbouring pixel values. We can start from any neighbouring pixel and then we can transverse either in clockwise or anti-clockwise direction but we must use the same order for all the pixels. Since there are 8 neighbouring pixels – for each pixel, we will perform 8 comparisons. The results of the comparisons are stored in a 8-bit binary array.

If the current pixel value is greater or equal to the neighbouring pixel value, the corresponding bit in the binary array is set to 1 else if the current pixel value is less than the neighbouring pixel value, the corresponding bit in the binary array is set to 0.

The whole process is shown in the image below (Figure 2). The current (central) pixel has value 7. We start comparing from the neighbouring pixel where the label 0. The value of the neighbouring pixel with label 0 is 2. Since it is less than the current pixel value which is 7, we reset the 0th bit location in the 8 bit binary array to 0. We then iterate in the counter-clockwise direction. The next label location 1 have value 7 which is equal to the current pixel value, so we set the 1st bit location in the 8 bit binary to 1. We then continue to move to the next neighbouring pixel until we reach the 8th neighbouring pixel. Then the 8-bit binary pattern is converted to a decimal number and the decimal number is then stored in the corresponding pixel location in the LBP mask.

Once we have calculated the LBP Mask, we calculate the LBP histogram. The LBP mask values range from 0 to 255, so our LBP Descriptor will be of size 1x256. We then normalize the LBP histogram. The image below shows the scheme of the algorithm -

2. Convert to grayscale image.
4. Calculate the LBP Histogram and normalize it.

One advantage of LBP is that it is illumination and translation invariant. We have selected a 8 point neighbourhood, but most implementations use a circular neighbourhood as shown below. In the code, we will use a circular neighbourhood.

The original LBP algorithm has been further optimised to give better results. One such implementation is the Uniform LBP. In the code, we will use Uniform LBP.

That’s LBP for you, now let’s get started with the programming part

* Image taken from Detección de objetos course by Antonio López Peña, Ernest Valveny, Maria Vanrell (UAB) on Coursera.

# Import the required modules

The first step is to import the required modules. Since I am working in IPython Notebook, I have imported the pylab module in inline mode since it seamlessly embeds the images within the notebook. Otherwise, a seperate window opens which makes working with IPython Notebooks cumbersome. We will use the cv2 module to read the images, perform color space transformations etc. The os module is used to perform path manipulations. We will leverage the local_binary_pattern function from the skimage.feature module to calculate the LBP mask. From the LBP mask we will calculate the LBP histogram using scipy.stats.itemfreq function and then we will use the sklearn.preprocessing.normalize function to normalize the histogram. cvutils is a utility package for working with computer vision and image processing packages. Use the command, pip install cvutils to install the package. Finally, the csv module provides functionality to parse text files.

%pylab inline --no-import-all
# OpenCV bindings
import cv2
# To performing path manipulations
import os
# Local Binary Pattern function
from skimage.feature import local_binary_pattern
# To calculate a normalized histogram
from scipy.stats import itemfreq
from sklearn.preprocessing import normalize
# Utility package -- use pip install cvutils to install
import cvutils
# To read class from file
import csv

# Prepare the training set

I have assorted 6 training images. The 6 images consists of 2 images of rocks, 2 images of grass and 2 images of checkered patterns. I have also created a class_train.txt file which looks like -

rock-1.jpg 0
rock-2.jpg 0
grass-1.jpg 1
grass-2.jpg 1
checkered-1.jpg 2
checkered-2.jpg 2

Each line consists of the image name followed by the class label. I have given the following labels for each class -

Class Label
Rock 0
Grass 1
Checkered 2

Although, I haven’t used the class labels in this tutorial but it is always better to prepare a structure that can be used later on. For example, these labels can be useful if let’s say an SVM is used for classification.

So, let’s write the code to store the path of all the images in the training set.

1 # Store the path of training images in train_images
2 train_images = cvutils.imlist("../data/lbp/train/")
3 # Dictionary containing image paths as keys and corresponding label as value
4 train_dic = {}
5 with open('../data/lbp/class_train.txt', 'rb') as csvfile:
8         train_dic[row[0]] = int(row[1])

In line 2, we store all the paths of the training images in a list named train_images. In line 4, we create a dictionary train_dic that will contain the image name and the corresponding class label. From line 5 to 8, we read lines from the class_train.txt document described above and then we add the key-value pair – (image name, class label) to train-dic.

train_images contains the following image paths -

['../data/lbp/train/rock-1.jpg', '../data/lbp/train/rock-2.jpg', '../data/lbp/train/grass-2.jpg', '../data/lbp/train/checkered-2.jpg', '../data/lbp/train/checkered-1.jpg', '../data/lbp/train/grass-1.jpg']

train-dic contains the following key-value pairs -

{'grass-1.jpg': 1, 'rock-1.jpg': 0, 'checkered-2.jpg': 2, 'rock-2.jpg': 0, 'checkered-1.jpg': 2, 'grass-2.jpg': 1}

# Calculate the LBP Histograms

Now the next step is to calculate the normalized LBP histograms of the training images. Here is the code for the same -

 1 # List for storing the LBP Histograms, address of images and the corresponding label
2 X_test = []
3 X_name = []
4 y_test = []
5
6 # For each image in the training set calculate the LBP histogram
7 # and update X_test, X_name and y_test
8 for train_image in train_images:
11     # Convert to grayscale as LBP works on grayscale image
12     im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
14     # Number of points to be considered as neighbourers
15     no_points = 8 * radius
16     # Uniform LBP is used
17     lbp = local_binary_pattern(im_gray, no_points, radius, method='uniform')
18     # Calculate the histogram
19     x = itemfreq(lbp.ravel())
20     # Normalize the histogram
21     hist = x[:, 1]/sum(x[:, 1])
22     # Append image path in X_name
23     X_name.append(train_image)
24     # Append histogram to X_name
25     X_test.append(hist)
26     # Append class label in y_test
27     y_test.append(train_dic[os.path.split(train_image)[1]])
28
29 # Display the training images
30 nrows = 2
31 ncols = 3
32 fig, axes = plt.subplots(nrows,ncols)
33 for row in range(nrows):
34     for col in range(ncols):
36         axes[row][col].axis('off')
37         axes[row][col].set_title("{}".format(os.path.split(X_name[row*ncols+col])[1]))

Let’s break up the above code. In line 2-4, we create 3 lists –

1. X_test - to store the normalized LBP Histograms
2. X_name - to store the address of images
3. y_test - to store the corresponding class label

Each index in all the 3 lists corresponds to the same image.

From line 8 to 27 we loop over all the images in the training set and calculate the normalized LBP histograms for the training images. So firstly in line 10, we read the current image using the cv2.imread function. We then convert the image to grayscale since LBP works on grayscale image. In line 17, we calculate the LBP mask. We set the radius of the neighbourhood to 3 and the number of points to be equal to 24. Once we have the mask, we calculate the LBP histogram in line 19 and normalize it in line 21. Next, we append the histogram to the list X_testin line 25. We also append the image name to the list X_name and the image class label to y_test.

Next from line 30 to 37, we display the 6 training images. The image generated is -

# Get the testing images

To test the performance of the LBP algorithm, I have again assorted 3 images of each class not present in the training set. We will read the paths of the 3 images and append them to a list named test_images. We will also create a dictionary test_dic similar to the dictionary train_dic created above. Here is the code for the same-

1 # Store the path of testing images in test_images
2 test_images = cvutils.imlist("../data/lbp/test/")
3 # Dictionary containing image paths as keys and corresponding label as value
4 test_dic = {}
5 with open('../data/lbp/class_test.txt', 'rb') as csvfile:
8         test_dic[row[0]] = int(row[1])

The contents of test_images and test_dic are -

# test_images
['../data/lbp/test/rock-1.png', '../data/lbp/test/checkered-1.jpg', '../data/lbp/test/grass-1.jpg']
# test_dic
{'rock-1.png': 0, 'grass-1.jpg': 1, 'checkered-1.jpg': 2}

# Calculate the Chi-squared distance for each testing image

We then calculate the normalized LBP histograms for each image in the testing set and then we compare the normalized LBP Histograms of the training images (calculated above) with these using the Chi-Squared distance metric. We then sort the results based on the Chi-Squared distance and display the results in sorted order. The lower the Chi-Squared distance, the better is the match.

 1 for test_image in test_images:
4     # Convert to grayscale as LBP works on grayscale image
5     im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
7     # Number of points to be considered as neighbourers
8     no_points = 8 * radius
9     # Uniform LBP is used
10     lbp = local_binary_pattern(im_gray, no_points, radius, method='uniform')
11     # Calculate the histogram
12     x = itemfreq(lbp.ravel())
13     # Normalize the histogram
14     hist = x[:, 1]/sum(x[:, 1])
15     # Display the query image
16     cvutils.imshow("** Query Image -> {}**".format(test_image), im)
17     results = []
18     # For each image in the training dataset
19     # Calculate the chi-squared distance and the sort the values
20     for index, x in enumerate(X_test):
21         score = cv2.compareHist(np.array(x, dtype=np.float32), np.array(hist, dtype=np.float32), cv2.cv.CV_COMP_CHISQR)
22         results.append((X_name[index], round(score, 3)))
23     results = sorted(results, key=lambda score: score[1])
24     # Display the results
25     nrows = 2
26     ncols = 3
27     fig, axes = plt.subplots(nrows,ncols)
28     fig.suptitle("** Scores for -> {}**".format(test_image))
29     for row in range(nrows):
30         for col in range(ncols):
32             axes[row][col].axis('off')
33             axes[row][col].set_title("Score {}".format(results[row*ncols+col][1]))

From line 1 to 14, we calculate the normalized LBP histograms as in the case of training images. In line 21, we calculate the Chi-Squared Distance of the testing image with all the training images using the cv2.compareHist function. Then, we sort the scores in line 22. From line 25 to 33, we display the training images with the corresponding score.

# Results

Let’s check out the results.

Input Image – ROCK Class

Here is an input image of a rock texture.

Matching Scores - Below are the sorted results of matching. The top 2 scores are of rock texture.

Input Image – CHECKERED Class

Here is an input image of a checkered texture .

Matching Scores - Below are the sorted results of matching. The top 2 scores are of checkered texture.

Input Image – GRASS Class

Here is an input image of a grass texture.

Matching Scores - Below are the sorted results of matching. The top 2 scores are of grass texture.

Inference – In each case, the best 2 results outperform the next matches by atleast a single significant digit – sufficient to prove the potency of LBP for texture matching.