Lecture Notes: Basic Image Processing¶
Before jumping to build powerful and intelligent models for visual recognition it is always important to look at some pixels. Looking at images and pixels and transforming them in various ways gives us often valuable intuitions on how to find things about images.
torch.setdefaulttensortype('torch.FloatTensor') -- use floats as the default data type.
image = require 'image' -- load the torch image library.
-- Download image using this line or manually from the following url, or use any other image.
os.execute('wget http://www.cs.virginia.edu/~vicente/images/google_android.jpg');
rgb_image = image.load('google_android.jpg')
itorch.image(rgb_image)
The rgb_image variable contains a FloatTensor of size channels x height x width corresponding to the dimensions of the image. Each entry is between 0 and 1.
print('Number of channels: ' .. rgb_image:size(1))
print('Image height: ' .. rgb_image:size(2))
print('Image width: ' .. rgb_image:size(3))
print('Tensor type: ' .. torch.type(rgb_image))
print('Max value: ' .. rgb_image:max())
print('Min value: ' .. rgb_image:min())
1. Image channels¶
We can slice the image into each R, G, and B channels and show them separately:
local red_image = rgb_image[{{1}, {}, {}}]
local green_image = rgb_image[{{2}, {}, {}}]
local blue_image = rgb_image[{{3}, {}, {}}]
itorch.image({red_image, green_image, blue_image})
Each image in the above code is a one-channel image (e.g. grayscale image) corresponding to each RGB channel. You can clearly notice the Android figure looks brighter in the Green channel. We could also show each channel by setting to zero the channels corresponding to the other images.
local red_image = rgb_image:clone()
red_image[{{2}, {}, {}}]:zero()
red_image[{{3}, {}, {}}]:zero()
local green_image = rgb_image:clone()
green_image[{{1}, {}, {}}]:zero()
green_image[{{3}, {}, {}}]:zero()
local blue_image = rgb_image:clone()
blue_image[{{1}, {}, {}}]:zero()
blue_image[{{2}, {}, {}}]:zero()
itorch.image({red_image, green_image, blue_image})
2. Converting to Grayscale¶
How do we convert an RGB image to grayscale? A simple way would be to average all three RGB channels. Note the division by 3, since each channel has values between 0 and 1, we want to make sure the resulting grayscale image also has values between 0 and 1.
local red_image = rgb_image[{{1}, {}, {}}]
local green_image = rgb_image[{{2}, {}, {}}]
local blue_image = rgb_image[{{3}, {}, {}}]
local gray_image = (red_image + green_image + blue_image) / 3
itorch.image(gray_image)
A better way to convert a color RGB image into a grayscale image is using a weighted average for instance:
gray_image = 0.4 $*$ red_image + 0.4 $*$ green_image + 0.2 $*$ blue_image;
Why? To reproduce better how most humans perceive the images. We do not have the same sensitivity for all three channels, the coefficients reflect this. We are considerbly less sensative to blue. Here is a more detailed exposition about this on wikipedia: https://en.wikipedia.org/wiki/Grayscale#Colorimetric_.28luminance-preserving.29_conversion_to_grayscale
3. Other Color Spaces¶
In addition to RGB images, we can represent images as HSL, where each channel corresponds to Hue, Saturation and Lightness instead. Other color spaces include: HSV, Lab, YUV, etc. Sometimes representing an image in some of these other ways might be more beneficial for a given analysis. For instance if we want to focus only on Hue and Saturation but we do not care too much about Lightness. Torch has convenient operations to make this transformations.
local hsl_image = image.rgb2hsl(rgb_image)
local h_channel = hsl_image[{{1}, {}, {}}]
local s_channel = hsl_image[{{2}, {}, {}}]
local l_channel = hsl_image[{{3}, {}, {}}]
itorch.image({h_channel, s_channel, l_channel})
Notice how the Hue channel tries to output a flat value for regions that are supposed to be the same color (e.g. t-shirts) regardless of luminance. The Saturation channel tries to output higher (brighter) values for regions that have more intense colors (e.g. the Android), and the luminance channel outputs something similar to the original image. You can also deduce that the RGB color space outputs roughly three different versions of luminance.
4. Brightness¶
Making an image brighter is achieved by multiplying the pixels in the image by a scalar (making sure the values are still between 0 and 1)
local brighter_image = rgb_image:clone()
-- Go through each channel.
for c = 1, rgb_image:size(1) do
-- Go through each row.
for i = 1, rgb_image:size(2) do
-- Go through each column.
for j = 1, rgb_image:size(3) do
brighter_image[{c, i, j}] = 1.8 * rgb_image[{c, i, j}] -- scale pixel intensity.
-- make sure the intensity is within 0 and 1.
if brighter_image[{c, i, j}] > 1 then
brighter_image[{c, i, j}] = 1
end
end
end
end
-- The above code is slow.
itorch.image({rgb_image, brighter_image})
Modifying each pixel intensity one by one using three for loops is slow. The efficient way to do this would be to program the three for loops in a lower level language like C/C++. Fortunately Torch has many efficient tensor operations implemented in C/C++. This means that if we can find a way to implement the above code using functions provided by Torch we not need to write any C/C++ to obtain good performance. For instance:
local brighter_image = rgb_image:clone() -- copy the original image.
brighter_image:mul(1.8) -- in-place multiplication with scalar.
brighter_image:cmin(1) -- in-place assignment of min value between current value and 1.
-- This implementation is efficient because mul, and cmin are implemented efficiently.
itorch.image({rgb_image, brighter_image})
-- Let's also try to make the image darker.
local darker_image = rgb_image:clone()
darker_image:mul(0.6)
-- no need to make sure the image is within 0 and 1 in this case.
itorch.image({rgb_image, darker_image})