Labels

Deep papers: A survey on Image Data Augmentation for Deep Learning part 1


data augmentation comes in handy when we have a small number of training samples or unbalanced datasets. in this survey, Connor Shorten and Taghi M. Khoshgoftaar discussed data augmentation techniques and how they affect model performance. we are going to test those technics with a small dataset with two classes slightly unbalanced. authors mainly break down augmentation techniques into two branches, which are basic image manipulations and deep learning approaches. in the first part, we only discuss basic image manipulations. those techniques are also known as geometric transformations.

Why augmentations?

maybe you think why augmentation, we can try to gather more training samples. that's not the optimal solution always. sometimes it's. but when gathering more training samples will be more expensive, maybe there is some level of rarity issues that prevent gathering more training samples and also there may be privacy issues, maybe health issues like CT scanning likewise, gathering more training samples always not the best way to go. in such cases data augmentation really helpful.

Benefits

first things first, you can increase your training samples effectively. we all know deep neural networks need lots of data. in case I mention deep neural networks here because that's the go-to solution for computer vision problems these days. if you don't have enough data your model may overfit or even it's not overfitted it performs poorly because it is trained on far fewer scenarios that can occur in real situations. data basic data augmentation technics are powerful enough to solve this problem by increasing training samples.

sometimes sample size not be the case but possible combinations. let's say you have a dataset of humans and there are too few samples that humans are captured by side angles. when a model trained on that data set in the inference that has very little possibility to successfully identify a human in an image who captured inside angle. likewise, we need as much as a possible combination to train our models. then those models can perform better. also, a very common situation is models trained on images that have identical angles which also can affect model performance. these kinds of problems also can solve easily with augmentation. in classification and detecting models most of the time models are unable to perform well when objects are in different positions rather than the centre of the frame. that's happening because of positional bias. that's mean when models are trained on perfectly centred training images it's difficult to do classification or identification when objects are off from the centre of the frame. this kind of positional bias also can solve with translation augmentation techniques.

How augmentations works

images are three-dimensional metrics in another way 2d-tensors. wich consists of width, height and colour channels as depth. all basic augmentations are done by manipulating those metrics. let's say we need to flip an image left to right, let's see what happens to metric.

left - image matrix, right - image, top row - original and bottom row - flipped

you can see that after flipping how image matrics change by changing position. in other manipulation such as colour space transformation, cropping, rotation, translations and random noise injection all are manipulations of the image matrix.

Augmentations

we can put basic image augmentation methods into six categories. which are flip, colour space transformations, cropping, rotation, translation, and noise injection. let us see how each augmentation works and compare their contribution to increasing model performance by comparing with baseline model performance. in here baseline model means model trained on the original dataset without any augmentation technique applied.

flipping

flip can be divided into two subgroups, vertical and horizontal flip. in vertical flip image turn up-side-down and in horizontal flip image turn right to left. there is a drawback in flip. which is, let's say in digit recognizer if we apply any kind of flip above mention post augmented result may loss label information, in other words, this method not label-preserved in some datasets but can benefit in other. that's a thing to care about before applying flip augmentations.

import tensorflow as tf
import PIL.Image
image = PIL.Image.open('path to image')
# horizontal flip
tf.image.flip_left_right(image)
# vertical flip
tf.image.flip_up_down(image)
view raw flipping.py hosted with ❤ by GitHub

horizontal flip

vertical flip

colourspace manipulation

there are few augmentation methods in colour space manipulations. basically, they can divide into two groups, colour schema changes and image property adjustments. in colour schema changes we can change RGB colour schema to other colour schemas. not only RGB but others also. in image property adjustments colour channel isolation, brightness, contrast, gamma, hue and saturation adjustment. we treated these image property adjustments as a whole group in this comparison with the baseline model.

colour channel isolation


def rgb_channel_isolation(path, dst=None):
img = PIL.Image.open(path)
img = img_to_array(img)
src_path = path.split('/')[-1].split('.')[0]
if img.shape[2] == 1:
ValueError()
print(f'ValueError: image should contain more than {img.shape[2]}')
elif dst is None:
r = img.copy()
g = img.copy()
b = img.copy()
r[:, :, 1:3] = 0
g[:, :, 0] = 0
g[:, :, 2] = 0
b[:, :, 0:2] = 0
return r, g, b
view raw channel_isolation.py hosted with ❤ by GitHub


isolate RGB channels by replacing 0's in others

brightness adjustments


# delta should be a float between -1 to 1
# too high and too low delta values destroy image details
delta = 0.4
tf.image.adjust_brightness(image, delta=delta)
view raw brightness.py hosted with ❤ by GitHub
brightness adjustments

brightness adjustments result in different lighting effects which are very useful if our dataset doesn't have images that take in different light conditions. light conditions refer to brighter and darker light conditions.

contrast adjustment


# a multiflier can be positive or negative
cf = 2
tf.image.adjust_contrast(image, contrast_factor=cf)
view raw contrast.py hosted with ❤ by GitHub
contrast adjustment

by definition, colour contrast is changing of appearance of one colour when surrounded by other colours. as you can see above image shows when negative contrast factors result in a negative effect. this method will help to identify certain colours in images.

hue adjustments


# delta mustbe in between -1 to 1
delta = -0.3
tf.image.adjust_hue(image, delta=delta)
view raw hue.py hosted with ❤ by GitHub
hue adjustments

by adjusting hue we can generate different colour casting effects which are helpful to create different lighting effects. most low-quality cameras and low-quality smartphone cameras have a certain blue colour tone so when our models run on certain devices it will help to train on different colour cast images.

saturation adjustments


# saturation factor used as a multiplier to change RGB saturation
sf = 0.3
tf.image.adjust_saturation(image, saturation_factor=sf)
view raw saturation.py hosted with ❤ by GitHub
saturation adjustment

when saturation increases colour intensity increases and when decrease colours get pale. this is helpful to create different colour intensity effects in our dataset.

gamma transformation


# when gamma to large image getting darker and distroy details
# even too close image getting brighter
# this performe gamma corelation
# gamma must be non-negative real number
g = 0.125
tf.image.adjust_gamma(image, gamma=g)
view raw gamma.py hosted with ❤ by GitHub
gamma correlation transformation

this is a colour correction method used to reproduce colours accurately images which not properly corrected.

change colour schema


colour schema changes have to be done because maybe colours are not an effective factor when it comes to inference. as a rule of thumb if the naked human eye can classify or identify objects or things without RGB colour CNN's also able to. changing RGB to greyscale will reduce computational needs because it reduces 3 deep colour channels into 1 deep colour channel. also, there may be a need to change the colour schema to identify certain characteristics of an image.

RGB to GrayScale

RGB to BGR

RGB to YCbCr

Cropping

cropping is a very handy augmentation method and also a bit dangerous because it will sometimes destroy label information. mainly cropping can use in two ways, central cropping and random cropping. central crop cut a specified amount of area by anchoring the centre of the image. random cropping is different. it cut randomly from any position. cropping has a few benefits, it reduces positional bias and also with this method we train on multi-scale by using resize along with the crop.

central crop

random crop

Rotation

by the rotating images, we can generate new images with random angles which provide more possible scenarios to our dataset to generalize its performance. as the authors mention in the original paper recognition task such as handwriting rotation is effective in 1-20o other than that range not effective.

rotation


Translation

the translation is consist of moving images up, down, left and right inside the frame. this results in different positions for the model to train. the gap is created by moving images that can be filled with constants (0's or 255's) or nearest values. other than that TensorFlow has a few other options as well.

translations

How these techniques affected model training 

how each augmentation technique affected model training (VGGNet16)
left - validation accuracy, right - validation lost

in inference with hold out data,

How to use augmentation in training

actually, it can do in two ways. one is applying augmentation on the fly and the other is creating a dataset by applying augmentations. the first method is better if you have high computational power because it will take a lot of computational resources when your images are high resolution. if it's not you can create a dataset from the original dataset by randomly applying augmentation.

apply augmentation while training

there are two ways to do this. one is using ImageDataGenerator class in Keras preprocessing module, by the way, it's most popular and easy. but if you want more control over your augmentations you can create your own generator using Keras subclass API. the second method is using Keras preprocessing layers directly into the model. one benefit of the second method is when you import your model augmentation layers also imported automatically. so you don't need to recreate the same logic again.

image data generator class
tf.keras.preprocessing.image.ImageDataGenerator(
featurewise_center=False, samplewise_center=False,
featurewise_std_normalization=False, samplewise_std_normalization=False,
zca_whitening=False, zca_epsilon=1e-06, rotation_range=0, width_shift_range=0.0,
height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0,
channel_shift_range=0.0, fill_mode='nearest', cval=0.0,
horizontal_flip=False, vertical_flip=False, rescale=None,
preprocessing_function=None, data_format=None, validation_split=0.0, dtype=None
)
view raw image_data_generator.py hosted with ❤ by GitHub

using preprocessing layers
import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow.keras import layers
data_augmentation = tf.keras.Sequential([
layers.RandomFlip("horizontal_and_vertical"),
layers.RandomRotation(0.2),
])
model = tf.keras.Sequential([
# Add the preprocessing layers you created earlier.
resize_and_rescale,
data_augmentation,
layers.Conv2D(16, 3, padding='same', activation='relu'),
layers.MaxPooling2D(),
# Rest of your model.
])
view raw layers.py hosted with ❤ by GitHub

apply augmentation and create a dataset

by using the TensorFlow image module and addons module we can create a program to generate a randomly augmentation applied dataset from the original dataset.

import os
import random
import numpy as np
import PIL.Image
import tensorflow as tf
import tensorflow_addons as tfa
import tensorflow_io as tfio
from tensorflow.keras.utils import array_to_img, img_to_array
def flip(path, dst, save_original=True):
"""
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
dst_lr_path = os.path.join(dst, f'_flip_l_r_{image}')
dst_ud_path = os.path.join(dst, f'_flip_u_d_{image}')
img = PIL.Image.open(image_path)
flip_lr = array_to_img(tf.image.flip_left_right(img_to_array(img)))
flip_ud = array_to_img(tf.image.flip_up_down(img_to_array(img)))
# save flipped image in destination
flip_lr.save(dst_lr_path)
flip_ud.save(dst_ud_path)
if save_original:
img.save(os.path.join(dst, image))
def rgb_channel_isolation(path, dst, save_original=True):
"""
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path))
r, g, b = img, img, img
r[:, :, 1:3] = 0
g[:, :, 0] = 0
g[:, :, 2] = 0
b[:, :, 0:2] = 0
# red component
r = array_to_img(r)
r.save(os.path.join(dst, f'_isolate_r_{image}'))
# green component
g = array_to_img(g)
g.save(os.path.join(dst, f'_isolate_g_{image}'))
# blue component
b = array_to_img(b)
b.save(os.path.join(dst, f'_isolate_b_{image}'))
if save_original:
# original
img = array_to_img(img)
img.save(os.path.join(dst, image))
def random_changes_to_color_properties(path, dst, delta=None, gamma_transformation=True, change_contrast=True,
factor=None, copies=1, save_original=True):
"""
:type copies: int
:type factor: int
:type change_contrast: bool
:type gamma_transformation: bool
:type delta: float
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path))
if delta is None:
for i in range(copies):
change_factor = np.round(np.random.uniform(-1, 1), 2)
bc_img = array_to_img(tf.image.adjust_brightness(img, change_factor))
hue_img = array_to_img(tf.image.adjust_hue(img, change_factor))
sat_img = array_to_img(tf.image.adjust_saturation(img, change_factor))
# save transformed images
bc_img.save(os.path.join(dst, f'bc_{i}_{image}'))
hue_img.save(os.path.join(dst, f'hue_{i}_{image}'))
sat_img.save(os.path.join(dst, f'sat_{i}_{image}'))
elif isinstance(delta, float):
for i in range(copies):
change_factor = np.round(np.random.uniform(-1 * delta, 1 * delta), 2)
bc_img = array_to_img(tf.image.adjust_brightness(img, change_factor))
hue_img = array_to_img(tf.image.adjust_hue(img, change_factor))
sat_img = array_to_img(tf.image.adjust_saturation(img, change_factor))
# save transformed images
bc_img.save(os.path.join(dst, f'bc_{i}_{image}'))
hue_img.save(os.path.join(dst, f'hue_{i}_{image}'))
sat_img.save(os.path.join(dst, f'sat_{i}_{image}'))
if factor is None:
if gamma_transformation:
for i in range(copies):
gamma = np.round(np.random.uniform(1, 5), 2)
gamma_img = array_to_img(tf.image.adjust_gamma(img, gamma))
gamma_img.save(os.path.join(dst, f'gamma_img_{i}_{image}'))
if change_contrast:
for i in range(copies):
change_factor = np.round(np.random.uniform(-5, 5), 2)
cont_img = array_to_img(tf.image.adjust_contrast(img, change_factor))
cont_img.save(os.path.join(dst, f'cont_img_{i}_{image}'))
elif isinstance(factor, int):
if gamma_transformation:
for i in range(copies):
gamma = np.round(np.random.uniform(1, factor), 2)
gamma_img = array_to_img(tf.image.adjust_gamma(img, gamma))
gamma_img.save(os.path.join(dst, f'gamma_img_{i}_{image}'))
if change_contrast:
for i in range(copies):
factor = np.round(np.random.uniform(-1 * factor, factor), 2)
cont_img = array_to_img(tf.image.adjust_contrast(img, factor))
cont_img.save(os.path.join(dst, f'cont_img_{i}_{image}'))
if save_original:
array_to_img(img).save(os.path.join(dst, image))
def cropping(path, dst, central=True, random=False, fraction_low=0.5, fraction_high=0.9, random_width=224,
random_height=224, copies=3, save_original=True):
"""
:type copies: int
:type random_height: int
:type random_width: int
:type fraction_high: float
:type fraction_low: float
:type random: bool
:type central: bool
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path))
if central:
for i in range(copies):
crop_area = np.round(np.random.uniform(fraction_low, fraction_high), 2)
cc_img = tf.image.central_crop(img, central_fraction=crop_area)
array_to_img(cc_img).save(os.path.join(dst, f'cc_img_{i}_{image}'))
elif random:
for i in range(copies):
rc_img = tf.image.random_crop(img, size=[random_width, random_height, 3])
array_to_img(rc_img).save(os.path.join(dst, f'rc_img_{i}_{image}'))
if save_original:
array_to_img(img).save(os.path.join(dst, image))
def rotation(path, dst, angel=30, copies=3, save_original=True):
"""
:type angel: int
:type copies: int
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path))
for i in range(copies):
random_angel = np.random.randint(-1 * angel, angel)
ra_img = tfa.image.rotate(img, random_angel)
array_to_img(ra_img).save(os.path.join(dst, f'ra_img_{i}_{image}'))
if save_original:
array_to_img(img).save(os.path.join(dst, image))
def translation(path, dst, height_shift_range=0.3, width_shift_range=0.3, height_shift=True, width_shift=True,
copies=3, save_original=True):
"""
:type copies: int
:type width_shift: bool
:type height_shift: bool
:type width_shift_range: float
:type height_shift_range: float
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path))
for i in range(copies):
if height_shift and width_shift:
random_hshift = np.random.uniform(0.1, height_shift_range)
random_wshift = np.random.uniform(0.1, width_shift_range)
rs_img = (tf.keras.preprocessing.image.random_shift(img, random_wshift, random_hshift, channel_axis=2,
row_axis=0, col_axis=1,
fill_mode='reflect')).astype(np.uint8)
array_to_img(rs_img).save(os.path.join(dst, f'rs_img_{i}_{image}'))
elif height_shift and not width_shift:
random_hshift = np.random.uniform(0.1, height_shift_range)
rs_img = (tf.keras.preprocessing.image.random_shift(img, 0, random_hshift, channel_axis=2,
row_axis=0, col_axis=1,
fill_mode='reflect')).astype(np.uint8)
array_to_img(rs_img).save(os.path.join(dst, f'rs_h_img_{i}_{image}'))
elif not height_shift and width_shift:
random_wshift = np.random.uniform(0.1, width_shift_range)
rs_img = (tf.keras.preprocessing.image.random_shift(img, random_wshift, 0, channel_axis=2,
row_axis=0, col_axis=1,
fill_mode='reflect')).astype(np.uint8)
array_to_img(rs_img).save(os.path.join(dst, f'rs_img_{i}_{image}'))
if save_original:
array_to_img(img).save(os.path.join(dst, image))
def noise_injection(path, dst, magnitude=0.5, copies=3, save_original=True):
"""
:type copies: int
:type magnitude: float
:type save_original: bool
:type dst: basestring
:type path: basestring
"""
images = os.listdir(path)
for image in images:
image_path = os.path.join(path, image)
img = img_to_array(PIL.Image.open(image_path)) / 255.
for i in range(copies):
random_magnitude = np.round(np.random.uniform(0, magnitude), 2)
noise = np.random.normal(0, np.round(random_magnitude, decimals=3), size=img.shape)
ni_img = array_to_img((img + noise))
array_to_img(ni_img).save(os.path.join(dst, f'ni_img_{i}_{image}'))
if save_original:
array_to_img(img).save(os.path.join(dst, image))
view raw basic.py hosted with ❤ by GitHub

References 

Labels:

Comments

Post a Comment