Gaussian Deformation and the Dataset state

[tmquan]

Hi tensorpack,

What does the gaussian deformation do exactly? Is it similar to Elastic Deformation?
If that is the case, can we alter the magnitude as well as the size of randomly generated grid?

And does the dataset perform the deformation on the original data as the iteration increases?
For example, original dataset dset after the first iteration, has been applied some rotations, deformations, etc. Can we guarantee that in the next iteration, it will perform on the original data but not on the augmented data after iteration 1? (Probably, reset state of the dset will work but I am not sure)

Best regards,

[ppwwyyxx]

It generates a continuous vector field of NxM so that it gives each pixel a small displacement but continuous in spatial domain. The field is generated by a mixture of gaussian centered at several spatial locations.

You can adjust the parameters.

Augmentors doesn't gurantee they don't modify input image. They can do whatever to the input to speed up their processing. This specific augmentor doesn't modify the input image, though.
To avoid the modification goes to your original images (in case you keep them persistent in memory, for example), you can yield a deepcopy in your first dataflow.

[ppwwyyxx]

Just give it a second thought. Maybe it's not a very good idea to use a dangerous default, allowing the augmentors to modifiy the image.
Maybe we should use a safe default (make a copy explicity) and allow turning it off when users know it's OK.

[ppwwyyxx]

FYI, added a "copy" option to AugmentImageComponent but still defaults to false now.

[tmquan]

Hi @ppwwyyxx ,

I would like to revisit this one and am working on the image segmentation.
Specifically, I would like to implement a custom function on image augmentation which is so-called Elastic Deformation since the Gaussian Deformation is not robust to my problem and it requires the higher accuracy.

In Elastic Deformation, both 3-channel input and output images need to used the same seed number to generate the same vector flow from numpy.
Is it straightforward to add such this customization? An example of interface could be helpful a lot.

Thanks so much.

[ppwwyyxx]

class MyAug:
   def _get_augment_params(self, img):
        return any params that have to be kept same for both input and output
        # any random number has to be generated with self.rng
    def _augment(self, img, params):
        return new_img using img and params

df = MyDataFlow() # produce [input, output]
df = AugmentImageComponents(df, [MyAug()], (0,1))

[tmquan]

Hi @ppwwyyxx ,

I managed to complete the implementation ElasticDeformation augmentation.
It takes a random vector field prod(range(8, 16)) and truncates its boundary values.
Deformation rate can be controlled via this range.
Next, it performs warping the original image with this field.

It would be great if you can include in the next release.

Bests

class ElasticDeform:
    def __init__(self):
        pass
    
    def reset_state(self):
        self.rng = get_rng(self)
        
    def _augment(self, img, param):
        assert img.ndim in [2, 3], img.ndim
        du, dv = param
        shape = img.shape
        
        DU = cv2.resize(du, (shape[-2], shape[-2])) 
        DV = cv2.resize(dv, (shape[-2], shape[-2])) 
        X, Y = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]))
        indices = np.reshape(Y+DV, (-1, 1)), np.reshape(X+DU, (-1, 1))
        
        flow = img.copy()
        from scipy.ndimage.interpolation 	import map_coordinates
        for k in range(3):
            tmp = map_coordinates(img[...,k], indices, order=1)
            flow[...,k] = tmp.reshape(shape[0:2])
        flow = flow.reshape(shape)
        return flow
    def _get_augment_params(self, d):
        """
        get the augmentor parameters
        """
        size = self.rng.choice(range(8,16)) #8
        ampl = self.rng.choice(range(8,16)) #8
        du = self.rng.uniform(-ampl, ampl, size=(size, size))
        dv = self.rng.uniform(-ampl, ampl, size=(size, size))    
        
        # Dont distort at the boundaries
        du[ 0,:] = 0; du[-1,:] = 0; du[:, 0] = 0; du[:,-1] = 0;
        dv[ 0,:] = 0; dv[-1,:] = 0; dv[:, 0] = 0; dv[:,-1] = 0;
        return du, dv
    def _augment_return_params(self, d):
        """
        Augment the image and return both image and params
        """
        prms = self._get_augment_params(d)
        return (self._augment(d, prms), prms)

import glob
imgs = glob.glob(os.path.join('data/std/', 'lena.png'))
ds = ImageFromFile(imgs, channel=3, shuffle=False)
augmentors = [
    ElasticDeform(), 
    imgaug.ColorSpace(mode=cv2.COLOR_RGB2BGR),
]
ds = AugmentImageComponent(ds, augmentors)
ds = PrintData(ds, num=2) # only for debugging

gen = ds.get_data()
for dp in gen:
    newImg = np.array(dp)
    
    a = np.squeeze(cv2.imread(imgs[0]))
    b = np.squeeze(newImg)
    c = np.abs(a-b)
    concat = np.concatenate((a,b,c), axis=-2)
    concat = concat.astype(np.uint8)
    print concat.shape
    plt.figure(figsize=(20, 60))
    plt.imshow(concat, cmap=plt.cm.gray)
    plt.axis('off')
    plt.show()

[ppwwyyxx]

Thanks! The results look interesting. I should try it some day.
Please note that, I added models or augmentors only when they are very common (or sometimes for my personal convenient). I found everyone likes to do augmentation very differently, but then it's very hard for me to maintain all the pieces.
The nice thing is that you can just import and use your augmentor without adding it into tensorpack. I also had a bunch of private code with different ideas I tried, and there are also several augmentors in examples/ but not in the library. Because I'd rather keep the main library small and only include the extensions as examples.

[tmquan]

ElasticDeform is quite common in biomedical image processing, especially in segmentation.
If you look into the original U-Net paper, the authors also leveraged it intensively.

It is good to know that the main library should be kept small and common :)
I will close this issue for other people would like to perform their own custom augmentations.

Bests,

[PatWie]

I vote for adding this to the library.
@tmquan
Maybe the best way is to add an example and define it directly there.

[tmquan]

@PatWie : I am using the image2image translation for debugging the segmentation problem.
However, I met the issue such that discriminator curve and generator curve are not going as expected.
Once it is done, I will make a pull request :)

[zxpeter]

Hi @tmquan ,
Thanks for the script of ElasticDeformation.
While implementing it, I find your code may just suit for the square image, it works for me after changed the following lines:

DU = cv2.resize(du, (shape[-2], shape[-2])) 
DV = cv2.resize(dv, (shape[-2], shape[-2])) 
X, Y = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]))

to

DU = cv2.resize(du, (shape[1], shape[0])) 
DV = cv2.resize(dv, (shape[1], shape[0])) 
X, Y = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))