How to crop and resize an image from an HTML Canvas, Image or Video element using Tensorflow.js tf.image.cropAndResize

This is a quick post to show how you can use tf.image.cropAndResize to crop and resize an image to a square using the function tf.image.cropAndResize.

This function was created for a specific purpose to cutout the bounding boxes given by a face detection algorithm. But you can use it for other purposes too!

Your image can come from an HTML Canvas, an HTML Image or even from an HTML Video Element.
First we convert it to a Tensor using tf.browser.fromPixels:

 let imageTensor = tf.browser.fromPixels(imageData);

where imageData: (tf.PixelData | ImageData | HTMLImageElement | HTMLCanvasElement | HTMLVideoElement)

Since the image has pixel values from [0, 255] we need to normalize it with:

const offset = tf.scalar(255.0);
const normalized = imageTensor.div(offset);

Now we are almost ready to use the tf.image.cropAndResize() function:

tf.image.cropAndResize (image, boxes, boxInd, cropSize, method?, extrapolationValue?)

Parameters:

  • image: a batched tensor(4-D), i.e. a list of normalized images to resize in Tensor format.
  • boxes: a list of boxes to use in the cropping. Each box specifies coordinates as [top, left, bottom, right]. The coordinates are normalized, which means they have to be between [0,1]
  • boxInd – 1D Tensorf which specifies the index of the image which the box refers to.
  • cropSize – this controls the width and height to which the image is resized to after cropping

In this example, taken from a Tensorflow.js widget that I created for demonstrating the Fast Style Transfer, we are resizing an image and cropping it so it becomes a square:

const imgSize = Math.min(imageData.width, imageData.height);
const left = (imageData.width - imgSize) / 2;
const top = (imageData.height - imgSize) / 2;
const right = (imageData.width + imgSize) / 2;
const bottom = (imageData.height + imgSize) / 2;
let boxes = [[top / imageData.height, left / imageData.width, bottom / imageData.height, right / imageData.width]];
result = tf.image.cropAndResize(batched, boxes, [0], [size,size])

You can check out the Github Repository to see this function in action:

https://github.com/armindocachada/fast-style-transfer-tensorflowjs

Resources

Recommended Courses for Data Science

Recommended Websites

Tensorflow.js API Documentation
https://js.tensorflow.org/api/latest/#image.cropAndResize