What is object detection?

Object detection is a computer vision task in machine learning that involves identifying and localizing objects within an image or a video. Object detection is a crucial task for many applications, such as autonomous driving, surveillance systems, and medical imaging.

There are several popular object detection algorithms, including YOLO (You Only Look Once), Faster R-CNN (Region-based Convolutional Neural Network), and SSD (Single Shot Detector). These algorithms have different strengths and weaknesses, and the choice of algorithm depends on the specific requirements of the application.

How should I build this object detector?

I build an object detection as a play for the react play. React play is an open-source project containing apps(plays) developed under a react project. This is a react application, which means a javascript project. So I use the TensorFlow.js library. Tenseflow.js is a library for developing ML models in JavaScript and using ML directly in the browser or Node.js. There are already trained ML models. So here I use one (@tensorflow-models/coco-ssd) of them to detect the objects using video of the object.

Load the web camera and capture the object as a video

const webcamRef = useRef(null);
const canvasRef = useRef(null);

useEffect(() => {
    runCoco();
  }, []);

const runCoco = async () => {
  const net = await cocossd.load();
  setInterval(() => {
    detect(net);
  }, 10);
};

const detect = async (net) => {
    if (
      typeof webcamRef.current !== 'undefined' &&
      webcamRef.current !== null &&
      webcamRef.current.video.readyState === 4
    ) {
      const video = webcamRef.current.video;
      const videoWidth = webcamRef.current.video.videoWidth;
      const videoHeight = webcamRef.current.video.videoHeight;

      webcamRef.current.video.width = videoWidth;
      webcamRef.current.video.height = videoHeight;

      canvasRef.current.width = videoWidth;
      canvasRef.current.height = videoHeight;

      const obj = await net.detect(video);

      const ctx = canvasRef.current.getContext('2d');
      drawRect(obj, ctx);
    }
  };

Pass the video to the trained AI model and show the prediction results near the object.

const drawRect = (detections, ctx) => {
  detections.forEach((prediction) => {
    const [x, y, width, height] = prediction['bbox'];
    const text = prediction['class'];

    const color = Math.floor(Math.random() * 16777215).toString(16);
    ctx.strokeStyle = '#' + color;
    ctx.font = '18px Arial';

    ctx.beginPath();
    ctx.fillStyle = '#' + color;
    ctx.fillText(text, x, y);
    ctx.rect(x, y, width, height);
    ctx.stroke();
  });
};

Libraries used for application development.

@tensorflow/tfjs

Used to train and deploy machine learning models using JavaScript, making it easier to develop and deploy machine learning applications that run entirely in the browser.

@tensorflow-models/coco-ssd

pre-trained object detection model that can be used with TensorFlow.js. It is based on the Single Shot Multibox Detector (SSD) architecture and is trained on the Microsoft Common Objects in Context (COCO) dataset, which contains over 330,000 images with more than 2.5 million object instances labeled across 80 different object categories.

react-webcam

Popular open-source library for integrating webcam capture capabilities into React.js applications.

You can refer to the full code from the repo of the react play.

References:

1. Tenserflow.js

2. Pre-trained Models

3. Object detection pre-trained tenserflow.js model

Thank You!