Motion

Detection

in JavaScript





Paulo Ávila - 4 July 2013
BarcelonaJS

Who's Paulo?


Brazil > California > Bilbao > Barcelona

Information Architect - Waytostay

JavaScript developer for ~10 years


NOT expert in video or motion graphics

... nor <canvas>

Agenda


  1. Questions
  2. The Challenge (overview)
  3. Detect Movement
    1. Video to Canvas
    2. Canvas basics (context)
    3. Photoshop
  4. Detect Motion (direction)
  5. Applications

    The Challenge


    Perform real-time image
    stabilization in the browser


    1. Counteract motion
    2. Determine motion direction
    3. Detect movement*


    The Hope




    Motion Tracking in Adobe After Effects


    (credit: Andrew Kramer)

    <video>2<canvas>

    No access to image data in video tag
    Paint each frame onto <canvas>

    var srcVideo = document.getElementById('src-video'),
        srcCanvas = document.createElement('canvas'),
        srcCtx = srcCanvas.getContext('2d');
        
    srcCanvas.width = srcVideo.width;
    srcCanvas.height = srcVideo.height;
    
    function frameLoop() {
        srcCtx.drawImage(srcVideo, 0, 0, srcVideo.width, srcVideo.height);
        
        setTimeout(frameLoop, 1000 / 40);
    }

    <canvas>

    context = canvas.getContext('2d');

    context.getImageData(x, y, w, h);
    context.createImageData(w, h);

    imageData.data


    • Returns an array containing pixel data
    • Each series of 4 elements represents a pixel (rgba)

    • 4px by 3px grid
    • length = (4 * 3) * 4 = 48

    [
      r,g,b,a,   r,g,b,a,   r,g,b,a,   r,g,b,a,
      r,g,b,a,   r,g,b,a,   r,g,b,a,   r,g,b,a,
      r,g,b,a,   r,g,b,a,   r,g,b,a,   r,g,b,a
    ] 

    Detecting Movement


    function frameLoop() {
        // Create a blank image data where the blended image (diff) will be drawn
        diffImageData = srcCtx.createImageData(canvasWidth, canvasHeight);
    
        // Get image data from the image drawn on the source canvas.
        srcFrameImageData = srcCtx.getImageData(0, 0, canvasWidth, canvasHeight);
    
        // Create an image if the previous image doesn’t exist (1st iteration)
        if (!prevFrameImageData) {
            prevFrameImageData = srcFrameImageData;
        }
    
        movementPxCount = compareFrames(diffImageData,
                                        srcFrameImageData,
                                        prevFrameImageData);
    
        // Store the current frame's image data for the next iteration.
        prevFrameImageData = srcFrameImageData;
    
        setTimeout(frameLoop, 1000 / 40);
    }

    Comparing Frames

    Demo

    // diff between 2 frames' luma to be considered as movement (a.k.a. sensitivity)
    var LUMA_DIFF_THRESHOLD = 20;
    
    function compareFrames(deltaImgData, imgData1, imgData2) {
      var imgDataSubPixels1 = imgData1.data,
          imgDataSubPixels2 = imgData2.data,
          deltaImgDataSubPixels = deltaImgData.data,
          subPixelCount = imgDataSubPixels1.length;
    
      // For each sub-pixel channel (rgba) of every pixel from the image data...
      while (i < subPixelCount) {
        // Get the grayscale (achromatic) value of the pixel.
        pxLuminance1 = luma(imgDataSubPixels1[i],
                            imgDataSubPixels1[i + 1],
                            imgDataSubPixels1[i + 2]);
        pxLuminance2 = luma(imgDataSubPixels2[i],
                            imgDataSubPixels2[i + 1],
                            imgDataSubPixels2[i + 2]);
    
        // Get the difference in brightness of both pixels.
        pxLuminanceDiff = Math.abs(pxLuminance1 - pxLuminance2);
    
        // If brightness difference is "significant", set it to white, else black.
        if (pxLuminanceDiff >= LUMA_DIFF_THRESHOLD) {
          pxBinaryDiff = 255;
          diffPxCount++;
        } else {
          pxBinaryDiff = 0;
        }
    
        // Write the difference in luminosity to the blended canvas context.
        // We write the same value to all color channels to show it in white since
        // there's no noticeable performance difference if we only use one color.
        deltaImgDataSubPixels[  i  ] = pxBinaryDiff;   // r
        deltaImgDataSubPixels[i + 1] = pxBinaryDiff;   // g
        deltaImgDataSubPixels[i + 2] = pxBinaryDiff;   // b
        deltaImgDataSubPixels[i + 3] = 255;            // a
    
        // Advance to the next pixel (a set of rgba channels).
        i = (i + 4);
      }
    
      return diffPxCount;
    }
    function luma(r, g, b) {
      return (r + g + b) / 3;
      //return (r+r+r + g+g+g+g+g+g + b) / 10;
    }

    Photoshop


    Visual demo of what's going on...

    • Blend mode: Difference
      • Compares two layers
      • Looks at their pixel color information
      • Subtracts the darker of the two from the other

    Direction of Motion



    1. Get number of pixels
    2. Shift the previous frame to one direction
    3. Get number of pixels
    4. If fewer pixels when shifted
      1. direction of shift is the same as motion
    5. else
      1. direction of motion is opposite as the shift

    Repeat for y axis

    Applications


    • Suggestions?

    Keep in touch!


    Open Source Project: Shaky Shake JS



    Google+: http://goo.gl/XbFFd


    GitHub:  https://github.com/demoive


    Twitter: @demoive

    Motion Detection in JavaScript

    By Paulo Ávila

    Motion Detection in JavaScript

    • 2,310