OpenCV

Get started with OpenCV on Android™

As our smartphones become more powerful, we can do more advanced things that previously required a high-end PC. One way to make use of the robust processing power of your smartphone is Computer Vision – the ability for a device to acquire, process, analyse and understand images the same way images are perceived by human eyes. Basically, we can use the powerful CPU in modern smartphones to interpret the images captured through the camera. Examples of use cases are face detection and recognition or simple post-processing of photographs. The best approach to using Computer Vision on Android is through library called OpenCV. Read on as Erik Hellman, research engineer at Sony, explains more.

Erik Hellman, research engineer at Sony.

The OpenCV library of programming functions has become the de-facto industry standard for software based computer vision. It is completely open-sourced (BSD licensed) and has support for most major platforms, including Android. The current version is 2.4.3, and you can download the Open CV SDK for free and start writing your OpenCV-based Android apps from the OpenCV website.

In this tutorial I will cover three things that will help you to get started with OpenCV on Android:

  1. How to acquire images from the Android camera through OpenCV.
  2. How to process an image to retrieve the parts within a certain colour range only.
  3. How to display the result of the processed image.

Before you can do any of the above, you need to load the OpenCV library. OpenCV for Android is designed as a standalone library APK that you install from Google Play. You also need to include the OpenCV SDK as an Android library project to your own Android project. Instructions on how to do this can be found in the OpenCV online documentation.

How to acquire camera images using OpenCV
In this first example, we will rely on the preview frames from the camera since we can retrieve these images relatively fast. We need to pick an appropriate size for the preview frames, as too small frames will result in a bad result when we do the processing, and too large frames will slow down everything to an unacceptable level. Since smartphone cameras often have a different set of supported sizes for preview frames, we need to pick a minimum acceptable size and go through all of them to find the correct one. OpenCV has its own Java API, and doesn’t rely on the Java-based Android Camera API. The following code opens the camera and sets the preview size to an acceptable level.

private void setupCamera() {

    if (mCamera != null) {
       VideoCapture camera = mCamera;
       mCamera = null; // Make it null before releasing...
      camera.release();
    }

    mCamera = new VideoCapture(mCameraId);

    List<Size> previewSizes = mCamera.getSupportedPreviewSizes();
    double smallestPreviewSize = 1280 * 720; // We should be smaller than this...
    double smallestWidth = 480; // Let's not get smaller than this...
    for (Size previewSize : previewSizes) {
        if (previewSize.area() < smallestPreviewSize && previewSize.width >= smallestWidth) {
            mPreviewSize = previewSize;
        }
    }

mCamera.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, mPreviewSize.width);
mCamera.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, mPreviewSize.height);
}

An important thing to remember when working with OpenCV is that we need a lot of RAM for the images we are processing. OpenCV uses matrices to represent all images so to be as efficient as possible, we will allocate the needed matrices once and re-use them for every frame.

Once we have the camera setup and the matrices allocated, we can start to grab preview frames for processing. The following code shows how we do this with OpenCV.

while (mDoProcess && mCamera != null) {
    boolean grabbed = mCamera.grab();
    if (grabbed) {
        mCamera.retrieve(mCurrentFrame,
Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB);
        // Process mCurrentFrame…
    }
}

Screen shot showing the camera frame. In this example app, you can tap a colour to display that colour only.

How to process a camera frame
We now have a preview frame in mCurrentFrame that we can use for processing. In this case, we will use the function inRange to create a mask that covers only a specific colour interval. For the best result, we will first convert the grabbed frame to HSV-format. Once we have the mask, we draw the original frame to a new matrix using the mask we got from the inRange function. The following code shows how to convert the format from RGB to HSV, generate the colour mask and drawing the result.

// Convert mCurrentFrame to HSV format and store
the result in mCurrentFrameHsv
Imgproc.cvtColor(mCurrentFrame, mCurrentFrameHsv,
Imgproc.COLOR_RGB2HSV);
// Generate the color mask based on the lower and upper
color limits
Core.inRange(mCurrentFrameHsv, mLowerColorLimit,
mUpperColorLimit, mInRangeResult);
// Clear the final image before we copy the new filtered image
mFilteredFrame.setTo(new Scalar(0, 0, 0));
// Copy the original frame using the color mask
mCurrentFrame.copyTo(mFilteredFrame, mInRangeResult);

Screen shot showing the processed camera frame when a colour has been selected. As you can see, only the red colour range is displayed.

How to display the result of the processed image
The resulting bitmap can now be used to draw to a SurfaceView or similar to show the result. The easiest way to do this is to convert the OpenCV matrix to an Android Bitmap object. There is a Util class that provides a function for this.

// Convert the matrix to an Android bitmap
Utils.matToBitmap(result, resultBitmap, true);

From this point, you can simply draw the Bitmap as usual on a SurfaceView, or assign it to an ImageView.

Deeper dive and more OpenCV code examples are available
This was a very quick walkthrough of OpenCV. To get a deeper look into this, you can download more extensive versions of these OpenCV code examples. Feel free to use this anyway you want to experiment with different OpenCV functions. There are lots of more tutorials available on the OpenCV website, and you can find additional code examples at other resources online.

So are you interested in OpenCV? Do you plan to try it out? Let us know in the comments below!

More information

Comments 13

Sort by: Newest | Oldest | Most popular

  1. By Camila Colanjo

    #1

    package com.example.testecores;

    import java.util.List;

    import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
    import org.opencv.android.Utils;
    import org.opencv.core.Core;
    import org.opencv.core.Mat;
    import org.opencv.core.Scalar;
    import org.opencv.highgui.Highgui;
    import org.opencv.highgui.VideoCapture;
    import org.opencv.imgproc.Imgproc;

    import android.graphics.Bitmap;
    import android.hardware.Camera.Size;
    import android.os.Bundle;
    import android.app.Activity;
    import android.view.Menu;

    public class MainActivity extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    }

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
    // Inflate the menu; this adds items to the action bar if it is present.
    getMenuInflater().inflate(R.menu.main, menu);
    return true;
    }

    private void setupCamera() {

    VideoCapture mCamera;
    if (mCamera != null) {
    VideoCapture camera = mCamera;
    mCamera = null; // Make it null before releasing…
    camera.release();
    }

    VideoCapture mCameraId;
    mCamera = new VideoCapture(mCameraId);

    List previewSizes = mCamera.getSupportedPreviewSizes();
    double smallestPreviewSize = 1280 * 720; // We should be smaller than this…
    double smallestWidth = 480; // Let’s not get smaller than this…
    org.opencv.core.Size mPreviewSize;
    for (org.opencv.core.Size previewSize : previewSizes) {
    if (previewSize.area() = smallestWidth) {
    mPreviewSize = previewSize;
    }
    }

    mCamera.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, mPreviewSize.width);
    mCamera.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, mPreviewSize.height);
    }
    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

    Object mCurrentFrameHsv;
    Mat mCurrentFrame;
    // Convert mCurrentFrame to HSV format and store
    // the result in mCurrentFrameHsv
    Imgproc.cvtColor(mCurrentFrame, (Mat) mCurrentFrameHsv, Imgproc.COLOR_RGB2HSV);
    Object mLowerColorLimit;
    Object mInRangeResult;
    Object mUpperColorLimit;
    // Generate the color mask based on the lower and upper
    // color limits
    Core.inRange(mCurrentFrameHsv, mLowerColorLimit, mUpperColorLimit, mInRangeResult);
    Mat mFilteredFrame;
    // Clear the final image before we copy the new filtered image
    mFilteredFrame.setTo(new Scalar(0, 0, 0));
    // Copy the original frame using the color mask
    mCurrentFrame.copyTo(mFilteredFrame, mInRangeResult);
    Bitmap resultBitmap;
    Mat result;
    Utils.matToBitmap(result, resultBitmap, true);
    return null;
    }
    }
    This is not working

  2. By thin norakseiha

    #3

    I do interesting OpenCV, but I have problem with document guide or tutorial.

  3. By Eddie Patton

    #5

    Hi
    I compiled and ran your code for Nexus 7 and “Unfortunately, OpenCVTutorial has stopped”

    I can email you the full log, but here is a snipped when thing start to go bad. If you get a chance to look at it, I’d love to get it running.

    Cheers2u

    01-15 11:15:29.407: I/ActivityManager(509): Start proc com.example.opencvtutorial_sony for activity com.example.opencvtutorial_sony/.OpenCVTutorial_sony: pid=27137 uid=10049 gids={50049}
    01-15 11:15:29.427: D/dalvikvm(27137): Debugger has detached; object registry had 1 entries
    01-15 11:15:29.447: W/dalvikvm(27137): VFY: unable to find class referenced in signature (Lorg/opencv/core/Size;)
    01-15 11:15:29.447: W/dalvikvm(27137): VFY: unable to resolve instance field 1237
    01-15 11:15:29.447: D/dalvikvm(27137): VFY: replacing opcode 0×53 at 0×0006
    01-15 11:15:29.447: W/dalvikvm(27137): Unable to resolve superclass of Lcom/example/opencvtutorial_sony/OpenCVTutorial_sony$OpenCVLoaderCallback; (742)

  4. By Tharaka Nirmana

    #6

    Hi, I am a final year student doing IT. For my final year research, I have to do an android project. It is something to do with the lottery system. All that I want is to extract the lottery numbers and the draw date from the lottery after capturing it from the android camera. I have been trying and trying but still not satisfied. I contacted some indian developers but still I was unable to get what I want. I did post on stack over flow as well:

    1. http://stackoverflow.com/questions/16663504/extract-text-from-a-captured-image?noredirect=1#comment23973954_16663504

    2. http://stackoverflow.com/questions/16874949/extracting-information-from-captured-image-in-android/16880026?noredirect=1#comment24352493_16880026

    3. http://stackoverflow.com/questions/17237676/object-detection-for-android-with-tesseract-or-opencv?noredirect=1#comment24978228_17237676

    Please help me out with this. It would be a great, very great help for me and my group members!!!

    Thank you!

  5. By Narasimha Kaushik

    #7

    Dear Mr Hellman,

    I am keen on developing vision apps on the go on my android phone. I have downloaded the OpenCV manager which comes with the built-in OpenCV 2.4.5. Now which compiler-IDE do I install and how do I integrate the OpenCV library to get started with vision apps?

  6. By Kamil Sindi

    #8

    For some odd reason I’m getting the error:

    E/dalvikvm(22477): Could not find class ‘com.sonymobile.opencvtutorial.OpenCVTutorial$OpenCVLoaderCallback’, referenced from method com.sonymobile.opencvtutorial.OpenCVTutorial.surfaceCreated

    I can clearly see the subclass OpenCVLoaderCallback. Can’t figure out what I’m getting this error.

    • By Kamil Sindi

      #9

      Fyi I get this warning before: Unable to resolve superclass of Lcom/sonymobile/opencvtutorial/OpenCVTutorial$OpenCVLoaderCallback; (60)

      I’m using Android 2.4.1 on Nexus 4 and OpenCV 2.4.3.1.

      • By Erik Hellman

        #10

        Besides the code linked in this tutorial you also need to download the OpenCV Android SDK from opencv.org and include the SDK Library Project before building. Once you’ve downloaded and unpacked the SDK zip-file you can look in the sub-folder sdk/java and you’ll see a standard Android project. This is the Library Project you need to include. If you are using Eclipse you can find the guide for that at http://developer.android.com/tools/projects/projects-eclipse.html#SettingUpLibraryProject

        • By Kamil Sindi

          #11

          Thanks. That was the problem. The library wasn’t being reference correctly.

  7. By Zachary Klippenstein

    #12

    The ant build file in the sample zip apparently points to the author’s home directory:

    BUILD FAILED
    /Users//OpenCVTutorial/build.xml:90: Cannot find /Users/erikhe
    lman/android-sdk-macosx/tools/ant/build.xml imported from /Users//OpenCVTutorial/build.xml

    • By Zachary Klippenstein

      #13

      Never mind, running android update project fixes it.

1-13 of 13 comments.