License Plate Recognition in CSharp

From Emgu CV: OpenCV in .NET (C#, VB, C++ and more)
Jump to navigation Jump to search

System Requirement

Component Requirement Detail
Emgu CV Version 2.0.0.0 Alpha and above
Operation System All OS Except Windows Phone Tesseract do not support Windows Phone C++ compiler.

License Plate Recognition

According to wikipedia

Automatic number plate recognition (ANPR; see also other names below) is a mass surveillance method that uses optical character recognition on images to read the license plates on vehicles. As of 2006, systems can scan number plates at around one per second on cars traveling up to 100 mph (160 km/h).[citation needed] They can use existing closed-circuit television or road-rule enforcement cameras, or ones specifically designed for the task. They are used by various police forces and as a method of electronic toll collection on pay-per-use roads and monitoring traffic activity, such as red light adherence in an intersection.
ANPR can be used to store the images captured by the cameras as well as the text from the license plate, with some configurable to store a photograph of the driver. Systems commonly use infrared lighting to allow the camera to take the picture at any time of the day. A powerful flash is included in at least one version of the intersection-monitoring cameras, serving both to illuminate the picture and to make the offender aware of his or her mistake. ANPR technology tends to be region-specific, owing to plate variation from place to place.

This tutorial's approach to ANPR is divided into two stage

  • In the first stage, we perform license plate region detection
  • In the second stage, we perform OCR on the license plate to recover the license number

Assumption

This tutorial assumes that ANPR is performed on European license plate. Within the source code, you will find the following lines of code that indicates only rectangle with width-height ratio in the range of (3.0, 8.0) is considered. 

double whRatio = (double)box.size.Width / box.size.Height;
if (!(3.0 < whRatio && whRatio < 8.0))
...

If you are performing ANPR on different region, you will have to change this threshold to best match the characteristic of the license plate from that region.

Recognition Accuracy

This tutorial is written to demonstrate how a simple ANPR system can be implement. This system is not robust and recognition accuracy might be low. A few places in this algorithm that affect the recognition accuracy includes

  • The license plate region detection is not robust. The contour extraction algorithm requires high contrast. If the car is white or silver, the license plate region is less likely to be recovered.
  • The OCR engine is not tuned for ANPR. If the license plate in your region contains a certain limited set of characters, you should tuned the OCR to be more sensitive to the specific character set. Visit tesseract OCR engine for more information.

Complete Source Code

Emgu CV 3.x

Click to view source code 

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Drawing;
using System.Text;
using Emgu.CV;
using Emgu.CV.CvEnum;
using Emgu.CV.OCR;
using Emgu.CV.Structure;
using Emgu.CV.Util;
using Emgu.Util;

namespace LicensePlateRecognition
{
   /// <summary>
   /// A simple license plate detector
   /// </summary>
   public class LicensePlateDetector : DisposableObject
   {
      /// <summary>
      /// The OCR engine
      /// </summary>
      private Tesseract _ocr;

      /// <summary>
      /// Create a license plate detector
      /// </summary>
      /// <param name="dataPath">
      /// The datapath must be the name of the parent directory of tessdata and
      /// must end in / . Any name after the last / will be stripped.
      /// </param>
      public LicensePlateDetector(String dataPath)
      {
         //create OCR engine
         _ocr = new Tesseract(dataPath, "eng", OcrEngineMode.TesseractCubeCombined);
         _ocr.SetVariable("tessedit_char_whitelist", "ABCDEFGHIJKLMNOPQRSTUVWXYZ-1234567890");
      }

      /// <summary>
      /// Detect license plate from the given image
      /// </summary>
      /// <param name="img">The image to search license plate from</param>
      /// <param name="licensePlateImagesList">A list of images where the detected license plate regions are stored</param>
      /// <param name="filteredLicensePlateImagesList">A list of images where the detected license plate regions (with noise removed) are stored</param>
      /// <param name="detectedLicensePlateRegionList">A list where the regions of license plate (defined by an MCvBox2D) are stored</param>
      /// <returns>The list of words for each license plate</returns>
      public List<String> DetectLicensePlate(
         IInputArray img, 
         List<IInputOutputArray> licensePlateImagesList, 
         List<IInputOutputArray> filteredLicensePlateImagesList, 
         List<RotatedRect> detectedLicensePlateRegionList)
      {
         List<String> licenses = new List<String>();
         using (Mat gray = new Mat())
         using (Mat canny = new Mat())
         using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
         {
            CvInvoke.CvtColor(img, gray, ColorConversion.Bgr2Gray);
            CvInvoke.Canny(gray, canny, 100, 50, 3, false);
            int[,] hierachy = CvInvoke.FindContourTree(canny, contours, ChainApproxMethod.ChainApproxSimple);
            
            FindLicensePlate(contours, hierachy, 0, gray, canny, licensePlateImagesList, filteredLicensePlateImagesList, detectedLicensePlateRegionList, licenses);
         }
         return licenses;
      }

      private static int GetNumberOfChildren(int[,] hierachy, int idx)
      {
         //first child
         idx = hierachy[idx,2];
         if (idx < 0)
            return 0;
         
         int count = 1;
         while (hierachy[idx,0] > 0)
         {
            count++;
            idx = hierachy[idx,0];
         }
         return count;
      }

      private void FindLicensePlate(
         VectorOfVectorOfPoint contours, int[,] hierachy, int idx, IInputArray gray, IInputArray canny,
         List<IInputOutputArray> licensePlateImagesList, List<IInputOutputArray> filteredLicensePlateImagesList, List<RotatedRect> detectedLicensePlateRegionList,
         List<String> licenses)
      {
         for (; idx >= 0;  idx = hierachy[idx,0])
         {
            int numberOfChildren = GetNumberOfChildren(hierachy, idx);      
            //if it does not contains any children (charactor), it is not a license plate region
            if (numberOfChildren == 0) continue;

            using (VectorOfPoint contour = contours[idx])
            {
               if (CvInvoke.ContourArea(contour) > 400)
               {
                  if (numberOfChildren < 3)
                  {
                     //If the contour has less than 3 children, it is not a license plate (assuming license plate has at least 3 charactor)
                     //However we should search the children of this contour to see if any of them is a license plate
                     FindLicensePlate(contours, hierachy, hierachy[idx, 2], gray, canny, licensePlateImagesList,
                        filteredLicensePlateImagesList, detectedLicensePlateRegionList, licenses);
                     continue;
                  }

                  RotatedRect box = CvInvoke.MinAreaRect(contour);
                  if (box.Angle < -45.0)
                  {
                     float tmp = box.Size.Width;
                     box.Size.Width = box.Size.Height;
                     box.Size.Height = tmp;
                     box.Angle += 90.0f;
                  }
                  else if (box.Angle > 45.0)
                  {
                     float tmp = box.Size.Width;
                     box.Size.Width = box.Size.Height;
                     box.Size.Height = tmp;
                     box.Angle -= 90.0f;
                  }

                  double whRatio = (double) box.Size.Width/box.Size.Height;
                  if (!(3.0 < whRatio && whRatio < 10.0))
                     //if (!(1.0 < whRatio && whRatio < 2.0))
                  {
                     //if the width height ratio is not in the specific range,it is not a license plate 
                     //However we should search the children of this contour to see if any of them is a license plate
                     //Contour<Point> child = contours.VNext;
                     if (hierachy[idx, 2] > 0)
                        FindLicensePlate(contours, hierachy, hierachy[idx, 2], gray, canny, licensePlateImagesList,
                           filteredLicensePlateImagesList, detectedLicensePlateRegionList, licenses);
                     continue;
                  }

                  using (UMat tmp1 = new UMat())
                  using (UMat tmp2 = new UMat())
                  {
                     PointF[] srcCorners = box.GetVertices();
                     
                     PointF[] destCorners = new PointF[] {
                        new PointF(0, box.Size.Height - 1),
                        new PointF(0, 0),
                        new PointF(box.Size.Width - 1, 0), 
                        new PointF(box.Size.Width - 1, box.Size.Height - 1)};
                     
                     using (Mat rot = CameraCalibration.GetAffineTransform(srcCorners, destCorners))
                     {
                        CvInvoke.WarpAffine(gray, tmp1, rot, Size.Round(box.Size));           
                     }

                     //resize the license plate such that the front is ~ 10-12. This size of front results in better accuracy from tesseract
                     Size approxSize = new Size(240, 180);
                     double scale = Math.Min(approxSize.Width/box.Size.Width, approxSize.Height/box.Size.Height);
                     Size newSize = new Size( (int)Math.Round(box.Size.Width*scale),(int) Math.Round(box.Size.Height*scale));
                     CvInvoke.Resize(tmp1, tmp2, newSize, 0, 0, Inter.Cubic);

                     //removes some pixels from the edge
                     int edgePixelSize = 2;
                     Rectangle newRoi = new Rectangle(new Point(edgePixelSize, edgePixelSize),
                        tmp2.Size - new Size(2*edgePixelSize, 2*edgePixelSize));
                     UMat plate = new UMat(tmp2, newRoi);

                     UMat filteredPlate = FilterPlate(plate);

                     Tesseract.Character[] words;
                     StringBuilder strBuilder = new StringBuilder();
                     using (UMat tmp = filteredPlate.Clone())
                     {
                        _ocr.Recognize(tmp);
                        words = _ocr.GetCharacters();

                        if (words.Length == 0) continue;

                        for (int i = 0; i < words.Length; i++)
                        {
                           strBuilder.Append(words[i].Text);
                        }
                     }

                     licenses.Add(strBuilder.ToString());
                     licensePlateImagesList.Add(plate);
                     filteredLicensePlateImagesList.Add(filteredPlate);
                     detectedLicensePlateRegionList.Add(box);

                  }
               }
            }
         }
      }

      /// <summary>
      /// Filter the license plate to remove noise
      /// </summary>
      /// <param name="plate">The license plate image</param>
      /// <returns>License plate image without the noise</returns>
      private static UMat FilterPlate(UMat plate)
      {
         UMat thresh = new UMat();
         CvInvoke.Threshold(plate, thresh, 120, 255, ThresholdType.BinaryInv);
         //Image<Gray, Byte> thresh = plate.ThresholdBinaryInv(new Gray(120), new Gray(255));

         Size plateSize = plate.Size;
         using (Mat plateMask = new Mat(plateSize.Height, plateSize.Width, DepthType.Cv8U, 1))
         using (Mat plateCanny = new Mat())
         using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
         {
            plateMask.SetTo(new MCvScalar(255.0));
            CvInvoke.Canny(plate, plateCanny, 100, 50);
            CvInvoke.FindContours(plateCanny, contours, null, RetrType.External, ChainApproxMethod.ChainApproxSimple);

            int count = contours.Size;
            for (int i = 1; i < count; i++)
            {
               using (VectorOfPoint contour = contours[i])
               {

                  Rectangle rect = CvInvoke.BoundingRectangle(contour);
                  if (rect.Height > (plateSize.Height >> 1))
                  {
                     rect.X -= 1; rect.Y -= 1; rect.Width += 2; rect.Height += 2;
                     Rectangle roi = new Rectangle(Point.Empty, plate.Size);
                     rect.Intersect(roi);
                     CvInvoke.Rectangle(plateMask, rect, new MCvScalar(), -1);
                     //plateMask.Draw(rect, new Gray(0.0), -1);
                  }
               }

            }

            thresh.SetTo(new MCvScalar(), plateMask);
         }

         CvInvoke.Erode(thresh, thresh, null, new Point(-1, -1), 1, BorderType.Constant, CvInvoke.MorphologyDefaultBorderValue);
         CvInvoke.Dilate(thresh, thresh, null, new Point(-1, -1), 1, BorderType.Constant, CvInvoke.MorphologyDefaultBorderValue);

         return thresh;
      }

      protected override void DisposeObject()
      {
         _ocr.Dispose();
      }
   }
}

Emgu CV 2.x

Click to view source code 

using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using Emgu.Util;
using Emgu.CV;
using Emgu.CV.Structure;
using tessnet2;
using System.Diagnostics;

namespace LicensePlateRecognition
{
   /// <summary>
   /// A license plate detector
   /// </summary>
   public class LicensePlateDetector : DisposableObject
   {
      private Tesseract _ocr;

      /// <summary>
      /// Create a license plate detector
      /// </summary>
      public LicensePlateDetector()
      {
         //create OCR
         _ocr = new Tesseract();

         //You can download more language definition data from
         //http://code.google.com/p/tesseract-ocr/downloads/list
         //Languages supported includes:
         //Dutch, Spanish, German, Italian, French and English
         _ocr.Init("eng", false);
      }

      /// <summary>
      /// Detect license plate from the given image
      /// </summary>
      /// <param name="img">The image to search license plate from</param>
      /// <param name="licensePlateList">A list of images where the detected license plate region is stored</param>
      /// <param name="filteredLicensePlateList">A list of images where the detected license plate region with noise removed is stored</param>
      /// <param name="boxList">A list where the region of license plate, defined by an MCvBox2D is stored</param>
      /// <returns>The list of words for each license plate</returns>
      public List<List<Word>> DetectLicensePlate(Image<Bgr, byte> img, List<Image<Gray, Byte>> licensePlateList, List<Image<Gray, Byte>> filteredLicensePlateList, List<MCvBox2D> boxList)
      {
         //Stopwatch w = Stopwatch.StartNew();
         List<List<Word>> licenses = new List<List<Word>>();
         using (Image<Gray, byte> gray = img.Convert<Gray, Byte>())
         using (Image<Gray, Byte> canny = new Image<Gray, byte>(gray.Size))
         using (MemStorage stor = new MemStorage())
         {
            CvInvoke.cvCanny(gray, canny, 100, 50, 3);

            Contour<Point> contours = canny.FindContours(
                 Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
                 Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_TREE,
                 stor);
            FindLicensePlate(contours, gray, canny, licensePlateList, filteredLicensePlateList, boxList, licenses);
         }
         //w.Stop();
         return licenses;
      }

      private void FindLicensePlate(
         Contour<Point> contours, Image<Gray, Byte> gray, Image<Gray, Byte> canny,
         List<Image<Gray, Byte>> licensePlateList, List<Image<Gray, Byte>> filteredLicensePlateList, List<MCvBox2D> boxList,
         List<List<Word>> licenses)
      {
         for (; contours != null; contours = contours.HNext)
         {
            Contour<Point> approxContour = contours.ApproxPoly(contours.Perimeter * 0.05, contours.Storage);

            if (approxContour.Area > 100 && approxContour.Total == 4)
            {
               //img.Draw(contours, new Bgr(Color.Red), 1);
               if (!IsParallelogram(approxContour.ToArray()))
               {
                  Contour<Point> child = contours.VNext;
                  if (child != null)
                     FindLicensePlate(child, gray, canny, licensePlateList, filteredLicensePlateList, boxList, licenses);
                  continue;
               }

               MCvBox2D box = approxContour.GetMinAreaRect();

               double whRatio = (double)box.size.Width / box.size.Height;
               if (!(3.0 < whRatio && whRatio < 8.0))
               {
                  Contour<Point> child = contours.VNext;
                  if (child != null)
                     FindLicensePlate(child, gray, canny, licensePlateList, filteredLicensePlateList, boxList, licenses);
                  continue;
               }

               Image<Gray, Byte> plate = gray.Copy(box);
               Image<Gray, Byte> filteredPlate = FilterPlate(plate);

               List<Word> words;
               using (Bitmap bmp = filteredPlate.Bitmap)
                  words = _ocr.DoOCR(bmp, filteredPlate.ROI);

               licenses.Add(words);
               licensePlateList.Add(plate);
               filteredLicensePlateList.Add(filteredPlate);
               boxList.Add(box);
            }
         }
      }

      /// <summary>
      /// Check if the four points forms a parallelogram
      /// </summary>
      /// <param name="pts">The four points that defines a polygon</param>
      /// <returns>True if the four points defines a parallelogram</returns>
      private static bool IsParallelogram(Point[] pts)
      {
         LineSegment2D[] edges = PointCollection.PolyLine(pts, true);

         double diff1 = Math.Abs(edges[0].Length - edges[2].Length);
         double diff2 = Math.Abs(edges[1].Length - edges[3].Length);
         if (diff1 / edges[0].Length <= 0.05 && diff1 / edges[2].Length <= 0.05
            && diff2 / edges[1].Length <= 0.05 && diff2 / edges[3].Length <= 0.05)
         {
            return true;
         }
         return false;
      }

      /// <summary>
      /// Filter the license plate to remove noise
      /// </summary>
      /// <param name="plate">The license plate image</param>
      /// <returns>License plate image without the noise</returns>
      private static Image<Gray, Byte> FilterPlate(Image<Gray, Byte> plate)
      {
         Image<Gray, Byte> thresh = plate.ThresholdBinaryInv(new Gray(120), new Gray(255));

         using (Image<Gray, Byte> plateMask = new Image<Gray, byte>(plate.Size))
         using (Image<Gray, Byte> plateCanny = plate.Canny(new Gray(100), new Gray(50)))
         using (MemStorage stor = new MemStorage())
         {
            plateMask.SetValue(255.0);
            for (
               Contour<Point> contours = plateCanny.FindContours(
                  Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
                  Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_EXTERNAL,
                  stor);
               contours != null; contours = contours.HNext)
            {
               Rectangle rect = contours.BoundingRectangle;
               if (rect.Height > (plate.Height >> 1))
               {
                  rect.X -= 1; rect.Y -= 1; rect.Width += 2; rect.Height += 2;
                  rect.Intersect(plate.ROI);

                  plateMask.Draw(rect, new Gray(0.0), -1);
               }
            }

            thresh.SetValue(0, plateMask);
         }

         thresh._Erode(1);
         thresh._Dilate(1);

         return thresh;
      }

      protected override void DisposeObject()
      {
         _ocr.Dispose();
      }
   }
}

Result

License Plate Recognition
Retrieved from "https://www.emgu.com/wiki/index.php?title=License_Plate_Recognition_in_CSharp&oldid=1963"