Design and Development of an Embedded Real Time Vision Enhancement System Using Image Fusion

Biography

6 pages
0 views

Please download to get full document.

View again

of 6
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Description
Accepted for Publication, Author Submitted Version can be downloaded for personal use "There may be changes as per the reviewer comments in the final version published. “© © 20xx IEEE. Personal use of this material is permitted. Permission from
Transcript
    Abstract   —   Vision enhancement using multi-view image fusion technique has diversified applications like surveillance systems to monitor activities in crowded areas. The images taken from different viewpoints need to be aligned and fused as they are not aligned to each other for providing better scene awareness. Most of the existing systems are not capable of providing real time information successfully. Thus there is need to develop robust real time system that performs automated image registration and fusion. This paper deals with the design and development of real time vision enhancement system for surveillance applications using real time image fusion. Image registration and fusion algorithm is developed and validated using MATLAB. The developed algorithm captures video from moving camera, where frames are aligned to same coordinate system using feature based image registration technique and fuses the registered frames using DWT based maximum selection image fusion. The developed feature based image registration algorithm obtains the corners in the reference and unregistered frames, calculates the corner matching point pairs and estimates the geometric transformation matrix with all distortion parameters such as rotation, translation and scaling factor. Using obtained estimated transformation matrix, the geometric transformation is done to unregistered frame to obtain the registered frame. The reference and registered frames are fused using DWT based maximum selection image fusion. After validation, the enhancement system is developed by making a target processor specific image registration and fusion Simulink model and porting on DM642EVM DSP processor with the help of Embedded Coder and CCStudio. The developed system is tested for   achieving functional requirements with various designed test cases and its performance is evaluated. The developed system meets all system specifications set. The developed vision enhancement system gives 30 fps and it is  jitter free. The response time of the developed system is 155 ms. The execution time of un-optimised implemented real time image fusion algorithm on DM642 processor gives 770 ms. The algorithm is optimised further to give average execution time of 740 ms. The developed system can be extended to two dissimilar cameras moving in different directions. Key Words: Vision Enhancement System, Image Registration, Image Fusion, DM642EVM Processor  N.Swathi, is with MSRSAS, Bangalore India (email:nimmagaddasriswathi@ gmail.com) Chandan N is with Department of Computer Engineering, MSRSAS, Bangalore India (e-mail: chandan@msrsas.org). Viswanath K, is with Department of EEE, MSRSAS, Bangalore India (e-mail: viswanath@msrsas.org).  Naveen K.S is now with System controls, Bangalore India.   I.   I  NTRODUCTION  Vision enhancement using Image Fusion plays an important role in diversified domains of research. Those fields include mainly medical imaging, microscopic imaging, remote sensing, computer vision, image processing, robotics, geosciences and robot vision. By obtaining information from one or more cameras and fusing one can monitor activities in crowded areas, buildings etc where surveillance is required as a preventive measure. Various streams of image processing require high spectral resolution in a single image. Thus traffic monitoring system, long range fusion systems all of them use image processing techniques. But most of the existing systems are not capable of providing this type of information successfully. Thus there is need for Real time image fusion systems which require automated image alignment. Also frequent problem arises when images taken, by different sensors or from different viewpoints or at different times, need to be compared and fused. The images need to be aligned with one another so that differences can be detected. All of these  problems, and many related variations, are solved by developing robust system that performs image registration [7]. This paper focuses on real time image fusion for providing vision enhancement by using specific image registration and fusion algorithms in a step by step manner. The main applications motivating this paper would be a) Medical imaging, b) Military and Traffic Surveillance Systems and c) Multi sensor data fusion systems. Particular applications mentioned are remote sensing, medical diagnosis, surveillance systems, and pattern recognition. These illustrate several of the motivating applications, such as the-state-of-the-art for image fusion. Medical Imaging: Image Fusion has got wide importance in medical imaging and diagnosis  purpose. For example in medical science magnetic resonance image (MRI) provides more information of anatomical structures whereas computed tomography (CT) provides detailed structures inside the body [1]. So by fusing both images using image fusion techniques information from both of the scans can be easily analysed hence this application provides better diagnosis. Surveillance Systems: Surveillance systems are gaining lot of attention these days where image fusion is crucial factor. Various places like malls, open streets, banks widely use large surveillance camera technology systems for close monitoring as a crime preventive measure and traffic watcher  .  Multi Sensor Data Fusion Systems: Images from various sensors (cameras) are fused for better information which is not possible by analysing single sensor images. Variety of applications like law Design and Development of an Embedded Real Time Vision Enhancement System using Image Fusion   N. Sri Swathi 1 , Chandan N. 2 , Viswanath K. 3 , Naveen K.S. 4   enforcement, security, medical diagnosis, monitoring environment, mineral resources identification uses this multi sensor fusion technique [2]. For example in night time CCD cameras can capture only visual information and IR cameras gives rich information only of high temperature objects whereas fusion of two videos will provide better perceptibility to human at any lighting conditions. Hence this multi sensor fusion detects moving objects during night time at any weather condition II.   P ROBLEM STATEMENT Vision Enhancement using Image Fusion has gained its importance in various domains and in surveillance applications particularly. The goal of image fusion is to integrate complementary multi-sensor and/or multi-view information into one new image containing information, the quality of which cannot be achieved otherwise and to provide  better visual understanding of certain phenomenon. The main aim of this paper is to design and develop a Real time Embedded based vision enhancement system using image fusion technique for high end surveillance/traffic applications III.   D ESIGN AND DEVELOPMENT V ISION E  NHANCEMENT OF S YSTEM USING R  EAL T IME F USION  The developed system should meet the following specifications set: The Vision Enhancement system should provide greater than 25 Frames per Second The Execution Time of un-optimized implemented real time image fusion algorithm on the DSP  processor should be less than 1 second which is a satisfactory response time to a user The average execution time of optimized real time image fusion algorithm implemented on DSP processor should be less than 1 second.  A. Top Level Design of Vision Enhancement System Fig 1.Top Level Design of Vision Enhancement System  In order to fulfill system objectives the Top level design that is to be developed is planned as seen in Figure 1 • CCD Camera: It  shows the input for the Real time vision enhancement system developed as Hi Focus Color CCD camera which is moving dynamic). This CCD camera has resolution of 720 X 480 pixels. The acquisition algorithm implemented on video processor grabs the frames from this camera. • Real Time Vision Enhancement system: The real  time image fusion algorithm which is developed and validated using MATLAB is converted in to Simulink model so that it can be dumped on to the DSP Video processor via simulink coder and code composer studio C code generation. Thus DM642EVM DSP runs this vision enhancement standalone application • DSP Video Processor: The video processing board used  in this application is DaVinci DM642EVM Version 3. It’s a higher end processor that consumes low power. • LCD Display: To display the final enhanced fused  output video Color TFT LCD Monitor is used. It is interfaced to video port 2 of DSP video processor successfully  B. Low Level Design of Vision Enhancement System   Fig 2.Low Level Design of Vision Enhancement System This section of gives the Low Level design details of the developed system. The real time vision enhancement system that is shown in Figure 1 as Top level diagram is further divided in to sub blocks as Image Acquisition, image pre- processing, image registration and image fusion for getting  better results. All these sub blocks are implemented at algorithm level on the DSP video processor. Then the final enhanced fused video is displayed on LCD monitor. In the later sections each sub block is explained further. Figure 2 shows the complete low level design details that are implemented in the next sections. Fig 3. Image Acquisition and Pre-Processing Subsystem CCD camera is interfaced with the DSP video processor. This image acquisition subsystem initially has to set the CCD camera properties like brightness, contrast, saturation and resolution etc for giving required frames for the next sub  blocks. Then the video frames are also grabbed by image acquisition subsystem. This pre-processing includes grayscale conversion of frames (so that working on intensity based frames increases speed of performance), image data type conversions (required as next sub blocks can work for only single or double data type frames) and resizing (as   implementing for smaller resolution frames will be easy). This is seen in Figure 3. The input frames obtained from preprocessing subsystem are given to this image registration subsystem for further implementation. Initially the frames captured from dynamic camera are given to corner detection block which would detect maximum required number of corner points from both the frames. These corners points obtained from frames are matched further in corner matching block for getting putative matching points to get the distance between the frames. Then the distortion between the frames is estimated in Transformation estimation block which would estimate rotation, translation and scale distortion between frames in the form of a transform matrix [5][7]. This transform matrix is given to apply geometric transformation and re-sampling  block which would perform geometric transformation by applying either affine, non-reflective similarity or projective transformations so that both the frames are aligned to same coordinate system [7]. This is seen in Figure 4 Fig 4. Image Registration Subsystem Thus the registered frames obtained from image registration subsystem are feed to image fusion subsystem. In this image fusion subsystem initially one level two dimension Discrete wavelet transform of the frames is performed using subband method by doing row-wise and column wise operation to obtain low frequency and high frequency components. Apply  pixel based algorithm for approximations which involves fusion based on taking the maximum valued pixels from approximations of wavelet decomposed subbands of frames. Then fusion decision is made for getting new coefficient matrix. The final fused output can be obtained by reconstructing the new coefficient matrix using inverse wavelet transform [4]. Figure 5 gives the image fusion subsystem design. Fig 5. Image Fusion Subsystem  In this image fusion subsystem initially one level two dimension Discrete wavelet transform of the frames is  performed using sub band method by doing row-wise and column wise operation to obtain low frequency and high frequency components. Apply pixel based algorithm for approximations which involves fusion based on taking the maximum valued pixels from approximations of wavelet decomposed sub bands of frames [3]. Then fusion decision is made for getting new coefficient matrix. The final fused output can be obtained by reconstructing the new coefficient matrix using inverse wavelet transform [11]. C. Software Implementation developed of Vision  Enhancement System This section provides the implementation of real time image fusion in MATLAB for validating the algorithm. The implementation starts with reading static images/frames, next Pre-processing of these frames is done, then feature based image registration is used for aligning frames to same coordinate system which are then fused to get enhanced output. Video frames are acquired from the recorded video using the data acquisition toolbox available in MATLAB. The acquired video frames are converted into grayscale images and resize to a suitable smaller dimension to speed up the  processing time. This part of section gives complete detailed flow of image Registration Fig 6. Image Registration Flowchart The preprocessed frames obtained from previous sub system are input to this sub block. As the output from dynamic camera will have frames which are not aligned to each other. Image registration of frames must be performed before fusing them. Figure 6 gives detailed software flow of image registration algorithm implemented. Initially two frames are read and one first frame is taken as reference frame whereas the next frame is considered as unregistered frame. Then corners detection of both frames is performed. If required   number of corners are detected their corner feature descriptors are extracted. Next corner points from both the frames are matched and locations of corresponding points are retrieved. Transform estimation is done to find the distortion between the reference and unregistered frames with parameters like rotation, translation and scale and a transform matrix is generated with those estimated parameters. Then finally  basing on that transform estimation matrix geometric transformation is done to align both frames to single coordinate system [11]. This part of section gives complete software flow of developed image fusion algorithm. Figure 7 gives the flow of the implemented image fusion algorithm. Initially the registered and reference frames from image registration subsystem are taken as input to this fusion subsystem. Fig 7. Image Fusion Flowchart Then one level two dimension discrete wavelet transform is applied to both reference and registered frames obtained from image registration subsystem. The wavelet transform is obtained by decomposition using row wise and column wise operation. That is independent wavelet decomposition of the two frames is performed until level one to get approximation (LL) and detail (LH, HL, HH) coefficients. Pixel based algorithm for approximations which involves fusion based on taking the maximum valued pixels from approximations of source frames is applied [12]. Then fusion decision is made to select with maximum pixel values thus gets a new concatenated fusion matrix. The final fused frames can be reconstructed using inverse wavelet transform.  D. Hardware Implementation Flow This section of design and development chapter givescomplete details of hardware implementation of the vision enhancement using real time image fusion application. Selection of suitable DSP Processor for Real Time Image Fusion Video Application. This section provides the  justification for selecting suitable digital signal processor required for implementing the validated real time image fusion algorithm. Initially let us brief the processor requirements for this application as follows: • Video Encoder for capturing video source   • Video Decoder for disp laying video output • Code Composer Studio Platinum software feature which includes 'C' Compiler, Assembler, Linker, Debugger, and DSP BIOS from Texas Instruments for algorithm development • JTAG Emulator hardware for dumping.   • Video processor for imaging application development Thus the Digital Media Processor TMS320DM642 (Version 3), which belongs to the DaVinci family of Texas Instruments C6000 series is selected. The approach followed for implementing Real time image fusion is with simulink, but however it is not possible to load simulink model directly to DM642EVM video processor. A simulink model which is set with target (DM642EVM V3) is developed. Thus the designed Simulink model facilitates to generate C code automatically corresponding to the desired  problem with the help of embedded target and Embedded Coder facility provided in MATLAB/Simulink [10]. However the DSP development software, Code Composer Studio can accept either C or assembly code to generate output (.out) file, which can be load on DSP chip. So Code composer studio version 3.3 used to generate .out file. This file now can be load into our DM642EVM processor. Figure 8 represents the interfacing between MATLAB, CCS and DM642EVM DSP . Fig 8. Interfacing flow between MATLAB (2012), CCS (v3.3) and DSP It   initially starts with the development of DM642 EVM board specific Simulink model along with its target preference block from Simulink with embedded coder (simulink coder).Then it is run on the simulator. If any errors found the configuration  parameters are checked accordingly. Later the incremental  building of the Simulink model is done. Finally the code is generated and downloaded as a project including all board specific header files and source files. This project in CCStudio is build and a.out file generated is loaded in to DM642 DSP chip and the project is run for results.  E. Simulink Model For Real Time Image Fusion Figure 9 gives in depth developed video registration and fusion algorithm Simulink model also the important blocks is explained in below section as follows: Video Capture Block: It is DM642 EVM Video ADC used as Video decoders to capture analog video. The block captures and buffers one frame (two fields for NTSC standard) of analog input video from the input ports, converts the buffered video to the   specified format, and then outputs the converted video frame as 8-bit unsigned integer data for further processing. Input to the DM642 EVM must be analog National Television Standards Committee (NTSC)or Phase Alternating Line (PAL) video format. For this video application Decoder type is selected to SAA7115,Input port to 0, Mode to NTSC, Output size to 720X480 and output format to Y (intensity  based). Video Display Block: It is DM642 EVM Video DAC used for Video encoder to display video. For this video application block parameters Mode is set to NTSC 720X480 Y. Corner Detection Block: It Calculate corner metric matrix and find corners in frames. The block parameters are set as Method to Local Intensity comparison, Intensity comparison to 0.001, Maximum angle to be considered a corner (in degrees) :157.5, Output to corner location, Maximum number of corners: 150, Minimum metric value that indicates a corner: 0.01, Neighborhood size (suppress region around detected corners):[21 21] and click Apply Fig 9. Over all Simulink Model for Real Time Fusion Corner Matching Block: It Find out matching corners in the current and previous video frames The Block parameters are set as Block size used in corner matching to 9, Maximum number of points to 150, Maximum feature difference for matching points to 81. Estimate Geometric   Transformation Block: It Find the transformation matrix that maps the largest number of points from Pts1 to Pts2.The block parameters are set as Transformation type to non reflective similarity, Enable Find and exclude outliners, Method to Random sample consensus (RANSAC), Distance threshold for determining inliers (in pixels): 2.5, Determine number of random samplings using: Desired confidence, Desired confidence (in %): 99.9, Maximum number of random samplings: 1000 and When Pts1 and Pts2 are built in integers, set transformation matrix data type to: single. Apply Geometric Transformation Block: It applies  projective or affine transformation to an image. The block  parameters are set as Transformation matrix source to Input  port, Interpolation method for calculating pixel value(s): Bilinear, Background fill value: 0, Output image size and  position: same as input image, Process pixels in: whole input image. To Frame Block  : It Specify sampling mode of output signal. For this application the image fusion sub model accepts only frame based inputs so this block is set to frame based. Zero-Order Hold Block: This block is used to convert continuous to discrete signal as DWT block allows only discrete data. Sample time set to -1 or inf. DWT Block:  This is inbuilt Simulink block used for calculating DWT of input frame or decomposing in to sub bands. The parameters are set as wavelet order to 2, and number of levels to 1 and output to single port. Embedded MATLAB Function: This Block is used for implementing image fusion rule of max pixels selection. This is easily implemented and code generation is done successfully. IDWT Block: This is inbuilt Simulink  block used for calculating inverse wavelet transform to reconstruct fused output. IV.   RESULTS  AND D ISCUSSIONS The developed real time image fusion algorithm shown in design and development section is tested with designed test cases that are shown in testing chapter for various sets of inputs as static input images, recorded video file frames and live video frames.  A.    MATLAB Results of Static Images: The developed MATLAB code is tested with various designed test cases on static images and the various stages of implementation gives step by step result which are shown as in Figure 10 that gives the results of reference image, unregistered image, registered image and finally fused image. Fig 10. (a) Ref img, (b) Unreg img, (c)Reg Img, (d) Fused img  B.    MATLAB Results of Recorded video frames: The developed MATLAB code is tested with various designed test cases on frames from recorded video file and the various stages of implementation gives step by step result which are shown as in Figure 11 that gives the results of reference frame, unregistered frame, color composite of reference and unregistered frames, registered frame and finally fused frame. Fig 11. (a) Ref frame, (b) Unreg frame, (c)color composite of Ref and unreg frames, (d) Reg frame, (e) Fused frame
Advertisement
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks