android手機遠程視頻移動檢測的實踐

家中老人高齡，爲防止意外跌倒，需要時刻看護，於是想到用視頻監控代替部分注意力。遠程視頻移動監測的方案有很多種，因爲以前在手機上做了類似工作，參見用安卓手機實現視頻監控，在此基礎之上增加移動監測報警功能。

服務端修改

移動監測功能在服務端（camera）實現，在以前架構基礎上，做出如下修改。

• 原以爲createCameraPreviewSession()（創建顯示預覽界面）時，用createCaptureSession()（創建畫面捕獲會話）可以使用任意數量surface作爲照相機圖像數據的輸出口，以前架構中用到了3個surface：視頻預覽、視頻編碼器、照片文件各用一個。這次增加第四個，用於視頻移動監測，但反覆調試仍無法創建畫面捕獲會話，SDK文檔中也沒有找到有關surface數量限制的記述。只好將第一個surface交給ImageReader，用ImageReader同時實現視頻預覽和移動監測功能。

// 創建ImageReader
openCvImageReader = ImageReader.newInstance(imageWidth, imageHeight, openCvFormat, 3)
// 創建imageAvailableListener
imageAvailableListener = ImageAvailableListener(openCvFormat, imageWidth, imageHeight, false, backgroundMat, previewThread)
// 在imageAvailableListener中注入movingAlarmListener（移動報警監聽器）
imageAvailableListener?.setMovingAlarmListener(movingAlarmListener)
// 打開ImageReader
openCvImageReader.setOnImageAvailableListener(imageAvailableListener, null)
// 定義ImageReader.surface
val openCvSurface = openCvImageReader.surface
// 創建作爲預覽的CaptureRequest.Builder
previewRequestBuilder = cameraDevice!!.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
// 將openCvImageReader的surface作爲CaptureRequest.Builder的目標
previewRequestBuilder.addTarget(openCvSurface)

• 創建畫面捕獲會話

// 創建CameraCaptureSession，該對象負責管理處理預覽請求和拍照請求，以及傳輸請求。最多隻能容納3個surface!
cameraDevice!!.createCaptureSession(listOf(openCvSurface, encoderInputSurface, imageReader.surface), object :
                    CameraCaptureSession.StateCallback() {
                    ...}

• 根據相機的分辨率，獲取最佳的預覽尺寸，具體算法請參考代碼中的註釋

    /**
     * 根據view的物理尺寸，參照相機支持的分辨率，使用最接近的長寬比，確定預覽畫面的尺寸
     * 循環測試相機支持的分辨率
     * 同時計算預覽時圖像放大比例
     * 測試條件：最佳寬度 = 0
     *           最佳高度 = 0
     *           如果view的寬度>= 相機分辨率寬度 且 view的高度 >= 相機分辨率高度  且
     *               最佳寬度  <= 相機分辨率寬度 且 最佳高度   <= 相機分辨率高度  且
     *               view的長寬比與相機分辨率長寬比之差 < 0.1
     *           則  最佳寬度 = 相機分辨率寬度
     *               最佳高度 = 相機分辨率高度
     * @author wxson
     * @param
     * viewWidth : view's viewWidth
     * viewHeight : view's viewHeight
     * @return false: fail   true: success
     */
    internal fun calcPreviewSize(viewWidth: Int, viewHeight: Int): Boolean {
        Log.i(TAG, "calcPreviewSize: " + viewWidth + "x" + viewHeight)
        val manager = app.getSystemService(Context.CAMERA_SERVICE) as CameraManager
        try {
            val characteristics = manager.getCameraCharacteristics(cameraId)
            val map = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)
            var bestWidth = 0
            var bestHeight = 0
            val aspect = viewWidth.toFloat() / viewHeight       // view的長寬比
            val sizes = map!!.getOutputSizes(ImageReader::class.java)   // 相機支持的分辨率
            for (i in sizes.size downTo 1) {
                val sz = sizes[i - 1]
                val w = sz.width
                val h = sz.height
                Log.i(TAG, "trying size: " + w + "x" + h)
                if (viewWidth >= w && viewHeight >= h && bestWidth <= w && bestHeight <= h
                    && Math.abs(aspect - w.toFloat() / h) < 0.1
                ) {
                    bestWidth = w
                    bestHeight = h
                }
            }
            Log.i(TAG, "best size: " + bestWidth + "x" + bestHeight)
            assert(!(bestWidth == 0 || bestHeight == 0))
            if (imageSize.width == bestWidth && imageSize.height == bestHeight)
                return false
            else {
                imageSize = Size(bestWidth, bestHeight)
                // 圖像放大率
                previewScale = Math.min(viewWidth.toFloat()/bestWidth, viewHeight.toFloat()/bestHeight)
                // 圖像顯示範圍
                viewRect = Rect(0, 0, (previewScale * bestHeight).toInt(), (previewScale * bestWidth).toInt())
                previewThread.canvasRect = viewRect
                return true
            }
        } catch (e: CameraAccessException) {
            Log.e(TAG, "calcPreviewSize - Camera Access Exception", e)
        } catch (e: IllegalArgumentException) {
            Log.e(TAG, "calcPreviewSize - Illegal Argument Exception", e)
        } catch (e: SecurityException) {
            Log.e(TAG, "calcPreviewSize - Security Exception", e)
        }
        return false
    }

• 在stringTransferListener中增加接受客戶端開關移動監測指令的功能，直接控制imageAvailableListener的外部開關變量motionDetectOn。

    private val stringTransferListener = object : IStringTransferListener {
        override fun onStringArrived(arrivedString: String, clientInetAddress: InetAddress) {
            Log.i(TAG, "onStringArrived")
            localMsgLiveData.postValue("arrivedString:$arrivedString clientInetAddress:$clientInetAddress")
            when (arrivedString){
            	...
                "Start Motion Detect" ->{
                    imageAvailableListener?.motionDetectOn = true
                }
                "Stop Motion Detect" ->{
                    imageAvailableListener?.motionDetectOn = false
                }
            }
        }

• 需要定義一個預覽用的線程PreviewThread

/**
 *  The thread used to display a content stream on
 *  @param textureView
 */
class PreviewThread(private val textureView: TextureView) : Runnable {
    lateinit var canvasRect: Rect
    // 定義接收preview image的Handler對象
    lateinit var revHandler: Handler
    class MyHandler(private val textureView: TextureView, private val dstRect: Rect) : Handler(){
        override fun handleMessage(msg: Message?) {
            val canvas = textureView.lockCanvas() ?: return
            if (msg != null){
                val bitmap: Bitmap? = msg.data.getParcelable("bitmap")
                if (bitmap != null) {
                    canvas.drawBitmap(bitmap, null, dstRect, null)
                }
            }
            textureView.unlockCanvasAndPost(canvas)
        }
    }

    override fun run() {
        Looper.prepare()
        revHandler = MyHandler(textureView, canvasRect)
        Looper.loop()
    }
}

• 在相機打開時啓動PreviewThread

    fun openCamera() {
        Log.i(TAG, "openCamera")
        previewThread = PreviewThread(textureView)
        setUpCameraOutputs(previewSurfaceWidth, previewSurfaceHigh)

        val manager = app.getSystemService(Context.CAMERA_SERVICE) as CameraManager
        try {
            // 打開攝像頭
            manager.openCamera(cameraId, stateCallback, null)
        } catch (e: CameraAccessException) {
            e.printStackTrace()
        } catch (e: SecurityException) {
            e.printStackTrace()
        } catch (e: NullPointerException) {
            e.printStackTrace()
        }
            // to start previewThread
            Thread(previewThread).start()
    }

• 在ImageReader中實現OnImageAvailableListener，這是實現移動監測的關鍵部分，主要動作有
  • 讀取圖像數據
  • 格式檢查
  • 圖像數據轉爲mat
  • 用圖像亮度mat作爲移動監測用的幀數據
  • 首幀處理
    • 用高斯濾波抑制噪聲
    • 提取輪廓
    • 存爲背景幀
  • 後續幀處理
    • 用高斯濾波抑制噪聲
    • 提取輪廓
    • 計算當前幀與背景幀的差值
    • 如果差值大於指定的閾值，通過movingAlarmListener向外部發出移動報警文字。
    • 存爲背景幀
  • 用圖像亮度mat和色度mat合成彩色mat
  • 彩色mat順時針旋轉90°
  • 彩色mat轉換爲 bitmap
  • 把bitmap發送給預覽線程previewThread，預覽線程用canvas.drawBitmap方法顯示圖像。

class ImageAvailableListener(private val openCvFormat: Int,
                             private val width: Int,
                             private val height: Int,
                             var motionDetectOn: Boolean,
                             private var backgroundMat: Mat?,
                             private val previewThread: PreviewThread) : ImageReader.OnImageAvailableListener {
    private val TAG = this.javaClass.simpleName
    private var isTimeOut = true
    private val timer = Timer(true)     // The timer for restraining contiguous alarms
    private lateinit var movingAlarmListener: IMovingAlarmListener

    override fun onImageAvailable(reader: ImageReader) {
        try{
            val image = reader.acquireLatestImage() ?: return
            // sanity checks - 3 planes
            val planes = image.planes
            assert(planes.size == 3)
            assert(image.format == openCvFormat)
            // https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888
            // Y plane (0) non-interleaved => stride == 1; U/V plane interleaved => stride == 2
            assert(planes[0].pixelStride == 1)
            assert(planes[1].pixelStride == 2)
            assert(planes[2].pixelStride == 2)

            val yPlane = planes[0].buffer
            val uvPlane = planes[1].buffer
            val yMat = Mat(height, width, CvType.CV_8UC1, yPlane)
            val uvMat = Mat(height / 2, width / 2, CvType.CV_8UC2, uvPlane)
            val cacheMat = yMat.clone()
            // start of motion detection
            if (motionDetectOn){
                if (backgroundMat == null){
                    // first image noise reduction
                    Imgproc.GaussianBlur(cacheMat, cacheMat, org.opencv.core.Size(13.0, 13.0), 0.0, 0.0)
                    backgroundMat = Mat()
                    Imgproc.Canny(cacheMat, backgroundMat, 80.0, 100.0)
                    return
                }
                else{
                    // skip interval frames
                    // next image noise reduction
                    Imgproc.GaussianBlur(cacheMat, cacheMat, org.opencv.core.Size(13.0, 13.0), 0.0, 0.0)
                    // get contours
                    val contoursMat = Mat()
                    Imgproc.Canny(cacheMat, contoursMat, 80.0, 100.0)
                    // get difference between two images
                    val diffMat = Mat()
                    Core.absdiff(backgroundMat, contoursMat, diffMat)
                    // Counts non-zero array elements.
                    val diffElements = Core.countNonZero(diffMat)
                    val matSize = diffMat.rows() * diffMat.cols()
                    val diff = diffElements.toFloat() / matSize
                    if (diff > 0.004) {
//                        Log.e(TAG, "object moving !! diff=$diff")
                        if (isTimeOut){
                            Log.i(TAG, "send MovingAlarm message out ")
                            movingAlarmListener.onMovingAlarm()
                            isTimeOut = false
                            timer.schedule(object : TimerTask(){
                                override fun run() {
                                    isTimeOut = true
                                }
                            }, 1000)
                        }
                    }
                    // save background image
                    backgroundMat = contoursMat.clone()
//                    // ***************** debug start *********************
//                    sendImageMsg(contoursMat)
//                    // ***************** debug end   *********************
                }
            }
            //**************************************************************************************
            // send image to previewThread
            val tempFrame = JavaCamera2Frame(yMat, uvMat, width, height, openCvFormat)
            val modified = tempFrame.rgba()
            sendImageMsg(modified)
            tempFrame.release()
            //**************************************************************************************
            image.close()
        }
        catch (e: Exception){
            Log.e(TAG, "onImageAvailable ", e)
        }
    }

    /**
     * This class interface is abstract representation of single frame from camera for onCameraFrame callback
     * Attention: Do not use objects, that represents this interface out of onCameraFrame callback!
     */
    interface CvCameraViewFrame {
        /**
         * This method returns RGBA Mat with frame
         */
        fun rgba(): Mat
        /**
         * This method returns single channel gray scale Mat with frame
         */
        fun gray(): Mat
    }

    private inner class JavaCamera2Frame() : CvCameraViewFrame {
        private var openCvFormat: Int = 0
        private var width: Int = 0
        private var height: Int = 0
        private var yuvFrameData: Mat? = null
        private var uvFrameData: Mat? = null
        private var rgba = Mat()

        constructor(Yuv420sp: Mat, w: Int, h: Int, format: Int): this() {
            yuvFrameData = Yuv420sp
            uvFrameData = null
            width = w
            height = h
            openCvFormat = format
        }

        constructor(Y: Mat, UV: Mat, w: Int, h: Int, format: Int) : this() {
            yuvFrameData = Y
            uvFrameData = UV
            width = w
            height = h
            openCvFormat = format
        }


        override fun gray(): Mat {
            return yuvFrameData!!.submat(0, height, 0, width)
        }

        override fun rgba(): Mat {
            if (openCvFormat == ImageFormat.NV21)
                Imgproc.cvtColor(yuvFrameData!!, rgba, Imgproc.COLOR_YUV2RGBA_NV21, 4)
            else if (openCvFormat == ImageFormat.YV12)
                Imgproc.cvtColor( yuvFrameData!!, rgba, Imgproc.COLOR_YUV2RGB_I420,4) // COLOR_YUV2RGBA_YV12 produces inverted colors
            else if (openCvFormat == ImageFormat.YUV_420_888) {
                assert(uvFrameData != null)
                //                Imgproc.cvtColorTwoPlane(yuvFrameData, uvFrameData, rgba, Imgproc.COLOR_YUV2RGBA_NV21);    //modified by wan
                Imgproc.cvtColorTwoPlane(yuvFrameData!!, uvFrameData!!, rgba, Imgproc.COLOR_YUV2BGRA_NV21)  //modified by wan
            } else
                throw IllegalArgumentException("Preview Format can be NV21 or YV12")

            return rgba
        }

        fun release() {
            rgba.release()
        }
    }

    private fun sendImageMsg(inputMat: Mat){
        val showMat = inputMat.clone()
        Core.rotate(showMat, showMat, Core.ROTATE_90_CLOCKWISE)
        // make bitmap for display
        val bitmap = Bitmap.createBitmap(height, width, Bitmap.Config.ARGB_8888)
        try{
            Utils.matToBitmap(showMat, bitmap)
        }
        catch (e: java.lang.Exception){
            Log.e(TAG, "Mat type: $showMat")
            Log.e(TAG, "modified.dims:" + showMat.dims() + " rows:" + showMat.rows() + " cols:" + showMat.cols())
            Log.e(TAG, "Bitmap type: " + bitmap.width + "*" + bitmap.height)
            Log.e(TAG, "Utils.matToBitmap() throws an exception: " + e.message)
            exitProcess(1)
        }
        val msg = Message()
        msg.data.putParcelable("bitmap", bitmap)
        previewThread.revHandler.sendMessage(msg)
    }

    private fun sendMovingMsg(){
    }

    fun setMovingAlarmListener(listener : IMovingAlarmListener){
        movingAlarmListener = listener
    }

}

客戶端修改

客戶端（monitor）修改比較簡單。

• 在界面上增加一個移動監測按鈕

    <com.google.android.material.floatingactionbutton.FloatingActionButton
        android:id="@+id/fab_motion_detect"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="bottom"
        android:layout_margin="16dp"
        android:layout_marginBottom="16dp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toStartOf="@+id/fab_transmit"
        app:layout_constraintStart_toEndOf="@+id/fab_capture"
        app:srcCompat="@drawable/ic_motion_detect" />

• 在程序中監聽這個按鈕，通過已經建立的socket連接，發送字符串指令到服務端。

private var isMotionDetectOn = false

        //定義移動探測浮動按鈕
        fab_motion_detect.setOnClickListener{
            view ->
            if (isMotionDetectOn){
                viewModel.sendMsgToServer("Stop Motion Detect")    // notify server to Stop Motion Detect
                Snackbar.make(view, "停止移動探測", Snackbar.LENGTH_SHORT).show()
                fab_motion_detect.backgroundTintList = ContextCompat.getColorStateList(this.activity!!.baseContext, R.color.button_light)
                isMotionDetectOn = false
            }
            else{
                viewModel.sendMsgToServer("Start Motion Detect")    // notify server to Start Motion Detect
                Snackbar.make(view, "開始移動探測", Snackbar.LENGTH_SHORT).show()
                fab_motion_detect.backgroundTintList = ContextCompat.getColorStateList(this.activity!!.baseContext, R.color.colorAccent)
                isMotionDetectOn = true
            }
        }

• 在客戶端增加對服務端信息的響應，如果服務端發出的是移動警告信息，則啓動系統鈴聲作爲警告鈴聲。

    override fun onActivityCreated(savedInstanceState: Bundle?) {
        super.onActivityCreated(savedInstanceState)
        Log.i(TAG, "onActivityCreated")
        ...
        val serverMsgObserver: Observer<String> = Observer { serverMsg -> remoteMsgHandler(serverMsg.toString()) }
        viewModel.getServerMsg().observe(this, serverMsgObserver)
        ...      
    }

    private fun remoteMsgHandler(remoteMsg: String){
        when (remoteMsg){
            "Moving Alarm" -> {
                showMsg(remoteMsg)
                viewModel.defaultMediaPlayer()
            }
        }
    }

    fun defaultMediaPlayer(){
        val ringtoneUri: Uri = RingtoneManager.getDefaultUri(RingtoneManager.TYPE_RINGTONE)
        val ringtone: Ringtone = RingtoneManager.getRingtone(app, ringtoneUri)
        if (ringtone.isPlaying)
            ringtone.stop()
        else
            ringtone.play()
    }

其它細節，請參考源代碼。
實踐中，發現由於不同手機鏡頭有差異，移動監測的靈敏度會有不同，需要調整移動監測處理的相關參數。
如果有問題，請聯繫我（[email protected]）。