YUV420格式的CVPixelBuffer转换为RGB格式

ARKit中提取到的CVPixelBuffer为YUV420格式，很多时候我们需要把它转换为RGB格式，然后再进行各种后续操作。这里我们利用Accelerate中提供一个函数来完成这种转换：

1、Declaration

func vImageConvert_420Yp8_CbCr8ToARGB8888(_ srcYp: UnsafePointer<vImage_Buffer>, _ srcCbCr: UnsafePointer<vImage_Buffer>, _ dest: UnsafePointer<vImage_Buffer>, _ info: UnsafePointer<vImage_YpCbCrToARGB>, _ permuteMap: UnsafePointer<UInt8>!, _ alpha: UInt8, _ flags: vImage_Flags) -> vImage_Error

2、Parameters

srcYp
A pointer to the vImage buffer that references the source Yp plane.
​
srcCbCr
A pointer to the vImage buffer that references the source CbCr plane.
​
dest
A pointer to the vImage buffer that references 8-bit ARGB interleaved destination pixels.
​
info
A pointer to vImage_YpCbCrToARGB, which contains info coefficient and prebias values.
​
permuteMap
An array of four 8-bit integers with the values 0, 1, 2, and 3, in some order. Each value specifies a channel from the source image that should be copied to that channel in the destination image. 0 denotes the alpha channel, 1 the red channel, 2 the green channel, and 3 the blue channel. 
​
alpha
A value for the alpha channel in dest.
​
flags
The options to use when performing the operation. If you plan to perform your own tiling or use multithreading, pass kvImageDoNotTile.

3、Return Value

kvImageNoError; otherwise, one of the error codes described in Data Types and Constants.

4、完整代码：

func pixelBufferToImage(pixelBuffer:CVPixelBuffer) -> UIImage {
    //01
    CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
    //02
    defer {
       CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
    }
        
    //03
    let lumaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
    //04
    let lumaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
    let lumaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)
    let lumaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0)
    var sourceLumaBuffer = vImage_Buffer(data: lumaBaseAddress, height: vImagePixelCount(lumaHeight), width: vImagePixelCount(lumaWidth), rowBytes: lumaRowBytes)
        
    let chromaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)
    let chromaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)
    let chromaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)
    let chromaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1)
    var sourceChromaBuffer = vImage_Buffer(data: chromaBaseAddress, height: vImagePixelCount(chromaHeight), width: vImagePixelCount(chromaWidth), rowBytes: chromaRowBytes)
        
    //05
    var rawRGBBuffer: UnsafeMutableRawPointer = malloc(lumaWidth * lumaHeight * 4)
    var rgbBuffer: vImage_Buffer = vImage_Buffer(data: rawRGBBuffer, height: vImagePixelCount(lumaHeight), width: vImagePixelCount(lumaWidth), rowBytes: lumaWidth * 4)
        
    //06
    guard var conversionInfoYpCbCrToARGB = _conversionInfoYpCbCrToARGB else {
        return UIImage()
    }
        
    //07
    guard vImageConvert_420Yp8_CbCr8ToARGB8888(&sourceLumaBuffer, &sourceChromaBuffer, &rgbBuffer, &conversionInfoYpCbCrToARGB, nil, 255, vImage_Flags(kvImageNoFlags)) == kvImageNoError else {
        return UIImage()
    }
      
    //08
    let colorSpace = CGColorSpaceCreateDeviceRGB()
    let ctx = CGContext(data: rgbBuffer.data, width: lumaWidth, height: lumaHeight, bitsPerComponent: 8, bytesPerRow: rgbBuffer.rowBytes, space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
​
    let imageRef = ctx!.makeImage()!
    let uiimage = UIImage(cgImage: imageRef)
       
    //09 
    rawRGBBuffer.deallocate()
    return uiimage
}
​
private var _conversionInfoYpCbCrToARGB: vImage_YpCbCrToARGB? = {
    var pixelRange = vImage_YpCbCrPixelRange(Yp_bias: 16, CbCr_bias: 128, YpRangeMax: 235, CbCrRangeMax: 240, YpMax: 235, YpMin: 16, CbCrMax: 240, CbCrMin: 16)
    var infoYpCbCrToARGB = vImage_YpCbCrToARGB()
    guard vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4!, &pixelRange, &infoYpCbCrToARGB, kvImage422CbYpCrYp8, kvImageARGB8888, vImage_Flags(kvImageNoFlags)) == kvImageNoError else {
        return nil
    }
    return infoYpCbCrToARGB
}()

【学习地址】：FFmpeg/WebRTC/RTMP/NDK/Android音视频流媒体高级开发

【文章福利】：免费领取更多音视频学习资料包、大厂面试题、技术视频和学习路线图，资料包括（C/C++，Linux，FFmpeg webRTC rtmp hls rtsp ffplay srs 等等）有需要的可以点击1079654574加群领取哦~

4.1 代码解析

01、在访问buffer内部裸数据的地址時（读或写都一样），需要先将其锁上，用完了再放开。

CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) 
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)

02、defer block 里的代码会在函数 return 之前执行

defer {
       CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
}

03、ARKit中通过ARFrame获取的CVPixelBuffer为YUV420格式，YUV420格式Y通道（Luminance）与UV通道（Chrominance）分开储存数据，可通过下面语句获取地址

let lumaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
let chromaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)

04、创建vImage_Buffer，参数中包含三个vImage buffer，前两个参数需要基于Y通道与UV通道地址初始化： srcYp：

let lumaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
let lumaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)
let lumaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0)
var sourceLumaBuffer = vImage_Buffer(data: lumaBaseAddress, height: vImagePixelCount(lumaHeight), width: vImagePixelCount(lumaWidth), rowBytes: lumaRowBytes)

srcCbCr：

let chromaWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)
let chromaHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)
let chromaRowBytes = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1)
var sourceChromaBuffer = vImage_Buffer(data: chromaBaseAddress, height: vImagePixelCount(chromaHeight), width: vImagePixelCount(chromaWidth), rowBytes: chromaRowBytes)

05、参数中第三个vImage buffer需要初始化一个新的buffer，dest：

var rawRGBBuffer: UnsafeMutableRawPointer = malloc(lumaWidth * lumaHeight * 4)
var rgbBuffer: vImage_Buffer = vImage_Buffer(data: rawRGBBuffer, height: vImagePixelCount(lumaHeight), width: vImagePixelCount(lumaWidth), rowBytes: lumaWidth * 4)

06、第四个参数info，

guard var conversionInfoYpCbCrToARGB = _conversionInfoYpCbCrToARGB else {
      return UIImage()
}
​
private var _conversionInfoYpCbCrToARGB: vImage_YpCbCrToARGB? = {
    var pixelRange = vImage_YpCbCrPixelRange(Yp_bias: 16, CbCr_bias: 128, YpRangeMax: 235, CbCrRangeMax: 240, YpMax: 235, YpMin: 16, CbCrMax: 240, CbCrMin: 16)
    var infoYpCbCrToARGB = vImage_YpCbCrToARGB()
    guard vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4!, &pixelRange, &infoYpCbCrToARGB, kvImage422CbYpCrYp8, kvImageARGB8888, vImage_Flags(kvImageNoFlags)) == kvImageNoError else {
        return nil
    }
    return infoYpCbCrToARGB
}()

07、利用函数进行转换

guard vImageConvert_420Yp8_CbCr8ToARGB8888(&sourceLumaBuffer, &sourceChromaBuffer, &rgbBuffer, &conversionInfoYpCbCrToARGB, nil, 255, vImage_Flags(kvImageNoFlags)) == kvImageNoError else {
      return UIImage()
}

至此已完成RGB格式的转换

08、这里我们把RGB格式的buffer转换为UIImage，注意bitmapInfo的值

let colorSpace = CGColorSpaceCreateDeviceRGB()
let ctx = CGContext(data: rgbBuffer.data, width: lumaWidth, height: lumaHeight, bitsPerComponent: 8, bytesPerRow: rgbBuffer.rowBytes, space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
​
let imageRef = ctx!.makeImage()!
let uiimage = UIImage(cgImage: imageRef)

09、最后别忘了释放rawRGBBuffer。

rawRGBBuffer.deallocate()

4.2 耗时测试

ARKit可以在ARWorldTrackingConfiguration中设定摄像头采集分辨率

worldTrackingSessionConfiguration.videoFormat = ARWorldTrackingConfiguration.supportedVideoFormats[0]

对于iphoneX有3种分辨率可供选择

ARWorldTrackingConfiguration.supportedVideoFormats[0]   1920*1440
ARWorldTrackingConfiguration.supportedVideoFormats[1]   1920*1048
ARWorldTrackingConfiguration.supportedVideoFormats[2]   1024*768

耗时统计

绝大多数时候我们只需要完成01-07步即可，不需要执行08步。

5、其他

当我们使用Vision框架中的Machine-Learning Image Analysis时，CVPixelBuffer格式会自动转换为RGB格式。

如果我们需要的是灰度图数据，直接使用Y通道数据即可。

原文链接：YUV420格式的CVPixelBuffer转换为RGB格式 - 简书