基于live555的rtsp播放器：数据接收（拉流）

live555的使用都是从研究源码中的testRTSPClient例子开始的，这个例子包含了RTSP消息交互和数据接收。

一.RTSP消息交互

一次基本的RTSP操作过程如下:C表示RTSP客户端,S表示RTSP服务端
1.第一步：查询服务器端可用方法
C->S:OPTIONrequest          //询问S有哪些方法可用
S->C:OPTIONresponse       //S回应信息的public头字段中包括提供的所有可用方法

2.第二步：得到媒体描述信息
C->S:DESCRIBE request    //要求得到S提供的媒体描述信息
S->C:DESCRIBE response  //回应媒体描述信息，一般是sdp信息

3.第三步：建立RTSP会话
C->S:SETUPrequest            //通过Transport头字段列出可接受的传输选项，请求S建立会话
S->C:SETUPresponse          //建立会话，通过Transport头字段返回选择的具体转输选项，并返回建立的Session ID;

4.第四步：请求开始传送数据
C->S:PLAY request               //C请求S开始发送数据
S->C:PLAYresponse              //S回应该请求的信息

5.第五步：数据传送播放中
S->C:发送流媒体数据             //通过RTP协议传送数据

6.  第六步：关闭会话，退出
C->S:TEARDOWN request    //C请求关闭会话
S->C:TEARDOWN response //S回应该请求
上述的过程只是标准的、友好的rtsp流程，但实际的需求中并不一定按此过程。
其中第三和第四步是必需的！第一步，只要服务器客户端约定好，有哪些方法可用，则option请求可以不要。第二步，如果我们有其他途径得到媒体初始化描述信息（比如http请求等等），则我们也不需要通过rtsp中的describe请求来完成。

关于RTSP消息格式和SDP协议格式详见：https://blog.csdn.net/caoshangpa/article/details/53191630

testRTSPClient例子的不足之处在于所有变量、函数及其回调都写在了一个文件中，不方便管理。而且收到数据后，只是简单的打印了数据流的类型、字节数和时间戳，并没有进行下一步的处理。
关于testRTSPClient中变量、函数及其回调的拆分，github上有个demo可以参考：https://github.com/drriguz/Kamera
这个demo中实现了h264视频流的解码和显示，例子中最可取的做法是将数据接收放到了一个线程中，也就是将env->taskScheduler().doEventLoop(&this->eventLoopWatchVariable);这一句直接放到了线程中。线程的结束通过eventLoopWatchVariable变量来控制。这样的话一个线程就是一个session，拉取多路流只需要创建多个session即可。

二.数据接收

继承父类MediaSink即可实现数据的接收。在子类的构造函数中，可以先根据MediaSubsession参数获取一些有用的信息。

1.视频信息（h264和h265（HEVC））

    if(strcmp(m_subsession.mediumName(), "video") == 0)
    {
        if(strcmp(m_subsession.codecName(), "H264") == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_H264;

            unsigned int extraSize = 0;
            uint8_t *extra = h264_parse_config(m_subsession.fmtp_spropparametersets(),extraSize);
            if(extra&&extraSize>8)
            {
                format->extraSize = extraSize;
                format->extra = new uint8_t[extraSize];
                memcpy(format->extra, extra, extraSize);
                delete [] extra;
            }

            delete format;
        }
        else if(strcmp(m_subsession.codecName(), "H265") == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_H265;

            unsigned int extraSizeVPS = 0, extraSizeSPS = 0, extraSizePPS = 0, extraSizeTotal = 0;
            uint8_t *extraVPS = h264_parse_config(m_subsession.fmtp_spropvps(),extraSizeVPS);
            uint8_t *extraSPS = h264_parse_config(m_subsession.fmtp_spropsps(),extraSizeSPS);
            uint8_t *extraPPS = h264_parse_config(m_subsession.fmtp_sproppps(),extraSizePPS);
            extraSizeTotal = extraSizeVPS + extraSizeSPS + extraSizePPS;
            if(extraVPS&&extraSPS&&extraPPS&&extraSizeTotal>12)
            {
                format->extraSize = extraSizeTotal;
                format->extra = new uint8_t[extraSizeTotal];
                memcpy(format->extra, extraVPS, extraSizeVPS);
                memcpy(format->extra+extraSizeVPS, extraSPS, extraSizeSPS);
                memcpy(format->extra+extraSizeVPS+extraSizeSPS, extraPPS, extraSizePPS);
                delete [] extraVPS;
                delete [] extraSPS;
                delete [] extraPPS;
            }

            delete format;
        }
    }

对于h264格式，fmtp_spropparametersets包含了从SDP中获取的SPS和PPS信息的base64编码，SPS和PPS中间用逗号隔开。解析后的数据extra在初始化视频解码器的时候需要用到。
一个典型的h264视频流的SDP信息如下图：

红圈处sprop-parameter-sets等号后面就是fmtp_spropparametersets函数获取到的值。
需要特别注意一点的是SPS和PPS信息并不是在SDP中强制提供的，即sprop-parameter-sets等号后面可以是空的。因此为了保险起见，在视频流中总首次接收到SPS和PPS信息时，需要再解码之前，再次传给解码器。
对于h265格式，VPS和SPS和PPS是分三次获取的，解析后要组装到一起，同理，组装后的数据extra在初始化视频解码器的时候需要用到。h265也要注意上面提到的问题。

2.音频信息（MPEG4-GENERIC（AAC）、PCMA（G711a）、PCMU（G711u）和G726）

之所以要处理这么多音频编码类型，是因为这些类型几乎所有的安防摄像机都支持。

    if(strcmp(m_subsession.mediumName(), "audio") == 0)
    {
        if(strcmp(m_subsession.codecName(), "MPEG4-GENERIC") == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_AAC;

            unsigned int extraSize = 0;
            uint8_t *extra = parseGeneralConfigStr(m_subsession.fmtp_config(),extraSize);
            if(extra)
            {
                format->extraSize = extraSize;
                format->extra = new uint8_t[extraSize];
                memcpy(format->extra, extra, extraSize);
                delete [] extra;
            }

            delete format;
        }
        else if(strcmp(m_subsession.codecName(), "PCMA") == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_PCM_ALAW;
            format->samplerate = m_subsession.rtpTimestampFrequency();
            format->channels = m_subsession.numChannels();
            format->bitspersample = 8;

            delete format;
        }
        else if(strcmp(m_subsession.codecName(), "PCMU") == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_PCM_MULAW;
            format->samplerate = m_subsession.rtpTimestampFrequency();
            format->channels = m_subsession.numChannels();
            format->bitspersample = 8;

            delete format;
        }
        else if(strncmp(m_subsession.codecName(), "G726", 4) == 0)
        {
            Format *format = new Format();

            format->codecID = AV_CODEC_ID_ADPCM_G726;
            format->samplerate = 8000;
            format->channels = 1;
            if(strcmp(m_subsession.codecName()+5, "40") == 0)
            {
                format->bitrate = 40000;
            }
            else if(strcmp(m_subsession.codecName()+5, "32") == 0)
            {
                format->bitrate = 32000;
            }
            else if(strcmp(m_subsession.codecName()+5, "24") == 0)
            {
                format->bitrate = 24000;
            }
            else if(strcmp(m_subsession.codecName()+5, "16") == 0)
            {
                format->bitrate = 16000;
            }

            delete format;
        }
    }

fmtp_config包含了采样率和通道等信息，由下图红圈处的rtpmap参数获取。这个信息SDP是必须提供的，因为在接收数据时无法再获取到这些信息了，它们将用于初始化音频解码器。

3.流数据处理
通过循环调用FrameSouce类的getNextFrame(m_receiveBuffer, RECEIVE_BUFFER_SIZE,afterGettingFrame, this, onSourceClosure, this);函数来获取数据，参数m_receiveBuffer是接收buffer，afterGettingFrame是回调函数。
在回调函数中afterGettingFrame(unsigned frameSize, unsigned numTruncatedBytes, struct timeval presentationTime, unsigned /*durationInMicroseconds*/)可以对接收到的音视频流数据进行处理。
此时获取到的是完整的帧数据，参数frameSize是数据大小，参数presentationTime的类型timeval是live555内置的时间结构，转换成pts的方法如下：

int64_t pts = (int64_t)presentationTime.tv_sec * INT64_C(1000000) + (int64_t)presentationTime.tv_usec;

这个时间是绝对时间，单位是微秒。
在处理数据之前，先看看live555官方的FAQ：http://www.live555.com/liveMedia/faq.html#testRTSPClient-how-to-decode-data
Question:I have successfully used the "testRTSPClient" demo application to receive a RTSP/RTP stream. Using this application code as a model, how can I decode the received video (and/or audio) data?
The "testRTSPClient" demo application receives each (video and/or audio) frame into a memory buffer, but does not do anything with the frame data. You can, however, use this code as a model for a 'media player' application that decodes and renders these frames. Note, in particular, the "DummySink" class that the "testRTSPClient" demo application uses - and the (non-static) "DummySink::afterGettingFrame()" function. When this function is called, a complete 'frame' (for H.264 or H.265, this will be a "NAL unit") will have already been delivered into "fReceiveBuffer". Note that our "DummySink" implementation doesn't actually do anything with this data; that's why it's called a 'dummy' sink.
If you want to decode (or otherwise process) these frames, you would replace "DummySink" with your own "MediaSink" subclass. Its "afterGettingFrame()" function would pass the data (at "fReceiveBuffer", of length "frameSize") to a decoder. (A decoder would also use the "presentationTime" timestamp to properly time the rendering of each frame, and to synchronize audio and video.)
If you are receiving H.264 video data, there is one more thing that you have to do before you start feeding frames to your decoder. H.264 streams have out-of-band configuration information ("SPS" and "PPS" NAL units) that you may need to feed to the decoder to initialize it. To get this information, call "MediaSubsession::fmtp_spropparametersets()" (on the video 'subsession' object). This will give you a (ASCII) character string. You can then pass this to "parseSPropParameterSets()" (defined in the file "include/H264VideoRTPSource.hh"), to generate binary NAL units for your decoder.
(If you are receiving H.265 video, then you do the same thing, except that you have three separate configuration strings, that you get by calling "MediaSubsession::fmtp_spropvps()", "MediaSubsession::fmtp_spropsps()", and "MediaSubsession::fmtp_sproppps()". For each of these three strings, in turn, pass them to "parseSPropParameterSets()", then feed the resulting binary NAL unit to your decoder.)
大致意思对h264或h265, 接收到的buffer是一个 "NAL unit"，需要注意的是这个NAL unit不带四字节起始码0x00000001，但是ffmpeg解码的时候需要在接收buffer前添加起始码，否则ffmpeg就解码错误no frame。

Block *block=new Block();
block->esType=ES_VIDEO;
block->codecType=CODEC_H264;
block->frameType=frameType;
block->pts=pts;
block->dts=pts;

uint8_t *receiveBufferAV = new uint8_t[frameSize + 4];
receiveBufferAV[0] = 0;
receiveBufferAV[1] = 0;
receiveBufferAV[2] = 0;
receiveBufferAV[3] = 1;
memcpy(receiveBufferAV + 4, m_receiveBuffer, frameSize);
block->buffer=receiveBufferAV;
block->bufferSize=frameSize + 4;

每个NAL unit的第一个字节是NAL头，通过NAL头可以解析出NAL unit的类型，对于h264：uint8_t nalType=(m_receiveBuffer[0] & 0x1f)。nalType=5表示当前NAL unit是IDR图像，nalType=1表示当前NAL unit是非IDR图像，nalType=7或8表示当前NAL unit是SPS或者PPS。

关于h264的格式分析详见：https://blog.csdn.net/caoshangpa/article/details/53019793?utm_source=blogxgwz3
SPS信息和PPS信息可用于初始化视频解码器，如前文所述，如果从SDP中未获取到，从这里也可以获取。
SPS中包含了视频的宽高和帧率等信息，如何解码SPS获取分辨率和帧率可参考：https://blog.csdn.net/caoshangpa/article/details/53083410?utm_source=blogxgwz6
需要注意的是帧率并不是SPS中必须包含的，因为有些流的帧率是可变的，解出来帧率值可能为0。
这里IDR图像一定是I帧，但是I帧不一定是IDR图像，关于I帧、P帧和B帧将在下篇文章中总结一下。
为了防止解码第一帧时出现马赛克，接收时需要判断接收到帧是否是IDR图像（注意不是判断I帧），如果是IDR图像，则开始解码。我的做法是SPS和PPS都接收到且传给了解码器，再判断当前帧是否是IDR图像，如果是则开始解码当前帧和之后收到的IDR图像和非IDR图像，也就是说SPS和PPS只用处理一次。大概如下：
0x00, 0x00, 0x00, 0x01, pps, 0x00, 0x00, 0x00, 0x01, sps, 0x00, 0x00, 0x00, 0x01, IDR frame...........
对于h265：uint8_t nalType=((m_receiveBuffer[0] & 0x7e)>>1)，具体类型判断如下。

uint8_t nalType=((m_receiveBuffer[0] & 0x7e)>>1);

if(nalType == 32)//VPS
{   
       
}
else if(nalType == 33)//SPS
{  
           
}
else if(nalType == 34)//PPS
{
                
}

if(nalType==19 || nalType==20)//key frame
{
     m_hasKeyFrame=true;
}

if(((nalType>=1 && nalType<=9) || (nalType>=16 && nalType<=18) || nalType==21) && m_hasKeyFrame)//I/P/B
{
               
}

音频数据的处理就简单很多，不需要添加起始码和判断帧类型。

    if(strcmp(m_subsession.mediumName(), "audio") == 0)
    {
        if(strcmp(m_subsession.codecName(), "MPEG4-GENERIC") == 0
                || strcmp(m_subsession.codecName(), "PCMA") == 0
                || strcmp(m_subsession.codecName(), "PCMU") == 0
                || strncmp(m_subsession.codecName(), "G726", 4) == 0)
        {
            Block *block=new Block();
            if(strcmp(m_subsession.codecName(), "MPEG4-GENERIC") == 0)
            {
                block->codecType=CODEC_AAC;
            }
            else if(strcmp(m_subsession.codecName(), "PCMA") == 0)
            {
                block->codecType=CODEC_G711A;
            }
            else if(strcmp(m_subsession.codecName(), "PCMU") == 0)
            {
                block->codecType=CODEC_G711U;
            }
            else if(strncmp(m_subsession.codecName(), "G726" , 4) == 0)
            {
                block->codecType=CODEC_G726;
            }
            block->esType=ES_AUDIO;
            block->pts=pts;
            block->dts=pts;

            uint8_t *receiveBufferAV = new uint8_t[frameSize];
            memcpy(receiveBufferAV, m_receiveBuffer, frameSize);
            block->buffer=receiveBufferAV;
            block->bufferSize=frameSize;
        }
    }

原创不易，转载请标明出处：https://blog.csdn.net/caoshangpa/article/details/112063679