投稿
  1. Paragoger衍生者 AIGC 训练营Home
  2. 那些年一起创业的日子
  3. 青龙云渲染(基于WebRTC的流媒体平台)

第二十三章 建立P2P 连接

chris • • 青龙云渲染(基于WebRTC的流媒体平台) • 60 views

什么是P2P连接

PeerConnection 是 WebRTC 连接流程中无法绕过的一个类，甚至可以认为这个类是整个连接流程的 Controller。当2个用户分别在2端试图连接对方的时候，首先由信令服务器服务，使得相互可以认识到对方的存在，之后，就需要建立一个稳定的长连接，此时并没有另外一个中央服务器处理这2个客户端的共有信息，我们认为是2个Endpoint到Endpoint的连接，也称为Peer2Peer的连接，简称P2P .

WebRTC中重要的类PeerConnection负责建立这个连接。 在这个连接的过程中，程序初始化了很多很多重要的内容，包括编码器，解码器，视频音频的内容，工作流线程等。所以可以说最重要的一个类都不为过。

时序图

假定读者已经阅读过「基础知识」中的内容，对信令（signaling）服务器、ICE 服务器等概念都有所了解，那么没有什么比一张时序图更加简洁清晰的了：

第二十三章 建立P2P 连接

这里针对时序图中的一些情况做具体说明：

  1. 上图不完全是 API 的调用流程，读者在编程时仍需参考 WebRTC 的文档或源码注释。
  2. 先进入房间的用户是发起方（Indicator），后进入房间的用户是参与者（Participant）。如果参与者进房时信令服务器已经有 offerSdp 甚至（发起方的）ICE candidate 信息了，则信令服务器可以将它们与 ICE server addr 一起返回给参与者。
  3. add audio & video tracks 不是连接流程中的关键步骤，也可以在 ICE 流程之后再执行。
  4. 在 SetLocalDescription 执行成功后，协商 SDP 和 ICE candidate 的流程便会同时开始。
  5. 通话双方均与选定的 ICE 服务器连接成功后，即可开始相互推流。
  6. 在 多人会议服务端架构 中，一般由 SFU 服务器同时充当 ICE 服务器的角色。

完整的建立P2P函数源代码

这里的函数源代码包含了很多我们产品的业务逻辑和配置，你需要自行吸收，并且可能由于WebRTC版本的差异，接口和参数会有不同。

        /// <summary>
        /// 创建标准管道。
        /// </summary>
        /// <returns></returns>
        bool QLSignalConnection::CreateP2PConnection()
        {
                if (mPeerConnectionFactory)
                {
                        QL_ERROR("Already created P2P Connection Factory,will return.");
                        return false;
                }

                //global threads
                //if networkThread is nullptr seems no difference 
                //If we create socket by ourself, it will cause except when we exit app,
                //The reason might be we have to clean thread info by ourselves. so just use default
                //networkThread = rtc::Thread::CreateWithSocketServer();
                //networkThread->Start();
                //networkThread = nullptr;

                //Those are ok
                globalWorkerThread = rtc::Thread::Create();
                globalWorkerThread->Start();
                globalSignalingThread = rtc::Thread::Create();
                globalSignalingThread->Start();

                //init video and audio factory
                std::unique_ptr<webrtc::VideoEncoderFactory> myVideoEncoderFactory = webrtc::CreateBuiltinVideoEncoderFactory();
                rtc::scoped_refptr<webrtc::AudioEncoderFactory> myAudioEncoderFactory = webrtc::CreateBuiltinAudioEncoderFactory();


                //try to add some constraint but failed
                //webrtc::CreatePeerConnectionFactory(rtc::Thread::Current(), rtc::Thread::Current(), rtc::Thread::Current(),nullptr,nu)
                webrtc::PeerConnectionFactoryDependencies deps;
                webrtc::PeerConnectionFactoryInterface::Options myOptions;
                //WE want to control not adjust birrate regarding CPU usage
                webrtc::PeerConnectionInterface::RTCConfiguration configuration;


                //Important ,Use NVENC
                if (QL::QLGlobalConfig::Get().IsUseNVENC())
                {
                        //myVideoEncoderFactory = UniEncoderFactory::CreateUniEncoderFactory();
                        myVideoEncoderFactory = QL::CreateBuiltinExternalVideoEncoderFactory();

                        QL_LOG("Create My NVENC encoder");
                }


                rtc::scoped_refptr<webrtc::AudioDeviceModule> myADM = nullptr;
                if (QL::QLGlobalConfig::Get().IsUseFakeADM())
                {
                        myADM = FakeADM::Create();
                        if (!myADM)
                        {
                                QL_ERROR("Failed to create dummy ADM");
                        }
                        else
                        {

                                QL_LOG("Create faked ADM successfully.");
                        }
                }
                else
                {
                        //myADM = webrtc::AudioDeviceModule::Create(webrtc::AudioDeviceModule::kPlatformDefaultAudio, webrtc::CreateDefaultTaskQueueFactory().get());
                }

                //detect supported encoders,should place before create p2p
                std::vector<webrtc::SdpVideoFormat> video_formats = myVideoEncoderFactory->GetSupportedFormats();
                for (const webrtc::SdpVideoFormat& myFormat : video_formats)
                {
                        QL_LOG(std::string("Supported video encoders:" + myFormat.name).c_str());
                }
                //detect current audio codecs,should place before create p2p
                std::vector<webrtc::AudioCodecSpec> audioCodecs = myAudioEncoderFactory->GetSupportedEncoders();
                for (const webrtc::AudioCodecSpec& myAudioCodes : audioCodecs)
                {
                        QL_LOG(std::string("Supported audio encoders:" + myAudioCodes.format.name).c_str());
                }

                //注意创建的时候globalSignalingThread和globalWorkerThread是必须的，否则信令不会收到
                mPeerConnectionFactory = webrtc::CreatePeerConnectionFactory(
                        networkThread.get() /* network_thread */,
                        globalWorkerThread.get()/* worker_thread */,
                        globalSignalingThread.get() /* signaling_thread */,
                        myADM /* default_adm */,
                        std::move(myAudioEncoderFactory),
                        webrtc::CreateBuiltinAudioDecoderFactory(),
                        std::move(myVideoEncoderFactory),
                        webrtc::CreateBuiltinVideoDecoderFactory(),
                        nullptr /* audio_mixer */,
                        nullptr /* audio_processing */);

                if (!mPeerConnectionFactory)
                {
                        QL_ERROR("Create P2P Connection Factory error.");
                        DeletePeerConnection();
                        return false;
                }



                //Config 更多的是针对于ICE等网络配置
                webrtc::PeerConnectionInterface::RTCConfiguration myRTCConfig;
                //允许动态切换编码器根据网络带宽情况
                myRTCConfig.allow_codec_switching = true;
                //CPU自适应编码或负载情况
                myRTCConfig.set_cpu_adaptation(true);
                myRTCConfig.set_dscp(true);

                myRTCConfig.set_prerenderer_smoothing(true);
                //只有这个配置，其余都废弃了
                myRTCConfig.sdp_semantics = webrtc::SdpSemantics::kUnifiedPlan;
                //默认就是DTLS的
                //这个配置被移动到CryptoOptions下面
                //myRTCConfig.enable_dtls_srtp = true;
                webrtc::CryptoOptions myCryptoOptions;
                std::vector<int> myWay= myCryptoOptions.GetSupportedDtlsSrtpCryptoSuites();
                myCryptoOptions.sframe.require_frame_encryption = true;
                myCryptoOptions.srtp.enable_encrypted_rtp_header_extensions = true;
                myCryptoOptions.srtp.enable_aes128_sha1_80_crypto_cipher = true;
                myRTCConfig.crypto_options = myCryptoOptions;
                //myRTCConfig.screencast_min_bitrate
                // 
                //禁止TCP方式传输(不使用TCP进行媒体传输)
                myRTCConfig.tcp_candidate_policy = webrtc::PeerConnectionInterface::kTcpCandidatePolicyDisabled;
                //优先选用WIFI
                myRTCConfig.candidate_network_policy = webrtc::PeerConnectionInterface::CandidateNetworkPolicy::kCandidateNetworkPolicyLowCost;
                //持续寻找ICe潜在对象信息
                //允许WebRTC在通讯过程中持续收集candidates，一旦网络状况发生改变，将使用当前有效的candidate pair取代失效的candidate pair进行数据传输。
                //本地环境下实测，禁用一个网卡时，WebRTC的数据传输会迅速调整到另一块网卡，视频流几乎没有卡顿出现
                myRTCConfig.continual_gathering_policy = webrtc::PeerConnectionInterface::ContinualGatheringPolicy::GATHER_CONTINUALLY;
                //设置ice通路方式
                myRTCConfig.type = webrtc::PeerConnectionInterface::IceTransportsType::kAll;

                if (QL::QLGlobalConfig::Get().IsUseTurnServer())
                {
                        //Please notice ,the turn server content should like as: "turn:10.2.3.4:3478" [turn:] as prefix
                        //
                        ////add turn server
                        webrtc::PeerConnectionInterface::IceServer turn_server;
                        turn_server.urls.push_back(QL::QLGlobalConfig::Get().TurnServer());
                        turn_server.username = QL::QLGlobalConfig::Get().TurnServerUsername();
                        turn_server.password = QL::QLGlobalConfig::Get().TurnServerPwd();
                        myRTCConfig.servers.push_back(turn_server);

                }
                if (QL::QLGlobalConfig::Get().IsUseStunServer())
                {
                        ////add stun server 
                        webrtc::PeerConnectionInterface::IceServer stunServer;
                        stunServer.urls.push_back(QL::QLGlobalConfig::Get().StunServer());
                        stunServer.username = QL::QLGlobalConfig::Get().StunServerUsername();
                        stunServer.password = QL::QLGlobalConfig::Get().StunServerPwd();
                        myRTCConfig.servers.push_back(stunServer);
                        QL_INFO("Add Turn server");
                }

                //设置端口，如果配置中>0的设置。
                webrtc::PeerConnectionInterface::PortAllocatorConfig myPortConfig;
                if (QLGlobalConfig::Get().MinPort() > 0)
                {
                        myPortConfig.min_port = QLGlobalConfig::Get().MinPort();
                }
                if (QLGlobalConfig::Get().MaxPort() > 0)
                {
                        myPortConfig.max_port = QLGlobalConfig::Get().MaxPort();
                }
                myRTCConfig.port_allocator_config = myPortConfig;


                //使用配置真实创建对象
                mPeerConnection = mPeerConnectionFactory->CreatePeerConnection(myRTCConfig, nullptr, nullptr, this);

                if (!mPeerConnection)
                {
                        QL_ERROR("Create P2P Connection error.");
                        DeletePeerConnection();
                        return false;
                }

                //add to global session to maintain p2p object
                QLRoomSession::Get().CurrentP2PConnection(mPeerConnection);

                //Try to set bitrate if we want to 
                //so far these codes are not affect logic since the CreateOffer was called by web client
                //If native client CreateOffer then these codes will be useful.
                if (!QL::QLGlobalConfig::Get().IsAutoBitrate())
                {
                        webrtc::BitrateSettings mySettings;
                        mySettings.max_bitrate_bps = QL::QLGlobalConfig::Get().MaxBitrate();
                        mySettings.min_bitrate_bps = QL::QLGlobalConfig::Get().MinBitrate();
                        mySettings.start_bitrate_bps = QL::QLGlobalConfig::Get().StartBitrate();

                        mPeerConnection->SetBitrate(mySettings);
                }


                //webrtc::RtpParameters parameters=mPeerConnection->GetSenders()[0].get()->GetParameters();
                //parameters.encodings[0].ss
                // 
                // 
                //mPeerConnection->AddIceCandidate();
                return mPeerConnection != nullptr;
        }

我们针对上述的源代码进行重要内容的解释。

重要的步骤有：

  1. 首先需要创建几个线程对象，Global Worker 线程和 Global Signaling线程network线程。否则webrtc的消息循环不能正常工作。导致ICE消息或者SDP讯息，回调函数等不能被正常触发。具体的关系比较复杂，可能需要看webrtc的源代码才能梳理清楚
  2. 创建P2P的时候，需要我们创建好2个工厂对象，一个是音频的，一个是视频的。如果采用webrtc内置的，就直接嗲用CreateBuiltinVideoEncoderFactory和CreateBuiltinAudioEncoderFactory方法，编码器对应的解码器这块，分别是CreateBuiltinAudioDecoderFactory和CreateBuiltinVideoDecoderFactory
  3. 第四个参数ADM很有意思，管理声卡相关的对象，我们有一次在Linux平台上的一台机器没有安装声卡，这个对象出错。解决的方法是我们使用一个Fake的ADM对象代替。成功解决。
  4. webrtc::PeerConnectionInterface::RTCConfiguration这个配置很重要。更多的是针对于ICE等网络配置。allow_codec_switching，可以允许动态切换编码器根据网络带宽情况。set_cpu_adaptation，CPU自适应编码或负载情况。
  5. 默认相关的DTLS设置
                //默认就是DTLS的
                //这个配置被移动到CryptoOptions下面
                //myRTCConfig.enable_dtls_srtp = true;
                webrtc::CryptoOptions myCryptoOptions;
                std::vector<int> myWay= myCryptoOptions.GetSupportedDtlsSrtpCryptoSuites();
                myCryptoOptions.sframe.require_frame_encryption = true;
                myCryptoOptions.srtp.enable_encrypted_rtp_header_extensions = true;
                myCryptoOptions.srtp.enable_aes128_sha1_80_crypto_cipher = true;
                myRTCConfig.crypto_options = myCryptoOptions;

设定DTLS加密协议中相关的加密等级等内容

  1. 其余的主要设置
                //禁止TCP方式传输(不使用TCP进行媒体传输)
                myRTCConfig.tcp_candidate_policy = webrtc::PeerConnectionInterface::kTcpCandidatePolicyDisabled;
                //优先选用WIFI
                myRTCConfig.candidate_network_policy = webrtc::PeerConnectionInterface::CandidateNetworkPolicy::kCandidateNetworkPolicyLowCost;
                //持续寻找ICe潜在对象信息
                //允许WebRTC在通讯过程中持续收集candidates，一旦网络状况发生改变，将使用当前有效的candidate pair取代失效的candidate pair进行数据传输。
                //本地环境下实测，禁用一个网卡时，WebRTC的数据传输会迅速调整到另一块网卡，视频流几乎没有卡顿出现
                myRTCConfig.continual_gathering_policy = webrtc::PeerConnectionInterface::ContinualGatheringPolicy::GATHER_CONTINUALLY;
                //设置ice通路方式
                myRTCConfig.type = webrtc::PeerConnectionInterface::IceTransportsType::kAll;
  1. 在整体的P2P连接建立后，继续建立几个重要的对象，分别是AddTrack ,增加视频，音频轨道。 CreateDataChannel, 创建数据通道管道。 SetREmoteDescription, 设置远端描述，告诉远端，我们已经配置好了我们的信息，远端可以连接了。 CreateAnswer， 发送回复消息给到远端。

这些步骤完成后，我们底层的ClientSink对象，会在WebSocket层面传递网络流出去。整个机制就流转起来了。

以下附上Fake ADM的源代码头文件

#ifndef QC_FAKE_AUDIO_CAPTURE_MODULE_H_
#define QC_FAKE_AUDIO_CAPTURE_MODULE_H_

#include <stddef.h>
#include <stdint.h>

#include <memory>

#include "api/scoped_refptr.h"
#include "api/sequence_checker.h"
#include "modules/audio_device/include/audio_device.h"
#include "modules/audio_device/include/audio_device_defines.h"
#include "rtc_base/synchronization/mutex.h"
#include "rtc_base/thread_annotations.h"
#include "rtc_base/ref_counted_object.h"

namespace rtc
{
    class Thread;
}  // namespace rtc


namespace QL
{
    class FakeADM : public webrtc::AudioDeviceModule
    {
    public:
        typedef uint16_t Sample;

        // The value for the following constants have been derived by running VoE
        // using a real ADM. The constants correspond to 10ms of mono audio at 44kHz.
        static const size_t kNumberSamples = 440;
        static const size_t kNumberBytesPerSample = sizeof(Sample);

        virtual void AddRef() const override;
        virtual rtc::RefCountReleaseStatus Release() const override;

        // Creates a FakeAudioCaptureModule or returns NULL on failure.
        static rtc::scoped_refptr<FakeADM> Create();

        // Returns the number of frames that have been successfully pulled by the
        // instance. Note that correctly detecting success can only be done if the
        // pulled frame was generated/pushed from a FakeAudioCaptureModule.
        int frames_received() const RTC_LOCKS_EXCLUDED(mutex_);

        int32_t ActiveAudioLayer(AudioLayer* audio_layer) const override;

        // Note: Calling this method from a callback may result in deadlock.
        int32_t RegisterAudioCallback(webrtc::AudioTransport* audio_callback) override
            RTC_LOCKS_EXCLUDED(mutex_);

        int32_t Init() override;
        int32_t Terminate() override;
        bool Initialized() const override;

        int16_t PlayoutDevices() override;
        int16_t RecordingDevices() override;
        int32_t PlayoutDeviceName(uint16_t index,
            char name[webrtc::kAdmMaxDeviceNameSize],
            char guid[webrtc::kAdmMaxGuidSize]) override;
        int32_t RecordingDeviceName(uint16_t index,
            char name[webrtc::kAdmMaxDeviceNameSize],
            char guid[webrtc::kAdmMaxGuidSize]) override;

        int32_t SetPlayoutDevice(uint16_t index) override;
        int32_t SetPlayoutDevice(WindowsDeviceType device) override;
        int32_t SetRecordingDevice(uint16_t index) override;
        int32_t SetRecordingDevice(WindowsDeviceType device) override;

        int32_t PlayoutIsAvailable(bool* available) override;
        int32_t InitPlayout() override;
        bool PlayoutIsInitialized() const override;
        int32_t RecordingIsAvailable(bool* available) override;
        int32_t InitRecording() override;
        bool RecordingIsInitialized() const override;

        int32_t StartPlayout() RTC_LOCKS_EXCLUDED(mutex_) override;
        int32_t StopPlayout() RTC_LOCKS_EXCLUDED(mutex_) override;
        bool Playing() const RTC_LOCKS_EXCLUDED(mutex_) override;
        int32_t StartRecording() RTC_LOCKS_EXCLUDED(mutex_) override;
        int32_t StopRecording() RTC_LOCKS_EXCLUDED(mutex_) override;
        bool Recording() const RTC_LOCKS_EXCLUDED(mutex_) override;

        int32_t InitSpeaker() override;
        bool SpeakerIsInitialized() const override;
        int32_t InitMicrophone() override;
        bool MicrophoneIsInitialized() const override;

        int32_t SpeakerVolumeIsAvailable(bool* available) override;
        int32_t SetSpeakerVolume(uint32_t volume) override;
        int32_t SpeakerVolume(uint32_t* volume) const override;
        int32_t MaxSpeakerVolume(uint32_t* max_volume) const override;
        int32_t MinSpeakerVolume(uint32_t* min_volume) const override;

        int32_t MicrophoneVolumeIsAvailable(bool* available) override;
        int32_t SetMicrophoneVolume(uint32_t volume)
            RTC_LOCKS_EXCLUDED(mutex_) override;
        int32_t MicrophoneVolume(uint32_t* volume) const
            RTC_LOCKS_EXCLUDED(mutex_) override;
        int32_t MaxMicrophoneVolume(uint32_t* max_volume) const override;

        int32_t MinMicrophoneVolume(uint32_t* min_volume) const override;

        int32_t SpeakerMuteIsAvailable(bool* available) override;
        int32_t SetSpeakerMute(bool enable) override;
        int32_t SpeakerMute(bool* enabled) const override;

        int32_t MicrophoneMuteIsAvailable(bool* available) override;
        int32_t SetMicrophoneMute(bool enable) override;
        int32_t MicrophoneMute(bool* enabled) const override;

        int32_t StereoPlayoutIsAvailable(bool* available) const override;
        int32_t SetStereoPlayout(bool enable) override;
        int32_t StereoPlayout(bool* enabled) const override;
        int32_t StereoRecordingIsAvailable(bool* available) const override;
        int32_t SetStereoRecording(bool enable) override;
        int32_t StereoRecording(bool* enabled) const override;

        int32_t PlayoutDelay(uint16_t* delay_ms) const override;

        bool BuiltInAECIsAvailable() const override { return false; }
        int32_t EnableBuiltInAEC(bool enable) override { return -1; }
        bool BuiltInAGCIsAvailable() const override { return false; }
        int32_t EnableBuiltInAGC(bool enable) override { return -1; }
        bool BuiltInNSIsAvailable() const override { return false; }
        int32_t EnableBuiltInNS(bool enable) override { return -1; }

        int32_t GetPlayoutUnderrunCount() const override { return -1; }

        //absl::optional<webrtc::AudioDeviceModule::Stats> GetStats() const override
        //{
        //    return webrtc::AudioDeviceModule::Stats();
        //}
#if defined(WEBRTC_IOS)
        int GetPlayoutAudioParameters(
            webrtc::AudioParameters* params) const override {
            return -1;
        }
        int GetRecordAudioParameters(webrtc::AudioParameters* params) const override {
            return -1;
        }
#endif  // WEBRTC_IOS

        // End of functions inherited from webrtc::AudioDeviceModule.

    protected:
        // The constructor is protected because the class needs to be created as a
        // reference counted object (for memory managment reasons). It could be
        // exposed in which case the burden of proper instantiation would be put on
        // the creator of a FakeAudioCaptureModule instance. To create an instance of
        // this class use the Create(..) API.
        FakeADM();
        // The destructor is protected because it is reference counted and should not
        // be deleted directly.
        virtual ~FakeADM();

    private:

        mutable webrtc::webrtc_impl::RefCounter mRefCount;

        // Initializes the state of the FakeAudioCaptureModule. This API is called on
        // creation by the Create() API.
        bool Initialize();
        // SetBuffer() sets all samples in send_buffer_ to `value`.
        void SetSendBuffer(int value);
        // Resets rec_buffer_. I.e., sets all rec_buffer_ samples to 0.
        void ResetRecBuffer();
        // Returns true if rec_buffer_ contains one or more sample greater than or
        // equal to `value`.
        bool CheckRecBuffer(int value);

        // Returns true/false depending on if recording or playback has been
        // enabled/started.
        bool ShouldStartProcessing() RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);

        // Starts or stops the pushing and pulling of audio frames.
        void UpdateProcessing(bool start) RTC_LOCKS_EXCLUDED(mutex_);

        // Starts the periodic calling of ProcessFrame() in a thread safe way.
        void StartProcessP();
        // Periodcally called function that ensures that frames are pulled and pushed
        // periodically if enabled/started.
        void ProcessFrameP() RTC_LOCKS_EXCLUDED(mutex_);
        // Pulls frames from the registered webrtc::AudioTransport.
        void ReceiveFrameP() RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);
        // Pushes frames to the registered webrtc::AudioTransport.
        void SendFrameP() RTC_EXCLUSIVE_LOCKS_REQUIRED(mutex_);

        // Callback for playout and recording.
        webrtc::AudioTransport* audio_callback_ RTC_GUARDED_BY(mutex_);

        bool recording_ RTC_GUARDED_BY(
            mutex_);  // True when audio is being pushed from the instance.
        bool playing_ RTC_GUARDED_BY(
            mutex_);  // True when audio is being pulled by the instance.

        bool play_is_initialized_;  // True when the instance is ready to pull audio.
        bool rec_is_initialized_;   // True when the instance is ready to push audio.

        // Input to and output from RecordedDataIsAvailable(..) makes it possible to
        // modify the current mic level. The implementation does not care about the
        // mic level so it just feeds back what it receives.
        uint32_t current_mic_level_ RTC_GUARDED_BY(mutex_);

        // next_frame_time_ is updated in a non-drifting manner to indicate the next
        // wall clock time the next frame should be generated and received. started_
        // ensures that next_frame_time_ can be initialized properly on first call.
        bool started_ RTC_GUARDED_BY(mutex_);
        int64_t next_frame_time_ RTC_GUARDED_BY(process_thread_checker_);

        std::unique_ptr<rtc::Thread> process_thread_;

        // Buffer for storing samples received from the webrtc::AudioTransport.
        char rec_buffer_[kNumberSamples * kNumberBytesPerSample];
        // Buffer for samples to send to the webrtc::AudioTransport.
        char send_buffer_[kNumberSamples * kNumberBytesPerSample];

        // Counter of frames received that have samples of high enough amplitude to
        // indicate that the frames are not faked somewhere in the audio pipeline
        // (e.g. by a jitter buffer).
        int frames_received_;

        // Protects variables that are accessed from process_thread_ and
        // the main thread.
        mutable webrtc::Mutex mutex_;
        webrtc::SequenceChecker process_thread_checker_;
    };
}

#endif  // QC_FAKE_AUDIO_CAPTURE_MODULE_H_

Cpp文件


#include "FakeADM.h"
#include <string.h>

#ifdef USE_M108_WEBRTC
#include "api/make_ref_counted.h"
#endif

#if _WIN32||_WIN64
#include "api/make_ref_counted.h"
#else
#endif

#include "api/units/time_delta.h"
#include "rtc_base/checks.h"
#include "rtc_base/thread.h"
#include "rtc_base/time_utils.h"
#include <boost/bind.hpp>

using ::webrtc::TimeDelta;

// Audio sample value that is high enough that it doesn't occur naturally when
// frames are being faked. E.g. NetEq will not generate this large sample value
// unless it has received an audio frame containing a sample of this value.
// Even simpler buffers would likely just contain audio sample values of 0.
static const int kHighSampleValue = 10000;

// Constants here are derived by running VoE using a real ADM.
// The constants correspond to 10ms of mono audio at 44kHz.
static const int kTimePerFrameMs = 10;
static const uint8_t kNumberOfChannels = 1;
static const int kSamplesPerSecond = 44000;
static const int kTotalDelayMs = 0;
static const int kClockDriftMs = 0;
static const uint32_t kMaxVolume = 14392;

namespace QL
{
    FakeADM::FakeADM()
        : audio_callback_(nullptr),
        recording_(false),
        playing_(false),
        play_is_initialized_(false),
        rec_is_initialized_(false),
        current_mic_level_(kMaxVolume),
        started_(false),
        next_frame_time_(0),
        frames_received_(0),
        mRefCount(0)
    {
        process_thread_checker_.Detach();
    }

    //#include <rtc_base/atomic_ops.h>
    void FakeADM::AddRef() const
    {
        mRefCount.IncRef();
    }

    rtc::RefCountReleaseStatus FakeADM::Release() const
    {
        rtc::RefCountReleaseStatus status = mRefCount.DecRef();
        return status;
    }

    FakeADM::~FakeADM()
    {
        if (process_thread_) {
            process_thread_->Stop();
        }
    }

    rtc::scoped_refptr<FakeADM> FakeADM::Create()
    {


#ifdef USE_M108_WEBRTC
         rtc::scoped_refptr<FakeADM> capture_module(new rtc::RefCountedObject<FakeADM>());
        //auto capture_module=rtc::make_ref_counted<FakeAudioCaptureModule>();

#elif _WIN32 ||_WIN64
                //auto capture_module = rtc::make_ref_counted<FakeAudioCaptureModule>();
        rtc::scoped_refptr<FakeADM> capture_module(new rtc::RefCountedObject<FakeADM>());
#else
                auto capture_module = new FakeAudioCaptureModule();
#endif


        if (!capture_module->Initialize()) {
            return nullptr;
        }
        return capture_module;
    }

    int FakeADM::frames_received() const {
        webrtc::MutexLock lock(&mutex_);
        return frames_received_;
    }

    int32_t FakeADM::ActiveAudioLayer(
        AudioLayer* /*audio_layer*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::RegisterAudioCallback(
        webrtc::AudioTransport* audio_callback) {
        webrtc::MutexLock lock(&mutex_);
        audio_callback_ = audio_callback;
        return 0;
    }

    int32_t FakeADM::Init() {
        // Initialize is called by the factory method. Safe to ignore this Init call.
        return 0;
    }

    int32_t FakeADM::Terminate() {
        // Clean up in the destructor. No action here, just success.
        return 0;
    }

    bool FakeADM::Initialized() const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int16_t FakeADM::PlayoutDevices() {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int16_t FakeADM::RecordingDevices() {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::PlayoutDeviceName(
        uint16_t /*index*/,
        char /*name*/[webrtc::kAdmMaxDeviceNameSize],
        char /*guid*/[webrtc::kAdmMaxGuidSize]) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::RecordingDeviceName(
        uint16_t /*index*/,
        char /*name*/[webrtc::kAdmMaxDeviceNameSize],
        char /*guid*/[webrtc::kAdmMaxGuidSize]) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SetPlayoutDevice(uint16_t /*index*/) {
        // No playout device, just playing from file. Return success.
        return 0;
    }

    int32_t FakeADM::SetPlayoutDevice(WindowsDeviceType /*device*/) {
        if (play_is_initialized_) {
            return -1;
        }
        return 0;
    }

    int32_t FakeADM::SetRecordingDevice(uint16_t /*index*/) {
        // No recording device, just dropping audio. Return success.
        return 0;
    }

    int32_t FakeADM::SetRecordingDevice(
        WindowsDeviceType /*device*/) {
        if (rec_is_initialized_) {
            return -1;
        }
        return 0;
    }

    int32_t FakeADM::PlayoutIsAvailable(bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::InitPlayout() {
        play_is_initialized_ = true;
        return 0;
    }

    bool FakeADM::PlayoutIsInitialized() const {
        return play_is_initialized_;
    }

    int32_t FakeADM::RecordingIsAvailable(bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::InitRecording() {
        rec_is_initialized_ = true;
        return 0;
    }

    bool FakeADM::RecordingIsInitialized() const {
        return rec_is_initialized_;
    }

    int32_t FakeADM::StartPlayout() {
        if (!play_is_initialized_) {
            return -1;
        }
        {
            webrtc::MutexLock lock(&mutex_);
            playing_ = true;
        }
        bool start = true;
        UpdateProcessing(start);
        return 0;
    }

    int32_t FakeADM::StopPlayout() {
        bool start = false;
        {
            webrtc::MutexLock lock(&mutex_);
            playing_ = false;
            start = ShouldStartProcessing();
        }
        UpdateProcessing(start);
        return 0;
    }

    bool FakeADM::Playing() const {
        webrtc::MutexLock lock(&mutex_);
        return playing_;
    }

    int32_t FakeADM::StartRecording() {
        if (!rec_is_initialized_) {
            return -1;
        }
        {
            webrtc::MutexLock lock(&mutex_);
            recording_ = true;
        }
        bool start = true;
        UpdateProcessing(start);
        return 0;
    }

    int32_t FakeADM::StopRecording() {
        bool start = false;
        {
            webrtc::MutexLock lock(&mutex_);
            recording_ = false;
            start = ShouldStartProcessing();
        }
        UpdateProcessing(start);
        return 0;
    }

    bool FakeADM::Recording() const {
        webrtc::MutexLock lock(&mutex_);
        return recording_;
    }

    int32_t FakeADM::InitSpeaker() {
        // No speaker, just playing from file. Return success.
        return 0;
    }

    bool FakeADM::SpeakerIsInitialized() const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::InitMicrophone() {
        // No microphone, just playing from file. Return success.
        return 0;
    }

    bool FakeADM::MicrophoneIsInitialized() const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SpeakerVolumeIsAvailable(bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SetSpeakerVolume(uint32_t /*volume*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SpeakerVolume(uint32_t* /*volume*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::MaxSpeakerVolume(
        uint32_t* /*max_volume*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::MinSpeakerVolume(
        uint32_t* /*min_volume*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::MicrophoneVolumeIsAvailable(
        bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SetMicrophoneVolume(uint32_t volume) {
        webrtc::MutexLock lock(&mutex_);
        current_mic_level_ = volume;
        return 0;
    }

    int32_t FakeADM::MicrophoneVolume(uint32_t* volume) const {
        webrtc::MutexLock lock(&mutex_);
        *volume = current_mic_level_;
        return 0;
    }

    int32_t FakeADM::MaxMicrophoneVolume(
        uint32_t* max_volume) const {
        *max_volume = kMaxVolume;
        return 0;
    }

    int32_t FakeADM::MinMicrophoneVolume(
        uint32_t* /*min_volume*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SpeakerMuteIsAvailable(bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SetSpeakerMute(bool /*enable*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SpeakerMute(bool* /*enabled*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::MicrophoneMuteIsAvailable(bool* /*available*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::SetMicrophoneMute(bool /*enable*/) {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::MicrophoneMute(bool* /*enabled*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::StereoPlayoutIsAvailable(
        bool* available) const {
        // No recording device, just dropping audio. Stereo can be dropped just
        // as easily as mono.
        *available = true;
        return 0;
    }

    int32_t FakeADM::SetStereoPlayout(bool /*enable*/) {
        // No recording device, just dropping audio. Stereo can be dropped just
        // as easily as mono.
        return 0;
    }

    int32_t FakeADM::StereoPlayout(bool* /*enabled*/) const {
        RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::StereoRecordingIsAvailable(
        bool* available) const {
        // Keep thing simple. No stereo recording.
        *available = false;
        return 0;
    }

    int32_t FakeADM::SetStereoRecording(bool enable) {
        if (!enable) {
            return 0;
        }
        return -1;
    }

    int32_t FakeADM::StereoRecording(bool* /*enabled*/) const {
        //RTC_DCHECK_NOTREACHED();
        return 0;
    }

    int32_t FakeADM::PlayoutDelay(uint16_t* delay_ms) const {
        // No delay since audio frames are dropped.
        *delay_ms = 0;
        return 0;
    }

    bool FakeADM::Initialize() {
        // Set the send buffer samples high enough that it would not occur on the
        // remote side unless a packet containing a sample of that magnitude has been
        // sent to it. Note that the audio processing pipeline will likely distort the
        // original signal.
        SetSendBuffer(kHighSampleValue);
        return true;
    }

    void FakeADM::SetSendBuffer(int value) {
        Sample* buffer_ptr = reinterpret_cast<Sample*>(send_buffer_);
        const size_t buffer_size_in_samples =
            sizeof(send_buffer_) / kNumberBytesPerSample;
        for (size_t i = 0; i < buffer_size_in_samples; ++i) {
            buffer_ptr[i] = value;
        }
    }

    void FakeADM::ResetRecBuffer() {
        memset(rec_buffer_, 0, sizeof(rec_buffer_));
    }

    bool FakeADM::CheckRecBuffer(int value) {
        const Sample* buffer_ptr = reinterpret_cast<const Sample*>(rec_buffer_);
        const size_t buffer_size_in_samples =
            sizeof(rec_buffer_) / kNumberBytesPerSample;
        for (size_t i = 0; i < buffer_size_in_samples; ++i) {
            if (buffer_ptr[i] >= value)
                return true;
        }
        return false;
    }

    bool FakeADM::ShouldStartProcessing() {
        return recording_ || playing_;
    }

    void FakeADM::UpdateProcessing(bool start) {
        if (start) {
            if (!process_thread_) {
                process_thread_ = rtc::Thread::Create();
                process_thread_->Start();
            }
#if _WIN32||_WIN64

            process_thread_->PostTask([this] { StartProcessP(); });

           
#else
            //process_thread_->PostTask(boost::bind(&FakeAudioCaptureModule::UpdateProcessing, this, start));
#endif
        }
        else {
            if (process_thread_) {
                process_thread_->Stop();
                process_thread_.reset(nullptr);
                process_thread_checker_.Detach();
            }
            webrtc::MutexLock lock(&mutex_);
            started_ = false;
        }
    }

    void FakeADM::StartProcessP() {
        RTC_DCHECK_RUN_ON(&process_thread_checker_);
        {
            webrtc::MutexLock lock(&mutex_);
            if (started_) 
            {
                // Already started.
                return;
            }
        }
        ProcessFrameP();
    }

    void FakeADM::ProcessFrameP() {
        RTC_DCHECK_RUN_ON(&process_thread_checker_);
        {
            webrtc::MutexLock lock(&mutex_);
            if (!started_) {
                next_frame_time_ = rtc::TimeMillis();
                started_ = true;
            }

            // Receive and send frames every kTimePerFrameMs.
            if (playing_) {
                ReceiveFrameP();
            }
            if (recording_) {
                SendFrameP();
            }
        }

        next_frame_time_ += kTimePerFrameMs;
        const int64_t current_time = rtc::TimeMillis();
        const int64_t wait_time =
            (next_frame_time_ > current_time) ? next_frame_time_ - current_time : 0;

#if _WIN32||_WIN64

        process_thread_->PostDelayedTask([this] { ProcessFrameP(); },TimeDelta::Millis(wait_time));
        
#else
        //process_thread_->PostDelayedTask(boost::bind(&FakeAudioCaptureModule::ProcessFrameP, this), webrtc::TimeDelta::Millis(wait_time));
#endif

    }

    void FakeADM::ReceiveFrameP() {
        RTC_DCHECK_RUN_ON(&process_thread_checker_);
        if (!audio_callback_) {
            return;
        }
        ResetRecBuffer();
        size_t nSamplesOut = 0;
        int64_t elapsed_time_ms = 0;
        int64_t ntp_time_ms = 0;
        if (audio_callback_->NeedMorePlayData(kNumberSamples, kNumberBytesPerSample,
            kNumberOfChannels, kSamplesPerSecond,
            rec_buffer_, nSamplesOut,
            &elapsed_time_ms, &ntp_time_ms) != 0) {
            //RTC_DCHECK_NOTREACHED();
        }
        RTC_CHECK(nSamplesOut == kNumberSamples);

        // The SetBuffer() function ensures that after decoding, the audio buffer
        // should contain samples of similar magnitude (there is likely to be some
        // distortion due to the audio pipeline). If one sample is detected to
        // have the same or greater magnitude somewhere in the frame, an actual frame
        // has been received from the remote side (i.e. faked frames are not being
        // pulled).
        if (CheckRecBuffer(kHighSampleValue)) {
            ++frames_received_;
        }
    }

    void FakeADM::SendFrameP() {
        RTC_DCHECK_RUN_ON(&process_thread_checker_);
        if (!audio_callback_) {
            return;
        }
        bool key_pressed = false;
        uint32_t current_mic_level = current_mic_level_;
        if (audio_callback_->RecordedDataIsAvailable(
            send_buffer_, kNumberSamples, kNumberBytesPerSample,
            kNumberOfChannels, kSamplesPerSecond, kTotalDelayMs, kClockDriftMs,
            current_mic_level, key_pressed, current_mic_level) != 0) {
            //RTC_DCHECK_NOTREACHED();
        }
        current_mic_level_ = current_mic_level;
    }

}

RA/SD 衍生者AI训练营。发布者：chris，转载请注明出处：https://www.shxcj.com/archives/6568

Like (0)
0 0

关于作者

chris的头像

chris

641 文章
5 评论
1 粉丝
这个人很懒，什么都没有留下～
Previous 2024-09-30 2:19 下午
Next 2024-09-30 2:22 下午

相关推荐

发表回复

Please Login to Comment
本文授权以下站点有原版访问授权 https://www.shxcj.com https://www.2img.ai https://www.2video.cn