使用Go語言開發流媒體視頻網站

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流媒體如今已經成爲工業上的一個重要技術了，比如：直播網站、視頻監控傳輸、APP直播等，如何實現一個高併發的視頻網站，這就涉及到語言技術的選型以及流媒體技術的使用，本節將主要介紹如何使用Golang來實現一個流媒體視頻網站。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"目錄","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼選擇Go以及Go的一些優勢","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GoLang的簡介以及實現一個webserver工具鏈","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Golang的channel併發模式","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用Golang完成一個流媒體網站","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網站部署","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"爲什麼選擇Go以及Go的一些優勢","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼會選擇Go來開發視頻網站呢？這其實主要體現在Go語言的優勢。那麼Go有哪些優勢呢？","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"開發效率高，不管用其他語言，都需要很多其他的配置或插件，就連全家桶配套齊全的Java語言都會需要一個Servlet引擎，如：tomcat、jetty等。但Go在這方面，提供了得天獨厚的能力。大部分功能和內容已經集成在了pkg。包括開發完整的開發工具鏈（tools、test、benchmark、builtin.etc），包括Go命令（go test、go install、go build）。這些都是完整的，直接下載Go後即可使用。包括音視頻相關的插件、配置，都已經被包含在了pkg。所以用go開發音視頻，可以完美詮釋go語言的優勢。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"另一方面，部署簡單，go屬於編譯性語言，而且是能夠編譯多個平臺可執行文件的語言。Compile once，run everywhere，直接編譯後生成二進制文件，直接運行。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"良好的native http庫、集成模板引擎，無需添加第三方框架。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"GoLang的簡介以及實現一個webserver工具鏈","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Go語言是一種編譯性語言，一個開源的編程語言，能夠讓構造簡單、可靠且高效的軟件變得容易。而且它的目標是兼具Python等動態語言的開發速度和集成C/C++等編譯語言的性能與安全性。它的主要特性是：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡潔、快速、安全、並行、內存管理、數組安全、編譯迅速。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Go中有一些常見的工具鏈，比如：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go build，編譯go文件，可以跨平臺編譯：env GOOS=linux GOARCH=amd64 go build，在CI/CD中，這是一個非常有用的命令。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go install，這也是編譯，但與build的區別是編譯後將輸出文件打包成庫放在pkg下。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go get，用於獲取go的第三方包，常見的是：go get -u git地址，表示從git上獲取某個資源並安裝到本地。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go fmt，統一代碼風格、排版，這將使得go代碼更加易讀、易理解。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go test，運行當前目錄下的tests，\"go test -v\" 會打印所有的信息，而\"go test\"只會打印測試的結果。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"go的test文件一般以XXX_test.go命名，這樣，在執行\"go test\"的時候，程序會自動去執行那些被加了test的文件，這是一種約定的方式。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"要點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"使用TestMain作爲初始化test，並且使用Run()來調用其它tests可以完成一些需要初始化操作的testing，如：音視頻資源數據庫、文件加載等，這些有的可能需要被多次使用，但在設計模式中，只會加載一次到內存，這樣，可以減少過多的內存佔用，同時可以一次性的進行清理。","attrs":{}}]}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"func TestMain(m *testing.M) {\n    fmt.Println(\"Test begin\")\n    m.Run()\n}\n","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"如果沒在其中加Run()，除了TestMain的其它的tests都不被執行。","attrs":{}}]}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"func TestPrint(t *testing.T) {\n    fmt.Println(\"Test print\")\n}\n\nfunc TestMain(m *testing.M) {\n    fmt.Println(\"Test begin\")\n    //m.Run()\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按照上面說的，如果沒有執行Run()方法，則TestPrint函數不會被執行。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Golang的channel併發模式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Go 中，既然有了協程，那麼這些協程之間如何通信呢？Go 提供了一個 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"channel（通道）","attrs":{}},{"type":"text","text":" 來解決。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"聲明一個 channel","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Go 語言中，聲明一個 channel 非常簡單，使用內置的 make 函數即可，如下:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"ch:=make(chan string)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中 chan 是一個關鍵字，表示是 channel 類型。後面的 string 表示 channel 裏的數據是 string 類型。通過 channel 的聲明也可以看到，chan 是一個集合類型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定義好 chan 後就可以使用了，一個 chan 的操作只有兩種：發送和接收:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"發送：向 chan 發送值，把值放在 chan 中，操作符爲 chan = SAVE_STATICS_INTERVAL {\n\t\tdiffTimestamp := (nowInMS - v.WriteBWInfo.LastTimestamp) / 1000\n\n\t\tv.WriteBWInfo.VideoSpeedInBytesperMS = (v.WriteBWInfo.VideoDatainBytes - v.WriteBWInfo.LastVideoDatainBytes) * 8 / uint64(diffTimestamp) / 1000\n\t\tv.WriteBWInfo.AudioSpeedInBytesperMS = (v.WriteBWInfo.AudioDatainBytes - v.WriteBWInfo.LastAudioDatainBytes) * 8 / uint64(diffTimestamp) / 1000\n\n\t\tv.WriteBWInfo.LastVideoDatainBytes = v.WriteBWInfo.VideoDatainBytes\n\t\tv.WriteBWInfo.LastAudioDatainBytes = v.WriteBWInfo.AudioDatainBytes\n\t\tv.WriteBWInfo.LastTimestamp = nowInMS\n\t}\n}\n\nfunc (v *VirWriter) Check() {\n\tvar c core.ChunkStream\n\tfor {\n\t\tif err := v.conn.Read(&c); err != nil {\n\t\t\tv.Close(err)\n\t\t\treturn\n\t\t}\n\t}\n}\n\nfunc (v *VirWriter) DropPacket(pktQue chan *av.Packet, info av.Info) {\n\tlog.Warningf(\"[%v] packet queue max!!!\", info)\n\tfor i := 0; i < maxQueueNum-84; i++ {\n\t\ttmpPkt, ok := maxQueueNum-2 {\n\t\t\t\tlog.Debug(\"drop audio pkt\")\n\t\t\t\t maxQueueNum-10 {\n\t\t\t\tlog.Debug(\"drop video pkt\")\n\t\t\t\t= maxQueueNum-24 {\n\t\tv.DropPacket(v.packetQueue, v.Info())\n\t} else {\n\t\tv.packetQueue = SAVE_STATICS_INTERVAL {\n\t\tdiffTimestamp := (nowInMS - v.ReadBWInfo.LastTimestamp) / 1000\n\n\t\t//log.Printf(\"now=%d, last=%d, diff=%d\", nowInMS, v.ReadBWInfo.LastTimestamp, diffTimestamp)\n\t\tv.ReadBWInfo.VideoSpeedInBytesperMS = (v.ReadBWInfo.VideoDatainBytes - v.ReadBWInfo.LastVideoDatainBytes) * 8 / uint64(diffTimestamp) / 1000\n\t\tv.ReadBWInfo.AudioSpeedInBytesperMS = (v.ReadBWInfo.AudioDatainBytes - v.ReadBWInfo.LastAudioDatainBytes) * 8 / uint64(diffTimestamp) / 1000\n\n\t\tv.ReadBWInfo.LastVideoDatainBytes = v.ReadBWInfo.VideoDatainBytes\n\t\tv.ReadBWInfo.LastAudioDatainBytes = v.ReadBWInfo.AudioDatainBytes\n\t\tv.ReadBWInfo.LastTimestamp = nowInMS\n\t}\n}\n\nfunc (v *VirReader) Read(p *av.Packet) (err error) {\n\tdefer func() {\n\t\tif r := recover(); r != nil {\n\t\t\tlog.Warning(\"rtmp read packet panic: \", r)\n\t\t}\n\t}()\n\n\tv.SetPreTime()\n\tvar cs core.ChunkStream\n\tfor {\n\t\terr = v.conn.Read(&cs)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif cs.TypeID == av.TAG_AUDIO ||\n\t\t\tcs.TypeID == av.TAG_VIDEO ||\n\t\t\tcs.TypeID == av.TAG_SCRIPTDATAAMF0 ||\n\t\t\tcs.TypeID == av.TAG_SCRIPTDATAAMF3 {\n\t\t\tbreak\n\t\t}\n\t}\n\n\tp.IsAudio = cs.TypeID == av.TAG_AUDIO\n\tp.IsVideo = cs.TypeID == av.TAG_VIDEO\n\tp.IsMetadata = cs.TypeID == av.TAG_SCRIPTDATAAMF0 || cs.TypeID == av.TAG_SCRIPTDATAAMF3\n\tp.StreamID = cs.StreamID\n\tp.Data = cs.Data\n\tp.TimeStamp = cs.Timestamp\n\n\tv.SaveStatics(p.StreamID, uint64(len(p.Data)), p.IsVideo)\n\tv.demuxer.DemuxH(p)\n\treturn err\n}\n\nfunc (v *VirReader) Info() (ret av.Info) {\n\tret.UID = v.Uid\n\t_, _, URL := v.conn.GetInfo()\n\tret.URL = URL\n\t_url, err := url.Parse(URL)\n\tif err != nil {\n\t\tlog.Warning(err)\n\t}\n\tret.Key = strings.TrimLeft(_url.Path, \"/\")\n\treturn\n}\n\nfunc (v *VirReader) Close(err error) {\n\tlog.Debug(\"publisher \", v.Info(), \"closed: \"+err.Error())\n\tv.conn.Close(err)\n}","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"播放","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"視頻流媒體播放，支持多種協議：rtmp、flv、hls，我們先看看rtmp：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"var (\n\tSTOP_CTRL = \"RTMPRELAY_STOP\"\n)\n\ntype RtmpRelay struct {\n\tPlayUrl string\n\tPublishUrl string\n\tcs_chan chan core.ChunkStream\n\tsndctrl_chan chan string\n\tconnectPlayClient *core.ConnClient\n\tconnectPublishClient *core.ConnClient\n\tstartflag bool\n}\n\nfunc NewRtmpRelay(playurl *string, publishurl *string) *RtmpRelay {\n\treturn &RtmpRelay{\n\t\tPlayUrl: *playurl,\n\t\tPublishUrl: *publishurl,\n\t\tcs_chan: make(chan core.ChunkStream, 500),\n\t\tsndctrl_chan: make(chan string),\n\t\tconnectPlayClient: nil,\n\t\tconnectPublishClient: nil,\n\t\tstartflag: false,\n\t}\n}\n\nfunc (self *RtmpRelay) rcvPlayChunkStream() {\n\tlog.Debug(\"rcvPlayRtmpMediaPacket connectClient.Read...\")\n\tfor {\n\t\tvar rc core.ChunkStream\n\n\t\tif self.startflag == false {\n\t\t\tself.connectPlayClient.Close(nil)\n\t\t\tlog.Debugf(\"rcvPlayChunkStream close: playurl=%s, publishurl=%s\", self.PlayUrl, self.PublishUrl)\n\t\t\tbreak\n\t\t}\n\t\terr := self.connectPlayClient.Read(&rc)\n\n\t\tif err != nil && err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\t//log.Debugf(\"connectPlayClient.Read return rc.TypeID=%v length=%d, err=%v\", rc.TypeID, len(rc.Data), err)\n\t\tswitch rc.TypeID {\n\t\tcase 20, 17:\n\t\t\tr := bytes.NewReader(rc.Data)\n\t\t\tvs, err := self.connectPlayClient.DecodeBatch(r, amf.AMF0)\n\n\t\t\tlog.Debugf(\"rcvPlayRtmpMediaPacket: vs=%v, err=%v\", vs, err)\n\t\tcase 18:\n\t\t\tlog.Debug(\"rcvPlayRtmpMediaPacket: metadata....\")\n\t\tcase 8, 9:\n\t\t\tself.cs_chan maxQueueNum-2 {\n\t\t\t\t maxQueueNum-10 {\n\t\t\t\t= maxQueueNum-24 {\n\t\tsource.DropPacket(source.packetQueue, source.info)\n\t} else {\n\t\tif !source.closed {\n\t\t\tsource.packetQueue = duration {\n\t\tsource.flushAudio()\n\n\t\tsource.seq++\n\t\tfilename := fmt.Sprintf(\"/%s/%d.ts\", source.info.Key, time.Now().Unix())\n\t\titem := NewTSItem(filename, int(source.stat.durationMs()), source.seq, source.btswriter.Bytes())\n\t\tsource.tsCache.SetItem(filename, item)\n\n\t\tsource.btswriter.Reset()\n\t\tsource.stat.resetAndNew()\n\t} else {\n\t\tnewf = false\n\t}\n\tif newf {\n\t\tsource.btswriter.Write(source.muxer.PAT())\n\t\tsource.btswriter.Write(source.muxer.PMT(av.SOUND_AAC, true))\n\t}\n}\n\nfunc (source *Source) parse(p *av.Packet) (int32, bool, error) {\n\tvar compositionTime int32\n\tvar ah av.AudioPacketHeader\n\tvar vh av.VideoPacketHeader\n\tif p.IsVideo {\n\t\tvh = p.Header.(av.VideoPacketHeader)\n\t\tif vh.CodecID() != av.VIDEO_H264 {\n\t\t\treturn compositionTime, false, ErrNoSupportVideoCodec\n\t\t}\n\t\tcompositionTime = vh.CompositionTime()\n\t\tif vh.IsKeyFrame() && vh.IsSeq() {\n\t\t\treturn compositionTime, true, source.tsparser.Parse(p, source.bwriter)\n\t\t}\n\t} else {\n\t\tah = p.Header.(av.AudioPacketHeader)\n\t\tif ah.SoundFormat() != av.SOUND_AAC {\n\t\t\treturn compositionTime, false, ErrNoSupportAudioCodec\n\t\t}\n\t\tif ah.AACPacketType() == av.AAC_SEQHDR {\n\t\t\treturn compositionTime, true, source.tsparser.Parse(p, source.bwriter)\n\t\t}\n\t}\n\tsource.bwriter.Reset()\n\tif err := source.tsparser.Parse(p, source.bwriter); err != nil {\n\t\treturn compositionTime, false, err\n\t}\n\tp.Data = source.bwriter.Bytes()\n\n\tif p.IsVideo && vh.IsKeyFrame() {\n\t\tsource.cut()\n\t}\n\treturn compositionTime, false, nil\n}\n\nfunc (source *Source) calcPtsDts(isVideo bool, ts, compositionTs uint32) {\n\tsource.dts = uint64(ts) * h264_default_hz\n\tif isVideo {\n\t\tsource.pts = source.dts + uint64(compositionTs)*h264_default_hz\n\t} else {\n\t\tsampleRate, _ := source.tsparser.SampleRate()\n\t\tsource.align.align(&source.dts, uint32(videoHZ*aacSampleLen/sampleRate))\n\t\tsource.pts = source.dts\n\t}\n}\nfunc (source *Source) flushAudio() error {\n\treturn source.muxAudio(1)\n}\n\nfunc (source *Source) muxAudio(limit byte) error {\n\tif source.cache.CacheNum() < limit {\n\t\treturn nil\n\t}\n\tvar p av.Packet\n\t_, pts, buf := source.cache.GetFrame()\n\tp.Data = buf\n\tp.TimeStamp = uint32(pts / h264_default_hz)\n\treturn source.muxer.Mux(&p, source.btswriter)\n}\n\nfunc (source *Source) tsMux(p *av.Packet) error {\n\tif p.IsVideo {\n\t\treturn source.muxer.Mux(p, source.btswriter)\n\t} else {\n\t\tsource.cache.Cache(p.Data, source.pts)\n\t\treturn source.muxAudio(cache_max_frames)\n\t}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於hls本身的流傳輸，還是遵循http協議：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"const (\n\tduration = 3000\n)\n\nvar (\n\tErrNoPublisher = fmt.Errorf(\"no publisher\")\n\tErrInvalidReq = fmt.Errorf(\"invalid req url path\")\n\tErrNoSupportVideoCodec = fmt.Errorf(\"no support video codec\")\n\tErrNoSupportAudioCodec = fmt.Errorf(\"no support audio codec\")\n)\n\nvar crossdomainxml = []byte(`\n\n\t\n\t\n`)\n\ntype Server struct {\n\tlistener net.Listener\n\tconns *sync.Map\n}\n\nfunc NewServer() *Server {\n\tret := &Server{\n\t\tconns: &sync.Map{},\n\t}\n\tgo ret.checkStop()\n\treturn ret\n}\n\nfunc (server *Server) Serve(listener net.Listener) error {\n\tmux := http.NewServeMux()\n\tmux.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\n\t\tserver.handle(w, r)\n\t})\n\tserver.listener = listener\n\thttp.Serve(listener, mux)\n\treturn nil\n}\n\nfunc (server *Server) GetWriter(info av.Info) av.WriteCloser {\n\tvar s *Source\n\tv, ok := server.conns.Load(info.Key)\n\tif !ok {\n\t\tlog.Debug(\"new hls source\")\n\t\ts = NewSource(info)\n\t\tserver.conns.Store(info.Key, s)\n\t} else {\n\t\ts = v.(*Source)\n\t}\n\treturn s\n}\n\nfunc (server *Server) getConn(key string) *Source {\n\tv, ok := server.conns.Load(key)\n\tif !ok {\n\t\treturn nil\n\t}\n\treturn v.(*Source)\n}\n\nfunc (server *Server) checkStop() {\n\tfor {\n\t\t> 24 & 0xff\n\n\tpio.PutU8(h[0:1], uint8(typeID))\n\tpio.PutI24BE(h[1:4], int32(dataLen))\n\tpio.PutI24BE(h[4:7], int32(timestampbase))\n\tpio.PutU8(h[7:8], uint8(timestampExt))\n\n\tif _, err := writer.ctx.Write(h); err != nil {\n\t\treturn err\n\t}\n\n\tif _, err := writer.ctx.Write(p.Data); err != nil {\n\t\treturn err\n\t}\n\n\tpio.PutI32BE(h[:4], int32(preDataLen))\n\tif _, err := writer.ctx.Write(h[:4]); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc (writer *FLVWriter) Wait() {\n\tselect {\n\tcase > 4\n\ttag.mediat.soundRate = (flags >> 2) & 0x3\n\ttag.mediat.soundSize = (flags >> 1) & 0x1\n\ttag.mediat.soundType = flags & 0x1\n\tn++\n\tswitch tag.mediat.soundFormat {\n\tcase av.SOUND_AAC:\n\t\ttag.mediat.aacPacketType = b[1]\n\t\tn++\n\t}\n\treturn\n}\n\nfunc (tag *Tag) parseVideoHeader(b []byte) (n int, err error) {\n\tif len(b) < n+5 {\n\t\terr = fmt.Errorf(\"invalid videodata len=%d\", len(b))\n\t\treturn\n\t}\n\tflags := b[0]\n\ttag.mediat.frameType = flags >> 4\n\ttag.mediat.codecID = flags & 0xf\n\tn++\n\tif tag.mediat.frameType == av.FRAME_INTER || tag.mediat.frameType == av.FRAME_KEY {\n\t\ttag.mediat.avcPacketType = b[1]\n\t\tfor i := 2; i < 5; i++ {\n\t\t\ttag.mediat.compositionTime = tag.mediat.compositionTime<<8 + int32(b[i])\n\t\t}\n\t\tn += 4\n\t}\n\treturn\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家知道在數據傳輸、數據存儲時，爲了保證數據的正確性，需要採用檢錯的手段來處理，crc在很多檢錯手段中是最出名的一種，其檢錯能力極強，開銷小，易於用編碼器及檢測電路實現。所以這裏對於 ts 格式的多路調製，採用了 crc32 法校驗：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"func (muxer *Muxer) Mux(p *av.Packet, w io.Writer) error {\n\tfirst := true\n\twBytes := 0\n\tpesIndex := 0\n\ttmpLen := byte(0)\n\tdataLen := byte(0)\n\n\tvar pes pesHeader\n\tdts := int64(p.TimeStamp) * int64(h264DefaultHZ)\n\tpts := dts\n\tpid := audioPID\n\tvar videoH av.VideoPacketHeader\n\tif p.IsVideo {\n\t\tpid = videoPID\n\t\tvideoH, _ = p.Header.(av.VideoPacketHeader)\n\t\tpts = dts + int64(videoH.CompositionTime())*int64(h264DefaultHZ)\n\t}\n\terr := pes.packet(p, pts, dts)\n\tif err != nil {\n\t\treturn err\n\t}\n\tpesHeaderLen := pes.len\n\tpacketBytesLen := len(p.Data) + int(pesHeaderLen)\n\n\tfor {\n\t\tif packetBytesLen <= 0 {\n\t\t\tbreak\n\t\t}\n\t\tif p.IsVideo {\n\t\t\tmuxer.videoCc++\n\t\t\tif muxer.videoCc > 0xf {\n\t\t\t\tmuxer.videoCc = 0\n\t\t\t}\n\t\t} else {\n\t\t\tmuxer.audioCc++\n\t\t\tif muxer.audioCc > 0xf {\n\t\t\t\tmuxer.audioCc = 0\n\t\t\t}\n\t\t}\n\n\t\ti := byte(0)\n\n\t\t//sync byte\n\t\tmuxer.tsPacket[i] = 0x47\n\t\ti++\n\n\t\t//error indicator, unit start indicator,ts priority,pid\n\t\tmuxer.tsPacket[i] = byte(pid >> 8) //pid high 5 bits\n\t\tif first {\n\t\t\tmuxer.tsPacket[i] = muxer.tsPacket[i] | 0x40 //unit start indicator\n\t\t}\n\t\ti++\n\n\t\t//pid low 8 bits\n\t\tmuxer.tsPacket[i] = byte(pid)\n\t\ti++\n\n\t\t//scram control, adaptation control, counter\n\t\tif p.IsVideo {\n\t\t\tmuxer.tsPacket[i] = 0x10 | byte(muxer.videoCc&0x0f)\n\t\t} else {\n\t\t\tmuxer.tsPacket[i] = 0x10 | byte(muxer.audioCc&0x0f)\n\t\t}\n\t\ti++\n\n\t\t//關鍵幀需要加pcr\n\t\tif first && p.IsVideo && videoH.IsKeyFrame() {\n\t\t\tmuxer.tsPacket[3] |= 0x20\n\t\t\tmuxer.tsPacket[i] = 7\n\t\t\ti++\n\t\t\tmuxer.tsPacket[i] = 0x50\n\t\t\ti++\n\t\t\tmuxer.writePcr(muxer.tsPacket[0:], i, dts)\n\t\t\ti += 6\n\t\t}\n\n\t\t//frame data\n\t\tif packetBytesLen >= tsDefaultDataLen {\n\t\t\tdataLen = tsDefaultDataLen\n\t\t\tif first {\n\t\t\t\tdataLen -= (i - 4)\n\t\t\t}\n\t\t} else {\n\t\t\tmuxer.tsPacket[3] |= 0x20 //have adaptation\n\t\t\tremainBytes := byte(0)\n\t\t\tdataLen = byte(packetBytesLen)\n\t\t\tif first {\n\t\t\t\tremainBytes = tsDefaultDataLen - dataLen - (i - 4)\n\t\t\t} else {\n\t\t\t\tremainBytes = tsDefaultDataLen - dataLen\n\t\t\t}\n\t\t\tmuxer.adaptationBufInit(muxer.tsPacket[i:], byte(remainBytes))\n\t\t\ti += remainBytes\n\t\t}\n\t\tif first && i < tsPacketLen && pesHeaderLen > 0 {\n\t\t\ttmpLen = tsPacketLen - i\n\t\t\tif pesHeaderLen <= tmpLen {\n\t\t\t\ttmpLen = pesHeaderLen\n\t\t\t}\n\t\t\tcopy(muxer.tsPacket[i:], pes.data[pesIndex:pesIndex+int(tmpLen)])\n\t\t\ti += tmpLen\n\t\t\tpacketBytesLen -= int(tmpLen)\n\t\t\tdataLen -= tmpLen\n\t\t\tpesHeaderLen -= tmpLen\n\t\t\tpesIndex += int(tmpLen)\n\t\t}\n\n\t\tif i < tsPacketLen {\n\t\t\ttmpLen = tsPacketLen - i\n\t\t\tif tmpLen <= dataLen {\n\t\t\t\tdataLen = tmpLen\n\t\t\t}\n\t\t\tcopy(muxer.tsPacket[i:], p.Data[wBytes:wBytes+int(dataLen)])\n\t\t\twBytes += int(dataLen)\n\t\t\tpacketBytesLen -= int(dataLen)\n\t\t}\n\t\tif w != nil {\n\t\t\tif _, err := w.Write(muxer.tsPacket[0:]); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t\tfirst = false\n\t}\n\n\treturn nil\n}\n\n//PAT return pat data\nfunc (muxer *Muxer) PAT() []byte {\n\ti := 0\n\tremainByte := 0\n\ttsHeader := []byte{0x47, 0x40, 0x00, 0x10, 0x00}\n\tpatHeader := []byte{0x00, 0xb0, 0x0d, 0x00, 0x01, 0xc1, 0x00, 0x00, 0x00, 0x01, 0xf0, 0x01}\n\n\tif muxer.patCc > 0xf {\n\t\tmuxer.patCc = 0\n\t}\n\ttsHeader[3] |= muxer.patCc & 0x0f\n\tmuxer.patCc++\n\n\tcopy(muxer.pat[i:], tsHeader)\n\ti += len(tsHeader)\n\n\tcopy(muxer.pat[i:], patHeader)\n\ti += len(patHeader)\n\n\tcrc32Value := GenCrc32(patHeader)\n\tmuxer.pat[i] = byte(crc32Value >> 24)\n\ti++\n\tmuxer.pat[i] = byte(crc32Value >> 16)\n\ti++\n\tmuxer.pat[i] = byte(crc32Value >> 8)\n\ti++\n\tmuxer.pat[i] = byte(crc32Value)\n\ti++\n\n\tremainByte = int(tsPacketLen - i)\n\tfor j := 0; j < remainByte; j++ {\n\t\tmuxer.pat[i+j] = 0xff\n\t}\n\n\treturn muxer.pat[0:]\n}\n\n// PMT return pmt data\nfunc (muxer *Muxer) PMT(soundFormat byte, hasVideo bool) []byte {\n\ti := int(0)\n\tj := int(0)\n\tvar progInfo []byte\n\tremainBytes := int(0)\n\ttsHeader := []byte{0x47, 0x50, 0x01, 0x10, 0x00}\n\tpmtHeader := []byte{0x02, 0xb0, 0xff, 0x00, 0x01, 0xc1, 0x00, 0x00, 0xe1, 0x00, 0xf0, 0x00}\n\tif !hasVideo {\n\t\tpmtHeader[9] = 0x01\n\t\tprogInfo = []byte{0x0f, 0xe1, 0x01, 0xf0, 0x00}\n\t} else {\n\t\tprogInfo = []byte{0x1b, 0xe1, 0x00, 0xf0, 0x00, //h264 or h265*\n\t\t\t0x0f, 0xe1, 0x01, 0xf0, 0x00, //mp3 or aac\n\t\t}\n\t}\n\tpmtHeader[2] = byte(len(progInfo) + 9 + 4)\n\n\tif muxer.pmtCc > 0xf {\n\t\tmuxer.pmtCc = 0\n\t}\n\ttsHeader[3] |= muxer.pmtCc & 0x0f\n\tmuxer.pmtCc++\n\n\tif soundFormat == 2 ||\n\t\tsoundFormat == 14 {\n\t\tif hasVideo {\n\t\t\tprogInfo[5] = 0x4\n\t\t} else {\n\t\t\tprogInfo[0] = 0x4\n\t\t}\n\t}\n\n\tcopy(muxer.pmt[i:], tsHeader)\n\ti += len(tsHeader)\n\n\tcopy(muxer.pmt[i:], pmtHeader)\n\ti += len(pmtHeader)\n\n\tcopy(muxer.pmt[i:], progInfo[0:])\n\ti += len(progInfo)\n\n\tcrc32Value := GenCrc32(muxer.pmt[5 : 5+len(pmtHeader)+len(progInfo)])\n\tmuxer.pmt[i] = byte(crc32Value >> 24)\n\ti++\n\tmuxer.pmt[i] = byte(crc32Value >> 16)\n\ti++\n\tmuxer.pmt[i] = byte(crc32Value >> 8)\n\ti++\n\tmuxer.pmt[i] = byte(crc32Value)\n\ti++\n\n\tremainBytes = int(tsPacketLen - i)\n\tfor j = 0; j < remainBytes; j++ {\n\t\tmuxer.pmt[i+j] = 0xff\n\t}\n\n\treturn muxer.pmt[0:]\n}\n\nfunc (muxer *Muxer) adaptationBufInit(src []byte, remainBytes byte) {\n\tsrc[0] = byte(remainBytes - 1)\n\tif remainBytes == 1 {\n\t} else {\n\t\tsrc[1] = 0x00\n\t\tfor i := 2; i < len(src); i++ {\n\t\t\tsrc[i] = 0xff\n\t\t}\n\t}\n\treturn\n}\n\nfunc (muxer *Muxer) writePcr(b []byte, i byte, pcr int64) error {\n\tb[i] = byte(pcr >> 25)\n\ti++\n\tb[i] = byte((pcr >> 17) & 0xff)\n\ti++\n\tb[i] = byte((pcr >> 9) & 0xff)\n\ti++\n\tb[i] = byte((pcr >> 1) & 0xff)\n\ti++\n\tb[i] = byte(((pcr & 0x1) << 7) | 0x7e)\n\ti++\n\tb[i] = 0x00\n\n\treturn nil\n}\n\ntype pesHeader struct {\n\tlen byte\n\tdata [tsPacketLen]byte\n}\n\n//pesPacket return pes packet\nfunc (header *pesHeader) packet(p *av.Packet, pts, dts int64) error {\n\t//PES header\n\ti := 0\n\theader.data[i] = 0x00\n\ti++\n\theader.data[i] = 0x00\n\ti++\n\theader.data[i] = 0x01\n\ti++\n\n\tsid := audioSID\n\tif p.IsVideo {\n\t\tsid = videoSID\n\t}\n\theader.data[i] = byte(sid)\n\ti++\n\n\tflag := 0x80\n\tptslen := 5\n\tdtslen := ptslen\n\theaderSize := ptslen\n\tif p.IsVideo && pts != dts {\n\t\tflag |= 0x40\n\t\theaderSize += 5 //add dts\n\t}\n\tsize := len(p.Data) + headerSize + 3\n\tif size > 0xffff {\n\t\tsize = 0\n\t}\n\theader.data[i] = byte(size >> 8)\n\ti++\n\theader.data[i] = byte(size)\n\ti++\n\n\theader.data[i] = 0x80\n\ti++\n\theader.data[i] = byte(flag)\n\ti++\n\theader.data[i] = byte(headerSize)\n\ti++\n\n\theader.writeTs(header.data[0:], i, flag>>6, pts)\n\ti += ptslen\n\tif p.IsVideo && pts != dts {\n\t\theader.writeTs(header.data[0:], i, 1, dts)\n\t\ti += dtslen\n\t}\n\n\theader.len = byte(i)\n\n\treturn nil\n}\n\nfunc (header *pesHeader) writeTs(src []byte, i int, fb int, ts int64) {\n\tval := uint32(0)\n\tif ts > 0x1ffffffff {\n\t\tts -= 0x1ffffffff\n\t}\n\tval = uint32(fb<<4) | ((uint32(ts>>30) & 0x07) << 1) | 1\n\tsrc[i] = byte(val)\n\ti++\n\n\tval = ((uint32(ts>>15) & 0x7fff) << 1) | 1\n\tsrc[i] = byte(val >> 8)\n\ti++\n\tsrc[i] = byte(val)\n\ti++\n\n\tval = (uint32(ts&0x7fff) << 1) | 1\n\tsrc[i] = byte(val >> 8)\n\ti++\n\tsrc[i] = byte(val)\n}","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"scheduler","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"scheduler主要是來調度任務，那麼主要是哪些任務呢？主要是那些普通api無法立即給結果的任務。比如：我們視頻網站需要一些視頻審覈、數據恢復的需求。這時候，我們需要做一些short delay，用戶看不到，但後臺還是存在的。這就需要scheduler異步處理。還比如有些週期性的任務。在Scheduler中，還存在Timer，定時器主要用來作定時處理task的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，我們的架構圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ea/ea2504ab42a76f615c4fbe151aab2776.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在本小節中，我們採用runner的生產、消費者模式實現。具體代碼如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"package taskrunner\n\nimport (\n)\n\ntype Runner struct {\n Controller controlChan\n Error controlChan\n Data dataChan\n dataSize int\n longLived bool\n Dispatcher fn \n Executor fn\n}\n\nfunc NewRunner(size int, longlived bool, d fn, e fn) *Runner {\n return &Runner {\n  Controller: make(chan string, 1),\n  Error: make(chan string, 1),\n  Data: make(chan interface{}, size),\n  longLived: longlived,\n  dataSize: size,\n  Dispatcher: d,\n  Executor: e,\n }\n}\n\nfunc (r *Runner) startDispatch() {\n defer func() {\n  if !r.longLived {\n   close(r.Controller)\n   close(r.Data)\n   close(r.Error)\n  }\n }()\n\n for {\n  select {\n  case c :== cl.concurrentConn {\n  log.Printf(\"Reached the rate limitation.\")\n  return false\n }\n\n cl.bucket  /etc/timezone\n\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n        g++ \\\n        ca-certificates \\\n        wget && \\\n    rm -rf /var/lib/apt/lists/*\n\nENV GOLANG_VERSION 1.15.1\nRUN wget -nv -O - https://studygolang.com/dl/golang/go1.15.1.linux-amd64.tar.gz \\\n     | tar -C /usr/local -xz\n\n\nENV GOPROXY=https://goproxy.cn,direct\nENV GO111MODULE=on\nENV GOPATH /go\nENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH\n\nWORKDIR /go/src\nCOPY . .\nWORKDIR /go/src/video-service\nRUN  sed -i \"/runmode/crunmode=pro\" /go/src/video-service/conf/app.conf\nRUN export CGO_LDFLAGS_ALLOW='-Wl,--unresolved-symbols=ignore-in-object-files' && \\\n    go install -ldflags=\"-s -w\" -v /go/src/video-service\n\nFROM ubuntu:16.04\nWORKDIR /video-service\n\nRUN mkdir -p log\nCOPY --from=build /go/bin/video-service /video-service\nCMD [\"./video-service\"]\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"k8s部署服務的腳本很簡單，通過簡單的yml或json格式的數據調用k8s本身的api服務，即可完成k8s對服務的部署。接下來，補充部署腳本：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n  labels:\n    app: video-service\n  name: video-service\n  namespace: system-server\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: video-service\n  template:\n    metadata:\n      labels:\n        app: video-service\n    spec:\n      containers:\n        - image: {{ cluster_cfg['cluster']['docker-registry']['prefix'] }}video-service\n          imagePullPolicy: Always\n          name: video-service\n          ports:\n            - containerPort: 1000\n          #livenessProbe:\n            #httpGet:\n              #path: /api/v1/healthz\n              #port: 1000\n              #scheme: HTTP\n            #initialDelaySeconds: 15\n            #periodSeconds: 10\n            #timeoutSeconds: 3\n            #failureThreshold: 5\n          volumeMounts:\n            - name: video-service-config\n              mountPath: /video-service/conf\n      volumes:\n        - name: video-service-config\n          configMap:\n            name: video-service-config\n      nodeSelector:\n        video-service: \"true\"\n      restartPolicy: Always\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"執行編譯打包後，部署腳本：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"sh build/build.sh\nkubectl create -f deploy.yml","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏使用K8s部署，k8s的主要好處就是，在執行命令後，會自動給我們創建底層容器pod，並且管理這些容器，我們來看看服務啓動的情況：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"tess@cm001:~$ kubectl describe po video-service-646ccc4c5c-qrpgp -n system-server \nName: video-service-646ccc4c5c-qrpgp\nNamespace: system-server\nPriority: 0\nNode: cm001/10.11.32.21\nStart Time: Wed, 21 Apr 2021 15:38:48 +0800\nLabels: app=video-service\n pod-template-hash=646ccc4c5c\nAnnotations: cni.projectcalico.org/podIP: 20.247.87.138/32\nStatus: Running\nIP: 20.247.87.138\nControlled By: ReplicaSet/video-service-646ccc4c5c\nContainers:\n video-service:\n Container ID: docker://f284ceac649f4e1a29ac77cdd425ccc852caf67cc9c133144a7d1c8747e32aea\n Image: minicub/video-service\n Image ID: docker-pullable://minicub/video-service@sha256:3036927b55c6be9efc4cf34d6ad04aba093e04b49807c18ce0f7a418b2577bf4\n Port: 1000/TCP\n Host Port: 0/TCP\n State: Running\n Started: Wed, 28 Apr 2021 16:01:15 +0800\n Ready: True\n Restart Count: 1\n Environment: \n Mounts:\n /video-service/conf from video-service-config (rw)\n /var/run/secrets/kubernetes.io/serviceaccount from default-token-62wgr (ro)\nConditions:\n Type Status\n Initialized True \n Ready True \n ContainersReady True \n PodScheduled True \nVolumes:\n video-service-config:\n Type: ConfigMap (a volume populated by a ConfigMap)\n Name: video-service-config\n Optional: false\n default-token-62wgr:\n Type: Secret (a volume populated by a Secret)\n SecretName: default-token-62wgr\n Optional: false\nQoS Class: BestEffort\nNode-Selectors: video-service=true\nTolerations: node.kubernetes.io/not-ready:NoExecute for 300s\n node.kubernetes.io/unreachable:NoExecute for 300s\nEvents: ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏使用K8s部署到機器上。部署後的服務訪問地址：","attrs":{}},{"type":"link","attrs":{"href":"http://10.11.32.21:1000","title":"","type":null},"content":[{"type":"text","text":"http://10.11.32.21:1000","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，我們打開網址，訪問視頻，我們可以看到一些界面效果：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cb/cb5079f519f5e6660c297f01fd9b7a00.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}