FlatBuffers是google最新針對遊戲開發退出的高性能的跨平臺序列化工具,目前已經支持C++, C#, Go, Java, JavaScript, PHP, and Python (C和Ruby正在支持中),相對於json和Protocol Buffers,FlatBuffers在序列化和反序列化方面表現更爲優異,而且需要的資源更少,更適合大部分移動應用的使用場景。
除了高性能和低內存消耗的特點,FlatBuffers還有其他一些優勢,官方的總結說明如下:
- 不用解析也能對數據進行訪問
FlatBuffers使用二進制結構化的方法來存儲數據,可以不用先對整段數據進行解析就可以直接在二進制結構中訪問到指定成員的信息。並且FlatBuffers還支持存儲數據結構的前後兼容。 - 內存佔用小,運行效率高
FlatBuffers使用ByteBuffer進行數據存儲,在FlatBuffers的序列化和反序列化過程中不會額外佔用其他內存空間,FlatBuffers讀取數據的速度和直接從內存讀取原始數據相當,僅僅多了一個相對尋址的耗時。同時,數據存儲在ByteBuffer中還能夠比較容易地支持內存映射(mmap)和流式讀寫,進一步降低對內存的消耗。 - 靈活
FlatBuffers支持選擇性地寫入數據成員,這不僅爲某一個數據結構在應用的不同版本之間提供了兼容性,同時還能使程序員靈活地選擇是否寫入某些字段及靈活地設計傳輸的數據結構。 - 體積小,集成成本低
項目中集成FlatBuffers只需要很少的額外代碼。 - 強類型約束
如果數據結構不符合FlatBufferss文件的描述,那麼會在編譯期間就發現問題,而不會在運行時才暴露問題。 - 使用方便
可以兼容json等其他格式的解析。 - 跨平臺
FlatBuffers和Protocol Buffers是比較相似的,但是FlatBuffers不需要在讀取成員變量之前必須將數據完全解析成對象,因爲它所有信息的讀取都是在對應的ByteBuffer中進行的,少了這些解析時必須爲對象和成員變量分配的內存空間,就降低了解析過程中的內存消耗。json相對於FlatBuffers來說可讀性更好,但是缺點也是明顯的,那就是它的性能太低了。JSON作爲數據交換格式,被廣泛用戶各種動態語言之間(當然也包括靜態語言)。它的優點是易於理解(可讀性好),同時它的最大的缺點那就是解析時的性能問題了。而且因爲它的動態類型特點,你的代碼可能還需要多寫好多類型、數據檢查邏輯。
下文簡介FBS的使用
FlatBuffers使用項目的整體流程:
在使用 FlatBuffers 時,我們需要以特殊的格式定義我們的結構化數據,保存爲 .fbs 文件。 FlatBuffers 項目爲我們提供了編譯器,可用於將 .fbs 文件編譯爲Java文件,C++文件等,以用於我們的項目。 FlatBuffers 編譯器在我們的開發機,比如Ubuntu,Mac上運行。這些源代碼文件是基於 FlatBuffers 提供的Java庫生成的,同時我們也需要利用這個Java庫的一些接口來序列化或解析數據。
我們將 FlatBuffers 編譯器生成的Java文件及 FlatBuffers 的Java庫導入我們的項目,就可以用 FlatBuffers 來對我們的結構化數據執行序列化和反序列化了。
下載、編譯FlatBuffers編譯器
在https://github.com/google/flatbuffers/releases 獲取官方發佈的打包好的版本:
$ git clone https://github.com/google/flatbuffers.git
Cloning into 'flatbuffers'...
remote: Counting objects: 7340, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 7340 (delta 16), reused 0 (delta 0), pack-reused 7290
Receiving objects: 100% (7340/7340), 3.64 MiB | 115.00 KiB/s, done.
Resolving deltas: 100% (4692/4692), done.
Checking connectivity... done.
下載代碼之後,我們需要用cmake工具來爲flatbuffers生成Makefile文件並編譯:
$ cd flatbuffers/
$ cmake CMakeLists.txt
-- The C compiler identification is AppleClang 7.3.0.7030031
-- The CXX compiler identification is AppleClang 7.3.0.7030031
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/netease/Projects/OpenSource/flatbuffers
$ make && make install
安裝之後執行如下命令以確認已經裝好:
$ flatc --version
flatc version 1.4.0 (Dec 7 2016)
flatc沒有爲我們提供 –help 選項,不過加了錯誤的參數時這個工具會爲我們展示詳細的用法:
$ flatc --help
flatc: unknown commandline argument: --help
usage: flatc [OPTION]... FILE... [-- FILE...]
--binary -b Generate wire format binaries for any data definitions.
--json -t Generate text output for any data definitions.
--cpp -c Generate C++ headers for tables/structs.
--go -g Generate Go files for tables/structs.
--java -j Generate Java classes for tables/structs.
--js -s Generate JavaScript code for tables/structs.
--csharp -n Generate C# classes for tables/structs.
--python -p Generate Python files for tables/structs.
--php Generate PHP files for tables/structs.
-o PATH Prefix PATH to all generated files.
-I PATH Search for includes in the specified path.
-M Print make rules for generated files.
--version Print the version number of flatc and exit.
--strict-json Strict JSON: field names must be / will be quoted,
no trailing commas in tables/vectors.
--allow-non-utf8 Pass non-UTF-8 input through parser and emit nonstandard
\x escapes in JSON. (Default is to raise parse error on
non-UTF-8 input.)
--defaults-json Output fields whose value is the default when
writing JSON
--unknown-json Allow fields in JSON that are not defined in the
schema. These fields will be discared when generating
binaries.
--no-prefix Don't prefix enum values with the enum type in C++.
--scoped-enums Use C++11 style scoped and strongly typed enums.
also implies --no-prefix.
--gen-includes (deprecated), this is the default behavior.
If the original behavior is required (no include
statements) use --no-includes.
--no-includes Don't generate include statements for included
schemas the generated file depends on (C++).
--gen-mutable Generate accessors that can mutate buffers in-place.
--gen-onefile Generate single output file for C#.
--gen-name-strings Generate type name functions for C++.
--escape-proto-ids Disable appending '_' in namespaces names.
--gen-object-api Generate an additional object-based API.
--cpp-ptr-type T Set object API pointer type (default std::unique_ptr)
--raw-binary Allow binaries without file_indentifier to be read.
This may crash flatc given a mismatched schema.
--proto Input is a .proto, translate to .fbs.
--schema Serialize schemas instead of JSON (use with -b)
--conform FILE Specify a schema the following schemas should be
an evolution of. Gives errors if not.
--conform-includes Include path for the schema given with --conform
PATH
FILEs may be schemas, or JSON files (conforming to preceding schema)
FILEs after the -- must be binary flatbuffer format files.
Output files are named using the base file name of the input,
and written to the current directory or the path given by -o.
example: flatc -c -b schema1.fbs schema2.fbs data.json
創建 .fbs 文件
flatc支持將爲 Protocol Buffers 編寫的 .proto 文件轉換爲 .fbs 文件,如:
$ ls
addressbook.proto
$ flatc --proto addressbook.proto
$ ls -l
total 16
-rw-r--r-- 1 netease staff 431 12 7 17:21 addressbook.fbs
-rw-r--r--@ 1 netease staff 486 12 1 15:18 addressbook.proto
Protocol Buffers消息文件中的一些寫法, FlatBuffers 編譯器還不能很好的支持,如option java_package,option java_outer_classname,和嵌套類。這裏我們基於 FlatBuffers 編譯器轉換的 .proto 文件來獲得我們的 .fbs 文件:
// Generated from addressbook.proto
namespace com.example.tutorial;
enum PhoneType : int {
MOBILE = 0,
HOME = 1,
WORK = 2,
}
namespace com.example.tutorial;
table Person {
name:string (required);
id:int;
email:string;
phone:[com.example.tutorial._Person.PhoneNumber];
}
namespace com.example.tutorial._Person;
table PhoneNumber {
number:string (required);
type:int;
}
namespace com.example.tutorial;
table AddressBook {
person:[com.example.tutorial.Person];
}
root_type AddressBook;
編譯.fbs文件
可以通過如下命令編譯 .fbs 文件:
$ flatc --java -o out addressbook.fbs
–java用於指定編譯的目標編程語言。-o 參數則用於指定輸出文件的路徑,如過沒有提供則將當前目錄用作輸出目錄。 FlatBuffers 編譯器按照爲不同的數據結構聲明的namespace生成目錄結構。對於上面的例子,會生成如下的這些文件:
$ find out
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}span.s1 {font-variant-ligatures: no-common-ligatures}
$ find out/
out/
out//com
out//com/example
out//com/example/tutorial
out//com/example/tutorial/_Person
out//com/example/tutorial/_Person/PhoneNumber.java
out//com/example/tutorial/AddressBook.java
out//com/example/tutorial/Person.java
out//com/example/tutorial/PhoneType.java
在Android項目中使用FlatBuffers
我們將前面由 .fbs 文件生成的Java文件拷貝到我們的項目中。我們前面提到的, FlatBuffers 的Java庫比較薄,當前並沒有發不到jcenter這樣的maven倉庫中,因而我們需要將這部分代碼也拷貝到我們的額項目中。 FlatBuffers 的Java庫在其repo倉庫的 java 目錄下。引入這些文件之後,我們的代碼結構如下
添加訪問 FlatBuffers 的類:
package com.netease.volleydemo;
import com.example.tutorial.AddressBook;
import com.example.tutorial.Person;
import com.example.tutorial._Person.PhoneNumber;
import com.google.flatbuffers.FlatBufferBuilder;
import java.nio.ByteBuffer;
/**
* Created by hanpfei0306 on 16-12-5.
*/
public class AddressBookFlatBuffers {
public static ByteBuffer encodeTest(String[] names) {
FlatBufferBuilder builder = new FlatBufferBuilder(0);
int[] personOffsets = new int[names.length];
for (int i = 0; i < names.length; ++ i) {
int name = builder.createString(names[i]);
int email = builder.createString("[email protected]");
int number1 = builder.createString("0157-23443276");
int type1 = 1;
int phoneNumber1 = PhoneNumber.createPhoneNumber(builder, number1, type1);
int number2 = builder.createString("136183667387");
int type2 = 0;
int phoneNumber2 = PhoneNumber.createPhoneNumber(builder, number2, type2);
int[] phoneNubers = new int[2];
phoneNubers[0] = phoneNumber1;
phoneNubers[1] = phoneNumber2;
int phoneNumbersPos = Person.createPhoneVector(builder, phoneNubers);
int person = Person.createPerson(builder, name, 13958235, email, phoneNumbersPos);
personOffsets[i] = person;
}
int persons = AddressBook.createPersonVector(builder, personOffsets);
AddressBook.startAddressBook(builder);
AddressBook.addPerson(builder, persons);
int eab = AddressBook.endAddressBook(builder);
builder.finish(eab);
ByteBuffer buf = builder.dataBuffer();
return buf;
}
public static ByteBuffer encodeTest(String[] names, int times) {
for (int i = 0; i < times - 1; ++ i) {
encodeTest(names);
}
return encodeTest(names);
}
public static AddressBook decodeTest(ByteBuffer byteBuffer) {
AddressBook addressBook = null;
addressBook = AddressBook.getRootAsAddressBook(byteBuffer);
return addressBook;
}
public static AddressBook decodeTest(ByteBuffer byteBuffer, int times) {
AddressBook addressBook = null;
for (int i = 0; i < times; ++ i) {
addressBook = decodeTest(byteBuffer);
}
return addressBook;
}
}
使用 flatbuf-gradle-plugin
我們有開發一個 FlatBuffers 的gradle插件,以方便開發 。這個插件的設計有參考Google的protobuf-gradle-plugin,功能與用法也與protobuf-gradle-plugin類似。在這個項目中,我們也有爲 FlatBuffers 的Java庫創建一個module。
參考:
1、http://www.jianshu.com/p/3bac6bc80db7
2、https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/