文章目錄

Android A/B System系列

Makefile

先說點無關的，Makefile這個東西還是要耐心點看，不要浮躁。這裏推薦一篇適合小白閱讀的講解：
Make 命令教程
另外複雜的邏輯，讀起來很繞，可以在Makefile裏面把變量打印出來看看：
makefile 打印變量的值

這一節的目標是搞清楚：執行make otapackage這個命令的時候，發生了什麼？
Makefile的路徑：build/make/core/Makefile

下面開始分析：
可以看到.PHONY: otapackage，它依賴於$(INTERNAL_OTA_PACKAGE_TARGET)。

接下來看$(INTERNAL_OTA_PACKAGE_TARGET)，先來看它生成的文件名字是什麼：

PRODUCT_OUT := $(TARGET_PRODUCT_OUT_ROOT)/$(TARGET_DEVICE)
- TARGET_PRODUCT_OUT_ROOT) := $(TARGET_OUT_ROOT)/product
  - TARGET_OUT_ROOT := $(OUT_DIR)/target
    - OUT_DIR := out
- TARGET_DEVICE := $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_DEVICE)
  - INTERNAL_PRODUCT := $(call resolve-short-product-name, $(TARGET_PRODUCT))
name = $(TARGET_PRODUCT)-ota-$(FILE_NAME_TAG)
- FILE_NAME_TAG := eng.$(USER)

如上是依賴關係，可以在build/make裏面搜索到，那麼總結起來，最終生成的文件是：

out/target/product/$(TARGET_OUT_ROOT)/$(TARGET_PRODUCT)-ota-eng.$(USER).zip

假設$(TARGET_OUT_ROOT)爲tardis，$USER爲root，那麼最終的結果爲：

 out/target/product/tardis/tardis-ota-eng.root.zip

接下來看$(INTERNAL_OTA_PACKAGE_TARGET)的依賴關係：

$(BRILLO_UPDATE_PAYLOAD)（for a/b system，這個之後再講）
$(BUILT_TARGET_FILES_PACKAGE)
$(HOST_OUT)
$(KEY_CERT_PAIR)

/build/core/Makefile

1. $(HOST_OUT)

HOST_OUT := $(HOST_OUT_ROOT)/$(HOST_OS)-$(HOST_PREBUILT_ARCH)

HOST_OUT_ROOT := $(OUT_DIR)/host
HOST_OS := linux
HOST_PREBUILT_ARCH := x86

$(HOST_OUT)實際爲out/host/linux-x86

2. $(KEY_CERT_PAIR)

KEY_CERT_PAIR := $(DEFAULT_KEY_CERT_PAIR)

DEFAULT_KEY_CERT_PAIR := $(DEFAULT_SYSTEM_DEV_CERTIFICATE)

所以，$(KEY_CERT_PAIR)默認值爲build/target/product/security/testkey

3. $(BUILT_TARGET_FILES_PACKAGE)

BUILT_TARGET_FILES_PACKAGE:= $(intermediates)/$(name).zip

intermediates := $(call intermediates-dir-for,PACKAGING,target_files)
intermediates-dir-for是函數，後面的是參數，這句話的意思是尋找out/target/product/$(TARGET_PRODUCT)/obj/中PACKAGING類型文件中以target_files開頭的目錄。
name
前面分析過，name是$(TARGET_PRODUCT)-ota-eng.$(USER)
假設$(TARGET_OUT_ROOT)爲tardis，$USER爲root，那麼name就是tradis-ota-eng.root
它的作用是生成了最終包的中間包。

Summary
總結來看，make otapackage這個命令，最終執行的是如下的命令，也就是生成完整包的命令。

./build/tools/releasetools/ota_from_target_files -v \
    -p out/host/linux-x86 \
    -k \build/target/product/security/testkey \
	out/target/product/tardis/obj/PACKAGING/target_files_intermediates/tardis-ota.eng.root.zip \
	out/target/product/tardis/tardis-ota.eng.root.zip

ota_from_target_files.py

ota_from_target_files.py喫一些參數，我這裏列舉一些常用的：

-p：Search for binaries run by this script
-i：Generate an incremental OTA using the given target-files zip as the starting build.
-k：Key to use to sign the package
-v：Show command lines being executed

這些命令的傳遞關係如圖所示：

ota_from_target_files(WriteABOTAPackageWithBrilloScript())
- brillo_update_payload
  - delta_generator(generate_delta_main.cc)

其中，ota_from_target_file將命令，參數翻譯成brillo_update_payload [cmd] [args]
brillo_update_payload包括如下命令：（每個命令後面帶參數，有很多，這裏不再贅述）

generate --xxxxx
hash
sign
properities

OTA Package Format

protobuf

protobuf是什麼？

xxx.proto文件表示一種結構化的存儲方式，它有如下特點：

smaller
faster
simpler

Google 官方解釋：Protocol Buffers

語法

For example：

message相當於jave的class，C的struct，message裏面開頭的變量有三種：

required：a well-formed message must have exactly one of this field.
optional：a well-formed message can have zero or one of this field (but not more than one).
repeated：this field can be repeated any number of times (including zero) in a well-formed message. The order of the repeated values will be preserved.

update_metadata.proto

payload.bin的結構在update_metadata.proto中看：/system/update_engine/update_metadata.proto

下面具體來看這個文件，其實在這個文件中已經有註釋：（struct delta_update_file）

除了一些基礎信息，內容如圖所示：

Header: magic/version/manifest_size/metadata_signature_size
protobuf:
- manifest[]:存儲blob[]的組織信息，在升級的時候告訴update_engine如何分解這些數據
- metadata_signature_message
blobs[]:數據
signature:簽名

20 // struct delta_update_file {
21 //   char magic[4] = "CrAU";
22 //   uint64 file_format_version;
23 //   uint64 manifest_size;  // Size of protobuf DeltaArchiveManifest
24 //
25 //   // Only present if format_version > 1:
26 //   uint32 metadata_signature_size;
27 //
28 //   // The Bzip2 compressed DeltaArchiveManifest
29 //   char manifest[];
30 //
31 //   // The signature of the metadata (from the beginning of the payload up to
32 //   // this location, not including the signature itself). This is a serialized
33 //   // Signatures message.
34 //   char medatada_signature_message[metadata_signature_size];
35 //
36 //   // Data blobs for files, no specific format. The specific offset
37 //   // and length of each data blob is recorded in the DeltaArchiveManifest.
38 //   struct {
39 //     char data[];
40 //   } blobs[];
41 //
42 //   // These two are not signed:
43 //   uint64 payload_signatures_message_size;
44 //   char payload_signatures_message[];
45 //
46 // };

下面這幅圖看起來更加清晰，簡而言之，update_metadata.proto這個文件中的7個message就是用來組裝DeltaArchiveManifest。DeltaArchiveManifest可以理解爲升級包的安裝說明書。這7個message分別是：

Extent
Signatures
PartitionInfo
ImageInfo
InstallOperation：每一個Partition分區都有一個InstallOperation數組
PartitionUpdate：Describes the update to apply to a single partition.
DeltaArchiveManifest

DeltaArchiveManifest
下面來看DeltaArchiveManifest的內容：

InstallOperation
block_size（差分包，文件系統的block size必須是4096）
signatures_offset/signtures_size
PartitionInfo
ImageInfo
PartitionUpdate
minor_version
max_timestamp

253 message DeltaArchiveManifest {
254   // Only present in major version = 1. List of install operations for the
255   // kernel and rootfs partitions. For major version = 2 see the |partitions|
256   // field.
257   repeated InstallOperation install_operations = 1;
258   repeated InstallOperation kernel_install_operations = 2;
259 
260   // (At time of writing) usually 4096
261   optional uint32 block_size = 3 [default = 4096];
262 
263   // If signatures are present, the offset into the blobs, generally
264   // tacked onto the end of the file, and the length. We use an offset
265   // rather than a bool to allow for more flexibility in future file formats.
266   // If either is absent, it means signatures aren't supported in this
267   // file.
268   optional uint64 signatures_offset = 4;
269   optional uint64 signatures_size = 5;
270 
271   // Only present in major version = 1. Partition metadata used to validate the
272   // update. For major version = 2 see the |partitions| field.
273   optional PartitionInfo old_kernel_info = 6;
274   optional PartitionInfo new_kernel_info = 7;
275   optional PartitionInfo old_rootfs_info = 8;
276   optional PartitionInfo new_rootfs_info = 9;
277 
278   // old_image_info will only be present for delta images.
279   optional ImageInfo old_image_info = 10;
280 
281   optional ImageInfo new_image_info = 11;
282 
283   // The minor version, also referred as "delta version", of the payload.
284   optional uint32 minor_version = 12 [default = 0];
285 
286   // Only present in major version >= 2. List of partitions that will be
287   // updated, in the order they will be updated. This field replaces the
288   // |install_operations|, |kernel_install_operations| and the
289   // |{old,new}_{kernel,rootfs}_info| fields used in major version = 1. This
290   // array can have more than two partitions if needed, and they are identified
291   // by the partition name.
292   repeated PartitionUpdate partitions = 13;
293 
294   // The maximum timestamp of the OS allowed to apply this payload.
295   // Can be used to prevent downgrading the OS.
296   optional int64 max_timestamp = 14;
297 }

InstallOperation

Type
- REPLACE：Replace destination extents with attached data.
- REPLACE_BZ：Replace destination extents with attached bzipped data.
- REPLACE_XZ：Replace destination extents with attached xz data.
- MOVE：Move source extents to destination extents.
- BSDIFF：The data is a bsdiff binary diff.
- SOURCE_COPY：Copy from source to target partition without data.（適用於source和target一樣的情況）
- SOURCE_BSDIFF：
  - Read from source partition
  - Read bsdiff binary patch from update.zip
  - Write (a+b) results to target partition
- ZERO：Write zeros in the destination.
- DISCARD：Discard the destination blocks， reading as undefined.
- BROTLI_BSDIFF：Like SOURCE_BSDIFF, but compressed with brotli.
- PUFFDIFF：The data is in puffdiff format.
data_offset/data_length
Extent
- start_block
- num_block
data_sha256_hash/src_sha256_hash

154 message InstallOperation {
155   enum Type {
156     REPLACE = 0;  // Replace destination extents w/ attached data
157     REPLACE_BZ = 1;  // Replace destination extents w/ attached bzipped data
158     MOVE = 2;  // Move source extents to destination extents
159     BSDIFF = 3;  // The data is a bsdiff binary diff
160 
161     // On minor version 2 or newer, these operations are supported:
162     SOURCE_COPY = 4; // Copy from source to target partition
163     SOURCE_BSDIFF = 5; // Like BSDIFF, but read from source partition
164 
165     // On minor version 3 or newer and on major version 2 or newer, these
166     // operations are supported:
167     REPLACE_XZ = 8; // Replace destination extents w/ attached xz data.
168 
169     // On minor version 4 or newer, these operations are supported:
170     ZERO = 6;  // Write zeros in the destination.
171     DISCARD = 7;  // Discard the destination blocks, reading as undefined.
172     BROTLI_BSDIFF = 10;  // Like SOURCE_BSDIFF, but compressed with brotli.
173 
174     // On minor version 5 or newer, these operations are supported:
175     PUFFDIFF = 9;  // The data is in puffdiff format.
176   }
177   required Type type = 1;
178   // The offset into the delta file (after the protobuf)
179   // where the data (if any) is stored
180   optional uint32 data_offset = 2;
181   // The length of the data in the delta file
182   optional uint32 data_length = 3;
183 
184   // Ordered list of extents that are read from (if any) and written to.
185   repeated Extent src_extents = 4;
186   // Byte length of src, equal to the number of blocks in src_extents *
187   // block_size. It is used for BSDIFF and SOURCE_BSDIFF, because we need to
188   // pass that external program the number of bytes to read from the blocks we
189   // pass it.  This is not used in any other operation.
190   optional uint64 src_length = 5;
191 
192   repeated Extent dst_extents = 6;
193   // Byte length of dst, equal to the number of blocks in dst_extents *
194   // block_size. Used for BSDIFF and SOURCE_BSDIFF, but not in any other
195   // operation.
196   optional uint64 dst_length = 7;
197 
198   // Optional SHA 256 hash of the blob associated with this operation.
199   // This is used as a primary validation for http-based downloads and
200   // as a defense-in-depth validation for https-based downloads. If
201   // the operation doesn't refer to any blob, this field will have
202   // zero bytes.
203   optional bytes data_sha256_hash = 8;
204 
205   // Indicates the SHA 256 hash of the source data referenced in src_extents at
206   // the time of applying the operation. If present, the update_engine daemon
207   // MUST read and verify the source data before applying the operation.
208   optional bytes src_sha256_hash = 9;
209 }

PartitionInfo

size
hash

130 message PartitionInfo {
131   optional uint64 size = 1;
132   optional bytes hash = 2;
133 }

Extent

117 message Extent {
118   optional uint64 start_block = 1;
119   optional uint64 num_blocks = 2;
120 }

For example, a file stored in blocks 9, 10, 11, 2, 18, 12 (in that order) would be stored in extents { {9, 3}, {2, 1}, {18, 1}, {12, 1} }
{9,3}的意思是，start_block = 9，連續的有三個(9,10,11)，num_block = 3

ImageInfo

141 message ImageInfo {
142   optional string board = 1;
143   optional string key = 2;
144   optional string channel = 3;
145   optional string version = 4;
146 
147   // If these values aren't present, they should be assumed to match
148   // the equivalent value above. They are normally only different for
149   // special image types such as nplusone images.
150   optional string build_channel = 5;
151   optional string build_version = 6;
152 }

PartitionUpdate

partition_name
run_postinstall
postinstall_path
filesystem_type
old_partition_info/new_partition_info（PartitionInfo）：全包升級，只有new_partition_info有內容；差分升級，有- - old_partition_info/new_partition_info都有內容
operations（InstallOperation）

211 // Describes the update to apply to a single partition.
212 message PartitionUpdate {
213   // A platform-specific name to identify the partition set being updated. For
214   // example, in Chrome OS this could be "ROOT" or "KERNEL".
215   required string partition_name = 1;
216 
217   // Whether this partition carries a filesystem with post-install program that
218   // must be run to finalize the update process. See also |postinstall_path| and
219   // |filesystem_type|.
220   optional bool run_postinstall = 2;
221 
222   // The path of the executable program to run during the post-install step,
223   // relative to the root of this filesystem. If not set, the default "postinst"
224   // will be used. This setting is only used when |run_postinstall| is set and
225   // true.
226   optional string postinstall_path = 3;
227 
228   // The filesystem type as passed to the mount(2) syscall when mounting the new
229   // filesystem to run the post-install program. If not set, a fixed list of
230   // filesystems will be attempted. This setting is only used if
231   // |run_postinstall| is set and true.
232   optional string filesystem_type = 4;
233 
234   // If present, a list of signatures of the new_partition_info.hash signed with
235   // different keys. If the update_engine daemon requires vendor-signed images
236   // and has its public key installed, one of the signatures should be valid
237   // for /postinstall to run.
238   repeated Signatures.Signature new_partition_signature = 5;
239 
240   optional PartitionInfo old_partition_info = 6;
241   optional PartitionInfo new_partition_info = 7;
242 
243   // The list of operations to be performed to apply this PartitionUpdate. The
244   // associated operation blobs (in operations[i].data_offset, data_length)
245   // should be stored contiguously and in the same order.
246   repeated InstallOperation operations = 8;
247 
248   // Whether a failure in the postinstall step for this partition should be
249   // ignored.
250   optional bool postinstall_optional = 9;
251 }

Signatures

122 message Signatures {
123   message Signature {
124     optional uint32 version = 1;
125     optional bytes data = 2;
126   }
127   repeated Signature signatures = 1;
128 }

Generate Flow

generate_delta_main.cc
關於這一部分，放在system/engine/payload_generator 文件夾下。先從generate_delta_main.cc開始看，剛開始看的時候，完全不知道從哪裏分析，好在log還很完整，所以可以通過log來學習，generate_delta_main.cc做了如下事情：

將flags轉化爲PayloadGenerationConfig
Generate：GenerateUpdatePayloadFile()
Hash：CalculateHashForSigning()
Sign：AddSignatureToPayload()

delta_diff_generator.cc
GenerateUpdatePayloadFile()

全包：FullUpdateGenerator()，請看Full OTA Package
差分：ABGenerator()，請看Delta OTA Package

log會打印出：

Generating full update(or delta update)
Partition name: xxx
Partition size: xxxx
Block count: xxxx

For example：

Generate

Full OTA Package

從FullUpdateGenerator()往下看：
/system/update_engine/payload_generator/full_update_generator.cc
幾個重要的變量：

partition_blocks：partition有多少個block
chunk_blocks：一個chunk有多少個block
num_chunks：partition有多少個chunk

num_chunks=(partition_blocks + chunk_blocks - 1)/chunk_blocks

For example:
boot.img partition size是16777216，block size = 4096，那麼：

partition_blocks：4096
chunk_blocks：512
num_chunks：(4096+512-1)/512=8

FullUpdateGenerator::GenerateOperations()把每個partition，分成了多個chunk，一個thread處理一個chunk中的數據（thread_pool的大小與CPU有關，這裏max_threads=16）。

/system/update_engine/payload_generator/blob_file_writer.cc

StoreBlob

/system/update_engine/payload_generator/payload_file.cc
全部的partition都寫完以後，開始Write payload file：PayloadFile::WritePayload()

Writing final delta file header
Writing final delta file protobuf
Writing final delta file data blobs

到這裏generate就結束了。

Delta OTA Package

system/update_engine/payload_generator/ab_generator.cc

GenerateOperations()
- DeltaReadPartition()
- FragmentOperation()
- SortOperationByDestination(aops)
- MergeOperations()
- AddSourceHash()

bool ABGenerator::GenerateOperations(
    const PayloadGenerationConfig& config,
    const PartitionConfig& old_part,
    const PartitionConfig& new_part,
    BlobFileWriter* blob_file,
    vector<AnnotatedOperation>* aops) {
  TEST_AND_RETURN_FALSE(old_part.name == new_part.name);

  ssize_t hard_chunk_blocks = (config.hard_chunk_size == -1 ? -1 :
                               config.hard_chunk_size / config.block_size);
  size_t soft_chunk_blocks = config.soft_chunk_size / config.block_size;

  aops->clear();
  TEST_AND_RETURN_FALSE(diff_utils::DeltaReadPartition(aops,
                                                       old_part,
                                                       new_part,
                                                       hard_chunk_blocks,
                                                       soft_chunk_blocks,
                                                       config.version,
                                                       blob_file));
  LOG(INFO) << "done reading " << new_part.name;

  TEST_AND_RETURN_FALSE(
      FragmentOperations(config.version, aops, new_part.path, blob_file));
  SortOperationsByDestination(aops);

  // Use the soft_chunk_size when merging operations to prevent merging all
  // the operations into a huge one if there's no hard limit.
  size_t merge_chunk_blocks = soft_chunk_blocks;
  if (hard_chunk_blocks != -1 &&
      static_cast<size_t>(hard_chunk_blocks) < soft_chunk_blocks) {
    merge_chunk_blocks = hard_chunk_blocks;
  }

  TEST_AND_RETURN_FALSE(MergeOperations(
      aops, config.version, merge_chunk_blocks, new_part.path, blob_file));

  if (config.version.minor >= kOpSrcHashMinorPayloadVersion)
    TEST_AND_RETURN_FALSE(AddSourceHash(aops, old_part.path));

  return true;
}

在接着往下看之前，需要了解幾個基本概念：
Partition Type：

Raw Partition: ex boot.img
FileSystem Partition: system.img

FileSystem Type:

raw
ext2
squashfs

帶文件系統的Partition會先解析爲一個一個的file，可以用過xxx.map查看。這些Partition最終會被拆分成一個一個的block

system/update_engine/payload_generator/block_mapping.h
BlockMapping:

block id: The block id assigned to this unique block
byte_offset: The location on this unique block on disk
block data: The location of block data

AnnotatedOperation:

op: The InstallOperation, as defined by the protobuf.

DeltaReadPartition()

/system/update_engine/payload_generator/delta_diff_utils.cc
統計Zero block和新舊相同的block

DeltaReadPartition()
- DeltaMoveAndZeroBlocks()
  /system/update_engine/payload_generator/delta_diff_utils.cc
  以文件爲單位做差分，每個文件都會生成對應的operations
DeltaReadPartition()
- vector<FileDeltaProcessor> file_delta_processors
  - file_delta_processors.emplace_back()
- MergeOperation(aops)
- DeltaReadFile()
  - ReadExtentsToDiff()

FragmentOperation()

/system/update_engine/payload_generator/delta_diff_utils.cc
假如operation中有超過一個extents，就會被拆分

SortOperationByDestination(aops)

/system/update_engine/payload_generator/delta_diff_utils.cc
假如operation中有超過一個extents，就會被拆分

MergeOperations()

/system/update_engine/payload_generator/delta_diff_utils.cc
對operation進行排序，也就是vector<AnnotatedOperation>* aops

Android A/B System - Generate OTA Package

文章目錄

Makefile

ota_from_target_files.py

OTA Package Format

protobuf

protobuf是什麼？

語法

update_metadata.proto

Generate Flow

Generate

Full OTA Package

Delta OTA Package

DeltaReadPartition()

FragmentOperation()

SortOperationByDestination(aops)

MergeOperations()

AddSourceHash()

Hash

Signature

Reference

985 碩士程序員，空窗 4 個月沒有 Offer！

營銷系統黑名單優化：位圖的應用解析

一文搞懂 Spring 循環依賴

我真的從測試轉成了開發......

nginx添加相應配置，通過瀏覽器訪問或curl時返回客戶端對應公網IP

賽博鬥地主——使用大語言模型扮演Agent智能體玩牌類遊戲。

python內置函數——sorted

[oeasy]python020在遊戲中體驗數值自由_勇闖地下城_終端文字遊戲

爲何我建議你學會抄代碼

抖音面試：說說延遲任務的調度算法？

Android9.0源碼學習-Sensor Framework

leetcode No315. Count of Smaller Numbers After Self

認真努力 - 我的2019

leetcode No198. House Robber

Android A/B System - Generate OTA Package

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結