Models And Schedulers
By incorporating multiple frameworks and also custom backends, the TensorRT Inference Server supports a wide variety of models. The inference server also supports multiple scheduling and batching configurations that further expand the class of models that the inference server can handle.
融合跟踪框架和自定义后台处理程序，trtis支持多种模型、多种策略和批量配置，未来可以拓展trtis能处理的模型种类。
This section describes stateless, stateful and ensemble models and how the inference server provides schedulers to support those model types.
这部分描述stateless,stateful和ensemble模型，trtis时怎样提供支持这些模型的策略。
Stateless Models
With respect to the inference server’s schedulers, a stateless model (or stateless custom backend) does not maintain state between inference requests. Each inference performed on a stateless model is independent of all other inferences using that model.
关于is的策略，stateless的模型或后台处理不会保存推理请求的状态。stateless模型的每次推理与使用该模型的其他推理是彼此独立的。
Examples of stateless models are CNNs such as image classification and object detection. The default scheduler or dynamic batcher can be used for these stateless models.
举例，像图像分类或者检测等CNN模型。这样的stateless模型可以使用trtis中的默认策略或动态批量。
RNNs and similar models which do have internal memory can be stateless as long as the state they maintain does not span inference requests. For example, an RNN that iterates over all elements in a batch is considered stateless by the inference server if the internal state is not carried between inference requests. The default scheduler can be used for these stateless models. The dynamic batcher cannot be used since the model is typically not expecting the batch to represent multiple inference requests.
RNN或者类似模型有内部存储也能是stateless的，只要他们的状态不会跨越推理请求。比如，在所有批量单元中迭代的RNN可以看作stateless，前提是内部的state不会在推理请求之间产生影响。默认的策略也能用在这些stateless模型上，但是动态批量不能，因为它是典型的不希望批量代表多推理请求。（批量代表一次请求而不是多次？）
Stateful Models
With respect to the inference server’s schedulers, a stateful model (or stateful custom backend) does maintain state between inference requests. The model is expecting multiple inference requests that together form a sequence of inferences that must be routed to the same model instance so that the state being maintained by the model is correctly updated. Moreover, the model may require that the inference server provide control signals indicating, for example, sequence start.
根据is策略，stateful模型或者后台处理保存推理请求之间的状态。模型期望来自一个序列的多个推理请求必须进入同一个模型实例，这样模型才能依靠保存的状态正确更新。然而，模型可能要求is提供指示控制信号，比如，序列的开端。
The sequence batcher must be used for these stateful models. As explained below, the sequence batcher ensures that all inference requests in a sequence get routed to the same model instance so that the model can maintain state correctly. The sequence batcher also communicates with the model to indicate when a sequence is starting, when a sequence is ending, when a sequence has a inference request ready for execution, and the correlation ID of the sequence.
有序批量必须用这些stateful模型，有序批量保证所有的推理请求是有序进入相同的模型实例，模型也因此能正确保存其状态。同时，有序批量能与模型进行通讯，告诉模型序列什么时候开始，什么时候结束，模型什么时候可接受新的推理请求，以及序列的相关ID等。
As explained in Client API for Stateful Models, when making inference requests for a stateful model, the client application must provide the same correlation ID to all requests in a sequence, and must also mark the start and end of the sequence. The correlation ID allows the inference server to identify that the requests belong to the same sequence.
如Client API的描述，当stateful模型处理推理请求时，客户端应用需要给所有序列中的请求提供相同的相关ID。这个ID允许is区分请求属于哪个序列。
Control Inputs
For a stateful model to operate correctly with the sequence batcher, the model must typically accept one or more control input tensors that the inference server uses to communicate with the model. The nvidia::inferenceserver::ModelSequenceBatching::Controlsection of the sequence batcher configuration indicates how the model exposes the tensors that the sequence batcher should use for these controls. All controls are optional. Below is portion of a model configuration that shows an example configuration for all the available control signals:
stateful模型为了能正确处理同一个序列的请求，需要接受一个或多个控制输入张量来与trt is进行通讯。
所有控制张量时可配置的，下面是一个例子：
sequence_batching {
control_input [
{
name: "START"
control [
{
kind: CONTROL_SEQUENCE_START
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: "END"
control [
{
kind: CONTROL_SEQUENCE_END
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: "READY"
control [
{
kind: CONTROL_SEQUENCE_READY
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: "CORRID"
control [
{
kind: CONTROL_SEQUENCE_CORRID
data_type: TYPE_UINT64
}
]
}
]
}
-
Start: The start input tensor is specified using CONTROL_SEQUENCE_START in the configuration. The example configuration indicates that the model has an input tensor called START with a 32-bit floating point data-type. The sequence batcher will define this tensor when executing an inference on the model. The START tensor must be 1-dimensional with size equal to the batch-size. Each element in the tensor indicates if the sequence in the corresponding batch slot is starting or not. In the example configuration, fp32_false_true indicates that a sequence start is indicated by tensor element equal to 1, and non-start is indicated by tensor element equal to 0.
-
End: The end input tensor is specified using CONTROL_SEQUENCE_END in the configuration. The example configuration indicates that the model has an input tensor called END with a 32-bit floating point data-type. The sequence batcher will define this tensor when executing an inference on the model. The END tensor must be 1-dimensional with size equal to the batch-size. Each element in the tensor indicates if the sequence in the corresponding batch slot is ending or not. In the example configuration, fp32_false_true indicates that a sequence end is indicated by tensor element equal to 1, and non-end is indicated by tensor element equal to 0.
-
Ready: The ready input tensor is specified using CONTROL_SEQUENCE_READY in the configuration. The example configuration indicates that the model has an input tensor called READY with a 32-bit floating point data-type. The sequence batcher will define this tensor when executing an inference on the model. The READY tensor must be 1-dimensional with size equal to the batch-size. Each element in the tensor indicates if the sequence in the corresponding batch slot has an inference request ready for inference. In the example configuration, fp32_false_true indicates that a sequence ready is indicated by tensor element equal to 1, and non-start is indicated by tensor element equal to 0.
-
Correlation ID: The correlation ID input tensor is specified using CONTROL_SEQUENCE_CORRID in the configuration. The example configuration indicates that the model has an input tensor called CORRID with a unsigned 64-bit integer data-type. The sequence batcher will define this tensor when executing an inference on the model. The CORRID tensor must be 1-dimensional with size equal to the batch-size. Each element in the tensor indicates the correlation ID of the sequence in the corresponding batch slot.
Scheduling Strategies
The sequence batcher can employ one of two scheduling strategies when deciding how to batch the sequences that are routed to the same model instance. These strategies are Direct and Oldest.
当决定怎样让有序批量作用于同一个模型实例时，序列批量能处理一半的部署策略。这些策略是直接而过时的：
Direct
With the Direct scheduling strategy the sequence batcher ensures not only that all inference requests in a sequence are routed to the same model instance, but also that each sequence is routed to a dedicated batch slot within the model instance. This strategy is required when the model maintains state for each batch slot, and is expecting all inference requests for a given sequence to be routed to the same slot so that the state is correctly updated.
As an example of the sequence batcher using the Direct scheduling strategy, assume a TensorRT stateful model that has the following model configuration:
name: “direct_stateful_model”
platform: “tensorrt_plan”
max_batch_size: 2
sequence_batching {
max_sequence_idle_microseconds: 5000000
direct { }
control_input [
{
name: “START”
control [
{
kind: CONTROL_SEQUENCE_START
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: “READY”
control [
{
kind: CONTROL_SEQUENCE_READY
fp32_false_true: [ 0, 1 ]
}
]
}
]
}
input [
{
name: “INPUT”
data_type: TYPE_FP32
dims: [ 100, 100 ]
}
]
output [
{
name: “OUTPUT”
data_type: TYPE_FP32
dims: [ 10 ]
}
]
instance_group [
{
count: 2
}
]
The sequence_batching section indicates that the model should use the sequence batcher and the Direct scheduling strategy. In this example the model only requires a start and ready control input from the sequence batcher so only those controls are listed. The instance_group indicates two instances of the model should be instantiated and max_batch_size indicates that each of those instances should perform batch-size 2 inferences. The following figure shows a representation of the sequence batcher and the inference resources specified by this configuration.
_images/sequence_example0.png
Each model instance is maintaining state for each batch slot, and is expecting all inference requests for a given sequence to be routed to the same slot so that the state is correctly updated. For this example that means that the inference server can simultaneously perform inference for up to four sequences.
Using the Direct scheduling strategy, the sequence batcher:
Recognizes when an inference request starts a new sequence and allocates a batch slot for that sequence. If no batch slot is available for the new sequence, the server places the inference request in a backlog.
Recognizes when an inference request is part of a sequence that has an allocated batch slot and routes the request to that slot.
Recognizes when an inference request is part of a sequence that is in the backlog and places the request in the backlog.
Recognizes when the last inference request in a sequence has been completed. The batch slot occupied by that sequence is immediately reallocated to a sequence in the backlog, or freed for a future sequence if there is no backlog.
The following figure shows how multiple sequences are scheduled onto the model instances using the Direct scheduling strategy. On the left the figure shows several sequences of requests arriving at the inference server. Each sequence could be made up of any number of inference requests and those individual inference requests could arrive in any order relative to inference requests in other sequences, except that the execution order shown on the right assumes that the first inference request of sequence 0 arrives before any inference request in sequences 1-5, the first inference request of sequence 1 arrives before any inference request in sequences 2-5, etc.
The right of the figure shows how the inference request sequences are scheduled onto the model instances over time.
_images/sequence_example1.png
The following figure shows the sequence batcher uses the control input tensors to communicate with the model. The figure shows two sequences assigned to the two batch slots in a model instance. Inference requests for each sequence arrive over time. The START and READY rows show the input tensor values used for each execution of the model. Over time the following happens:
The first request arrives for the sequence in slot0. Assuming the model instance is not already executing an inference, the sequence scheduler immediately schedules the model instance to execute because an inference request is available.
This is the first request in the sequence so the corresponding element in the START tensor is set to 1. There is no request available in slot1 so the READY tensor shows only slot0 as ready.
After the inference completes the sequence scheduler sees that there are no requests available in any batch slot and so the model instance sits idle.
Next, two inference requests arrive close together in time so that the sequence scheduler sees them both available in their respective batch slots. The scheduler immediately schedules the model instance to perform a batch-size 2 inference and uses START and READY to show that both slots have an inference request avaiable but that only slot1 is the start of a new sequence.
The processing continues in a similar manner for the other inference requests.
_images/sequence_example2.png
Oldest
With the Oldest scheduling strategy the sequence batcher ensures that all inference requests in a sequence are routed to the same model instance and then uses the dynamic batcher to batch together multiple inferences from different sequences into a batch that inferences together. With this strategy the model must typically use the CONTROL_SEQUENCE_CORRID control so that it knows which sequence each inference request in the batch belongs to. The CONTROL_SEQUENCE_READY control is typically not needed because all inferences in the batch will always be ready for inference.
As an example of the sequence batcher using the Oldest scheduling strategy, assume a Custom stateful model that has the following model configuration:
name: “oldest_stateful_model”
platform: “custom”
max_batch_size: 2
sequence_batching {
max_sequence_idle_microseconds: 5000000
oldest
{
max_candidate_sequences: 4
preferred_batch_size: [ 2 ]
}
control_input [
{
name: “START”
control [
{
kind: CONTROL_SEQUENCE_START
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: “END”
control [
{
kind: CONTROL_SEQUENCE_END
fp32_false_true: [ 0, 1 ]
}
]
},
{
name: “CORRID”
control [
{
kind: CONTROL_SEQUENCE_CORRID
data_type: TYPE_UINT64
}
]
}
]
}
input [
{
name: “INPUT”
data_type: TYPE_FP32
dims: [ 100, 100 ]
}
]
output [
{
name: “OUTPUT”
data_type: TYPE_FP32
dims: [ 10 ]
}
]
The sequence_batching section indicates that the model should use the sequence batcher and the Oldest scheduling strategy. The Oldest strategy is configured so that the sequence batcher maintains up to 4 active candidate sequences from which it prefers to form dynamic batches of size 2. In this example the model requires a start, end, and correlation ID control input from the sequence batcher. The following figure shows a representation of the sequence batcher and the inference resources specified by this configuration.
_images/dyna_sequence_example0.png
Using the Oldest scheduling strategy, the sequence batcher:
Recognizes when an inference request starts a new sequence and attempts to find a model instance that has room for a candidate sequence. If no model instance has room for a new candidate sequence, the server places the inference request in a backlog.
Recognizes when an inference request is part of a sequence that is already a candidate sequence in some model instance and routes the request to that model instance.
Recognizes when an inference request is part of a sequence that is in the backlog and places the request in the backlog.
Recognizes when the last inference request in a sequence has been completed. The model instance immediately removes a sequence from the backlog and makes it a candidate sequence in the model instance, or records that the model instance can handle a future sequence if there is no backlog.
The following figure shows how multiple sequences are scheduled onto the model instance specified by the above example configuration. On the left the figure shows four sequences of requests arriving at the inference server. Each sequence is composed of multiple inference requests as shown in the figure. The center of the figure shows how the inference request sequences are batched onto the model instance over time, assuming that the inference requests for each sequence arrive at the same rate with sequence A arriving just before B, which arrives just before C, etc. The Oldest strategy forms a dynamic batch from the oldest requests but never includes more than one request from a given sequence in a batch (for example, the last two inferences in sequence D are not batched together).
_images/dyna_sequence_example1.png
Ensemble Models
An ensemble model represents a pipeline of one or more models and the connection of input and output tensors between those models. Ensemble models are intended to be used to encapsulate a procedure that involves multiple models, such as “data preprocessing -> inference -> data postprocessing”. Using ensemble models for this purpose can avoid the overhead of transferring intermediate tensors and minimize the number of requests that must be sent to the inference server.
组合模型表示一个pipeline，这个pipeline由一个或多个模型、连接模型的输入输出张量构成。组合模型是用来封装多模型的处理，比如，数据预处理->推理->后处理。使用组合模型可以避免张量在不同模型中作用的开销，最小化发送到is请求的数量。
The ensemble scheduler must be used for ensemble models, regardless of the scheduler used by the models within the ensemble. With respect to the ensemble scheduler, an ensemble model is not an actual model. Instead, it specifies the dataflow between models within the ensemble as Step. The scheduler collects the output tensors in each step, provides them as input tensors for other steps according to the specification. In spite of that, the ensemble model is still viewed as a single model from an external view. Ensemble Image Classification Example Application is an example that performs image classification using an ensemble model.
组合策略必须用于组合模型，尽管策略是根据其包含的模型组织的。关于组合策略，组合模型可能不是一个模型。相反，它描述的是模型间的数据流动方向。策略收集每一步的输出张量，并按照说明提供给其他的步骤作为输入。尽管如此，外部来看，组合模型仍然被当成一个模型。综合图片分类Ensemble Image Classification Example Application例子就是用组合模型做图片分类的例子。
Note that the ensemble models will inherit the characteristics of the models involved, so the meta-data in the request header must comply with the models within the ensemble. For instance, if one of the models is stateful model, then the inference request for the ensemble model should contain the information mentioned in the previous section, which will be provided to the stateful model by the scheduler.
注意组合模型会继承模型包含的特点，所以请求头的meda数据需遵守组合模型中各自模型要求。比如，如果一个模型是stateful，组合模型的推理请求需要包含前面部分说过的信息，当然在该stateful模型提供的策略中也会有清晰描述。
As a running example, consider an ensemble model for image classification and segmentation that has the following model configuration:
给一个running例子，考虑一个图片分类和分割的组合模型：
name: "ensemble_model"
platform: "ensemble"
max_batch_size: 1
input [
{
name: "IMAGE"
data_type: TYPE_STRING
dims: [ 1 ]
}
]
output [
{
name: "CLASSIFICATION"
data_type: TYPE_FP32
dims: [ 1000 ]
},
{
name: "SEGMENTATION"
data_type: TYPE_FP32
dims: [ 3, 224, 224 ]
}
]
ensemble_scheduling {
step [
{
model_name: "image_preprocess_model"
model_version: -1
input_map {
key: "RAW_IMAGE"
value: "IMAGE"
}
output_map {
key: "PREPROCESSED_OUTPUT"
value: "preprocessed_image"
}
},
{
model_name: "classification_model"
model_version: -1
input_map {
key: "FORMATTED_IMAGE"
value: "preprocessed_image"
}
output_map {
key: "CLASSIFICATION_OUTPUT"
value: "CLASSIFICATION"
}
},
{
model_name: "segmentation_model"
model_version: -1
input_map {
key: "FORMATTED_IMAGE"
value: "preprocessed_image"
}
output_map {
key: "SEGMENTATION_OUTPUT"
value: "SEGMENTATION"
}
}
]
}
The ensemble_schedulingsection indicates that the ensemble scheduler will be used and that the ensemble model consists of three different models. Each element in step section specifies the model to be used and how the inputs and outputs of the model are mapped to tensor names recognized by the scheduler. For example, the first element in step specifies that the latest version of image_preprocess_modelshould be used, the content of its input “RAW_IMAGE”is provided by “IMAGE”tensor, and the content of its output “PREPROCESSED_OUTPUT”will be mapped to “preprocessed_image”tensor for later use. The tensor names recognized by the scheduler are the ensemble inputs, the ensemble outputs and all values in the input_mapand the output_map.
有ensemble_scheduling表示会使用组合策略，由3个模型组成。step中的每个单元是一个模型，给出了模型的输入输出及被映射到策略能识别的名称。例如，step中第一个模型是最新版image_preprocess_model，输入是“RAW_IMAGE”由“IMAGE”张量提供，输出会以“PREPROCESSED_OUTPUT”命名用以让其后的“preprocessed_image”使用。张量名称能被策略识别到作为组合输入，所有的键值对都存储在input_map和output_map中。
The models composing the ensemble may also have dynamic batching enabled. Since ensemble models are just routing the data between composing models, the inference server can take requests into an ensemble model without modifying the ensemble’s configuration to exploit the dynamic batching of the composing models.
所有的模型组成的组合模型也接受动态批量处理。因为组合模型只是指示了数据在所有组成模型间的流向，is能将许多请求送进一个组合模型而不用修改模型的配置来指明让这些模型支持动态批量处理。
Assuming that only the ensemble model, the preprocess model, the classification model and the segmentation model are being served, the client applications will see them as four different models which can process requests independently. However, the ensemble scheduler will view the ensemble model as the following.
假设只有组合模型，预处理模型，分类模型和分割模型提供服务，客户端会看到四个不同的模型，他们能独立接受请求。然而，组合策略会按如下策略处理组合模型。
When an inference request for the ensemble model is received, the ensemble scheduler will:
当is上的组合模型接受一个请求时
Recognize that the “IMAGE”tensor in the request is mapped to input “RAW_IMAGE”in the preprocess model.
识别请求中的“IMAGE”张量被映射到预处理模型中的“RAW_IMAGE”。
Check models within the ensemble and send an internal request to the preprocess model becuase all the input tensors required are ready.
检查组合模型，发送内部请求给预处理模型，因为所有给预处理的数据已经准备完毕。
Recognize the completion of the internal request, collect the output tensor and map the content to “preprocessed_image”which is an unique name known within the ensemble.
识别内部请求已经完成，收集输出张量，然后映射到组合策略中唯一的名称“preprocessed_image”中。
Map the newly collected tensor to inputs of the models within the ensemble. In this case, the inputs of “classification_model”and “segmentation_model”will be mapped and marked as ready.
映射新收集的张量作为下个模型的输入。在这种情况下，“classification_model”和“segmentation_model”将会被标记为ready。
Check models that require the newly collected tensor and send internal requests to models whose inputs are ready, the classification model and the segmentation model in this case. Note that the responses will be in arbitrary order depending on the load and computation time of individual models.
检查接受新收集张量为输入的模型，发送内部请求到需求数据已经准备完成的模型，也就是例子中的分类和分割模型。注意，返回是乱序的，取决于模型的load和单独处理时间。
Repeat step 3-5 until no more internal requests should be sent, and then response to the inference request with the tensors mapped to the ensemble output names.
重复步骤3-5直到没有内部请求发送，然后让输出映射到相应的组合名称变量中。