grpc-gateway 返回值中默認值爲什麼不顯示？

概述

問題

GPRC的RESTFUL API返回結果中忽略默認值或者空值

解決辦法

解決辦法其實就是把mux.marshalers[”*“]的marshaler的EmitDefaults=true，可以看下這個答案 https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags 提供的答案就是

gwmux := runtime.NewServeMux(runtime.WithMarshalerOption(runtime.MIMEWildcard, &runtime.JSONPb{OrigName: true, EmitDefaults: true}))

詳細說明

在升級proto2到proto3的過程中，因爲結構體的類型從指針變成了屬性本身，導致到grpc gateway返回reponse的時候，由於某些屬性的值是默認值，導致respose中缺失該屬性。

// proto 2
type B {
    b *int json:"b,omitempty"
}

type A {
    a *string json:"Aa,omitempty"
    b *B      json:"Ab,omitempty"
}
respA := A{a: new("hello", b: &B{b: new(0)})}
// json marshal respA result
{Aa:"hello", Ab:{b:0}}

// proto 3
type B {
    b int json:"b,omitempty"
}

type A {
    a string json:"Aa,omitempty"
    b *B     json:"Ab,omitempty"
}
respA := A{a: "hello", b: &B{b: 0}}
// json marshal respA result
{Aa:"hello"}

已上面代碼爲例，在proto2情況下， A.b 是指針，默認值爲 nil, 所以B.a=0的時候，json marshal之後的結果依然可以輸出，但是對於proto3的情況下，因爲A.b的值是默認值，所以在marshal的時候會導致被忽略。通常而言，在RESTFUL API的接口設計中，空值與默認值並不代表同一種情況。

Golang GRPC的omitempty

在使用proto3的時候，發生生成的gprc代碼裏面，結構體屬性默認帶的tag裏面就有omitempty，也沒找到生成選項裏面可以關掉這個tag，google了一下之後發現有些回答中提到 http://www.itkeyword.com/doc/9393181324974692x230/golang-protobuf-remove-omitempty-tag-from-generated-json-tags omitempty json tag is hard-coded into the protoc-gen-go，大家有興趣可以打開上面link中提到的源代碼，這裏貼出涉及到的代碼

jsonName := *field.Name
tag := fmt.Sprintf("protobuf:%s json:%q", g.goTag(message, field, wiretype), jsonName+",omitempty")

所以基本上可以找到的答案都是建議使用sed -i替換掉這個tag。。

GRPC-Gateway簡介

按照上面的方法，替換掉gprc生成的code中的omitempty之後，發現返回的API結果裏面默認值屬性還是被忽略掉了，覺得十分奇怪。通過再次搜索以及閱讀grpc-gateway源代碼之後，發現是gprc-gateway的鍋。 GRPC gateway的工作原理簡答而言

提供對外的RESTFUL API
收到請求之後，把結果編譯成proto.Message
轉發請求給GPRC
收到proto.Message之後，編譯成json，返回給調用者

通常而言，在使用grpc-gateway的時候，要先生成一個mux

    "github.com/grpc-ecosystem/grpc-gateway/runtime"
    grpcService := grpc.NewServer()
    mux := runtime.NewServeMux()

// mux.go
// NewServeMux returns a new ServeMux whose internal mapping is empty.
func NewServeMux(opts ...ServeMuxOption) *ServeMux {
    serveMux := &ServeMux{
        handlers:               make(map[string][]handler),
        forwardResponseOptions: make([]func(context.Context, http.ResponseWriter, proto.Message) error, 0),
        marshalers:             makeMarshalerMIMERegistry(),
        incomingHeaderMatcher:  DefaultHeaderMatcher,
    }

    for _, opt := range opts {
        opt(serveMux)
    }
    return serveMux
}

扒源代碼可以看到，NewServeMux定義了marshalers屬性，下面可以看下它的make方法

// MIMEWildcard is the fallback MIME type used for requests which do not match
// a registered MIME type.
const MIMEWildcard = "*"

var (
    defaultMarshaler = &JSONPb{OrigName: true}
)

// makeMarshalerMIMERegistry returns a new registry of marshalers.
// It allows for a mapping of case-sensitive Content-Type MIME type string to runtime.Marshaler interfaces.
//
// For example, you could allow the client to specify the use of the runtime.JSONPb marshaler
// with a "application/jsonpb" Content-Type and the use of the runtime.JSONBuiltin marshaler
// with a "application/json" Content-Type.
// "*" can be used to match any Content-Type.
// This can be attached to a ServerMux with the marshaler option.
func makeMarshalerMIMERegistry() marshalerRegistry {
    return marshalerRegistry{
        mimeMap: map[string]Marshaler{
            MIMEWildcard: defaultMarshaler,
        },
    }
}

makeMarshalerMIMERegistry定義了一個默認的marshal方法，其中用到了&JSONPb模塊。

當gprc-gateway轉發請求的時候，會用到生成好的如下代碼

inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
        ...
resp, md, err := request_XXX_Get_0(rctx, inboundMarshaler, client, req, pathParams)
        ...
forward_XXX_Get_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)

這裏也可以順帶串下grpc-gateway的workflow request_XXX_Get_0負責轉發請求給GRPC server，具體的邏輯也比較簡單，不再贅述

if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil {
        return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}

msg, err := client.Get(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))

看下forward_XXX_Get_0這個函數，仔細看代碼的話，會發現生成的代碼裏面有一個定義把custom函數定義到grpc的系統函數

var (
    forward_XXX_Get_0 = runtime.ForwardResponseMessage
)

// ForwardResponseMessage forwards the message "resp" from gRPC server to REST client.
func ForwardResponseMessage(ctx context.Context, mux *ServeMux, marshaler Marshaler, w http.ResponseWriter, req *http.Request, resp proto.Message, opts ...func(context.Context, http.ResponseWriter, proto.Message) error) {
    md, ok := ServerMetadataFromContext(ctx)
    if !ok {
        grpclog.Printf("Failed to extract ServerMetadata from context")
    }

    handleForwardResponseServerMetadata(w, mux, md)
    handleForwardResponseTrailerHeader(w, md)
    w.Header().Set("Content-Type", marshaler.ContentType())
    if err := handleForwardResponseOptions(ctx, w, resp, opts); err != nil {
        HTTPError(ctx, mux, marshaler, w, req, err)
        return
    }

    buf, err := marshaler.Marshal(resp)
    if err != nil {
        grpclog.Printf("Marshal error: %v", err)
        HTTPError(ctx, mux, marshaler, w, req, err)
        return
    }

    if _, err = w.Write(buf); err != nil {
        grpclog.Printf("Failed to write response: %v", err)
    }

    handleForwardResponseTrailer(w, md)
}

從代碼的comment就可以看着，這個函數的主要功能就是用處理resp，編譯成json字符，返回結果.

這裏仔細看下MarshalerForRequest這個函數，它決定了grpc-gateway怎麼樣編譯反編譯proto信息

// MarshalerForRequest returns the inbound/outbound marshalers for this request.
// It checks the registry on the ServeMux for the MIME type set by the Content-Type header.
// If it isn't set (or the request Content-Type is empty), checks for "*".
// If there are multiple Content-Type headers set, choose the first one that it can
// exactly match in the registry.
// Otherwise, it follows the above logic for "*"/InboundMarshaler/OutboundMarshaler.
func MarshalerForRequest(mux *ServeMux, r *http.Request) (inbound Marshaler, outbound Marshaler) {
    for _, acceptVal := range r.Header[acceptHeader] {
        if m, ok := mux.marshalers.mimeMap[acceptVal]; ok {
            outbound = m
            break
        }
    }

    for _, contentTypeVal := range r.Header[contentTypeHeader] {
        if m, ok := mux.marshalers.mimeMap[contentTypeVal]; ok {
            inbound = m
            break
        }
    }

    if inbound == nil {
        inbound = mux.marshalers.mimeMap[MIMEWildcard]
    }
    if outbound == nil {
        outbound = inbound
    }

    return inbound, outbound
}

函數runtime.MarshalerForRequest，這個函數返回兩個mashaler，一個用來編譯request，一個用來編譯request， 從下面的代碼可以看出，如果沒有執行http HEADER的類型與value，最終都會用到之前定義好的MIMEWildcard(*)，所以默認的json編譯選項用的是&JSONPb結構體。 在轉發請求的函數中，可以看到buf, err := marshaler.Marshal(resp), 可以看下默認outbound的行爲是怎樣的。 首先得看下defaultMarshaler = &JSONPb{OrigName: true}是個什麼東西，跳轉之後可以看到如下代碼

type JSONPb jsonpb.Marshaler
// Marshal marshals "v" into JSON
// Currently it can marshal only proto.Message.
// TODO(yugui) Support fields of primitive types in a message.
func (j *JSONPb) Marshal(v interface{}) ([]byte, error) {
    if _, ok := v.(proto.Message); !ok {
        return j.marshalNonProtoField(v)
    }

    var buf bytes.Buffer
    if err := j.marshalTo(&buf, v); err != nil {
        return nil, err
    }
    return buf.Bytes(), nil
}

func (j *JSONPb) marshalTo(w io.Writer, v interface{}) error {
    p, ok := v.(proto.Message)
    if !ok {
        buf, err := j.marshalNonProtoField(v)
        if err != nil {
            return err
        }
        _, err = w.Write(buf)
        return err
    }
    return (*jsonpb.Marshaler)(j).Marshal(w, p)
}

能看到，當最後調用marshal的時候，會從Marshal調用到marshalTo，最後一行用一種奇特的用法向上調用引用package的Marshal函數，之前從來沒有見過這種用法，也是漲知識了。接下來看下jsonpb裏面的代碼

// jsonpb.go
// Marshaler is a configurable object for converting between
// protocol buffer objects and a JSON representation for them.
type Marshaler struct {
    // Whether to render fields with zero values.
    EmitDefaults bool
}

type JSONPBMarshaler interface {
    MarshalJSONPB(*Marshaler) ([]byte, error)
}

// Marshal marshals a protocol buffer into JSON.
func (m *Marshaler) Marshal(out io.Writer, pb proto.Message) error {
    writer := &errWriter{writer: out}
    return m.marshalObject(writer, pb, "", "")
}

// marshalObject writes a struct to the Writer.
func (m *Marshaler) marshalObject(out *errWriter, v proto.Message, indent, typeURL string) error {
    if jsm, ok := v.(JSONPBMarshaler); ok {
        b, err := jsm.MarshalJSONPB(m)
        if err != nil {
            return err
        }
        out.write(string(b))
        return out.err
    }

    s := reflect.ValueOf(v).Elem()

    // Handle well-known types.
    if wkt, ok := v.(wkt); ok {
        switch wkt.XXX_WellKnownType() {
        case "DoubleValue", "FloatValue", "Int64Value", "UInt64Value",
            "Int32Value", "UInt32Value", "BoolValue", "StringValue", "BytesValue":
            // "Wrappers use the same representation in JSON
            //  as the wrapped primitive type, ..."
            sprop := proto.GetProperties(s.Type())
            return m.marshalValue(out, sprop.Prop[0], s.Field(0), indent)
        case "Any":
            // Any is a bit more involved.
            return m.marshalAny(out, v, indent)
        }
    }

上面是jsonpb的代碼，代碼太長了，這裏只截取關鍵的部分，看到看到Marshaler結構體裏的EmitDefaults屬性，這個屬性從變量名即可看出是用來控制json的output default值的行爲的。 這裏也提到了一個接口MarshalJSONPB(*Marshaler) ([]byte, error) 可以自定義marshal的行爲，等有空可以研究下在哪裏調用。默認情況下都會走到第二個if判斷裏面，這裏用到了一個接口方法XXX_WellKnownType，從 grpc-go 官方文檔可以看到如下描述

Protobufs come with a set of predefined messages, called well-known types (WKTs). These types can be useful either for interoperability with other services, or simply because they succinctly represent common, useful patterns. For example, the Struct message represents the format of an arbitrary C-style struct.

Pre-generated Go code for the WKTs is distributed as part of the Go protobuf library, and this code is referenced by the generated Go code of your messages if they use a WKT.

Generally speaking, you shouldn’t need to import these types directly into your code. However, if you need to reference one of these types directly, simply import the github.com/golang/protobuf/ptypes/[TYPE] package, and use the type normally.

所以jsonpb代碼解析pb message結構體的時候使用時WKTs的type，最後看下omitempty的相關行爲，截取部分代碼如下

for i := 0; i < s.NumField(); i++ {
        value := s.Field(i)
        valueField := s.Type().Field(i)
        if strings.HasPrefix(valueField.Name, "XXX_") {
            continue
        }

        // IsNil will panic on most value kinds.
        switch value.Kind() {
        case reflect.Chan, reflect.Func, reflect.Interface:
            if value.IsNil() {
                continue
            }
        }

        if !m.EmitDefaults {
            switch value.Kind() {
            case reflect.Bool:
                if !value.Bool() {
                    continue
                }
            case reflect.Int32, reflect.Int64:
                if value.Int() == 0 {
                    continue
                }
            }
        }

    }

能看到當EmitDefaults爲false的時候，jsonpb包會忽略掉默認值的field，所以在返回的http reponse中看不到這些默認項。

解決辦法

解決辦法其實就是把mux.marshalers[”*“]的marshaler的EmitDefaults=true，可以看下這個答案 https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags 提供的答案就是

gwmux := runtime.NewServeMux(runtime.WithMarshalerOption(runtime.MIMEWildcard, &runtime.JSONPb{OrigName: true, EmitDefaults: true}))