grpc-gateway 返回值中默认值为什么不显示？

概述

问题

GPRC的RESTFUL API返回结果中忽略默认值或者空值

解决办法

解决办法其实就是把mux.marshalers[”*“]的marshaler的EmitDefaults=true，可以看下这个答案 https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags 提供的答案就是

gwmux := runtime.NewServeMux(runtime.WithMarshalerOption(runtime.MIMEWildcard, &runtime.JSONPb{OrigName: true, EmitDefaults: true}))

详细说明

在升级proto2到proto3的过程中，因为结构体的类型从指针变成了属性本身，导致到grpc gateway返回reponse的时候，由于某些属性的值是默认值，导致respose中缺失该属性。

// proto 2
type B {
    b *int json:"b,omitempty"
}

type A {
    a *string json:"Aa,omitempty"
    b *B      json:"Ab,omitempty"
}
respA := A{a: new("hello", b: &B{b: new(0)})}
// json marshal respA result
{Aa:"hello", Ab:{b:0}}

// proto 3
type B {
    b int json:"b,omitempty"
}

type A {
    a string json:"Aa,omitempty"
    b *B     json:"Ab,omitempty"
}
respA := A{a: "hello", b: &B{b: 0}}
// json marshal respA result
{Aa:"hello"}

已上面代码为例，在proto2情况下， A.b 是指针，默认值为 nil, 所以B.a=0的时候，json marshal之后的结果依然可以输出，但是对于proto3的情况下，因为A.b的值是默认值，所以在marshal的时候会导致被忽略。通常而言，在RESTFUL API的接口设计中，空值与默认值并不代表同一种情况。

Golang GRPC的omitempty

在使用proto3的时候，发生生成的gprc代码里面，结构体属性默认带的tag里面就有omitempty，也没找到生成选项里面可以关掉这个tag，google了一下之后发现有些回答中提到 http://www.itkeyword.com/doc/9393181324974692x230/golang-protobuf-remove-omitempty-tag-from-generated-json-tags omitempty json tag is hard-coded into the protoc-gen-go，大家有兴趣可以打开上面link中提到的源代码，这里贴出涉及到的代码

jsonName := *field.Name
tag := fmt.Sprintf("protobuf:%s json:%q", g.goTag(message, field, wiretype), jsonName+",omitempty")

所以基本上可以找到的答案都是建议使用sed -i替换掉这个tag。。

GRPC-Gateway简介

按照上面的方法，替换掉gprc生成的code中的omitempty之后，发现返回的API结果里面默认值属性还是被忽略掉了，觉得十分奇怪。通过再次搜索以及阅读grpc-gateway源代码之后，发现是gprc-gateway的锅。 GRPC gateway的工作原理简答而言

提供对外的RESTFUL API
收到请求之后，把结果编译成proto.Message
转发请求给GPRC
收到proto.Message之后，编译成json，返回给调用者

通常而言，在使用grpc-gateway的时候，要先生成一个mux

    "github.com/grpc-ecosystem/grpc-gateway/runtime"
    grpcService := grpc.NewServer()
    mux := runtime.NewServeMux()

// mux.go
// NewServeMux returns a new ServeMux whose internal mapping is empty.
func NewServeMux(opts ...ServeMuxOption) *ServeMux {
    serveMux := &ServeMux{
        handlers:               make(map[string][]handler),
        forwardResponseOptions: make([]func(context.Context, http.ResponseWriter, proto.Message) error, 0),
        marshalers:             makeMarshalerMIMERegistry(),
        incomingHeaderMatcher:  DefaultHeaderMatcher,
    }

    for _, opt := range opts {
        opt(serveMux)
    }
    return serveMux
}

扒源代码可以看到，NewServeMux定义了marshalers属性，下面可以看下它的make方法

// MIMEWildcard is the fallback MIME type used for requests which do not match
// a registered MIME type.
const MIMEWildcard = "*"

var (
    defaultMarshaler = &JSONPb{OrigName: true}
)

// makeMarshalerMIMERegistry returns a new registry of marshalers.
// It allows for a mapping of case-sensitive Content-Type MIME type string to runtime.Marshaler interfaces.
//
// For example, you could allow the client to specify the use of the runtime.JSONPb marshaler
// with a "application/jsonpb" Content-Type and the use of the runtime.JSONBuiltin marshaler
// with a "application/json" Content-Type.
// "*" can be used to match any Content-Type.
// This can be attached to a ServerMux with the marshaler option.
func makeMarshalerMIMERegistry() marshalerRegistry {
    return marshalerRegistry{
        mimeMap: map[string]Marshaler{
            MIMEWildcard: defaultMarshaler,
        },
    }
}

makeMarshalerMIMERegistry定义了一个默认的marshal方法，其中用到了&JSONPb模块。

当gprc-gateway转发请求的时候，会用到生成好的如下代码

inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
        ...
resp, md, err := request_XXX_Get_0(rctx, inboundMarshaler, client, req, pathParams)
        ...
forward_XXX_Get_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)

这里也可以顺带串下grpc-gateway的workflow request_XXX_Get_0负责转发请求给GRPC server，具体的逻辑也比较简单，不再赘述

if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil {
        return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}

msg, err := client.Get(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))

看下forward_XXX_Get_0这个函数，仔细看代码的话，会发现生成的代码里面有一个定义把custom函数定义到grpc的系统函数

var (
    forward_XXX_Get_0 = runtime.ForwardResponseMessage
)

// ForwardResponseMessage forwards the message "resp" from gRPC server to REST client.
func ForwardResponseMessage(ctx context.Context, mux *ServeMux, marshaler Marshaler, w http.ResponseWriter, req *http.Request, resp proto.Message, opts ...func(context.Context, http.ResponseWriter, proto.Message) error) {
    md, ok := ServerMetadataFromContext(ctx)
    if !ok {
        grpclog.Printf("Failed to extract ServerMetadata from context")
    }

    handleForwardResponseServerMetadata(w, mux, md)
    handleForwardResponseTrailerHeader(w, md)
    w.Header().Set("Content-Type", marshaler.ContentType())
    if err := handleForwardResponseOptions(ctx, w, resp, opts); err != nil {
        HTTPError(ctx, mux, marshaler, w, req, err)
        return
    }

    buf, err := marshaler.Marshal(resp)
    if err != nil {
        grpclog.Printf("Marshal error: %v", err)
        HTTPError(ctx, mux, marshaler, w, req, err)
        return
    }

    if _, err = w.Write(buf); err != nil {
        grpclog.Printf("Failed to write response: %v", err)
    }

    handleForwardResponseTrailer(w, md)
}

从代码的comment就可以看着，这个函数的主要功能就是用处理resp，编译成json字符，返回结果.

这里仔细看下MarshalerForRequest这个函数，它决定了grpc-gateway怎么样编译反编译proto信息

// MarshalerForRequest returns the inbound/outbound marshalers for this request.
// It checks the registry on the ServeMux for the MIME type set by the Content-Type header.
// If it isn't set (or the request Content-Type is empty), checks for "*".
// If there are multiple Content-Type headers set, choose the first one that it can
// exactly match in the registry.
// Otherwise, it follows the above logic for "*"/InboundMarshaler/OutboundMarshaler.
func MarshalerForRequest(mux *ServeMux, r *http.Request) (inbound Marshaler, outbound Marshaler) {
    for _, acceptVal := range r.Header[acceptHeader] {
        if m, ok := mux.marshalers.mimeMap[acceptVal]; ok {
            outbound = m
            break
        }
    }

    for _, contentTypeVal := range r.Header[contentTypeHeader] {
        if m, ok := mux.marshalers.mimeMap[contentTypeVal]; ok {
            inbound = m
            break
        }
    }

    if inbound == nil {
        inbound = mux.marshalers.mimeMap[MIMEWildcard]
    }
    if outbound == nil {
        outbound = inbound
    }

    return inbound, outbound
}

函数runtime.MarshalerForRequest，这个函数返回两个mashaler，一个用来编译request，一个用来编译request， 从下面的代码可以看出，如果没有执行http HEADER的类型与value，最终都会用到之前定义好的MIMEWildcard(*)，所以默认的json编译选项用的是&JSONPb结构体。 在转发请求的函数中，可以看到buf, err := marshaler.Marshal(resp), 可以看下默认outbound的行为是怎样的。 首先得看下defaultMarshaler = &JSONPb{OrigName: true}是个什么东西，跳转之后可以看到如下代码

type JSONPb jsonpb.Marshaler
// Marshal marshals "v" into JSON
// Currently it can marshal only proto.Message.
// TODO(yugui) Support fields of primitive types in a message.
func (j *JSONPb) Marshal(v interface{}) ([]byte, error) {
    if _, ok := v.(proto.Message); !ok {
        return j.marshalNonProtoField(v)
    }

    var buf bytes.Buffer
    if err := j.marshalTo(&buf, v); err != nil {
        return nil, err
    }
    return buf.Bytes(), nil
}

func (j *JSONPb) marshalTo(w io.Writer, v interface{}) error {
    p, ok := v.(proto.Message)
    if !ok {
        buf, err := j.marshalNonProtoField(v)
        if err != nil {
            return err
        }
        _, err = w.Write(buf)
        return err
    }
    return (*jsonpb.Marshaler)(j).Marshal(w, p)
}

能看到，当最后调用marshal的时候，会从Marshal调用到marshalTo，最后一行用一种奇特的用法向上调用引用package的Marshal函数，之前从来没有见过这种用法，也是涨知识了。接下来看下jsonpb里面的代码

// jsonpb.go
// Marshaler is a configurable object for converting between
// protocol buffer objects and a JSON representation for them.
type Marshaler struct {
    // Whether to render fields with zero values.
    EmitDefaults bool
}

type JSONPBMarshaler interface {
    MarshalJSONPB(*Marshaler) ([]byte, error)
}

// Marshal marshals a protocol buffer into JSON.
func (m *Marshaler) Marshal(out io.Writer, pb proto.Message) error {
    writer := &errWriter{writer: out}
    return m.marshalObject(writer, pb, "", "")
}

// marshalObject writes a struct to the Writer.
func (m *Marshaler) marshalObject(out *errWriter, v proto.Message, indent, typeURL string) error {
    if jsm, ok := v.(JSONPBMarshaler); ok {
        b, err := jsm.MarshalJSONPB(m)
        if err != nil {
            return err
        }
        out.write(string(b))
        return out.err
    }

    s := reflect.ValueOf(v).Elem()

    // Handle well-known types.
    if wkt, ok := v.(wkt); ok {
        switch wkt.XXX_WellKnownType() {
        case "DoubleValue", "FloatValue", "Int64Value", "UInt64Value",
            "Int32Value", "UInt32Value", "BoolValue", "StringValue", "BytesValue":
            // "Wrappers use the same representation in JSON
            //  as the wrapped primitive type, ..."
            sprop := proto.GetProperties(s.Type())
            return m.marshalValue(out, sprop.Prop[0], s.Field(0), indent)
        case "Any":
            // Any is a bit more involved.
            return m.marshalAny(out, v, indent)
        }
    }

上面是jsonpb的代码，代码太长了，这里只截取关键的部分，看到看到Marshaler结构体里的EmitDefaults属性，这个属性从变量名即可看出是用来控制json的output default值的行为的。 这里也提到了一个接口MarshalJSONPB(*Marshaler) ([]byte, error) 可以自定义marshal的行为，等有空可以研究下在哪里调用。默认情况下都会走到第二个if判断里面，这里用到了一个接口方法XXX_WellKnownType，从 grpc-go 官方文档可以看到如下描述

Protobufs come with a set of predefined messages, called well-known types (WKTs). These types can be useful either for interoperability with other services, or simply because they succinctly represent common, useful patterns. For example, the Struct message represents the format of an arbitrary C-style struct.

Pre-generated Go code for the WKTs is distributed as part of the Go protobuf library, and this code is referenced by the generated Go code of your messages if they use a WKT.

Generally speaking, you shouldn’t need to import these types directly into your code. However, if you need to reference one of these types directly, simply import the github.com/golang/protobuf/ptypes/[TYPE] package, and use the type normally.

所以jsonpb代码解析pb message结构体的时候使用时WKTs的type，最后看下omitempty的相关行为，截取部分代码如下

for i := 0; i < s.NumField(); i++ {
        value := s.Field(i)
        valueField := s.Type().Field(i)
        if strings.HasPrefix(valueField.Name, "XXX_") {
            continue
        }

        // IsNil will panic on most value kinds.
        switch value.Kind() {
        case reflect.Chan, reflect.Func, reflect.Interface:
            if value.IsNil() {
                continue
            }
        }

        if !m.EmitDefaults {
            switch value.Kind() {
            case reflect.Bool:
                if !value.Bool() {
                    continue
                }
            case reflect.Int32, reflect.Int64:
                if value.Int() == 0 {
                    continue
                }
            }
        }

    }

能看到当EmitDefaults为false的时候，jsonpb包会忽略掉默认值的field，所以在返回的http reponse中看不到这些默认项。

解决办法

解决办法其实就是把mux.marshalers[”*“]的marshaler的EmitDefaults=true，可以看下这个答案 https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags 提供的答案就是

gwmux := runtime.NewServeMux(runtime.WithMarshalerOption(runtime.MIMEWildcard, &runtime.JSONPb{OrigName: true, EmitDefaults: true}))