golang操作mongodb的驅動mongo-go-driver的事務支持

mongodb要支持事務，需要滿足以下條件：

• 4.0以上版本；
• 安裝後時以replication set（複本集）模式啓動；
• storageEngine存儲引擎須是wiredTiger (支持文檔級別的鎖)，4.0以上版本已經默認是這個，參考；

安裝mongodb server 4.0以上版本

下載地址 目前最新的release版本是4.0.5，package 類型是server：

• 可根據自己的系統平臺選擇相應的安裝包進行安裝
• 可下載源碼包進行編譯安裝

我用的是ubuntu 16.04系統 x64，所以我直接下載該該平臺的server deb包進行安裝

ubuntu@VM-0-3-ubuntu:~$ wget https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/4.0/multiverse/binary-amd64/mongodb-org-server_4.0.5_amd64.deb
ubuntu@VM-0-3-ubuntu::~$ dkpg -i mongodb-org-server_4.0.5_amd64.deb
ubuntu@VM-0-3-ubuntu:~$ whereis mongod  #可以看到安裝後的程序和配置文件路徑
mongod: /usr/bin/mongod /etc/mongod.conf /usr/share/man/man1/mongod.1.gz

replication set模式：

• 就是多個server進程組合成一個集羣，會自動推選出一個Primary server來對外，提供數據庫服務，其它的server就是Secondary角色，還有一種不存儲數據只選舉充數的arbiter。

• Primary接收所有數據的寫操作並記錄到操作日誌。

• 所有Secondary server複製Primary server的操作日誌並更新到自己的數據集，異步的，所以有數據一致性問題。

• server之間相互有心跳。當Primary一段時間(可配置，默認10秒)內沒有心跳，Secondary們就再選出一個Primary，一個合適的secondary server會發起選舉自薦爲Primary，讓其它server投票。如果本來數據庫進程是偶數，就需要增加一個arbiter角色的server集成奇數個，以在選舉時確定“大多數”。選舉過程中不再接收寫操作。

如果有條件，最好準備多臺主機。我測試，只有一臺主機，所以運行3個端口的進程。

打開mongod.conf配置文件，做點修改：

ubuntu@VM-0-3-ubuntu:~$ sudo vi /etc/mongod.conf  #看到文件內容如下

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017      #端口
  bindIp: 0.0.0.0  #地址默認是127.0.0.1 ，如果想要網絡上能訪問到，最好改成外網地址或全0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

replication:     #此行默認是加#註釋掉的，需要去掉#，開啓複本集模式，再加上下面一行
  replSetName: rs1   #rs1是複本集的名字，可自定義，但集羣中所有server的配置必須一樣
#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

修改配置文件後，複製兩份：

ubuntu@VM-0-3-ubuntu:~$ sudo cp /etc/mongod.conf /etc/mongod1.conf
ubuntu@VM-0-3-ubuntu:~$ sudo cp /etc/mongod.conf /etc/mongod2.conf

打開/etc/mongod1.conf，修改三行：

ubuntu@VM-0-3-ubuntu:~$ sudo vi /etc/mongod1.conf

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb1   #此處的數據存儲路徑目錄改一下，以區分
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod1.log #此處的日誌文件路徑改一下，以區分

# network interfaces
net:
  port: 27018				#因在同一主機上，需要一個不同的端口，改爲27018
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

replication:
  replSetName: rs1
#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

同樣修改/etc/mongod2.conf，對應修改即可，端口27019。

然後啓動三個mongod服務進程：

ubuntu@VM-0-3-ubuntu:~$ sudo mongod --fork -f /etc/mongod.conf  #--fork表示後臺進程運行
ubuntu@VM-0-3-ubuntu:~$ sudo mongod --fork -f /etc/mongod1.conf
ubuntu@VM-0-3-ubuntu:~$ sudo mongod --fork -f /etc/mongod2.conf
ubuntu@VM-0-3-ubuntu:~$ ps -ef | grep mongod #可查看一下三個進程

安裝mongodb shell客戶端

下載地址 package 類型選 shell

ubuntu@VM-0-3-ubuntu:~$ wget https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/4.0/multiverse/binary-amd64/mongodb-org-shell_4.0.5_amd64.deb
ubuntu@VM-0-3-ubuntu::~$ dkpg -i mongodb-org-shell_4.0.5_amd64.deb
ubuntu@VM-0-3-ubuntu:~$ whereis mongo  #可以看到安裝後的程序和配置文件路徑
mongo: /usr/bin/mongo /usr/share/man/man1/mongo.1.gz

然後需要用客戶端連接一個server，並初始化複本集：

ubuntu@VM-0-3-ubuntu:~$ mongo          #運行mongo客戶端，默認是連接127.0.0.1:27107/test，也可以指定服務器地址，比如 mongodb://127.0.0.1:27107/test,test表示數據庫名。

MongoDB shell version v4.0.5
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("76f42f17-4012-4ad0-b386-7e66459af51d") }
MongoDB server version: 4.0.5
Server has startup warnings: 
2018-12-24T22:37:51.871+0800 I STORAGE  [initandlisten] 
2018-12-24T22:37:51.871+0800 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-12-24T22:37:51.871+0800 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-12-24T22:37:52.628+0800 I CONTROL  [initandlisten] 
2018-12-24T22:37:52.628+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-24T22:37:52.628+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-12-24T22:37:52.628+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-12-24T22:37:52.628+0800 I CONTROL  [initandlisten] 
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

#執行如下命令初始化三個成員，rs1是集羣名；members裏的_id是序號，可自定義，不同就行；host是ip:端口，我用的是本地127.0.0.1，如果要外網訪問，請設置相應的外網ip。
> rs.initiate({_id:rs1,members:[{_id:0,host:127.0.0.1:27107},{_id:0,host:127.0.0.1:27108},{_id:0,host:127.0.0.1:27109}]})

#成功會就會開始推選primary，一般是客戶端連的這個server.

rs1:PRIMARY> rs.status()   #查看一下複本集的狀態，顯示如下

{
        "set" : "rs1",
        "date" : ISODate("2018-12-25T06:02:09.320Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1545717724, 1),
                        "t" : NumberLong(1)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1545717724, 1),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1545717724, 1),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1545717724, 1),
                        "t" : NumberLong(1)
                }
        },
        "lastStableCheckpointTimestamp" : Timestamp(1545717714, 1),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "127.0.0.1:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 55458,
                        "optime" : {
                                "ts" : Timestamp(1545717724, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-12-25T06:02:04Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1545662551, 1),
                        "electionDate" : ISODate("2018-12-24T14:42:31Z"),
                        "configVersion" : 1,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 55189,
                        "optime" : {
                                "ts" : Timestamp(1545717724, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1545717724, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-12-25T06:02:04Z"),
                        "optimeDurableDate" : ISODate("2018-12-25T06:02:04Z"),
                        "lastHeartbeat" : ISODate("2018-12-25T06:02:07.796Z"),
                        "lastHeartbeatRecv" : ISODate("2018-12-25T06:02:07.795Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "127.0.0.1:27017",
                        "syncSourceHost" : "127.0.0.1:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 1
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:27019",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 55189,
                        "optime" : {
                                "ts" : Timestamp(1545717724, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1545717724, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-12-25T06:02:04Z"),
                        "optimeDurableDate" : ISODate("2018-12-25T06:02:04Z"),
                        "lastHeartbeat" : ISODate("2018-12-25T06:02:07.796Z"),
                        "lastHeartbeatRecv" : ISODate("2018-12-25T06:02:07.795Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "127.0.0.1:27017",
                        "syncSourceHost" : "127.0.0.1:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1545717724, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1545717724, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

可以確定一下server的存儲引擎是不是wiredTiger:

ubuntu@VM-0-3-ubuntu:~$ echo "db.serverStatus()" | mongo | grep wiredTiger
			"name" : "wiredTiger",
        "wiredTiger" : {

如果沒有輸出，則不是wiredTiger，再看看具體是什麼。因爲輸出內容較多，在終端上很難找，可以輸出到文件去後再找storageEngine：

ubuntu@VM-0-3-ubuntu:~$ echo "db.serverStatus()"| mongo > a.log
ubuntu@VM-0-3-ubuntu:~$ vi a.log

本來mongod默認是wiredTiger，我就遇到一個問題：之前裝過舊版本的mongod，其存儲引擎是mmapv1 ，在dbpath路徑下留下了數據文件，換成4.0的mongod server後，依然設置了該數據目錄，導致它去讀了舊的數據文件，就初始化成了mmapv1 引擎，最後測試事務的時候總報錯不支持“transaction numbers”,需要支持文檔級別鎖的存儲引擎。

Mongo-go-driver測試事務

在gopath項目裏使用go get或者go vendor下載github.com\mongodb\mongo-go-driver開源包，具體過程略。

mongo-go-driver是mongo官方的golang驅動庫，目前還有頻繁修改。

驅動源碼裏，連接server過程內會先生成連接池，然後返回有一個client對象，通過client對象可以對server裏的數據庫集合進行讀寫。但是任何讀寫操作本身是不帶session對象的，所以在操作前會先生成一個默認的session對象，然後再從連接池中取一個連接來進行通信。而事務相關的接口是在session接口內，包括Transaction的Start、Abort、Commit，但session接口裏並沒有其它CRUD相關方法。

研究時走了些彎路，先看一下如下代碼：

import (
	"context"
	"github.com/mongodb/mongo-go-driver/mongo"
	"net/url"
    "fmt"
)

func main(){
    connectString := "mongodb://127.0.0.1/test"
	dbUrl, err := url.Parse(connectString)
	if err != nil {
		fmt.Println(err)
         return
	}

	client, err = mongo.Connect(context.Background(), connectString)
	if err != nil {
		fmt.Println(err)
         return
	}

    db := client.Database(dbUrl.Path[1:])
    
    ctx := context.Background()
	defer db.Client().Disconnect(ctx)
	
	col := db.Collection("test")
	//先在事務外寫一條id爲“111”的記錄
    _,err = col.InsertOne(ctx, bson.M{"_id": "111", "name": "ddd", "age": 50})
    if(err != nil){
        fmt.Println(err)
        return
    }
	
	session,err := db.Client().StartSession()
	if(err != nil){
        fmt.Println(err)
        return
    }
    defer db.Client().EndSession()
    
    //開始事務
    err := session.StartTransaction()
    if(err != nil){
    	fmt.Println(err)
    	return
    }

    //在事務內寫一條id爲“222”的記錄
    _, err = col.InsertOne(ctx, bson.M{"_id": "222", "name": "ddd", "age": 50})
    if(err != nil){
    	fmt.Println(err)
    	return
    }

    //寫重複id
    _, err = col.InsertOne(ctx, bson.M{"_id": "111", "name": "ddd", "age": 50})
    if err != nil {
    	session.AbortTransaction(ctx)
    }else {
    	session.CommitTransaction(ctx)
    }
} 

//最終成功寫入的數據有"111","222"兩條，顯然，理所當然認爲的後兩條數據並沒有在事務內。

爲什麼讀寫操作沒有在事務內？找任何一個操作源碼看看，比如InsertOne:

//cellection.go

// InsertOne inserts a single document into the collection.
func (coll *Collection) InsertOne(ctx context.Context, document interface{},
	opts ...*options.InsertOneOptions) (*InsertOneResult, error) {

	if ctx == nil {
		ctx = context.Background()
	}

	doc, err := transformDocument(coll.registry, document)
	if err != nil {
		return nil, err
	}

	doc, insertedID := ensureID(doc)

    //下句很重要,從Context中獲取的session
	sess := sessionFromContext(ctx) 

	err = coll.client.ValidSession(sess)
	if err != nil {
		return nil, err
	}

	wc := coll.writeConcern
	if sess != nil && sess.TransactionRunning() {
		wc = nil
	}
	oldns := coll.namespace()
	cmd := command.Insert{
		NS:           command.Namespace{DB: oldns.DB, Collection: oldns.Collection},
		Docs:         []bsonx.Doc{doc},
		WriteConcern: wc,
		Session:      sess,
		Clock:        coll.client.clock,
	}

	// convert to InsertManyOptions so these can be argued to dispatch.Insert
	insertOpts := make([]*options.InsertManyOptions, len(opts))
	for i, opt := range opts {
		insertOpts[i] = options.InsertMany()
		insertOpts[i].BypassDocumentValidation = opt.BypassDocumentValidation
	}

    
	res, err := driver.Insert(
		ctx, cmd,
		coll.client.topology,
		coll.writeSelector,
		coll.client.id,
		coll.client.topology.SessionPool,
		coll.client.retryWrites,
		insertOpts...,
	)

	rr, err := processWriteError(res.WriteConcernError, res.WriteErrors, err)
	if rr&rrOne == 0 {
		return nil, err
	}

	return &InsertOneResult{InsertedID: insertedID}, err
}

//session.go

// sessionFromContext checks for a sessionImpl in the argued context and returns the session if it
// exists
func sessionFromContext(ctx context.Context) *session.Client {
    //從帶key-Value的Context裏取出session
	s := ctx.Value(sessionKey{})
	if ses, ok := s.(*sessionImpl); ses != nil && ok {
		return ses.Client
	}

	return nil
}

// driver/insert.go

func Insert(
	ctx context.Context,
	cmd command.Insert,
	topo *topology.Topology,
	selector description.ServerSelector,
	clientID uuid.UUID,
	pool *session.Pool,
	retryWrite bool,
	opts ...*options.InsertManyOptions,
) (result.Insert, error) {

	ss, err := topo.SelectServer(ctx, selector)
	if err != nil {
		return result.Insert{}, err
	}

     //Session爲nil，則新建一個默認的session
	// If no explicit session and deployment supports sessions, start implicit session.
	if cmd.Session == nil && topo.SupportsSessions() {
		cmd.Session, err = session.NewClientSession(pool, clientID, session.Implicit)
		if err != nil {
			return result.Insert{}, err
		}
		defer cmd.Session.EndSession()
	}

	insertOpts := options.MergeInsertManyOptions(opts...)

	if insertOpts.BypassDocumentValidation != nil && ss.Description().WireVersion.Includes(4) {
		cmd.Opts = append(cmd.Opts, bsonx.Elem{"bypassDocumentValidation", bsonx.Boolean(*insertOpts.BypassDocumentValidation)})
	}
	if insertOpts.Ordered != nil {
		cmd.Opts = append(cmd.Opts, bsonx.Elem{"ordered", bsonx.Boolean(*insertOpts.Ordered)})
	}

	// Execute in a single trip if retry writes not supported, or retry not enabled
	if !retrySupported(topo, ss.Description(), cmd.Session, cmd.WriteConcern) || !retryWrite {
		if cmd.Session != nil {
			cmd.Session.RetryWrite = false // explicitly set to false to prevent encoding transaction number
		}
		return insert(ctx, cmd, ss, nil)
	}

	// TODO figure out best place to put retry write.  Command shouldn't have to know about this field.
	cmd.Session.RetryWrite = retryWrite
	cmd.Session.IncrementTxnNumber()

	res, originalErr := insert(ctx, cmd, ss, nil)

	// Retry if appropriate
	if cerr, ok := originalErr.(command.Error); ok && cerr.Retryable() ||
		res.WriteConcernError != nil && command.IsWriteConcernErrorRetryable(res.WriteConcernError) {
		ss, err := topo.SelectServer(ctx, selector)

		// Return original error if server selection fails or new server does not support retryable writes
		if err != nil || !retrySupported(topo, ss.Description(), cmd.Session, cmd.WriteConcern) {
			return res, originalErr
		}

		return insert(ctx, cmd, ss, cerr)
	}

	return res, originalErr
}

它這樣設計，說明它希望在調用InsertOne裏傳入一個帶session-value的context對象給它。問題關鍵是：sessionFromContext()方法還只認sessionKey{}這個key，而sessionKey struct是沒有導出的，我們項目包裏是訪問不到，所以我們不能通過context.WithValue()方法建建一個session-value-context對象給它。

那麼它包內肯定有構建session-value-context對象的，那就找sessionKey{}的出現位置，發現：

// UseSession creates a default session, that is only valid for the
// lifetime of the closure. No cleanup outside of closing the session
// is done upon exiting the closure. This means that an outstanding
// transaction will be aborted, even if the closure returns an error.
//
// If ctx already contains a mongo.Session, that mongo.Session will be
// replaced with the newly created mongo.Session.
//
// Errors returned from the closure are transparently returned from
// this method.
func (c *Client) UseSession(ctx context.Context, fn func(SessionContext) error) error {
	return c.UseSessionWithOptions(ctx, options.Session(), fn)
}

// UseSessionWithOptions works like UseSession but allows the caller
// to specify the options used to create the session.
func (c *Client) UseSessionWithOptions(ctx context.Context, opts *options.SessionOptions, fn func(SessionContext) error) error {
	defaultSess, err := c.StartSession(opts)
	if err != nil {
		return err
	}

	defer defaultSess.EndSession(ctx) //如果事務沒有顯示的Commit，內部會Abort。
    
    //在這裏了
	sessCtx := sessionContext{
		Context: context.WithValue(ctx, sessionKey{}, defaultSess),
		Session: defaultSess,
	}

	return fn(sessCtx)
}

顯然，只有如下closure閉包方式才能使用事務接口，修改golang測試代碼如下：

import (
    "context"
    "github.com/mongodb/mongo-go-driver/mongo"
    "net/url"
    "fmt"
)

func main(){
    connectString := "mongodb://127.0.0.1/test"
    dbUrl, err := url.Parse(connectString)
    if err != nil {
        panic(err)
	}

    client, err = mongo.Connect(context.Background(), connectString)
    if err != nil {
        panic(err)
	}

    db := client.Database(dbUrl.Path[1:])
    
    ctx := context.Background()
    defer db.Client().Disconnect(ctx)
    
    col := db.Collection("test")
    
    //先在事務外寫一條id爲“111”的記錄
    _,err = col.InsertOne(ctx, bson.M{"_id": "111", "name": "ddd", "age": 50})
    if(err != nil){
        fmt.Println(err)
        return
    }
    
    //第一個事務：成功執行
    db.Client().UseSession(ctx, func(sessionContext mongo.SessionContext) error {
        err = sessionContext.StartTransaction()
        if(err != nil){
            fmt.Println(err)
            return err
        }

        //在事務內寫一條id爲“222”的記錄
        _, err = col.InsertOne(sessionContext, bson.M{"_id": "222", "name": "ddd", "age": 50})
        if(err != nil){
            fmt.Println(err)
            return err
        }

        //在事務內寫一條id爲“333”的記錄
        _, err = col.InsertOne(sessionContext, bson.M{"_id": "333", "name": "ddd", "age": 50})
        if err != nil {
            sessionContext.AbortTransaction(sessionContext)
            return err
        }else {
            sessionContext.CommitTransaction(sessionContext)
        }
        return nil
    })
	
    //第二個事務：執行失敗，事務沒提交，因最後插入了一條重複id "111",
    err = db.Client().UseSession(ctx, func(sessionContext mongo.SessionContext) error {
        err := sessionContext.StartTransaction()
        if(err != nil){
            fmt.Println(err)
            return err
        }

        //在事務內寫一條id爲“222”的記錄
        _, err = col.InsertOne(sessionContext, bson.M{"_id": "444", "name": "ddd", "age": 50})
        if(err != nil){
            fmt.Println(err)
            return err
        }

		//寫重複id
        _, err = col.InsertOne(sessionContext, bson.M{"_id": "111", "name": "ddd", "age": 50})
        if err != nil {
            sessionContext.AbortTransaction(sessionContext)
            return err
        }else {
            sessionContext.CommitTransaction(sessionContext)
        }
        return nil
    })
} 

//最終數據只有 "111","222","333" 三條，事務測試成功。