note
burrow code version: v0.30.3
Note that the blob show burrow consensus flow of single node.
ABCI Application
Burrow use tendermint(lib) as consensus engine, which finish consensus via abci.
// Application is an interface that enables any finite, deterministic state machine
// to be driven by a blockchain-based replication engine via the ABCI.
// All methods take a RequestXxx argument and return a ResponseXxx argument,
// except CheckTx/DeliverTx, which take `tx []byte`, and `Commit`, which takes nothing.
type Application interface {
// Info/Query Connection
Info(RequestInfo) ResponseInfo // Return application info
SetOption(RequestSetOption) ResponseSetOption // Set application option
Query(RequestQuery) ResponseQuery // Query for state
// Mempool Connection
CheckTx(RequestCheckTx) ResponseCheckTx // Validate a tx for the mempool
// Consensus Connection
InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain w validators/other info from TendermintCore
BeginBlock(RequestBeginBlock) ResponseBeginBlock // Signals the beginning of a block
DeliverTx(RequestDeliverTx) ResponseDeliverTx // Deliver a tx for full processing
EndBlock(RequestEndBlock) ResponseEndBlock // Signals the end of a block, returns changes to the validator set
Commit() ResponseCommit // Commit the state and return the application Merkle root hash
}
implement
type App struct {
// Node information to return in Info
nodeInfo string
// State
blockchain *bcm.Blockchain
validators Validators
mempoolLocker sync.Locker
authorizedPeers AuthorizedPeers
// We need to cache these from BeginBlock for when we need actually need it in Commit
block *types.RequestBeginBlock
// Function to use to fail gracefully from panic rather than letting Tendermint make us a zombie
panicFunc func(error)
checker execution.BatchExecutor
committer execution.BatchCommitter
txDecoder txs.Decoder
logger *logging.Logger
}
consensus flow
step1: checkTx
When client send BroadcastTx to node via transact grpc service, Transactor will call kern.Node.Mempool().CheckTx() to delivery tx to tendermint mempool, and then app.CheckTx() will be called by tendermint that application should check tx by its specified way(ie, check account permission and balance).
call stack
github.com/hyperledger/burrow/execution/contexts.(*CallContext).Execute at call_context.go:34
github.com/hyperledger/burrow/execution.(*executor).Execute at execution.go:254
github.com/hyperledger/burrow/consensus/abci.ExecuteTx at execute_tx.go:30
github.com/hyperledger/burrow/consensus/abci.(*App).CheckTx at app.go:189
github.com/tendermint/tendermint/abci/client.(*localClient).CheckTxAsync at local_client.go:99
github.com/tendermint/tendermint/proxy.(*appConnMempool).CheckTxAsync at app_conn.go:114
github.com/tendermint/tendermint/mempool.(*CListMempool).CheckTx at clist_mempool.go:281
github.com/tendermint/tendermint/mempool.Mempool.CheckTx-fm at mempool.go:18
github.com/hyperledger/burrow/execution.(*Transactor).CheckTxAsyncRaw at transactor.go:237
github.com/hyperledger/burrow/execution.(*Transactor).CheckTxSyncRaw at transactor.go:206
github.com/hyperledger/burrow/execution.(*Transactor).CheckTxSync at transactor.go:134
github.com/hyperledger/burrow/execution.(*Transactor).BroadcastTxSync at transactor.go:83
github.com/hyperledger/burrow/rpc/rpctransact.(*transactServer).BroadcastTxSync at transact_server.go:55
github.com/hyperledger/burrow/rpc/rpctransact._Transact_BroadcastTxSync_Handler.func1 at rpctransact.pb.go:505
github.com/hyperledger/burrow/rpc.unaryInterceptor.func1 at grpc.go:29
github.com/hyperledger/burrow/rpc/rpctransact._Transact_BroadcastTxSync_Handler at rpctransact.pb.go:507
google.golang.org/grpc.(*Server).processUnaryRPC at server.go:1024
google.golang.org/grpc.(*Server).handleStream at server.go:1313
google.golang.org/grpc.(*Server).serveStreams.func1.1 at server.go:722
runtime.goexit at asm_amd64.s:1357
- Async stack trace
google.golang.org/grpc.(*Server).serveStreams.func1 at server.go:720
CheckTx() check tx according to tx type. For example, call type will check permission of create/call contract, and account balance.
func (app *App) CheckTx(req types.RequestCheckTx) types.ResponseCheckTx {
const logHeader = "CheckTx"
defer func() {
if r := recover(); r != nil {
app.panicFunc(fmt.Errorf("panic occurred in abci.App/CheckTx: %v\n%s", r, debug.Stack()))
}
}()
checkTx := ExecuteTx(logHeader, app.checker, app.txDecoder, req.GetTx())
logger := WithEvents(app.logger, checkTx.Events)
if checkTx.Code == codes.TxExecutionSuccessCode {
logger.InfoMsg("Execution success")
} else {
logger.InfoMsg("Execution error",
"code", checkTx.Code,
"log", checkTx.Log)
}
return checkTx
}
Each type is corresponding to one context
type Type uint32
// Types of Payload implementations
const (
TypeUnknown = Type(0x00)
// Account transactions
TypeSend = Type(0x01)
TypeCall = Type(0x02)
TypeName = Type(0x03)
TypeBatch = Type(0x04)
// Validation transactions
TypeBond = Type(0x11)
TypeUnbond = Type(0x12)
// Admin transactions
TypePermissions = Type(0x21)
TypeGovernance = Type(0x22)
TypeProposal = Type(0x23)
TypeIdentify = Type(0x24)
)
Each type is corresponding to one context
// account
SendContext
CallContext
NameContext
// Validation
BondContext
UnboundContext
// admin
GovernanceContext
IdentifyContext
PermissionsContext
ProposalContext
step2: begin block
BeginBlock is used to check If validators given from tendermint is different from burrow.
func (app *App) BeginBlock(block types.RequestBeginBlock) (respBeginBlock types.ResponseBeginBlock) {
app.block = &block
defer func() {
if r := recover(); r != nil {
app.panicFunc(fmt.Errorf("panic occurred in abci.App/BeginBlock: %v\n%s", r, debug.Stack()))
}
}()
if block.Header.Height > 1 {
var err error
previousValidators := validator.NewTrimSet()
// Tendermint runs two blocks behind plus we are updating in end block validators updated last round
err = validator.Write(previousValidators,
app.validators.Validators(BurrowValidatorDelayInBlocks+TendermintValidatorDelayInBlocks))
if err != nil {
panic(fmt.Errorf("could not build current validator set: %v", err))
}
if len(block.LastCommitInfo.Votes) != previousValidators.Size() {
err = fmt.Errorf("Tendermint passes %d validators to BeginBlock but Burrow's has %d:\n %v",
len(block.LastCommitInfo.Votes), previousValidators.Size(), previousValidators.String())
panic(err)
}
for _, v := range block.LastCommitInfo.Votes {
err = app.checkValidatorMatches(previousValidators, v.Validator)
if err != nil {
panic(err)
}
}
}
return
}
call stack
github.com/hyperledger/burrow/consensus/abci.(*App).BeginBlock at app.go:135
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync at local_client.go:231
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync at app_conn.go:69
github.com/tendermint/tendermint/state.execBlockOnProxyApp at execution.go:280
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:131
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
step3: deliverTx
DeliverTx() execute tx according to tx type. new app state and block will be generated Once all DeliverTx are executed.
func (app *App) DeliverTx(req types.RequestDeliverTx) types.ResponseDeliverTx {
const logHeader = "DeliverTx"
defer func() {
if r := recover(); r != nil {
app.panicFunc(fmt.Errorf("panic occurred in abci.App/DeliverTx: %v\n%s", r, debug.Stack()))
}
}()
checkTx := ExecuteTx(logHeader, app.committer, app.txDecoder, req.GetTx())
logger := WithEvents(app.logger, checkTx.Events)
if checkTx.Code == codes.TxExecutionSuccessCode {
logger.InfoMsg("Execution success")
} else {
logger.InfoMsg("Execution error",
"code", checkTx.Code,
"log", checkTx.Log)
}
return DeliverTxFromCheckTx(checkTx)
}
call stack
github.com/hyperledger/burrow/execution/contexts.(*CallContext).Deliver at call_context.go:134
github.com/hyperledger/burrow/execution/contexts.(*CallContext).Execute at call_context.go:49
github.com/hyperledger/burrow/execution.(*executor).Execute at execution.go:254
github.com/hyperledger/burrow/consensus/abci.ExecuteTx at execute_tx.go:30
github.com/hyperledger/burrow/consensus/abci.(*App).DeliverTx at app.go:212
github.com/tendermint/tendermint/abci/client.(*localClient).DeliverTxAsync at local_client.go:88
github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync at app_conn.go:73
github.com/tendermint/tendermint/state.execBlockOnProxyApp at execution.go:293
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:131
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
step4: end block
EndBlock() is used to update validators.
func (app *App) EndBlock(reqEndBlock types.RequestEndBlock) types.ResponseEndBlock {
var validatorUpdates []types.ValidatorUpdate
defer func() {
if r := recover(); r != nil {
app.panicFunc(fmt.Errorf("panic occurred in abci.App/EndBlock: %v\n%s", r, debug.Stack()))
}
}()
err := app.validators.ValidatorChanges(BurrowValidatorDelayInBlocks).IterateValidators(func(id crypto.Addressable, power *big.Int) error {
app.logger.InfoMsg("Updating validator power", "validator_address", id.GetAddress(),
"new_power", power)
validatorUpdates = append(validatorUpdates, types.ValidatorUpdate{
PubKey: id.GetPublicKey().ABCIPubKey(),
// Must ensure power fits in an int64 during execution
Power: power.Int64(),
})
return nil
})
if err != nil {
panic(err)
}
return types.ResponseEndBlock{
ValidatorUpdates: validatorUpdates,
}
}
call stack
github.com/hyperledger/burrow/consensus/abci.(*App).EndBlock at app.go:228
github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync at local_client.go:239
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync at app_conn.go:77
github.com/tendermint/tendermint/state.execBlockOnProxyApp at execution.go:300
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:131
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
step5: commit
Commit() is used to commit state and block.
app.committer.Commit() is used to commit state and block which generated in deleverTx step.
app.blockchain.CommitBlock() is only used to record last block height and apphash which saved in db as checkpoint that we can load checkpoint Once restart.
If tendermint found the appHash passed in is different from its appHash, It will generate an empty block.
func (app *App) Commit() types.ResponseCommit {
defer func() {
if r := recover(); r != nil {
app.panicFunc(fmt.Errorf("panic occurred in abci.App/Commit: %v\n%s", r, debug.Stack()))
}
}()
blockTime := app.block.Header.Time
app.logger.InfoMsg("Committing block",
"tag", "Commit",
structure.ScopeKey, "Commit()",
"height", app.block.Header.Height,
"hash", app.block.Hash,
"block_time", blockTime,
"last_block_time", app.blockchain.LastBlockTime(),
"last_block_duration", app.blockchain.LastCommitDuration(),
"last_block_hash", app.blockchain.LastBlockHash(),
)
// Lock the checker while we reset it and possibly while recheckTxs replays transactions
app.checker.Lock()
defer func() {
// Tendermint may replay transactions to the check cache during a recheck, which happens after we have returned
// from Commit(). The mempool is locked by Tendermint for the duration of the commit phase; during Commit() and
// the subsequent mempool.Update() so we schedule an acquisition of the mempool lock in a goroutine in order to
// 'observe' the mempool unlock event that happens later on. By keeping the checker read locked during that
// period we can ensure that anything querying the checker (such as service.MempoolAccounts()) will block until
// the full Tendermint commit phase has completed.
if app.mempoolLocker != nil {
go func() {
// we won't get this until after the commit and we will acquire strictly after this commit phase has
// ended (i.e. when Tendermint's BlockExecutor.Commit() returns
app.mempoolLocker.Lock()
// Prevent any mempool getting relocked while we unlock - we could just unlock immediately but if a new
// commit starts gives goroutines blocked on checker a chance to progress before the next commit phase
defer app.mempoolLocker.Unlock()
app.checker.Unlock()
}()
} else {
// If we have not be provided with access to the mempool lock
app.checker.Unlock()
}
}()
appHash, err := app.committer.Commit(&app.block.Header)
if err != nil {
panic(errors.Wrap(err, "Could not commit transactions in block to execution state"))
}
err = app.checker.Reset()
if err != nil {
panic(errors.Wrap(err, "could not reset check cache during commit"))
}
// Commit to our blockchain state which will checkpoint the previous app hash by saving it to the database
// (we know the previous app hash is safely committed because we are about to commit the next)
err = app.blockchain.CommitBlock(blockTime, app.block.Hash, appHash)
if err != nil {
panic(fmt.Errorf("could not commit block to blockchain state: %v", err))
}
app.logger.InfoMsg("Committed block")
return types.ResponseCommit{
Data: appHash,
}
}
call stack
github.com/hyperledger/burrow/consensus/abci.(*App).Commit at app.go:258
github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync at local_client.go:215
github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync at app_conn.go:81
github.com/tendermint/tendermint/state.(*BlockExecutor).Commit at execution.go:212
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:166
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
empty block
It is strange that tendermint will generate an empty block instantly followed by previous block which contain of tx.
In fact, appHash and validatorsHash is recorded in header of next tendermint block, Specially validators that tendermint has newest info(ie, the power of validator).
So It is necessary to send an empty block to refresh validators info in application.
You can get it in abci.Header structure.
type Header struct {
// basic block info
Version Version `protobuf:"bytes,1,opt,name=version,proto3" json:"version"`
ChainID string `protobuf:"bytes,2,opt,name=chain_id,json=chainId,proto3" json:"chain_id,omitempty"`
Height int64 `protobuf:"varint,3,opt,name=height,proto3" json:"height,omitempty"`
Time time.Time `protobuf:"bytes,4,opt,name=time,proto3,stdtime" json:"time"`
// prev block info
LastBlockId BlockID `protobuf:"bytes,5,opt,name=last_block_id,json=lastBlockId,proto3" json:"last_block_id"`
// hashes of block data
LastCommitHash []byte `protobuf:"bytes,6,opt,name=last_commit_hash,json=lastCommitHash,proto3" json:"last_commit_hash,omitempty"`
DataHash []byte `protobuf:"bytes,7,opt,name=data_hash,json=dataHash,proto3" json:"data_hash,omitempty"`
// hashes from the app output from the prev block
ValidatorsHash []byte `protobuf:"bytes,8,opt,name=validators_hash,json=validatorsHash,proto3" json:"validators_hash,omitempty"`
NextValidatorsHash []byte `protobuf:"bytes,9,opt,name=next_validators_hash,json=nextValidatorsHash,proto3" json:"next_validators_hash,omitempty"`
ConsensusHash []byte `protobuf:"bytes,10,opt,name=consensus_hash,json=consensusHash,proto3" json:"consensus_hash,omitempty"`
AppHash []byte `protobuf:"bytes,11,opt,name=app_hash,json=appHash,proto3" json:"app_hash,omitempty"`
LastResultsHash []byte `protobuf:"bytes,12,opt,name=last_results_hash,json=lastResultsHash,proto3" json:"last_results_hash,omitempty"`
// consensus info
EvidenceHash []byte `protobuf:"bytes,13,opt,name=evidence_hash,json=evidenceHash,proto3" json:"evidence_hash,omitempty"`
ProposerAddress []byte `protobuf:"bytes,14,opt,name=proposer_address,json=proposerAddress,proto3" json:"proposer_address,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
step6: Begin Block
call stack is similar to BeginBlock above.
github.com/hyperledger/burrow/consensus/abci.(*App).BeginBlock at app.go:135
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync at local_client.go:231
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync at app_conn.go:69
github.com/tendermint/tendermint/state.execBlockOnProxyApp at execution.go:280
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:131
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
step7: End Block
call stack is similar to endBlock above.
github.com/hyperledger/burrow/consensus/abci.(*App).EndBlock at app.go:228
github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync at local_client.go:239
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync at app_conn.go:77
github.com/tendermint/tendermint/state.execBlockOnProxyApp at execution.go:300
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:131
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335
step8: Commit
call stack is similar to commit above.
github.com/hyperledger/burrow/consensus/abci.(*App).Commit at app.go:258
github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync at local_client.go:215
github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync at app_conn.go:81
github.com/tendermint/tendermint/state.(*BlockExecutor).Commit at execution.go:212
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock at execution.go:166
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit at state.go:1431
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit at state.go:1350
github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1 at state.go:1285
github.com/tendermint/tendermint/consensus.(*State).enterCommit at state.go:1322
github.com/tendermint/tendermint/consensus.(*State).addVote at state.go:1819
github.com/tendermint/tendermint/consensus.(*State).tryAddVote at state.go:1642
github.com/tendermint/tendermint/consensus.(*State).handleMsg at state.go:709
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine at state.go:660
runtime.goexit at asm_amd64.s:1357
- Async stack trace
github.com/tendermint/tendermint/consensus.(*State).OnStart at state.go:335