git submodule python實現

概述

Git 原生的sumodules 功能無法實現將子模塊存放在父級目錄， 而放入子目錄時候會存在多份代碼，交叉依賴的問題。所以需要搭建一個腳本工具來解決多個模塊之間的依賴關係。

解決模塊依賴

• 通過配置文件指定git 倉庫url、名稱、分支、版本

• 支持解析原生的.gitmodules 文件 （優先解析）

[submodule "modules/my-module-base"]
	path = modules/my-module-base
	url = http://gitlab.xxx.com/my-project/my-module-base.git
[submodule "modules/my-module-config"]
	path = modules/my-module-config
	url = http://gitlab.xxx.com/my-project/my-module-config.git

• 支持dependent.jsong格式配置， 第一次自動生成模板json文件，需要手動再修改成項目所依賴的子模塊

  {
      "my-module-base": {
          "url": "http://gitlab.xxx.com/my-project/my-module-base.git",
          "path": "../base",
          "branch": "master",
          "version": ""
      },
      "my-module-config": {
          "url": "http://gitlab.xxx.com/my-project/my-module-config.git",
          "path": "../config",
          "branch": "master",
          "version": ""
      }
  }
  

  .gitmodules 和 depentent.json 二選一，前者 優先解析

• 日誌文件git_multi_module.log

  2021-04-16 16:48:25,347 MainThread   INFO   151: Git multi-modules checkout
  2021-04-16 16:48:25,347 MainThread   INFO    53: No search `.gitmodules` file, sik to git submodules parser.
  2021-04-16 16:48:25,347 MainThread   INFO    90: Load ./dependent.json file.
  2021-04-16 16:48:25,347 MainThread   INFO   111: Clone http://gitlab.xxx.com/my-project/my-module-base.git master to ../base
  2021-04-16 16:48:27,936 MainThread   INFO   114: Done
  2021-04-16 16:48:27,936 MainThread   INFO   111: Clone http://gitlab.xxx.com/my-project/my-module-config.git master to ../config
  2021-04-16 16:48:29,477 MainThread   INFO   114: Done
  

使用步驟

• 打包腳本 pyinstaller git_multi_module.py -F 生的Exe;
• 拷貝git_multi_module.exe文件到工程目錄；
• 雙擊exe文件運行直接解析.submodule 文件，如果存在則解析並跳過後面步驟，否則檢測並生產dependent.json 文件；
• 修改dependent.json 成指定模塊地址；
• 再次運行git_multi_module.exe文件拉取子模塊代碼；

Visual stdio 配置

• 拉取 developer-manual 工程倉：http://gitlab.xxx.com/my-project/developer-manual.git
• 同記錄拉取開發工程如： my-module-config, 配置依賴模塊， 如：

{
    "my-module-base": {
        "url": "http://gitlab.xxx.com/my-project/my-module-base.git",
        "path": "my-module-base",
        "branch": "master",
        "version": ""
    }
}

• 打開VS 中的 項目 -》 屬性 -》 配置屬性 -》 生成事件 -》 預先生成事件 -》 命令行 填寫入：

$(RootDir)developer-manual\tools\git_submodules\git_multi_module.exe $(SolutionDir)dependent.json $(RootDir) 1

• 編譯前自動拉取依賴倉

1>------ 已啓動生成:  項目: my_module_config, 配置: Debug Win32 ------
1>  2021-04-19 14:49:11,670 MainThread   INFO   156: Git multi-modules checkout
1>  2021-04-19 14:49:11,670 MainThread   INFO    40: Root path: D:\xxx_workplace\my-module-config\..\
1>  2021-04-19 14:49:11,670 MainThread   INFO    41: Config file: D:\xxx_workplace\my-module-config\dependent.json
1>  2021-04-19 14:49:11,671 MainThread   INFO    42: Reset before pull: True
1>  2021-04-19 14:49:11,671 MainThread   INFO    58: No search `.gitmodules` file, skip git submodules parser.
1>  2021-04-19 14:49:11,672 MainThread   INFO    95: Load D:\xxx_workplace\my-module-config\dependent.json file.
1>  2021-04-19 14:49:11,672 MainThread   INFO   103: Pull http://gitlab.xxx.com/my-project/my-module-base.git master to D:\xxx_workplace\my-module-base
1>  2021-04-19 14:49:12,542 MainThread   INFO   109: Done
1>  my_module_config.vcxproj -> D:\xxx_workplace\xxx-module-config\..\bin\Debug\my_module_config.dll

工具腳本源碼

git_multi_module.py

from logging.handlers import RotatingFileHandler

import os
import json
import logging
from git import Repo


class GitMultiModule(object):
    dependent_file = "./dependent.json"
    git_modules_file = ".gitmodules"

    dependent_template = {
        "pos-module-base": {
            "url": "https://github.com/xxx/xxx.git",
            "path": "",
            "branch": "master",
            "version": ""
        }
    }

    _dependents_ = dict()

    def __init__(self, file_name="", root_path_="./",  reset_before_pull_=False):
        if file_name:
            self.dependent_file = file_name
        self.root_path = root_path_
        self.reset_before_pull = reset_before_pull_
        logging.info("Root path: {}".format(self.root_path))
        logging.info("Config file: {}".format(self.dependent_file))
        logging.info("Reset before pull: {}".format(self.reset_before_pull))

        if not self.load_git_modules_file():
            self.load_file()

    def make_dependent_template(self) -> bool:
        logging.info("Make dependent template json file...")
        with open(self.dependent_file, "w", encoding="utf-8") as f:
            json.dump(self.dependent_template, f, ensure_ascii=False, indent=4)
        logging.info("Done")
        msg = "Please configure the `{}` file with your project and restart this tools"
        logging.info(msg.format(self.dependent_file))
        return False

    def load_git_modules_file(self) -> bool:
        if not os.path.exists(self.git_modules_file):
            logging.info("No search `{}` file, skip git submodules parser.".format(self.git_modules_file))
            return False

        logging.info("Loading `{}` file...".format(self.git_modules_file))
        from configparser import ConfigParser
        cf = ConfigParser()
        cf.read(self.git_modules_file)
        modules = cf.sections()
        git_submodules = dict()

        for section in modules:
            try:
                _, name, _ = section.split('"')
                m_path = cf.get(section, "path")
                m_url = cf.get(section, "url")
                m_version = cf.has_option(section, "version") and cf.get(section, "version") or ""
                m_branch = cf.has_option(section, "branch") and cf.get(section, "branch") or "master"
                git_submodules[name] = dict(path=m_path, url=m_url, branch=m_branch, version=m_version)
            except Exception as err:
                logging.error("Load submodules section error:{}".format(err))
        self._dependents_ = git_submodules

        if git_submodules:
            return True
        else:
            return False

    def load_file(self) -> bool:
        if not os.path.exists(self.dependent_file):
            logging.error("No search `{}` file!".format(self.dependent_file))
            return self.make_dependent_template()

        try:
            with open(self.dependent_file, encoding="utf-8") as f:
                json_obj = json.load(f)
                if isinstance(json_obj, dict):
                    self._dependents_ = json_obj
                    logging.info("Load {} file.".format(self.dependent_file))
                    return True
        except Exception as err:
            logging.error("Load dependent file error:{}".format(err))
        return False

    def pull(self, m_url, m_branch, m_path, m_version):
        try:
            logging.info("Pull {} {} to {}".format(m_url, m_branch, m_path))
            r = Repo(m_path)
            self.reset_before_pull and r.git.execute("git reset HEAD --hard")
            if r.active_branch.name != m_branch:
                r.git.checkout(m_branch)
            r.remote().pull()
            logging.info("Done")
        except Exception as err:
            _ = self
            logging.error("Pull error:{}".format(err))

    def clone(self, m_url, m_branch, m_path, m_version):
        try:
            logging.info("Clone {} {} to {}".format(m_url, m_branch, m_path))
            r = Repo.clone_from(m_url, m_path)  # 拉取遠程代碼
            r.git.checkout(m_branch)
            logging.info("Done")
        except Exception as err:
            _ = self
            logging.error("Pull clone {}".format(err))

    def clone_or_pull(self, m_url, m_branch, m_path, m_version):
        m_path = os.path.abspath(self.root_path + m_path)
        git_path = os.path.join(m_path, ".git")
        if os.path.isdir(git_path):
            self.pull(m_url, m_branch, m_path, m_version)
        else:
            self.clone(m_url, m_branch, m_path, m_version)

    def check_modules(self):
        if not self._dependents_:
            logging.error("Dependent configuration error!")
            return

        for module_name, config in self._dependents_.items():
            m_url = config.get("url")
            m_path = config.get("path")
            m_version = config.get("version") or ""
            m_branch = config.get("branch") or "master"
            self.clone_or_pull(m_url, m_branch, m_path, m_version)


if __name__ == '__main__':
    log_fmt = r"%(asctime)s %(threadName)s %(levelname)6s %(lineno)5d: %(message)s"
    log_file = "git_multi_module.log"
    log_size = 1024 * 124 * 5
    handlers = [
        logging.StreamHandler(),
        RotatingFileHandler(log_file, maxBytes=log_size, backupCount=5, encoding="utf-8")
    ]
    log_param = dict(level=logging.INFO, handlers=handlers, format=log_fmt)
    logging.basicConfig(**log_param)

    logging.info("Git multi-modules checkout")

    import sys
    argv_len = len(sys.argv)
    config_file = argv_len > 1 and sys.argv[1] or ""
    root_path = argv_len > 2 and sys.argv[2] or "./"
    reset = argv_len > 3 and sys.argv[3] == "1" or False

    g = GitMultiModule(config_file, root_path, reset)
    g.check_modules()