关于看kubelet启动代码，本来是想了解一个问题，就是关于CNI，二进制部署的时候，使用flannel组网，在启动kublet时并没有传递network-plugin参数，那必然有默认值。
其实flannel的本质也是使用brigde方式进行pod的网络接口配置操作，这和docker本身默认的docker0是一致的，但是这里需要明白一点的kubelet的network-plugin的参数最后是传递给dockershim的来处理的，因此dockershim去判断使用docker自己的还是cni插件去做。无论是docker自己去做还是flannel去做，是dockershim决定的。如何决定的呢？
因此有两个参数需要深究:
从kubelet的启动参数
| --network-plugin | <Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle. This docker-specific flag only works when container-runtime is set to docker. |
| --container-runtime | The container runtime to use. Possible values: 'docker', 'remote', 'rkt (deprecated)'. (default "docker") |
//pkg/kubelet/dockershim/docker_service.go
func NewDockerService(config *ClientConfig, podSandboxImage string, streamingConfig *streaming.Config, pluginSettings *NetworkPluginSettings,
...
// dockershim currently only supports CNI plugins.
pluginSettings.PluginBinDirs = cni.SplitDirs(pluginSettings.PluginBinDirString)
cniPlugins := cni.ProbeNetworkPlugins(pluginSettings.PluginConfDir, pluginSettings.PluginBinDirs)
cniPlugins = append(cniPlugins, kubenet.NewPlugin(pluginSettings.PluginBinDirs))
netHost := &dockerNetworkHost{
&namespaceGetter{ds},
&portMappingGetter{ds},
}
plug, err := network.InitNetworkPlugin(cniPlugins, pluginSettings.PluginName, netHost, pluginSettings.HairpinMode, pluginSettings.NonMasqueradeCIDR, pluginSettings.MTU)
if err != nil {
return nil, fmt.Errorf("didn't find compatible CNI plugin with given settings %+v: %v", pluginSettings, err)
}
ds.network = network.NewPluginManager(plug)
glog.Infof("Docker cri networking managed by %v", plug.Name())
...
}
// pkg/kubelet/dockershim/network/plugin.go
// InitNetworkPlugin inits the plugin that matches networkPluginName. Plugins must have unique names.
func InitNetworkPlugin(plugins []NetworkPlugin, networkPluginName string, host Host, hairpinMode kubeletconfig.HairpinMode, nonMasqueradeCIDR string, mtu int) (NetworkPlugin, error) {
if networkPluginName == "" {
// default to the no_op plugin
plug := &NoopNetworkPlugin{}
plug.Sysctl = utilsysctl.New()
if err := plug.Init(host, hairpinMode, nonMasqueradeCIDR, mtu); err != nil {
return nil, err
}
return plug, nil
}
pluginMap := map[string]NetworkPlugin{}
从以上代码可以看出，我们在二进制部署的时候，是没有配置network-plugin参数的，即代码中会进入NoopNetworkPlugin分支，
从/var/log/message中也能看到kubelet的日志“Docker cri networking managed by kubernetes.io/no-op"。
一句话：dockershim会判断传递的network-plugin参数，没配怎么处理，配了怎么处理。
以下代码是看kubelet的默认参数的是如何定义的。
（一） kubelet的启动入口：
app.NewKubeletCommand（）
#kubernets/cmd/kubelet/kubelet.go
func main() {
rand.Seed(time.Now().UTC().UnixNano())
command := app.NewKubeletCommand(server.SetupSignalHandler())
logs.InitLogs()
defer logs.FlushLogs()
if err := command.Execute(); err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
os.Exit(1)
}
}
# kubernetes/cmd/app/server.go
func NewKubeletCommand(stopCh <-chan struct{}) *cobra.Command {
cleanFlagSet := pflag.NewFlagSet(componentKubelet, pflag.ContinueOnError)
cleanFlagSet.SetNormalizeFunc(flag.WordSepNormalizeFunc)
kubeletFlags := options.NewKubeletFlags()
kubeletConfig, err := options.NewKubeletConfiguration()
// programmer error
if err != nil {
glog.Fatal(err)
}
... （中间将来会运行）
// keep cleanFlagSet separate, so Cobra doesn't pollute it with the global flags
kubeletFlags.AddFlags(cleanFlagSet)
options.AddKubeletConfigFlags(cleanFlagSet, kubeletConfig)
options.AddGlobalFlags(cleanFlagSet)
cleanFlagSet.BoolP("help", "h", false, fmt.Sprintf("help for %s", cmd.Name()))
以上代码中，options.NewKubeletFlags，以及下面的kubeletFlays.AddFlags是将配置参数默认值的地方
这里options.NewKubeletFlags中，会初始化kubelet的各种配置。
func NewKubeletFlags() *KubeletFlags {
remoteRuntimeEndpoint := ""
if runtime.GOOS == "linux" {
remoteRuntimeEndpoint = "unix:///var/run/dockershim.sock"
} else if runtime.GOOS == "windows" {
remoteRuntimeEndpoint = "tcp://localhost:3735"
}
return &KubeletFlags{
EnableServer: true,
ContainerRuntimeOptions: *NewContainerRuntimeOptions(),
CertDirectory: "/var/lib/kubelet/pki",
RootDirectory: defaultRootDir,
MasterServiceNamespace: metav1.NamespaceDefault,
MaxContainerCount: -1,
MaxPerPodContainerCount: 1,
MinimumGCAge: metav1.Duration{Duration: 0},
NonMasqueradeCIDR: "10.0.0.0/8",
RegisterSchedulable: true,
ExperimentalKernelMemcgNotification: false,
RemoteRuntimeEndpoint: remoteRuntimeEndpoint,
NodeLabels: make(map[string]string),
VolumePluginDir: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/",
RegisterNode: true,
SeccompProfileRoot: filepath.Join(defaultRootDir, "seccomp"),
HostNetworkSources: []string{kubetypes.AllSource},
HostPIDSources: []string{kubetypes.AllSource},
HostIPCSources: []string{kubetypes.AllSource},
// TODO(#58010:v1.13.0): Remove --allow-privileged, it is deprecated
AllowPrivileged: true,
// prior to the introduction of this flag, there was a hardcoded cap of 50 images
NodeStatusMaxImages: 50,
}
}
在这里有linux环境下，RemoteRuntimeEndpoint的定义。也有ContaineRuntimeOptions的初始化
func NewContainerRuntimeOptions() *config.ContainerRuntimeOptions {
dockerEndpoint := ""
if runtime.GOOS != "windows" {
dockerEndpoint = "unix:///var/run/docker.sock"
}
return &config.ContainerRuntimeOptions{
ContainerRuntime: kubetypes.DockerContainerRuntime,
RedirectContainerStreaming: false,
DockerEndpoint: dockerEndpoint,
DockershimRootDirectory: "/var/lib/dockershim",
PodSandboxImage: defaultPodSandboxImage,
ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute},
ExperimentalDockershim: false,
//Alpha feature
CNIBinDir: "/opt/cni/bin",
CNIConfDir: "/etc/cni/net.d",
}
}
====================分割线===================
之所以有上面的东西，其实我本意是想知道，既然我们在二进制部署时，没有配置network-plugin，那么kubelet在创建pod时，究竟容器的网络是如何创建的。
从配置中能看到的是flannel启动时，将本节点的网段传递了docker启动参数。docker启动后会以此设置docker0网桥的网段。
其实我们有以下几个疑问:
- flannel这个插件的工作原理，为什么network-plugin没有体现flannel， flannel最终实现跨节点的pod通信？
- kubelet对network-plugin参数的默认值是多少？针对该配置，对pod的网络流程的影响？
（一）先回答第一个问题：
flannel在cni中是一个相对于calico等第三方cni比较特殊的一个插件，特殊在哪呢？https://github.com/containernetworking/plugins中也有说明，flannel属于Meta: other plugins类，不能独立工作，需要有Main类支持才可以。而flannel本质是调用Main类中的bridge，这个和docker 默认的NetworkMode是一致的。因此二进制部署的时候，kubelet没有指定flannel相关的配置，pod的网络也配好了，肯定是docker自己配置的。
（二）再来看第二个问题：