kaldi win10 vs2019 CPU和GPU下遇到的問題

1 需要組件

請按照順序安裝。

1.1 nvidia driver

nvidia win10 顯卡型號和驅動下載地址直接雙擊exe安裝
cuda_10.2.89_441.22_win10.exe下載地址直接雙擊exe安裝
cudnn-10.2-windows10-x64-v7.6.5.32.zip下載地址
請將cudnn解壓的庫複製到C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2下即可。
【配置PATH環境變量】

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\libnvvp
C:\Program Files\NVIDIA Corporation\NVSMI

1.2 vs2019社區版

vs2019_community__1503913078.1588313013.exe vs社區版
C++桌面版全選即可

1.3 cmake

cmake-3.17.3-win64-x64.zip，解壓到C:\ProgramFiles下
【或者】不用下載了，vs2019內置了make，將其添加到PATH中即可
【配置PATH環境變量】

C:\ProgramFiles\cmake-3.7.0-win64-x64\bin
# 或者
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin

1.4 git和TortoiseGit

請安裝Git_V2.5.1_64_bit_setup.1441791170.exe
和TortoiseGit-1.8.15.0-64bit.msi
以及TortoiseGit-LanguagePack-1.8.15.0-64bit-zh_CN.msi中文語言包
【因爲所有相關操作都在git bash上操作，windos命令行不支持mv cp等命令】

2 kaldi vs工程配置

2.1 kaldi

在這裏可以看一下kaldi/windows文件夾的Reda

git clone https://github.com/kaldi-asr/kaldi.git C:\anjos\vs\kaldi-win

2.2 拷貝變量文件

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【複製 variables.props.dev 一份爲 variables.props】
(2) 【複製 kaldiwin_openblas.props 一份爲 kaldiwin.props】
(3) 【複製 cuda_7.0.props 爲 cuda_10.2.props】
(4) 【複製portaudio.props 爲 portaudio.props.dev】

2.3 支持cuda GPU

因爲安裝了cudav10.2，所以發現cuda7.0已經過時，同時發現C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\BuildCustomizations下是**CUDA 10.2.props**。因此做如下變更：
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【將 generate_solution.pl 文件中的所有cuda 7.0替換爲cuda 10.2】
(2) 【將 generate_solution.pl 文件中的所有cuda_7.0替換爲cuda_10.2】
(3) 【在 variables.props 的<PropertyGroup Label="UserMacros">中增加】

<CUDADIR>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2</CUDADIR>
<NVTOOLSDIR>C:\Program FIles\NVIDIA Corporation\NvToolsExt\</NVTOOLSDIR>

(4) 【在variables.props的<ItemGroup>中修改】

    <BuildMacro Include="NVTOOLSDIR">
      <Value>$(NVTOOLSDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>

(5)【在 cuda_10.2.props <AdditionalIncludeDirectories> 子項下添加修改 cuda的include文件夾】

<AdditionalIncludeDirectories>$(CUDADIR)\include;$(NVTOOLSDIR)\include;$(CUBDIR);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>

(6)【在 cuda_10.2.props <AdditionalLibraryDirectories> 子項下添加修改 cuda的lib文件夾】

<AdditionalLibraryDirectories>$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib\x64;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

(7)【在 cuda_10.2.props <AdditionalDependencies> 子項下添加修改 cuda的依賴庫】

<AdditionalDependencies>cublas.lib;cuda.lib;cudart.lib;nvToolsExt64_1.lib;%(AdditionalDependencies)</AdditionalDependencies>

最後樣子：

  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(CUDADIR)\include;$(NVTOOLSDIR)\include;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_CUDA;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>cublas.lib;cuda.lib;cudart.lib;nvToolsExt64_1.lib;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>

2.4 支持vs 2019

在C:\anjos\vs\kaldi-win\windows下操作，將generate_solution.pl文件中增加vs2019相關：

my %TOOLS=( default=> "14.1",
            vs2015 => "14.0",
            vs2017 => "14.1",
			vs2019 => "v14.2"
            );
my %FORMAT=( default=> "14.10",
             vs2015 =>  "14.00",
             vs2017 =>  "14.10",
			 vs2019 => "v14.20"
             );
my %TOOLSET=( default=> "v141",
              vs2015 => "v140",
              vs2017 => "v141",
			  vs2019 => "v142"
              );

2.5 openfst

這時候配置openfst發現直接用源碼其實是不ok的，kaldi用的openfst版本是1.6.2，不能直接拉openfst GitHub最新版。這裏openfst官網有提供一個版本：

wget http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.2.tar.gz
wget https://pan.baidu.com/s/1o8SghqM
tar -xvf openfst-1.6.2.tar.gz
patch -b -p1 < openfst-1.6.2.patch

以vs2019方式打開C:/anjos/vs/kaldi-win/tools/openfst/openfst.sln，選擇x64/Release & Debug配置分別進行編譯即可，產生庫在C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64.

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在 variables.props 的 <PropertyGroup Label="UserMacros"> 中修改】：

<OPENFST>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\</OPENFST>
<OPENFSTLIB>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64</OPENFSTLIB>

(2) 【在 openfstwin_debug.props 和 openfstwin_release.props 修改將<AdditionalDependencies>子項下內容】：

<AdditionalDependencies>libfst.lib;%(AdditionalDependencies)</AdditionalDependencies>

(2) 【在 openfstwin_debug.props 和 openfstwin_release.props 修改將<AdditionalLibraryDirectories>子項下內容】【這個要區分環境】：

<AdditionalLibraryDirectories>$(OPENFSTLIB)\Release;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

<AdditionalLibraryDirectories>$(OPENFSTLIB)\Debug;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

2.6 openblas

下載OpenBLAS-v0.2.14-Win64-int32.zip，解壓至C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32
下載mingw64_dll.zip，解壓至C:\anjos\vs\kaldi-win\tools\mingw64_dll
在C:\anjos\vs\kaldi-win\windows下操作：
【在variables.props的<PropertyGroup Label="UserMacros">中修改】

<OPENBLASDIR>C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32</OPENBLASDIR>

2.7 PortAudio庫

wget http://www.portaudio.com/archives/pa_stable_v190600_20161030.tgz -O C:\anjos\vs\kaldi-win\tools
wget http://www.steinberg.net/sdk_downloads/asiosdk2.3.zip -O C:\anjos\vs\kaldi-win\tools
#C:\anjos\vs\kaldi-win\tools\portaudio
tar -xvf pa_stable_v190600_20161030.tgz
#C:\anjos\vs\kaldi-win\tools\ASIOSDK2.3
unzip asiosdk.zip
# C:\anjos\vs\kaldi-win\tools\portaudio\src\hostapi\asio\ASIOSDK下
# 是C:\anjos\vs\kaldi-win\tools\ASIOSDK2.3的一系列文件
mv ASIOSDK2.3/* portaudio/src/hostapi/asio/ASIOSDK

msvc\portaudio.sln，選擇x64/Release配置進行編譯
【打開 C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\vportaudio.sln，分別在 x64|Debug 和 x64|Release 下編譯生成項目】
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在variables.props的<PropertyGroup Label="UserMacros">中修改】

<PORTAUDIO>C:\anjos\vs\kaldi-win\tools\portaudio</PORTAUDIO>
<PORTAUDIOLIB>C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\x64</PORTAUDIOLIB>

(2)【在 variables.props 的 <ItemGroup> 中修改】

    <BuildMacro Include="PORTAUDIO">
      <Value>$(PORTAUDIO)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
      <BuildMacro Include="PORTAUDIOLIB">
      <Value>$(PORTAUDIOLIB)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>

(3)【在 kaldiwin.props <AdditionalIncludeDirectories> 子項下添加修改 PortAudio的include文件夾】

<AdditionalIncludeDirectories>..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common</AdditionalIncludeDirectories>

(4)【在 kaldiwin.props <AdditionalLibraryDirectories> 子項下添加修改 PortAudio的lib文件夾】

<AdditionalLibraryDirectories>$(OPENBLASDIR)\lib\;%(AdditionalLibraryDirectories);$(PORTAUDIOLIB)\Debug</AdditionalLibraryDirectories>

(5)【在 kaldiwin.props <AdditionalDependencies> 子項下添加修改 PortAudio的依賴庫】

<AdditionalDependencies>libopenblas.dll.a;%(AdditionalDependencies);portaudio_x64.lib;pa_ringbuffer.obj</AdditionalDependencies>

【對variables.props也做一樣修改如下：】

portaudio.props最後樣子：

  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(PORTAUDIO)\src\common;$(PORTAUDIO)\src\os\win;$(PORTAUDIO)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common</AdditionalIncludeDirectories>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(PORTAUDIOLIB);$(PORTAUDIOLIB)\Debug;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>portaudio_x64.lib;pa_ringbuffer.obj;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>

2.8 pthread-win

wget http://downloads.sourceforge.net/project/pthreads4w/pthreads-w32-2-9-1-release.zip
mkdir pthreads
cd pthreads
unzip C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release.zip

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在 variables.props 的 <PropertyGroup Label="UserMacros"> 中修改】：

<PTHREADW>C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release\Pre-built.2</PTHREADW>

2.9 cub庫

cub-1.8.0下載地址
也可以直接用nvidia內嵌的cub版本C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust\system\cuda\detail（這裏面有一個cub文件夾）：
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在variables.props的<PropertyGroup Label="UserMacros">中修改】

<CUBDIR>C:\anjos\vs\kaldi-win\tools\cub-1.8.0</CUBDIR>

(2) 【在kaldiwin.props的<AdditionalIncludeDirectories>中增加變量】

<AdditionalIncludeDirectories>..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common;$(CUBDIR)</AdditionalIncludeDirectories>

2.10 最後庫界面

(1) 【kaldiwin.props】

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ImportGroup Label="PropertySheets" />
  <PropertyGroup Label="UserMacros">
    <!-- Change the following paths so they are correct on your machine -->
    <!-- Do not modify anything before this line -->
    <MKLDIR>C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\</MKLDIR>
    <OPENBLASDIR>C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32</OPENBLASDIR>
    <OPENFST>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2</OPENFST>
    <OPENFSTLIB>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64</OPENFSTLIB>
    <CUBDIR>C:\anjos\vs\kaldi-win\tools\cub-1.8.0</CUBDIR>
    <PORTAUDIO>C:\anjos\vs\kaldi-win\tools\portaudio</PORTAUDIO>
    <PORTAUDIOLIB>C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\x64</PORTAUDIOLIB>
	<PTHREADW>C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release\Pre-built.2</PTHREADW>
	<NVTOOLSDIR>C:\Program FIles\NVIDIA Corporation\NvToolsExt</NVTOOLSDIR>
	<CUDADIR>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2</CUDADIR>
    <!-- Do not modify anything after this line -->
  </PropertyGroup>
  <PropertyGroup />
  <ItemDefinitionGroup />
  <ItemGroup>
    <BuildMacro Include="OPENBLASDIR">
      <Value>$(OPENBLASDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="OPENFST">
      <Value>$(OPENFST)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="OPENFSTLIB">
      <Value>$(OPENFSTLIB)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="MKLDIR">
      <Value>$(MKLDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="CUBDIR">
      <Value>$(CUBDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="PORTAUDIO">
      <Value>$(PORTAUDIO)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
      <BuildMacro Include="PORTAUDIOLIB">
      <Value>$(PORTAUDIOLIB)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="NVTOOLSDIR">
      <Value>$(NVTOOLSDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
  </ItemGroup>
</Project>

(2)【kaldiwin.props】

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalOptions>/bigobj  %(AdditionalOptions)</AdditionalOptions>
      <AdditionalIncludeDirectories>..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common;$(CUBDIR)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_OPENBLAS;HAVE_LAPACK_CONFIG_H;LAPACK_COMPLEX_STRUCTURE;USE_ONLY_PRERECORDED;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(OPENBLASDIR)\lib\;%(AdditionalLibraryDirectories);$(PORTAUDIOLIB)\Debug</AdditionalLibraryDirectories>
      <AdditionalDependencies>libopenblas.dll.a;%(AdditionalDependencies);portaudio_x64.lib;pa_ringbuffer.obj</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

(3)【cuda_10.2.props】

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(CUDADIR)\include;$(NVTOOLSDIR)\include;$(CUBDIR);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_CUDA;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib\x64;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>cublas.lib;cuda.lib;cudart.lib;nvToolsExt64_1.lib;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

(4) 【portaudio.props】

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(PORTAUDIO)\src\common;$(PORTAUDIO)\src\os\win;$(PORTAUDIO)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common</AdditionalIncludeDirectories>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(PORTAUDIOLIB);$(PORTAUDIOLIB)\Debug;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>portaudio_x64.lib;pa_ringbuffer.obj;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

3 kaldi vs工程生成

3.1 生成腳本

# 僅CPU
./generate_solution.pl --vsver vs2019 --enable-openblas
# 或也可以啓用cuda GPU(這裏我使用的是GPU)
./generate_solution.pl --vsver vs2019 --enable-cuda --enable-openblas
get_version.pl

如果僅是啓用CPU會出現錯誤信息，不用管：

ERROR?: file \c\anjos\vs\kaldi-win\windows\..\src\cudadecoder\batched-static-nnet3-kernels.cc not found - project kaldi-cudadecoder
ERROR?: file \c\anjos\vs\kaldi-win\windows\..\src\cudadecoder\cuda-decoder-kernels.cc not found - project kaldi-cudadecoder

3.2 最終編譯

打開C:\anjos\vs\kaldi-win\kaldiwin_vs2019_OPENBLAS_CUDA\kaldiwin_vs2019.sln，在選擇x64/Release & Debug配置分別進行編譯即可

Reference

kaldi window install
Building of Kaldi VS2015 Project on Windows
Win10 VS 2019 preview+CUDA10.1+cudnn7.5.1 配置kaldi