錯誤的locale設置導致Impala crash
今天在編譯cdh5.16.2版本的Impala並加載數據時，發現三個impalad同時crash。在impalad.ERROR裏可以看到錯誤信息：
terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid
Wrote minidump to /home/quanlong/workspace/Impala-CDH/logs/data_loading/minidumps/impalad/07f37528-511c-4325-654b5bb6-1ae3dbd5.dmp
既然有生成minidump文件，不妨就看一下crash在哪了。解析出來如下：
Operating system: Linux
0.0.0 Linux 4.15.0-99-generic #100~16.04.1-Ubuntu SMP Wed Apr 22 23:56:30 UTC 2020 x86_64
CPU: amd64
family 6 model 158 stepping 10
1 CPU
GPU: UNKNOWN
Crash reason: SIGABRT
Crash address: 0x3e8000026e5
Process uptime: not available
Thread 213 (crashed)
0 libc-2.23.so + 0x35428
1 libc-2.23.so + 0x3702a
2 libstdc++.so.6.0.21 + 0x1381c0
3 libc-2.23.so + 0x79242
4 libc-2.23.so + 0x79242
5 libstdc++.so.6.0.21 + 0x8c880
6 libstdc++.so.6.0.21 + 0x8f84d
7 libstdc++.so.6.0.21 + 0xa2b80
8 libstdc++.so.6.0.21 + 0x8d6b6
9 libstdc++.so.6.0.21 + 0x8d701
10 libstdc++.so.6.0.21 + 0x8d919
11 libstdc++.so.6.0.21 + 0x138968
12 libstdc++.so.6.0.21 + 0xb65af
13 libstdc++.so.6.0.21 + 0xb0714
14 libstdc++.so.6.0.21 + 0xa126c
15 libstdc++.so.6.0.21 + 0xa24d8
16 impalad!std::_Rb_tree_const_iterator<std::pair<long const, impala::HdfsPartitionDescriptor*> >::operator->() const [stl_tree.h : 278 + 0xf]
17 impalad!boost::filesystem::path::codecvt() + 0x53
18 impalad!impala::HdfsScanNodeBase::Prepare(impala::RuntimeState*) [hdfs-scan-node-base.cc : 209 + 0x5]
19 impalad!std::allocator<impala::ScalarExpr*>::allocator() [allocator.h : 113 + 0x3]
20 impalad!std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> >::vector() [stl_vector.h : 257 + 0xc]
21 impalad!std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >::pair<int const&, 0ul>(std::tuple<int const&>&, std::tuple<>&, std::_Index_tuple<0ul>, std::_Index_tuple<>) [tuple : 1102 + 0x18]
22 impalad!std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >::pair<int const&>(std::piecewise_construct_t, std::tuple<int const&>, std::tuple<>) [tuple : 1091 + 0x1b]
23 impalad!std::tuple<int const&>::tuple(std::tuple<int const&>&&) [tuple : 409 + 0x13]
24 impalad!__gnu_cxx::new_allocator<std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > > >::construct<std::pair<int const, std::vector<impala::ScalarExpr*> >, const std::piecewise_construct_t&, std::tuple<int const&>, std::tuple<> > [new_allocator.h : 120 + 0x15]
25 impalad!__gnu_cxx::__aligned_buffer<std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > > >::_M_ptr() [aligned_buffer.h : 64 + 0xc]
26 impalad!std::tuple_element<0ul, std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > > >::type& std::get<0ul, int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >(std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >&) [utility : 144 + 0xc]
27 impalad!std::__detail::_Equal_helper<int, std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >, std::__detail::_Select1st, std::equal_to<int>, unsigned long, false>::_S_equals(std::equal_to<int> const&, std::__detail::_Select1st const&, int const&, unsigned long, std::__detail::_Hash_node<std::pair<int const, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >, false>*) [hashtable_policy.h : 1337 + 0x34]
28 impalad!impala::ScalarExpr** std::__copy_move_a<false, impala::ScalarExpr* const*, impala::ScalarExpr**>(impala::ScalarExpr* const*, impala::ScalarExpr* const*, impala::ScalarExpr**) [stl_algobase.h : 396 + 0x17]
29 impalad!__gnu_cxx::__normal_iterator<impala::ScalarExpr**, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > > std::__copy_move_a2<false, __gnu_cxx::__normal_iterator<impala::ScalarExpr* const*, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >, __gnu_cxx::__normal_iterator<impala::ScalarExpr**, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > > >(__gnu_cxx::__normal_iterator<impala::ScalarExpr* const*, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >, __gnu_cxx::__normal_iterator<impala::ScalarExpr* const*, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >, __gnu_cxx::__normal_iterator<impala::ScalarExpr**, std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > >) [stl_algobase.h : 434 + 0x4f]
30 impalad!std::allocator<std::_Rb_tree_node<std::string> >::allocator(std::allocator<std::_Rb_tree_node<std::string> > const&) [allocator.h : 116 + 0x13]
31 impalad!std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >::_Rb_tree_impl<std::less<std::string>, true>::_Rb_tree_impl(std::less<std::string> const&, std::allocator<std::_Rb_tree_node<std::string> > const&) [stl_tree.h : 469 + 0xc]
32 impalad!std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >::_Rb_tree(std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >&&) [stl_tree.h : 703 + 0xc]
33 impalad!std::set<std::string, std::less<std::string>, std::allocator<std::string> >::set(std::set<std::string, std::less<std::string>, std::allocator<std::string> >&&) [stl_set.h : 209 + 0x1e]
34 impalad!std::pair<std::string const, std::set<std::string, std::less<std::string>, std::allocator<std::string> > >::pair<std::string, std::set<std::string, std::less<std::string>, std::allocator<std::string> >, void>(std::pair<std::string, std::set<std::string, std::less<std::string>, std::allocator<std::string> > >&&) [stl_pair.h : 152 + 0x35]
35 impalad!__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::construct<std::pair<const std::basic_string<char>, std::set<std::basic_string<char> > >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > [new_allocator.h : 120 + 0xb]
36 impalad!std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > >::_S_construct<std::pair<const std::basic_string<char>, std::set<std::basic_string<char> > >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > [alloc_traits.h : 253 + 0x22]
37 impalad!__gnu_cxx::new_allocator<std::_Rb_tree_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::construct<std::basic_string<char>, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&> [new_allocator.h : 120 + 0xb]
38 impalad!__gnu_cxx::__aligned_buffer<std::pair<std::string const, impala::RuntimeProfile::Counter*> >::_M_ptr() const [aligned_buffer.h : 68 + 0xc]
39 libstdc++.so.6.0.21 + 0xce9e8
40 impalad!bool std::operator< <char, std::char_traits<char>, std::allocator<char> >(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [basic_string.h : 2590 + 0x13]
41 impalad!std::less<std::string>::operator()(std::string const&, std::string const&) const [stl_function.h : 371 + 0x13]
42 impalad!std::_Rb_tree<std::string, std::pair<std::string const, impala::RuntimeProfile::Counter*>, std::_Select1st<std::pair<std::string const, impala::RuntimeProfile::Counter*> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::string const, impala::RuntimeProfile::Counter*> >*, std::_Rb_tree_node<std::pair<std::string const, impala::RuntimeProfile::Counter*> >*, std::string const&) [stl_tree.h : 1265 + 0x13]
43 linux-gate.so + 0xc45
44 libc-2.23.so + 0x115876
45 libc-2.23.so + 0x115876
46 impalad!impala::MonotonicStopWatch::Start() [stopwatch.h : 110 + 0x5]
47 impalad!impala::ScopedTimer<impala::MonotonicStopWatch>::ScopedTimer(impala::RuntimeProfile::Counter*, bool const*) [runtime-profile-counters.h : 486 + 0xc]
48 impalad!impala::ThreadResourceMgr::ResourcePool::num_optional_threads() const [thread-resource-mgr.h : 134 + 0xe]
這是在讀數據時，調用boost的filesystem::path::codecvt()然後crash了。Google一下發現這原來是個known issue:
https://impala.apache.org/docs/build/html/topics/impala_known_issues.html
有一段寫道 “Impala should tolerate bad locale settings”
If the LC_* environment variables specify an unsupported locale, Impala does not start.
Apache Issue: IMPALA-532
Workaround: Add LC_ALL=“C” to the environment settings for both the Impala daemon and the Statestore daemon. See Modifying Impala Startup Options for details about modifying these environment settings.
Resolution: Fixing this issue would require an upgrade to Boost 1.47 in the Impala distribution.
意思是說錯誤的 locale 設置會讓impala啓動失敗。但事實上我這邊啓動是成功的，只是讀數據時失敗了。點進去看提到的 IMPALA-532，發現在comments裏有提到說雖然Impala依賴的boost版本已經升到1.47以上了（cdh5.16.2的impala用的是boost-1.57.0-p3），但在讀數據時還會出這個問題。解決辦法就是讓impala進程有正確的locale變量，於是改爲這樣啓動：
export LC_ALL=en_US.UTF-8
bin/start-impala-cluster.py
即在啓動impala minicluster前設置好LC_ALL，問題就不出現了。生產環境中遇到這個問題，應該需要改impala啓動腳本，或者系統全局設置。
待續思考
我這臺機器之前是沒有這個問題的，最近死機後被我強制重啓了兩回，然後就出現了這個問題。我另外還有一臺沒問題的ubuntu，對比兩臺機器的locale輸出：
左邊是正常機器上的輸出，右邊是有問題機器的輸出。在有問題的機器上其實還附帶了一個warning: “locale: Cannot set LC_ALL to default locale: No such file or directory”。
LANGUAGE環境變量似乎不相關，設了之後問題照樣重現。可能是系統某個文件被我強制重啓損壞了，導致處理空的LC_ALL時讀不到缺省的值。現在的解決辦法（即手動設非空的LC_ALL）相當於一個workaround吧，可能除了Impala外的應用也會受影響，還是得把系統級別的問題修復了。