最近在研究Spark的相關知識，本地搭建了一個開發環境Windows7+Eclipse+JDK1.7。
一. 日誌效率原因
開發時，控制檯輸出一大堆日誌信息，嚴重影響查看日誌效率。
從控制檯輸出日誌我們可以看出，應用程序是默認加載Spark-core包下面的log4j-defaults.properties日誌文件。查看log4j-defaults.properties文件
由上圖可知，Spark-core包設置默認的日誌級別爲info，所以我們纔看到一大堆日誌信i息。
那針對以上問題，在開發過程中我們如何解決？
二. 日誌級別解決方法
方式一.局部應用設置
針對SparkContext應用，Spark有專門的api設置日誌級別，如下：
上述方法，只針對SparkContext相關的應用，而對Spark-streaming等應用無效果。
方式二.全局應用設置
針對spark所有應用，可以Java工程目錄中新建/src/main/resources目錄，把log4j.properties放置該目錄。
log4j.properties生成：
1. Spark中conf默認配置文件是log4j.properties.template，可以將其改名爲log4j.properties;
2. 將Spark-core包中的log4j-default.properties內容複製到log4j.properties文件。
#log4j內容如下
- #
- # Licensed to the Apache Software Foundation (ASF) under one or more
- # contributor license agreements. See the NOTICE file distributed with
- # this work for additional information regarding copyright ownership.
- # The ASF licenses this file to You under the Apache License, Version 2.0
- # (the "License"); you may not use this file except in compliance with
- # the License. You may obtain a copy of the License at
- #
- # http://www.apache.org/licenses/LICENSE-2.0
- #
- # Unless required by applicable law or agreed to in writing, software
- # distributed under the License is distributed on an "AS IS" BASIS,
- # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- # See the License for the specific language governing permissions and
- # limitations under the License.
- #
- # Set everything to be logged to the console
- log4j.rootCategory=WARN, console
- log4j.appender.console=org.apache.log4j.ConsoleAppender
- log4j.appender.console.target=System.err
- log4j.appender.console.layout=org.apache.log4j.PatternLayout
- log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
- # Settings to quiet third party logs that are too verbose
- log4j.logger.org.spark-project.jetty=WARN
- log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
- log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
- log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
- log4j.logger.org.apache.parquet=ERROR
- log4j.logger.parquet=ERROR
- # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
- log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
- log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
在開發工程中，我們可以設置日誌級別爲WARN，即：
log4j.rootCategory=WARN, console
三. 日誌級別設置效果
