一次java 内存泄漏问题的解决过程

前段时间公司项目运行一段时间 cpu 就占用100%，然后服务就不可用了，  但是那段时间并发也没有升高，数据库，缓存也很正常，弄了很久都没有头绪。于是领导让我来解决这个问题。

登陆服务器 先用top 命令查看cpu 占用

top

发现 java 进程确实占用cpu 很高，继续查看java 内线程的cpu 占用

top -H -p 4536

找到两个线程占用cpu 很高，然后打印java 程序的线程，找到这两个线程的信息（线程号需要转成16进制）

jstack 4536> 4536.txt

发现这两个线程都是 gc 操作。有可能是jvm 内存占用满了，频繁full gc 导致的。

下载 阿里的 arthas  来看一下gc 的情况

https://github.com/alibaba/arthas/blob/master/README_CN.md

发现老年代都占用100%了，果然是频繁的full gc 导致的。而且full gc 都无法降低老年代的内存

打印一下 堆的信息

jmap -heap 4536
jstack 4536> 1.log

手动fullgc 也不行

jmap -histo:live 4536

发现有一个user对象占用很多，比较诡异，然后把堆的dump 对象下载下来，在本地用mat 查看

jmap -dump:live,format=b,file=4536 4536

https://www.eclipse.org/mat/

右键char[]  打开char[]，查看里面的对象

发现不仅user 对象比较多，而且char[] 里面有一个  对象也比较多，带有cookie 和 deleteMe 字段，这个是shiro 框架里面的内容。 我们使用了 shiro 和shiro-redis 作为权限控制框架。 有可能是这两个框架导致的。最后在网上找的果然是shiro-redis 导致的问题

http://www.findsrc.com/java/detail/8688

我们在github找到框架的源码

https://github.com/alexxiyang/shiro-redis

发现果然用到了threadlocal，作者的说法是shiro 在鉴权的时候频繁读取session 信息，所以用到了threadlocal，减小redis 的读取压力。所以可能就是这个原因导致的内存泄漏。

/**
 * doReadSession be called about 10 times when login.
 * Save Session in ThreadLocal to resolve this problem. sessionInMemoryTimeout is expiration of Session in ThreadLocal.
 * The default value is 1000 milliseconds (1s).
 * Most of time, you don't need to change it.
 */

最终解决办法。

升级shiro-redis 至 3.2.3  

shiroconfig在使用redissessiondao 时禁用threadLocal

@Bean
    public RedisSessionDAO sessionDAO(){
        RedisSessionDAO sessionDAO = new RedisSessionDAO(); // crazycake 实现
        sessionDAO.setSessionInMemoryEnabled(false);
        sessionDAO.setRedisManager(redisManager());
        sessionDAO.setSessionIdGenerator(sessionIdGenerator()); //  Session ID 生成器
        return sessionDAO;
    }

以下为shiro-redis源码

package org.crazycake.shiro;

import org.apache.shiro.session.Session;
import org.apache.shiro.session.UnknownSessionException;
import org.apache.shiro.session.mgt.eis.AbstractSessionDAO;
import org.crazycake.shiro.exception.SerializationException;
import org.crazycake.shiro.serializer.ObjectSerializer;
import org.crazycake.shiro.serializer.RedisSerializer;
import org.crazycake.shiro.serializer.StringSerializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.Serializable;
import java.util.*;

public class RedisSessionDAO extends AbstractSessionDAO {

	private static Logger logger = LoggerFactory.getLogger(RedisSessionDAO.class);

	private static final String DEFAULT_SESSION_KEY_PREFIX = "shiro:session:";
	private String keyPrefix = DEFAULT_SESSION_KEY_PREFIX;

	private static final long DEFAULT_SESSION_IN_MEMORY_TIMEOUT = 1000L;
	/**
	 * doReadSession be called about 10 times when login.
	 * Save Session in ThreadLocal to resolve this problem. sessionInMemoryTimeout is expiration of Session in ThreadLocal.
	 * The default value is 1000 milliseconds (1s).
	 * Most of time, you don't need to change it.
	 */
	private long sessionInMemoryTimeout = DEFAULT_SESSION_IN_MEMORY_TIMEOUT;

	private static final boolean DEFAULT_SESSION_IN_MEMORY_ENABLED = true;

	private boolean sessionInMemoryEnabled = DEFAULT_SESSION_IN_MEMORY_ENABLED;

	// expire time in seconds
	private static final int DEFAULT_EXPIRE = -2;
	private static final int NO_EXPIRE = -1;

	/**
	 * Please make sure expire is longer than sesion.getTimeout()
	 */
	private int expire = DEFAULT_EXPIRE;

	private static final int MILLISECONDS_IN_A_SECOND = 1000;

	private IRedisManager redisManager;
	private RedisSerializer keySerializer = new StringSerializer();
	private RedisSerializer valueSerializer = new ObjectSerializer();
	private static ThreadLocal sessionsInThread = new ThreadLocal();
	
	@Override
	public void update(Session session) throws UnknownSessionException {
		this.saveSession(session);
		if (this.sessionInMemoryEnabled) {
			this.setSessionToThreadLocal(session.getId(), session);
		}
	}
	
	/**
	 * save session
	 * @param session
	 * @throws UnknownSessionException
	 */
	private void saveSession(Session session) throws UnknownSessionException {
		if (session == null || session.getId() == null) {
			logger.error("session or session id is null");
			throw new UnknownSessionException("session or session id is null");
		}
		byte[] key;
		byte[] value;
		try {
			key = keySerializer.serialize(getRedisSessionKey(session.getId()));
			value = valueSerializer.serialize(session);
		} catch (SerializationException e) {
			logger.error("serialize session error. session id=" + session.getId());
			throw new UnknownSessionException(e);
		}
		if (expire == DEFAULT_EXPIRE) {
			this.redisManager.set(key, value, (int) (session.getTimeout() / MILLISECONDS_IN_A_SECOND));
			return;
		}
		if (expire != NO_EXPIRE && expire * MILLISECONDS_IN_A_SECOND < session.getTimeout()) {
			logger.warn("Redis session expire time: "
					+ (expire * MILLISECONDS_IN_A_SECOND)
					+ " is less than Session timeout: "
					+ session.getTimeout()
					+ " . It may cause some problems.");
		}
		this.redisManager.set(key, value, expire);
	}

	@Override
	public void delete(Session session) {
		if (session == null || session.getId() == null) {
			logger.error("session or session id is null");
			return;
		}
		try {
			redisManager.del(keySerializer.serialize(getRedisSessionKey(session.getId())));
		} catch (SerializationException e) {
			logger.error("delete session error. session id=" + session.getId());
		}
	}

	@Override
	public Collection<Session> getActiveSessions() {
		Set<Session> sessions = new HashSet<Session>();
		try {
			Set<byte[]> keys = redisManager.keys(this.keySerializer.serialize(this.keyPrefix + "*"));
			if (keys != null && keys.size() > 0) {
				for (byte[] key:keys) {
					Session s = (Session) valueSerializer.deserialize(redisManager.get(key));
					sessions.add(s);
				}
			}
		} catch (SerializationException e) {
			logger.error("get active sessions error.");
		}
		return sessions;
	}

	@Override
	protected Serializable doCreate(Session session) {
		if (session == null) {
			logger.error("session is null");
			throw new UnknownSessionException("session is null");
		}
		Serializable sessionId = this.generateSessionId(session);  
        this.assignSessionId(session, sessionId);
        this.saveSession(session);
		return sessionId;
	}

	@Override
	protected Session doReadSession(Serializable sessionId) {
		if (sessionId == null) {
			logger.warn("session id is null");
			return null;
		}

		if (this.sessionInMemoryEnabled) {
			Session session = getSessionFromThreadLocal(sessionId);
			if (session != null) {
				return session;
			}
		}

		Session session = null;
		logger.debug("read session from redis");
		try {
			session = (Session) valueSerializer.deserialize(redisManager.get(keySerializer.serialize(getRedisSessionKey(sessionId))));
			if (this.sessionInMemoryEnabled) {
				setSessionToThreadLocal(sessionId, session);
			}
		} catch (SerializationException e) {
			logger.error("read session error. settionId=" + sessionId);
		}
		return session;
	}

	private void setSessionToThreadLocal(Serializable sessionId, Session s) {
		Map<Serializable, SessionInMemory> sessionMap = (Map<Serializable, SessionInMemory>) sessionsInThread.get();
		if (sessionMap == null) {
            sessionMap = new HashMap<Serializable, SessionInMemory>();
            sessionsInThread.set(sessionMap);
        }

		removeExpiredSessionInMemory(sessionMap);

		SessionInMemory sessionInMemory = new SessionInMemory();
		sessionInMemory.setCreateTime(new Date());
		sessionInMemory.setSession(s);
		sessionMap.put(sessionId, sessionInMemory);
	}

	private void removeExpiredSessionInMemory(Map<Serializable, SessionInMemory> sessionMap) {
		Iterator<Serializable> it = sessionMap.keySet().iterator();
		while (it.hasNext()) {
			Serializable sessionId = it.next();
			SessionInMemory sessionInMemory = sessionMap.get(sessionId);
			if (sessionInMemory == null) {
				it.remove();
				continue;
			}
			long liveTime = getSessionInMemoryLiveTime(sessionInMemory);
			if (liveTime > sessionInMemoryTimeout) {
				it.remove();
			}
		}
	}

	private Session getSessionFromThreadLocal(Serializable sessionId) {

		if (sessionsInThread.get() == null) {
			return null;
		}

		Map<Serializable, SessionInMemory> sessionMap = (Map<Serializable, SessionInMemory>) sessionsInThread.get();
		SessionInMemory sessionInMemory = sessionMap.get(sessionId);
		if (sessionInMemory == null) {
			return null;
		}
		long liveTime = getSessionInMemoryLiveTime(sessionInMemory);
		if (liveTime > sessionInMemoryTimeout) {
			sessionMap.remove(sessionId);
			return null;
		}

		logger.debug("read session from memory");
		return sessionInMemory.getSession();
	}

	private long getSessionInMemoryLiveTime(SessionInMemory sessionInMemory) {
		Date now = new Date();
		return now.getTime() - sessionInMemory.getCreateTime().getTime();
	}

	private String getRedisSessionKey(Serializable sessionId) {
		return this.keyPrefix + sessionId;
	}

	public IRedisManager getRedisManager() {
		return redisManager;
	}

	public void setRedisManager(IRedisManager redisManager) {
		this.redisManager = redisManager;
	}

	public String getKeyPrefix() {
		return keyPrefix;
	}

	public void setKeyPrefix(String keyPrefix) {
		this.keyPrefix = keyPrefix;
	}

	public RedisSerializer getKeySerializer() {
		return keySerializer;
	}

	public void setKeySerializer(RedisSerializer keySerializer) {
		this.keySerializer = keySerializer;
	}

	public RedisSerializer getValueSerializer() {
		return valueSerializer;
	}

	public void setValueSerializer(RedisSerializer valueSerializer) {
		this.valueSerializer = valueSerializer;
	}

	public long getSessionInMemoryTimeout() {
		return sessionInMemoryTimeout;
	}

	public void setSessionInMemoryTimeout(long sessionInMemoryTimeout) {
		this.sessionInMemoryTimeout = sessionInMemoryTimeout;
	}

	public int getExpire() {
		return expire;
	}

	public void setExpire(int expire) {
		this.expire = expire;
	}

	public boolean getSessionInMemoryEnabled() {
		return sessionInMemoryEnabled;
	}

	public void setSessionInMemoryEnabled(boolean sessionInMemoryEnabled) {
		this.sessionInMemoryEnabled = sessionInMemoryEnabled;
	}

	public static ThreadLocal getSessionsInThread() {
		return sessionsInThread;
	}
}