http://www.techpaste.com/2013/10/weblogic-admin-server-startup-hanging-coming-running-state/
While starting weblogic admin server, weblogic server not starting or weblogic server hangs at startup at one point with out any further logs on stdout. This might come due to many reasons, we use to frequently see these errors on old weblogic setups and finally started keeping record of solutions which we tried to fix them. Please follow below solutions and see which one works for your environment for issues like weblogic server not starting or weblogic server hangs at startup.
Solution 1
Usually it used to get stuck at starting with below logs:
<Oct 22, 2013 3:42:23 AM PDT> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true> <Oct 22, 2013 3:42:23 AM PDT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true> <Oct 22, 2013 3:42:23 AM PDT> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath: /mydomain/WLS/user_projects/domains/wls_mydomain/lib/mbeantypes/csp-id-asserter.jar:/mydomain/WLS/user_projects/domains/wls_mydomain/lib/mbeantypes/jps-wls-trustprovider.jar> <Oct 22, 2013 3:42:23 AM PDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-50-153520-1.6.0_37-20121220-0843-linux-x86_64 from Oracle Corporation> <Oct 22, 2013 3:42:24 AM PDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server Temporary Patch for BUG16362974 Tue Feb 19 23:49:30 CST 2013 WebLogic Server Temporary Patch for BUG16104758 Thu Jan 17 00:49:16 CST 2013 WebLogic Server Temporary Patch for BUG16180221 Wed Jun 05 12:46:15 MDT 2013 WebLogic Server Temporary Patch for BUG14595754 Wed Oct 03 10:10:13 CDT 2012 WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050 > <Oct 22, 2013 3:42:26 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING> <Oct 22, 2013 3:42:26 AM PDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool> <Oct 22, 2013 3:42:26 AM PDT> <Notice> <Log Management> <BEA-170019> <The server log file /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
weblogic server hangs at startup and used to get stuck at above point and will not even log anything. After many tries we came to know the issue was with data folder inside /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data folder, so
to check if its really issue with data folder we renamed/moved the data folder to data_backup and tried to start it again. We saw the server recreated the data folder by itself and started up fine without any hangs.
So we came to know the issue is with the data folders some file corruption, so we extracted the old backups which gets stored as EmbeddedLDAPBackup.1.zip in /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data/ldap/backup folder and replaced all existing EmbeddedLDAP.data, EmbeddedLDAP.index, EmbeddedLDAP.delete, EmbeddedLDAP.tran, EmbeddedLDAP.twpos files with the backed up file and restarted the Admin Server. this time it came up fine, once its up, to avoid this situation again follow steps mentioned here: (i.e: Login to WebLogic console, click on domain name on left pane –> go to security tab –> Embedded LDAP tab –> click on “Refresh replica at startup” option.)
Solution 2
If you are getting below error while starting the managed server try to remove the edit.lok file, it will fix the issue or regenerate the boot.properties file.
<Oct 19, 2012 8:01:41 PM MMT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.> <Oct 19, 2012 8:01:41 PM MMT> <Critical> <Security> <BEA-090403> <Authentication for user denied> <Oct 19, 2012 8:01:41 PM MMT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user denied weblogic.security.SecurityInitializationException: Authentication for user denied at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:965) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050) at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873) at weblogic.security.SecurityService.start(SecurityService.java:141) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) Truncated. see log file for complete stacktrace Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261) at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110) at java.security.AccessController.doPrivileged(Native Method) at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Truncated. see log file for complete stacktrace
Solution 3
If you are getting below error before starting the server or just after starting the server and weblogic server not starting then please execute below mentioned steps to fix it.
Delete all the corrupted DAT files under /opt/domain/server_name/servers/AdminServer/data/store/diagnostics/*.DAT and try to restart the server again.
####<Oct 25, 2006 11:03:15 AM MEST> <Notice> <WebLogicServer> <mynew-srv> <AdminServer> <main> <<WLS Kernel>> <> <> <1161766995356> <BEA-000360> <Server started in RUNNING mode> ####<Oct 25, 2006 11:07:29 AM MEST> <Error> <Diagnostics> <mynew-srv> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel> <> <26be653e45def710:-52c11f7a:10e7eb579cc:-7fe1-0000000000000001><1161767249271> <BEA-320103> <There was an error creating the DiagnosticDataRuntimeMBean instance with the given logical name EventsDataArchive. weblogic.diagnostics.accessor.DataAccessServiceCreateException: weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService(DiagnosticDataAccessServiceFactory.java:76) at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessService (DiagnosticDataAccessRuntime.java:164) at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessRuntime(DiagnosticDataAccessRuntime.java:100) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime$1.run(DiagnosticAccessRuntime.java:144) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime (DiagnosticAccessRuntime.java:140) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:94) at weblogic.diagnostics.instrumentation.EventQueue.getArchive(EventQueue.java:338) at weblogic.diagnostics.instrumentation.EventQueue.access$200(EventQueue.java:46) at weblogic.diagnostics.instrumentation.EventQueue$ArchiveEventsAction.run(EventQueue.java:371) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121) at weblogic.diagnostics.instrumentation.EventQueue.timerExpired(EventQueue.java:316) at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:259) at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) at weblogic.work.ExecuteThread.run(ExecuteThread.java:179) weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:794) at weblogic.store.io.file.Heap.open(Heap.java:181) at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:83) at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:332) at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:201) at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:54) at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.getArchiveSchema(PersistentStoreDataArchive.java:309) at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.<init>(PersistentStoreDataArchive.java:268) at weblogic.diagnostics.archive.wlstore.EventsPersistentStoreDataArchive.<init>(EventsPersistentStoreDataArchive.java:53) at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createEventsDataAccessService(DiagnosticDataAccessServiceFactory.java:174) at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService(DiagnosticDataAccessServiceFactory.java:61) at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessService(DiagnosticDataAccessRuntime.java:164) at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessRuntime(DiagnosticDataAccessRuntime.java:100) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime$1.run(DiagnosticAccessRuntime.java:144) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:140) at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:94) at weblogic.diagnostics.instrumentation.EventQueue.getArchive(EventQueue.java:338) at weblogic.diagnostics.instrumentation.EventQueue.access$200(EventQueue.java:46) at weblogic.diagnostics.instrumentation.EventQueue$ArchiveEventsAction.run(EventQueue.java:371) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121) at weblogic.diagnostics.instrumentation.EventQueue.timerExpired(EventQueue.java:316) at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:259) at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) at weblogic.work.ExecuteThread.run(ExecuteThread.java:179) java.io.IOException: Error reading from file, No such file or directory, errno=2 at weblogic.store.io.file.direct.DirectIONative.read(Native Method) at weblogic.store.io.file.direct.DirectFileChannel.read(DirectFileChannel.java:133) at weblogic.store.io.file.StoreFile.read(StoreFile.java:275) at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:792) at weblogic.store.io.file.Heap.open(Heap.java:181) at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:83) at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:332) at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:201) at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:54) at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.getArchiveSchema(PersistentStoreDataArchive.java:309)
Solution 4
If you are getting below error while starting the server and weblogic server not starting then follow below mentioned steps to fix this issue.
Note: Always keep backup of files before removing it.
1: Edit the file, use plain text username and password instead of encrypted like
Username: weblogic
password: Weblogic1
After start the server from command line, it will encrypt again automitically.
2: Remove the ManagedServerDir/data/ldap directory completely or restore it from the automated backups taken @ /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data/ldap/backup folder.
3: Now try to start the server from the command prompt and provide the username/password used to login to the Admin Console.
####<May 18, 2011 12:05:53 PM IST> <Critical> <Security> <CALTP8BB14> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275028553535> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> ####<May 18, 2011 12:05:53 PM IST> <Critical> <WebLogicServer> <CALTP8BB14> <AdminServer> <main> <<WLS Kernel>> <> <> <1275028553535> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted. weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted. at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:959) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050) at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875) at weblogic.security.SecurityService.start(SecurityService.java:141) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) at weblogic.work.ExecuteThread.run(ExecuteThread.java:173) javax.security.auth.login.FailedLoginException: [Security:090303]Authentication Failed: User weblogic weblogic.security.providers.authentication.LDAPAtnDelegateException: [Security:090295]caught unexpected exception at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:243) at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110) at java.security.AccessController.doPrivileged(Native Method) at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
These are few solutions which we implement to make things work when you have issues like weblogic server not starting or weblogic server hangs at startup . If you have few other solutions which is commonly used by you then please post in comments it will help others to debug there issue.