General Error Messages

There are many types of threads; however, essentially they fall into two categories: Java threads and native threads. All the threads running within a JVM are Java threads (java.lang.Thread class inside Java). The native code (C++/C) creates native threads that are scheduled and managed by the operating system. Here are the key differences between the two types:

• Java threads are created and managed by LiveCycle code, application server, or the JVM itself.

• Operating system tools (such as perfmon or Task Manager) know only about native threads.

Because the operating system has no visibility into Java threads, when you monitor threads using operating system tools such as perfmon, you are monitoring only native threads. The only way to get details into Java threads is to get a Java thread dump. The process to get a Java thread dump varies depending on your application server and JVM. See the manufacturer’s documentation.

The implementation of the JVM is done in C/C++ code and that JVM code maps Java threads to native threads. This mapping can be either 1:1 (1 Java thread to 1 native thread) or N:1 (multiple Java threads to 1 native thread). The details of how this mapping works are specific to the JVM vendor, but 1:1 mapping is a typical default. This mapping means that each Java thread has a corresponding native thread. The number of Java threads has no limit; however, because 1:1 mapping is typical and the number of native threads is limited, you can run out of Java threads as well. This limit applies per process (JVM being a single process) and varies with each operating system. You can assume that the limit is in the thousands, but less than 10,000. Regardless of this number, having many hundreds of threads is a performance problem because the operating system has to schedule up to that many threads.

Another common issue for threads pertains to memory allocations. When a new Java thread is allocated, a fixed amount of memory is required for the thread's stack. This thread stack space is a parameter (-Xss option for Sun™ JVM), and the default is ~512 KB. Therefore, if you have 1000 threads, 500 MB of memory is required just for the thread's stacks. This memory will compete with all the other memory allocations being done in the JVM, such as what LiveCycle allocates, and will create memory allocation issues.

In practice, when the JVM cannot allocate memory or create threads, it throws an OutOfMemory exception back to the caller. Along with this exception is a stack trace and a reason for throwing the exception. This reason is very important to note; it will give you further clues to what may be wrong.

The following code is an example of a message that displays two errors and their associated reason codes:

"unable to create new native thread: java.lang.OutOfMemoryError: unable to create new native thread java.lang.OutOfMemoryError: Java heap space"

These errors mean that the JVM could not create more threads for one of these reasons:

• The per-process thread limit was reached.

• The thread stack cannot be allocated.

To determine the exact cause, you must get a thread dump (also known as Java jump). A thread dump is typically called javacore.xxxx.txt and resides under an application server's log directories. A lot of information is inside the thread dump, but you can quickly determine the number of threads by counting the occurrences of the TID: token on the list. A typical entry looks like this:

"Thread-1227" (TID:0x106948F0, sys_thread_t:0x78996DA0, state:R, native ID:0x191C) prio=5 
4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method) 
4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code)) 
4XESTACKTRACE at java.io.BufferedInputStream.fill(BufferedInputStream.java(Compiled Code)) 
4XESTACKTRACE at java.io.BufferedInputStream.read1(BufferedInputStream.java(Compiled Code)) 
4XESTACKTRACE at java.io.BufferedInputStream.read(BufferedInputStream.java(Compiled Code)) 
4XESTACKTRACE at com.sun.jndi.ldap.Connection.run(Connection.java(Compiled Code)) 
4XESTACKTRACE at java.lang.Thread.run(Thread.java:567)

If you find thousands of threads, you are probably running out of threads. Developers should be able to identify obvious culprits by scanning the stack traces of these threads.

If the thread count is in the hundreds, the reason for the java.lang.OutOfMemory error is not the thread limit. Reduce the thread stack size (-Xss option mentioned above), rerun LiveCycle, and see if the problem disappears.

Content Services (deprecated) features various in-memory caches that significantly improve performance, but consume considerable Java heap memory. You may encounter OutOfMemory exceptions if you run the Document Management service for Content Services on hardware that only meets the minimum hardware requirements.

You can control memory usage by setting the JVM arguments -Dhibernate.cache.use_second_level_cache=false and -Dhibernate.cache.use_query_cache=false.

1. Check the JNDI tree on the LiveCycle application server. Does the name used appear in the tree?

   • If yes, it is most likely that your code did not properly set up the InitialContext object being used for the look-up, and the look-up is being done on a JNDI tree that is not the one that the resource is listed in. For the property values to use, see the Installing and Deploying LiveCycle document for your application server.

   • If no, continue to step 2.

2. Does the resource appear in the JNDI tree under a name other than that listed in the look-up?

   • If yes, you are using the incorrect look-up name. Provide the correct name.

   • If no, continue to step 3.

3. Review the application server logs during startup. If the application server was configured to make this resource available but something is going wrong, an exception appears here. Is there an exception?

   • If yes, review the exception and stack trace. If the NameNotFoundException is a symptom of another problem based on your investigation of the server logs, go to the troubleshooting steps for that problem.

   • If no, continue to step 4.

4. If the resource is not listed in the JNDI tree, and no exception appears at startup to explain why it is not available, the most probable issue is that the application server was not configured properly to make that resource available. Review the application server configuration. Was it configured to make this resource available?

   • If no, see the Installing and Deploying LiveCycle guide for your application server.

   • If yes, this problem is not one of the common ones that cause this issue. Contact Adobe Enterprise Support.

(Known issue) If ECM Connector for EMC Documentum is included in your LiveCycle for JBoss installation, the following error message appears in the server logs every time you restart the server:

An org.jboss.logging.appender.FileAppender object is not assignable to an org.apache.log4j.Appender variable

To stop unnecessary ERROR and WARNING log messages, generated by IBM FileNet, from appearing in the JBoss Application Server log file, make the following modification to the log4j.xml file located at [jboss_root]/server/all/conf.

1. Locate the log4j.xml file and open it in an editor.

2. Add the following text to the [Category] section:

        <category name="com.filenet"> 
            <priority value="FATAL"/> 
        </category>

1. Save and close the file.

2. Restart the application server.

You have a WebLogic time-out issue if you receive the following error message:

<Warning> <com.adobe.workflow.AWS> <ap-sun4> <Server_127> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-58E59A31956BB0D8F0AB> <> <1178316054656> <000000> <javax.ejb.TransactionRolledbackLocalException: EJB Exception: ; nested exception is: javax.ejb.TransactionRolledbackLocalException: EJB Exception: ; nested exception is: weblogic.transaction.internal.TimedOutException: Transaction timed out after 299 seconds 

To resolve this issue, increase the WebLogic JTA time-out value to a value greater than 300 seconds. (See “Configuring the WebLogic transaction time-out” in the Preparing to Install LiveCycle document.

You have a WebLogic EAR file deployment issue if you receive the following error message:

    Could not start application adobe-livecycle-weblogic. 
    com.adobe.livecycle.cdv.CDVException[ALC-LCM-030-113]: Failed to deploy EAR.

To resolve this issue, check the WebLogic Server Administration Console to ensure that it is not locked, which is indicated by the Lock & Edit button being selected. If it is locked, Configuration Manager shows the deployment process as 16% complete and the WebLogic Server Administration Console shows the EAR file as deployed but in an installed state. If the WebLogic Server Administration Console is not locked, Configuration Manager can deploy the EAR files.

To resolve this issue, go to the WebLogic Server Administration Console, ensure that it is unlocked, and redeploy the EAR files.

You have a WebLogic EAR file deployment issue (Solaris) if you receive the following error message:

    java.lang.OutOfMemoryError: PermGen space

To resolve this issue, increase the MaxPermSize from 256 to 512. You can change this value from the WebLogic Server Administration Console.

There is a known issue that WebLogic Server running on Windows fails to deploy LiveCycle modules because the server time-out setting of 5 seconds is too short. You must manually configure this setting as follows:

• Go to [appserverdomain] and open startWeblogic.cmd in an editor.

• Add the following parameter:

        -Dweblogic.client.socket.ConnectTimeout = <timeout value>

If your LiveCycle process uses an e-mail endpoint, you might receive an error similar to the following when the e-mail endpoint is invoked.

SECJ0305I: The role-based authorization check failed for naming-authz operation NameServer:bind_java_object. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: CosNamingCreate, CosNamingDelete, CosNamingWrite.

This error occurs due to missing permissions for the CORBA naming service groups in WebSphere.

Perform the following steps to resolve this issue:

1. In the WebSphere administration console, select Environment > Naming > CORBA Naming service groups.

2. Add the following privileges:

   • Cos Naming Write

   • Cos Naming Delete

   • Cos Naming Create

3. Restart the WebSphere application server.

In case of a WebSphere cluster deployment, you might see multiple entries of error logs similar to the following:

org.hibernate.event.def.AbstractFlushingEventListener performExecutions Could not synchronize database state with session 
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.alfresco.repo.domain.hibernate.NodeImpl#10] 
at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1769)at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2412) 
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2312) 
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)

These logs do not affect the functionality. However, to suppress these logs, you need to change the logging levels using the WebSphere Administration Console. To change the logging levels:

1. Log on to the WebSphere Administrative Console.

2. Click Troubleshooting > Logs and Trace.

3. In the right pane, click the name of the application server.

4. Click Change Log Detail Levels.

5. In the Configuration tab, expand All Components > org.hibernate.event.* > org.hibernate.event.def.*.

6. Click org.hibernate.event.def.AbstractFlushingEventListener. A pop-up menu appears.

7. In the pop-up menu, click Message and Trace Levels > Fatal.

8. Click OK and then click Save directly to the master configuration.

This section explains how to correct a failed deployment if you receive the following error message when attempting to deploy the adobe-livecycle-websphere.ear file:

Could not deploy adobe-livecycle-websphere.ear. com.adobe.livecycle.cdv.CDVException[ALC-LCM-030-112]: Failed to deploy EAR. Could not deploy adobe-livecycle-websphere.ear.

To avoid receiving warning messages in the SystemOut.log file that are related to the deprecated usage of direct JNDI lookup, you can modify the WebSphere logging level.

To suppress the warning message J2CA0294W from the SystemOut.log, you can change the logging level to *=info:com.ibm.ejs.j2c.ConnectionFactoryBuilderImpl=severe.

To avoid the WebSphere installation from logging several unnecessary log messages, you can increase the logging level to “Warning” so that messages at lower level are not logged.

1. Log in to WebSphere Administrative Console through the URL http://[hostname]:9060/admin and

2. In the navigation tree, click Troubleshooting and select Logs and Trace.

3. In the right pane, click the name of the application server and then click Change Log Detail Levels.

4. Select Runtime and enter org.apache.xml.security.*

5. Click Message And Trace Levels, and select Warning.

6. Select Save runtime changes to configuration check box.

7. Click OK.

Your WebSphere Application Server may give exceptions similar to the ones described below.

         Could not connect to Inbox. Error message: com.ibm.jsse2.util.h: 
            No trusted certificate found; nested exception is: 
                javax.net.ssl.SSLHandshakeException: 
        com.ibm.jsse2.util.h: No trusted certificate found

        [5/28/08 13:15:30:283 CDT] 00000025 SystemOut     O 
        CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN 
        "CN=imap.gmail.com, O=Google Inc, L=Mountain View, ST=California, C=US" 
        was sent from target host:port "null:null". The signer may need to be 
        added to local trust store "D:/servers/websphere6.1/profiles/AppSrv01 
        /config/cells/MN-TOBIKONode01Cell/nodes/MN-TOBIKONode01/trust.p12" 
        located in SSL configuration alias "NodeDefaultSSLSettings" loaded from 
        SSL configuration file "security.xml".  The extended error message from 
        the SSL handshake exception is: "No trusted certificate found". 
        [5/28/08 13:15:30:283 CDT] 00000025 SystemOut     O  
        [5/28/08 13:15:30:283 CDT] 00000025 ExceptionUtil E    
        CNTR0020E: EJB threw an unexpected (non-declared) exception during 
        invocation of method "doSupports" on bean "BeanId(adobe-core-websphere 
        #adobe-dscf.jar#EjbTransactionCMTAdapter, null)". Exception data: 
        java.lang.RuntimeException: Could not connect to Inbox. Error message: 
        com.ibm.jsse2.util.h: No trusted certificate found; 
            nested exception is: 
                javax.net.ssl.SSLHandshakeException:  
        com.ibm.jsse2.util.h: No trusted certificate found

While bootstrapping User Manager components on WebSphere Application Server, the following exception message will appear only once after the application is started:

00000043 javaURLContex E   NMSV0310E: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names. Exception stack trace: 
javax.naming.ConfigurationException [Root exception is javax.naming.NameNotFoundException: Name comp/env/ejb not found in context "java:".]

This error can be safely ignored.

When a DSC is invoked from within a transaction started by another application deployed as an EAR on the same instance of WebSphere on which LiveCycle is deployed, the DSC call fails with the following error message:

LocalException E   CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "getObjectType" on bean

This error is encountered for WebSphere only when the adobe-utilities.jar file is used and Platform.UTIL.getTransactionManager() is the user that starts the transaction manager.

To resolve this issue, do not use adobe-utilities.jar to start the transaction manager. Rather, use the following code to create the UserTransaction:

InitialContext initialContext = new InitialContext(); 
UserTransaction ut = (UserTransaction)initialContext.lookup("java:comp/UserTransaction"); 
ut.begin();