ORA-02049: timeout: distributed transaction waiting for lock

Problem :

The issue what we were getting while Product sync. Whenever any product sync got failed due to any valid issue (product already exists, Product not found) the transaction is not getting closed. As a result even if after correcting, if we try to sync the same or other products all of them getting failed with the same error. We found one workaround that is after restarting SOA Manage Server & DB restart issue is not coming anymore. But again once there is any valid error while product sync, subsequent all request are getting failed.

Cause :

Trnsactions was not getting closed after Throw. As a result all the subsequent request are causing DB_Lock.

Solution :

Ideally timeout parameters should be set as below :

syncMaxWaitTime < BPEL EJB's transaction timeout < Global Transaction Timeout (JTA timeout) < DISTRIBUTED_LOCK_TIMEOUT

1. SyncMaxWaitTime already tune to 240sec.

 


2. EJB level timeout setting :

BPELActivityManagerBean (300sec)
BPELDeliveryBean (300sec)
BPELDispatcherBean (300sec)
BPELEngineBean (300sec)
BPELFinderBean (300sec)
BPELInstanceManagerBean (300sec)
BPELProcessManagerBean (300sec)
BPELSensorValuesBean (300sec)
BPELServerManagerBean (300sec)

3. JTA time out already set as 360sec also Abandon Timeout Seconds as 400sec.

4. We can increase this more than 360 (more than JTA timeout). We made it 405sec which is just more than Abandon Timeout.

Remove unused files/artifacts from MDS

Whenever we want to delete some files/artifacts from MDS by using below steps we can delete data directly from MDS :

1. Execute wlst.cmd :
--------------------------------------
cd ${ORACLE_HOME}/oracle_common/common/bin

Execute wlst.sh file:
----------------------
wlst.sh


2. Connect to WL Server :
--------------------------------------
connect('weblogic','welcome1','t3://hostname:port')


3. Delete from MDS :
--------------------------------------
deleteMetadata(application='soa-infra', server='soa_server1',docs='/apps/AIAMetaData/##Artifacts Path##')

java.lang.NoClassDefFoundError

java.lang.NullPointerException at oracle.apps.aia.core.eh.InvokeBusinessErrorHandler.process(InvokeBusinessErrorHandler.java:24)

To work around the exception and missing /classes issue associated with the java.lang.NoClassDefFoundError, please take the following steps:  

Firstly check below steps :

> Check to see if aia.jar exists in the SOA CLASSPATH.
> Verify whether or not aia.jar also resides under the "domain/lib" folder.
If aia.jar is present in either location, please remove the aia.jar file from each location and then retest.
If issue is resolved , the following 5 steps is not required.


1) Copy aia.jar located under $AIAHOME/lib to $SOA_HOME/soa/modules/oracle.soa.ext.11.1.1/
2) Use command - jar -xvf aia.jar to extract all of its contents.
3) Create a directory named "classes" under $SOA_HOME/soa/modules/oracle.soa.ext.11.1.1 directory and copy the extracted contents of aia.jar into newly created  "classes" directory.
4) After this you need to run the build.xml file located in the same directory.
5) Remove the aia.jar FROM this location $SOA_HOME/soa/modules/oracle.soa.ext.11.1.1 that is copied in Step 1 (If it exists)
6) Bounce SOA server and then retest.
This should correct the java.lang.NoClassDefFoundError.

Once the above is complete, to address the email notification portion, the following changes are required:

Update the AIAConfiguraitonProperties.xml file with the following changes:

1) Under EH Module properties section
<Property name="EH.HOSTNAME">managedserver1.hostname</Property> <!-- Use one of the SOA Servers instead of Load balancer hostname for this property-->
<Property name="EH.PORT">managedserver1.port</Property>   <!-- Used the SOA Port and not the LBR port for this property-->
2) Upload to MDS
3) Reload using AIA UI.
4) Retest.

==================================================================================================================================================================

If still issue not got resolved, please check whether the class file got corrupted not (try to decompile). If got corrupted, please load a new class file 

Add Security Header

Namespace add :

xmlns:ns7="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd

Variable Declaration :

    <variable name="securityContext" element="ns7:Security"/>
    <variable name="userNameToken" element="ns7:UsernameToken"/>
    <variable name="pswd" element="ns7:Password"/>

Assign Security Header :

    <assign name="Assign_WS_Header">

      <copy>

        <from expression="aia:getServiceProperty('{http://xmlns.oracle.com/ABCSImpl/Siebel/Industry/Comms/ProcessPayableInvoiceXXXXProvABCSImpl}ProcessPayableInvoiceXXXXProvABCSImpl','Routing.billformatter.pwd',false())"/>

        <to variable="pswd" query="/ns7:Password"/>

      </copy>

      <copy>

        <from expression="aia:getServiceProperty('{http://xmlns.oracle.com/ABCSImpl/Siebel/Industry/Comms/ProcessPayableInvoiceXXXXProvABCSImpl}ProcessPayableInvoiceXXXXProvABCSImpl','Routing.billformatter.username',false())"/>

        <to variable="userNameToken" query="/ns7:UsernameToken/ns7:Username"/>

      </copy>

      <bpelx:insertAfter>

        <bpelx:from variable="pswd" query="/ns7:Password"/>

        <bpelx:to variable="userNameToken"

        query="/ns7:UsernameToken/ns7:Username"/>

      </bpelx:insertAfter>

      <bpelx:append>

        <bpelx:from variable="userNameToken" query="/ns7:UsernameToken"/>

      <bpelx:to variable="securityContext" query="/ns7:Security"/>

      </bpelx:append>

    </assign>

</sequence>

Invocation :

<invoke name="Invoke_test" bpelx:invokeAsDetail="no"

                  inputVariable="Invoke_test_InputVariable"

                  outputVariable="Invoke_test_OutputVariable"

                  partnerLink="billformatter" portType="ns1:Billformatter"

                  bpelx:inputHeaderVariable="securityContext"

                  operation="resendBill"/>

          <assign name="Reply_ResendBillOutput">