APS 11.1.2.x Out of Memory Errors

APS occasionally crashes in our Production environment. Our AnalyticProviderServices0.log reveals many “out of memory” errors within all of the java mess in that log file:

AnalyticProviderServices0.log:

 ####<Sep 3, 2014 10:03:12 AM EDT> <Error> <Kernel> <EPMPRWSA> <AnalyticProviderServices0> <[STANDBY] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1409752992821> <BEA-000802> <ExecuteRequest failed 
 java.lang.OutOfMemoryError: getNewTla. 
 java.lang.OutOfMemoryError: getNewTla 
 
 ####<Sep 3, 2014 10:01:32 AM EDT> <Error> <Kernel> <EPMPRWSA> <AnalyticProviderServices0> <[STANDBY] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1409752892040> <BEA-000802> <ExecuteRequest failed 
 java.lang.OutOfMemoryError.

AnalyticProviderServices0-diagnostic.log:

[2014-09-03T09:53:32.993-04:00] [AnalyticProviderServices0] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: Thread-39] [userId: <anonymous>] [ecid: 0000KWgfAIy8tlvpKWg8ye1K0ntz000002,1:23165] [APP: APS#11.1.2.0] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Wed Sep 03 09:53:32 EDT 2014 errorMessage=DFW-99997 executionContextId=null]
java.lang.OutOfMemoryError

To resolve this issue, we applied the following changes to the APS service:

We host APS on WIndows, so we did the following:

Open Regedit and navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Hyperion Solutions\AnalyticProviderServices0\HyS9aps

Export the key for a backup (File Export) in case something goes wrong when editing.

Note the highest number after JVMOption. For me, the last entry was JVMOption42.

Create a new “String Value” with the following name: JVMOption43 (since the last entry was 42) and set the value to Value:”-XXtlasize:min=4k,preferred=64k” without quotes.

Create another “String Value” with the name of JVMOption44 (after 43) and set the following value:”-XXlargeObjectLimit:4k” without quotes.

Restart APS when finished for the changes to take effect.

For Linux, the following will fix it:

Update the MIDDLEWARE_HOME/user_projects/epmsystem1/bin/deploymentScripts/setCustomParamsAnalyticProviderServices.sh and modify the following line from:

USER_MEM_ARGS="-XX:PermSize=64m -XX:MaxPermSize=256m -XXtlasize:min=16k,preferred=196k "

to:

USER_MEM_ARGS="-XX:PermSize=64m -XX:MaxPermSize=256m -XXtlasize:min=16k,preferred=250k -XXlargeObjectLimit=4k"

Restart APS when finished for the changes to take effect.

Good luck!

[ad#ad-post]