Weblogic -- Weblogic Java processes hang- Admin Console may work slow -- Watch out /dev/random!

I have faced with this incident in a Weblogic 10.3.6 environment running with 64 bit jdk 7 upgdate 5 on a Redhat Linux 6.3 64 bit system.
Okay, before characterizing the problem and explaining the workarounds, Lets first start with /dev/random..
/dev/random is a special file .. It is a blocking number generator. /dev/random is based on the entropy pool, which is compromised by  random bits that are assumed to be unknown..  The bits are calculated from user-triggered events (keystroke, disk I/O, mouse clicks etc
The mechanism is like;
When random bits are requested, return that many bits derived from the entropy pool (by a cryptographic hash function, say) and decrement the estimate of the number of random bits remaining in the pool. If not enough unknown bits are available, wait until enough are available.
So it is blocking.

Okay, lets take a look to the /dev/urandom..
/dev/urandom is an unlimited, non-blocking random source, which reuses the internal pool to produce more pseudo-random bits.

The key difference above is that /dev/random is a blocking number generator, so it may block the processes which are trying get random number from it.

So, this blocking situation may arise if the remaining bits decrease in the entropy pool..

But how this affectes Weblogic processes?

We have /dev/urandom set in our java.security files by default, (java.security file is in the JRE folder of the JDK that our WEblogic server uses -- jre\lib\security\java.security) ,so we should not get affected from it..

Actually , yes we may be affected. Our weblogic processes may hang and these hang situation can make us feel the slowness even in the weblogic console screens..

So lets look at this this bug : 

JDK-6202721 : SHA1PRNG reads from /dev/random even if /dev/urandom selected

If you do

import java.security.SecureRandom;

class JRand {

  public static void main(String args[]) throws Exception {

    System.out.println("Ok: " +

      SecureRandom.getInstance("SHA1PRNG").nextLong());

  }

}

then SecureRandom will read from /dev/random even if securerandom.source is configured to use /dev/urandom. This is a problem if /dev/urandom was chosen because /dev/random is not working properly.

The root cause is that 4705093 assigned special meaning to the string "/dev/urandom".

So it is magic. When you say /dev/urandom, it goes to /dev/urandom :)

Workaround 1)

The workaround is specfiying securerandom.source=file:/dev/./urandom rahter than securerandom.source=file:/dev/urandom to eliminate the magic. :)

Workaround 2)

Using Oracle Linux 's "rngd" which can generate entropy data for the /dev/random pool following certain rules that can defined in a configuration file.

Workaround 3)

In setDomainEnv.sh
if [ "${USER_MEM_ARGS}" != "" ] ; then
MEM_ARGS="${USER_MEM_ARGS}
export MEM_ARGS
fi
MEM_ARGS="${MEM_ARGS} -Djava.security.egd=file:/dev/./urandom"
Workaround 4)
While configuring your domain;
export CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"
/u01/app/oracle/product/fmw/wlserver_12.1/common/bin/config.sh

So, the behaviour of java for /dev/urandom is confusing but JDK 8 seems promising for clearing this confusion..

If you call: 

 o new SecureRandom() on Linux and the default values are used, it will read from /dev/urandom and not block. (By default on Solaris, the PKCS11 SecureRandom is used, and also calls into /dev/urandom.) 

 o SecureRandom.getInstance("SHA1PRNG") and do not specify a seed, *OR* new SecureRandom() but have specified an alternate java.security.egd besides "file:/dev/urandom", it will use the SHA1PRNG which calls into /dev/random and may potentially block. 

 o SecureRandom.getInstance("NativePRNG"), it will depend on what java.security.egd is pointing to.