OutOfMemory Exception?

[Mic]

I would appreciate some help regarding Java's memory model.

At times I get

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at org.apache.commons.httpclient.util.TimeoutController.execute(TimeoutController.java:61)
        at org.apache.commons.httpclient.util.TimeoutController.execute(TimeoutController.java:82)
        at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory.createSocket(ControllerThreadSocketFactory.java:95)
        at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:128)
        at wora.service.access.http.MeasurementProtocolSocketFactory.createSocket(MeasurementProtocolSocketFactory.java:61)

There I am creating a new socket for my http client:

     public Socket createSocket(final String host, final int port,
			final InetAddress localAddress, final int localPort,
			final HttpConnectionParams params) throws IOException {
		startTime.set(System.currentTimeMillis());
		Socket socket;
		try {
			socket = delegate.createSocket(host, port, localAddress, localPort,
					params);
			return socket;
		} catch (OutOfMemoryError e) {
			ServerHealthBacking.logIt();
			throw (e);
		}
	}

As you can see above I already introduced some logging when this OutOfMemory occurs. The memory looks like this at that point of time:

heap: init = 0(0K) used = 28684528(28012K) committed = 45522944(44456K) max = 166526976(162624K) commited usage=63% max usage=17%
non heap: init = 8552448(8352K) used = 41372552(40402K) committed = 42303488(41312K) max = 100663296(98304K) commited usage=97% max usage=41%

Code Cache - NOW : init = 163840(160K) used = 6639296(6483K) committed = 6651904(6496K) max = 33554432(32768K) commited usage=99% max usage=19%
Code Cache - COLLECTION : null
Eden Space - NOW : init = 524288(512K) used = 1419056(1385K) committed = 2949120(2880K) max = 10354688(10112K) commited usage=48% max usage=13%
Eden Space - COLLECTION : init = 524288(512K) used = 0(0K) committed = 2949120(2880K) max = 10354688(10112K) commited usage=0% max usage=0%
Survivor Space - NOW : init = 65536(64K) used = 67472(65K) committed = 327680(320K) max = 1245184(1216K) commited usage=20% max usage=5%
Survivor Space - COLLECTION : init = 65536(64K) used = 67472(65K) committed = 327680(320K) max = 1245184(1216K) commited usage=20% max usage=5%
Tenured Gen - NOW : init = 1441792(1408K) used = 27198000(26560K) committed = 42246144(41256K) max = 154927104(151296K) commited usage=64% max usage=1
7%
Tenured Gen - COLLECTION : init = 1441792(1408K) used = 16836616(16442K) committed = 28061696(27404K) max = 154927104(151296K) commited usage=59% max
usage=10%
Perm Gen - NOW : init = 8388608(8192K) used = 34733256(33919K) committed = 35651584(34816K) max = 67108864(65536K) commited usage=97% max usage=51%
Perm Gen - COLLECTION : init = 8388608(8192K) used = 33552040(32765K) committed = 33554432(32768K) max = 67108864(65536K) commited usage=99% max usage
=49%

So I am not really a professional regarding Java's memory model, but why does the VM not just increase the commited space since max space is not full?

Are there any tuning options instead of increasing the total memory, which is not an option in my case?

Thanks for your help!
Michael

 

[Verjigorm]

You can add VM arguments like: -Xms128M -Xmx512M to set the min and max memory for the JVM.

JDK's Monitoring & Troubleshooting Tools - java.OutOfMemoryException

 

[Mic]

You can add VM arguments like: -Xms128M -Xmx512M to set the min and max memory for the JVM.

JDK's Monitoring & Troubleshooting Tools - java.OutOfMemoryException

That's a good tip but I want to go with that amount of memory and maybe make the VM use it more efficiently. Apparently there is enough memory available as I would think when looking at the memory dump.

 

[maki]

You should never catch an Error unless you know exactly what you are doing.

Do you know how many Threads you're actually starting before you get the OOME?

 

[Mic]

You should never catch an Error unless you know exactly what you are doing.

Do you know how many Threads you're actually starting before you get the OOME?

I tend to believe that I know what I am doing, especially as I am throwing that catched exception again.

thread count=86, daemon threads=74, peak=88, total started=10381

Maybe one interesting point is that the exception does not occur at times of high traffic on my portal. Tomcat's number of used threads has been at 2 at that time while it is at 50 at other times and no exception does occur.

 

[maki]

Sounds funny, but this special kind of OOME (unable to create new native thread) can indicate that your VM is too big, leaving not enough resources for the OS.

Maybe you could try to reduce the VM Memory and check if you're able to start more threads than before.

 

[Mic]

Sounds funny, but this special kind of OOME (unable to create new native thread) can indicate that your VM is too big, leaving not enough resources for the OS.

Maybe you could try to reduce the VM Memory and check if you're able to start more threads than before.

At the moment it looks like this...

# free
total used free shared buffers cached
Mem: 1048576 833156 215420 0 0 0
-/+ buffers/cache: 833156 215420
Swap: 0 0 0

How could I get a "low memory alert" from Debian?

 

[Der Müde Joe]

>How could I get a "low memory alert" from Debian?

try "top"
when linux starts swapping you will know

EDIT:
looks like you do not have any swap?
Swap: 0 0 0

mine is like:
total used free shared buffers cached
Mem: 3088884 1786932 1301952 0 187864 622076
-/+ buffers/cache: 976992 2111892
Swap: 3927884 0 3927884

 

[maki]

How could I get a "low memory alert" from Debian?

You probably won't, all you'll get is an OOME "unable to create new native thread"

Could you try to reduce the stack size (-Xss256k, AFAIK the default is 512k)?

 

[Mic]

>How could I get a "low memory alert" from Debian?

try "top"
when linux starts swapping you will know

EDIT:
looks like you do not have any swap?
Swap: 0 0 0

mine is like:
total used free shared buffers cached
Mem: 3088884 1786932 1301952 0 187864 622076
-/+ buffers/cache: 976992 2111892
Swap: 3927884 0 3927884

I think missing swap display is due to that it is just a virtual server.

Problem with top is that I can't have an eye on it until that OOME occurs every 3 days...

 

[Mic]

You probably won't, all you'll get is an OOME "unable to create new native thread"

Could you try to reduce the stack size (-Xss256k, AFAIK the default is 512k)?

Maybe there is any debian log showing a thread limit reached?

I will try reducing the memory, but since the OOME occurs not really often it will take some time to see or not see the effects...

 

[maki]

Maybe JMeter could help reproducing the problem.

Personally i think it's the OS that's running out of Resources, not the JVM, but i could be wrong there.

 

[Mic]

Maybe JMeter could help reproducing the problem.

Personally i think it's the OS that's running out of Resources, not the JVM, but i could be wrong there.

I already did some load testing but could not reproduce it. It seems not to be related to load... but maybe really to that virtual server's Debian reaching any limits. I will investigate with my provider.

Thanks for giving me that idea! I could have been debugging the VM's memory for months...