Paul McLachlan
2010-11-07 08:10:09 UTC
I’d like to chronicle my experiences fixing a memory leak in our
enterprise GWT application when running on Internet Explorer.
A few facts getting started:
1. Using a click-recording tool, QA could get our application to
leak hundreds of megabytes of memory in Internet Explorer.
2. No leak measured using Java memory analysis tools in hosted mode.
3. Non-trivial application - we have over 100K SLOC just for GWT
(ie, not counting server side or any non .java files)
Reproducibility was handled by QA, but the first problem was working
out what was going on. We didn't see these kinds of problems with
Firefox & this strongly implies some kind of Internet Explorer
circular reference strangeness between DOM elements and javascript.
We spent some time playing with Drip and sIEve, but we were advised
not to use these tools (misleading output) and didn't have wonderful
success with them in any case.
A bit more googling found a javascript memory leak analyzer from
Microsoft at: http://blogs.msdn.com/b/gpde/archive/2009/08/03/javascript-memory-leak-detector-v2.aspx
. That's actually the v2 and most google hits send you to a (now
removed v1), so it's a little difficult to find.
In any case, the results of using Microsoft's javascript memory leak
detector are basically unintelligible in a regular production mode GWT
compile. I had more luck when compiling with -style PRETTY
(obviously), and I also turned on -draftCompile. I think I remember
that -draftCompile did less inlining, which made the generated
javascript closer to our Java code. This was important because the
output of the tool is basically a series of leaks, like:
DIV leaked due to onclick from stack XYZ
where "XYZ" is a javascript stack trace of where the onclick event
handler was set. By clicking back up the stack you can generally get
a reasonable idea of which widget in your application is causing the
problem.
At this point, I didn't actually trust the tool - so from a
methodology perspective my first step was to validate the tools
output.
I commented out code and otherwise configured our application down to
a bare-bones "login, display a couple of things, logout" script that I
could run in a loop. Having done so, I could demonstrate that:
a) that operational loop actually leaked in IE
b) the tool reported about 15 elements as being leaked
Then, I proceeded to ... try to "fix" those leaks.
First attempt was to click the ClickListener (or ClickHandler or
KeyPressHandler or whatever). I mean, calling Handler.remove(), or
removeClickListener() during onDetach(). My theory was that my
ClickHandler was a Java inner class and it somehow just had an inline
reference to the parent object and etc.
No joy. The tool still reported it as a leak. I've since spent a lot
more time going through GWT's implementation of event handling and
it's pretty clear that:
a) the intent is to not have to do this; and
b) it doesn't set element.onclick = null
I understand the argument with b) is that you don't have to. I wasn't
feeling very trusting at this point (I had a memory leak tool from
Microsoft that seemed to disagree with that), so I thought I'd test
it.
Reading http://javascript.crockford.com/memory/leak.html gave me an
idea, so I wrote a little helper method:
public native static void purgeEventHooks( Element elem, boolean
recurse ) /*-{
try {
elem.onclick = null;
elem.ondblclick = null;
elem.onmousedown = null;
elem.onmouseup = null;
elem.onmouseover = null;
elem.onmouseout = null;
elem.onmousemove = null;
elem.onkeydown = null;
elem.onkeypress = null;
elem.onkeyup = null;
elem.onchange = null;
elem.onfocus = null;
elem.onblur = null;
elem.onlosecapture = null;
elem.onscroll = null;
elem.onload = null;
elem.onerror = null;
elem.onmousewheel = null;
elem.oncontextmenu = null;
elem.onpaste = null;
if (recurse) {
var a = elem.childNodes;
if (a) {
l = a.length;
for (i = 0; i < l; i += 1) {
purgeEventHooks(elem.childNodes[i], recurse);
}
}
}
} catch( e ) {
// ignore
}
}-*/;
And then proceeded to call it from onDetach() on the "leaked" element.
Aha - magic -- the Microsoft javascript leak tool no longer reports
that element as a leak!
However, it seems quite possible that all I've managed to do is fool
the leak tool, as opposed to actually fix any leak. So I resolve to
fix all of the leaks on this code path. I eventually managed to do
this (gosh, it was painful -- mostly because it's very difficult from
a javascript stacktrace to where onclick was called to work out which
custom GWT widget is ultimately involved).
Now I have the leak tool reporting no leaks. So, I turn it off and
put the application back into a loop in Internet Explorer --- half
expecting to see it still leaking tens of megabytes and being back
where I started having wasted several days. But... it didn't leak.
IE memory usage varied by a few hundred KB (up and down) throughout
the course of a long test, but no memory leaks.
Soooo - now I'm at a point where I trust the leak tool, and I think
there's some kind of really fundamental problem in this theory that
you don't have to clear DOM element event hooks. So I build a "hello
world" type GWT application intending to, you know, prove this to the
world.
No joy. In that application, (which was just a button added and
removed from a SimplePanel on the RootPanel), it wasn't necessary to
clear the DOM element event hooks to have IE clear everything up. So
there's some "more complicated than trivial" case in which this is
triggered. Unfortunately. Also unfortunately, I have no-where near
the kind of time to go back and trial-and-error work out what that
condition is.
So, now I can't explain why clearing the handlers helps, but I can say
that it does. And I am faced with the really daunting task of trying
to go through our app and explicitly clear event handlers onDetach.
Don't really want to do that -- actually, all I really want to do is
have Widget.onDetach() call, basically DOM.unsinkEvents().
Mucking around in the GWT code, I find a hook that will let me do
that. This is a bit... well, 'distasteful'. But, it worked for me
and got us out of a tight spot / will also let us clear up some of the
other mitigation code we've had in place around this issue.
The observation is that DOM.setElementListener is called on attach
with a value, and on detach with null. And it calls into an abstract
DOMImpl method that I can override with deferred binding (.gwt.xml
<replace-with>).
The implementation of DOMImpl that I use looks like this:
public class NoMemLeaksDOMImplIE8 extends
com.google.gwt.user.client.impl.DOMImplIE8 {
public NoMemLeaksDOMImplIE8() {
super();
}
static native void backupEventBits(Element elem) /*-{
elem.__eventBits_apptio = elem.__eventBits;
}-*/;
static native int getEventBitsBackup(Element elem) /*-{
return (elem.__eventBits_apptio || 0);
}-*/;
static void hookEventListenerChange( DOMImpl impl, Element elem,
EventListener listener ) {
if( listener == null ) {
// Called from onDetach() for Widget (among other places).
//
// Basically, we're going to be detached at this point.
// So.... set all event handlers to null to avoid IE
// circular reference memory leaking badness
backupEventBits( elem );
impl.sinkEvents(elem,0);
} else {
int backup = getEventBitsBackup( elem );
if( backup != 0 ) {
impl.sinkEvents( elem, backup );
}
}
}
public void setEventListener( Element elem, EventListener listener)
{
super.setEventListener( elem, listener );
hookEventListenerChange( this, elem, listener );
}
}
Note that I have to back up the event bits & restore them in case the
element is re-attached back into the DOM later (otherwise it won't get
any of the events it is expecting).
In any case, your actual mileage may vary, but this helps us a LOT.
Lessons learned:
1) Microsoft javascript memory leak detection tool rocks. It is
available from here: http://blogs.msdn.com/b/gpde/archive/2009/08/03/javascript-memory-leak-detector-v2.aspx
It has an automation mode where you could put an "zero leaks"
assertion as part of an automated test.
2) There is some way you can set up a GWT app so that you need to
clear event handlers in order to avoid leaks in IE6, IE7 & IE8. No
plain GWT manipulations I found (removing the clicklisteners, etc)
helped at all. When you get into this state, basically every single
widget you have with an event listener will leak. Microsoft's leak
tool isn't lying, but neither can I explain why or what it is that
pushes the app over this threshold.
3) You can successfully hack the DOM Impl to have Widget.onDetach()
clear the event hooks for you, meaning you don't have to refactor all
your code to follow this pattern.
I hope this helps someone.
- Paul
enterprise GWT application when running on Internet Explorer.
A few facts getting started:
1. Using a click-recording tool, QA could get our application to
leak hundreds of megabytes of memory in Internet Explorer.
2. No leak measured using Java memory analysis tools in hosted mode.
3. Non-trivial application - we have over 100K SLOC just for GWT
(ie, not counting server side or any non .java files)
Reproducibility was handled by QA, but the first problem was working
out what was going on. We didn't see these kinds of problems with
Firefox & this strongly implies some kind of Internet Explorer
circular reference strangeness between DOM elements and javascript.
We spent some time playing with Drip and sIEve, but we were advised
not to use these tools (misleading output) and didn't have wonderful
success with them in any case.
A bit more googling found a javascript memory leak analyzer from
Microsoft at: http://blogs.msdn.com/b/gpde/archive/2009/08/03/javascript-memory-leak-detector-v2.aspx
. That's actually the v2 and most google hits send you to a (now
removed v1), so it's a little difficult to find.
In any case, the results of using Microsoft's javascript memory leak
detector are basically unintelligible in a regular production mode GWT
compile. I had more luck when compiling with -style PRETTY
(obviously), and I also turned on -draftCompile. I think I remember
that -draftCompile did less inlining, which made the generated
javascript closer to our Java code. This was important because the
output of the tool is basically a series of leaks, like:
DIV leaked due to onclick from stack XYZ
where "XYZ" is a javascript stack trace of where the onclick event
handler was set. By clicking back up the stack you can generally get
a reasonable idea of which widget in your application is causing the
problem.
At this point, I didn't actually trust the tool - so from a
methodology perspective my first step was to validate the tools
output.
I commented out code and otherwise configured our application down to
a bare-bones "login, display a couple of things, logout" script that I
could run in a loop. Having done so, I could demonstrate that:
a) that operational loop actually leaked in IE
b) the tool reported about 15 elements as being leaked
Then, I proceeded to ... try to "fix" those leaks.
First attempt was to click the ClickListener (or ClickHandler or
KeyPressHandler or whatever). I mean, calling Handler.remove(), or
removeClickListener() during onDetach(). My theory was that my
ClickHandler was a Java inner class and it somehow just had an inline
reference to the parent object and etc.
No joy. The tool still reported it as a leak. I've since spent a lot
more time going through GWT's implementation of event handling and
it's pretty clear that:
a) the intent is to not have to do this; and
b) it doesn't set element.onclick = null
I understand the argument with b) is that you don't have to. I wasn't
feeling very trusting at this point (I had a memory leak tool from
Microsoft that seemed to disagree with that), so I thought I'd test
it.
Reading http://javascript.crockford.com/memory/leak.html gave me an
idea, so I wrote a little helper method:
public native static void purgeEventHooks( Element elem, boolean
recurse ) /*-{
try {
elem.onclick = null;
elem.ondblclick = null;
elem.onmousedown = null;
elem.onmouseup = null;
elem.onmouseover = null;
elem.onmouseout = null;
elem.onmousemove = null;
elem.onkeydown = null;
elem.onkeypress = null;
elem.onkeyup = null;
elem.onchange = null;
elem.onfocus = null;
elem.onblur = null;
elem.onlosecapture = null;
elem.onscroll = null;
elem.onload = null;
elem.onerror = null;
elem.onmousewheel = null;
elem.oncontextmenu = null;
elem.onpaste = null;
if (recurse) {
var a = elem.childNodes;
if (a) {
l = a.length;
for (i = 0; i < l; i += 1) {
purgeEventHooks(elem.childNodes[i], recurse);
}
}
}
} catch( e ) {
// ignore
}
}-*/;
And then proceeded to call it from onDetach() on the "leaked" element.
Aha - magic -- the Microsoft javascript leak tool no longer reports
that element as a leak!
However, it seems quite possible that all I've managed to do is fool
the leak tool, as opposed to actually fix any leak. So I resolve to
fix all of the leaks on this code path. I eventually managed to do
this (gosh, it was painful -- mostly because it's very difficult from
a javascript stacktrace to where onclick was called to work out which
custom GWT widget is ultimately involved).
Now I have the leak tool reporting no leaks. So, I turn it off and
put the application back into a loop in Internet Explorer --- half
expecting to see it still leaking tens of megabytes and being back
where I started having wasted several days. But... it didn't leak.
IE memory usage varied by a few hundred KB (up and down) throughout
the course of a long test, but no memory leaks.
Soooo - now I'm at a point where I trust the leak tool, and I think
there's some kind of really fundamental problem in this theory that
you don't have to clear DOM element event hooks. So I build a "hello
world" type GWT application intending to, you know, prove this to the
world.
No joy. In that application, (which was just a button added and
removed from a SimplePanel on the RootPanel), it wasn't necessary to
clear the DOM element event hooks to have IE clear everything up. So
there's some "more complicated than trivial" case in which this is
triggered. Unfortunately. Also unfortunately, I have no-where near
the kind of time to go back and trial-and-error work out what that
condition is.
So, now I can't explain why clearing the handlers helps, but I can say
that it does. And I am faced with the really daunting task of trying
to go through our app and explicitly clear event handlers onDetach.
Don't really want to do that -- actually, all I really want to do is
have Widget.onDetach() call, basically DOM.unsinkEvents().
Mucking around in the GWT code, I find a hook that will let me do
that. This is a bit... well, 'distasteful'. But, it worked for me
and got us out of a tight spot / will also let us clear up some of the
other mitigation code we've had in place around this issue.
The observation is that DOM.setElementListener is called on attach
with a value, and on detach with null. And it calls into an abstract
DOMImpl method that I can override with deferred binding (.gwt.xml
<replace-with>).
The implementation of DOMImpl that I use looks like this:
public class NoMemLeaksDOMImplIE8 extends
com.google.gwt.user.client.impl.DOMImplIE8 {
public NoMemLeaksDOMImplIE8() {
super();
}
static native void backupEventBits(Element elem) /*-{
elem.__eventBits_apptio = elem.__eventBits;
}-*/;
static native int getEventBitsBackup(Element elem) /*-{
return (elem.__eventBits_apptio || 0);
}-*/;
static void hookEventListenerChange( DOMImpl impl, Element elem,
EventListener listener ) {
if( listener == null ) {
// Called from onDetach() for Widget (among other places).
//
// Basically, we're going to be detached at this point.
// So.... set all event handlers to null to avoid IE
// circular reference memory leaking badness
backupEventBits( elem );
impl.sinkEvents(elem,0);
} else {
int backup = getEventBitsBackup( elem );
if( backup != 0 ) {
impl.sinkEvents( elem, backup );
}
}
}
public void setEventListener( Element elem, EventListener listener)
{
super.setEventListener( elem, listener );
hookEventListenerChange( this, elem, listener );
}
}
Note that I have to back up the event bits & restore them in case the
element is re-attached back into the DOM later (otherwise it won't get
any of the events it is expecting).
In any case, your actual mileage may vary, but this helps us a LOT.
Lessons learned:
1) Microsoft javascript memory leak detection tool rocks. It is
available from here: http://blogs.msdn.com/b/gpde/archive/2009/08/03/javascript-memory-leak-detector-v2.aspx
It has an automation mode where you could put an "zero leaks"
assertion as part of an automated test.
2) There is some way you can set up a GWT app so that you need to
clear event handlers in order to avoid leaks in IE6, IE7 & IE8. No
plain GWT manipulations I found (removing the clicklisteners, etc)
helped at all. When you get into this state, basically every single
widget you have with an event listener will leak. Microsoft's leak
tool isn't lying, but neither can I explain why or what it is that
pushes the app over this threshold.
3) You can successfully hack the DOM Impl to have Widget.onDetach()
clear the event hooks for you, meaning you don't have to refactor all
your code to follow this pattern.
I hope this helps someone.
- Paul
--
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
To post to this group, send email to google-web-toolkit-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to google-web-toolkit+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
To post to this group, send email to google-web-toolkit-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to google-web-toolkit+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.