Debugging Memory Leaks

I haven't currently got any, but I'm aware that with developing a game engine, and with my TV3D and other modules underneath, there is a good chance I'm going to come up with one at some stage. I'm wondering what people consider the best practice to spot these early on.

At the moment, I have a GCCollect in the main loop, which I figure is harmless ( except for possible speed issues, but I develop in debug mode, so who cares? ) and means that when I see memory increasing it's because there's a leak and not because the garbage collector can't be bothered to do anything about all the memory I'm finished with.

Is this a decent practice, and do I need to do anything else?

Speed issues?
The garbage had to be collected any way, so surly haveing control over it and ensureing that it is done every frame is a good thing?

well general wisdom on garbage collectors is that they know best when to empty themselves, not you. Perhaps with a game, emptying every frame is more conducive to a consistent framerate, but even then, I don't know if that's true. I've no prior experience with GC's.

There is no memory leak on the BM side.
The only leaks you have to take care of is stuff you get from C side ie everything you get by BytePtr. And there is no way from within BM to find those. You need the process manager or getting some usefull coding style (ie assign it to an object which you use within BM instead of relying on C++ code unstructure and free the byteptr memory in the Method Delete ).

Using unmanaged code which is not encapsulated within managed objects is something hardly liked in a managed environment as it gives you nearly a guarantee that you will have leaks at some point.

I don't think you've understood my question. It's not particularly related to any external modules, although that did occur to me later.

Managed or unmanaged, Garbage Collector or no, anyone can write code which has memory leaks. I'm simply asking the best practices to spot them early.

Thinking out aloud.
Could you use GCCollect within an 'if my_memcheck=1' condition at the beginning and end of each function/method and write out entry/exit gcmemalloced() values into a type. At the end of the program write out the type values along with the function name to a text file. Maybe add them to a function global as exit_gcmemalloced()-entry_gcmemalloced() to see whether it's building up memory?
Could be really clever and add it to a debug module along with 'number of times called' with 'total ms' and 'average ms' to see which functions are memory and ms intensive.

When I found out about memory leaks, I already had a fair bit of code with the problem. It took nearly a month for me to track down and fix all of them.

Now, I'm writing the status of memory to a debug file every frame, as well as use ProcessExplorer to keep an eye on it. I often leave the program running overnight to check whether the mem usage creeps up or whether there's any long term crash/bug problems.

I guess it's similar to how most people handle the problem.

BM has nearly no internal memory leaks anymore.
Thus using GCMemAlloced is quite useless, as it won't give you any type of general memory usage information. Only its own data, which don't leak.

As mentioned: Most common leak is usage of BytePtr within BM but not attached to a GC managed object. Thats a big no no and anyone using such constructs doesn't deserve anything else than memory leaks. (sorry but a little question to those people that don't use BytePtr only as fields within types: Have you ever asked yourself why Microsoft introduced .NET and C# and apple Objective C, if the C++ pointer hacking wouldn't have been system stability critical? If it wouldn't be millions of times easier to introduce a memory leak in unmanaged than finding it again?)

@Dreamora.
Gabriel is talking about programming errors resulting in additional memory being retained when it would normally be released. In terms of this particular program that could be considered a memory 'leak' whether or not the issue is caused by BMax code or not.
In these situations GCMemalloced() will be useful... no?

Yes, thats right. sorry if my reaction might have been over the top.

But the only way getting a BM internal leak is a serious design error in the object hierarchy, as only cyclic references which don't break up the "parent - child" double links are able to create such leaks. So as long as you have a destroy method that null-ifies all references on the type instance, this type of BM internal leak can't happen. (yes it needs to be an own method. Null-ifying within Delete is worthless, as delete won't be called until the references are nullified)

All other things within BM are no leaks (theoretically not even above) as there are still references on the given object and I only count stuff as leak that is still allocated but not referenced by anything anymore. Last thing like that were leaking were sounds, when I remember correctly. (not fully sure if they are completely leak free by now)

How about, for example, ...
1) Create 1000 objects in for/next loop.
2) Attempt to delete those objects but mistype 100 instead of 1000 without noticing.
?

I'm not sure why you don't consider them memory leaks.
BRL program a product called Bmax. If BRL had a hiccup while coding and memory was mis-used you're saying it's a memory leak in Bmax... Yep, agreed.
If I program a product called MyProduct and had a hiccup while coding and memory was mis-used you're saying it's *NOT* a memory leak in MyProduct... ? Then I disagree.
A problem where memory is taken and not released when it should be is a memory leak whatever the product.
IMO.

Don't understand the "example"
You create 1000 objects, ok.
As what?
- local? -> go out of scope automatically and are freed

- local but added to a list? -> list.clear() or someList = null -> out of scope and freed as well

Actually I don't see how you want to create a leak this way, its all managed.


On the "MyProduct": I call it a memory leak there as well. But the BM functionality won't give you any way to find the leak. BM itself does not leak unless you have a serious object design error in your OO constructs which you better hunt on paper where you easily spot it instead of using memory supervicing and stuff like that.
So the only leak you get is the leak from interfacing from outside which GCMemAlloced won't tell you anything about nor does it worry about that as it isn't BMs problem or area of interest.


I hope you see my point now.
Leaks within BM mean a serious design error which you better hunt on your class design and functionality paper and not at runtime as this won't help you much. (thats the way Microsoft hunted bugs in Win98 and ME and we all know the result)


And if you train yourself to only use unmanaged stuff linked to a managed object, you will find yourself in the happy situation where memory leaks become a very rare thing.

I don't *WANT* to create the leak.. that's why memory leaks are programming errors.
Maybe it was a poor example. Basically, if you create 1000 objects, stick them on a list but only remove 100 when you were supposed to remove 1000, you'll get a 'memory leak'.

BM function helps you find 'Memory leaks' in your own code via the gcmemalloced() command.
<edit> In case it explains better...

Type ttest
	Field COUNT:Int
	Field NAME:String
End Type
test_list:TList=CreateList()
For x = 1 To 100
	GCCollect()
	GC1:Int=GCMemAlloced()
	For x = 1 To 1000
	   temp:TTest = New TTEST
		TEMP.COUNT=X
		TEMP.NAME="basdkjhfsaeifkashfklasehfklasdhfkasdhfklasdhflkash"
	   ListAddLast test_list,temp
	Next
	GCCollect()
	GC2:Int=GCMemAlloced()
	For x=1 To 100   ' This should read 1000 but ***programming error*** I put 100.
	 	test_list.remove(test_list.first())
	Next
	GCCollect()
	gc3:Int=GCMemAlloced()
Next
Print gc1 + " " + gc2 + " " + gc3

Thats no leak.
A leak is uncontrolled lose of memory.
Yours will stay at that point.

But perhaps you start to see the point of managed functionality and modern mechanics:

TList.clear() is one
For local t:TTest = eachin test_list is the other

This example is one of those good example that show how B3D programmers approach a solution from the counter productive way. You want to perform an operation on all entries in a list, so do it with the correct mechanics and you won't run into problems.
Thats quite exactly what I meant by serious problems in the design. Stuff that is designed this way is great for C++ ... so for 10 year old apps. But we are in 2006 and BM offers some of the modern mechanics and it is just nonsense to not use them just because one is used to "how the old stuff" worked and I don't mind if such behavior results in many hours of debugging. perhaps people then will learn the new possibilities offered to them and understand their true power.

PS: Even in the old way you wouldn't do it with a number based loop. You would do: while not test_list.isEmpty() remove thats good and secure code as it has a clearly stated break condition. using count variables on a dynamic structure like a list is garbage. thats good for arrays where an indexed access exists.

I'm also not used to these garbage collectors.
Mostly I have the taskmanager open behind the running app and try to watch out for leaks arising. This takes a lot of time and patience.

But how about calling "GCSuspend" and "GCResume" to track leaks? Or is that too simple? :)

I like this. It could be quite difficult to implement cleanly, but I'll have a think about this, because it could prove invaluable if I can implement it right.



Sounds good. I'm finding that my program burns memory like it's going out of fashion just for using local variables in the main loop to cycle through objects in a tlist. It worried me at first but then I realised that the GC was just taking it's sweet time to clean up after me. Now that I've added a GCCollect in the main loop, this seems to work reasonably well though.


That's because it's an example. I don't know if you're being intentionally obtuse to annoy TonyG, but his examples make perfect sense to me. If ( to use his example ) you create 1000 objects and delete 999 of them, but it's in your main game loop, it IS a memory leak. You only need to miss one. As much as you like to go on about managed functionality, BMax object control and being leakproof, it's really easy to get memory leaks, and not just with cyclic references ( which are a reality in most complex projects anyway. ) There are all sorts of ways to allocate memory both internally and externally which the GC does not even manage. If the GC isn't managing it, you are and if you're NOT, you've got memory leaks.


I'm not sure what you mean. If I shutdown the garbage collector, I'm just going to stop things from being cleaned up, so surely that makes it even harder for me to spot leaks? I've been doing the opposite, and forcing the GC to collect every frame specifically because that makes it easier for me to spot memory usage increasing. How would I do what you're suggesting?

Here's a memory leak. Not blitzmax's fault but still. I believe GCMemAlloced() will show you that you have a memory leak. But you have to find it.

'main loop
repeat
update()
draw()
listaddlast nodelist,new node 'oops
forever

There is no leak.
You just uncontrolled add useless stuff without ever removing anything.

You just flood the memory because the design leaks not the program. this most likely happens if there is not enough time invested into designing the software but only writting it and that although at least 50% of the time is actually designing. At least if you plan to write it in a way that prevents you from rewritting it 3 times.

As per Gabe.
In a way I hope you ARE being obtuse as your inability to understand this thread is startling.
Anyway, if memory is taken and not released then memory has leaked. 'Managed functionality' can't compensate for 'stupidity'.

Leak: memory that can not be accessed anymore but that is still allocated.
Within a managed environment, normally this can't happen. In BM its possible to create that as BM does not have root references. So it is possible to have a whole cyclic group of objects within GC memory pool without the ability to access and thus remove it anymore. The chance that this happens raises with structures that have bidirectional parent-child relationships.
Another possibility is using the MemAlloc functionality or getting memory blocks from extern via BytePtr that is not manually freed through "free".

The above is just stupidity or program design error and has nothing to do with the phenomenom called memory leak.

Defition of memory leak: (which clearly states memory that is not needed anymore -> no usable reference to it anymore)
http://www.webopedia.com/TERM/M/memory_leak.html
http://www.pcmag.com/encyclopedia_term/0,2542,t=memory+leak&i=46775,00.asp

Doesn't that second link cover this situation?

Yep, the program has allocated extra memory but has not freed it up.

Yep, not c++ but it's MyProduct, I've allocated memory to hold data and, as the programmer, not remembered to deallocate it.

Yep again. memory has been allocated and not deallocated.

Yep, that'd happen.
Alternatively, you could ignore you posted that link and refer to the first.
Not sure why you didn't link to our friend
<edit> In fact the first definition is open to interpretation as well.