Loading time

It takes like 2 minutes to load the world for my RPG, every time I run the program. The map is 300x300 tiles, and each tile has 1 to 5 layers.
It takes almost a minute to create the tiles and a minute to load them. But the collisionsetup function finishes instantly.
I can't work like this, its too slow.
Does anyone have any tips on how to speed this up?
There are no entrances or guys on this map but there will be.
The map file is over 5 MB.

Method Load()
	Local file:tstream
	
	file=ReadFile("map/"+id+".dat")

	sizex=Readint(file)
	sizey=Readint(file)
	
	array=New ttile[sizex+1,sizey+1]

	
	Local forx,fory
	Local tile:ttile
	Local layer
	Local idread
	Local amread
	Local levelread
	Local xread,yread
	Local battleidread
	Local guy:tguy
	Local f
	Local tilelayered:ttilelayered
	Local rgb
	
	tfont.draw "creating tiles",10,10
	Flip
	Cls
	For fory=1 To sizey
		For forx=1 To sizex
			tile=New ttile
			array[forx,fory]=tile
			tile.x=forx
			tile.y=fory
			tile.guylist=CreateList()
		Next
	Next
	
	For fory=1 To sizey
		If fory Mod 20=0 Then
			tfont.draw "loading tiles: "+100*fory/sizey+"%",10,10
			Flip
			Cls
			If KeyHit(key_escape) Then End
		EndIf
		For forx=1 To sizex
			tile=array[forx,fory]
			For layer=1 To ttile.layeram
				idread=Readint(file)
				If idread<>0 Then
					tilelayered=New ttilelayered
					tile.layers[layer]=tilelayered
					
					tilelayered.kind=ttilekind.array[idread]
					
					xread=Readint(file)
					yread=Readint(file)
					If xread=forx And yread=fory Then 'common speedup
						tilelayered.drawtile=tile
					Else
						tilelayered.drawtile=array[xread,yread]
					EndIf
					
					battleidread=Readint(file)
					tilelayered.battleid=battleidread
					If battleidread>battleidlast Then battleidlast=battleidread
					
					tilelayered.scalex=ReadFloat(file)
					tilelayered.scaley=ReadFloat(file)
					
					tilelayered.handlex2=Readint(file)
					tilelayered.handley2=Readint(file)
					
					

					tilelayered.color=New Byte[3]
					For rgb=0 To 2
						tilelayered.color[rgb]=ReadByte(file)
					Next
					If tilelayered.color[0]=255 ..
					 And tilelayered.color[1]=255 ..
					 And tilelayered.color[2]=255 Then
						tilelayered.color=colorfull
					EndIf

				EndIf
			Next
			
			'load entrance on tile
			idread=Readint(file)
			If idread<>0 Then
				tentrance.make(tile)
				tile.entrance.starttilex=Readint(file)
				tile.entrance.starttiley=Readint(file)
				tile.entrance.mapid=idread
				tile.entrance.areaid=Readint(file)
				tile.entrance.startareaid=Readint(file)
				ListAddLast tentrance.list,tile.entrance
			EndIf
			
			'load guys on tile
			amread=Readint(file)
			If amread<0 Or amread>25 Then RuntimeError "guy am read="+amread
			If amread<>0 Then
				For f=1 To amread
					idread=Readint(file)
					levelread=Readint(file)
					guy=tguy.make(tguykind.array[idread],False,levelread,False)
					guy.id=Readint(file)
					guy.x=Readint(file)
					guy.y=Readint(file)
					guy.dir=tdir.array[Readint(file)]
					guy.map=Self
				Next
			EndIf

		Next
	Next
	
	'battle id
	battleidhigh=Readint(file)
	Local i
	For f=1 To battleidhigh
		For i=1 To tguykind.am
			battleappear[f,i]=ReadShort(file)
			battlelevel[f,i,1]=ReadShort(file)
			battlelevel[f,i,2]=ReadShort(file)
		Next
	Next
	
	Local temp=Readint(file)
	If temp<>999 Then RuntimeError "the sentinel failed!"

	CloseFile file

	tfont.draw "setting tile collision",10,10
	Flip
	Cls
	
	self.collisionsetup()
EndMethod




Method CollisionSetup()
	Local forx,fory
	Local areax,areay
	Local tile:ttile
	Local addx,addy
	Local tilelayered:ttilelayered
	Local layer
	Local x1,x2,y1,y2
	
	x1=1; x2=sizex
	y1=1; y2=sizey
	
	
	For fory=y1 To y2
		For forx=x1 To x2
	
			tile=array[forx,fory]
			
			For areay=0 To 2
				For areax=0 To 2
					tile.battleid[areax,areay]=-1
				Next
			Next

			For layer=ttile.layeram To 1 Step -1

				tilelayered=tile.layers[layer]
				If tilelayered<>Null Then
				
					addx=tile.x-(tilelayered.drawtile.x-(tilelayered.kind.sizex-1))
					addy=tile.y-..
					 (tilelayered.drawtile.y+tilelayered.kind.drawtileyless-(tilelayered.kind.sizey-1))

					For areay=0 To 2
						For areax=0 To 2
							If tilelayered.kind.area[addx,addy][areax,areay]=1 Then
								If tile.collision[areax,areay]=0 Then
									tile.collision[areax,areay] = tilelayered.kind.walk
								EndIf
							EndIf
							If layer<ttile.layeram Then
								If tilelayered.kind.areawalk[addx,addy][areax,areay]=1..
								 Or tilelayered.kind.area[addx,addy][areax,areay]=1 Then
									If tile.battleid[areax,areay]=-1 Then
										tile.battleid[areax,areay]=tilelayered.battleid
									EndIf
								EndIf
							EndIf
						Next
					Next

				EndIf
			Next
		Next
	Next
EndMethod

There could be several things causing the slowdown, I suggest profiling your code to find out for certain where the speed problem is. By profiling, I mean use the MilliSecs() function to find out how long certain sections of code are taking to execute.

Some other info needed:
- What format are the images saved in
- How large is each tile (in pixels)

Something which usually helps in any project - insert a call to FlushMem after creating / loading each tile.

Well first I would suugest not making your tiles Types. Use plain arrays. Second, I just tried this with my own map editor, organize the map file into large common chunks. So for each layer put all of the data together in one chunck then you can do something like this:

For Local y:Int=0 Until mapheight
Local bptr:Byte Ptr = Varptr newmap.layers[l].tiles[0,y]
mfs.ReadBytes( bptr,mapwidth*4 )
Next

I found that the reading of each tile value one at a time with ReadInt is extreemly slow. With the above method you are rading huge chunks of data very quickly. In my test I was able to to read a 3 layer map that was 366 megabytes in 2 seconds. Before I changed to doing the above it was taking almost a minute. Note that you have to deal with the fact that the blitz code is no longer deal with big/little endedness in this situation.

EDIT:

looks like you can even handle the endedness with by using a BankStream. Read the data into a bank and then create a BankStream on that bank.

well I'm trying this full on and not getting good results. I'll let you know if I find the problem. Right now it looks like using Banks and BankStreams is just as slow as normal file streams.

.... EDIT

Okay I can now load a 5000x5000 map with 3 full layers in the above way in about 12 seconds. This is not using the endian streams.

ok, I'm not at home at the moment so can't do any testing of the code, but I've noticed a lack of flushmem in your for loops (especially ones that are instancing types each iteration).

This may well be one of the causes.

Thanks for quick replies

OMG I thought yall were crazy
I just stuck flushmem in the loop and it loads in like 2 seconds!
Could someone tell me what exactly is being flushed that makes this take 60 seconds...

FlushMem
For fory=1 To sizey
	For forx=1 To sizex
		tile=New ttile
		array[forx,fory]=tile
		tile.x=forx
		tile.y=fory
		tile.guylist=CreateList()
	Next
Next

But this takes 2 seconds

For fory=1 To sizey
	FlushMem
	For forx=1 To sizex
		tile=New ttile
		array[forx,fory]=tile
		tile.x=forx
		tile.y=fory
		tile.guylist=CreateList()
	Next
Next

Its not loading any images. Just makeing types and loading their data for a file. I could have mentioned that, but, nah



Thats interesting
Ive never used the readbytes command, but I think I could use it. What does endian mean by the way?

I am also loading thousands of x and y handles for character images, 1 by 1, with readint() and I want it sped up, because it takes like 10 seconds. with flushmem ;)
What if I just loaded the whole file as a chunk in a 1D array?

Endian refers to the why in which the particular hardware platform stores multibyte values. Intel chips are usually little indian and motorola chips are usually big endian.

0xABCD writting to a file on Windows would be

DCBA

and on mac

ABCD

I think I got that right. If you want binary files to be cross platform for your program make sure you write them and read them with the same endian stream types.

I think the flushmem is deleting your unused lists created during each for/next (90,000 of them?)
Silly test...

start_time=MilliSecs()
For x = 1 To 300
  FlushMem()
  For y = 1 To 300
    mylist:TList=CreateList()
  Next
Next
end_time=MilliSecs()
Print end_time-start_time

and take out the flushmem to see the difference.

well all my lists have a different varaible pointing to them.
this is what I am really doing.
But ya, flushmem makes it go way faster even though it is not deleteing any lists here.
It must be CreateList() that is leaveing all the garbage!

Local listarray:TList[300+1,300+1]

start_time=MilliSecs()

For x = 1 To 300
  FlushMem()
  For y = 1 To 300
    listarray[x,y]=CreateList()
  Next
Next

end_time=MilliSecs()
Print end_time-start_time

Scott Shaver: thanks for the help, how did you find out about the readbytes command?

Isn't learning about your new fa. language fun?!

Glad that seems to have cured it. Note to self - Flush! Fluch and thrice Flush

IPete2.

Digging through the module code to figure out why it is so slow. :)

There's a thread where the suggestion was a single flushmem should be OK *unless* a function or for/next was doing a lot of work. Maybe this is that situation... possibly?

Okay so here is what I ended up doing to speed up the loading of huge maps in my editor:

' this is a speed optimization, istead of reading each int one at a time
' we read them in in blocks, much faster but we have to deal with the endian stuff
' ourselves
Local temp:Int[] = New Int[mcw]
Local bptr:Byte Ptr = Varptr temp[0]
' cell tile index data
For Local y:Int=0 Until mch
	FlushMem
	mfs.ReadBytes( bptr,mcw*4 )
	For Local x:Int=0 Until mcw
?BigEndian
		Local t ' only do the byte swapping if on a big endian system, file is little endian
		t=bptr[(x*4)];bptr[(x*4)]=bptr[(x*4)+3];bptr[(x*4)+3]=t
		t=bptr[1];bptr[(x*4)+1]=bptr[(x*4)+2];bptr[(x*4)+2]=t
?
		newmap.layers[l].tiles[x,y]=temp[x]
	Next
Next

Arrg, Having readable text for data files is so much more convenient.

Can you give us a rundown of how much faster your system was with just a flushmem in the loop.

How much better did it get by going to a binary format?

Actually with regards to the endian thing, I believe 0xABCD is stored on Intel as CDAB - remember, each $xx is 1 byte.

haha, I dont know what the hell is going on anymore/

I would hazzard a guess that the lower memory footprint causes less allocations/swaps/cache misses (take a pick).

http://www.blitzwiki.org/index.php/FlushMem

Not sure if this was addressed to me but my map format has always been binary. Loading each int with readint was taking around 1 minute 30 seconds. Using a bank to load each horizontal row took about 1 minute. Doing what I did in the code above took it down to about 15 seconds. Understand that I'm test this with a outrageously large map. 5000x5000 with 3 layers so I'm loading 75,000,000 integers (the tile index for each cell) and 75,000,000 bytes (the alpha values for each cell). The total map file size on disk is 366 meg.

Adding the flushmem in my case has no effect what so ever. But I'm not creating any objects in my code for the map cells, I use an object that holds straight arrays.

ReadBank() is the new ReadBytes().