Simultaneous load files and do stuff

Here's an idea to throw about .... I know lots of people have been wanting to load files (or do other stuff) at the same time as keeping things in the game running, like animations, or menu activity, etc. And while threads on the Windows version has been one possibility, it's not widespread.

So the idea is ... what if you break all the large files down into lots of small files, where each small file is a chunk of a bigger one. Then you just load one small file per frame, do your animations etc, and proceed from there?

Would that be fast enough to maintain a smooth framerate? Although it would involve a `file seek` operation for each file, and hoping that all the files are stored fairly sequentially and not fragmented/scattered, maybe it would work?

Running an mpeg animation from disk spools the data one chunk at a time from a single file. The only difference here would be that each chunk has a file `header` and seek operation associated with it?

Thoughts?

Nope. Threading is the only intelligent option.

It works fine for my purposes, but my game is not exactly going overboard on graphics and speed is not as critical as in some games. I'm not sure I'd wanna do it while playing a level of a 2d shooter, but with a menu or something like that, it works fine.

If you stream the files yourself, simply have resource manager that you can update and which loads a small part of the file each frame, keeps track of progress etc.
That used to work sweet on the old windows games I used to do for a living :)

Yeah, ozak is right, while BlitzMax doesn't support threading, you can simulate it with some careful programming. You can create like an update function, which handles loading data section at a time. If you add an inner loop to that, you could allow it to run for a certain number of milliseconds before returning, thus leaving the rest of your program running smoothly.

I wrote a blitz3d module to handle this kind of background loading, but never put it into full use, so its hard to say how well it'll run. Still figuring out BlitzMax so it's hard to say how easily the same module will be implementable in BlitzMax.

While this is technically true, it's practically useless. First of all, the ammount of data you can load today in say, 2 millisecs is going to be less than you can load in 5 years. Second of all, the instance your application tries to do I/O it gets kicked out of the short term scheduling queue, and doesn't get back in, until I/O is finished.

So while it's technically possible it will come with a huge performance hit compared to using threading, where your game will keep runnning until it's preempted.

This is BlitzPlus/3d code but it demonstartes anyway.
You could get 1000 bytes in a frame even while doing alot of stuff:

;
;REQUIRES
;
;an image called s.png, prefferably 256*256 or smaller: My Size 4.47 KB: Demo suited to run at 60 fps withough filr reading to observe effects.
;
;a file called "file.txt". Here its 232 KB.


Graphics 640,480,0,2

SetBuffer BackBuffer()
SeedRnd MilliSecs()

img=LoadImage("s.png")
If( Not img) RuntimeError"U Need an image called s.png!, or else change it to w~ever"
MaskImage img,$FF,$FF,$FF

tmr=CreateTimer(60)

ClsColor 128,0,0


FILE=OpenFile("File.txt")
If(Not FILE) RuntimeError"U Need a file called File.txt!, or else change it to w~ever"



fps#=0
Dontrefreshfps=False
ex=False
While Not KeyHit(1)

	Cls
	
	For u=0 To 1000
		If(Eof(FILE))
			Notify"Finished reading file"
			ex=True
			Exit
		EndIf
		ReadByte(FILE)
		Read_Bytes=Read_Bytes+1
	Next
	If ex Exit
	
	;STUFF TO SIMULATE...STUFF GOING ON BEGINS HERE
	
	;to simulate stuff happening=>realtime stuff, lots of commands as well as code=>instead of looping cause they rep. individual commands
	
	For u=0 To 2	;any more and it runs less than 60 fps WITHOUGHT file streaming
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	DrawImage img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img))
	
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0
	If(ImagesCollide(img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0,img,Rnd(640-ImageWidth(img)),Rnd(480-ImageHeight(img)),0))hohoho=0

	For t=0 To 50
		x=x*b+Sin(3)*Cos(3454+ef)/(Rnd(2)+3556)
		x=x+1
		x=x*1
		b=12
		x=x*(b Shl 1)
		x=x*b+Sin(3)*Cos(3454+ef)/(Rnd(2)+3556)
		x=x+1
		x=x*1
		b=12
		x=x*(b Shl 1)
		x=x*b+Sin(3)*Cos(3454+ef)/(Rnd(2)+3556)
		x=x+1
		x=x*1
		b=12
		x=x*(b Shl 1)
		x=x*b+Sin(3)*Cos(3454+ef)/(Rnd(2)+3556)
		x=x+1
		x=x*1
		b=12
		x=x*(b Shl 1)
	Next
	
	Next
	
	;STUFF TO SIMULATE...STUFF GOING ON ENDS HERE
		
	lengthOfFrame=MilliSecs()-LastFrame
	LastFrame=MilliSecs()
	
	If Not Dontrefreshfps
		;as a fraction of a second
		fps#=1000.0/lengthOfFrame
	EndIf
	
	Text 0,0,"fps("+Int%(fps)+");"
	Text 0,20,"Read :"+Read_Bytes+"Bytes of File"
	
	Dontrefreshfps=Dontrefreshfps+1
	If(Dontrefreshfps>30)Dontrefreshfps=False ;and so it'll refresh
	
	Flip()
	
	If(KeyDown(56) And KeyDown(62))Exit
	
	WaitTimer(tmr)
Wend

CloseFile(FILE)
FreeTimer(tmr)
FreeImage(img)

End

Not exactly five years lol.

well, im shure one can split the data flow in some cases, i wrote a file share engine that handles "unlimited" amount of different upploads and downloads in main app by read/writing X bytes in each file, and i never had any problems with it, it might not be the fastest solution but it worked very nicely :)
im shure one can use a simular teqnicue for loading game data.

i dont think it whod be impossible to create some sort of class for this, any way for loading images, sound and sutch...

I think what FlameDuck is getting at is that doing it any other way than using threading is actually a bit crazy because of the severe performance penalty involved.

If you need to load lots of data (preloading the next level with lots of textures) and you want the user to sit there looking at an interesting screen, rather than a static "Loading..." then you'd best come up with an impresseive-yet-cpu-trivial effect.

If you need to load/save only a small amount of data (16Mb or less) then consider if a 0.5s pause during a menu is really that much of a bad thing. Many hard drives have 8 or 16MB cache these days, so your output gets pumped through at 100MB/s+ and will take less than 20ms (best-case scenario).

The difficulty comes if you need to stream a lot of data during play (recording a demo, or so the user can watch a replay and you have a lot going on in-game). Probably the best option at the moment would be to keep all the data in RAM.

I know what he is saying, and i know it will be slower, but i dont think it will be that slow(depending how one solve it), and if i have to make a shoise, to ether load graphic for 5sec and have a feazed lockdown the hole time(showing some static booring looading screen), or load for 7-8sec and be able to do fully graphical representation of what is loaded and how, i think i whod prefer that, extra so if i want some sort of graphical consol showing evrything loaded by engine and still be able to render it smothly.

shure, this solution might not be soutibel for all types of massive data, but im shure it whod be usable still.

i dont care mutch about performence during a loading screen, what i realy want it to know the system havent locked down or freazed, somthing that might be hard to tell with a static loading screen, shure one can do that after each LoadImage/sound or whatever, but it still feals a bit static to me.

Threading is exactly as you describe, load the files in small chunks. It's just that Windows does that for you. The only difference is that Windows doesnt need to open/close file buffers in each frame.

I know what threads is and how it works, that is Not the point... im saying that one can work around it or emulate it, That is all i say.

a nother problem is that bmax is not writen for Threads, Mark has said this himself and stuff might crash for strange reasons if you put stuff in threads, but if you insted emulate loading functions thay might be alot slower, but insted it shod always work, and on all platforms without any DLL's or other local OS specefic libs.
the only thread stuff i have seen so far in bmax only works for win32 platforms.... an alternative solution might work on any OS, if loading data "threaded" is all we want to do.

ther is alot of people ho say "you must use threads" and my point is, that you dont... it might be the most optimized solution in most cases, but ther is still other areas that other simular solutions might work very nicely for a specefik task.

I think even though Max isn't written for threads and isn't threadsafe, threads could be used in a couple of places if you're very careful, and file loading is probably one of those places.

Mark's understandably reluctant to add official threaded file loading support to Max, because of the support nightmare that would create, but that doesn't mean that 3rd-party mods can't be created and used.

I tried to do a sound background loader (with Antony's thread module), just threading a function with a LoadSound and it crashed, even if the main thread was doing nothing in the meantime... I think loading (at least with Bmax load functions) + threading is probably not a good idea atm.

But, if anyone achieves something I'd like to hear about it

Oh yeah, it would be very unsafe to use Max's loading commands in another thread, but you should have no problem reading in a file using C io.

it seems that Mark is working on a simple implementation threads (atleast file-wise).. atleast from what I could gather from his last worklog.. :)