Loadimage speeds.

I did a very, very simple test with loading the same image, several thousand times in several different formats. I came up with:

2195ms - png (with transparency. Seems to load much faster)
6699ms - bmp (256 color)
3332ms - bmp (24bit [16mil])
9873ms - tga

I did this test because i'm hoping to find or maybe even create the best pixel perfect vs speed image format. I would have figured a bitmap would load faster, as it doesn't have crazy compression, but it seems gif and jpeg are the winners (and losers of quality).
This test was using the LoadImage() function. I'm just curious if anyone has any theories on how to load images more quickly using a custom format, or by manipulation.

Please discuss.

You could have guessed that from the start, since the GIF and JPG file versions were probably way smaller than anything else.

Yes but need more time to make pixeldata out of them, at least JPG.

Sounds strange that it is that massive ... are you on a notebook?
Because it seems quite heavily dependent on the harddisk read performance if BMP takes 3+ seconds even thought its plain readin of pure byte data.

Or did you just overdo it massively with the size ie nothing that has to do with reality in 2D through 3D.

Actually, the gif and jpeg times I did were flawed [oops =(] in the test after attempting a render with them. The bmp's and png's were accurate tests.

Dreamora, the image is a small 32x64 sprite, and I loaded it 5,000 times in a loop before checking the time used. However, i'm a bit confused with png. Although bitmap is larger, the png is also compressed with...zlib I believe. I even tried an incbin and png's are still quite a bit faster.

I'll screw around for a while, see where I end up. If I find anything significant i'll add it to the code archives.

One of my personal projects loads several thousand small images, taking around 10 seconds. While I plan to add several thousand more I hope to reduce the time it takes to load these as best I can.

5000 times: hopefully by storing them in a list, right?
Otherwise you are testing how far the GC can be trash flooded until it breaks ;-)

lol yes.

Ever since I took your advice on using lists, they have been extremely useful to me.

Fine fine ...

One thing that might have an impact here btw is that the libraries that are used are differently optimized. So it can easily "play out" well for a format one wouldn't have assumed to be on place 1 :) (thought I'm happy that it seems like png is the winner as I use it for everything :) )

The bmp and tga loaders are written in BlitzMax which I would expect to be slower than one written in C (like the png loader).

For example, the bmp loader is not what I'd call optimized :

				hsize=ReadInt(stream)
				pad=ReadInt(stream)
				hoffset=ReadInt(stream)
				size=ReadInt(stream)
				width=ReadInt(stream)
				height=ReadInt(stream)

I suggest you try loading the bmp into a bank first and then creating a ramstream to load the bmp from the bank... should be faster as that way you are kind of creating a "buffered" stream.

Or create yourself a buffered stream, which loads the file in large chunks, rather than picking bits out of the stream as is required.

Disk I/O is your enemy :-p

Any chance you can post the example code you use for testing?

I Fancy trying it out the equiv on FreeImage...

Mind you, since the image is so small... disk access times aren't very imortant :-p

I imagine there would be a slightly faster way of loading images. from working with XNA and it's content pipeline it transforms all images (png, jpeg, tga) to a ready-to-load directly into graphics memory format. Made for much faster load times as opposed to png conversion. Only draw back is all the image files ended up being very bloated in size.

It is the memoryBMP used by XNA. Anyway, I'm making an application that is loading about 1000 PNGs of 100x100 pix in less than 400 milisecs on my labtop. The speed increase was done by storing all the PNG files in a single big file, and load each PNG passig a single stream pointing to the big file where everything is stored in order. something like

Local Stream:OpenStream("MyBigfile.dat")
Image1 = loadimage(stream)
Image2 = LoadImage(Stream)
Image3 = LoadImage(Stream)
...

I agree that different loaders go about things in different ways and it depends on how well they buffer the disk i/o and also how they buffer the decompression of data. The BMP reader looks hideously slow if it's reading every Integer separately.

I personally found, in my own tests, that loading purely RAW data is faster than any other format. Although there is more data to load than, say, a PNG, for larger images it proved to be about 2-3 times faster to read as raw data. Ie just a dump of the contents of a pixmap. No header data or compression.

I would think possibly that some form of run-length-encoding might prove to be a bit faster than more complicated compression schemes, when it comes to decompressing it, because then you can do little loops which output a local variable several times. I would think that has to be faster than the BMP loader. IFF/ILBM is a fairly fast format which uses run-length.

If you are going to write your own format I think you should seriously consider a format which can store multiple images in a single file, and which can encode them as if they were animation frames, to take advantage of only having to store the changes between `frames`, plus being able to reference a larger window of similar data when you want to do compression. Going beyond single isolated file compression is how archivers like LZX improved over the older single-file LHA, for example. You might even try interleaving data from several similar images. This is how the MNG format can give you better compression than PNG.

If you want to go with the fastest loading possible, you might focus on spending as much brute force time as you like in compressing the file, but make sure that your technique stores the output in such a way that when it comes time to decompress it the decompression is VERY simple. You could implement your own exhaustive search for repeated sequences and replace them with `offset,length` pairs, which decompress pretty quickly.

Wow. I spent a good while making a response just to have my page error and my textbox clear :'(. Shortened version:

Ctrl+a and Ctrl+C combo for life.

@Ziggy
I haven't tested the speed of multiple file streams in a single package, although I tried the same principle by using tilesets.
From about 30 different tilesets (all png), containing a total of between 4-5k 48x48 tiles, I was able to load them all within 440ms, and process them into seperate image handles, and clear the tilesets within 18ms. After seeing that, I think it's safe to say that the main speed decrease is due to disk calls, not so much format.

That being said, I simply having a stream-based package to load large quantities of images is probobly the best path to go. Going with raw and other differently optimized formats might save some time, but the amount of tiles me and ziggy are using in our applications is pretty rare. So if we are able to load this in less than a second, I think I went the wrong way looking for a better format, rather than a better method.

I am still a bit interested in another method though. I may have a hard time explaining it as i'm pretty new to blitzmax and a lot of its capabilities. Is it possible to write a pre-processed stream, so that you don't have to read the data seperately into each of a types values?

ex:

Type MyImageType
	Field Image:TImage
	Field Name:String
	Field Description:String
	Field Rotation:Int
	Field ScaleX:Int
	Field ScaleY:Int
	'etc
End Type

Rather than having To Load it with:
Local Stream:TStream = OpenStream("file")
Imagetype.image = read. ..
imagetype.name = read. ..
imagetype.description = read ..

Is there a way to save the Type in a way that you can simply load it's values without using readline/readint, etc. And have values for it saved in preprocessed form?
Maybe by saving the memory found in the memory pointer to a stream, then to load it...loading it into memory and giving the ImageType a pointer to the memory?

There is a lot in between what I just said and what I don't know, so that might make no sense. If something like that is possible, it would be extremely easy, and MUCH faster to load images (like imaginary said, in raw data), models (for 3d), world data (in games), etc.

Slightly off topic, but on the same lines of loading things faster.

You may take a look to reflection. Anyway, in my experience, compressed formats do load faster than uncompressed ones in most computers, becouse disk access is slower than decompression algorithms in current computers. In addition, disk streams are buffered in BlitzMax, that does it even faster when packaging media in a single stream. (no need to create a buffer for each file read).

I would think that if you are loading lots of small files then the disk has extra time whereby it has to move to search for the file, whereas if it were one big file or several big files with relatively little or no fragmentation it is probably faster?

I would think possibly if you loaded all your type data into a single array or bank to act as a buffer for the disk access, and then once it is all loaded simply copy it over to your individual types, that may be faster, but it depends on whether the disk access already has buffering.

that fragmentation stuff is most likely one of the reasons why commercial games always use zips even if not encrypted or anything. just to make sure that the data is much more likely in the same place to be handled as a virtual file system and loaded selectively when needed.

Fragmentation and bad file placement actually makes a serious difference on loadup of media ... the more files there are, the worse it can get depending on the harddisk layout of the files.

Also the creation of a single data buffer for all the images has a deep performance improvement. The buffer has information of the 'next' image while your still loading the first one. When loading multiple files, a buffer is created and destroyed for each file. To the fragmentation issue, I'm not sure if a unfragmented disk will work better on a contiguous big file, or in contiguous several unfragmented files. Haven't tested, anyway, data storage format will make a diference. In my experience, dealing with the UAC for each file will be a lot slower than dealing with it a single time when using Vista