Large Files

Can BlitzMax handle reading/writing large files? i am talking about files whose filesize goes beyond the Integer limit? It looks to me like all of BlitzMax's file handling uses integers, which cause it to fail when dealing with really large files? I am trying to copy a 5GB file using BlitzMax and it is failing and acting quite weird.

Can you somehow split the process into smaller chunks?

I am just trying to copy one large file from one place to another. If all of the stream seeking commands operate with Integers, there will not be a way to move to a point in the file past the Integer barrier.

If I were to go through all of the bmx files that do the streams and file handling and switch the variables to long, would it work? Or is there something somewhere I do not have access to that affects this process?

I wrote this whole backup application. But never tested any files larger than 2GB while developing it. Now that I am trying to actually use it, I have run in to this roadblock. Can any one give me an idea on how I could copy a file larger than 2GB? Without shelling out to use the copy command :)

Some way to use streams, or maybe call some C code to copy large files for me?

Even simple things like the FileSize command just returns an Integer that has rolled over to a negative number.

Edit: It seems the CopyFileEx WinAPI function can do it, and it has a callback function so I would be able to get progress from it as it copies. Can anybody point me in the right direction to Implement this in BlitzMax?

I also found this which can read file sizes larger than 2GB:
http://www.blitzmax.com/codearcs/codearcs.php?code=1688

So, part of the problem is solved.

Thanks for any help that can be offered. This is driving me nuts as I wrote this backup app and tried to implement it as my backup solution in our office today and as soon as it hit a 3GB zip file, it failed miserably.

Feel like I'm talking to myself. :)

I figured it out, hopefully this will help somebody else at some point.

Here is the code that does it:

Extern "win32"
Function CopyFileExW:Int(SrcFile:Byte Ptr, DestFile:Byte Ptr, CallBack:Byte Ptr, Data:Int Ptr, Cancel:Int Ptr, Flags:Int)
End Extern

Local src:String = "srcfile.ext"
Local dst:String = "destfile.ext"
CopyFileExW(src.ToWString(), dst.ToWString(), Callback, Null, Null, 0)

Function Callback(lFileSize:Long, lBytesTransfered:Long, lStreamSize:Long, lStreamBytesTransfered:Long, iStreamNumber:Int, iCallBackReason:Int, iSrcHandle:Int, iDestHandle:Int, data:Int)

debuglog lBytesTransfered

End Function

The filesize limit also depends on what file system you're using. NTFS, for example, doesn't have a filesize limit (supposedly).

Russell

Well, all is not exactly well. That callback works fine on my notebook running vista. But that callback fails miserable on my desktop running XP.

For now, I have disabled the callback.

Does anybody know how to make that callback work? Or can at least give me some ideas on how I need to declare it and call it?

Thanks for any help.

Out of interest what code were you using in native Bmax?

I'm sure you could probably manage it with Pub.Stdc and fread/fwrite

That code I posted is exactly what I was using to test it.

Here is a slightly modified version, as I have been playing with it trying to get it to work.

SuperStrict

Extern "win32"
  Function CopyFileExW:Int(SrcFile$W, DestFile$W, CallBack:Byte Ptr, Data:Int Ptr, Cancel:Int Ptr, Flags:Int)
End Extern

Local src:String = "1.txt"
Local dst:String = "2.txt"
CopyFileExW(src, dst, CallBack, Null, Null, 0)
DebugLog "hello"
End

Function Callback:Int(lFileSize:Long, lBytesTransfered:Long, lStreamSize:Long, lStreamBytesTransfered:Long, iStreamNumber:Int, iCallBackReason:Int, iSrcHandle:Int, iDestHandle:Int, data:Int)
  DebugLog lBytesTransfered
  Return 0
End Function

I get slightly different results now. The copy happens, the callback gets called, but then the program freezes as though the CopyFileEx never returns.

If I do not give a Callback, like this:

CopyFileExW(src, dst, Null, Null, Null, 0)

Then the file is copied and the program ends like it should.

Sorry, I thought your original post suggested you tried Bmax (copyfile or streams or something) and got a problem and *then* tried CopyFileExW.

Oh I did. BlitzMax's CopyFile cannot handle any filesize above the 2GB barrier. It uses streams and Integers. And since Integers cannot be larger than 2GB, it fails.

Well, I shouldn't say "it fails", but once it does the copy, the CopyFile command never returns. So, the application hangs up.

I can see that copyfile uses copystream but not sure what you mean about it using integers.
Doesn't it just copy bytes?
I might be missing something though.

I *think* he means that you can't specify an offset to read from greater than an int is capable of storing.

... but I thought the intent was to copy the file (copyfile or copystream) and I can't see where an integer comes into it.
Doesn't matter I suppose just interested.

I am looking into the BlitzMax side again. The whole problem on the BlitzMax side could have come from the FileSize function not working, which caused something elseof mine to fail, and not the CopyFile.

But, I just did a CopyFile on a 3.22GB file and it copied, but the CopyFile command never returned, so the Blitz app hung.

Just did a little more testing, and from what I can see, the CopyFile command (and more specifically, the CopyStream Function) just continues right on reading past EOF when copying a file larger than the 2GB Integer barrier.

Function CopyStream( fromStream:TStream,toStream:TStream,bufSize=4096 )
	Assert bufSize>0
	Local buf:Byte[bufSize]
	While Not fromStream.Eof()
		toStream.WriteBytes buf,fromStream.Read( buf,bufSize )
	Wend
End Function

It never seems to find fromStream.Eof. And the bufSize variable just starts getting larger and larger.

I am going to post this into the BlitzMax bugs section.