Reading and Writing Stream problems

This one's been bugging me for a few days now, and I can't seem to locate the error.

The program is essentially a simple relational database wherein all the actual data is stored in a Bank field in the Table type, with two other types (FieldGroup and Field) used to hold offset information and fieldnames, etc.

However, my code to save and load the data to the hard drive for further use isn't working properly - the data is garbled somewhere along the line and thus becomes unusable. Whether or not it works depends on the data being used, and if it does work, saving after adding further data to the database will bring up the same error after loading.

Sorry if it's not too clear.

The save function is:

'Data Saving
Function SaveData()
	
	Local FileName:String = Input("Enter Filename: ")
	
	Local file:TStream = WriteStream(FileName)
		
	'save totals
	WriteInt(file, TotalTables)
	WriteInt(file, TotalFieldGroups)
	WriteInt(file, TotalFields)
	
	'save tables, then field groups, then fields
	For Local tbl:DBTable = EachIn DBTableList
		WriteLine(file, tbl.Name)
		WriteInt(file, tbl.Id)
		WriteInt(file, tbl.Records)
		WriteInt(file, tbl.MaxRecords)
		WriteInt(file, tbl.MaxSize)
		WriteInt(file, tbl.Mode)
		WriteInt(file, tbl.RecordSize)
		
		For Local a = 0 To tbl.MaxRecords - 1
			WriteInt(file, tbl.Offsets[a])
		Next
		
		'Bank Sizes
		WriteInt(file, BankSize(tbl.DataBank))
		WriteInt(file, BankSize(tbl.RecordFlags))
		
		WriteBank(tbl.DataBank, file, 0, BankSize(tbl.DataBank))
		WriteBank(tbl.RecordFlags, file, 0, BankSize(tbl.RecordFlags))
		
	Next
		
	For Local grp:DBFieldGroup = EachIn DBFieldGroupList
		WriteLine(file, grp.Name)
		WriteInt(file, grp.Id)
		WriteInt(file, grp.Size)
		WriteInt(file, grp.Table)
		WriteInt(file, grp.Offset)
	Next
		
	For Local fld:DBField = EachIn DBFieldList
		WriteLine(file, fld.Name)
		WriteLine(file, fld.StrValue)
		WriteLine(file, fld.DefaultStr)
		WriteInt(file, fld.Id)
		WriteInt(file, fld.DataType)
		WriteInt(file, fld.Size)
		WriteInt(file, fld.BitSize)
		WriteInt(file, fld.Offset)
		WriteInt(file, fld.Group)
		WriteInt(file, fld.Table)
		WriteInt(file, fld.IntValue)
		WriteInt(file, fld.DefaultInt)
		WriteInt(file, fld.Record)
	Next
		
	Print "Saved!"
	
	CloseStream file
	
End Function

The load function is

Function LoadData()
	
	For Local tbl:DBTable = EachIn DBTableList
		ListRemove(DBTableList, tbl)
	Next
	
	For Local grp:DBFieldGroup = EachIn DBFieldGroupList
		ListRemove(DBFieldGroupList, grp)
	Next
	
	For Local fld:DBField = EachIn DBFieldList
		ListRemove(DBFieldList, fld)
	Next
	
	Print "Existing Data is wiped!"
	
	
	
	
	Local FileName:String = Input("Enter Filename: ")
	
	Local file:TStream = ReadStream(FileName)
		
	'load in totals
	TotalTables = Readint(file)
	TotalFieldGroups = Readint(file)
	TotalFields = Readint(file)
	
	'load tables, then field groups, then fields
	
	For Local count = 1 To TotalTables
		Local tbl:DBTable = New DBTable
		tbl.Name = ReadLine(file)
		tbl.Id = Readint(file)
		tbl.Records = Readint(file)
		tbl.MaxRecords = Readint(file)
		tbl.MaxSize = Readint(file)
		tbl.Mode = Readint(file)
		tbl.RecordSize = Readint(file)
		
		tbl.Offsets = New Int[tbl.MaxRecords]
		For Local a = 0 To tbl.MaxRecords - 1
			tbl.Offsets[a] = Readint(file)
		Next
		
		Local DataBankSize = Readint(file)
		Local RecordFlagSize = Readint(file)
		
		tbl.DataBank = CreateBank(DataBankSize)
		tbl.RecordFlags = CreateBank(RecordFlagSize)
		
		ReadBank(tbl.DataBank, file, 0, DataBankSize)
		ReadBank(tbl.RecordFlags, file, 0, RecordFlagSize)
		
		ListAddLast(DBTableList, tbl)
	Next
	
	For Local count = 1 To TotalFieldGroups
		Local grp:DBFieldGroup = New DBFieldGroup
		grp.Name = ReadLine(file)
		grp.Id = Readint(file)
		grp.Size = Readint(file)
		grp.Table = Readint(file)
		grp.Offset = Readint(file)
		
		ListAddLast(DBFieldGroupList, grp)
	Next
	
	For Local count = 1 To TotalFields
		Local fld:DBField = New DBField
		fld.Name = ReadLine(file)
		fld.StrValue = ReadLine(file)
		fld.DefaultStr = ReadLine(file)
		fld.Id = Readint(file)
		fld.DataType = Readint(file)
		fld.Size = Readint(file)
		fld.BitSize = Readint(file)
		fld.Offset = Readint(file)
		fld.Group = Readint(file)
		fld.Table = Readint(file)
		fld.IntValue = Readint(file)
		fld.DefaultInt = Readint(file)
		fld.Record = Readint(file)
		
		ListAddLast(DBFieldList, fld)
	Next
	
	Print "Loaded!"
	
	CloseStream file 
	
End Function

I'm wondering if I'm doing something wrong with the way I'm saving and loading the data. I've scoured the code for hours trying to find if I'm forgetting to save an important piece of data, or if I'm correctly creating the Table, FieldGroup and Field types and adding them to their list (although I'm convinced that's not the problem, as the data read in is garbled before I add the object to the list).

Note that I've mainly only encountered a problem with the "Field" data - never with the FieldGroup or Table data (indeed, the banks are being read and stored correctly). The field data ends up with offsets in the tens of thousands, bitsizes far too big and results in garbled data. On one occasion, however, I did get a error reading a databank (trying to access a memory location out of range), but never had enough time to find the exact cause - it's likely part of the field offset problem.

Any help is greatly, greatly appreciated.

I don't see anything wrong right off the bat. You might try adding a number of debuglog statements to make sure the values you are writing are really what you think they are.

Another possible issue is that you are mixing binary and text mode commands. Don't use WriteLine and ReadLine, use WriteInt to write the length of the string data and then WriteString to write the char data. The when reading use ReadInt and ReadString. I'm not sure but the use of Write/Read Line might be throwing you off somewhere.

Does the problem alway happen? What data causes it to happen? What data doesn't cause it to happen?

I'll certainly give the WriteInt/WriteString combo a whirl, I might get lucky but you never know.

The problem doesn't always happen, but I've yet to properly isolate the cases in which it does occur. I have stepped through the code using the debugger, and the values I'm writing are the ones that should be written. When read, they're screwed up - but only for the Field types - the preceeding Tables and FieldGroups are read fine.

One case in which it definitely happens is as follows:

There are two objects of type Table, two of type FieldGroup, and four of type Field (meaning each table has one field group, with two fields in it). When created in one sitting, this saves and loads perfectly well.

When I then add another Table, with one Field Group and two Fields, then save and reload that (including when starting the program from scratch) the problem occurs - when I select an option to view the contents, the fields themselves are incorrectly read due to the screwed field data.

In that case we need to see the rest of your code. The entire type definitions for Table, FieldGroup and Field. I suspect you have something messed up in your list handling.

Ok, the lists are set up as follows:

Global DBTableList:TList
Global DBFieldGroupList:TList
Global DBFieldList:TList

and then initialised upon the running of the code with

'Initiate System
Function InitialiseDBOne()

	DBTableList = New TList
	DBFieldGroupList = New TList
	DBFieldList = New TList
	
End Function

Each type includes methods, including one to create an object and add it to the relevant list

Table:

Type DBTable
	
	Field Name:String		'name of the table
	Field Id:Int			'Id code for the table
	Field Records:Int		'number of records
	Field MaxRecords:Int	'maximum number of records
	Field MaxSize:Int		'maximum size of area (for dynamic tables)
	Field Mode:Int			'dynamic or static mode
	Field RecordSize:Int	'size of a record (static only)
	Field DataBank:TBank	'bank used to store all data
	Field RecordFlags:TBank	'bank used to store flags for record existence
	Field Offsets:Int[]		'array used to store offsets of records (dynamic). May become bank
	
	'Function to create a new table
	Function CreateTable:DBTable(Name:String, Id:Int, MaxRecords:Int, Mode:Int, MaxSize:Int)
	
		Local this:DBTable = New DBTable
		
		this.Name = Name
		this.Id = Id
		this.Records = 0
		this.MaxRecords = MaxRecords
		this.Mode = Mode
		this.RecordSize = -1
		this.Offsets = New Int[MaxRecords]
		
		TotalTables = TotalTables + 1
		
		ListAddLast(DBTableList, this)
		Return this
				
	End Function
		
	
End Type

FieldGroup

Type DBFieldGroup

	Field Name:String		'name of the field group
	Field Id:Int			'Id code for the group
	Field Size:Int			'size of field group in bytes (depends on component fields)
	Field Table:Int			'ID of parent table
	Field Offset:Int		'offset of field group from start of record

	Function CreateFieldGroup:DBFieldGroup(Name:String, Id:Int, Table:Int)
	
		Local this:DBFieldGroup = New DBFieldGroup
		
		this.Name = Name
		this.Id = Id
		this.Size = -1
		this.Table = Table
		this.Offset = -1
		
		TotalFieldGroups = TotalFieldGroups + 1
		
		ListAddLast(DBFieldGroupList, this)
		Return this
		
	End Function
	
End Type

Field

Type DBField
	
	Field Name:String		'name of the field
	Field Id:Int			'ID code for the field
	Field DataType:Int		'type of data (String, int, BitPattern etc.)
	Field Size:Int			'size of data (e.g. characters, max value)
	Field BitSize:Int		'size of data in bits
	Field Offset:Int		'offset in bits of field from group
	Field Group:Int			'ID of parent group
	Field Table:Int			'ID of parent table
	
	'Values
	Field StrValue:String	'string value (if string)
	Field IntValue:Int		'int value (if int)
	Field DefaultStr:String	'default string value
	Field DefaultInt:Int	'default int value
	Field Record:Int		'current record stored
	
	Function CreateField:DBField(Name:String, Id:Int, DataType:Int, Size:Int, Group:Int, Table:Int, Str:String = "", Integer:Int = 0)
	
		Local this:DBField = New DBField
		
		this.Name = Name
		this.Id = Id
		this.DataType = DataType
		this.Size = Size
		this.Offset = -1
		this.Group = Group
		this.Table = Table
		this.DefaultInt = Integer
		this.DefaultStr = Str
		
		If this.DataType = 0 Then
			this.BitSize = this.Size * 8
		Else If this.DataType = 2 Then
			this.BitSize = this.Size
		Else
			this.BitSize = GetBitSize(this.Size)
		End If
		
		TotalFields = TotalFields + 1
		
		ListAddLast(DBFieldList, this)
		Return this
	
	End Function
	
	Method ReadValue(rec:Int)
		Record = rec
		If DataType = 0 Then ThisReadStr() Else ThisReadInteger()
	End Method
	
	Method WriteValue(rec:Int)
		Record = rec
		If DataType = 0 Then ThisWriteStr() Else ThisWriteInteger()
	End Method
	
	Method ThisReadStr()
	
		'get record offset
		Local RecordOff
		
		'Get table for databank
		For Local tbl:DBTable = EachIn DBTableList
			If tbl.id = Table Then
				RecordOff = tbl.RecordSize * record
				StrValue = ReadStr(tbl.DataBank, RecordOff + (Offset / 8), Size)
			End If
		Next
	
	End Method
	
	Method ThisWriteStr()
	
		'get record offset
		Local RecordOff		
	
		'get table for databank
		For Local tbl:DBTable = EachIn DBTableList
			If tbl.id = Table Then
				RecordOff = tbl.RecordSize * record
				WriteStr(tbl.DataBank, RecordOff + (Offset / 8), StrValue)
			End If
		Next
	
	End Method
	
	Method ThisReadInteger()
	
		'get record offset
		Local RecordOff
	
		'get table for databank
		For Local tbl:DBTable = EachIn DBTableList
			If tbl.id = Table Then
				RecordOff = tbl.RecordSize * record
				IntValue = ReadInteger(tbl.DataBank, (RecordOff * 8) + Offset, BitSize)
			End If
		Next
	
	End Method
	
	Method ThisWriteInteger()
	
		'get record offset
		Local RecordOff
	
		'get table for databank
		For Local tbl:DBTable = EachIn DBTableList
			If tbl.id = Table Then
				RecordOff = tbl.RecordSize * record
				WriteInteger(tbl.DataBank, (RecordOff * 8) + Offset, BitSize, IntValue)
			End If
		Next
	
	End Method
	
	Method StoreString(str:String)
		
		If str.length <= Size Then
			StrValue = str
		Else
			StrValue = str[0..(size-1)]
		End If
		
	End Method
	
	Method StoreInt(integer:Int)
		
		If integer > Size Then
			integer = 0
		End If
		
		IntValue = integer
		
	End Method
	
End Type

Throughout the code, the objects are only accessed via those methods, and directly during the Save and Load functions.

I don't suppose you can send me a zip file with the program and source I would like to see it running in front of me. Check my profile for an email addr.

I've sent the two source files, haven't got access to the built files right now as I'm posting from work (no net access at home at the moment, sadly).

Thanks

Okay I got the files, I'm having a hard time making it screw up.

Test 1:
Started a new database
Added a new table "test"
added a Group "group1"
added field "string" type String Max chars 10 Default 0123456789
added field "int" type Int Max value 255 Default 0
Saved db1
reloaded no problem

Test 2:
Load db1
Add 3 records to table "test"
abcdefghij,1
bcdefghijk,2
cdefghijkl,3
view records, okay
save db2
quit
run
load db2
view records, okay

Test 3:
Added a new table "second"
added a Group "group2"
added field "string" type String Max chars 10 Default 0123456789
added field "int" type Int Max value 255 Default 0
Saved db3
quit
run
load db3
view records, okay

Test 4:
Add 1 records to table "second"
abcdefghij,1
view records, okay
save db4
quit
run
load db4
view records, okay

Test 5:
Added a new table "third"
added a Group "group3"
added field "string" type String Max chars 10 Default 0123456789
added field "int" type Int Max value 255 Default 0
Saved db4
quit
run
load db4
view records, okay

EDIT: just notice that if you enter a string in a string field that has a max chars of 10 and the string is longer than 10 it truncates the string to 9 characters. Still saves and load okay though.

I haven't been able to lock it down yet but there is something wrong with tables that have more than one int field.


EDIT:

okay the problem shows when you have two or more tables where one or more of them have multiple int fields. You don't even have to save/load the database.

EDIT:

well you've got me it appears to be totally random when it fails, except that I've been able to make it fail with tables that use multiple int fields.

Another possible problem, create a single table with a single group with a string and an int field. enter a single record then try to view the record. the program doesn't list the value of the int field only the string field and it seems to hang until you hit enter.

Here is the out of the session where i did this.

DatabaseOne Main Menu
1. Load Database
2. Save Database
3. Create Table
4. Edit Table
5. Data Entry
6. View Records
7. View Tables
8. Exit
9. DEBUG TABLE BANKS
Select option: 3

Table Name: table1
Maximum Records: 10
Mode (Static/Dynamic): static

Field Group Name: group1
Number of Fields: 2

Field Name: string1
Data Type (String/Int/Bit): string
Size (Characters/Max Value/Bits): 10
Default: empty

Field Name: int1
Data Type (String/Int/Bit): int
Size (Characters/Max Value/Bits): 255
Default: 0

Another Field Group (y/n)? n
Now Packing...
Memory Used: 6730

DatabaseOne Main Menu
1. Load Database
2. Save Database
3. Create Table
4. Edit Table
5. Data Entry
6. View Records
7. View Tables
8. Exit
9. DEBUG TABLE BANKS
Select option: 7

0. table1 (STATIC, 0/10)
View any table, -1 to exit: 0

Table: table1

Group: group1
0, string1 (STRING, 10 Chars)
1, int1 (INT, Max 255)

Link Field (-1 to exit): -1
Memory Used: 6730

DatabaseOne Main Menu
1. Load Database
2. Save Database
3. Create Table
4. Edit Table
5. Data Entry
6. View Records
7. View Tables
8. Exit
9. DEBUG TABLE BANKS
Select option: 5
0, table1 (0/10)
Enter data for which table? 0

Group: group1
string1: Scott
int1: 1
Memory Used: 6752

DatabaseOne Main Menu
1. Load Database
2. Save Database
3. Create Table
4. Edit Table
5. Data Entry
6. View Records
7. View Tables
8. Exit
9. DEBUG TABLE BANKS
Select option: 6
0, table1 (1/10)
View data for which table? 0
0, Scott 

Group: group1
string1: Scott 
Memory Used: 6762

DatabaseOne Main Menu
1. Load Database
2. Save Database
3. Create Table
4. Edit Table
5. Data Entry
6. View Records
7. View Tables
8. Exit
9. DEBUG TABLE BANKS
Select option: 7

0. table1 (STATIC, 1/10)
View any table, -1 to exit: 0

Table: table1

Group: group1
0, string1 (STRING, 10 Chars)
1, int1 (INT, Max 255)

Link Field (-1 to exit): 
Process terminated

Thanks for your help so far Scott, I'll get a good look at those issues tonight.

I think I may have solved any direct saving/loading issue. The way I was writing strings to the databank, I neglected to fill the remainder of the allocated space (e.g. writing a four-character string to a ten-character space, I'd need to fill the last six with spaces) which then caused problems.

This space would have been a list of null characters (byte value #00). This is fine, until you read the value from the bank - instead of getting, say, "TEST", you'd get "TEST" followed by 16 null values, which completely screws up the saving and subsequent loading. This explains why it only occurs in the Field type - as it is only fields that read and write data to Banks. By filling the remainder with space characters, and using Trim() after reading them, the problem seems to have disappeared.

I'm getting a few other problems which may be related to what you've pointed out, so I'll give those a whirl. It's also good to note that sometimes the Output screen of BlitzMax fails to display properly - it hangs as you have indicated. It should work fine in a normal command window when running the EXE directly. If not, it's definitely a bug (probably a bug anyway, but I'm baffled by it).