Two WIP modules...

Hallo again !

Since it's the weekend again, I've been playing around with a couple of new modules, neither finished as yet, but perhaps of interest anyway :-)

The first, BaH.Persistence, is an Object serializer, which can handle most things you can throw at it. The only things it doesn't like very much are multi-dimension arrays and Byte Ptr data. The arrays issue I think is due to the Reflection module.
It uses the Libxml module to build and load the persistence data.

Perhaps the most interesting bit about the module is that it was designed from the beginning to properly handle circular references. This means that you can serialize a TList and be sure that when you de-serialize it, you will end up with a TList in exactly the same state as it was in at the time.

...

The second, BaH.XLWriter, allows you to create .xlsx files using a very simple interface. xlsx is the file format for Excel 2007, and is also readable by apps such as Numbers (Apple) and Open Office. It is actually a special zip file containing lots of different types of xml files.

It's currently in the very early stages of development, but it is already able to create basic spreadsheets of text, numbers and formulas. At the moment, it is using BaH.Libxml, BaH.RegEx and wxMax. wxMax is required for access to the Zip output stream, which means it can create all the files for the zip as it creates the zip itself - ie. no external files created.

...

The Libxml that both modules require is the one currently in SVN at maxmods. I should hopefully be releasing the update fairly soon.

You can find the mods at the same location.


Docs are a bit on the sparse side for both at the moment, but there is a small example for each, and suggestions/comments are very welcome.

Anyway, thought you might be interested to see what's on-going :-)

Probably a stupid question, but what exactly is an object serializer (what does it do)?


This seems interesting... :-)

Imagine you have an Object instance of a Type, say :

Type TRect
	Field x:Int
	Field y:Int
	Field w:Int
	Field h:Int
End Type

...and you want to save it somewhere, but you don't want to have to write some save code for all your types...

Print TPersist.Serialize(New TRect)

...

<?xml version="1.0"?>
<bmo>
  <TRect ref="01VKY0">
    <field name="x" type="int">0</field>
    <field name="y" type="int">0</field>
    <field name="w" type="int">0</field>
    <field name="h" type="int">0</field>
  </TRect>
</bmo>

Yes, you could store that away in 16 bytes using a custom format, but then you'd have to write the custom format, and test it....

That may be extremely useful... :-)

Two excellent additions. Well done Brucey!

For those still not catching on to how useful TPersist is, this real world example might help. I just converted this xml loading code from:

Local node:TxmlNode
'	Load and Parse the file
LoadXML( "Settings/" +_s_file)	
' Window Info 
WindowRect	= GetRect(RootNode,"Window")

'Textfields	
node = FindChild(,"Textfields")				
MaxCentsTextfieldDigits = GetInt(node,"max_cents_textfield_digits")
MaxDollarsTextfieldDigits = GetInt(node,"max_dollars_textfield_digits")
MaxAccountTextfieldEntries = GetInt(node,"max_account_cache_entries")

'AppointmentWindow Info 
node = FindChild(Null,"AppointmentWindow")
NumCols = GetInt(node,"num_cols")
f_MouseScrollSpeed	= GetFloat(node,"mouse_scroll_speed")
AppointmentXScroll	= GetInt(node,"appointment_x_scroll")
AppointmentYScroll	= GetInt(node,"appointment_y_scroll")
b_Timeline			= GetInt(node,"b_timeline")
b_XSliderPane		= GetInt(node,"b_xslider_pane")
b_YSliderPane		= GetInt(node,"b_yslider_pane")

'Appointment
node = FindChild(Null,"Appointment")
b_AlwaysDisplayTimeInAppointments = GetInt(node,"show_time")
b_UseTreatmentCollections= GetInt(node,"use_collections")
LoyaltyPointRatio= Max(1,GetInt(node,"loyalty_point_ratio"))

'	GeneralDialog
node =  FindChild(Null,"GeneralDialog")
b_AddNumbersToMonthNames= GetInt(node,"b_add_numbers_to_month_names")
NumYearsInYearCombo= GetInt(node,"num_years_in_year_combo")
MinYear= GetInt(node,"min_year")
MaxEntriesInStaffList= GetInt(node,"max_entries_in_staff_list")

'	Treatment Dialog
node= FindChild(,"TreatmentDialog")
b_DisplayActiveTreatmentsOnly= GetInt(node,"b_display_active_treatments_only")
UndefinedTreatmentDuration= GetInt(node,"undefined_treatment_duration")
UndefinedTreatmentTurnaround= GetInt(node,"undefined_treatment_turnaround")

'	Appointment Menu
node = FindChild(,"AppointmentMenu")
s_CustomQuickNote= GetString(node,"s_custom_quicknote")
CustomQuickNoteDuration= GetInt(node,"custom_quicknote_duration")
Cleanup()
Return Self

to

Return TSettings(TPersist.Deserialize(LoadString( "Settings/" +_s_file)))

And of course, the save function is now a one-liner also.

This:

Local node:TxmlNode

'	Load and Parse the file
LoadXML( "Settings/" +_s_file)	

' Window
SetRect(,WindowRect,"Window")

'Textfields					
node 								= FindChild(,"Textfields")				
SetInt(node,MaxCentsTextfieldDigits,"max_cents_textfield_digits")
SetInt(node,MaxDollarsTextfieldDigits,"max_dollars_textfield_digits")
SetInt(node,MaxAccountTextfieldEntries,"max_account_cache_entries")
				
'AppointmentWindow Info 
node 								= FindChild(,"AppointmentWindow")
SetInt(node,NumCols,"num_cols")
SetFloat(node,f_MouseScrollSpeed,"mouse_scroll_speed")
SetInt(node,AppointmentXScroll,"appointment_x_scroll")
SetInt(node,AppointmentYScroll,"appointment_y_scroll")
SetInt(node,b_Timeline,"b_timeline")
SetInt(node,b_XSliderPane,"b_xslider_pane")
SetInt(node,b_YSliderPane,"b_yslider_pane")

'	Appointment
node 								= FindChild(,"Appointment")
SetInt(node,b_AlwaysDisplayTimeInAppointments,"show_time")
SetInt(node,b_UseTreatmentCollections,"use_collections")
SetInt(node,LoyaltyPointRatio,"loyalty_point_ratio")

'	GeneralDialog
node 								=  FindChild(,"GeneralDialog")
SetInt(node,b_AddNumbersToMonthNames,"b_add_numbers_to_month_names")
SetInt(node,NumYearsInYearCombo,"num_years_in_year_combo")
SetInt(node,MinYear,"min_year")
SetInt(node,MaxEntriesInStaffList,"max_entries_in_staff_list")

'	TreatmentlDialog
node 								=  FindChild(,"TreatmentDialog")
SetInt(node,b_DisplayActiveTreatmentsOnly,"b_display_active_treatments_only")
SetInt(node,UndefinedTreatmentDuration,"undefined_treatment_duration")
SetInt(node,UndefinedTreatmentTurnaround,"undefined_treatment_turnaround")

'	AppointmentMenu
node 								=  FindChild(,"AppointmentMenu")
SetString(node,s_CustomQuickNote ,"s_custom_quicknote")
SetString(node,CustomQuickNoteDuration ,"custom_quicknote_duration")

'Overwrite the original file.
SaveXML("Settings/" +_s_file)

'Close and free the xml file
Cleanup()

Becomes

New TPersist.SerializeToFile(Self, "Settings/" +_s_file)

You've got to love that! :-)

Didn't reflection already have serialization?

It uses reflection to provide the serialization.

Just looked at the docs, I thought there was a standard serializer already, my bad.

XLWriter is now able to add font-formatting to cells, which is kinda cool - that's things like different fonts, sizes, bold, italic etc.
...which adds to text, numbers and formula in cells.

Now that the core for styles is implemented it should be fairly trivial to get borders, fills, and colours on board too. And then cell formatting (like text-align, or numbers with 2 decimal places, etc)

:o)

This sounds all very exciting :-)

Borders and colours working now. (See provided examples for usage).

Fill and formatting styles to come.

excuse my ignorance but what url am i needing for the persistence and xlwriter modules for svn?

will the same address give me all the other bah.* modules?

cheers
glenn

will bah.xlwriter also let me READ xls files?

You should find them at

http://maxmods.googlecode.com/svn/trunk



No. The idea is to use Excel/Numbers for that.

cheers

Yep, just for creating xlsx files.

Unless you were wanting to create your own spreadsheet app, there wouldn't be too much need for xls(x) reading.

The XLWriter module makes it easier to create formatted spreadsheet-compatible documents.
Formats like CSV don't give you formatting - which is something you might wish to provide for your end-users.

Basic formatting support has now been added to the latest version (see example_01). Also improved the documentation.

I often need to take spreadsheet data and save as csv or txt so i can read the data into a program, manipulate it, then write something back out as csv or txt, and finally resave in xls format with formatting. I was just hoping to save a step of converting from xls to csv/txt and back again.

BaH.Persistence sounds great.

I'd like to be able to combine it with xsl transforms, so that I can make a "write" xml stylesheet, that will format the xml in a certain way and also a "read" stylesheet that will transform my custom xml into the format that BaH.Persistence expects.

I realize I can do this simply by reading and transforming the resulting xml files, but I'd like to see it integrated into BaH.Persistence, so that I could pass the loading and saving stylesheets as a parameter (or something like that) to (de)serialize.

Could this be made possible?

@Glenn
Bit of a big job, and not something I'd really fancy attempting - building a reader for some proprietary binary file format, which would have limited mileage anyway.

@Peter
It certainly sounds possible, and would be quite useful.
Not sure if it wants direct integration into that module, as you then add libxslt requirements onto apps that might not necessarily use that functionality.

that would be why i never attempted it too. Not to mention the fact that it would be too hard for me :)

@Brucey

Ok sure, but if there could be an alternative to SerializeToFile() and SerializeFromFile()?

Something like:
SerializeFrom(doc:TxmlDoc) and Serialize:TxmlDoc(obj:object)

then the post and pre transforms could easily be done with an TxsltStylesheet.applyStylesheet(doc) command.

Okay, how's about :

Method SerializeToDoc:TxmlDoc(obj:Object)

...for serializing.

You'll have to free the returned object yourself.

For DeSerializing, I've changed the DeSerialize() Function to take an Object instead of a String parameter. If you pass in a TxmlDoc (rather than a String), it will use that doc as the tree to process. Internally, it uses :

Method DeSerializeFromDoc:Object(xmlDoc:TxmlDoc)

But you don't need to call that specifically if you don't want to.
Again, the code won't free up the TxmlDoc you pass in, so you can do what you like with it otherwise, before you Free() it.

Absolutely great!

[EDIT] If you want I can make an example that you can include, when it's ready.

@Brucey: Feature request

It would be nice to have a metadata switch to
include only the fields with {serialize}.
How about that?

The problem I have with an inclusion switch is for cases such as :

You have a Type, which includes a TList that you wish to also serialize. You tag your TList field with the metadata, which enables serialization for that field. However, since the TList has no metadata for its own fields, none of the TList content is serialized.

I much prefer catch-all.

What about a tag for *not* serializing?
Unless you have a good reason for otherwise? :-)

> What about a tag for *not* serializing?

Yes, this will solve my problem!
Saving Meshes with Persistence.mod seems to be no
good idea. My TList file is bigger then 60 Mb with just one object.
(no bitmaps, a simple cube)

> Unless you have a good reason for otherwise? :-)

Not yet :-)

Compressing in memory before you save will reduce that size drasticly.
You can get an idea of how much by zipping the xml.

@Brucey: Could som space be saved by using a different format for arrays?

Taking a hint from collada and svg:

<field name="numbers" type="array:Int">
<val>1</val>
<val>2</val>
<val>3</val>
<val>4</val>
<val>5</val>
<val>6</val>
</field>

could be 

<field name="numbers" type="array:Int">1 2 3 4 5 6</field>

Fine for numbers, but not so great for strings - unless you encode them.

I don't see a problem doing it for numbers though.

Hmm.. looks like I've found a reflection bug with arrays and Long/Doubles :

        <field name="numbersl" type="array:Long">
          <val>1</val>
          <val>8589934592</val>
          <val>2</val>
          <val>12884901888</val>
          <val>3</val>
          <val>17179869184</val>
        </field>

...
	Field numbersl:Long[]
	this.numbersl = [ 1:Long, 2:Long, 3:Long, 4:Long, 5:Long, 6:Long ]

Ho hum...

Bug posted HERE with fix ;-)

Hokay... latest version changes for BaH.Persistence :

* Added support for {nopersist} metadata tag for fields.

Example :

Type TRect
	Field x:Int
	Field y:Int
	Field w:Int
	Field h:Int
	Field ignoreMe:String = "Hello" {nopersist}
End Type

...

        <TRect ref="0A01TK">
          <field name="x" type="int">0</field>
          <field name="y" type="int">0</field>
          <field name="w" type="int">0</field>
          <field name="h" type="int">0</field>
        </TRect>

* Added Peter's "compression" technique for arrays of numeric values. String and Object arrays remain the same.
Note : Also introduced a TPersist.BMO_VERSION constant which enables backwards read compatibility with previous serialization formats. If you are building your own XML, you should probably use the constant when setting the "ver" attribute of the root "bmo" node.
eg.

<bmo ver="1">

* Enabled global access to the "format" field, which now defaults to False. Setting this to True will enable "pretty" XML formatting of the serialized data. Run the example to see both types.

Any other ideas or queries?... :-)

Absolutely excellent. Works like a charm.

One more question:
Why is there no method "deserializefromfile"?
"deserializefromdoc" needs a TXMLdoc??

At the moment I helped myself this way:

Function SaveObject(obj:Object, file:String) 
	' Test for existing file TODO
	CreateFile (file) 
	Local f:TStream = OpenFile (file) 
	Local tp:TPersist = New TPersist
	tp.SerializeToString(obj) 
	WriteString (f, tp.SerializeToString(obj)) 
	CloseFile (f) 
End Function

Function LoadObject:Object (file:String) 
	Local f:TStream = OpenFile (file) 
	Local st:String = ReadString (f, StreamSize(f)) 
	Local obj:Object = TPersist.DeSerialize(st) 
	CloseFile f
	Return obj
End Function

But I think this is not the way it' s meant to be used.

I suppose it never occurred to me :-)

So... I've added... DeSerializeFromFile:Object(filename:String) method.

While I was tinkering, I've also added support for Streams.

There's now a method SerializeToStream(obj:Object, stream:TStream) which will write the serialized object to the passed in Stream.
It is up to the user to Close() that stream when they are finished with it.

The DeSerialize() function now also accepts a TStream object, which it will attempt to read serialized data from.
I've also added a DeSerializeFromStream:Object(stream:TStream) method, if you want it.

The example now demonstrates using the stream too.
It now effectively shows the following :
Object -> String -> Object -> File (via Stream) -> Object (via Stream) -> String (formatted)

:o)

does the serialization module use XML to save the objects?

Yes.