Request: CTYPE and CARRAY

To be brief here is what I want to do:

CTYPE vert
  FIELD x,yz
EndType

local verts:vert[] 
' verts now designated a compact C-type array
verts = verts[100] 

verts[0].x = ??
...

...and...

CTYPE vert
  FIELD x,yz
EndType

local verts:vert[] 

b:TBank = CreateBank(100*sizeof(vert))

verts = vert ptr(b.BankBuf())

verts[0].x = ??
...

Why? Because this is a much more efficient way of manipulating and storing data for OpenGL. I can't comment on DirectX because I don't/won't use it and prefer OpenGL because it's xplatform (ala bmax) and much more, well, "direct" than DirectX.

If there already is an efficient way to do what I want then please tell me. Without it or if the solution is to resort to part-C then I may as well dump bmax altogether.

1) this isn't a problem, at least I don't see any beside the missing new in front of vert[100] which could perhaps create problems.

2) why do you create an array of vert just to create a bank out of it instead of using the array itself? Does not make much sense beside create overhead.

Even after your illustration, I have no idea what you're aiming for...

Eg, a chunk of vert info that can be passed enmass to OpenGL... but still editable without jumping through hoops (pointer casting and poking for instance).

Maybe you missed the "CTYPE" above, which isn't a spelling mistake? A normal TYPE, aka an OBJECT, has header info (at least) and an array-of-TYPES is actually an array of references/addresses of objects (other than simple types?). The point is that the actual data is not contiguous and would need packing before being passed to opengl. There are other situations, apart from opengl stuff, where c-type arrays would be useful and/or needed for performance.

I would be surprised if Mark doesn't already recognise this requirement since any 3D engine created without it is going to be either inflexible or slow or both! His silence on this worries me and with the lack of action in these forums I'm beginning to wonder if bmax has not proved very popular. I hope to be wrong but just in case I'm spending this week learning cocoa.

they work the same as in C++

Just seen Noels post. A concrete example: In my win-purebasic app I currently store and manipulate my vertex data using an c-type array of GL_T2F_C4F_N3F_V3F format, which happens to be the exact format being used for VBO's (Vertex Buffer Objects). Because a c-type array is used, reading data from a file, doing any manipulation and passing it to opengl is about as efficient as it ever will be. There is no overhead in the data structure, it's already in the VBO format required, I don't do any pointer casting (maybe one at the beginning of a loop) and I don't do any peeking/poking on a field-by-field basis. In other word I create, edit and use the data in-situ where-as in bmax I would be forced into using objects along with their inherent overhead, I would have to pack/unpack data and take performance hits on pointer referencing/dereferenceing, peeking/poking etc, etc. Way way WAY too much hard graft for such straight forward and common task. An analogy: imagine no strings and instead every letter being an object... how long before you start screaming?

No they don't work the same as C++. They are derived from structures built in C++, however as dmoc said, they always have a header, which we have no control over. C++ doesn't explicitly introduce any sort of header to user defined structures.

Types in bmax are not well suited when calling external functions that only expect one element too. After some troubleshooting and searching the forums, I found you can use the varptr 'operator' with the first field of your type definition to get the actual memory address of the instance in memory (ugly). Defining arrays within a Type are not part of the struct, but also a reference type, so you must declare padding bytes to achieve the same thing.. Gets a little ugly when you have larger arrays.

If you are dealing with arrays of Types, you are out of luck. In order to better interface to external APIs, it is a requirement that bmax support a "C" style structure type, or it will become a bottle neck.

Edit: and to clarify.

Types in bmax are always reference types, and not 'value' types.

CType TSomeStruct
  Field x:Int, y:Int
End CType

Local myarray:TSomeStruct[100]

Assuming a base address of 0x00000000, in memory this should look like the following, to match a C++ or C style structure:

1st element
0x00000000 x
0x00000004 y

2nd element
0x00000008 x
0x0000000C y

3rd element
0x00000010 x
0x00000014 y

Notice, it is a continuous block of memory.

However, with bmax's object type, this might look like:

1st element
0x00000000 header data
0x00000010 x
0x00000014 y

2nd element
0x00000050 header data
0x00000060 x
0x00000064 y

3rd element
0x00001240 header data
0x00001250 x
0x00001254 y

etc...

It is references to objects elsewhere in memory, so before passing to an external function, as dmoc said, the data must be packed as my first example of memory layout, using peeks, pokes, mem-copies, etc.

maybe, I wouldn't know since I don't program in C++. No disrespect intended but we are talking bmax here. However I do remember a little C++ and I don't think using C++ prevents the programmer from storing data in the form most appropriate for the task and in many cases in 3d gfx this will still be packed c-type arrays. Imangine a detailed model with 1000's of verts - would you want to store each vert as an object? Don't get me wrong, I'm not against the use of objects and think bmax's implementation is about right for what still is BASIC (at it's heart). But if this is at the expense of allowing direct efficient data access/ manipulation then bmax will be a good stepping stone to a more "serious" language... but not much else.

No I wouldn't store each vertex as an object, I would simply create a [x,3] array as I would in C

But you can't. See StuC's post above.

That is a very simple example, and yes your example would work (not the normal solution in C), but in most cases, Vertices and other elements exist as structures, not n dimensional arrays.

struct vert3 
{
  float x, y, z;
};

vert3 vertar[100];

vertar[0].x = 1;
vertar[0].y = 1;
vertar[0].z = 1;

is more readable and maintainable than

float vertar[100][3];

vertar[0][0] = 1;   // x
vertar[0][1] = 1;   // y
vertar[0][2] = 1;   // z

As soon as you change from elements of the same size, you a out of luck.

struct colourargb
{
   BYTE a, b, g, r;
};

struct vert3 
{
  float       x, y, z;
  float       normal;
  colourargb  colour;
  vert3       *next
};

vert3 vertar[100];

vertar[0].x        = 1;
vertar[0].y        = 1;
vertar[0].z        = 1;
vertar[0].colour.a = 0;
...  etc

Hi,

There are no plans for 'C' like structs or arrays, as they would add a level of complexity that I don't feel comfortable with.

You'd end up with 2 kinds of types/arrays, and the rules for converting stuff, indexing stuff, finding the address of stuff (which suddenly becomes a lot more important) etc would get a lot more complex.

If it makes you feel any better, I've been faced with similar issues myself lately. My solution has been to go the 'mess with pointers' route - but to do it in a nice, encapsulated way.

Yes, C like structs/arrays would have made this easier, but this low level stuff represents such a small fragment of the overall code that I don't feel it justifies major language modifications.

If you are working on a project that does require a large amount of low level code, then BlitzMax is probably not the right language to do it in.

As for interfacing with 3rd party libs etc, this is never going to be 'perfect' - in fact, only C/C++ will ever really be perfect for this, due to them being so low level you can pretty much emulate any kind of 'type'.

This wasn't an easy decision to make - ie: how much 'language simplicity' to sacrifice in the name of 'language interoperability'? - and the answer is incredibly subjective as it really revolves around what you want to do with the language. So I don't think there are really any easy answers here!

Mark,

Firstly, congratulations on a thriving community and I really do like the advances you have made with Blitz Max over previous version.

Now, I understand your dilemma. I thought dmoc had the right idea suggesting CType, rather than just Type. So there is a clear distinction with regards to it's construction and implementation. Typical users do not have to use it.

I'd suggest that even the syntax in code be different for accessing the members (fields), such as "->".. This makes it obvious all throughout the code that this is a different type.

e.g.

CType colourargb
   Field a:Byte
   Field b:Byte
   Field g:Byte
   Field r:Byte
End CType

CType vert3 
   Field x:Double
   Field y:Double
   Field z:Double
   Field normal:Double

   Field colour:colourargb
End CType

Local myvertar[100]:vert3

myvertar[0]->x = 5
myvertar[0]->colour->a = 255

etc...

My day job involves development primarily with C and C++, to a lesser degree Delphi and sporadically .Net (but this is increasing). I disagree that C/C++ is the only language that is perfect for this, as all the languages I previously mentioned provide some form of data structure that maps directly to its layout in memory. I've written code that allows them all to interoperate.

I think a form of 'struct' is a fundamental requirement of any modern programming environment, BMax included, whether it be for interoperating or not.

Lacking a means to create a complex data structure a-la 'struct', will require additional work for your developers to represent this in code. It will require complex peeks/pokes and data manipulation, creating messy and difficult to maintain code.

You will see benefits in performance, as these structs can be allocated as single blocks of memory.

With much of BMax's functionality developed as modules, which in turn interoperate with external libraries, OpenGL being the primary example, I think you are already "commiting" yourself to providing a fairly standard 'struct' type. I use "committing" loosely, because obviously you decide what language features go in to BMax :)

I notice that FASM even allows declaration of 'C' style structures, so perhaps you can leverage this?

Look forward to your thoughts,

Cheers,

Stu

I agree. While it is possible to live without a feature like this, leaving it out for the sake of making the language a tiny bit easier is a bad mistake.

A command like 'extern' is a good example of how ctype would work. Currently a newbie blitzmax user doesn't have to ever deal with extern, but imagine if you had left 'extern' out because you didn't want to perhaps confuse that user. You would be left with a language missing a vital feature.

Just my 2 pennies.

Hi,

This is a bit different from extern, which was easy to do (since Blitz objects are very similar to C++ objects) and didn't really change the behaviour of anything, just added some limitations.

That said, I am starting to think a bit more seriously about this and will report back soon after a bit of a head scratch!

I'm firmly on the fence about this. On the one hand, there's the option for better C interopability, but on the other hand there's potential overcomplication of code (not a problem for newcomers as I think the majority of things would be optional).

So, really, this doesn't matter that much to me. If I need to write code that works with C, I'll work with pointers -- or I could just write my own Type that creates a memory block that imitates C structures (and arrays).

I'm happy to see this post has finally generated some discussion and especially happy to see Mark now chipping in. I have to disagree strongly with Noel because my point of posting *is* that without a C-type structures code *will* become over complicated and inefficient. Also it is not as easy as just working with pointers/blocks (see previous posts for why).

I can appreciate the difficulty of shoe-horning c-types/arrays into bmax at this late stage. Personally I'd welcome a syntax overload as opposed to trying to make bmax compiler/run-time code "smarter". Question is what is the minimum needed with the least impact on bmax as it is? Considering that fundamentally what is required is a 1:1 correspondance with C maybe a CSECTION/ END CSECTION type of statement will do? This would allow a clear and seperate code path in bmax's compiler code and because it's 1:1 should be relatively simple to implement (mostly same code as already exists?). It would also be the most flexible solution.

I have already provided an example of where C-type access is needed (vbo/opengl) but maybe others would like to chip in and the requirements will be clearer? (although to me and maybe others it's a no-brainer). I can offer another example: in the PC world I never got into mmx/sse type stuff because of the number of different implementations. Some/most compilers provide options and/or produce single exe's with run-time optimisations to make best use of whatever hw it finds. Clearly a case for leaving it to the compiler if possible to decide what's best. However, on the Mac the Altivec instructions/facility is pretty much standard and I would very much like to make use of it. AFAIK Altivec requires "unit stride", ie, no gap between the data or in other words a C-type structure/array format. FYI Altivec allows the same operation/s to be performed on a block of data very quickly. Examples would be creating the "tweens" of a 3d model, updating particles, real-time procedural textures, etc. Many more uses become apparent the more you think about it.

To Mark: sorry for not raising this issue months ago. It was only last week that I started playing with bmax and did not realise until then how far you had objectified things. I really hope you can come up with a solution.

PS: Other issues: xplatform *safe* threading, network server and IPC (see other post/s)

Noel,

This is not just an interoperability issue, this is a fundamental and performant language feature, only to benefit BMax and it's community. I'm not sure I understand how it "over" complicates the code - it indeed does the opposite, by simplifying it. See my previous examples above.

In essence it is syntatically very similar to a regular "Type" declaration in bmax. I don't even see a need to elevate this to an 'object' type, I think it is fine to treat it like any other first class data type (int, float, byte, etc).


I think you've made a point in favour of a 'C' style struct by suggesting that if you have the need to write code that works with C (or infact Delphi, .Net). To elaborate, by using pointers or writing a type that must create memory blocks that imitate the layour of a complex data structure (I've dropped the 'c' name, because interoping could invole any language with this ability), you've considerably over-complicated your code, reduced readability and maintainability.
You've now introduced additional memory requirements to create this 'fake' structure (which will be accessed by either memcopy's or peeks/pokes), overhead in copying to and from this structure and therefore performance.

Here is a more realistic example:

Noel, this is what you would have to do, and I think you may agree that adding a 'C' structured type is far less of an 'overcomplication' than the hoops required in this example ;-)

Type colourargb
   Field a:Byte
   Field b:Byte
   Field g:Byte
   Field r:Byte
End Type
' struct size 4 bytes

Type vert3 
   Field x:Double
   Field y:Double
   Field z:Double
   Field normal:Double

   Field colour:colourargb
End Type
' struct size 36 bytes

Type cube
	Field name:Byte[32]
	Field verts:vert3[8]
End Type
' struct size 320 bytes

Print "colourargb size: " + SizeOf(colourargb)
Print "vert3 size: " + SizeOf(vert3)
Print "cube size: " + SizeOf cube 


Local mycube:cube = New cube
Print "mycube size: " + SizeOf mycube

For i = 0 Until mycube.verts.length
	mycube.verts[i] = New vert3
	mycube.verts[i].colour = New colourargb
Next



' cube size is 320 bytes
' layout:
' 0..31    : name
' 32..319  : vert3[8]

' vert3 size is 36 bytes
' layout:
' 0..7     : x
' 8..15    : y
' 16..23   : z
' 24..31   : normal
' 32..6    : colourargb

' colourargb size is 4 bytes
' layout:
' 0				 : a
' 1				 : g
' 2				 : b
' 3				 : r

' create bank:

Local cubemem:TBank = CreateBank(320)

' copy cube.name
For i = 0 Until mycube.name.length
  PokeByte(cubemem, i, mycube.name[i])
Next

vert3size = 36
vert3start = 32
offset = 0
' copy cube.verts
For i = 0 Until mycube.verts.length
  PokeDouble(cubemem, i * vert3size + offset, mycube.verts[i].x)
	offset :+ 8

  PokeDouble(cubemem, i * vert3size + offset, mycube.verts[i].y)
	offset :+ 8

  PokeDouble(cubemem, i * vert3size + offset, mycube.verts[i].z)
	offset :+ 8

  PokeDouble(cubemem, i * vert3size + offset, mycube.verts[i].normal)
  offset :+ 8

  PokeByte(cubemem, i * vert3size + offset, mycube.verts[i].colour.a)
  offset :+ 1
  
  PokeByte(cubemem, i * vert3size + offset, mycube.verts[i].colour.b)
  offset :+ 1
  
  PokeByte(cubemem, i * vert3size + offset, mycube.verts[i].colour.g)
  offset :+ 1
  
  PokeByte(cubemem, i * vert3size + offset, mycube.verts[i].colour.r)
  
  offset = 0
  
Next

' now we can pass packed data structure off to external library
' call some lib function

' upon return, where the data may have been modified, we must copy
' back to our local objects

offset = 0
' copy cube.verts
For i = 0 Until mycube.verts.length
  mycube.verts[i].x = PeekDouble(cubemem, i * vert3size + offset)
	offset :+ 8

  mycube.verts[i].y = PeekDouble(cubemem, i * vert3size + offset)
	offset :+ 8

  mycube.verts[i].z = PeekDouble(cubemem, i * vert3size + offset)	
  offset :+ 8

  mycube.verts[i].normal = PeekDouble(cubemem, i * vert3size + offset)
  offset :+ 8

  mycube.verts[i].colour.a = PeekByte(cubemem, i * vert3size + offset)
  offset :+ 1
  
  mycube.verts[i].colour.b = PeekByte(cubemem, i * vert3size + offset)
  offset :+ 1
  
  mycube.verts[i].colour.g = PeekByte(cubemem, i * vert3size + offset)
  offset :+ 1
  
  mycube.verts[i].colour.r = PeekByte(cubemem, i * vert3size + offset)
  
  offset = 0
  
Next

With a 'C' style or complex data type, it would be as simple as:

CType colourargb
   Field a:Byte
   Field b:Byte
   Field g:Byte
   Field r:Byte
End CType
' struct size 4 bytes

CType vert3 
   Field x:Double
   Field y:Double
   Field z:Double
   Field normal:Double

   Field colour:colourargb
End CType
' struct size 36 bytes

CType cube
	Field name:Byte[32]
	Field verts:vert3[8]
End CType
' struct size 320 bytes

Print "colourargb size: " + SizeOf(colourargb)
Print "vert3 size: " + SizeOf(vert3)
Print "cube size: " + SizeOf cube 


Local mycube:cube = New cube

' call external lib function, passing mycube pointer directly

' no need to copy back to anything, as it was already modified in place.

So, aside from reducing the complexity, we have reduced:

Original:

mem allocs : 18
mem pokes (copy to bank) : 96
mem peeks (copy from bank): 64

Proposed 'c' style:
mem allocs : 1
mem pokes : 0
mem peeks : 0

That is 145 memory operations down to 1. Not bad. This also doesn't change based on the number of vertices in our 'object', cube in this case.

Let's extrapolate that out a little, to some arbitrary object with 1000 vertices:

Original:

mem allocs : 2002
mem peeks : 16032
mem pokes : 16000

Proposed 'c' style:
mem allocs : 1
mem pokes : 0
mem peeks : 0

Quite a difference.


Cheers,

Stu

Mark,

we really appreciate you looking in to this.

I understand that my example is worst case, and could be optimized a bit by doing memcopy's using the first field address to get the beginning of each vert3 and subsequently each colourargb type, but still there is many kb of data being pumped around unnecessarily, as you are well aware..

Cheers, and look forward to your post head scratch thoughts :)

Stu

Hi,

Well, for starters, your example is clearly an interoperability example - you wouldn't be passing stuff like that around within a Max program!

The way you've done it is fairly clumsy too. Better is something like...

Local buf:byte ptr=MemAlloc(320)
MemCopy buf,name.ToCString(),name.length
MemCopy buf+blah1,vert1,SizeOf(vert)
MemCopy buf+blah2,vert2,SizeOf(vert) 'etc...

No, not pretty either, but not quite as bad as you're making out. I would also imagine this code would be 'encapsulated' inside a more Max like object, so the 'ugliness' is at least localized!

Side note: Max will autoconvert an object to a byte pointer by taking the address of the first field, so it is actually easy to pass single structs to libs.

As for your hypothetical 'C' version, there is a nasty 'dual usage' of CType vars. Within a CType, it represents an 'instance', eg:

Field color:colorargb

...however, here it's magically a pointer...

Local cube:mycube=New mycube.

The fix of course is to 'C-ize' it even more (or, invent some weird 'context sensitive' behaviour - no!), eg:

Local cube:mycube Ptr=New mycube 'now we have an 'array' of mycubes

...which is getting slightly ugly, or just plain...

Local cube:mycube 'an instance!

...which is nice until you have to pass it to someone and VarPtr it. Not to mention the new implied side issue of copying/passing/returning stuff 'by value'.

My concern is that, before long, there are Ptr's and Varptr's all over the shop, the request for '->' in addition to '.' (to avoid "(p[0])." - max's equivalent to C's "(*p).") and it all just starts going in a direction I'm not particularly interested in.

I like Max being 'its own' language as I have directions I want to take it in that don't have a lot to do with C/C++, and I don't want to see it's 'identity' polluted with too much nasty stuff.

Still thinking though!

I don't even think this is an issue of language "aesthetics". Another way of viewing it is that bmax provides "object" access at one extreme, byte access at the other extreme and nothing in between. The thing inbetween is c-type access and I don't see why is needs to pollute the langauge. My suggestion of a "CSECTION" command (actually a bmax compiler directive) means little if any change to syntax. C-type access could be restricted to banks, limiting the impact on current bmax code, and allowing normal bmax array and c-type array access within the CSECTION (again with little/no change to syntax of code in the CSECTION or elsewhere).

Keep thinking Mark ;-)

Hi Mark -

perhaps there is a bug with the sizeof operator as it returned 4 and 8 for the sizes of my example object objects. It did not return the true data size. The first code example is actually runnable. That is why the Print sizeof(cube), etc are in the example program.

Also, when I created the arrays of objects in my example, if I didn't 'new' the colourargb and vert3 objects, no matter what element within the array I modified, it effected all array elements, indicating references, and not values.

So, as I mentioned, you could do mem copys, but I chose not to, as the sizeof operator was not working as I expected and each vert3 and colourargb was a reference.

This would have still have to be:

' ... all code to create and initialize the cube object, including all it's vert3 and colourargb components

Local buf:byte ptr=MemAlloc(320)

MemCopy buf,name.ToCString(),name.length

MemCopy buf+blah1,varptr(mycube.verts[0].x), 32
blah1 :+ 32
MemCopy buf+blah1,varptr(mycube.verts[0].colour), 4
blah1 :+ 4

MemCopy buf+blah1,varptr(mycube.verts[1].x), 32
blah1 :+ 32
MemCopy buf+blah1,varptr(mycube.verts[1].colour), 4
blah1 :+ 4

... etc

We still have :

mallocs : 18 mem
to bank copies : 16 memcopy
from bank copies: 16 memcopy

This still jumps up quite a bit when you get to 1000.

Also the is the requirement to increment a variable for the pointer offset. The amount of data being copied is still the same too, just less operations.

Yes, you can encapsulate it in an object, but there is no automatic way to do this, and so it must be manually written for each object type.

By suggesting the use of a "->" operator for these 'c' style types, I meant no relationship to C or C++ - it was merely a means to identify these types differently in code. Someone perusing some bmax code could see that a->x or b[10]->y referenced 'c' style objects, where as d.k or l[12].g referenced a blitz style object, and so they are different syntactically and semantically.

There is going to be a lot of unneccassary memory allocation and copying going on, that is my concern. The use of external physics engines is another example where passing arrays of vertices (or other data) is such a common exercise. Being this is per frame, it means potentially 100's of kbs to in excess of MBs of data per second. With the availability of a C style structure, there will be no need for any of this.

Blitz already has a number of advanced features - direct use of OpenGl, 'extern' to interface to external libraries. This is beyond a lot of average users, so I think there is a real need to provide the 'C' style structure for completeness.

Cheers,

Stu

Seems at least one other objectised-BASIC has run into this issue so Mark gets a bonus point if he provides a solution!

Stu: Took you a while to come up with something that looked that bad, eh? ;)

You're outdoing yourself there, because that could be simplified a great deal (you're doing more than just over -- complicating it, you're trying to make it look bad).

Since I don't feel like recreating your example, here's how I handle OpenGL vertex buffers:

(Note: this is not the actual code, but a shortened illustration because I don't have the code on this PC)

Local ptr:Byte Ptr = MemAlloc(VertexArr.Length*12)  ' Let's just say the size of vertices is 12 (X, Y, and Z floats)
Local stream:TBankStream = CreateBankStream(ptr,VertexArr.Length*12)

For Local i:CVertex = EachIn VertexArr
 stream.WriteBytes(i.X, 12) ' Writes all 3 floats to the stream
Next

glInterleavedArrays(GL_V3F, 0, ptr)

Can't guarantee that code would work as I don't remember exactly how to use TBankStream or TStream.WriteBytes anymore (haven't touched BlitzMax lately as I've been studying for class and working on a new version of my interpreter [written in C, not BlitzMax]).

Also...


That'd be wrong -- you're still going to be passing that large amount of data, just in a slightly different format.

My point, anyways, is that there's always a 'clean' (one of those 'in the eye of the beholder' things) solution to what you want to do. You don't need C structures and arrays to write code that'll work fine.

Also, you should be writing C code that works with BlitzMax first, not vice versa. Make any sense? Didn't think so -- that's me for you.

In any case, we agree to disagree on this subject. I've already lost where the heck I was going with this post...

Sorry StuC, but if you want -> and other complicated stuff, why don't you use C++ as you try to make BM to C++ 2 just because you are not willing to adapt the programming style of BM.

Me thinks people fail to grasp point of post, sigh. I remember people arguing *against* dll access for B3D because they couldn't see a use for it, sigh. Seems like more of the same, sigh. Good luck to the one's who do "get it". Personally I'm into the practicality of a language not it's aesthetics, I'll leave that to the academics and purists in their imaginary ivory towers. See yer when/if c-type access is provided.

Noel,

I am requesting this to help you and make bmax a better platform that will surpass it's competition. I want to see bmax take the crown. Be number one.
For my day job, I write highly optimized C and C++ (incl. multithreaded) code that is deployed to ten's of thousands of machines around the world. One of the many optimizations is code that reduces memory allocations and copying, by writing custom memory allocation schemes, and optimizing the layout of structures in memory. Believe me, I have many years of experience with this and do know a thing or two about what I am talking about ;-) This will make the difference of many FPS.. Isn't that what we're all going for? :)

I am not arguing that you need C structures for the code to work fine. We can certainly agree on that. You are unfortunately missing the point. It is a matter of performance and complexity. You again are proving our reasoning for requesting this - by the very fact you are required to stream the objects into some other buffer. This means overhead. Considering bmax is a gaming and graphics / media oriented development platform, I would like to think that performance is a primary consideration and goal.

My code was generalised to show examples of complex data types, not a simple 3d vertex. You are showing a simple structure. I discussed that already.


You are absolutely incorrect. By providing a C structured type, the memory that is used by this type in bmax is passed by reference, as a pointer to the calling library, so there is a 4 byte push onto the stack to the address of the vertex memory, rather than a complete copy to a new bank. ..and then a copy back once the vertices have been manipulated!


Not sure of your point here? I'm not talking about writing code that I want to interface to bmax. I'm talking about interfacing to the many well designed, tested and supported libraries out there that already have structured requirements for their APIs. OpenGL is one, but there are many others - physics libraries. Sound libraries. Etc..


Dreamora - I don't want that member access operator at all. It was merely a suggestion to Mark, to differenciate a bmax object type from a bmax 'structured' type, since he was concerned about confusion. I don't care one iota that bmax looks very little like C++. I write and debug software in several languages for my day job, that look nothing like C++, and I like them.

Cheers,

Stu

This is where varptr comes in which is mainly for communication with C/ASM and not for BM internally as it bypasses the GC.

I think something along these lines would be a great addition, Stu.

varptr only works on a single variable which will not work when you need a structure of variables.

A maybe a more practical example of something this might be for loading and saving of game data files.

Right now if you want to do so, you either know the combination of data sizes and read them one by one, or you read them into a bank and convert them.

It would be nice (and easier to maintain and debug) if I could do:

Struct HighScoreRecord
    Field Place%
    Field Score%
    Field Initials[3]
End Struct

Then I could do something like:

WriteBinary HighScoreRecord, sizeof(HighScoreRecord)

Obviously this is a simplistic example as well, however, I could see it being used to load and save complex Level data. Being able to read chunks of files into a Stucture like that could make the difference between long load times and fast load times.

Not sure what you are stating here? This argument has nothing to do with the GC. varptr provides a memory reference to an instance of a variable, and has little to do with the GC. Arcadenut is absolutely write, and an excellent example of where this applies only to bmax, and not about interfacing to external modules.

Let me state again. For an array based object type, blitz max stores them as an array of pointers.

Type vert3
  Field x:Double
  Field y:Double
  Field z:Double
End Type

Local verts:vert3[100]

"An array is a collection of similar elements stored in adjacent memory locations". This is not an array of vert3s. If you do not understand what an array of pointers is, what an array of struct's is, and how they differ in implementation, you will not understand why this is such an important feature.

Let me try to clarify for you, that this is not a major language feature, per se, as bmax syntactically already has a 'structured type'. "Type" fulfills this need.
This is a compiler and runtime implementation detail, so the impact on all of you coders should be very minimal from a language point of view, but major for those that need it.

I feel this argument is closed. I was only trying to help those who lack the technical expertise to understand what the implementation details were. Mark has already responded and decided to look into this, as he too understands this issue.

Cheers,

Stu

varptr gives you the pointer to an object as well.

WriteBinary varptr HighScoreRecord, sizeof(HighScoreRecord) 

would theoretically work.
The only problem with that is that arrays in objects are not inline in BM, so the only thing you would get for Initials is its reference / ptr.


But how this should affect the loading times? Don't see the point there, as it is only making a difference in the loading code.


Again: this is not C or C++. Most examples to show are C/C++ that don't work in any other language so I don't see the point in forcing in that direction ...
Just because C does it this way it must not be a good thing.
Creating a module for squencial fileaccess for example would be a solution as well (greetings from VB for example).

Perhaps freeing the mind from C and looking for all solutions??

varptr gives you the pointer to an object as well.

WriteBinary varptr HighScoreRecord, sizeof(HighScoreRecord) 

would theoretically work.
The only problem with that is that arrays in objects are not inline in BM, so the only thing you would get for Initials is its reference / ptr.


But how this should affect the loading times? Don't see the point there, as it is only making a difference in the loading code.


This is not C or C++. Most examples to show are C/C++ that don't work in any other language so I don't see the point in forcing in that direction ... Just to make C hackers (not programmers as they would think of a way before bringing up language bound samples) live easier?

Creating a module for squencial fileaccess for example would be a solution as well (greetings from VB for example).

Perhaps freeing the mind from C and looking for all solutions might help ... or going back to C if it is loved that much that all other languages should be like it.

Dreamora - this is so much more of a compiler implementation than an language issue. You have totally missed the point. C has nothing to do with it and I am not a C or C++ advocate at all. We are using C as an example, since many people know it.
ALL the examples will work in Delphi, VB, VB.Net, C#, C++, C, Pascal and many others. I've written software in all of them.

...and there appears to be a bug in the sizeof operator.
Which leads me to another point. sizeof happens to come from c / c++ (and many other langs). It's primary use in bmax is to identify the size of the type in memory so you can do binary streaming, and manipulate the object in many other low level ways. More C/C++ similarities include the :+, :-, etc (albeit C/C++ is +=, -=), the extern keyword. Whether you like it or not, bmax has many similarity to C.

My large example above (using colourargb, vert3 and cube types) has print sizeof(colorargb), etc.. They are incorrect. You can cut and paste the code upto the prints to see.

Anyhoo, it is entirely an implementation detail. Most programming languages since the beginning of time have always had some sort of complex data type that maps back to it's layout in memory.

A Delphi example for you to get off of C:
Type
  HighScoreRecord = Record
    Place : long;
    Scord : long;
    Initials: Array[0..2] of Char;
  End;

VB.Net:
Structure HighScoreRecord 
  Place As Integer
  Score As Integer
  Initials as System.Char[3]
End Structure 

VB 6.0:
Type HighScoreRecord 
  Place As Long
  Score As Long
  Initials As Char[3]
End Type

C++
class HighScoreRecord
{
public:
  long place;
  long score;
  char initials[3];
}

etc..

I don't care about C, Delphi or C++. I care about bmax in this instance.

I am at a loss as to why you are trying to make your life more difficult. We already have a structured type - no change here. All we are asking is that it is an exact representation in memory (or some way to 'flag' that it is). Nothing more.

Being a sucker for punishment I'll try one last time: @Dreamora - you have to ask yourself if you are following this thread? It's because bmax has moved *toward* a C++ type language and *away* from a BASIC/C type language that this issue arises. With respect, you need to re-read this post from the top and if you have a solution for efficient storage and manipulation of bulk data, regardless of what it may be eventually passed to, then surprise us all (including Mark) with a concrete and complete example.

The simplest version I can think of with respect of the actual working:

Varptr returns a pointer to the actual memory chunk in which the object is and not to the BM Object representation as sizeof actually does (otherwise it would be 8 bytes too large).

Then the only thing that would be needed would be a inlinefield beside field which takes care that you can define arrays inline so that they are saved in the object memory chunk itself and not as pointers to somewhere else.


Wouldn't that be simpler?

YES! You are heading in the right direction. Something to allow us to identify we want the Type to have inline members. There are a few more implementation details I have previously discussed, so we won't go over them again.

Cheers,

Stu