Array limits

I want to use a REALLY big Array in one of my programs. I currently have this because I dont need it that long at the moment:

Global Dictionary:String[27,27,27,27,27,27,27,27,27,27,27,27,5]

My problem is that when i run this i get a memory exception error and i know its because of the size because a smaller version of it works. This poses a problem as the array will need to have about 20 of the 27's (but this could go up to 30 if i wanted it to work perfectly).

There is a good reason because i want to store words in it and to do so i have each part of the array = a letter so 1 is A, 2 is B etc (With 0 being no letter) and then the last bit store certain details about the word.

Is there a way to get around the restricted length or are there other languages allow larger sizes?

Clearly there is a better way to store your data than that.

The only restriction is "How much memory is available to the system". That code works perfectly well here, and I can assure you that even larger arrays are possible.

This works fine too:

Global Dictionary:String[..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,..
27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,5]

But it takes a while because of the rediculous ammounts of memory required. In your case getting around the limitation would seem to be a matter of "buying more RAM".

That array is trying to create a block of memory THIS BIG: 750,473,176,484,995,605

That's nearly a gigabyte of gigabytes.

I bet the guys at MSN and Google combined don't have that much memory.

Whatever you are trying to do is obviously not correct.

What are you smoking?

Dunno, but it looks effective. I want some.

FlameDuck, there is NO WAY you can find a single computer with that much memory to consume, not until some time.

You got bitten by an apparent BlitzMax bug (aka "when basing language knowledge upon a compiler implementation is no good").
Try that

Global Dictionary:String[1000,1000,1000,1000]
Print SizeOf(Dictionary)

Actually this raises a good quesiton for me, seeing as the first post was declaring massive multi-dimensional arrays by accident, how do you declare and array and populate it all on one like like oin other languages? Can this be done in Blitz Plus or only Max or neither?

Matt, What happens when you "PRINT ASC("A")-65" (in max equivalent)? Wy cant you just DIM(Numberofwords)?

You can't. You could probably use data statements and a loop to do it though. I hate using data statements though. And they may not even be supported by Blitzmax for all I know... I think Blitzmax has no goto or labels?


"This poses a problem as the array will need to have about 20 of the 27's (but this could go up to 30 if i wanted it to work perfectly).

There is a good reason because i want to store words in it and to do so i have each part of the array = a letter so 1 is A, 2 is B etc (With 0 being no letter) and then the last bit store certain details about the word."


I see what you are trying to do. You're trying to access the array by putting the word in. Like this:

Dictionary(Chr("A"), Chr("P"), Chr("P"), Chr("L"), Chr("E"), 0, 0, 0, 0, 0)

But if you were able to do that, you would need enough memory to hold every single possible combination of letters in the english language. There is no way you can do that with the amount of ram computers currently have, or will have in the foreseeable future.

If you want to store words in a way which is easily retriveable, then there are better ways to to it.

I'm thinking you'll need to use multiple arrays indexing one another. And you might have to limit yourself to storing the index of words that start with a particular letter, and how many words are in that list. Then from there you might be able to do the same for the second letter, but from the third on, you may have to simply do a standard search through the remaining words which should not be that many.

Ie:
If your word list is
APPLE
ABACUS
APRICOT
PENCIL
ZEBRA

Then you would store that the A words start at index 0, and that the AB words start at index 1. You could use a two dimensional array for this: Dim Index(27, 27) Heck, you could go in 5 letters if you wanted to waste 57 megs of ram... Four might be more realistic.

I'm thinking there's gotta be a better way though.

can anyone answer my question from above?

sswift just did.

You mean this line: Dictionary(Chr("A"), Chr("P"), Chr("P"), Chr("L"), Chr("E"), 0, 0, 0, 0, 0)? with Dim in the front? I'll just test ... OK fine thanks.

Now how about for two dimensional arrays? Of course we don't need to discuss jagged arrays!

Um... considering that that array in the original posting requires 750,000 Terabytes (or 750 Petabytes), and that windows 2000 and XP can only address a maximum of 4GB of memory, this is not going to work 'perfectly well', no matter how much RAM you slap into your system. You may be able to declare it, but certainly won't be able to populate and retrieve data from all of it.

Your own example requires 27^572+5 bytes.
Google calculator maxes out at 27^215, which is 5.53616843 × 10^307 bytes. That's a number over three hundred digits long, and still only an insignificant fraction of the total array size you specified.

"Oops?"

Grey:
No, he means about using a data statement and a for loop. But I don't know about Blitzmax.

Yeah I realise the data statement is possible but I wanted to initilise the Dimension when it's declared on the same line C/Delphi/VB style.

Funny thing was I went into Blitz IDE and did

Dim Dictionary(Chr("A"), Chr("P"), Chr("P"), Chr("L"), Chr("E"), 0, 0, 0, 0, 0)

and of course it worked because it made a huge array!
Then of course it fails with

Notify(dictionary(0))

with "incorrect number of dimensions", doh.

So there is no way of initialising an array when declaring it?

Matt has obviously misunderstood how multi-dimensional arrays use memory. That is one Huuuuuuuuuuuuuuuuge array! :) I don't think I've ever used more that a 3 dimensional array - possibly a 4 dimensional one.

Clearly, arrays aren't the answer to this problem. For storing and retrieving words in a data structure, look into using 'hash tables'. They're perfect for this task.

Mhmm Hash Table, Reminds me of HashCake :]

Thanks for all the help on this.

I came up with this one night when i couldn't sleep trying to work out a way of storing data and getting it quickly for a chatbot. It was about 5 in the moring and i had been up most the previous day so i should have realised it wouldnt work.



I think i misunderstand multidimentional arrays fullstop.



Thats a really cool word. I didnt even know it existed :P (well i guess it doesnt technically)

I guess its back to doing my thinking during the day.

Is that more or less than a Brazillian? (One for regular forum readers, there :)

Don't knock it. I've had some of my best "Eureka!" moments while on the verge of falling asleep, etc. Amazing what your brain can do when you stop stressing it to find a solution. :)

"I think i misunderstand multidimentional arrays fullstop."

It's quite simple:

Dim A(5) creates the following, where each number is a single integer*:

[12345]

But Dim A(4,5) creates the following:

[12345][12345][12345][12345]

So you now have four sets of five integers.

And Dim A(3, 4, 5) creates this:

[12345][12345][12345][12345] [12345][12345][12345][12345] [12345][12345][12345][12345]

Which is three sets of four sets of five integers.

I think you can guess that Dim A(2,3,4,5) would do!

When you're working with multi dimensional arrays like this, the size increases greatly with each additional array element you add. Dim A(12, 12, 12, 12) will have 12^4 integers in it, or 12*12*12*12. That's 20736 integers, and I didn't increase the numbers very much from my examples above!


* In any language except Blitz that is. Blitz actually creates an extra element because beginners don't understand the concept of looping from 0 to SizeOfArray-1, and they try to start at 1 instead of 0. Personally I don't think Blitz should do this because it will just confuse people later.

LOL :)

FYI sswift, BlitzMax does do that (ie a 50 element array contains elements 0-49), goto and labels do exist, but goto only works in non-strict mode.

MattVonFat, theres some more info on arrays and multi-arrays (that I, Rob Farley and Big10p helped write) on my Tutorial website if you're interested.

Oh, the word 'petabyte' has been around for a long time... It's just that the systems that can handle those amounts of data haven't.
Here's the official list:

- Byte (1 byte. Duh.)
- Kilobyte (1,024 bytes)
- Megabyte (1,048,576 bytes)
- Gigabyte (1,073,741,824 bytes)
- Terabyte (1,099,511,627,776 bytes)
- Petabyte (1,125,899,906,842,624 bytes)
- Exabyte (1,152,921,504,606,846,976 bytes)
- Zettabyte (1,180,591,620,717,411,303,424 bytes)
- Yottabyte (1,208,925,819,614,629,174,706,176 bytes)

Even today there are already storage solutions that you can buy that will give you more than a petabyte, for example: (random link) http://www.enterprisestorageforum.com/hardware/news/article.php/3522666

But even though you won't be seeing storage devices with an Exabyte of space anytime soon, they are still real metrics to represent amounts of data, as in "The combined capacity of all harddrives manufactured in 1995 was 20 Petabytes" (supposedly true)

Now, think about this for a moment: over the past 16 years or so harddrives have increased in size over ten-thousand fold: from ~40 megabytes to 500 GB (half a terabyte) in a single 3.5" harddrive unit. Full terabyte units are expected to be just a couple of years away, and when you look at the progress already made, it's really not so farfetched that petabyte and exabyte range storage would be commonly available a couple of decades from now as well...

But until windows can handle more than 4GB memory addressing, you can still forget about using that array declared in the initial posting in this thread. ;-)

And you can bet by the time Exabyte storage comes about I will have found another way to do what I want...using even more memory :P

You're right ofcourse. I hadn't done the the math, just a small test program which defined the array and itterated through it printing the contents.

Since it took forever to start up, printed a rediculous ammount of 0's, and was swapping memory like crazy I just assumed it worked.

My bad.

>I came up with this one night when i couldn't sleep trying
>to work out a way of storing data and getting it quickly
>for a chatbot.

What exactly do you need to do?

Your approach above saves words into an array and enables you to retrieve the words letter by letter, but in the process uses ridiculous amounts of memory.

You can readily do the same by using a one dimensional array

Dim a$(number of words) and then use Len (a$(number)) to find the length of that particular variable. Then you can use Left$, Right$ and Mid$ to retrieve the individual letters.

This approach is much less memory-intensive, but also much slower. However, it may fit a chat-bot very well, atleast test it and make up your mind.

>I think i misunderstand multidimentional arrays fullstop.

There is a reason it is called multiDIMENSIONAL.

Dim a$(10) can be visualized as a vertical column of cells where data can be stored.

Dim a$(64,64) can be visualized as a chess-board with vertical and horizontal cells where data can be stored.

Dim a$(10,10,10) can be visualized as an appartment-building where data can be stored in every appartment.

Dim a$(10,10,10,10) can be visualized as a road with 10 of the abovementioned appartment-buildings at the side.

As with everything, if you play around with arrays, you will learn to use them correctly.


Andy

I don't know how you can store words in an array like that anyway. Surely you'd need to know the word in order to look it up?

"ABC" would be stored at Array(1,2,3,0,0,0,0,0,0....) ?

What you you put there anyway? A "1" to signify that the word exists?

;pseudo-code
age("mary") = 29
age("billy") = 26
age("eunice") = 22

If you want to store a list of key-value pairs where the keys are strings, use a map container (a.k.a. associative array, dictionary, hash, hash table.)

Take an array with say 5000 elements (these are called buckets.) For a given string key, to find out which element your value will be stored in you perform a "hash" algorithm on the string, MODding the result by your number of buckets. Any arbitrary function will do, but the more it scatters commonly used keys (english) evenly throughout the number set, the better. Here's an example of the one I use:

For character_index = 1 To Len(key$)
	Local character$ = Mid$(key$, character_index, 1)
	bucket_index% = (bucket_index% * 123) + Asc(character)		; arbitrary math, whee!
Next

Since more than one string can map to the same bucket, you also need a way to store multiple key-value pairs in each bucket. This is commonly done with a linked-list.

Presto - you can use strings (or arbitrarily large numbers, or a bunch of values, or anything you could write a hashing algorithm for) as keys for data. Every lookup, you pay a trivial cost for the hashing algorithm and you have to sort through a minimum of entries to find the correct one (on average half of the current number of entries divided by the number of buckets.)

I've an implementation in Blitz3d of map containers here: http://blitzbasic.com/codearcs/codearcs.php?code=1439

[sarcastic]
Hash tables! Why didn't I think of that!?
[/sarcastic]

:P

Yes, I saw your earlier post, but I don't think anyone else did. :(

I know - it's like talking to a brick wall, sometimes. :P