challenge: find 2 most important ints from array

Miscellaneous Forums/General Discussion/challenge: find 2 most important ints from array

Ok,

Imagine for example these numbers:

4,9,3,4,9,2,8,4

What I'd like to see is a simple function (no tons o' bloat worth o' types and functions.. just something small!) that gives me the number that occurs the most, and the 2nd-most, on a any array size, of ints. The two most important numbers, so to say..

The result should in this case be:

best: 4
2nd-best: 9

If you're up to it, return an array in which all variables are sorted on quantity, so that there's also a third place, fourth place, etc. (depending on the availability of such variables ofcoz), but best and 2nd-best will do, for me.
In case of more than one 2nd-best, just pick some. Imagine a sorting routine where multiple candidates share the same rank.

One may use Bmax, but I estimate it could be uni-blitzcode.

One last thing: assume that I really want to use full int values, so you can't temporalily overwrite (parts/bits) of them.

Well there are several strategies for solving this particular problem - most would probably revolve around an associative array, which in BlitzMAX, pretty much means a TMap. If you care to write your own priority queue that could also be an interesting solution. Or if speed isn't really your worry, a jagged array could provide a solution.

Is the max number going to be 2^16 or -+2^16? Oh I see you say full int values.
And how many elements are we looking at?

It's for my texture generator. I want to apply this thing so it can pick the two most important colors in a range of 8, it'll be to convert images to characters (that usually have one foreground and background color per line orso, or per character even. So speed is a little bit of an issue, since I'm not going to wait minutes or even some seconds on a result.

This int, in this case just is $00rrggbb, so I require at least 24bits worth of information to store in that int. I could say that '8' is the limit, as this is a sorta classic limitation (per line). But I figured that an algorithm usually doesn't mind values so I figured that a generic routine would do the trick.

I actually did some pattern'izing like this some while ago, but that was on indexed images, this one is meant for normal bitmap images (it's just another texture generator filter).
In the indexed version I only had 16 possible colors, so I just counted whatever indexes there were in the line o' 8 pixels, and put their SHL8 quantities in an array, OR'ed with the index. Then I sorted that array, since the quantity was SHL8'ed, it sorted on quantity. Then I AND'ed the index from the first 8 bits again, et voila. It would btw be so nice if BMax could store the index after sorting somewhere, so one would be able to apply it's changed state on other stuff.

First, run through the variable and pick out all the possible numbers. Store those in an array. Then, in another array, count how many times each element of array 2 appears in array 1.. Re-order arrays 2 and 3 based on highest to lowest. Then just look at array 2 as it is now a list of most common to least common numbers.

Code-wise this is up to you. But that is loggically what you wanna do. You might be able to cut out bits, depending on what you really want to do. Such as not re-ordering the arrays, but just checking it for the highest and second highest numbers. Which should be faster.

I'm sure there are other places to trim, but this is basicly what you wanna accomplish.

Here you go. I'll try and add comments to the code later

To summarize the code below: It'll work on any size array with any number of entries in any range. In fact, if you want to change it from int's to double or bytes or whatever you only have to change 2 function parameters. It will sort by the # of times a number has appeared in the array, and returns a key/value data pair that you can use to retrieve the both the numeric value + the number of times that number appeared in the array. Not including the hash table code (since hash tables really should be a standard part of BMAX, imo) and the test code, this is only 63 lines long with liberal whitespace.

First, you'll need to include my THashTable code (newly updated for this challenge...had to add a few functions)

''Constructor( capacity:int ) - returns a new Hashtable object with a maximum capacity = to 
''							the parameter provided. Utilize this function to create 
''							any and all hashtable objects.
''
''
''InsertEntry(name:string,o:object) - add an object to the table, the name is used to
''                                    to generate an index 0-Capacity, selecting the list. 
''							      multiple entries with the same name are permitted using
''								 this method. If you want to guarantee that only one 
''								 entry exists per a given name, use the 
''								 insertUniqueName( ) method
''
''insertUniqueName( name:string, o:object) - this method guarantees that the object specified
''									    will be the only object for that particular name.
''
''insertUniqueNamedObject( name:string, obj:object) - this method guarantees that there will only be 
''											    one copy of the object specified by the obj parameter
''											    in the hashtable. There may be multiple entries for
''											    the name however. Using this instead of the 
''											    standard InsertEntry() method when you want to make
''											    sure the same object (assuming it is the same name)
''											    is not duplicated in the table.
''
''RemoveNamed(name:string) - Removes all entries with the name specified from the hash table.
''                           Note, this does not return those objects. 
''
''RemoveNamedObject(name:string, obj:object) - removes any entries from the bucket specified by the name
''										 parameter whose object value matches the object specified
''										 by the obj parameter. This method lets you quickly remove
''										 a single object without removing all objects that share the
''										 same name (such as RemoveNamed() does.)
''
''RemoveObject( obj:object ) - searches every bucket in the hash table for the specified object and removes
''						   all traces of it from the hash table. Note: This method can be relatively slow
''                            on large tables, and if possible you should use RemoveNamedObject() instead.
''							
''GetEntry:object(name:string) - grab the first entry for a given name. You must cast it back to its original 
''                               type before using it,myobject=myType(GetEntry(name:string))
''
''GetMultipleEntries:object [](name:string) - returns an array of objects (entries) that match the
''									          given name parameter. This is useful when you intend
''									          to store a list of objects with a given name index
''
''Grow( growthsize:int ) - Increases the number of buckets in the hash array by growthsize.
''
''removeAllEntries() - removes all objects from the hash table                                    
''
''removeNamed( name )	removes all entries with the given name from the hash table
''
''
''									Regarding Unique Entries
''                                  ------------------------
''If you want to ensure there is only one entry for a given name key, there are two approaches you can take.
''If you want or need to ensure there is no entry for a given name key before you do an insert - or if there
''is an entry already present you want to handle it in some manner you should
''call getEntry( name ). If this is null, it's safe to insert. If this returns an object calling InsertEntry()
''will insert another entry with the same name. While you can access multiple entries with the same name via
''the method GetMultipleEntries(name), doing so is not ideal if you simply want a single entry per name.
''So the first approach is to call getEntry, and deal with the object returned (if any) and then do your 
''insertion.
''The second approach comes into play if you don't need access to any entry that might already be stored 
''under the insert name, but simply want to guarantee the entry you are inserting is the only one for
''that name. Calling insertUniqueName(name) will erase any entries already stored under "name" and guarantees that 
''there is only one unique entry for the name provided.


'''''''''''''''''''''''''''
''
''HASHTABLE OBJECT
''
''- M.Laurenson/Defoc8 2006
''- S.Hofslund/SculptureOfSoul 2006
''
''- major modifications: the introduction of a capacity parameter, getMultipleEntries(),
''  insertUniqueName(), insertUniqueNamedObject(), removeNamed(), removeNamedObject(), removeObject(), and 
''  Grow() methods and the Constructor() function some error checking code, document modification
''  as well as a new and much more effective hash algorithm added by S.Hofslund (SculptureOfSoul)
''''''''''''''''''''''''''''


Type THashTable

 Field _table:TList[]
 Field _capacity:Int
 ?debug
 Global _indirectconstruct:Int
 ?
 Method New()
  ?debug
  Assert _indirectconstruct, "Use THashtable.Constructor() to create a new hash table."
  ?
 EndMethod

 Function Constructor:THashTable( capacity:Int )
 ?debug
  _indirectconstruct = True
 ?
  Local retobj:THashTable = New THashTable
   retobj._capacity=capacity
   retobj._table=retobj._table[..capacity]
	For Local n:Int=0 Until capacity
     retobj._table[n]=New TList
	Next
  ?debug
  _indirectconstruct = False
  ?
  Return retobj
 EndFunction

 Method Grow( growthsize:Int )
  If growthsize <= 0 Return  

  Local oldtable:TList[] 
  oldtable = _table
  _capacity = (_capacity + growthsize)
  _table = New TList[_capacity]
  
  For Local i:Int = 0 Until _capacity
	_table[i] = New TList
  Next
  'regenerate indexes for and reinsert all of the old tables entries
  For Local n:Int = 0 Until oldtable.length
   For Local entry:gHashEntry = EachIn oldtable[n]
	InsertEntry( entry.name, entry.obj )
   Next
  Next

 EndMethod

 Method genIndex(name:String)
  Local val:Int=0
  Local temp:Int	
   For Local n:Int=0 Until name.length
'the following is commented out because it is neither as fast or as efficient as the one used below
'	val:+ (name[n])^2 + (name[n]*(n^2)) + (name[n]Mod 3 * name[n])
	val= (val Shl 3) + val + name[n] 
'commented out for the same reason 
'	val:+ (name[n])^(((name.length + 1 - n)Mod 2) + 2)
   Next
'call Abs on the index because it might be negative (in the case of an integer overflow). 
  Return  Abs(val Mod (_capacity ) )
 EndMethod

 Method insertEntry(name:String ,obj:Object)
  Local index:Int=genIndex(name$)
  Local entry:gHashEntry=New gHashEntry
    entry.name=name
    entry.obj=obj 
    entry.link=_table[index].AddLast(entry)
 EndMethod

'object must have an overriden compare method!
 Method insertSortedEntry(name:String,obj:Object,ascending:Int=True)
  Local index:Int=genIndex(name$)
  Local entry:gHashEntry=New gHashEntry
    entry.name=name
    entry.obj=obj 
    entry.link=_table[index].AddLast(entry)
    _table[index].Sort(ascending)
 EndMethod 

 Method sortEntriesNamed(name:String,ascending=True)
  Local index:Int=genIndex(name)
  _table[index].sort(ascending)
 EndMethod

 Method sortAll(ascending:Int=True)
  For Local iter:Int=0 Until _table.length
   _table[iter].sort(ascending)
  Next
 EndMethod

 Method insertUniqueNamedObject( name:String, obj:Object )
  Local index:Int = genIndex(name)
  internal_removeNamedObject( obj, index )
  Local entry:gHashEntry = New gHashEntry
   entry.name = name
   entry.obj = obj
   entry.link = _table[index].Addlast(entry)
 EndMethod
 
 Method insertUniqueName(name:String,obj:Object)
  Local index:Int=genIndex(name$)
  internal_removeNamed( name, index )
  Local entry:gHashEntry=New gHashEntry
    entry.name=name
    entry.obj=obj 
    entry.link=_table[index].AddLast(entry)
 EndMethod

 
 Method getEntry:Object(name:String)
  Local index:Int=genIndex(name)
  Local link:TLink = _table[index]._head
  Local entry:Object
 Rem
 the old for eachin variety of the loop, replaced below by the hand rolled loop. Uncomment this and comment the 
the below code to see for yourself the speed difference it makes (only noticeable when doing 1000's of operations though)
   For Local entry:gHashEntry=EachIn _table[index]
    If(entry.name=name)
     Return(entry.obj)
    EndIf
   Next
 EndRem
   
  'Print "_table[index].count() =" + _table[index].count()
  For Local iter = 0 Until _table[index].count()
    'handrolling the loop as it turns out to be much faster than a For..Eachin
    entry = link._succ._value   

    'do the keys match?
    If gHashEntry(entry).name = name
      Return gHashEntry(entry).obj
    EndIf
    link = link._succ
     'Return(entry.obj)
    'EndIf
   Next
  Return Null
 EndMethod

 Method getMultipleEntries:Object[](name:String)
  Local index:Int = genIndex(name)
  Local retarray:Object[] = New Object[getEntryCount(index)]
  Local objectcount:Int

   For Local entry:gHashEntry=EachIn _table[index]
     If entry.name = name
	  retarray[objectcount] = entry.obj
	  objectcount:+ 1
	 EndIf
   Next
		
	'resize the array to objectcount elements.
    	retarray = retarray[..objectcount]    
	Return retarray
 EndMethod

 Method getAllEntryPairsAsArray:gHashEntry[]()
	Local numEntries:Int
	Local retArray:gHashEntry[]
	Local entryCount:Int 
	
	  For Local n:Int=0 Until _capacity
   		numEntries:+ getEntryCount(n)
      Next

	retArray = New gHashEntry[numEntries]
	entryCount = 0
	
	 For Local iter = 0 Until _capacity
		For Local entry:gHashEntry = EachIn _table[iter]
			retArray[entryCount] = entry
			entryCount:+ 1
		Next
	 Next
	
	retArray = retArray[..entryCount]
	Return retArray
 EndMethod 

 Method getAllEntryPairsAsList:TList()
	Local retlist:TList = CreateList()
	
	
	 For Local iter = 0 Until _capacity
		For Local entry:gHashEntry = EachIn _table[iter]
			
			retlist.addlast(entry)
				
		Next
	 Next
	
	Return retlist
 EndMethod 

Rem

The below are leftovers from a variety of tests. I figured someone might find these of interest. Anyhow, except in
certain special situations, I found them to be slower than the code I'm using for getMultipleEntries. Feel free
to experiment though.

 Method getMultipleEntries2:Object[](name:String)
  Local index:Int = genIndex(name)
  Local objectcount:Int

   For Local entry:gHashEntry=EachIn _table[index]
     If entry.name = name
	  objectcount:+ 1
	 EndIf
   Next
	Local retarray:Object[] = New Object[objectcount]
	objectcount = 0
   For Local entry:gHashEntry = EachIn _table[index]
     If entry.name = name
	  retarray[objectcount] = entry.obj
	  objectcount:+ 1
	 EndIf
   Next
	'resize the array to objectcount elements.
	'MemCopy( retarray, retarray, SizeOf(retarray[0]) * objectcount)
    
	'retarray = retarray[..objectcount]    'above method is faster
	Return retarray
 EndMethod
EndRem
Rem
 Method getMultipleEntries3:Objwrapper(name:String)
  Local index:Int = genIndex(name)
  Local retarray:Object[] = New Object[getEntryCount(index)]
  Local objectcount:Int
  Local wrapper:objwrapper = New objwrapper
   For Local entry:gHashEntry=EachIn _table[index]
     If entry.name = name
	  retarray[objectcount] = entry.obj
	  objectcount:+ 1
	 EndIf
   Next

	'resize the array to objectcount elements.
	'MemCopy( retarray, retarray, SizeOf(retarray[0]) * objectcount)
    wrapper.objarray = retarray
    wrapper.length = retarray.length
	
	Return wrapper
 EndMethod
EndRem

'simply a slightly faster version of removeNamed. Faster because the index is provided
'and doesn't need to be generated. This method is called by insertUniqueName
 Method internal_removeNamed( name:String, index:Int )
  For Local entry:gHashEntry = EachIn _table[index]
   If(entry.name = name)
    entry.link.Remove()
   EndIf
  Next
 EndMethod

 Method removeNamed:Int( name:String )	'returns the # of objects removed
  Local index:Int = genIndex(name)
  Local remove_count:Int = 0
  For Local entry:gHashEntry = EachIn _table[index]
   If(entry.name = name)
    entry.link.Remove()
    remove_count:+ 1
   EndIf
  Next
  Return remove_count
 EndMethod

 Method internal_removeNamedObject( obj:Object, index:Int )
   For Local entry:gHashEntry = EachIn _table[index]
	If(entry.obj = obj)
	 entry.link.Remove()
	EndIf
   Next
  EndMethod

 Method removeNamedObject:Int( name:String, obj:Object ) 'returns # removed
   Local index:Int = genIndex(name)
   Local remove_count:Int = 0
   For Local entry:gHashEntry = EachIn _table[index]
	If(entry.obj = obj)
	 entry.link.Remove()
	 remove_count:+ 1
	EndIf
   Next
   Return remove_count
  EndMethod
'this loops through every bucket in the hash table
 Method removeObject:Int( obj:Object )	'returns # removed
  Local remove_count:Int = 0
  For Local iter = 0 Until _table.length
   For Local entry:gHashEntry = EachIn _table[iter]
    If entry.obj = obj
     entry.link.remove()
     remove_count:+ 1
    EndIf
   Next
  Next
  Return remove_count
 EndMethod 

 Method removeAllEntries() 
  For Local n:Int=0 Until _capacity
   _table[n].clear()
  Next
 EndMethod 

 Method getEntryCount:Int(index:Int)
  If (index >= 0) And (index < _capacity)
   Return _table[index].count()
  EndIf
 EndMethod
EndType

Type gHashEntry 
 Field name:String
 Field obj:Object
 Field link:TLink
 Method Compare(pHashEntry:Object)
  'Try
?debug 
  Print "gHashEntry.compare called"
?
 ' Print "Compare result: " + obj.Compare2(gHashEntry(pHashEntry).obj)
  Return obj.Compare(gHashEntry(pHashEntry).obj)
  'Catch ex:Object
  'EndTry
 EndMethod
EndType


And then here is the actual source code, broken in to a few functions so you can access the data at various stages of the analysis.

The function call to AnalyzeArray() first creates a hash-table with "Key/data" pairs of the format (Number, #OfTimesAppearingInArray)

Next, the function AnalyzeTable() gets all of the entries from the hash table and sorts them and returns the sorted list.

PrintResults then loops through the hash table entry pairs (of type gHashEntry), and prints the "Key/data" pairs.

The function AnalyzeArray just combines all of the above steps and functions into one function. If you trace through this function and the functions it calls it should make things clearer. Note that AnalyzeArray() also returns the sorted result list (as a Tlist), so you can easily get rid of the call to "PrintResults()" in AnalyzeArray (it's really just in there for testing & to display that this works) and then just do whatever you want to the returned list.

Anyhow, here is the code

Strict

Include "THashtable.bmx"


	Function CreateArrayAnalysisTable:THashTable( array:Int[] )
	
		Local hash:THashtable = THashtable.Constructor( array.length )
		Local entry:Object 
		
		For Local iter = 0 Until array.length
			
			entry = hash.getEntry( String(array[iter]) )
			
				
			If(entry)
				hash.insertUniqueName( String(array[iter]) , String((Int(String(entry))+1 )))		
			Else
				hash.insertentry( String(array[iter]), "1" ) 
			EndIf
			
					
		Next
		
		
		Return hash
		
	
	EndFunction
	

	Function AnalyzeTable:TList( hash:THashtable )
		
		Local allEntries:TList
		
		allEntries = hash.GetAllEntryPairsAsList()
		allEntries.Sort(False)
		
		Return allEntries		
	
	EndFunction
	
	
	Function PrintResults( resultlist:TList )
		For Local entry:ghashentry = EachIn resultlist
			Print "Number:" + entry.name + "~t~t~tCount: " + String(entry.obj)
		Next
	EndFunction

	Function AnalyzeArray:TList( array:Int[] )
		
		Local hash:Thashtable 
		Local resultlist:TList
		
		hash = CreateArrayAnalysisTable( array )
		resultlist = AnalyzeTable( hash )
		PrintResults( resultlist )
		
		Return resultlist
		
	EndFunction
	
	
'-------------------------------------------------------------------------------------------------
'                                TEST DATA AND CODE BELOW
'-------------------------------------------------------------------------------------------------

		
Global testarray:Int[] = New Int[9]
testarray[0] = 4
testarray[1] = 5
testarray[2] = 6
testarray[3] = 7
testarray[4] = 6
testarray[5] = 5
testarray[6] = 6
testarray[7] = 7
testarray[8] = 8

Global testarray2:Int[] = New Int[12]
testarray2[0] = -53
testarray2[1] = 326
testarray2[2] = 53
testarray2[3] = -326
testarray2[4] = 326
testarray2[5] = 32
testarray2[6] = -3
testarray2[7] = -53
testarray2[8] = 0
testarray2[9] = 12
testarray2[10] = -53
testarray2[11] = 32

Print "*********Analyzing TestArray**********"
AnalyzeArray( testarray )
Print "**************************************"
Print "~n~n~n"

Print "*********Analyzing TestArray2*********"
AnalyzeArray( testarray2 )
Print "**************************************"



Note I've included some test data in the code that runs and prints out the results.

The output is as follows for the data shown below
Global testarray:Int[] = New Int[9]
testarray[0] = 4
testarray[1] = 5
testarray[2] = 6
testarray[3] = 7
testarray[4] = 6
testarray[5] = 5
testarray[6] = 6
testarray[7] = 7
testarray[8] = 8


*********Analyzing TestArray**********
Number:6			Count: 3
Number:7			Count: 2
Number:5			Count: 2
Number:8			Count: 1
Number:4			Count: 1
**************************************



Global testarray2:Int[] = New Int[12]
testarray2[0] = -53
testarray2[1] = 326
testarray2[2] = 53
testarray2[3] = -326
testarray2[4] = 326
testarray2[5] = 32
testarray2[6] = -3
testarray2[7] = -53
testarray2[8] = 0
testarray2[9] = 12
testarray2[10] = -53
testarray2[11] = 32




*********Analyzing TestArray2*********
Number:-53			Count: 3
Number:326			Count: 2
Number:32			Count: 2
Number:53			Count: 1
Number:-326			Count: 1
Number:-3			Count: 1
Number:0			Count: 1
Number:12			Count: 1
**************************************



******Be sure to replace the line "Include "Thashtable.bmx" in the above source with the appropriate path of the Hastable code provided above!!******

Lastly, if you just want to get at the values of the "two best" numbers, grab the first 2 entries from the list returned by "AnalyzeArray()" and then cast them to a "gHashEntry" and get their ".name" field - which is a string representation of the number - and then cast it to int, like so

local resultlist:tlist
local bestresult:int
local secondbestresult:int

resultlist = AnalyzeArray( somearray )
bestresult = int(gHashEntry(resultlist.ValueAtIndex( 0 )).name)
secondbestresult = int(gHashEntry(resultlist.ValueAtIndex( 1 )).name)


If you want to know how many times the "best" value showed up in the array, you need to access the .obj field of the item in the list (each entry has a .name/.obj field which correspond to the "number/#oftimesappearing" key/data pair).

So to get the # of times the best value appeared we do this

local resultlist:tlist
local bestresult:int
local bestresultcount:int


resultlist = AnalyzeArray( somearray )
bestresult = int(gHashEntry(resultlist.ValueAtIndex( 0 )).name)
bestresultcount = int(gHashEntry(resultlist.ValueAtIndex( 0 )).obj)


Aren't you basically asking for a histogram which would be the same thing used to turn a truecolor image into a reduced palette image?

Note: If you want to speed up my hash table code (the hash tables themselves are fast - in my testings they were marginally faster than Lua's hash tables and those are highly optimized and written in C), the main bottleneck is that it's converting int's to strings to store and then back.

The hashtable itself (like TLists) can only store objects. What the hashtable stores is an internal type called a "gHashEntry" that has the fields "name" + "obj", that store the name (key) used to identify the object, and the object itself in "obj".

The hashtable does however provide you with the ability to sort entries, as do Tlists and whatever. All you have to do is write your own overloaded Compare method for the object type you are storing (look at the help section under "objects" for the footprint of this function and what it needs to return and why).

In fact, in the code above sort is already being called on the result list (which is a list of gHashEntry's), so all you'd have to do to get correct behavior for a custom int wrapper is to write an appropriate Compare method.

Note: The gHashEntry type calls Compare on the "obj" field of itself (the object it is storing) so everything will work correctly (as it currently does in the code above.)

Like I said before. The logic here is to first determine how many unique numbers there are. Then to count how many times they appear in the first list.

You can prolly do that all in one pass. With a lot less code.

say... with a matrix that looks like Numbers[unique][total]

A simple group of a few for loops and ifs and then a sort...

first for loop runs through all the numbers, and the second one which is nested in the first runs through the matrix. If the number is present it increases total by 1, if it isn't, it creates a new unique number and sets total to 1. If there are 10 numbers, all unique, it'll run through 55 times. If they are 10 numbers, all the same, it'll run through 10 times.

You'll end up with a table that looks something like:

52 , 3
12 , 8
1 , 1
37, 3

and if you sort it by the totals.

12 , 8
52 , 3
37, 3
1 , 1

I can't imagine it is any more than 10-20 lines to get the info, then more for the sort, but you can sort it in whatever advanced way you want. You can even just parse it and not re-arange it if you are just looking for 1-2 of the biggest.

It is important to note. All the code in this thread is A. In a language I've never used, and B. Too complex for me to look at and be able to understand. I don't have BMax, so I can't go break it to see what is doing what... but it seems like entirely too much code for such a simple problem.

psuedo-code

A1 = {1,2,3,4,4,4,7,7,9,10}

for (n = 1, n <= numbersInA1, n++){

   go = true

   for (i = 1, i <= numbersInM1, i++){

      if (A1[n] == M1[i][1]) {  // M1[i][1] is the first in the matrix, M1[i][2] is the total...

         M1[i][2]++
         go = false
         i = numbersInM1

      }

   }

   if (go == true) {

      M1[i+1][1] = A1[n]
      M1[i+1][2] = 1

   }

}

// Now M1 is a list of groups of 2 numbers. the first is 
//unique number, and the second is how many times it appears in the first list.

// in this case that is [{1,1},{2,1},{3,1},{4,3},{7,2},{9,1},{10,1}]



It's not that much code really. If a hashtable was a "built-in" type in BMax, and if I consolidated my code and combined my "different step functions" into one big nasty ugly function, I could probably get it down to 30 lines or so with sorting and output.

I first tried an approach similar to what you are suggesting Dampe, and it would work - sort of. The problem is that you either need two arrays as you mentioned. So each time you hit a number in the array you are analyzing, you have to search the entire length of your array to see if you've already seen that number. Not so bad on small arrays - very bad on huge arrays.

That is the same basic logic though that my example above uses. Instead of searching through an array to see if the number has already been found, it uses the number as a hashtable key and if there is already an entry there, it increases it by 1, otherwise it creates the entry and sets it to 1. Same basic logic - it's just that the lookup will be much faster when dealing with large arrays.

Friend of mine made a one-small-function solution:
Local a:Int[]=[43,43,1,6,1004,43,8,1],value:Int[8],count:Int[8]

count_thing a,8,value,count

For Local t:Int=0 To 7
	Print LSet(value[t],5)+" - "+count[t]
Next

End

Function count_thing(array:Int[], size:Int, values:Int[] Var, counters:Int[] Var)
	Local values_stored:Int=0
	Local i:Int
	Local index:Int
	
	For i = 0 To size-1
		For index = 0 To values_stored-1
			If values[index] = array[i] Exit
		Next
		
		If index = values_stored
			values[index] = array[i]
			counters[index] = 1
			values_stored:+1
		Else
			counters[index]:+1
		EndIf
	Next
	
	Local n:Int = values_stored - 1
	Local sorting:Int = 1
	Local temp:Int
	
	While sorting
		sorting = 0
		n:-1
		For i = 0 To n-1
			If counters[i] < counters[i+1]
				temp = values[i+1]
				values[i+1] = values[i]
				values[i] = temp
				temp = counters[i+1]
				counters[i+1] = counters[i]
				counters[i] = temp

				sorting = 1
			EndIf
		Next
	Wend
End Function


Shouldn't be too hard to make it a bit more BB-neutral (read: non-Max)


It works like a charm!

SOS. Can you explain the hashtable thing. I'm competent logically, what makes me a shite coder is that I can't follow peoples code most of the time. So I can't really learn these things.. and Don't have the drive to read mountains of books and so on.

So If you explain how this hashtable thing works. That would be golden.

Well, just for laughs I thought I'd see what my code looked like condensed into a single function. 27 lines of code and that includes output display code.

here it is

Strict

Include "ThashtableLean.bmx"
	
	Function AnalyzeArray:TList( array:Int[] )
		Local hash:THashtable = THashtable.Constructor( array.length )
		Local entry:Object 
		Local allEntries:TList
		
		For Local iter = 0 Until array.length
			entry = hash.getEntry( String(array[iter]) )
				If(entry)
					hash.insertUniqueName( String(array[iter]) , String((Int(String(entry))+1 )))		
				Else
					hash.insertentry( String(array[iter]), "1" ) 
				EndIf
		Next		

		allEntries = hash.GetAllEntryPairsAsList()
		allEntries.Sort(False)

		For Local entry:ghashentry = EachIn allEntries
			Print "Number:" + entry.name + "~t~t~tCount: " + String(entry.obj)
		Next
		
		Return allEntries
	EndFunction


edit: double post.

No problem Dampe. Basically, a hashtable is an associative array where you can store a "key,data" pair. So, if you wanted to store say a character (and by character I mean a player object or something, not a letter :P) by name, you'd make his name the "key" and then the character object itself would be the data. This facilitates fast lookup of random data that is best indexed with a string or some other "weird" key value. If you knew you had 10 characters and you knew exactly who you'd have up front, it'd of course be much more logical (and faster) to use an array. Hashtables are best when the dynamic is data and you want to be able to look it up based on a unique key (sorry for being redundant but just trying to be clear.)

Basically, what the hash table does is it converts the string you provide to it as a key to a number. This is called "hashing" and the algorithm to generate that # is the hashing algorithm. So, let's just say our hashing algorithm just adds up the ASCII values of each letter of the string provided as key, for illustrative purposes.

So, let's use the following pseudo-code


myObject:someobject
myObject.name = "Test"

'the first parameter to insertEntry is the key
'the second parameter is the object we want to store
'under that key value
hashtable.insertEntry( myObject.name , myObject )


So the code above is inserting an entry into the hashtable with the key "myObject.name", which evaluates to "Test".

So summing the ASCII value for Test we get

T......e........s..........t
84 + 101 + 115 + 116

or 416.

Now, what good does that # do us, and how does the hashtable use that as an index?

Well, the hashtable itself is just an "array of lists" of some size. That size is up to you when you create the hashtable - in the code in the post above I call THashtable.constructor( array.length ) - so the hashtable will have the same # of array elements as the length of the array.

For example purposes though, let's pretend we have a hashtable of size 5...0-4. So the hashtable itself is a 5 element array of Tlists.

So how do we use our generated # above, 416 (the hash value) to index this array? Well, the hashtable does this internally by doing a mod operation that conceptually looks like:

hash-value mod size-of-hash-table

This will always return a result from 0 to size-of-hash-table (well, actually it'll be from 0 to the size of the hashtable minus 1, which will then work perfectly as an index to the hashtables internal array)

so if we insert the #'s we get

416 mod 5

which = 1. (if you need to know how a mod operation works, just ask)

So the key "Test" generates a hash value of 1, which is the index where the hashtable will store the object.

So now we've stored "MyObject" with the key "Test". We can do a lookup like so

hashtable.getEntry( "Test" )

The only parameter is the key value to lookup. Now the hashtable does the same thing - it hashes the key - which will again = 416 which will then have the mod operation performed on it to produce the index of "1", and then the hash-table will return the value of the object stored at index 1.

Internally it's a bit more complex than this b/c of the possibility of "hash collisions" which are the result of 2 different keys producing the same result index. For example, using the hashing algorithm above (which is a very bad one, but it's good for demonstrative purposes) the key "Tets" would also = 416, and so would "steT" or any other arrangement of those same 4 letters. So now we've got multiple keys returning the same hash value? What happens here?

Well, internally the hashtable doesn't just store the object at the index that was generated. It stores a container object (this is the gHashEntry I've talked about in the above posts). This container object stores the "key" AND the "object". This way, multiple objects can be stored at the same index even if they have different keys, since the hash-table only returns the object whose key matches the one provided.

Hopefully this example will make it clearer. Lets say we have 3 objects, and all of the keys produce the same hash value, so they are all going to be stored internally at the same index of the hashtable (remember too that the hashtable is an array of Lists - so each time a hash collision results the object is simply added to the end of the list of the hash table)

So here's some more pseudo-code
object1:someobjecttype
object2:someobjecttype
object3:someobjecttype

'here we make a 3 element hashtable (0-2)
hash:thashtable = Thashtable.constructor( 3 )

object1.name = "Test"
object2.name = "tseT"
object3.name = "Tets"

hash.insertEntry( object1.name, object1 )
hash.insertEntry( object2.name, object2 )
hash.insertEntry( object3.name, object3 )



After this code runs, the hash table will have 3 entries. But since "Test", "tseT" and "Tets" all evaluate to the same value (1) the hashtable internally looks like this

[0] -> points to an empty list
[1] -> gHashEntry -> gHashEntry -> gHashEntry
[2] -> points to an empty list

those 3 gHashEntrys are the container objects I mentioned. The first one, which holds the first entry (object1) will look like this

ghashEntry.name "Test"
gHashEntry.obj = Object1

the 2nd one would be

gHashEntry.name = "tseT"
gHashEntry.obj = Object2

and I'm sure you can guess what the 3rd one would look like.

So now, even though all of the objects are stored in the same element of the hash table, we can still do a lookup and get the proper result with a call like so
hash.getEntry( "tseT" )


This works because when the hashtable is looking for the result it compares the key provided ("tseT") to the key of each entry in the list at the index of the hash value of that key.

So first it will reach the first entry ("Test") and compare it's stored key against "tseT"...nope, not the right object...then it goes to the next object in the list - and that one's key (or name) does match "tseT", and so it returns the associated object. It will return Object2.

Ack...this is a long post and I"m not sure if I made it real clear how this works. I have trouble illustrating concepts in any sort of brief manner, so I apologize :). Anyways, hope this helps a bit.

Ok... I think I get what you are saying. What I don't get is how this is any faster. beer, wine, or whisky. A shot is a shot.

If you have a big list of hash values, and a big list of unique numbers. You still have to look through them till you find the right one..

At least, you didn't mention how it doesn't.

If you mentioned how it does that. I missed it... haha


ALSO.. you said in mine it has to look through the 2 arrays or in my case matrix all the way each time. It doesn't, it can stop if it finds the number in the list.

Well, the key to getting the best speed out of a hash table is to make sure it's size is relative to the amount of data you're going to store in it. If you made a hashtable of size 1, it's really just a list, and just as slow.

If you know you're going to need to store say 50 elements, and you make you're hash table 150 elements big, you'll have very few collisions (if any), and so it'll basically be just as fast as an array lookup (the hash algorithm I'm using - borrowed from some brilliant comp-scientists - is extremely fast and very good at distributing hash results evenly). It'll only be marginally slower than an array lookup because it has to hash the key - but I've tested it and gotten 1 million plus lookups + insertions in about .3 seconds. The speed will vary a bit based on the # of collisions you have, but unless you have like 5000 objects in a 50 element hashtable, the speed hit will be extremely minimal.

The array lookup method would still be faster on small arrays - but having to scan even a 50 element array 50 times (once per each element in the array) could - worst case scenario - result in 2500 iterations, and that's probably going to be slower. If you were looking at a 50,000 element array, or, if the array was an array of pixels (a bmp), you might have millions of elements, obviously the hashtable would be exponentially faster.

I'm not saying that my implementation of the challenge is necessarily better than CS_TBL's friends - it's just that it'll be faster on large arrays and is flexible enough to handle arrays of any size and (in the condensed version at least) all you have to do is change the parameter type of AnalyzeArray() and it can handle doubles, floats, strings, objects (that you have written a compare method for or have one built in), basically anything at all.

For instance, changing the parameter to "string[]" and then calling AnalyzeArray on the following array
Global testarray:String[] = New String[9]
testarray[0] = "Woot"
testarray[1] = "Toot"
testarray[2] = "Woot"
testarray[3] = "Toot"
testarray[4] = "Hola"
testarray[5] = "blah"
testarray[6] = "Testing"
testarray[7] = "blah"
testarray[8] = "blah"


results in the following
Number:blah			Count: 3
Number:Woot			Count: 2
Number:Toot			Count: 2
Number:Testing			Count: 1
Number:Hola			Count: 1



Also note: if I disabled the output (print is slow) and just called AnalyzeArray() 1000 times with teh above data it only took 19 millisecs. So, it only took about 2 nanoseconds per iteration. Granted, I'm sure it's receiving some kind of caching "bonus" working on the same data with each call but still, that's fast.

I still don't really understand why. But we are getting closer.

"If you know you're going to need to store say 50 elements, and you make you're hash table 150 elements big, you'll have very few collisions (if any), and so it'll basically be just as fast as an array lookup"

I'm only unclear on this bit here. What is the hashtable doing that enables it to be faster here. Doesn't it still have to look at all the possible options? I'm missing some key piece of info here that muddles up my brain.

Well, let's say we have an array of 50,000 elements. Now, let's say the first # is 12...so now we scan the rest of the 49,999 elements and add up all the 12's.

Now you move on to the next element of the array, let's say element 2 is 520. So now we scan through the rest of the 49,998 elements and add 1 for each 520 we encounter. Etc etc, all the way through the array. Needless to say, that's a lot of array scanning.

You could say, well, first scan the array to see which unique #'s it has and create a new array with only those #'s, that way you won't have to scan 50,000 elements each time if there are only say, 500 unique #'s. Only problem is, to build up this list of unique #'s you have to scan your new "index" array for each element of the original array to see if it exists. So you'll still have TONS of array scanning. Also, you're index array will originally have to be as large as your original array since your original array *might* be composed of all unique #'s. In other words, you are using double the amount of space as your original array (which might be alot if you were doing this with a large .bmp)

The hash table does it different. It goes through the 50,000 elements ONCE and ONCE only. Each time it gets to the next element it converts it to a string and then hashes that and checks if a value (the count value) already exists. If it does, it increases that value by 1. If it does not, it sets that value to 1. Then it moves on to the next element in the array and does the same.

So basically, instead of scanning through a full array, it instead creates a hash value and uses that as an index into it's internal array. It's *not* scanning the array when it adds an entry. It simply tacks that entry onto the back of the list at the index that the hash algorithm returns.

The only time it does any scanning is when you are retrieving values - and even then it only scans if there have been hash collisions. So as long as you make your hash table big enough, you'll have very few collisions (1000 objects in a hashtable size 1000 usually results in an average of maybe 2 objects per index, with a max of maybe 6-8 objects in a single index with standard semi-random data). So in other words, on average it'll have to scan through 2 entries and at most 6-8 (and oftentimes there is only one entry and so no scanning is needed.)

Simplified, you can think of it this way, a hash-table is an array that uses a string as it's index, not a number. The only difference is that internally it changes that string into a #. The only reason it does any scanning is when more than one string index generated the same # index, and that scanning is only done on retrieval.*


*-well, that's not 100% always true. I do have insertion functions that scan - these are useful when you want to overwrite the object stored for a particular key. The standard insertion simply tacks the object to the end of the list at that index.

That doesn't make sense. There must be some internal scanning somewhere. Otherwise, it is just magic.

"The hash table does it different. It goes through the 50,000 elements ONCE and ONCE only. Each time it gets to the next element it converts it to a string and then hashes that and checks if a value (the count value) already exists. If it does, it increases that value by 1. If it does not, it sets that value to 1. Then it moves on to the next element in the array and does the same."

How does it check to see if the value already exists... doesn't that require checking the list?

I understand all this other stuff.. What I don't understand is the how it can check all the values, without checking all the values.. and if it IS checking all the values... Then... How can it be faster?

For the sake of discussion let's exclude hash collisions and the resulting "scanning" that takes place, since it is theoretically possible that you won't have a hash collision with a certain subset of data, and then it behaves essentially just like an array.

So what's the difference? Well, the difference is this - in your proposed solution you have the original array, and then you create two arrays - let's call them the "already-exists" and "count". You insert each number you see in the original array into the "already-exists" array at element X, and then increase the count at element X on the count array.

So if your original array is like so

1 2 4 2 5 1

The first scan would put 1 into already-exists[0], and then add 1 to the same index of the count array -- count[0] = count[0] + 1.

So now you move on to the second element of the original array. Now - to see if this value exists - you have to scan the "already-exists" array. Each time you move to the next element in the original array, you have to scan the length of the already-exists array.

So if you had 50,000 elements in your original array, you'd have to scan your 50,000 element "already-exists" array to see if the number was seen before. Granted, you might not have to scan the full length of the array if the # is found earlier, but you still have to do some scanning.
The reason this will take forever is because at element 49,000 you may have 49,000 lookups in the "already-exists" array (if the number isn't found and the whole array is scanned), and then when you move to the next element, 49,001 - you may have 49,001 lookups and then 49,002 and so on and so forth.

The hash table doesn't need to scan itself to see if the number exists - instead it uses the number itself to generate a hash-value that corresponds to an element within itself. If something is already stored there, then obviously the number already exists. In fact, we can store the "count" value there.

The whole point of a hash table is that it doesn't need to scan it's entire length to find an element - it uses the key (whatever key you provide it - in this case it uses the number you provided and converts it to a string and hashes it) and that key value IS the index where that value is stored.

So if you have the following hash lookup

hash.getEntry( 53352 )

the hash table doesn't scan it's entire length to see if 53352 exists anywhere. It converts 53352 to it's string equivalent "53352", and then converts that string to a # (using a hashing algorithm like the one described in an earlier post). This number is then used as an index to the hash tables internal array.

So no scanning is done. Instead, it generates the index based on the key provided, and then looks in that index.* If something is there, we know the # already exists (because something was stored with the same key - in this case the same number). If nothing is there, we know the # doesn't exist.

I don't know how to break it down any more - I'm sure it could be said clearer but probably not by me :P. The difference between a hash table and an array - in the way we are using them here - is like the difference between indexing an array with a known index versus scanning a list for an element.

What is faster:

someArray[200]

or

someList.FindValue( someValue )

Obviously the array is, because it can nearly instantly jump right to index 200 and retrieve the value there (if any). The list however, has to scan through each element and see if that particular element = someValue. Worst case scenario, the list reaches the end (X number of scans where X is the length of the list) and didn't find the value. The array is always constant time - either there is a value at 200 or there isn't.

Basically, the hashtable behaves like the array in the example above. It uses the provided key as an index into it's array and there's either a value there or there isn't. It doesn't really need to scan at all**. The array method proposed here by you & others requires you to scan through some index array to see if the # already exists, for *every* single # in the original array. While this is faster than a hash lookup on a small array, the hash table will end up being much faster on a large array.

*Actually, the hash algorithm generates a number based on an input string. THIS number will ALWAYS be the same for a given key. Just like an encryption algorithm always produces the same results on the same input with the same key. However, the hash table is only an X element array, so we need to turn the number generated (which can be very large) into a value between 0-X so that we can use it to index the hash table. That's what the mod operation is for (described in the post a few up) - it takes the number generated and reduces it to between 0-X.

**omitting hash collisions. I probably shouldn't have mentioned them in the above posts because they don't have that much bearing on how a hash table works on the conceptual level, and by mentioning them I only made the whole concept harder to understand.

Better yet, here's a quick illustration of the concept. Remember, with a hash table you use a string as an index, where a normal array uses a number.

i.e.

normal array:
array[5] = 25

That sets the the 6th element of the array (the indexes are zero based, hence 6th and not 5th) to 25

hashtable approach:

hash["my-key"] = 25

This sets the index "my-key" to 25. So instead of using a number as an index, it uses a string.

So now, let's look at the problem again. You've got some array

1 2 3 4 5 6 7 8 9 0 1

Your approach says make a new array that you scan through each time to see if the element already exists (scanning an array in this fashion is like scanning a list).

So once you reach the zero, you have to scan the index array to see if it already exists, so the scan looks like this

Scan: 1->2->3->4->5->6->7->8->9->end

nope, doesn't exist, we reached the end before finding it.

The hash table just turns that zero into a character "0", and uses that as an index

hash["0"]

so then we just create an if clause (if you look at my code you'll see this), and do something like this

if hash["0"]
//if it returns a value we know it exists so we can increase it with some code here
else
//if we reach this, we know there is no value stored at the index "0" - in other words we haven't seen the number yet, so set the value to 1 with some code here

endif


So that's why it's not doing any scanning. It's just saying - hey, is there a value at index "0" yet? - and then responding appropriately if there is or isn't.

"The hash table doesn't need to scan itself to see if the number exists - instead it uses the number itself to generate a hash-value that corresponds to an element within itself. If something is already stored there, then obviously the number already exists. In fact, we can store the "count" value there."

NOW I get it.

Hello.

Wouldn't a binary tree be better for this?

Goodbye.