Dirty syntax highlighting example

Hi. Finally got it working, so I only need cleanup, optimization etc.
Just thought I'd post a working sample which only checks/colorcodes the last string the user wrote.

I'll post a clean example in the Code Archives once I got it sorted out.
Just thought I'd give something back since I got so much (and quick) help on the subject :)

If, then, else, end are colored white, and =, <, > are colored green. Just as an example.

Edit: Changing lines with the mouse goes very wrong. Will fix later :)

SuperStrict

Local MyWindow:TGadget=CreateWindow("Syntax highlighting", 0,0,640,480)
Global MyText:TGadget=CreateTextArea(0,0,GadgetWidth(MyWindow),GadgetHeight(MyWindow),MyWindow)
SetGadgetLayout MyText,2,2,2,2

Local GUIFont:TGuiFont=LoadGuiFont( "Courier New",12)
SetTextAreaFont(MyText,GUIFont)


SetTextAreaColor(MyText,1,81,107,True)
SetTextAreaColor(MyText,255,255,50,False)

Global Keywords:String[] = ["print","if", "then", "else", "end"]
Global Operators:String[] = ["=", "<", ">"]

Local cursorpos:Int

Function IsKeyword:Int(test:String)

	test = test.tolower()
	For Local i:Int = 0 To Keywords.length-1				
	
		If Keywords[i].tolower() = test Then
			Return True
		End If

	Next
	
	Return False

End Function

Function IsOperator:Int(test:String)

	test = test.tolower()
	For Local i:Int = 0 To Operators.length-1				
	
		If Operators[i].tolower() = test Then
			Return True
		End If

	Next
	
	Return False

End Function


Repeat
  WaitEvent()
  Select EventID()
  Case EVENT_WINDOWCLOSE
     End
  Case EVENT_GADGETSELECT
    cursorpos=TextAreaCursor(MyText)
  End Select

  Local CurLine:String = TextAreaText(MyText, TextAreaLine(MyText, cursorpos), 1, TEXTAREA_LINES)
  Local LastWord:String = CurLine[CurLine.findLast(" ")..CurLine.length]
  LastWord = LastWord.trim()

  If (IsKeyword(LastWord))
	  FormatTextAreaText(MyText,255,255,255,0,Cursorpos-LastWord.length,LastWord.length)
  Else If (IsOperator(LastWord))
      FormatTextAreaText(MyText, 50, 255, 50,0,Cursorpos-LastWord.length,LastWord.length)
  Else
	  FormatTextAreaText(MyText,255,255,50,0,Cursorpos-LastWord.length,LastWord.length)

  End If


  SetStatusText MyWindow, LastWord
Forever
End

To program a highlighter, you need to program a lexer.

Function Test:Short(Blub:Int=8, Blub2:String="hello")
EndFunction

-- Terminals
Letter = [ "A".."Z" ] | [ "a".."z" ] ;
Digit = [ "0".."9" ] ;

-- Nonterminals
Types = "Int" | "String" | "Short" ;
Ident = Letter { Letter | Digit } ;
Parameter = Ident ":" Types ;
ParameterList = "(" Parameter { "," Parameter } ")" ;
Function = "Function" Parameter ParameterList "EndFunction" ;

Like this.

Very compilcated :)
Have fun!

cu olli

And as next, It is also nearly impossible with Bmax to write
a highlighter with a good speed.(espacially in Win32) Thats because of the richedit control, which is used. There are several tweaks for this, but mostly they are already implemented in BMax.
I have tried a highlighter by myself, and finally comes to the conclusion that it isn't worth the time.
My Highlighter was able to parse a text of around 150kb in nearly 12 seconds (very bad,but better than Bmax IDE). I was
able to change the behaviour of the highlighter to just highlight the current visible region of the text. which increased the loading time to 10% of the previous. but this had some issues with the scrollbars, as the jump repeadly from beginning to the end. So not a good solution.

@Vertex: At first he only needs a Tokenizer (a simplier Form of a Lexer)
For Example:
You have the delimiters " ( , ) :
and your text to analyse is :
MyType:String = GetString("Hello")
this will return this tokens:
MyType -> : ->Stirng etc.
this tokens you could save with its position and then highlight them if needed. The only thing you have to do is to check if a Token is in the range of "" (in this case Hello ). More complicated is the adding of multirem support.

Cheers, klepto2

Actually I don't need a tokenizer for my project as it's kinda special. I just need fast syntax highlighting and I can assume there's space between each of my words :)
But yes, for true programming language highlighting it would be needed.

I can see the speed issues, but I have a couple of ideas I need to try out. I'll let you know how it goes :)

Here are two functions that will be very usefull (but win32 only).

Function TextAreaFirstVisibleLine:Int(TextArea:TGadget)
	Local hwnd=QueryGadget(TextArea,QUERY_HWND)
	If hwnd Return SendMessageA(hwnd,EM_GETFIRSTVISIBLELINE,0,0)
End Function


Function GetlastVisibleLine:Int(Richedit:TGadget,font:TGuiFont)
	

	Local H:Int = GadgetHeight(richedit)    		  
	Local F:Int = bbFontHeight(font.Handle)  
	Local Ht:Float = (H/F)	

	Return TextAreaFirstVisibleLine(richedit)+Ht	
End Function   

Maybe these will help you

Ahh yeah. Those would be handy :)

I am working on a very fast method for finding words.
It is based of making a data tree of all the words to search. This uses each letter of the word to calculate a pointer to the next character in the word.

It uses lots of memory, but these days we have lots!

It is fast because the work is being done as the file loads. The search time is as fast on 1,000,000 words as it is to look thru 100 words. The search speed is based off the size of the word you are looking up.

This version only uses Spaces as word separators, so 'TEST' is dif from 'TEST?'


This example uses the Data Tree to find words fast in big files. It can load the works of Shakespeare in 1.5 seconds. It can find as fast as you type.

Klepto , I just noticed you are using bbFontHeight.
This is commented out in bmax 1.18 font.cpp and font.h as far as I can tell.
How are you get this to work??
I need a function like your GetlastVisibleLine, but it fails to compile for me!

Yes, I have chaanged the source to get it work ;)

Here a list of what you have to add:

First uncomment the bbFontHeight function in font.cpp
then add in font.h this, that the extern "C" block looks like this in the end.

Int			bbFontDescent( BBFont *font );
Int			bbFontHeight( BBFont *font );
};

After that you have to edit the Win32Gui.bmx file

and add this function:

Function bbFontHeight( font )

and finally rebuild the module.

Should also work with bbFontwidth

BtW: In my Highlighter I have used the TMap and it was really fast with more then 16000 unique Keywords. Will yours be faster than Tmap?
And the problem for a highlighter isn't the comparison with keywords. Unfortunatly the bottleneck of fast highlightning in Bmax is the Richedit Control. The only way to speed it up, would be to Heck in the OnPaint Event and afaik impossible in BMax :(

Thanks for the info. It would be nice to have the font height as a standard command , not a hidden/commented out one!

Tmap looks good, but I have not used it. So I can not really say which is faster.
My method uses each letter of the words ascii value to calc an offset in a 'branch' which points to the next branch. Looping thru the words characters ends up the position of the data.
A clever aspect of this is that A,AN,AND and ANDREW use the same branches.
I just did a rough test. Looking thru The complete work of shakspeare (142160 different words) for the word SUPER .I asked it to look for it 100,000 times and it took 1.45 seconds.