Wow! That has problems. And yet, by a miracle, it works for the example. Try changing CY a little, say to 110.
I've tweaked it to match the method I outlined. And I let b move over a larger range, to emphasize that the projection is really onto the infinite line through a and c, i.e. the point o may be outside the triangle.
I left in the Z's, which are irrelevant for 2D. So this already works for 3D. You just have to find nice way to draw everything. In fact, this method works for any number of dimensions, although you would need more letters.
Global RX#, RY#, RZ#
Local AX#, AY#, AZ#
Local BX#, BY#, BZ#
Local CX#, CY#, CZ#
Local UX#,UY#,UZ#
Local VX#,VY#,VZ#
AX = 100 : AY = 200 : AZ = 0
BX = 250 : BY = 100 : BZ = 0
CX = 300 : CY = 250 : CZ = 0
Graphics 800,600,32,2
SetBuffer BackBuffer()
While Not KeyDown(1)
Cls
If tog = 0 Then bx = bx + 1
If Tog = 1 Then bx = bx - 1
If bx > 400 Or bx < 70 Then tog = tog Xor 1
Line AX,AY,BX,BY
Line BX,BY,CX,CY
Line CX,CY,AX,AY
Vector(AX,AY,AZ , BX,BY,BZ) ; R is vector ab
VX = RX
VY = RY
VZ = RZ
UnitVector(AX,AY,AZ, CX,CY,CZ) ; R is unit vector in direction of ac
d# = Dot( VX,VY,VZ, RX,RY,RZ )
RX = RX * d
RY = RY * d
RZ = RZ * d ; now vector R is ao, the projection of ab onto ac.
Line BX, BY, AX + RX, AY + RY ; point o is point a plus vector R.
Flip
Wend
Function Dot#(X1#,Y1#,Z1#, X2#,Y2#,Z2#)
Return x1*x2 + y1*y2 + Z1*Z2
End Function
Function UnitVector(X1#, Y1#, Z1#, X2#, Y2#, Z2#) ; from 1 to 2
Local L#
RX# = X2#-X1#
RY# = Y2#-Y1#
RZ# = Z2#=Z1#
L# = Sqr((RX# * RX#) + (RY# * RY#) + (RZ# * RZ#))
RX# = RX# / L#
RY# = RY# / L#
RZ# = RZ# / L#
End Function
Function Vector(X1#, Y1#, Z1#, X2#, Y2#, Z2#)
Local L#
RX# = X2#-X1#
RY# = Y2#-Y1#
RZ# = Z2#=Z1#
End Function
This could be made a little more efficient. If you look closely you will see that the normalizing factor 1/Length( vector ac) has, in effect, been applied twice. Thus you are really dividing by length^2, so you could eliminate a square root and some divisions.
I can implement this if a little extra speed is of any significance.