fun rotation math that i just cant get right

im trying to do a little rotation trick, but am having a hard time coming up with a proper solution. hoping someone here may be able to chime in and tell me where im being a goofball :)

my goal is this... i have two entities which must always face each other. locking them to always directly facing each other is easy (PointEntity) but doesnt have the desired effect im looking for. what i would like to do is force the nodes to look at each other, but can vary their look angle by 45 deg on either side of directly looking at each other. my algorithm has been this:

1) get current entity1 rotation

2) pointentity to entity2 and get direct facing rotation

3) if the difference between current and direct is > my allowed angle difference, adjust current rotation to allowed angle degs off center

i may need to provide a working example, but here is my code. it works well except when the node crosses the 179/-179 border. then the calculation goes awry and jumps the rotation to the opposite allowed offset. in this code, nodes are essentially TEntity objects and rotations are only done around the Y axis.

Local current_rot:Vector3df = node1.GetRotation(True)
PointEntity(node1.GetNodeObject(), node2.GetNodeObject())
Local direct_rot:Vector3df = node1.GetRotation(True)
If Abs(current_angle) > angle_limit Then
   Local diff:Float = (Abs(current_angle) - angle_limit)
   Local angle_adjust:Float
   If current_angle < 0 Then angle_adjust = direct_rot.GetY() - angle_limit Else angle_adjust = direct_rot.GetY() + angle_limit
   current_rot.SetY(angle_adjust)
EndIf
node1.SetRotation(current_rot, True)

any thoughts would be much appreciated :)

It's a 2D rotation, but working the similar way I think.
Hope this helps.

Type face
	Global faces:TList = New TList
	
	Field x:Float
	Field y:Float
	Field rot:Float = 0
	Field radius:Float = 50
	Field anglimit:Float = 45
	
	Method render()
		SetColor(80, 80, 80)
		DrawOval(x - radius / 2, y - radius / 2, radius, radius)
		
		SetColor(255, 0, 0)
		DrawLine(x, y, Cos(rot) * radius * 1.5 + x, Sin(rot) * radius * 1.5 + y)
		
		SetColor(0, 0, 255)
		DrawLine(x, y, Cos(rot + anglimit) * radius * 1.2 + x, Sin(rot + anglimit) * radius * 1.2 + y)
		DrawLine(x, y, Cos(rot - anglimit) * radius * 1.2 + x, Sin(rot - anglimit) * radius * 1.2 + y)
	End Method
	
	Method New()
		faces.AddFirst(Self)
	End Method
	
	
End Type

Function faceeach(f1:face, f2:face)
	Local ca:Float = f1.rot
	Local ta:Float = ATan2(f2.y - f1.y, f2.x - f1.x)
	
	Local la:Float = f1.anglimit
	Local da:Float = ta - ca
	Local dif:Float = Abs(da) - la
	
	If dif > 0 Then
		Local da1:Float = Cos(da + la)
		Local da2:Float = Cos(da - la)
		If da1 >= da2 Then
			f1.rot = ta + la
		Else
			f1.rot = ta - la
		End If
	EndIf
End Function

Graphics(640, 480)

Local f1:face = New face
Local f2:face = New face

While Not KeyHit(KEY_ESCAPE)
	
	If MouseDown(1)
		f1.x = MouseX()
		f1.y = MouseY()
	End If
	
	If MouseDown(2)
		f2.x = MouseX()
		f2.y = MouseY()
	End If
	
	faceeach(f1, f2)
	faceeach(f2, f1)
	Flip
	Cls

	For Local f:face = EachIn face.faces
		f.render()
	Next
	
Wend

*EDIT* Sorry, this code is not correct...
I'll modify it.
But at least you can use it for testing.. lol


*EDIT2* It's working properly now I think.
The COS(da+la) is used instead of angle comparison.
It choose the smaller angle when diff > limit, in order to avoid the jumpping.

greetings Tommo! thank you very much for the working example :) the example is _exactly_ what im looking to do and is a real beauty to see in action finally :) i almost abandoned the idea completely :\ unfortunately when i translated it to using MiniB3D it still functions but while the rotations are limited to the angle limit, they stop just outside of the angle limit so they nodes can never look at each other. i will whip up a little cone-based example later today. there may be some intricacy in MiniB3D causing some issue. here is what i have (maybe you can spot something till then:

Local current_rot:Vector3df = node1.GetRotation(True)
Local current_pos:Vector3df = node1.GetPosition(True)
Local other_pos:Vector3df = node2.GetPosition(True)

Local ca:Float = current_rot.GetY()
Local ta:Float = ATan2(other_pos.GetZ() - current_pos.GetZ(), other_pos.GetX() - current_pos.GetX())

Local la:Float = angle_limit
Local da:Float = ta - ca
Local dif:Float = Abs(da) - la
Local da1:Float = 0
Local da2:Float = 0

If dif > 0 Then
	da1 = Cos(da + la)
	da2 = Cos(da - la)
	If da1 >= da2 Then
		current_rot._y = ta + la
	Else
		current_rot._y = ta - la
	End If
	
	current_rot._y = current_rot._y + angle_limit
EndIf

node1.SetRotation(current_rot, True)

here is an example in MiniB3D. use arrow keys to move the right player, WASD for the left player:

[EDIT] i was able to get it working finally the original way i was trying to accomplish it and provided the faceeach2 function below. the mathematical way you were trying to do it is probably more effecient/fast so if you feel like taking a crack at it please do. if not, i think im good! thank you very much for your help :)

Framework SiDesign.MiniB3D

Graphics3D 640,480,0,2
Local camera:TCamera = CreateCamera()
PositionEntity camera,0,25,-25
RotateEntity camera,55,0,0

Local face1:face = face.Create()
Local face2:face = face.Create()

PositionEntity face1.Entity(),-17, 0, 17
RotateEntity face1.Entity(), 0, -90, 0

PositionEntity face2.Entity(),17, 0, -17
RotateEntity face2.Entity(), 0, 90, 0

While Not KeyDown( KEY_ESCAPE )
	
	If KeyDown(KEY_UP) Then MoveEntity(face2.Entity(), 0, 0 ,1)
	If KeyDown(KEY_DOWN) Then MoveEntity(face2.Entity(), 0, 0 ,-1)
	If KeyDown(KEY_LEFT) Then TurnEntity(face2.Entity(), 0, 5, 0)
	If KeyDown(KEY_RIGHT) Then TurnEntity(face2.Entity(), 0, -5, 0)

	If KeyDown(KEY_W) Then MoveEntity(face1.Entity(), 0, 0 ,1)
	If KeyDown(KEY_A) Then TurnEntity(face1.Entity(), 0, 5, 0)
	If KeyDown(KEY_D) Then TurnEntity(face1.Entity(), 0, -5, 0)
	If KeyDown(KEY_S) Then MoveEntity(face1.Entity(), 0, 0 ,-1)

	faceeach2(face1, face2)
	faceeach2(face2, face1)
	
RenderWorld
Flip
Wend

End 

Type face
	Global faces:TList = New TList
	
	Field x:Float
	Field y:Float
	Field rot:Float = 0
	Field radius:Float = 50
	Field anglimit:Float = 45
	
	Field face_mesh:TEntity
	Field face_front:TEntity
	
	Function Create:face()
		Local new_face:face = New face
		new_face.Init()
		Return new_face
	EndFunction

	Method Entity:TEntity()
		Return face_mesh
	EndMethod

	Method New()
		faces.AddFirst(Self)
	End Method

	Method Init()
		face_mesh = CreateCube()
		face_front = CreateSphere()
		ScaleEntity(face_front, .5, .5, .5)
		PositionEntity(face_front, 0, 0, 3)
		EntityParent(face_front, face_mesh)
	EndMethod
	
End Type

Function faceeach(f1:face, f2:face)
	Local ca:Float = EntityYaw(f1.Entity(), True)
	Local ta:Float = ATan2(EntityZ(f2.Entity(), True) - EntityZ(f1.Entity(), True), EntityX(f2.Entity(), True) - EntityX(f1.Entity(), True))
	
	Local la:Float = 45
	Local da:Float = ta - ca
	Local dif:Float = Abs(da) - la
	
	If dif > 0 Then
		Local da1:Float = Cos(da + la)
		Local da2:Float = Cos(da - la)
		If da1 >= da2 Then
			RotateEntity(f1.Entity(), 0, ta + la, 0, True)
		Else
			RotateEntity(f1.Entity(), 0, ta - la, 0, True)
		End If
	EndIf
End Function

Function faceeach2(f1:face, f2:face)
		Local angle_limit:Int = 45
		Local current_rot:Float = EntityYaw(f1.Entity(), True)
		Local current_rot_x:Float = EntityPitch(f1.Entity(), True)

		' find the rotation to the other boxer
		PointEntity(f1.Entity(), f2.Entity())		
		Local final_rot:Float = EntityYaw(f1.Entity(), True)
				
		Local current_angle:Float = 0
		Local adjust_angle:Float = 0

		' check if we crossed the 180 border
		If (current_rot < -90 And current_rot > -179.99999 And final_rot > 0) Or (current_rot > 90 And current_rot < 179.99999  And final_rot < 0)
			
			current_angle = (180 - Abs(current_rot)) + (180 - Abs(final_rot))

			If (current_angle) > angle_limit Then  
				' turn left here 
				adjust_angle = (Abs(current_angle) - angle_limit)
				If current_rot < 0 Then adjust_angle :* -1
			EndIf
		Else 
			current_angle = current_rot - final_rot	
			If Abs(current_angle) > angle_limit Then
				adjust_angle = (Abs(current_angle) - angle_limit)
				If current_angle > 0 Then adjust_angle :* -1
			EndIf
		EndIf
				
		' restore original rotation
		RotateEntity(f1.Entity(), current_rot_x, current_rot, 0, True)
				
		' adjust to within bounds
		If adjust_angle <> 0 Then TurnEntity(f1.Entity(), 0, adjust_angle, 0, True)

EndFunction

Here's modified faceeach.
I just change the Atan with your pre-point way to get target rotation.
A mod operation can help you limit the angle within 360.
Happy coding. :-)

Function faceeach(f1:face, f2:face)
	Local ca:Float = EntityYaw(f1.Entity(), True)
	PointEntity(f1.Entity(), f2.Entity())
	Local ta:Float = EntityYaw(f1.Entity(), True)
	
	Local la:Float = 45
	Local da:Float = ta - ca
	Local dif:Float = Abs(da) - la
	
	If dif > 0 Then
		Local da1:Float = Cos(da + la)
		Local da2:Float = Cos(da - la)
		If da1 >= da2 Then
			'use +360 mod 360 to limit angle within 0-360
			RotateEntity(f1.Entity(), 0, (ta + la + 360) Mod 360, 0, True)
		Else
			RotateEntity(f1.Entity(), 0, (ta - la + 360) Mod 360, 0, True)
		End If
	Else
		RotateEntity(f1.Entity(), 0, ca, 0, True) 'restore current angle
	EndIf
End Function