http://blitzmax.com/codearcs/codearcs.php?code=471This code tells you if two line segments intersect, and if so, where.
Assuming P3 is moving, and P4 is the location it's trying to move to, then your two line segments are P1-P2 and P3-P4.
http://blitzmax.com/codearcs/codearcs.php?code=450This code calculates the "normal" of a line. Multiply the two components of this normal by N and add the result to the position of the intersection you calculated above, and you've moved P3's intersection point back to the correct side of the line by N units.
Lastly, you want to determine if P3 is on the correct side of the line.
You have the normal of the line, N1, which you calculated above.
Now, calculate a normal which points from P1 to P3, or P2 to P3. Either one, doesn't matter:
Calculate vector/unnormalized normal:
N2x# = P3x#-P1x#
N2y# = P3y#-P1y#
Calculate length of vector:
D# = Sqr(N2x#*N2x# + N2y#*N2y#)
Normalize normal (make its length 1):
N2x# = N2x# / D#
N2y# = N2y# / D#
Now, all you have to do is do a 2D dot product between the normal of the line, and the normal from P1 to P3:
Dot# = N1x#*N2x# + N1y#*N2y#
The Dot Product will be 1 if the normals point in the same direction, 0 if they point 90 degrees off from one another, and -1 if they point in exact opposite directions. (And any number in between if they're only roughly the same.)
In other words, if P3 lay in the line P1-P2, then the normal from either of those points would be pointing along the line, and would be 90 degrees off of the line's normal which points perpendicular to the line, and the dot product would be 0. But if it were positioned where the red dot is in the example above, then the normal pointing at it, and the line's normal would be pointing roughly in the same direction and the dot product would be greater than 0.
And I think that's all you need to do. :-)