I'm a bit rusty at this because I haven't coded physics in a few months, but looking at my own code which handles balls bouncing off levels, it looks like you failed to do several things like multiply the result by -2.0, and you're missing some abs things I have in another spot.
Here's my code. Make of it what you will. :-)
; -------------------------------------------------------------------------------------------------------------------
; This subroutine animates the sphere.
; -------------------------------------------------------------------------------------------------------------------
.Animate_Sphere
; Check to see if the entity collided with the level last frame.
Entity_Hit = EntityCollided(ballpos, COLLIDE_Level)
; If the entity collided with the level, then store the time of the last collision.
If Entity_Hit Then Last_Collide_Time = Time_Now
; Calculate motion friction:
; Calculate the entity's current velocity.
Velocity# = Sqr(Vx#^2 + Vy#^2 + Vz#^2)
; If the entity is not traveling fast enough to be motion blurred:
If Velocity# < 6.0
; If motion blur is on, turn it off:
If Motion_Blur = True
SPS_DELETE_CHILD_EMITTERS(ballpos)
Motion_Blur = False
EndIf
EndIf
; If the entity is moving, adjust it's velocity:
If Velocity# > 0
; Calculate the direction vector.
; The direction vector has a length of 1.
Direction_X# = Vx# / Velocity#
Direction_Y# = Vy# / Velocity#
Direction_Z# = Vz# / Velocity#
; Compute air friction.
; Air friction is dependent on the speed of the entity, and will prevent it from accelerting forever.
Air_Friction_Force# = AIR_FRICTION_CONSTANT# * Velocity#^2.0
Velocity# = Velocity# - (Air_Friction_Force# * Time_Delta_Sec#)
; If the entity is colliding with the level this frame, or still sticking to the level from it's last
; collision, then apply ground friction.
If (Time_Now - Last_Collide_Time) <= Player_Friction_Stick_Time
; Compute ground friction. Ground friction is not dependent on the speed of the entity.
Velocity# = Velocity# - (GROUND_FRICTION_CONSTANT# * Time_Delta_Sec#)
EndIf
; Make sure the entity's velocity doesn't go below 0.
; It is impossible to have a negative velocity in physics and "bad things" happen if you try to.
If (Velocity# < 0) Then Velocity# = 0
; Convert the entity's velocity and direction back into a motion vector.
Vx# = Direction_X# * Velocity#
Vy# = Direction_Y# * Velocity#
Vz# = Direction_Z# * Velocity#
; If the entity collided with the level, make it bounce.
If Entity_Hit > 0
; Calculate bounce:
; Get the normal of the surface which the entity collided with.
Nx# = CollisionNX(ballpos, 1)
Ny# = CollisionNY(ballpos, 1)
Nz# = CollisionNZ(ballpos, 1)
; Multiply the entity's motion vector by the surface normal.
; This gives us the part of the vector which points in the same direction as the surface the object collided with.
Elasticity# = 0.5
Vx# = Vx# - Vx#*Abs(Nx#)*(1.0 - Elasticity#)
Vy# = Vy# - Vy#*Abs(Ny#)*(1.0 - Elasticity#)
Vz# = Vz# - Vz#*Abs(Nz#)*(1.0 - Elasticity#)
; Compute the dot product of the entity's motion vector and the normal of the surface collided with.
VdotN# = Vx#*Nx# + Vy#*Ny# + Vz#*Nz#
; Calculate the normal force.
NFx# = -2.0 * Nx# * VdotN#
NFy# = -2.0 * Ny# * VdotN#
NFz# = -2.0 * Nz# * VdotN#
; Add the normal force to the direction vector.
Vx# = Vx# + NFx#
Vy# = Vy# + NFy#
Vz# = Vz# + NFz#
; Do not allow the entity to move vertically.
; If Vy# > 0 Then Vy# = 0
EndIf
EndIf
; Apply directional thrust:
; If the entity collided with the level, apply directional thrust.
;If Entity_Hit > 0
; Take thrust in object space, and translates it to an XYZ vector in world space.
;TFormVector 0, 0, Thrust#, ballpos, 0
; Add any thrust being applied this frame.
; There's a very good reason why this is done AFTER the friction is calculated.
; It involves inequalities in force cause by variable framerates.
Vx# = Vx# + (Thrust_X# * Time_Delta_Sec#)
Vz# = Vz# + (Thrust_Z# * Time_Delta_Sec#)
;EndIf
; Apply gravity:
Vy# = Vy# - (GRAVITY# * Time_Delta_Sec#)
; Move and rotate the entity:
; We rotate the entity by the actual distance moved and not by the velocity because if we rotate according
; to the velocity then the entity will roll when it's up against a wall and not moving.
OldX# = NewX#
OldZ# = NewZ#
TranslateEntity ballpos, Vx#*Time_Delta_Sec#, Vy#*Time_Delta_Sec#, Vz#*Time_Delta_Sec#, True
NewX# = EntityX#(ballpos, True)
NewZ# = EntityZ#(ballpos, True)
Mx# = (NewX# - OldX#)
Mz# = (NewZ# - OldZ#)
; Rotate the entity the right amount for it's radius and the distance it has moved along the X and Z axis.
; This is kinda a hack and only designed for rolling on planes but you won't notice the diffrence.
XAngleAdjust# = (Mx# / BallRadius#) * (180.0/Pi)
ZAngleAdjust# = (Mz# / BallRadius#) * (180.0/Pi)
TurnEntity ball,ZAngleAdjust#,0,-XAngleAdjust#,True
Return