Different type of buttons and rotate

Graphics 800,600,16,2

SetBuffer BackBuffer()
Color 255,0,255
Plot 0,0
LockBuffer()
Global mask_color%=ReadPixelFast(0,0)
UnlockBuffer()

button_image=CreateImage(200,50)
SetBuffer ImageBuffer(button_image)
Color Rand(0,255),Rand(0,255),150
Rect 0,0,200,50
Color 255,0,255
Oval 10,10,180,30

SetBuffer BackBuffer()

While KeyHit(1)=0
Cls

x=200
y=200
DrawImage button_image,x,y

overlapping=0
If MouseX()>x And MouseX()<x+200 And MouseY()>y And MouseY()<y+50
SetBuffer ImageBuffer(button_image)
LockBuffer()
If ReadPixelFast(MouseX()-x,MouseY()-y)<>mask_color Then overlapping=1
UnlockBuffer()
SetBuffer BackBuffer()
EndIf

Color 255,255,255
If overlapping=1 Then Text 0,0,"overlapping"

Flip
Wend


If i rotate the image with the RotateImage() -function, how can i check overlapping then?

You could use the Command "ImagesCollide" to check if they are overlapping. You only have to use a little dummy image under the mousepointer, something like
mous=createimage(1,1)
MaskImage mous,255,255,255

To allow a button with a sensitive and an unsensitive zone you may need to use 2 button images layers, where one of them is holding the mask information only.

Please note: rotating an image may be VERY slow.
Here's an example. You'll also notice the shape of the button degenerates increasingly:

Graphics 800,600,16,2

SetBuffer BackBuffer()
Color 255,0,255
Plot 0,0
LockBuffer()
Global mask_color%=ReadPixelFast(0,0)
UnlockBuffer()

TFormFilter 0
button_image=CreateImage(200,50)
SetBuffer ImageBuffer(button_image)
Color Rand(0,255),Rand(0,255),150
Rect 0,0,200,50
Color 255,0,255
Oval 10,10,180,30

button_image_masked=CopyImage(button_image)
MaskImage button_image_masked, 255,0,255

mous=CreateImage(1,1)
MaskImage mous, 255,255,255


SetBuffer BackBuffer()

While KeyHit(1)=0
 Cls
 a=(a +1) Mod 360
 RotateImage button_image_masked, a
 RotateImage button_image, a
 x=200
 y=200
 mx=MouseX()
 my=MouseY()
 DrawImage button_image_masked,x,y
 DrawImage button_image,x,y
 DrawImage mous,mx,my
 overlapping=0
 If ImagesCollide(mous,mx,my,0,button_image_masked,x,y,0)
  overlapping=1
 EndIf

 Color 255,255,255
 If overlapping=1 Then Text 0,0,"overlapping"
 VWait: Flip 0
Wend

Here's the same code, only using a copy of the original button image for every rotation. This will not only prevent degeneration by stepwise rotation, it's also a lot faster for some reason:

Graphics 800,600,16,2

SetBuffer BackBuffer()
Color 255,0,255
Plot 0,0
LockBuffer()
Global mask_color%=ReadPixelFast(0,0)
UnlockBuffer()

TFormFilter 0
button_image=CreateImage(200,50)
SetBuffer ImageBuffer(button_image)
Color Rand(0,255),Rand(0,255),150
Rect 0,0,200,50
Color 255,0,255
Oval 10,10,180,30
button_image_bk=CopyImage(button_image)

button_image_masked=CopyImage(button_image)
MaskImage button_image_masked, 255,0,255
button_image_masked_bk=CopyImage(button_image_masked)

mous=CreateImage(1,1)
MaskImage mous, 255,255,255


SetBuffer BackBuffer()

While KeyHit(1)=0
 Cls
 a=(a +1) Mod 360
 FreeImage button_image_masked
 button_image_masked=CopyImage(button_image_masked_bk)
 FreeImage button_image
 button_image=CopyImage(button_image_bk)

 RotateImage button_image_masked, a
 RotateImage button_image, a
 x=200
 y=200
 mx=MouseX()
 my=MouseY()
 DrawImage button_image_masked,x,y
 DrawImage button_image,x,y
 DrawImage mous,mx,my
 overlapping=0
 If ImagesCollide(mous,mx,my,0,button_image_masked,x,y,0)
  overlapping=1
 EndIf

 Color 255,255,255
 If overlapping=1 Then Text 0,0,"overlapping"
 VWait: Flip 0
Wend

To get a bearable speed I'd suggest to stay away from realtime rotation and use precalculate rotated images instead. Or, much more elegant and ultrafast: use 3D quads and sprites to simulate 2D images. This way you can roatae, scale etc. thousands of "images" each frame. Tho it may tak some time to become familar with this "2D made of 3D" stuff.

If you choose the 2d-in-3d route, you may well want to consider BlitzMax, as all that is done for you =]