Creating Track Map

We need to create a map in 2d displaying position of a car on the track in 3d world. Is it possible to add a circular mask to the displayed map image.

Yes, first draw the map onto a temp. image.
This image needs to be masked. Then draw the circle mask on it. The circle map is another transparent image: the circle should be transparent and the rest should be filled with the transparent color of the first image.
The temp. image and the circle mask image should have different transparent colors and the transparent color of the temp. image should not appear in the tiles of the map:

	;setup graphics
	Graphics 800, 600, 0, 2
	SetBuffer BackBuffer()
	
	;create mask image (black circle on purple background)
	mask = CreateImage(100, 100)
	SetBuffer ImageBuffer(mask)
	Color 255, 255, 0
	Rect 0, 0, 100, 100
	Color 0, 0, 0
	Oval 0, 0, 100, 100
	
	;create temp image (purple=transparent)
	temp = CreateImage(100, 100)
	MaskImage temp, 255, 255, 0
	MidHandle temp
	
	;main loop
	Repeat
	
		;draw to temp image
		SetBuffer ImageBuffer(temp)
		Cls
		
		;draw random text
		Color 255, 255, 255
		For i = 0 To 50
			p$ = ""
			For t = 1 To 6
				p$ = p$ + Chr$(Rand(65, 65+26))
			Next
			Text Rand(100), Rand(100), p$, 1, 1
		Next
		
		;draw circle mask
		DrawImage mask, 0, 0
		
		;draw on backbuffer
		SetBuffer BackBuffer()
		
		;draw background lines
		Cls
		Color 0, 0, 255
		For i = 0 To 800 Step 20
		Line i, 0, i, 600
		Line 0, i, 800, i
		Next
		
		;draw temp image, leaving out purple parts
		DrawImage temp, 400 + Sin(time) * 200, 300
		time = time + 1
		
		Flip
		
	Until KeyHit(1)
	
	End

What bram32 says is good, but it is usually preferable to only use 3d commands in 3d mode as 2d graphics are much slower on most cards when used in combination with 3d graphics. In this case you would almost do the same thing but use 3d sprites in much the same manner as what bram32 suggests.

If you want to use only 3d commands, you could consider using a 2nd camera to record the track instead of using 2d map tiles. The round shape can be a round mesh that is textured with the map:

;-----------------------------------------------------------------------------------------------------
;											setup graphics
;-----------------------------------------------------------------------------------------------------

	Graphics3D 800, 600, 0, 2
	SetBuffer BackBuffer()
	
;-----------------------------------------------------------------------------------------------------
;											setup cameras
;-----------------------------------------------------------------------------------------------------
	;setup 1st camera
	camera = CreateCamera()
	MoveEntity camera, 0, 10, -10
	
	;setup 2nd camera
	camera2 = CreateCamera(camera)
	MoveEntity camera2, 0, 10, 0
	RotateEntity camera2, 90, 0, 0
	CameraProjMode camera2, 2
	CameraZoom camera2, 0.01
	
;-----------------------------------------------------------------------------------------------------
;											create round shape
;-----------------------------------------------------------------------------------------------------
	
	;create texture for map
	tex = CreateTexture(512, 512)
	
	;create round flat mesh
	mesh = CreateMesh()
	surf = CreateSurface(mesh)
	
	;create vertices
	AddVertex(surf, 0, 0, 0, 0.5, 0.5)
	For i = 0 To 36
	
		x# = Cos(i * 10)
		y# = Sin(i * 10)
	
		AddVertex(surf, x, y, 0, x / 2 + 0.5, y / 2 + 0.5)
		
	Next
	
	;create triangles
	For i = 0 To 36
	
		i2 = (i + 1) Mod 37 + 1
		AddTriangle surf, 0, i + 1, i2
		
	Next
	
	;flip mesh
	FlipMesh mesh
	
	;position mesh in upperleft corner of 1st camera
	PositionEntity mesh, -7.5, 15, 0
	;link to camera
	EntityParent mesh, camera
	;make full-bright
	EntityFX mesh, 1
	;scale mesh 2x
	ScaleEntity mesh, 2, 2, 2

;-----------------------------------------------------------------------------------------------------
;												create floor
;-----------------------------------------------------------------------------------------------------
	
	;create red dot texture 
	gtex = CreateTexture(32, 32)
	SetBuffer TextureBuffer(gtex)
	Color 255, 255, 0
	Rect 0, 0, 32, 32
	Color 255, 0, 0
	Oval 0, 0, 16, 16
	SetBuffer BackBuffer()
	ScaleTexture gtex, 50, 50
	
	;create ground plane
	plane = CreatePlane()
	EntityTexture plane, gtex
	
	
;-----------------------------------------------------------------------------------------------------
;												main loop
;-----------------------------------------------------------------------------------------------------

	Repeat
	
		;cursor keys to walk around
		If KeyDown(203) Then TurnEntity camera, 0, 1, 0
		If KeyDown(205) Then TurnEntity camera, 0, -1, 0
		If KeyDown(200) Then MoveEntity camera, 0, 0, +1
		If KeyDown(208) Then MoveEntity camera, 0, 0, -1
		
		;render 2nd camera onto texture "tex"
		CameraProjMode camera, 0
		CameraProjMode camera2, 2
		RenderWorld
		CopyRect 0, 0, GraphicsWidth(), GraphicsHeight(), 0, 0, BackBuffer(), TextureBuffer(tex)
		;apply "tex" to round mesh
		EntityTexture mesh, tex
		
		;render 1st camera
		CameraProjMode camera, 1
		CameraProjMode camera2, 0
		RenderWorld
		
		Flip
		
	Until KeyHit(1)
	
	End