Ugh, that would suck. What determines video memory? Wouldn't the computer automatically switch to virtual memory?
No -- videocard uses very fast dedicated memory on the video adapter itself. The only exception to this are the integrated onboard videocards that don't have their own memory but use the *much slower* normal system RAM for everything -- those can expand their memory and claim more RAM. (While depending on the amount of contiguous free system RAM available to them, AND the configuration settings that may limit how much RAM is available for them)
From a technical point of view the integrated adapters are very much inferior than the dedicated cards: much slower memory interactions, underpowered GPU, and typically a few generations behind in supported features. You're going to get much better frame rates on pretty much any dedicated 3D accelerator vs. onboard graphics.
Do you know how big this video memory is? (Or is it variable...?) Is it a system thing or a BlitzMax thing?
It's a system thing. Unfortunately, Blitzmax has no built-in way to tell you how much videoram is present and/or available. It pretty much boils down to you doing the math, determinging that you use never use more than e.g. 187 MB or video RAM, add some reasonable allowance for videoRAM used by the operating system itself, and slap a '3D video card with 256MB memory'-line on the minimum system requirements of your game.
Blitz3D had a way of telling you how much available video RAM there was in DirectX. BlitzMax cannot, but technically that information should be accessible
somehow. To my knowledge there is no way to query the available video memory under OpenGL.
Anyway, there are things you can do to reduce the need for video memory: for example, don't load every single background/sprite all at once at the beginning of your game, but only load them just before you need them and unload them once you are done.
If you don't have enough memory to hold everything at once (as is likely the case for you here), then you can do things like unload your menu screen once you leave the menu, unload the level 1 sprites before you load the level 2 sprites, etc. Video RAM is a finite resource...