CUDA questions

Hi all,

I've been thinking about doing something with nvidia's CUDA architecture lately, but don't really know enough about it to decide whether it's worthwhile...

In particular:

* How many 'threads' is a 8800GTX capable of running.

* Does each thread get its own 'program counter' or is the PC 'shared'. Or do some of the threads have unique PC's and some don't?

* If the PC is in fact shared, does that imply that there's no branching *at all*, eg: this code...

If blah
Do x
Do y
EndIf

...will take the same amount of time to execute regardless of whether or not blah is true or false?

Thanks in advance for any clues!

I have heard CUDA is not working on Vista yet. It still seems pretty beta.

You might ask this on NVidia's public developer forums. They are actually more active than the private registered developer forums:
http://developer.nvidia.com/forums/index.php

Shader Model 4.0 cards are good at dynamic branching in the shader pipeline, so I am guessing the same is true of general calculations.

As Leadwerks said CUDA is in kind of a Beta stage but i'm quite confident that we'll see some nice stuff with the GT200 beeing released later this year. But beside of this if you're going after it what do you do on ATI/Intel cards then? What do you do on weaker cards generally? Due to these limitations and that it isn't proofen technology yet, i wouldn't invest my time into it at this point and come back later when it is.

Did you read this:

http://en.wikipedia.org/wiki/CUDA

Seems to give some decent info.

Hi,

> I have heard CUDA is not working on Vista yet.

That's OK, either am I...

> It still seems pretty beta.

What, cuda or Vista?

Sorry, in an unusually jocular mood today...

I checked out the nvidia forums and their pdf reference stuff, but it's all pretty hazy to me still.

The wikipedia entry is a little more to the point - thanks for that.

As for what I'm actually doing, it's just a little side project (raytracing - meee too!) so I'm not expecting anything commerical/useful to come out of it and don't really mind how I get it done.

I have an 8800GTX.

Send me Blitz3D v2 and I will test it for you.

@Mark
You have time for side projects? I'm surprised... :O)

I have SLI 8800GTX puki. Get in line. I get it first. :)

http://www.tomshardware.com/2008/04/14/nvidia_gpu_physics_engine_up_and_running_almost_/

CUDA-related news.

Forget about Vista, I believe CUDA is for Nvidia GPU's only which limits it quite a lot for the time being.

8800 GTX has 16 Multiprocessors. Each is capable to process 32 threads of a warp in for clock cycles.

On the IF:
The compiler will replace branches with prediction (the gpu is capable of branch prediction) if the number of instructions controled by the IF is smaller than a given threshold. There are different thresholds depending on how divergent the possible warps are.
If it does not replace it, it will not jump into a "false" scope of an IF, but branch execution takes some time (the programming guide does not give any clock cycle information on this for IF thought)

Will have to check back with some other papers to say something specific on the PC. Theoretically they would need their own PC as they can be pushed from one to another stream processor on the same Multiprocessor but thats a pure theoretical consideration.

Programming on CUDA is a lot different than on regular C and its very important to follow the advices from the CUDA Programming Guide 1.1 (would highly suggest reading it, it will answer many questions you will find while trying to tinker with it) on how to access memory correctly etc if you want to use the GPU usefully. Because if you don't do it, your GPU will be idle a lot and in the end a P3 will outperform your 8800GTX easily in that case. (a P3 when using MMX and SSE will leave it in the dust as the vectorization of that is already very powerfull and gives speedup of 3-4 compared to regular code)

I've had the chance to attend a CUDA talk of an NVIDIA employee today at our university so I was able to get some more precise information:

IF:
if a branching statement is reached which does not get replaced with prediction, the warp will get split into 2 warps (divergent warps) which both will executed. At the end of the execution of both warps they are will be combined again into one warp and the result of the "wanted" warp will be used.

On the first part:

Each Shader Core consists of 2 SMs, each SM of 8 SPs

Those SPs have a pipeline delay of 6 clock cycles, this means that each thread will have a delay of 6 clock cycles between 2 following commands. For that reason each SP needs at lest 6 Threads to work on to at least have a chance to be busy all the time.

To compensate pipeline delay you need at least 192 threads
to compensate memory fetch delay thought you need at least 768 threads


blocks always should have a multiply of 32 threads assigned as each warp has a size of 32.
Better and more optimal is 64 thought, due to priority handling

As well, with the 8800GTX (G80) you have the not current CUDA model (1.0), the newer ones have 1.1
the only real difference is that the G80 does not support atomic operations. (__syncronizethread() exists on all, so thats no problem)

Hi,

Thanks for that, clears up quite a few things.

Doesn't sound like cuda is quite right for my needs, but still sounds hack-worthy.

That is very strange, because SM 3.0 cards behave this way in the shader pipeline, but SM 4.0 cards only execute one branch.

If the branch is short, that is what happens (for very divergent warps, the threshold is 7 ops, otherwise its 4). Even thought it will not ignore the other. It just fills the other warp with NOP.

Might be that with pure shaders it does actually branch (don't know, avoid it as I don't want to see regular cards die down. GPUs are better at brute force streaming than jumping) but for the purpose of CUDA it would be catastrophal.

As well you don't really need it within CUDA that much as in Shaders for example. You do pure math calculations and you first would have me to show the mathematical algorithm that has an IF in ... there are cases, but those are more or less all covered by the prediction (which is an evolved version of previous shader generations prediction).

mark: what did you want to acomplish with CUDA?

> mark: what did you want to acomplish with CUDA?

Raytracing.

Unfortunately, it doesn't look like it's a good fit for CUDA.

This baby looks the biz though: http://www.tilera.com/index.php

Raytracing in Blitz3D 2? Not sure we need that.

>Raytracing in Blitz3D 2? Not sure we need that.

Why not? We need it, but now we are speculating. Mark is working on his own private procject where he wanted to have raytracing, nothing to do with blitz3d 2 ;)

Hmm, I think raytracing can actually be done with CUDA. (there are implementations already, think one even opensource, where the CUDA is as fast as a dual core CPU or the like. And that althought the technology has only been available for a year and most didn't use it as on windows you had to decide between cuda or gaming driver till 169.21 drivers ie very recently). Problem is that the mathematical and especially parallel numerical research done in the past based on totally different programming paradigms than ultra parallel scalar operation usage.
Thats the reason why the PS3 might need forever to get to the same level as the X360, the Cell suffers from the same thing, its just a totally new way of programming.
The parallel numerical computation research group at ETH actually just started in researching the CUDA capabilities, CELL researches were started about a year ago, so an extremely young research and usage sector.

But especially due to the Cell I believe that the paradigms needed for this kind of processing unit will appear and that more the sooner than the later as they are important to IBM after all.

@josh: I would be more than happy if ATI would write a CUDA driver just to point that out even thought I'll never again get one. But for support of more intelligent agent behavior for example it would be great to have the possibility to use it in next generation games, because NNs are something that is quite easily doable due to the way the data in an NN is stored and how CUDA or better the GPU memory bar and shared memory work.

I've seen a few papers on raytrcing on the gpu/simd etc, but it was all pretty ugly stuff for not all that much benefit. One bunch of dudes even built their own FPGA which is an intriguing thought!

The technique I'm using is 'inspired' by Carmack's mention of spare voxel octrees, and the algorithm branches very unpredictably and really depends on having the ability to halt or 'early out' at an unpredictable time. Recursion woould also be nice (though not 100% necessary) as you'd be able to do reflection/refraction too, although this isn't the point of the algorithm. All in all, it just doesn't seem that cuda friendly.

The Tilera thing is interesting, but almost seems *too* complete - each core has its own MMU etc and can even run its own OS! What I'd really like for this is something with a simple, low-transistor count architecture (eg: MIPS?) which can be multicored to the max but is still a 'proper' CPU.

I don't actually think there will be a 'winner' in the CPU vs GPU wars - having the flexibility of a PC per core is sometimes good, having the grunt to do a gazillion things at once is sometimes good. Depends on the job/algorithm, and I suspect both CPU's and GPU's will just continue to grow more and more cores.

And no, this is nothing to do with Blitz...just a fun thang.

I don't think its even the intention to have winner in the CPU vs GPU "war". GPU makers try to offload the math centred stuff to that chip which is definitely better at it, which is clearly the GPU due to its high parallelism and the focus on just doing this and nothing else.
The CPU on the other hands has its benefits as well like Out of order execution and sophisticated branch prediction and the like.

My guess thought is that the next generation of NVIDIA chipsets with the next iteration of CUDA will get a few more interesting capabilities, at least thats how it sounded yesterday but he wasn't allowed (logically) to give any information on what exactly and especially when (thought thats predictable. Q3 to Q4 this year at latest, my guess, as the G80-G90 starts to wear off and the HD4000 are approaching the scene in the forseable future). But it sounds like they added and are adding a few nice new tricks to the ShaderCores and SMs.

For what you want to achieve CUDA actually could be used as well depending on how long the stuff is because you could use cuda to achive part results which can be highly parallelized, then do some work or branching with that on CPU and then work on GPU again. Each of that GPU sections would be its own kernel.
Requirement would be a large dataset to operate on as the initialize and setup (data copies to and from gpu) would otherwise cost more than you get from using the GPU.