Big massively long exotic source file

Well, it's been disputed once that people should split their big source files into smaller bits of monolithic modules to then include for the compile. But how efficient is it in a development of say a complex multi-linked intertwined system?

It's more a debate really that I wish to throw at some people here, as a lot of people will go on and rant about how inefficient it is to code large scale source modules. How exhausting it must be to scroll around the code and to actually find the places that need some tweaking, or to know where to insert new functions and link them inside the network of calls and schedulings.

Personally, I found it really well worth the try. I don't think there should be a general rule about big modules. I'm not the person that aims at making big monolythic modules, but sometimes it helps in the development cycle. For example, in the tabs where you've got your global variables, or your types, function names; they are all there, while you are developping. You know, when you are in this module, you don't need to look anywhere else, as everything is there. So, in that regards, performing massive searches or changes is much easier. All that is needed is to focus on that big singular module. debugging is simple, as most bug can be easily traced within the module. Development is straight forward, no juggling around source files to figure out where to add the functions. Re-structuring parts of the code gets relatively easy with a bunch of cut/copy/paste here and there. No biggie renaming variables/types/functions. Versionning of that module is just as easy, no questions asked.

Finally, integrating the BMS (Big Monolythic Sourcefile) into your game or project and testing updates is much easier, when you use say for example for your sprite engine. I'm all for monolythic modules for various engines or things like post process effect system, procedural texture builder, etc... I don't think it has anything to do with 3rd generation language coding, object oriented or what. Some people like to have lots of tabs in their sources, or large intro texts, many blank lines across the code, etc... I like monolythic efficient, tight modules.

I used to use one big source file. Now I don't. I've seen the light.

I use TortoiseSVN for incremental backups throughout development. Subversion folders have icons which instantly show you which folders contain modified files, and which files have been modified. If I change a code file and break something, I can compare the edited version with the most recent backup, side by side, and its very easy to see every change I've made to the code.

It is much MUCH easier to split code up into smaller files. What is the point of having 20,000 lines of code in the IDE when you're only working on a couple of functions?

All of my code is split into classes - one per file. My main code file is only about 20 lines long and other than that, I don't have any code that exists outside of any classes.

Keeping everything modular means that I can always find the bit of code I need to work on, and that I can find it quickly. It also means that when I'm done working on a class, I can close it and forget about it, safe in the knowledge that it will all make sense when/if I need to go back to it later.

If you want to write well-structured and easily maintainable code, you absolutely need to split it up. If you're a masochist then you'll likely get a lot of benefit from cramming everything into one massive file.

splitting-up & code-folding ftw!

Having a mammoth source at your nose is fun for an afternoon, but soon enough you'll find yourself killing your mousewheel.

When stuff works, put it away into another file.. why look at it ad infinitum when it works? When it works, it works, no need to see it working anymore.

Or would there be a point to have - in case of BMax - the whole lot of BRL modules in one long sourcefile together with your project? I guess not..

My current project has the most source files ever. Not because it's complex but I finally caved and moved from about 5-6 files to about 10 because when I wrote fairway, even 6 files wasn't enough to prevent some of them from becoming massive and hard to find stuff in.

What about code reuse? If everything is in one large file, code duplication seems your only option when you move on to the next project - and I can't think of anything positive to say about code duplication.

The problem with code reuse is that it's hard to say when such code is done. I think I'm not alone when I say that the second attempt of a certain function looks usually cleaner than the first attempt. As project designs grow, the fundamental functions often grow too. Some ppl may come up with the notion that OO/inheriting lets functionality grow in a structured way, however not everyone likes the complexity of OO, and it's not available for all Blitzes. Besides, *if* one could simply rewrite some classes here and there we'd be using version 20+ of tools made in the 90's by now. But no, after a few versions usually a complete rewrite from the bottom-up of something comes up.

But do you guys think it is like sacred, this thing with large source files over here in the forums? I remember once I had this problem where the Blitz3d IDE gave me problems with a 400K source, and I got lots of replies saying that; "Buddy, you should split that source up, and get organized", type of comment. I tought that was rather rude somehow, as my code is really well structured, functionnal, and so on. reason at that time I couldn't split the source was that I had to restructure the workings as I was developping, and change a lot of the labelling, as I went along. Sometimes your source can't be split easily, as a lot of the types and globals are used all around the source, and it just makes sense as a whole unit. I prefer to have everything in relation to a type in the same source file.

I am working with a 500 kb source code with no problems, using mostly includes.

_33 is absolutely right!

I think I'm going to take my current project of 330+ .bmx files and matching number of .cpp/.h files, and convert it into one huge bmx and one huge .cpp file.

That way, when I need to add a new type, I won't have to go through that terribly long process of creating a new .bmx file, and rather just add it to the bottom of the 80,000 line single bmx file :-)

Fantastic!

:o)

@Brucey: Point well said, it also is a lot easier to find this and that with separate files (mainly pertaining to large projects, try having 1000kb+ code in one file).

Why most posts from the BlitzMax community just mystifies me... Brucey, I'm sure you're quite productive the way you're working.

I use many source files and try and make them as small and as categorised as i can.
Its also easier to find and compare code in separate tabs.

However if posting a program on a website (like a forum) its easier to both post and pick up in a big chunk. (as _33 said its rude to moan like that. If people want to split it up then they can blimmin' well do it themselves :).

I split functions into thier respective source files, networking goes in networking, graphics goes in graphics, you get the idea.

The main thing that makes this easier is that you dont have to scroll through thousands of lines of code you just jump to the relevant souce file and go from there.

http://www.blitzbasic.com/Community/posts.php?topic=76369

I used to work on just a few *large* source files until recently when in the design process we decided to split things up like you would see in C++.

Productivity went through the roof, more than one person could work on the project at the same time as long as it wasn't the same source file etc....

In short, One BMX file for each class, and any subclasses. One BMX for the main file. INCLUDE the classes.

It's true that separate files are better in a team situation. In a big business you'd have a file for each class. I prefer to have files for related things like Game Screen/HUD, Player/Profile, Particles, Misc Functions, Game Types, Other Screens etc. Too many files and I find it hard to switch between then all quickly in the basic IDE of course. Probably I need to upgrade the IDE...

Isn't this IDE dependent? I know Blide keeps a 'shortcut' list of globals, functions, types in a tab that spans the whole project no matter how many source files.



Ok, but if you're module is 100k+ lines long that doesn't really narrow things down does it? I'd rather focus on a few lines of code in a type...



again...and I'm speaking from experience. When developing in blitzplus I had 'monolithic' files for my 1MB+ source code and debugging was anything far from simple. Now i can trace the problem down to a single source file that spans maybe 50-200 lines of code vs thousands.




Still don't see a problem here. example: I need to add some special functionality for my player. I'm going to go look in my player source file that is pretty independent of the whole game engine. It's an organization issue.



whenever I find myself copying/pasting code it's a red flag and I immediately look for a way to refactor the code in a more reusable method or function. For cutting, I have to cut, click a tab to another source file paste vs your cut...scroll...paste.


no biggie? I've had several nightmarish experiences renaming variables in 'monolithic' structures because encapsulation tends to be minimal.


Um I'll go ahead and ask: Why would it be any different vs several source files in a single folder that is called '[module name here]' and just versioning the main include file?

I like to keep it to 1 type per file. almost 0 globals or global functions. On rare occassion there's 2 types and the 2nd is more of a data type used by the first. I never had issue with finding what i need. Need to look at some scenegraph code? go to scenegraph.bmx. bullets? bullet.bmx. etc...
As long as you name it something that makes sense then I see none of the advantages the OP has described with 1 large file.

And with Blide searching the whole project isn't any more difficult than searching a single 100k+ line monolithic source file.

I agree. Generally i try not to use global variables, as they are declared in another .bb file. When an error pops up or code doesn't work right, it's alot harder tracking down global variables.

10 source files is hardly "the most source files ever". Unless you mean most source files ever for you.

My current game has 16 and its probably 10% finished. I'm only ever working on one or two at once and there rest are finished and I might never need to go near them again.

My current project at work (PHP) as 225 files and counting, I wouldn't know where to begin if it was all lumped in one file.

Holy smoke, 225 .. with that number it's not even about having all that code in 1 file but about what directory structure one needs..

What kind of project is that anyway?

I think breaking your code up is the only way to go once you reach a certain size.

correct. I just find too many files a pain.

There is one possible advantage of large source files that's been overlooked here - although whether it impacts on Blitzmax would be a matter for experiment.

The following from the SQLite site:

Please note that they don't use this file for actual development. :-)

Would that work for every language though?..and what are these extra optimizations when they are at home? Surely this depends on the compiler?

Yep, directory structure is very important, as is naming convention.


In-house shiny spanky CMS :)

Incidentally, that number was just for the PHP files, not Javascript (774) or CSS (255) although those numbers include YUI

Depends on how you define and measure efficiency, and which tools are at your disposal. For anyone working in the BlitzIDE, I don't think it makes any difference at all. One way or the other you're going to get into a whole heap of trouble after your code reaches a certain point of complexity and scale.

For more complex IDEs (IntelliJ and Eclipse spring to mind) which have functional refactoring tools, splitting your source up into units (classes), in separate files is absolutely crucial to maintaining a constant forward momentum.

There are two ways to get around that. 1) Get a better editor. 2) Be more consistent and intuitive in your naming conventions. Ideally you would do both.

No. This is a weakness of the BlitzIDE. Any slightly more advanced IDE (X-Develop, Visual Studio + ReSharper, Eclipse) will let you easily navigate, search and refactor entire projects, regardless of how many files, or even computers the components are distributed across.

Surely the larger the module, the more difficult it is to locate a bug? By splitting up a large file into units, and performing vigorous unit tests, debugging is a breeze - especially since you can determine exactly in which unit the bug occurs, and said unit is rarely more lines of code than you can reasonably grasp at once.

You don't need to "juggle about source files". You simply place the function/method where it logically belongs, that is in the class it manipulates, conforming to principles like high coherency and separation of concerns.

Ever tried it? I find that a good 70% of the bugs in my code is the result of a copy/paste gone wrong (where one variable wasn't renamed or some other trivial, yet crippling mistake). Also without refactoring tools, renaming Variables, Types and Functions is borderline suicidal, as a simple search and replace does not take scope into account which can cause disastrous/hilarious problems (depending on whether you're the one who has to fix it or not).

Sure. If you're the only person who's ever going to be working on it. If one or more people are working on it simultaneously, I wouldn't want to be the sucker tasked with merging several thousand lines of conflicting source code.

Why? A module is a module regardless of how many lines of source code it contains. Also maybe your sprite engine needs to do two slightly different things in two different games. Great. Now you have *2* monolithic source files to maintain, rather than just having to maintain the differences.

Well you're entitled to your opinion, but you're wrong. The central encapsulation aspect of the object oriented paradigm, addresses exactly this issue. A lot of things get a lot cleaner and easier to understand if split into logical objects that bahave in an expected manor.

Done many of those then?

And you where doing so well. You should of course import the classes.

And the type of application, and the target processor/platform.

Flameduck, really enjoyed reading your post! You have excellent judgement and of course I take note form the IDE concerns that you raised. Of course, I bumped into problems with Blitz3D IDE bugs on big source files. The response I had was to split my source module into smaller chunk of source files. Today, that same subproject is 2 files; a main function file, and a data file. I use another IDE than Blitz to maintain it.



Well, Flameduck, if you're asking me what are the biggest singular monolythic modules I ever made, they border 5 digits in number of lines, so it's nothing like 100K lines of code, but it is still to me a case of "big monolythic sourcefile".



I absolutely do not agree here. A general purpose sprite engine that does it all spritewise, that has it's own mini script language, and that is self contained on it's singular purpose of generatign sprites and managing them, shouldn't need constant tweaking to adapt to projects. What it should get are updates to it's features.

Hah! Well that's a first. :o>

Well no. I wasn't really asking about the size, so much as the amount. The more code you write, the more you tend to recognize patterns in the way you write code, and the more natural generalizing becomes.

Most certainly. In my current project, I think the largest source file (that I've written) is maybe 1500 lines of code. The only really large classes are those tasked with O-R mapping. Mapping the data from the object model used in the software, to the relational model used in the database, and vice versa.

However even with that small amount of code, the class is starting to turn into one of those monsters that are a nightmare to maintain, and is in desperate need of some refactoring. If only there was more time...

Ideologically I agree. In practice however that's not always feasible. There can be hundreds of practical, as well as technical reasons why this is not always possible (or even desirable). For instance a game developed for a mobile phone or similar platform, is going to have to be a lot tighter with resource management, than one written for a Proper Computer, where such tight resource budgets are undesirable, but you could still reuse perhaps 95% of the sprite library code.

I tend to end up very few but long source files... Which is also the main reason I really wish that the default IDE could to function collapsing. :-?

That way you can have both the convenience of having all your cod ein one place, without actually having the (temporarily) unnecessary parts clutter up the screen.

An IDE which would let you 'hide' and reshow functions or section of code would be useful. Or if you could tag sections of code in different ways, and request to only display certain types of code, and each code section can have multiple tags...