At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. I don't know how they convinced the users to do this and how they managed with the license but it was done.

What happened next was even worse.

There was no version control of course, the code being available on a shared disk on the local network of the company with the code copied over in multiple folders each having its own version, with no particular logic to it either, V1, V2, V2.3, V2a, V2_customer_name, V2_customer_name_fix, ...

After that, when there was a problem for a customer, the programmer went there to debug and modified the code on site. If the bug/problem was impacting other customers, we had to dispatch some guys for each customer to go copy/edit the code for all of them. But if the problem was minor, it was just modified there, and probably saved on the shared folder in some new folder.

What happened next was to be expected: there was no consensus on what was the final version, each customer having slightly different versions, with some still having bugs fixed years before for others.

Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later).

Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there).

Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes.

A few years later a preventable crisis was how revision control first became client-server.

Git is not a natural development at all. Obviously, it is a standard right now.

But I as a hobby coder at my teens I started out with FileZilla and copying over index2.php, index3.php, index_final.php, index_final_2.php and all of it worked well enough when at that point.

I took a little break from that hobby, and it still took me a lot of time to build intuition around Git when I started out professionally.

Obviously now that I have over a decade of professional experience I think it's a no brainer, but I don't think it's natural to understand it as it's suggested to you. I at least kind of had to go along with it and trust that it's good. It felt frustrating at first as many other things are. Git as a thing was FAR more difficult for me to understand in my 20s than PHP and how to do anything with it was for me at teens. Coding is raw logic, Git is about imagining teamwork at scale, which is a whole other thing.

The funny thing is that I feel like I was overall still more productive when I was at my teens building stuff solo with FileZilla and PHP, compared to a corporate environment now with all the processes and 1000s of engineers.

Git is still a baby as these things go. Software version control systems go back to the early 1960s and descended from manual processes used to manage documents in large organizations, blueprints, and such from the early 20th century. Don’t imagine the Manhattan Project functioned by just passing around bundles of paper! There were staffs of people whose job was to manage document control! And I don’t mean “control” in the sense of secrecy, but making sure everybody was on the the same revision at the same time.

And for a field so steeped in building its own tools and automating human effort in development it is indeed astonishing revision control took so long to be accepted.

Most standard tools we use today are not obvious to beginners nor would they be the first thing that beginners reach for.

But senior developers can understand the problems that they claim to address, and why they are important and common problems.

I'm 100% sure that once I left that the devs went back to remote server crash and burn FTP development...they couldn't be bothered with the "hassle" and unneeded headaches of git.

Have you considered introducing Mercurial or even Subversion?

While Git may be a kind of an industry 'standard', if you're starting from zero, some of its concepts may be a bit mind-bending for folks, and it has often been commented that Hg seems to have a more beginner-friendly interface.

And if branching isn't going to be used (a large strength of git/hg), then Subversion may have an even simpler mental model (svn of course does branching, but the others are more optimized for it).

If folks are doing FTP-push deployment, then moving to 'just' SVN-push (commit) deployment can be an improvement.

It only takes doing index73.php and index_old3.php for a few months and then eventually screwing up (unless you’re perfect) to realize how dumb putting numbers and _old4 at the end of names is. Then at that point, you naturally go look if there’s a better way.

That might be true now but it wasn’t always. Plenty of shops didn’t have any version control at all. And the build was whatever came off your local dev box. And “local dev box” wasn’t even a term…

It's a false dichotomy. Before git there were other version managements systems which would've fit your use case much better than git. Subversion is the easiest perhaps.

That said, if I was going to start out teaching someone coding today, version control would be timestamped .zip files.

Oh no, we're in a different time zone.

>We'll put the timezone in.

Oh no, it's daylight savings time.

Git is actually an unnatural development. Its UI is atrocious. And it worked like crap on Windows for forever.

Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.

Then Github won due to VC cash dumping and foisted Git on all of us. Source control was no longer fine.

Thankfully, there is now Jujutsu(jj) which works on top of the Git storage layer so you can forget about the craptastic UI of Git. Source control is now fine again.

SourceForge had a messy interface and frankly I hated it. Google Code was like frozen in time, like most Google projects. Microsoft’s thing — I don’t even remember the name — felt half assed and like they were going to abandon it… and they abandoned it. There were also others… none of which I even think were serious enough.

Also Git won because SVN and CVS were centralized VCSes and you needed a server. I can just make Git repos before I even know if a project is serious.

There were other distributed VCSes of course, like Hg, but they either cost $ or wasn’t backed by a major user… like Linux. I admittedly just waited this one out and chose Git because it was more popular.

SourceSafe did get abandoned... because your source wasn't actually safe in it.

Its replacement, Team Foundation Server got rebranded as Azure DevOps; but the product's still around, 19 years later.

Well, that’s in the eye of the beholder. Yes, I hate the ‘program command’ syntax (“git add”, “git commit” etc) but I just call git-add etc and for me those commands are pretty clear.

But I understand how git works. I imagine most people treat it as a black box and then its behavior probably is rather obscure. I don’t think it was intended for that use case.

I acknowledge that Git is really good on a technical level, but don't like the CLI experience regardless.

Instead, I much prefer to use tools like GitKraken (paid), SourceTree (free, limited platform support), Git Cola (free, lightweight, a bit basic) and know that most of my colleagues just use the Git integration in their IDE of choice.

Working that way, it's a decidedly pleasant experience, not unlike using visual merge tools like Meld, where you can also stage individual lines/code blocks and operate with branches and tags in a very visual and obvious manner.

That said, sometimes the support for Git LFS and things like submodules is all over the place and I've had cases where not adding node_modules or something like that to .gitignore has made the UI unresponsive with how much stuff initially shows up in the working copy, so sometimes you still have to drop down to the CLI and do a fix herer and there.

> Well, that’s in the eye of the beholder. […]

I would say it is not.

Usability testing is a thing, and products can get better and worse scores when trying to do the same task / reach the same result. I'm not aware of any published studies on the topic, but I would not be surprised if Git got lower scores than Hg on UI/UX, even though they can basically do the same thing (distributed development).

Given the GP trained people on multiple version control systems, including another distributed one (hg), and people mostly had a problem with Git, I would say Git is the problem.

Git itself is mediocre, and some other DVCS like Mercurial could have won out, although HgHub really doesn't have the same ring to it. The halo effect from the Linux kernel was also a factor. But the VC cash dump and Microsoft buyout came later, people used GitHub because it was better than the rest.

You can thank BitKeeper for all of this, and Andrew Tridgwell for forcing Linus Torvalds into creating git.

Gits internals can be confusing at first, but once you understand them there's really nothing you can't do. Being distributed by design also helps.

Plus it didn’t have a mechanism to send contributions. I think GitHub “won” because of its web-based PR system.

GitHub is to sourceforge what Facebook was to MySpace. MySpace was first but it was buggy as hell.

A more apt comparison would be comparing Facebook to a hypothetical social media site that when you click on a thumbnail of a user's image, you get a fullsize image of something like goatse...which thankfully doesn't exist(yet).

Lots of companies were doing Git and Mercurial "forges" at the time. Many of them were better than Github.

Everything was evolving nicely (including Git and Github) until Github used VC money to offer everything for free to blow everybody out in order to lock people into their product (for example--export of anything other than code is still hit or miss on Github).

At which point, everything in the source control space completely collapsed.

You might want to go review your history before accusing someone else of that.

Github took $100 million from VCs in 2012. It then took another $250 million from VCs in 2015. Who the hell else was even within an order of magnitude of that in the same timeframe? Nobody. (Gitlab took almost the same amounts but did so somewhere between 5-8 years later depending upon how you count).

Bitbucket got bought by Atlassian in 2012. Atlassian was bootstrapped until it took $60 million in VC in 2010 and had revenues (not profits) of about $100 million in that time frame. It had nowhere near the resources to be able to drop the equivalent of $350 million on Bitbucket between 2012-2015.

VCs didn't make git successful, developers did.

In any case,that doesn't really matter considering that git had big players adopting before GitHub got any sizeable investment. And I'm not just talking about Linux. Rails migrated towards it when GitHub was still in beta.

If you think SF+CVS is equivalent to GitHub+Git then you never used SF+CVS. Git won because of GitHub, specifically, not because generically it could be hosted on the Internet.

Atomic commits can also easily be moved around (since the repository is a tree of commits with the commits being leaves), and they also make merging simpler in many scenarios.

Larry McVoy was famously wont to engage in trolling upon the Linux Kernel Mailing List, whereupon he did boast of his BitKeeper, which possessed atomic changesets. Concurrently, he did deride Subversion for its lack of the same. Thus, a great drama did ensue, one which ultimately bestowed upon us the creation of git.

git has also succeeded as a DVCS where others have faltered, for various reasons. For there have been monotone, darcs and other such systems; yet, it is chiefly git that has endured.

Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.

It requires a level of trust and a level of performance, that I now believe doesn’t scale. Note that we had only a single junior developer, an intern, who wrote our bug tracking system but didn’t otherwise interact with the code base. The rest of the team consisted entirely of experienced senior developers, including management. I think at up to about 25 employees, the shortest working experience on the team (apart from the intern) might have been 10 years of shipping code.

But I really respect Rich for taking this approach which is why I referred to him by name.

Some elements are inevitably going to end up being de-prioritized, and pushed further into the future. Features that do end up having a lot of demand could remain a priority.

I don't think this is even a case of "ask for forgiveness, not permission" (assuming you do intend to actually work on w/e particular demands if they end up actually continuing to demand it), but a natural product of triage.

When my consultant contract finally came to an end in 2003 they were still using it.

Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow.

I left.

CVS was notorious for doing "text/binary" conversions (CR/LF line endings to CR and vice versa), sometimes inappropriately. More than once, it resulted in files where every other line was empty. I can very well see this happening several times, resulting in exponential growth of whitespaces.

But the seniors on the team pointed to CVS. Not the individual, not the "test" process, not the deploy checks.

Nobody noticed it slow cause everything ran on LAN. Only noticed the issue on 56k

The guy ate, slept, possibly got married while his finger was still accidentally holding down that key.

That's when I figured out what happened in 1998.

At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)

1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS.

2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master.

The list goes on!

Perhaps you should have proposed anything-but-TFS (git/mercurial/even svn - no need for distributed in a small company... https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs...) but internal.

Might be a relatively easy sell if you just setup a local migration of their data to repository with a decent web gui for viewing the commits and associating with tickets, given my experiences with TFS in terms of slowness, and crashes/data corruption.

Haven't followed that for a while, but it used to be the case that it was Mercurial who was principled/fanatic about never rewriting history, ever, while git happily let you mess up with hard deletes.

What censor is for is the case where something absolutely has to be removed from the repo, for legal or security. It allows you to do it in a clean standard way, even replacing it with, say, a key signed statement, without forcing regeneration of the whole repository or altering the tree in any way.

Just remove the contents while keeping the hash.

https://wiki.mercurial-scm.org/CensorPlan

I’d argue everything else becomes expensive quickly.

…but it is also imperative to remember any modification on checked out code is a branch, technically, regardless of the version control system used. This becomes important if your testing is expensive.

1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins.

2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month)

3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too

4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed

Any many more!

This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now,

Good times!

My favorite was at the company that was self hosting their code. The senior team lead wanted me to help him find a memory leak that plagued the product for months. Customers were told to restart the application every few weeks (this was a C++ application).

I sat down with the senior and looked at the code. I spotted the error.

I was like, "You know when you do new[] you need to use delete[]?" as all of his deletions were without [].

The look on his face was the best.

This seems like a pretty major lack of a specific piece of knowledge on the senior developers part, yes, but it seems like a much more unforgivable miss on the part of the code reviewers. Was the team stuck in a rut where only a single person (with coincidentally the same blind spot) was reviewing his code, or did multiple reviewers somehow miss this?

The fact that people study for exams has absolutely no correlation with how much they will remember or care for what they studied, none.

You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way

One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess.

I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow.

In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way.

But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in.

But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server.

Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere?

I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business.

I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.

That old project though was never fixed though, probably still running that way now.

For VB Sourcesafe was the go to tool if memory serves.

This is not a case of new vs old, rather incompetence vs competence.

After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.

I don't know if that made it better, I assume not much, VSS was really trash.

Well into the subversion era…

My attempt to introduce a "multissh" tool that automatically executed the same commands in each node at once was regarded with the highest suspicion and shot down. Shortly after I left, they had a multi-week outage caused by somebody fat-fingering the permissions on a network interface.

I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it

git simply didn't have the mind share jet, thus I went with what I knew

Whereas nearly every UNIX installation included version control systems for free (SCCS in AT&T UNIX SVR1-4, RCS in BSD UNIX or both) that worked exactly the same everywhere.

Some thought version control was an obvious improvement to make better, more bug-free software. Others had a fetish for doing things the hard way with little justification beyond emotional resistance.

He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it.

I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.

I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.

Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.

Enterprise and business users will wade through endless swamps of crap if the software provides enough value. This is a lesson in why "it must be perfect before we release it" is such nonsense - that just says the value your app provides is so low that users barely care and they'll abandon it at the first problem. If that's the case you're not providing anything worth paying for.

I was brought in when an old sheet cost them $10m on a bad trade because the yahoo finance end point it was hitting stopped responding and it just used the last value it had gotten - three months before.

- No one having any preconception of how you're /supposed to/ do things or whether you'd even be the type of person to know, so you just kinda figure it out yourself. You spend a lot of time on reading and learning skills. Version control? Wow, cool, what's git, let's try that! A new graphing library? Let's give that a shot, maybe it'll make things easier! You want XYZ? Let me go read about that for a day.

- No one having any idea what's even possible: being treated like a wizard for introducing the tiniest piece of automation or improvement that makes someone's day easier or doing something they never thought was possible. Lots of appreciation and excitement for showing and teaching people new things (... and I know this is somewhat selfish and ego driven, but who doesn't like being appreciated?)

- Similarly people having no idea how long those things should take which, tbh, can be a nightmare if you're not trusted and respected enough to be consulted but also great if people believe you when you say it's gonna take 3 months.

- Beyond the basics just being mostly kinda left alone to do your job however: no standups or tickets or the 30 other kinds of daily (micro)management that is probably necessary but ends up feeling tiresome and stifling at an individual level

- Not being part of software company "culture": no performance review driven development and promo packet madness, no weird rating and ranking systems, no OGPs or KPIs. No ladder. Your bosses think you did what was required of you, so then you're good, and if it's a good year you get a raise, and that's that. I do recognize that with a bad boss this can be a terrible and unfair spot to be in - but again, subjectively with a decent enough boss it felt like a lot less weight on my shoulders at the time.

- No hacker ninja pirate segway mini quadcopter you're the smartest people in the world and we're the best company to work for sort of b.s.

- Socializing with people who are good at and love to talk about stuff other than software

Reading over that, I'm thinking maybe I lucked out a lot and that wasn't most people's experience from that era. And there's some level of rose tinted glasses going on. And/or maybe my years in the rat race are starting to show :-)

Working directly for the users without any weird BA/PM/TA shit in between is glorious, both because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue), and in a place like that you can likely deploy it in the middle of the day and immediately improve their workflow.

It still amuses me that IT was located together with finance, because we did reports xD

A similar feeling on my end too :-) That might be it - trust sounds like a big part of it for me. Taking a chance on someone who you might eventually end up being good, rather than interviewing and evaluating them 7 ways till sunday. I understand the impulse. I wouldn't want to be a new engineer out of college today though - seems rough.

I did get paid less then than some new grads seem to be now so that might have been a factor in taking the pressure off.

> because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue)

Oh absolutely!

> It still amuses me that IT was located together with finance, because we did reports xD

It was communications for me, because the software tool we built was free to use on the web, and websites are communications, obviously :D

I once ran across a (c) program that had 26 variables, each one letter long, one for each letter of the alphabet. They were all global variables. And many of them were re-used for completely unrelated things.

As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices??

Honest question. How does our profession root out intrinsically obvious bad practices?

The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.

Holy smoke! That's actually the most creative solution (horrible, but creative) I've ever heard to fix a Heisenbug:

https://en.wikipedia.org/wiki/Heisenbug

You’d think this sort of thing would be a problem of the 90s or early 2000s, but I’d bet you there are any number of companies with similar situations today.

The funny thing about CVS and Subversion is that they were last version control systems that non-programmers could actually use. TortoiseCVS/TortoiseSVN were easy enough that with a lot of training you could get technical writers or artists to use it too. Valuable for game projects. With git, forget about it. It can't even handle binary files well.

Only Git had the brain damage to take the source control idea of "there are two areas: working or committed" and add the dumbass index/staging/stash areas that confuse the hell out of everything.

Hopefully they won't discover at some point that customers were in fact depending on those bugs. That's poison for the programmer's soul.

In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.

In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.

Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).

Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.

This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.

Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.

It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.

Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.

I presume there were other reasons for resigning, or you just like massive refactoring projects.

After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.

If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.

There are few efforts as truly thankless in many organizations as fixing things.

Oddly, the project was one of the best I've ever worked on.

Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.

Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.

I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.

[0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.

The fun/frustrating part was getting branching and "views" correct.

I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.

In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?

When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).

CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.

I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.

> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.

COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!

But we were using Subversion's lock system and had a proper CI/CD with Jenkins.

So you won, your project was worse!

The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.

IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).

I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).

I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.

I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)

The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo.

By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.

I don't know why the developers had chosen that method.

A lot of the worst sins happened on Oracle databases, mostly because they were the most popular at the time, and they were the only real DB cult, so the cottage industry of specializations that told other developers not to worry their pretty little heads was especially bad.

Now I just don’t involve him in anything unless forced.

I am somewhat surprised that a programmer who was unaware of arrays in Perl managed to have tests. But then again, he managed to implement their own version of arrays, maybe he came up with the concept of software testing by himself :-P

In a similar vein, an alternate explanation may just be that a lot of bioinformatics algorithms are described as string algorithms. It would be a pity if that was just taken too literally, but I've seen a lot of bioinformatics perl that works on strings in this way.

While trying to be charitable though, it's almost impossible to come up with a rationalization for the three character offsets string. Sometimes I think when people really go wrong is when they hear of some problem or are trying to be too clever and end up going beyond their abilities and stray into farce.

Never attribute to competence that which is adequately explained by incompetence.

My youngest worked in a furniture chain over the summer. And they got sent a big, heavy furniture set from the central warehouse, which the store actually didn't want. So, they sent it back. The problem was that the system didn't allow them to say: please, don't send this again. And the non-natural intelligence at the central base decided to send this set again. When my youngest started working - they were loading this set back for the seventh time.

Why 'loading'? Because no one could find a way in the program to send this ill-fated set back on the same truck that brought it. No one, except my youngest, that is. He managed to find the combination of keys and checkboxes that allowed them not to unload the unwanted set and ship it back on the same truck - and he immediately got a raise.

I suspect the set is still traveling. But now they only load-unload it at the central warehouse.

1) His libtgvoip code, previously used for audio calls, is the worst code I have ever had the displesaure of touching, it caused all kinds of issues ranging from instability to segfaults, and thankfully it was completely replaced by webrtc.

2) The android app is literally the smoothest and most responsive android app I've ever used.

Grishka, please stop being so salty. VoIP is hard to do right, but not impossibly so. VoIP being hard is still not an excuse for writing garbage C/C++ code (and I don't mean to offend personally here, but the code was really a mess, please stick to Java development on your own personal projects).

It was replaced by WebRTC that would sometimes just suddenly disconnect your calls, right.

> I managed to improve the situation after weeks of refactoring and testing

Did you submit any PRs with your improvements? I used to review and merge those, unlike the Android app devs.

2) It might provide the best UX in the entire universe, but that still doesn't justify having a two-megabyte Java file with over a hundred anonymous inner classes that gaslight you because it's often a `new FrameLayout(activity){` that overrides onLayout and does something completely different. So you see a FrameLayout, you assume it behaves like one, you add views to it, then it doesn't work the way you expect. You start questioning your sanity and only then do you realize that it's one of these stupid things.

Oh and did I mention that even just editing ChatActivity.java is an exercise in frustration? At the time, I had a 2012 MacBook Pro that worked fine for everything I did on it, except editing 2-megabyte Java sources in Android Studio. It would sometimes take a solid 10 seconds for keystrokes to register.

In other words, it might be the best Android app in the world, but it blows up in your face nearly every time you try to add a new feature.

And, honestly, these things are connected. It takes more needless work from the developer to maintain the user-visible pleasantness while adding new features. The code is much more fragile than it could've been which means bugs are much easier to introduce and much harder to diagnose and fix.

And I'm not asking for a complete rewrite. Just losslessly restructure the existing code, extract all those custom FrameLayouts and ViewGroups into separate classes, split those 2000-line if statements into separate methods, make some things (like ChatMessageCell) into sensible class hierarchies, introduce constants or enums to replace magic numbers, all that sort of stuff. This will not affect the UX, but it would make bugs harder to introduce and easier to deal with.

I'm generally not a fan of the developer-experience-focused approach to software development (Electron, React, and the web dev "trust me bro" attitude should not exist), but sensible code structure is where I draw the line.

I started working with Pavel Durov in 2011 on the VKontakte Android app, after winning several of his developer contests, and it all just went from there. I was the last one of the "old" team to transfer to Telegram. My main job was actually the libtgvoip[1] library and its integration into the Android app (the other platforms were done by their corresponding devs).

I got fired in 2019 because the calls weren't working satisfactorily for Pavel. He got furious when he was unable to call some journalist, cursed at me in front of the entire team and sent me a termination notice. In hindsight, I should've definitely not taken building a VoIP library from scratch alone upon myself. It's a massive undertaking. But at least I learned a lot, and many people say that my implementation performed noticeably better than the current WebRTC-based one ¯\_(ツ)_/¯

I'm now working on two fediverse projects at the same time — Smithereen[2], a VK-like ActivityPub server, and the Mastodon Android app[3].

[1] https://github.com/grishka/libtgvoip

[2] https://github.com/grishka/Smithereen

[3] https://github.com/mastodon/mastodon-android

Honestly does not sound like a very healthy person to work for anyway.

I managed to improve the situation after weeks of refactoring and testing for my library, and thankfully now it was completely replaced by webrtc.

I feel a lot of this is the difference between theory and practice. Sure each of these things are bad, but probably a lot might have been the right choice at the time, and in a way, most companies, even most projects running for many years, end with similar quirky processes, messes, and hacks.

It's the war stories of what used to be, often only told by the code and the database as the creators have long left.

When I look at our couple year old startup, we have some of these things. Prototypes that just keep functioning and only get more attention when they break, manual processed that sort of make sense but also don't, integrations with systems that were build in the early 2000's (it works, but it ain't pretty). We fix many, but I'm sure some will survive way too long.

As software engineers, especially online, we like to discuss the ideal way to do things, best practices, testing strategies, redundancies, the whole nine yards. When time is of the essence and stakes are high, it all goes out of the window and you gotta just make it work in whatever way is possible to fight another day.

The old codebase was an hairy ball of scala using a very outdated version of a (now) infamous actor framework. Before they figured out that untyped messages kinda left out one of the major selling point of Scala.

The code was readable, but the authors had this strange idea that every little piece of logic should be part of its own "Actor". An actor is pretty much equivalent to a class, and each one of them had their own little file. With this many classes, with very specialized purposes, you ended up with 90 character identifier names.

To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic. Generally, at the end, you find that most of the code is about passing around a value, and there is this one file where there is actual logic applied to the value.

It wasn't that awful. The thing was that it was impossible to change anything: no documentation, no test, no bug tracking, not even any PR process, laconic commit messages, no Wiki pages. And the actor framework made it very difficult to add tests. But later developers did manage it pretty well, they drew fences around the old services, with an HTTP API to communicate with it. And all later additions were part of different services that were very cleanly and consistently designed.

I could talk through pain points they were having, we’d come up with a solution together, I’d hack up a quick prototype and launch it just to them to try out. We’d tweak it over a couple of weeks and when it was good I’d launch it to all customers.

Messy, ugly code base. But it worked well because it wasn’t over-managed. Just developers doing development. Amazing what happens when you get out of the way of smart, talented people and let them do their work.