Requirements isn't just making lists and tickets in swimlanes. It's actually learning the domain you are building for. Sure you can still build for something you don't really understand, but it'll be garbage. In fact, writing this garbage is OK if you accept that it's just a drafting process for learning the domain challenges and potential solutions. There are no shortcuts to quality.

My advice for stakeholders is to make sure you have access to the person(s) building your application and see if they understand your domain (you can do this without constantly pestering them!). Beware bullshitters using buzzwords.

My advice for designers / programmers is to make sure your stakeholders & domain experts are keen and engaged, otherwise getting requirements and understanding is like pulling teeth. Sometime a stakeholder doesn't even want to solve the problem. They may have wanted to go in a different direction entirely or maybe there are politics involved that has them miffed. Nothing sinks a project like lazy or uninterested stakeholders.

Fast forward to today, I'm in a different part of the industry, we're building things that aren't quite as complex but they take longer to finish and the experience is very frustrating. I feel like I'm working in teams where many people don't really know what they were doing but we keep getting pushed for higher story point "velocity" and whatnot. Quality is suffering, tech debt is piling up. This whole thing feels broken.

I can vouch for this. At the other end of the spectrum, programmers getting thick with their domain experts is like having clairvoyance. I can sometimes spot problems before my PM does!

For the most part, it's those types of issues that lead to a system not working as intended, or being buggy as hell. If you can get a good setup going where the requirements are clear and don't change and the people on the project communicate well and have full control over the process then things will work out fine. That's seemingly what happened in the source article.

> The stakeholders have no idea what they want

And yet, you want to put together a project plan up front based on them telling you exactly what to build without assuming there will be sweeping changes? That makes no sense!

Building iteratively recognizes this reality that people aren't sure what they want or need, rather than peevishly ignoring it. The goal is to figure out what would be valuable in a short period of time; if it isn't valuable at all, that's fine, it was not a huge investment and can just be thrown away; if it is valuable but not quite right or not as useful in its current form as it could be in a different form, but the change needed is large or fundamental, that's still fine, even sweeping changes are no big deal on top of something small; and if it's already valuable just the way it is, that's great, move on to the next iteration.

This seems to make a lot of people uneasy, I guess because it provides no formula for answering the question of where things will stand in a year, or even in six months. But it honestly recognizes that nobody knows, rather than setting the expectation that where things will stand is exactly where they project plan says they will, with everyone inevitably getting mad that reality didn't match that plan, or worse, being exactly where the plan said you'd be, except the thing the plan called for was the wrong thing to build and the whole thing was a useless waste of time.

I also think you can do iterative development with having a clear long-term objective (requirements) in mind. I would even argue it helps a lot.

I suggest looking up "What made Apollo a success" (https://ntrs.nasa.gov/citations/19720005243), they explain it quite well.

I never understood why software engineers had such a high tolerance for this shit. Is it because changes are technically possible at any point?

For example, if I order flowers for a wedding, I need to order at least a few months in advance depending on the time of year, because flowers are a seasonal agricultural product. And I can't change my order once it's locked in because they can't go back in time and plant more flowers.

We should treat software engineering more like that. There's no reason we should allow product people to abuse the flexibility of software. You can still be agile, but you don't have to be at the beck and call of people whose preferences change with the direction of the wind.

If you think "agile" is the reason requirements change...I don't know what to tell you. Requirements will always change, full stop. It's like a force of nature, there is no universe in which everyone just "knows" what to build up front and has a fully spec'd out API that you can go off into a cave and implement. Real life never works that way, and software engineering is not the right profession for you if you need that.

But that doesn't mean you have to stop what you're doing and go chase those requirements.

Why did the requirements change? Is it mandatory that the change happen right now? What research was done to support the requirements change? Was the original requirement bad in the first place? Was insufficient alignment and understanding achieved at the start of the project? Do you actually talk to the stakeholders at all, or does your PM just forward you their emails and expect you to do whatever is in there?

There's a lot of room between "design everything up-front and never change anything" and "allow requirements to change arbitrarily".

The arbiter of the changing requirements is the one paying for the work.

But it's not that binary. "Agile" is the reason requirements are allowed to change constantly. At worst, it can be like trying to steer down the freeway by slamming the steering wheel from one extreme to the other. And pure waterfall is also blatantly unworkable.

The real question is, at what rate do you allow changes to be made to the specification/requirements? How much dampening do you apply? And maybe under that, there's another question: How fast can you respond to the real world, and still maintain a coherent direction? The faster the better, but don't try to respond faster than you can maintain coherence.

And agile doesn't make you respond to change quicker, it makes it slower since it is done in 2 weeks sprints. Normally a team could adapt the moment new information comes up, strict adherence to scrum agile would push that for the next sprint.

Agile does make scheduling new changes effortless though, encouraging new changes to be made all the time, but it doesn't make the team react quickly to those changes and nor does it remove the total cost of a change. I don't think that is a good thing to encourage, in such a system no wonder people get used to changing things all the time so nobody really knows what things are supposed to be.

What do I mean by that: I work for big corporations mostly and quite often there is the average case that is about 90%-99% of all incoming work.

Then there is a myriad of special cases some of which happen every third year on a blood moon, if an eclipse is happening at the same time and the witches chant in the woods...

Instead of managing these unicorn-cases by hand, they have to be implemented in code and lead to bugs and a ton more code to review and maintain.

Speaking of maintenance... oh, don't get me started on that one, it's a sore spot!

In particular, these edge cases tend to be more complex, require more knowledge and have less margin of errors than the standard errors happening 90% of the time. Leaving them as a gift for the future maintainers is a special kind of dick move.

by whom exactly? I mean, I agree that it's shitty (I was such a maintainer already), but who is responsible (the whole machinery or a specific role?) and how can we change it?

I personally try to add meaningful comments to code that may seem "strange". Like code that if I read it and it was written by someone else I would ask myself "Why so complicated?" - that way I hope to improve the situation a bit for future maintainers.

If there is specific finger pointing needed IMHO it would land either on the manager or the product owner for miscalculating the impact of that decision.

PS: To your point on "weird" code, imagine a code that just asserts for some conditions with a "if these conditions are true call XXXX team for help" message on it. That's how I'm representing an unhandled edge case.

Sometimes these things are worth it because the manual process is error prone and perhaps frustrating/stressful (these things can't be measured as easily). But sometimes they are not examined critically at all.

How often? How many? How important? Are there simpler solutions?

Kind of depends on the receiving end of the questions in my experience. Some people are happy if you push back and keep things simple, others have problems with that.

I wonder how much does scale affect these basic principles. I would assume that with scale you already have a lot of problems that push even harder towards not implementing every single thing.

This is where trade-offs in engineering and product are important.

Is the edge case safety critical or poses a safety risk? If so, it definitely should be considered and handled.

Does the edge case block a critical user flow (eg. purchase/checkout step)? If so, it should probably be considered.

Does the edge case result in some non-critical piece of UX having the wrong padding in some lesser trodden user flow? Possibly acceptable.

Started as a Windows line of business software developer almost 20 years ago. At first we got clear specifications with UI mockups and a description what each and every button should do. When there were questions, these specs where updated and we implemented and tested and fixed until our boss was happy.

Over the years we got more and more customers yet less and less time for tests and fixes. So we switched to "agile" and dropped the specs, instead wrote quick notes about what must (roughly) be done. At first all were happy. But now we have a huge amount of features that not one dev knows. You have to ask and search around until you find all the lose specs.

Now I'm managing such a team of developers and have the same issue. I don't have the time to write a clean specification, yet alone discuss it with the actual customer, which anyway doesn't really understand all the implications. So they start coding by adding more if and else blocks to an already bloated code base.

Pity the days we would start with a class diagram or just some quick drawing about how the components would interact and be testable.

If it weren't for users my software would be perfect ;)

We are inventing the field as we go. Have been for the last decades.

Also it's one of those fields where the expert in the topic must build a system for something he is not an expert with.

Again and again.

And most project are custom.

Agile has nothing to do with it.

It's the nature of IT right now.

If you've got that - why do you need the developer? Genuine question. If the technology requirements are that well defined why on earth do i need to pay engineer level salaries for such basic work? As you said, and i agree:

>> Give a dev a clear API and a well defined set of criteria, and most will write code that works very well

At that point of having clear criteria, there are only 2 tasks left:

    1. write some code that satisfies this outcome
    2. write it in a way such that it can be cheaply changed in future

But but but performance! durability! ...

As a customer of this software developement process, these are just the next cycle of requirements to feed into the machine - hey take this thing you did, make it faster while obeying rule #2 above.

It would not surprise me if ChatGPT were able to cover 80%+ of a problem that well defined.

I totally get the attraction to focus on the fun bit, but it's not how the business world is designed to work in most cases.

For what it's worth i think you nailed the actual value opportunity for a strong dev:

>> no one actually knows what they're building

If you can nail that, you're always gonna be valuable. We talk about programming and coding but actually, this is more of the role. The coding part of the puzzle is easy in comparison.

As for:

>> and even if they do it keeps changing because "agile"

If your understanding of why requirements are in constant flux amounts to pinning it on "agile", you're not even on the field yet never mind winning the game.

Also, indeed, not all dev work is equally costly / hard / basic. It’s a crazy thought, but maybe that’s partially the reason why different devs can have different pay grades. :O

Someone still needs to do this work, and sometimes that's the developer. Isn't that what he was doing in this anecdote?

I'm probably romanticising it, but it was a time where software engineers and their skills, time and attention span were still respected.

But in actual professional coding practice over 10+ years, I can count on one hand the occasions where I had clean enough reqs and enough uninterrupted time to design software like that, and still have enough fingers left over to hold a pencil.

In particular, management was tech-savvy and everyone liked and respected each other. Most were also talented.

He did "invent" internal and external APIs for his service, just like developers do today: that's the part requiring developers to be like "engineers".

If you don’t understand a solution well enough to write it down, you don’t understand it well enough to implement it. (Much less get someone else to implement it.)

An overwhelming number of engineers I have worked with seem to think any kind of written design document is busy work, and not actually a part of the process of making something worth using. My experience shows it is not that at all: design docs are a tool to externalize your own thinking, and reflect on its quality without the overhead of keeping it in your mind. That’s their first and foremost purpose. To explain your rationale to others is a secondary objective.

I’m not talking about lengthy requirements documents or specifications, either. I mean a two or three page white paper outlining a problem or a proposed solution in plain English. This is something you can bang out in a half hour or hour if you’re diligent.

Many of the folks I work with never even seem to bother.

Agile is precisely the response to the fact that requirements keep changing, not the cause of it.

So: was it a good way to teach or a good way to filter out less motivated people?

Before then filtering out less motivated people wasn't as much of an issue, as they had little incentive to study CS in the first place.

In that days, designing on paper was probably just the default mode of working ...

Maybe the new way of doing thing is still more efficient in the end, but maybe not.

If it's important print it out, in large format.

While he was talking, I started outlining a solution on a piece of paper. At school, I couldn't use a computer during class, so I'd gotten used to programming (and debugging!) on paper.

When I arrived at home, I typed it in and tried it out. Could I possibly reproduce that Multics story I'd read about?

It didn't compile the first time, but after correcting a variable name, it worked perfectly.

I remember a number of years where the edit->compile->print loop was much better with listings than terminals, simply because you couldn't see/navigate/edit your code as efficiently. It eventually improved with better/visual editors, larger terminals and faster compiling and running.

An analogy might be early firearms, where things like wet gunpowder, unreliable flintlocks and pushing everything down the barrel made for an inefficient transition time from the longbow.

> Former Multician here. Andre was super-smart, but it is perhaps also relevant that even the most junior developers (as I was then) had quiet private space to work with large desks. All design was done offline. Multics terminals were like type-writers (although video did show up near the end), hence no stream of popups. The environment both allowed for and demanded focus and concentration. This is no longer so.

The first time I was maybe 10–12 years old. We were visiting my grandparents, who had no computer (and there were no smartphones). They did, however, have a (mechanical) typewriter that had the ability to type in both black and red with a switch. Back then my main hobby was Turbo Pascal, so I used the typewriter to write a Pascal program that I would later type up into the PC and run when we got back home. We spent a week at my grandparents, so I had plenty time to think it through, debug it by hand, and re-type sections that were faulty.

The second time relates to an esoteric programming language I invented called Ziim (https://esolangs.org/wiki/Ziim), and more specifically, the binary addition function you see on that page. It's huge and complicated... and it had a bug, which I knew because I ran it in an interpreter, but I had no idea where the bug was and how to fix it. — Around that time, I had a long bus ride coming up; sitting in a coach for like 6 hours or so. That was a perfect opportunity to debug this. I transferred the Ziim addition function to square-ruled paper using a pencil, and then, on the bus, I executed it by hand, step by step. I found the bug and was able to fix it. It required redoing the entire function layout.

I guess the moral of the story is that restricting your own ability to do things the “easy” way can sometimes lead to well thought out code. Despite, I don't do this regularly at all. I will just as soon write half-arsed snippets and iterate, grab a debugger to step through code, etc. Maybe I should rethink that?

In Ye Olde Days, "programmers," were usually little more than data entry clerks (often, women). The people who wrote the software would be in offices, filled with smoke, writing the programs on paper.

Compute time was expensive and precious. If you got a bug during your run, you wouldn't get a chance to fix it, until you could schedule another data entry session, and another CPU run.

It encouraged a "measure twice, cut once" approach. Since most software, in those days, was fairly humble, compared to the de rigueur, these days, it was easier to do. Also, software usually had extremely restricted I/O. UI was not even a term, and peripheral connection was a big deal.

These days, I often "throw stuff at the wall, and see what sticks," when writing software. It's easier for me to write some half-baked crap, and debug it in the IDE.

My software development tends to be "iterative." I write about that, here: https://littlegreenviper.com/miscellany/evolutionary-design-...

I remember an assignment in my university course, where I had to write a (simple, admittedly) OS kernel. I didn't know C very well, and I didn't know what kernels did beyond the theory, but we had to write a few hundred lines of C code to manage tasks. I knew that there would be absolutely no way I could debug this program if it didn't work, as it was all parallel code and a bug would mean mysterious race conditions.

I reasoned about the system as a whole, and I spent a few days writing small, self-contained functions that I thought a lot about. I then compiled and ran it, and, after the obligatory "missing semicolon" compilation errors, it worked first try.

(About this particular problem I have no idea.)

Control flow aside, the syntax looks surprisingly clean.

One can only become cynical and bitter.

Also, the whole extreme programming style was prototyped around hilariously big (for reasons called "USA" not bad code) payroll software for Chrysler.

Also, I'd say that manipulation of CSS classes and HTML elements with JS gives the developer quite a strong understanding of them.

Oh really, don't: that's what the discussion should be about.

You are certainly right to a point, but a better way to redirect positively is to ask: how can we come to a state where similar feats are possible today in a company setting?

i mean they might just be smarter, but that's far from the only explanation. they could be working harder, or have better mentorship, or not have to deal with productivity sinks you do, etc.

Do go check the code: https://multicians.org/vtoc_man.html

This is both bigger and more complex than most stuff people work on today (which is glorified glue code for bullshit CRUD/REST/etc). It might be smaller than the overall LoC of their project, but it's more than the units they deal with (and this is part of an overal OS code anyway).

This is a whole "manager, to manage file description information. It had to transport the file information between disk and memory, manage a shared memory buffer pool, and manage space on disk for the information."

Dare most current programmers to write such a thing, with the requirements and semantics it has on his version, today, in their language of choice.

Most would be lost at even contemplating that. Much less write it in paper and type it in and have it work.

a small fraction of that is urls for electronic parts, but nearly all (90%, say) is just english text. another 10% is stuff i quoted from other people (web pages, manufacturer app notes, books, etc.) it is a single file and will probably continue to be a single file forever, at least for the rest of the year. if i continue at the same pace, that will be 12 megabytes. there is nothing impossible about that at all

(if you're interested, git clone http://canonical.org/~kragen/sw/leatherdrink.git. it's currently almost 200 megs because i checked in the avr toolchain. there are also photos and videos and schematics and stuff)

in a programming language the growth would be fewer bytes per month, but perhaps by a factor of five

When I was 14 and went to highschool, for programming classes I did most of my code on paper. I was a poor kid in a poor country, so not only I did not have a PC at home, but the few "computers" we had in the computer lab at school were ZX Spectrum clones which didn't run Turbo Pascal, the programming language we used at that time.

"It can be done".

Thanks for the motivation!

P.S. Of course, I also read the article :).

Also "Hacki" and "HACK". All similar (of course), but all in some way lacking..

For me its like you can use paper for drawing but only 10 minutes a day. Or piano but only every second Monday for few hours.

When I have taken this approach - pencil and paper - I find that it accelerates the initial big decisions (e.g. "what will this module's responsibility be?") but sometimes causes rework when I discover that I was optimistic or unaware of how two areas of the code interact.

Perhaps I'm just not doing it thoroughly enough.

But the other missing piece to this is how much time did this actually take (there is an implication "designing" took its sweet time)? Would the modern way of coding on a computer be faster with the same number of incidents?

Simple. Agile/Scrum had not yet been conceived of.

Observability and trial-and-error tooling are a big deal. They enable the workflows that allow developers with a poor conception of the system to brute force their way to fitting requirements. The popular ethos around shipping the first thing that works, often, only works when you can see what you're doing, and the cost of bugs in production is low. That's software in 2024: mostly unimportant garbage.

For this kind of system at this point in time, a full design was the only way. The article hints at a touch of perfectionism (which probably didn't help the timeline), but it's not like he would have been more likely to guess and check his way to a working memory management system in less time with the sans-design development style popular today.

that said, i always love stories of people taking time to find aesthetics and quality with pen and paper

I saw him once a year, when my family came together for xmas at his house.

In the 12 months in-between I wrote tons of BASIC code with pencil. And then typed it in at xmas. That lasted for two years, until I had saved enough pocket money to buy my first PC.

The experience was very similar. Stuff just ran & worked as expected.

If you write code by pencil you just think a lot more before putting anything down on paper. As any corrections are a nightmare -- except for the last line. You can't just insert a code block etc.

Using his brain. Intensively ;-)

Reminds me a story that happened to me. I was doing some heavy changes to some library, then started running the tests. The tests were failing, and without looking at log I knew immediately where the bug was. So I fixed that, launch the test and again same story. And again. At the 4th time, the cause was not immediate so I was considering looking at the log when I noticed I was not actually running the program at all, but was executing something else!

So to find bug, you just need to ask yourself where it will fail, and fix that ;-)

Bug Kausing Almost Certain.

It Can Be Done (2003) - https://news.ycombinator.com/item?id=18415231 - Nov 2018 (18 comments)

Still surprising how much entrenched the computer industry was (and probably still is) within the military complex.

I'm curious if there is a list somewhere with big, consequential IT/computer programming stuff that has NOT had any direct connection with the military.

Being able to anticipate and find nice design level general solutions for hairy real-world problems is where systems design becomes most rewarding.

I never touched UML again after university ...

He started by sitting at his desk and drawing a lot of diagrams. I was the project coordinator, so I used to drop in on him and ask how things were going. "Still designing," he'd say. He wanted the diagrams to look beautiful and symmetrical as well as capturing all the state information.

Of course, this doesn't work for every problem / task.