> "The banking programming world is a completely different world than what most of us are used to"

If you're after another read on this topic, "An oral history of Bank Python" is good: https://calpaterson.com/bank-python.html (also previously on HN: https://news.ycombinator.com/item?id=29104047)

https://news.ycombinator.com/item?id=12097032

It's funny, people frequently assume this would be true but reality doesn't really bear this out. It's typically pretty average to even below average which contributes to the talent pipeline problem.

The other side of it is that its not the technical stuff like "knows COBOL" that is so immensely valuble. Any average dev can "learn COBOL" but that's not actually the valuable thing. The anecdotes of COBOL programmers coming out of retirement for 500k/yr contracts has little to do with COBOL, but their accumulated institutional knowledge of the giant ball of business logic encoded in that COBOL.

If these banks and other institutions actually did write fat paychecks to young mainframe programmers the demographic problem they're facing might not be so bad.

They wouldn't know that the reason the key export fell over when extracting data for your 10K filing because someone had decided/assumed/whatever that a certain record count would never go over 3000, so they hard coded the program to just error out if it went above that value.

The claims processing system I worked on used a number system that would only go to 4 digits, it would roll over to 0 if we ever processed more than 9999 claims overnight.

Our VP would not listen to us when we said we needed to change that. He said "Claims process fine every night! There is no problem." I had left before they got to that 10,000th claim (thankfully).

That said, anyone need an old COBOL developer?

Edit: now I’m the VP of Engineering. We do work on tech debt regularly.

The language you happen to be using in your system is almost inconsequential in the grans scheme of things, because it's like 1% of everything that's going on and that you need to know.

When I first started work I was involved in banking/insurance mainframe stuff, and it was honestly a pretty terrible working environment for reasons that spanned from the mundane (like dress codes) to the esoteric (like the horrible crufty legacy codebase). I would have put up with it if it paid well, but it didn't. In fact, it paid significantly worse than the job I'd had before which was installing physical cable plant (fiber optics and copper ethernet), the reason I took it was at least the office had air conditioning, but it certainly wasn't something I wanted to turn into a career.

As a mid-level windows sysadmin doing customer-facing phone support at a cloud provider, I made nearly 30% more than I had as a junior mainframe guy at a bank/insurance company. The difference in the skillsets and their commonality was massive, yet the pay was significantly worse for the mainframe work despite it being a rare skillset in a high value industry. I remembered when I was in this job that guys who worked on trading desk backend code were far better compensated while working in easier development environments on less esoteric codebases, the mainframe folks were the lowest paid of the developers at that company, at every level of seniority. The only person we worked with who was well compensated was an independent contractor who'd worked there previously for 30 years before retiring.

As you know, the pay wasn’t so hot for stuff like that back then. But it was an improvement over manually unloading flatbed trucks stacked with steel conduit, which was my prior line of work!

Today I think it’s much different. Some banks and insurance companies and the like still at least partially run on software as old as the Apollo missions. Maybe I’m wrong, but finding people still alive with that knowledge intact must be difficult and I imagine their compensation reflects that simple supply/demand formula. Shrug.

But the wages being offered for those roles are abysmal relative to what's available for other skills, and modern life is applying a great deal of pressure to chase larger paychecks, sadly.

I worked for a multinational bank headquartered in the U.S., whose corporate team came up with a sexy program to snatch up recent Computer Science graduates. The program offered them a "fair" market rate for an entry-level jobs in industry, and let the graduates try two to three different teams and/or departments in their first year. At the end, if the company still liked them, the candidate got to pick their permanent team for a full-time position. This position, however, did not see a bump in pay, and by that time the program effectively filtered out candidates that weren't performing at an accelerated level.

The results were great for the business: young programmers that were often more proficient in COBOL than their new peers were costing them $60k-$70k a year. The senior, or "tenured," peers were in the $200k-$400k range in some cases.

Whenever I hear about a skills shortage (in any field), I automatically append the phrase "at the salary we want to pay" and that completes the equation.

Also, there’s usually 2-3 layers of pimps (prime and subcontractors) who take a vig at each layer. Total comp for the contractor is usually rate divided by 3 or 4.

An engineer retires from his company, after a 40-year career. He left at a senior technical level.

Some time later, he’s contacted by his old company, begging him to help them fix a problem that has the current staff absolutely stymied. They have been at a standstill, for weeks.

He agrees, and shows up. He sits down at a workstation, examines the behavior, looks at some code, then says, after about five minutes: “Here’s your problem. If you just do this, it will fix it.”

He hands in an invoice for $10,000.

The beancounters flip their wig. “We can’t pay $10,000 for five minutes’ work! Itemize it, and tell us exactly why you think it’s worth it.”

He takes the invoice, turns it over, and writes on the back:

    1) Fixing the bug in five minutes. . . . . . . . . . . . . . . . . . . .
 . . . . . . . . . . . .$20
    2) Six years of college, and 40 years of experience, so I can fix the bug in five minutes. . . $9,980

    1. Hit machine with hammer ....... $1

    2. Knowing where to hit it ... $9,999

The artist takes out a drawing pad and says to the client, would you excuse me for 15 minutes, I need to draw in silence and solitude.

The client leaves and sits to a different table.

After 15 minutes, the artist hands him over the drawing and names a price.

So much? cries the client. This is quite expensive. You only drew for 15 minutes.

Well, says the artist. In order to draw this in 15 minutes, I have been practicing drawing for at least 5 hours every day for the last 50 years.

Absolutely love this story (and the mentioned artist as well, he was a huge inspiration for many).

"The earliest instance located by QI ((Quote Investigator)) appeared in “The Journal of the Society of Estate Clerks of Works” of Winchester, England in 1908.":

https://quoteinvestigator.com/2017/03/06/tap/

This, I went into programming over 40 years ago. At the time I worked in a warehouse, I had to take a 5% pay cut (IIRC). But in the long run it was worth it. Most people there stayed in the warehouse/manufacturing due to the pay.

The same code base passed down though generations.

Legacy code.

I'm already seeing the inheritance jokes write themselves.

Someone have chatgpt write the screenplay now.

CODE REDUX: Legacy Preserved

"In a world where the past and present collide, a family's legacy becomes the key to saving the future."

https://chat.openai.com/share/2f0c9eec-c3b8-4145-86d7-2c617a...

>Legacy code.

Vernor Vinge's A Deepness in the Sky depicts a human society thousands of years in the future, in which pretty much all software has already been written; it's just a matter of finding it. So programmer-archaeologists search archives and run code on emulators in emulators in emulators as far back as needed. https://garethrees.org/2013/06/12/archaeology/>

(Heck, recently I migrated a VM to its third hypervisor. It has been a VM for 15 years, and began as a physical machine more than two decades ago.)

And given how big the market share of Nordea is in Sweden (and other Nordic countries, for that matter) that would probably bring down the Swedish, and possibly Nordic economy. Which would then impact a lot of the EU as well, I guess.

Ever since reading this article I've wondered if these COBOL programmers that keep banks like this running are an enormously underestimated "bus factor" for many of the world's economies, and what kind of back-up plans they have for such a scenario.

[0] https://en.wikipedia.org/wiki/Nordea

Hmm, makes me wonder about potential similar situation with APL programmers - a language I know quite a few large companies use.

I mean, notoriously, stop working, for weeks: https://en.wikipedia.org/wiki/TSB_Bank_(United_Kingdom)#Migr...

(I've got to assume that loss of institutional knowledge was a big factor in that fiasco.)

I hope you came out with more of a scare than anything else?

Most of the time you try to reuse existing integration points from previous projects, as negotiating completely new interfaces had a lott of friction, both technical and business/compliance related, and could easily set your project back for more than a year or two.

Integrations are usually delivering structured documents before a certain time in the evening for overnight batch processing. The existing documents with data based on offsets have by design regions of not yet assigned space in them to accomodate future updates and use. You negotiate over the bytes of those that can be assigned to your needs if new info is required.

For data extraction you will sometimes find more 'modern' api's, but do not expect too many fine grained REST stuff. You'll often find yourself in meetings negotiating with regulation and compliance looking for ways to avoid costly new developments.

On a sidenote: I often found compliance people a lott more pragmatic and solution oriented than IT. Having to convince my team that 'the letter' of a regulation did not mean you have to interpret litterally in the most restrictive way possible what is written was often more a challange than getting a deal with compliance.

Now imagine hundreds of projects over many decennia of these strata of integrations, and you will start to get the first glimpse of why replacing these core systems is such a challange.

Somehow this resonates a lot with me even if though I've never worked on Mainframes or anything like that.

Witness modernist architecture: "you shall all live and work in soulless concrete boxes, because I the brilliant architect have decreed that ornament is superfluous, and I can shape you into Properly Thinking People by manipulating your environment."

Thank you for sharing!

She has all kinds of amusing stories of ye olden days. Before a proper computer system, everything was stored in physical documents of course, and her first project out of college was traveling to satellite offices to re-organize the file systems there to match the new strategy that HQ had come up with. The business of satellite office would grind to a halt while a team of people literally took every single document out, re-labeled it, and put it back. The whole project took over a year.

That gave some perspective about my job, lol.

At least, I hope they were satellites!

Now she can't tie her shoes consistently.

To everyone reading, capture the stories of your loved ones life accomplishments before it's too late!

I should make that my December project, to find a way to unlock those files.

Could be both: https://en.wikipedia.org/wiki/Fractional_Orbital_Bombardment...

https://www.smithsonianmag.com/science-nature/history-human-...

COBOL is not a "cool" language, but mainframes have been around long enough to be "retro cool" now, and most run Linux at least as an optional OS under some virtualization (IBM Z).

As a doctoral candidate in the noughties, I purchased a book about FORTRAN due to its "retro-coolness" and read it, and eventually took on a short university gig to earn the money that the book cost back, tutoring architects/engineers in FORTRAN 95 for a bit, which was fun; to date, I could not bring myself to do the same with COBOL, though, or not yet.

Because it is so verbose, if I had to use it I would probably write in another language and transpile to it.

One of the things I did was write a COBOL pre-processor (in C) to allow re-use of code and variables so the variables looked like locals rather than globals.

On the other hand, you could expect to learn a new operating system every year. Oh, we don't use DEC gear at this school, check out our spiffy Prime running PRIMOS. Lather, rinse, repeat.

Then I wrote a terminal emulator for Windows 3.1 that would parse the forms sent down by the runtime and display them as something that looked like a Windows app. This wasn't too difficult because mini & mainframe systems present data entry pages a whole screen at a time, like a windows form.

I would think that this is a task tailor-made for A.I./ML

That's pretty crazy

Sounds like having the whole team SSH into one server and doing all the work through the terminal

I'm imagining editing my co-workers files and just removing a random semicolon to mess with them

Legacies of this abound in Linux (even the ability to SSH in is a descendant of this world). Commands like "who" are there to show who's logged in, and there's even an entire process accounting system that can be switched on to bill your users for CPU time.

https://www.ibm.com/docs/en/explorer-for-zos/3.1.1?topic=mes...

I was using an Amiga at home and had been using Xenix and honest to god AT&T UNIX in my previous job so had some idea of what I was missing. I also used IBM S36s and weird Burroughs things so had a pretty wide education on different types of systems in a short time.

Source code management on mainframes has been around for decades now. Pansophic started selling Panvalet in 1970, and Broadcom still sells CA-Panvalet.

Closer to RCS than to Git in feature set.

> but many shops still have you log into the same "box" to do your development

It is very common to have separate LPARs for production, test and development, even if all three are running on the same physical hardware - the isolation is strong enough that it is rare for something running in one LPAR to cause a problem in another.

The largest shops will have physically separate mainframes for production and non-production.

Keep your skills up, people!

Considering what they are working with, I'm surprised they don't have even more downtime.

https://en.wikipedia.org/wiki/IBM_Information_Management_Sys...

The heyday of network model databases was after hierarchical, but before relational took over - basically the 1970s.

Hierarchical databases basically allow you to have parent-child relationships between tables, to represent many-to-one relations - something people commonly model nowadays using RDBMS foreign keys, but RDBMS foreign keys are much more flexible and can model other relationship types too, whereas hierarchical only allows parent-child.

Network model databases generalised hierarchical databases to allow more complex relationships than just parent-child. The biggest difference between them and the relational model, is they didn't have a query language like SQL, they offered a procedural API for processing one record at a time and manually navigating the links from table to table. Network model was standardised by CODASYL-the same committee which invented COBOL, with the result that they became particularly popular with COBOL applications.

The most famous network model database is probably Broadcom's IDMS (previously known as CA IDMS), which nowadays only runs on IBM and Fujitsu mainframes, but in past decades was supported on other mainframe and minicomputer platforms as well. Oracle also offers a CODASYL network model database for OpenVMS, which it got from Digital when it bought Rdb.

Graph databases have something in common with the historical network model, but some significant differences (1) graph databases have much more flexible schemas (defining arbitrary properties on a node as opposed to a rigid structure of fixed format record types), (2) graph databases generally have a query language with support for graph traversal operations, as opposed to the procedural API the network data model offered.

I wouldn't necessarily call that "advanced"... maybe more, lacking in requirements-analysis-time insight in ways to factor the business-domain into HAS-A component relationships, such that individual components and their ADTs can be shared across parent types [which are really just template/factory objects for a smaller set of actual types], and initialized with simple parameters that get used as formula variables with no piecewise logic.

To be clear, I write this as someone who works for a company that maintains a unified representation of data across the blockchain ecosystem — where each blockchain has its own peculiarities about what an "account" is, what a "transaction" can do, etc. Our data model only has one toplevel account type, one toplevel ledger-transaction type, etc. To handle the peculiarities, our data model instead has a large heirarchy of smaller data-objects hanging off of those toplevel ones; where any given data-object is sort of an "optional extension" that may or may not be there depending on how the toplevel object was created.

This approach allows us to just have one unified code-path that treats every account like every other account, every tx like every other tx, etc. We don't have duplicate code or a hierarchy of subclasses that all do things slightly differently; but instead, for any ledger-transaction, there may or may not be e.g. a strategy-pattern object hanging off that tx — and if there is, it gets used instead of the default static one. It's great for maintainability, testability, predictability, cacheability, and hundreds of other things.

I'd love to know if there's any good reason that a bank would actually want "50 different types of bank account" on an implementation level, rather than these all boiling down to one type with varying values of certain state variables + presence/absence of certain foreign-key relationships.

Other than maybe "some of these account types are actually a part of completely different data models living in third-party systems, that we acquired, and then never merged into our own systems." ;)

Not having the benefit of hindsight?

Some banks are hundreds of years old. Most of them were computerized seventy years ago if not earlier, back in the stone age of computers. These kinds of banks won't bet the house on a newfangled system every couple of years because some bright-eyed engineer told them it's the trend nowadays.

Not messing up their bookkeeping is their number one priority. People would riot if their bank told them "sorry, we no longer know how much money you had deposited with us".

The thing about banking, is that they keep records of everything — not just "state now", but all previous states, and all the state deltas, and all the commands that produced those deltas, and an audit log of who/where/when the requests were made to triggered those commands. Financial-ledger databases are the original CQRS event-streaming reducer systems.

And this actually means that it's very easy to produce a new system that is provably "at parity with" an existing system. No faith required. You take your complete historical CQRS event stream from your existing system, stream it through the new system, and see that it produces the same state that's in the existing system. If it does — and if you're operating on years of real data — then that's more evidence for exact parity than a test suite would ever be.

(You may also want to produce hypothetical CQRS command streams and run them through both the existing and new systems. This would mostly be useful for regression-testing of edge-case logic that is required for e.g. compliance, but so rare that it has yet to ever actually come up in practice.)

> Most of them were computerized seventy years ago if not earlier, back in the stone age of computers.

You can do everything I mentioned — factoring out your business rules into simpler rules that apply to a collection of orthogonal sub-ledgers — on paper. These ideas are not "newfangled"; while they were introduced into computing through the formalism of relational algebra (as database normalization), the ideas existed well before the mathematical formalism for them existed. Clever people have been simplifying their paper records into orthogonal sub-ledgers since the invention of double-entry book-keeping in 1494.

Since the post is about a Swedish bank, it might be interesting to note that the central bank in Sweden was founded in 1668.

https://en.wikipedia.org/wiki/Sveriges_Riksbank

Kinda makes you wonder if anyone opened an account then and deposited a dollar ("daler" in Swedish), and let it sit and accrue interest for the family for 350 years.

[0]. https://www.ibm.com/products/watsonx-code-assistant-z

Stories like this put in perspective people that are moaning about tech debt... by which they mean some functions written a year ago in a style they dislike.

Basically the only things computers were worth it for back then were scientific calculations and bookkeeping. Banking is obviously on the more complicated end of bookkeeping. Insurance did get a special COBOL variant because I guess they had some tasks shaped more like scientific calculations too.

At most banks nowadays, basic banking functions such as saving accounts are provided by off-the-shelf software, not written from scratch.

That's not to say that nothing gets written from scratch, but it is generally adding code to support the more exotic product offerings, integrations with other systems, institution-specific business rules and processes, etc - any off the shelf banking system is going to have basic bread-and-butter stuff like support for savings accounts already included.

This is true even for COBOL-based banking systems. Historically, many banks used CSC Hogan (now DXC Hogan)-and while many have moved away from it, some are still on it. Hogan runs on IBM mainframes (z/OS), and is written in COBOL and CICS. Basic stuff like savings accounts is supported by the vendor-provided COBOL code, but Hogan sites often end up writing their own custom COBOL code to support their own unique requirements.

> I surprised banks don't have a domain-specific language (DSL) to describe their businesses.

There is a long history of 4GL's being used in banking, going back decades. In prior decades, many of these 4GLs worked by generating COBOL code. Some of them were general-purpose, and used across many industries; others were exclusively used in banking. But, even those exclusively used in banking, rarely (to my knowledge) contained domain-specific banking features, and as such I'm not sure they really count as DSLs.

To give a specific example, FIS Global's core banking platform, Profile, is written in MUMPS (actually the open-source variant GT.M). Due to how horrid MUMPS is as a language, they created their own higher-level object-oriented language which compiles to MUMPS, called PSL (Profile Scripting Language), and gradually rewrote their banking platform in it. An old version of PSL was open-sourced over 10 years ago on Sourceforge, if anyone wants to look at it. [0] I don't think PSL itself has anything really banking-specific in it per se, but as is inevitable with a language developed to support a single application, the boundary between the language and the application is a little blurry, and you'll find a lot of banking-specific stuff in the open source release (possibly included by accident), especially inside the binary GT.M database dump it ships with.

I think the areas in which truly domain-specific languages are strongest in finance - such as modelling financial contracts - are generally the furthest away from the COBOL legacy.

[0] https://sourceforge.net/projects/pip/files/PIP/V0.2/ see also more easily digestible (somewhat improved) Git mirror at https://gitlab.com/YottaDB/DBMS/YDBPIP/

its struck me as interesting ever since. because over the course of my life and career since its seemed that 99%+ of the programmers I knew of were male.

I don't think there is any one right/ideal breakdown by gender, so I just like to have best sense for facts on the ground

This reminds me of a presentation[1] that showed pictures on how early computers were advertised and what roles programmers (women) took in the advertisements, which echoes your statement.

(For context, see response to a comment of mine here[2]; more[3] info[4])

[1] https://youtu.be/5zN83hvn68U [2] https://news.ycombinator.com/item?id=37411221 [3] https://www.bbc.co.uk/programmes/b08wmk5l#playt=0h10m07s [4] https://web.archive.org/web/20190713175933/https://datasocie...

The you had the markets division that would hire in externals to do in parallel dev and then drop the burning corps of the project on internal dev once they moved onto the next shining thing lol.

The timing was just interesting.