The Home Doctor - Practical Medicine for Every Household - is a 304 page doctor written and approved guide on how to manage most health situations when help is not on the way.

My mother has been working for one of the largest banks in the EU since before I was born and I’ve always been fascinated by her line of work, especially these last years since I’ve become a programmer myself. I’ve been asked to interview her plenty of times, and finally decided to do so.

Some notes

The banking programming world is a completely different world than what most of us are used to, and for the next couple of hours after posting this I’ll accept whatever questions you may have for her, but keep in mind that some questions may not be answered due to security concerns. I’ll be posting this to HN & Reddit and will answer questions over there as well.

I won’t write this as a Q&A, but more like I’m telling you her story.

UPDATE: I’ve added Q&A of some FAQ to the bottom of the post!

1991

The year she started internal training at Nordea, which back then was known as Nordbanken (The North Bank) but changed name to Nordea in 2001. During the training she had to take various tests, most notably an IQ-test to see if she had the intelligence to work within this field. Other tests includes a psychological checkup to make sure she had the psyche to handle that line of work and a multitasking test which she failed horribly where she got the score 22/100. She did very good on the other tests and among the 16 available positions, she managed to get one.

The position in question was an IBM Mainframe COBOL programmer, which to this day, 25 years later still work as for the same bank.

This position is the most important one in the bank, at least from a technical standpoint. If, let’s say, my mother and everyone on her team would quit their job, the bank would go under within a matter of weeks if they’re lucky. They have a rotation of people on her team being available 24/7. I remember when I was younger and she had to take a taxi to work in the middle of the night on a Sunday to fix a dead-lock problem.

COBOL…

…is not a fancy programming language like your functional Haskell or concurrent Golang— it’s an imperative, procedural language and since 2002, object-oriented. There’s nothing wrong with the language itself, the problem is that barely anyone knows it — at least not in the context of mainframe programming. My mother is the next youngest person on her team, and she’s born 1964, and the youngest person being 2 years younger. Since almost all of the largest banks in the world runs on IBM Mainframe with COBOL as the primary programming language, this is a global issue. The smaller banks however are better off which usually runs something like Java without mainframes.

My mother used to ask me if I wanted to learn, but since working with the more fancier stuff like Postgres, Redis, Node, Crystal, PHP, among others, I’ve always answered “Never, ever!”. I am still very interested in what she do, but these types of systems gives off the probably worst enterprisy-feeling that you can imagine, which I’d like to avoid.

I can only imagine the fat paycheck a 20-year old mainframe programmer would get though, because your age in this case would be invaluable.

Databases

Their primary database is called IMS, which is an hierarchical database built by IBM for the Apollo program. Internally they call it DL/1 which is short for Database Language One. They are trying to migrate to DB2 which is a relational database that speaks normal SQL, but considering the sheer volume of data that Nordea is storing, this is a task that is going to take years. It’s not as simple as just moving the data from IMS into DB2, they also have to update their modules to load & save data from DB2 instead of IMS and they have thousands of modules, many of which were developed by programmers that have either passed away or have retired.

Each transaction is stored in DB2. They’re avoiding writing to IMS as much as possible, only reading data from it until their newly purchased system is fully integrated and they can start storing data there.

IMS is extremely old and very slow (for some tasks).

Searching for data can take hours. Hah, and here we are, arguing that MySQL delivers 2ms better query speed than Postgres. That’s a bit ironic.

They also use a flat-file structure for a lot of tasks, which themselves has different flavors that IMS supports. One such example is GSAM which has forced my mother to take a taxi into work many times due to modules working on the same GSAM-file at the same time, producing deadlocks.

Let’s talk about the size of their databases for a moment. DB2 is only storing data about transactions, and each transaction is different in size depending on what type of account the transaction goes to/from. Private accounts like my personal bank account is orders of magnitude more simple than bank accounts used by business. Each transaction is somewhere between 2KB to 500 bytes, so let’s say the average is 1KB.

At the moment, their DB2 database holds 11 billion transactions, and the law requires them to save each transaction for 10 years, having settled for 11 years. At the moment, transactions are only 7 years old and the amount of transactions is estimated to grow roughly 5–8% each year, until they hit the 11 year mark where they can nuke transactions that are older than 11 years.

So at the moment, the DB2 database holds about 10TB of data, and that data is only transactions. In 4 years, it’ll be somewhere around 13–14TB of data.

IMS holds most other information. That kind of data differs a lot in size, so it’s hard to estimate how large it is, but I guess it’s much larger than their DB2 database.

They also store some data on tapes, where their programs tell a machine to load/save data from a specific tape, which is pretty cool!

IDE

Everyone has some form of IDE or text editor, right? Well, yes they do. The IDE is called ISPF which is like an entire OS. ISPF can be extended, and the part we would call an IDE is an extension to ISPF which is called Endevor.

ISPF is directly connected to the mainframe, and there’s no such thing as a local development environment here.

Roughly 80% of their systems are batch jobs. These are jobs that runs at a certain time or interval, doing some processing on their data or sends data to other banks/agencies/etc. For example when I buy a can of Coke, the money is withdrawn from my account balance, however the money is not actually transferred anywhere until one of their batch jobs does so. These jobs are usually executed during the night, which is usually why it takes a day before transactions between banks are completed. Transactions to the same bank are usually instant because it executes immediately.

Batch jobs are working with datasets on a gigabyte-, sometimes terrabyte-level and can in some cases take hours to complete. I’d love to see how IBMs mainframes all of a sudden starts working at full capacity the second the clock turns 00:00 in their datacenters, would be pretty cool!

The problems banks are facing

Banks that run on mainframes have quite a lot of issues that they need to take care of, but unfortunately time is scarce.

Programmers are getting older, not many wants to learn and the time before a new employee can stand on their own feet is 2–3 years, and even then there are a lot of gray areas for them.

There are programs that are decades old that nobody even knows what they do and the person who wrote it is long gone.

Many banks are trying to purchase shiny new systems to replace what they currently have, while almost all of their programmers are saying “this is not going to work, you cannot replace this monster that we have here”.

The reason their systems has become so huge is partially because they made the mistake early on to tailor their own system towards other systems. A great example here is how we build RESTful services, where programs that wants to get data from us have to tailor their system to match ours. This is not the case with some banks, instead they’re sending data to other systems (such as the tax agency) which may at any point update their systems which in turn breaks the current system the bank are using to send the data.

Banking systems are also extremely advanced. A personal bank account differs a lot from a business bank account, and there are at least 50 different types of bank accounts for each of them. And in Nordeas case, they also have the Swedish government accounts, which are different from both personal and business accounts. I think they have the Finnish government accounts and maybe a portion of Denmarks as well, which differs too.

It’s going to be very interesting to hear what Nordea and other banks will do these upcoming years, and what new tech they’ll be implemented on.
I may have forgot to ask something, so if you have any questions go ahead and ask, I’ll be forwarding them to her and write down her response for the next couple of hours.

Why did you choose to work with IBM Mainframe COBOL programming?
- I’ve always wanted to “work with computers”, but I wasn’t aiming for this specific job. I saw an ad in the papers and applied and got the job. I had a bit of computer background before I applied though.

What is the worst thing that you’ve seen on a day?
- My co-worker forgot to add a dot to the end of a statement for a module in the most critical part of our system which we call “The Cash Register”. It is the part that handles all the money. The result was that the entire bank was down for 16 hours straight due to the module continiuing executing statements when it actually should’ve stopped after that statement. It basically overloaded our systems, a self-DOS of sorts.

What do you think will happen in the future for banks that run on the same infrastructure as Nordea?
- Most banks has acknowledged that we need to replace the older mainframes with something more modern. Nordea has bought a new system which it has promised to replace the current one within 4 years, but Nordea takes care of multiple countries and a more reasonable number is 4 years per country, so 16 years in total.
Banks and finance that haven’t acknowledged this is going to have a very rough time ahead of them. I still think we’ll live on IBM infrastructure though.

What were the challenges you’ve faced as a female programmer that started in the 90's?
- No problem at all. I’ve got a couple of female co-workers, but most are men. It doesn’t bother me that much.

You’ve been working for the same entity and possibly the same code base for more than 20 years. Does it ever gets old?
- Yeah it definitely gets tiring after a while, like most other lines of work becomes. But during my time here I’ve build many completely new systems for various things in finance, and that has always been fun. Unfortunately from now on we will never build anything new, only maintain what we currently have and wait for the new system we’ve purchased to replace the current one.

How scary is it to write code for a bank?
- It is very scary, especially when we push changes to production which happens on Sundays. Whenever we push new changes to production, huge parts of the system has to be taken offline. One of those parts is the entire IMS. During that period anything can happen, but we have a very robust test environment so it usually goes through smoothly.

Have you caused any huge mistakes for the bank?
- Definitely have, quite a large mistake I did back in 1997, when my youngest son (that’s me, the writer) just started pre-school and my parental leave was over. We have a system for saving money for retirement. Those type of bank accounts wasn’t locked back then, and the law says you cannot withdraw money from that account before you’re 55 years old. Since the accounts wasn’t locked, it was possible to withdraw money if you had the bank account number, so the solution was simple; not give the customer their bank account number.
I managed to screw up by modifying a module that added their bank account number to the mail that was sent out to customers. So what happened was that customers started withdrawing money (which was not taxed yet!) from their retirement savings account before they were allowed to.
This triggered a huge inspection, the Swedish government stepped in, the financial inspection and the media were all over it. That was me.

What’s your working environment like?
- We’ve recently moved to a more “hip” location. We used to have personal desks, but now we have this “pick whatever spot is avaiable” open area. I dislike it a lot.