1) there are old 3rd party dependency incompatibilities that you can spin off and let live separately instead of doing a painful refactor, rebuilding in house, or kludgy gluing

2) there are deploy limitations on mission critical high available systems that should not hold up other systems deployment that have different priorities/sensitive business hours time windows

3) system design decisions that cannot be abstracted away are at the mercy of some large clients of the company that are unable or unwilling to change their way of doing things - you can silo the pain.

And to be clear, it's not that these things are "cost free". It's just a cost that is worth paying to protect the simpler monolith from becoming crap encrusted, disrupted with risky deploys, or constrained by business partners with worse tech stacks.

Correct me if I'm wrong, but isn't "microservices" when you make internal components into services by default, instead of defaulting to making a library or class?

But if you have 20 engineers and 60 services, you're likely in a world of pain. That's not microservices, it's distributed monolith and it's the one model that doesn't work (but everyone seems to do)

Though to understand why this is, you would have to know just how poorly our monolith was designed. That’s sometimes the issue with monoliths though, they allow you to “cut corners” and suddenly you end up with this huge spiderweb mess of a database that nobody knows who pulls what from because everyone has connected some sort of thing to it and now your monolith isn’t really a monolith because of it. Which isn’t how a monolith is supposed to work, but is somehow always how it ends up working anyway.

I do agree though, that the “DevOps” space for “medium-non-software-development” IT departments in larger companies is just terrible. We ended up outsourcing it, sort of, so that our regular IT operations partner (the ones which also help with networking, storage, backups, security and so on) also handle the management part of our managed Kubernetes cluster. So that once something leaves the build pipeline, it’s theirs. Which was surprisingly cheap by the way.

I do get where you’re coming from of course. If we had wanted to do it ourselves, we’d likely need to write “infrastructure as code” that was twice the size of the actual services we deploy.

I find this hard to relate to, the idea you have the discipline and culture to do microservices well if you can't do it with a monolith.

More likely is you migrate away from the monolith you never invested in fixing, and once you get to microservices you either call it a mistake and migrate back, or you have to eventually finally invest in fixing things.

Perhaps your microservice rewrite goes well because you now know your domain after building the monolith, that is another option.

If you gave me total control over everything, I’d probably build a couple of monoliths with some shared modules. But every time the data is centralised, it always somehow ends up being a total mess. With MicroServices you’ll still end up with a total mess in parts of the organisation, but at least it’ll be in something like PowerBI or even your datawarehouse and not directly in your master data.

Or to put it differently, for me MicroServices vs monoliths is all most completely an organisational question and not a technical one.

Yes, by the time you get to thousands of services you hopefully have moved past the distributed monololith, if you built one.

I called ours a macrolith.

One way to mitigate the growing pains of a monolithic backend is to split it into a set of independently deployable services that communicate via APIs. The APIs decouple the services from each other by creating boundaries that are hard to violate, unlike the ones between components running in the same process

We just go from 'one' to 'too many' as equally unworkable solutions to all of our problems, and each side (and each person currently subscribed to that 'side') knows the other side is wrong. The assumption is that this means their side is right instead of reality, which is nobody is right.

The moderates are right, but their answers are wiggly and they're too busy getting stuff done to argue with the purists. But the optics are bad so here we go again on another swing of the swingset.

'Some Services' is probably right, but it varies with domain, company size, and company structure (Conway's Law). And honestly developer maturity, so while 7 may be right for me and my peers today, in a year it might be 6, or 9. There's no clever soundbite to parrot.

Introducing the Revolutionary "Ten Service Applications" – because Ten is the Magic Number!

Tired of the endless debates about how many services your applications should have? Frustrated with the constant struggle to find the "Goldilocks" number of services? Look no further! The future of software design is here, and it's as easy as 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10!

The "Ten Service Applications" model is here to rescue you from your software design woes. We're not messing around with random numbers like 7 or 12 services. No, we've cracked the code, and it's all about that perfect "ten." You don't need any more services, and you definitely don't need any less. Ten is the answer to all your architectural problems!

So, are you ready to embrace the simplicity, predictability, and coolness of the "Ten Service Applications" model? Join the revolution today and experience software development like never before!

Act now, and we'll even throw in a bonus "Top 10 Services" list to inspire your next project. But remember, you only get 10 services, no more, no less—because why mess with perfection?

Going to definitely yoink it for next time I have this discussion. It applies to so many things!

The book isn't perfect. The use of afferent and efferent terminology and some of the arbitrary methods to put numbers on decisions weren't ideal. Most of the concepts are sound. The fact that almost every decision has cost/benefit and real world implications for a living product was refreshing. That a monolith can't be cut over instantly with zero effort to a perfect system is absolutely true.

It's good food for thought for anyone considering slicing up a monolith, but maybe don't follow it to the letter.

They don't seem to understand there's a useful middleground of adding fewer, larger data services, etc. It's like SOA isn't a hot topic so people aren't aware of it.

Being able to easily use different programming languages. Not every language is a good fit for every problem. Being able to write your machine learning deduction services in Python, your server side rendered UI in Rails and your IO and concurrency heavy services in Go might justify the additional overhead of having separate services for these three.

You should always look into existing languages first. There's a lot of "I rewrote this thing into $language for a 100x performance boost" posts, in a lot of cases the comments are like "Yeah but if you rewrote it in the original language you could make it a lot faster too".

For a cool visualisation of this problem, check out https://boringtechnology.club/

Due diligence in these cases is rare.

Never again. When I build teams forevermore, I pick the tech stack and I recruit people who know that tech stack, or want to learn that tech stack. And we stick to that tech stack whenever possible.

I didn’t know how to troubleshoot Node, but I did know how to read docs. Suddenly I can troubleshoot Node. Hooray.

I concur with your sentiment, and would also be an absolute dictator for tech stack and workload management. We’re not using Node, we’re avoiding React if at all possible, every dev touching the DB in any way will know SQL, and no one is using Scrum.

Even with just one, it isn't really a single one.

To pick on JS as example, not only there is JavaScript to learn, TypeScript might also be part of the stack, then there is the whole browser stack, and eventually node ecosystem as well.

Take the remaining UNIX/Windows related knowledge to get development and deployment into production going, SQL and the related stored procedures syntax for RDMS backends.

Eventually the need to know either C or C++, to contribute to V8 native extensions or WASM.

Now those folks need to learn about Go, Go's standard tooling, Go's ecosystem, IDE or programmer's editor customizations for Go code, and how to approach all the development workflows they are confortable of from the point of view of a Go developer.

That said, as the article also mentions, "micro" can be a bit of a misnomer; you could have a 50 person team working on a single "micro" service, in which case things like hiring and training are much more natural. (to add, I've never worked in a team of 50 on one product; 50 people is A Lot)

Too few developers use the facilities available for that kind of in-process isolation, even when it is possible. (Don't tell me Java isn't popular... It may be the new COBOL, but it's still mainstream.)

I think in these discussions, a lot of times people are taking past one another. If I start putting a JRE-targeted application together, I know I can eventually reach for isolated modules if I follow good internal practices, whereas if I'm in Python land it's pretty unattainable.

Also JNI won't be isolated which occasionally might be a factor (more likely for stability).

On the other hand one could make colocated multi-process services to work around this while avoiding all the other complexities of microservices.

If you have to do X,Y,Z to get A done, then you'll do X,Y,Z.

If your language gives you a shortcut and now you can get away with X,Y then you'll drop Z.

And even if your language doesn't give you those shortcuts, if Cool Language lets you do just X or X,Y, you'll naturally want to use Cool Language. So, its a losing game...

KISS is great when applied correctly, but you have to be able to know the level of complexity you'll actually need for the problem at hand, and it is so so very easy to over or under engineer something...

OSGi is the one that's deprecated and Java Modules is the one that can't actually provide that functionality yet, right? Or is it the other way round? Either way you get the point.

Can you actually use them to use two different versions of library C in the same application yet though?

> Eclipse keeps using OSGi just fine

Wasn't the impression I had the last time I tried to fix a bug in an eclipse plugin that touched on the OSGi parts. The codebase felt like a ghost town and I couldn't find any documentation for how it all worked or anyone who knew about it.

OSGi has an in-VM service registry which allows late binding of "services" by interface. This means you can do a lot of sophisticated mixing and matching of capabilities with OSGi, including the ability to load and use incompatible versions of the same library (if need be) in different parts of your code.

More than a decade ago, I built an app server system that used an OSGi base for server-side products and agent systems. I even had a flavor of JBoss running on OSGi (with dramatically faster start up times) before JBoss went OSGi on its own.

But now my teams do their work in Node. Isolation is by process, hopefully, maybe, and we're right smack back in version hell.

That said, almost always, if you are seriously trying to keep old unsupported dependencies running, you’re violating any reasonable security stance.

I’m not saying it never happens, but often the issues that the code is not understood, and no one is capable of supporting it, and so you are deep in the shit already.

You just invented microservices, lol.

https://docs.undo.io/java/Microservices.html

[Edit: for which, by the way, any feedback is welcome - it's technically quite promising but real world experience is worth a lot!]

I’m a fan of microservices btw.

Premature optimization and scaling is almost as bad of a form of technical debt as others when you have to optimize in a completely different manner and direction.

Increasing architectural complexity to enforce boundaries is never a solution to a lack of organizational discipline, but midsize tech companies _incessantly_ treat it like one. If you're having trouble because your domains lack good boundaries, then extracting services _is not going to go well_.

Quoted from: https://www.oreilly.com/library/view/software-architects-han...

By the time your devs are saying "this sucks" you've long overshot.

In my case however, I ended up applying the "LRM" strategy to my own work, where I'm both the decision maker and the sole implementer. This is where I see my mistake. In my defense, the software development books that argued for delaying decisions did not warn that this applies to larger decisions in projects developed by teams, and may not apply to small-scale design decisions made by an individual contributor or a small team in the scope of a small piece of work. It took me way too long to realize that, for most of my day-to-day choices, the cost function is pretty much flat.

My experience is that the new data available when postponing decisions can be very very valuable.

It's absolutely true that postponing a decision lets you take advantage of more data and experience. But making a decision now also generates new data - and often does it better and faster: you're directly observing how your choice plays out in the real world. For cheaply reversible decisions it means that, when things go bad, you can go back and make a new decision, this time informed by knowledge of what went wrong with your previous choice. If you squint, it almost looks like time travel :).

Not every decision is going to be like that, but e.g. in software design, I would often delay deciding between several possible approaches to seek out more information (and/or wait for it to come from customer side), only to later realize that in that time, I could've prototyped most or all of the options, or I could've picked one and made progress and undo it fast when more information came - and that either of those two approaches would've let me to gain the missing data faster, while also keeping forward momentum.

There's a reason many programmers (myself included) need to repeatedly hear the mantra that goes: "make it work, then make it right, then make it fast". In my case, postponing decisions is often a kind of analysis paralysis, and I'm slowly learning that in many cases, it's better to just pick any option, "make it work" in the dumbest, most straightforward way possible, and then reevaluate.

But as I said, I am learning this slowly. My mind knows better, but my heart still wants to delay by default :).

Very few things need to be perfect, or right first try. Waiting until the last possible moment is a version of perfectionism that is often counter-productive. Trying something out before it's needed gives room to experiment and discover new approaches.

[0] https://medium.com/@bre/the-cult-of-done-manifesto-724ca1c2f...

And yet we do this all the time. Your CI/CD blocking your PRs until tests pass? That's a costly technical solution to solve an issue of organizational discipline.

But architectural complexity has outsized long-term cost, and service-orientation in particular has a _lot_ of it. And in this particular case, it doesn't actually solve the problem, since you _can't_ successfully enforce those domain boundaries unless you already have them well-defined.

> But architectural complexity has outsized long-term cost

As do technical solutions, of course. CI/CD systems are very expensive, just from a monetary perspective, but also impose significant burdens to developers in terms of blocking PRs, especially if there are flaky or expensive tests.

> And in this particular case, it doesn't actually solve the problem, since you _can't_ successfully enforce those domain boundaries unless you already have them well-defined.

Ignoring microservices, just focusing on underlying SoA for a moment, the boundary is the process. That is an enforceable boundary. I think what you're saying amounts to, in microservice parlance, that there is no way to prevent a single microservice from crossing multiple bounded contexts, that it ultimately relies on developers. This is true, but it's also just as true for good monolithic designs around modules - there is no technical constraint for a module to not expand into domains, becoming cluttered and overly complex.

Microservices do not make that problem harder, but SoA does give you a powerful technical tool for isolation.

Not concisely in the general case, but in this case the difference is fairly straightforward - CI/CD doesn't affect the structure of your executing application at all, only the surrounding context. I don't want to spend the hours it would take to characterize architecture as distinct from implementation, but the vast number of textbooks on the topic all generally agree that there is one, though they draw the lines in slightly different places.

> I think what you're saying amounts to,

Very much no - my point is about the process of implementation. The services.. _do_ enforce boundaries, but the boundaries they enforce may not be good ones.

In order to successfully extract services from a monolith, you have to go through a process that includes finding and creating those domain boundaries for the domain being extracted. If it's your first time, you might be doing that implicitly and without realizing it's what you're doing, but under the hood it's the bulk of the mental effort.

The part where you actually _introduce a service_ can be anywhere from a tenth to half of the work (that fraction varies a lot by technical stack and depending on how coupled the domain in question is to the central behavioral tangle in the monolith), but by the time you've gotten the domain boundary created you've _already solved_ the original problem. Now you're facing a trade of "extract this well-factored domain out to a separate service application, to prevent its now-clear boundaries from being violated in the future". And I contend that that's a trade that should rarely be made.

The technical part of services is easy enough really but if the organisation leaks its boundaries, which always flow downstream into the actual teams, you are proper fucked.

It takes a different level of maturity, both in organisation and team, to build a service oriented software.

That's my experience at least.

People are not reliable enough to leave them as the only protection against system failure...

I don't think that rule can be applied as universally as you stated it. But then, I have never seen anybody breaking it in a bad way that did also break it in a good way, so the people that need to hear it will have no problem with the simplified version until they grow a bit.

Anyway, that problem is very general of software development methods. Almost every one of them is contextual. And people start without the maturity to discern the context from the advice, so they tend to overgeneralize what they see.

Because the complexity of that outer system is also important, but there are a few very major differences between the two that are probably too obvious to belabor. But in general, architectural complexity in the inner system costs a lot more than it does in the outer system, because it's both higher churn (development practices change much slower than most products) and higher risk (taking production systems offline is much less permissible than freezing deployments)

No, I'm not. Are you overlooking some of the impacts of your tests and most of the impact of static verification?

Those do absolutely impact your system, not only your environment. For tests it's good to keep those impacts at a minimum (for static verification you want to maximize them), but they still have some.

I'm not sure about the tests - it depends on what you'd consider an impact possibly; I've been trained by my context to design code to be decomposable in a way that _is_ easily testable, but I'm not sure that is really an 'impact of the tests' (and I'm fairly confident that it makes the abstractions less complex instead of more).

Would you mind explaining what you mean in more detail?

Like any good test system I have a large suite of "unit" tests that run quick that I run locally before committing code - it takes a few minutes to get high code coverage if you care about that metric. Even then I just run the tests for x86-64, if they fail on arm that is something for my CI system to figure out for me.

If you got _reamed_ you got off lucky: in plenty of places you'd be out on the street.

It may well be that you had it right but from context as given I hope this shows you some alternative perspective that might give you pause the next time you decide to throw out the rulebook, even in emergencies - especially in emergencies - these rules are there to keep you, your team and the company safe. In regulated industries you can multiply all of that by a factor of five or so.

Why? Because it's a "good practice"? They have 12 people and no customers, they can almost certainly adopt a very aggressive developer cycle that optimizes almost exclusively for happy-path velocity. You'd never do that at 50+ engineers with customers but for 12 engineers who have no customers? It's fine, in fact it's ideal.

> with massive downtime as a result.

They have no customers, downtime literally does not exist for them. You are following a dogmatic practice that is optimizing for a situation that literally does not exist within their company.

If you're wondering why software is crap: it is because every new generation of coders insists on making all the same mistakes all over again. Learn from the past, understand that 'good practice' has been established over many years of very expensive mistakes. 12 engineers is already a nice little recipe for pulling in 12 directions at once and even if they're all perfect they can still learn from looking at each others code and it will ensure that there are no critical dependencies on single individuals (which can bite you hard if one of them decides to leave, not unheard of in a startup) and that if need be labor can be re-divided without too much hassle.

Dogmatic adherence to a process that limits developer velocity and optimizes for correct code is very likely the wrong call when you have no customers.

I don't know the context and I don't know the particular business the OP is talking about. What I do know is that if you feel that your management is cargo culting development methodology (which really does happen) you can either engage them constructively or you can leave for a better company. Going in with a confrontational mindset isn't going to be a good experience for anybody involved. Case in point: the OP is still upset enough that he feels it necessary to vent about this in an online forum.

Note that this is the same person who in another comment wrote:

"On the flip side I’m trying to convince my CTO to fire half our engineering team - a group of jokers he hired during the run-up who are now wildly overpaid and massively under-delivering. With all the tech talent out there I’m convinced we’d replace them all within a week."

> Case in point: the OP is still upset enough that he feels it necessary to vent about this in an online forum.

I apologize for the conversation starter.

Your attitude towards code-review on the other hand is one I've seen before several times, and I was glad when each of those people were fired.

If we waited for reviews before merging, we’d be waiting weeks to merge a single PR. Thus, you wrote your code, opened a PR, did a self-review, then deployed it. We had millions of customers, downtime was a real possibility. So you’d watch metrics and revert if anything looked slightly off.

You would wake up to your PR being reviewed. Sometimes there would be mistakes pointed out, suggestions to improve it, etc. Sometimes it was just a thumbs up emoji.

The point is, there are many ways to skin this cat and to “ream” someone for merging without deploying is incredibly immature and uncreative. You can still review a merged PR.

> to “ream” someone for merging without deploying is incredibly immature and uncreative.

I'd agree, but I _highly_ doubt that description was an accurate one. Read through the other comments by the same person and you'll get a picture of their personality pretty quickly.

It's likely that there was already an ongoing conflict either in general or specifically between them about this issue. They probably got a moderately harsh comment to the effect of "hey, you're expected to wait for code-reviews now, knock it off"

You want to get the team to a point where you can disagree and commit, no code will ever be perfect and there is no reason spending 3-4 rounds of change requests trying. I think the worst code review I ever had, ended with me saying, “if you’re going to be this nitpicky, why don’t you take the ticket?” (It was extremely complex and hard to read — and there wasn’t any getting around it, lots of math, bit shifting, and other shenanigans. The reviewer kept making suggestions that would result in bugs, and then make more suggestions…)

He came back the next day and approved my PR once he understood the problem I was trying to solve.

Even these days, where I work on a close team IRL, I’ve been known to say, “if there are no objections, I’m merging this unreviewed code.” And then I usually get a thumbs up from the team, or they say something like “oh, I wanted to take a look at that. Give me a few mins I got sidetracked!” And I’ve even heard, “I already reviewed it, I just forgot to push approve!”

Communication is key in a team. Often, if the team is taking a long time to review, give them the benefit of the doubt, but don’t let yourself get blocked by a review.

Anything else is un-needed grooming that's more about the other developer's ego, not about good code (sometimes its to follow some other constraint, but its a good sign the person has a personality issue).

> They have 12 people and no customers, they can almost certainly adopt a very aggressive developer cycle that optimizes almost exclusively for happy-path velocity. You'd never do that at 50+ engineers with customers but for 12 engineers who have no customers? It's fine, in fact it's ideal.

12 engineers churning out code with no code-review at all? That'll produce velocity, for sure. It'll also produce not just an unmaintainable mess, but an interesting experiment, in which you get to find out which of your engineers are socially capable enough to initiate technical communication independently and construct technical rapport _without_ that process helping them to do so. Hope none of them hold strong technical opinions that clash!

But not being in a regulated industry doesn't mean there isn't a very good reason to have a code review in your process, assuming it is used effectively and not for nitpicking.

Not having a code review step is usually a bad idea, unless everybody on your team is of absolutely amazing quality and they never make silly mistakes. I've yet to come across a team like that, but maybe you are the exception to the rule.

During all this time, I've seen plenty of "small changes" having completely unexpected consequences, and sometimes all it would take to avoid would someone else seeing it from another perspective.

I agree with everything you say here, but honestly it’s quite ineffective. I wish we had found something better than unit tests (often misleading and fragile) and the ability of a qualified reviewer to maintain attention and context that a real review entails.

The primary value I've seen across teams has been more on having shared team context across a codebase, if something goes bump in the night you've got a sense on how that part of the codebase works. It's also a great opportunity for other engineers to ask "why" and explain parts that aren't obvious or other context that's relevant. We'll find the occasional architectural mismatch(although we like to catch those much earlier in the design process) and certainly prevented bugs from shipping but if that's the primary focus I think a team is missing a lot of the value from regular code reviews.

At the low end of engineering, yeah, code reviews matter a ton and do catch bugs even if they’re basically just peephole inspections.

The majority of SOA adoption in small-to-medium tech companies is driven by the wrong type of pain, by technical leaders that can see that if they had their domains already split out into services, their problems would not exist, but don't understand that reaching that point involves _solving their problems first_.

To me, what’s amazing is that in almost no organization is there a requirement to justify technically the addition of a new service, despite the cost and administrative and cognitive overhead of doing so.

_Microservices_ attract the criticism. It seems to assume something about the optimal size of services ("micro") that probably isn't optimal for all kinds of service you can think of.

Now, thanks to massive changes in how software is built (cloud, containers et al) it's a lot more standard for a normal "service" with no prefix to be built by a small team of developers, no micro- prefix needed.

There was a recent thread here where someone mentioned he personally was responsible for about four of their microservices...

> The term micro can be misleading, though - there doesn’t have to be anything micro about the services

There's a few things in play IMO.

One is lack of definition -- what's a "microservice" anyhow? Netflix popularized the idea of microservices literally being a few hundred lines of code maintained by a single developer, and some people believe that's what a microservice is. Others are more lax and see microservices as being maintained by small (4-10 person) development teams.

Another is that most people have not worked at a place where microservices were done well, because they were implemented by CTOs and "software architects" with no experience at companies with 10 developers. There are a lot of problems that come from doing microservices poorly, particularly around building distributed monoliths and operational overhead. It's definitely preferable to have a poorly-built monolith than poorly-built microservice architectures.

I've been at 4 companies that did microservices (in my definition, which is essentially one service per dev team). Three were a great development experience and dev/deploy velocity was excellent. One was a total clusterfuck.

> It's definitely preferable to have a poorly-built monolith than poorly-built microservice architectures.

I don't know about "definitely" at all. Having worked with some horrible monoliths, I really don't think I agree. Microservices can be done poorly but at minimum there's a fundamental isolation of components. If you don't have any isolation of components it was never even close to microservices/SoA, at which point, is it really a fair criticism?

How small is small? Even within this comment section there are people talking about a single developer being the sole maintainer of multiple microservices. I'm a strong advocate of (micro?)service architecture but I would never recommend doing the "all behavior is 100-line lambda functions" approach.

A horrible monolith vs horrible microservices is subjective, of course, but IMO having everything self-contained to one repository, one collection of app servers, etc. at least gives you some hope of salvation, often by building new functionality in separate services, ironically. Horrible microservices that violate data boundaries, i.e. multiple services sharing a database which is a sadly common mistake, is a much harder problem to solve. (both are bad, of course!)

> IMO having everything self-contained to one repository

I highly recommend keeping all microservices in a single repository. It's even more important in a microservice world to ensure that you can update dependencies across your organization atomically.

> Horrible microservices that violate data boundaries, i.e. multiple services sharing a database which is a sadly common mistake, is a much harder problem to solve.

But that's not microservices. Maybe this is in and of itself an issue of microservice architecture, the fact that people think they're implementing microservices when they're actually just doing SoA, but microservice architecture would absolutely not include multiple services with a single database, that would not be microservices.

So I think the criticism would be "people find it hard to actually implement microservices" and not "microservice architecture leads to these problems", because microservice architecture is going to steer you away from multiple services using one database.

1. The "data service" layer, which (if done improperly) is basically just a worse SQL implemented on top of HTTP but still centralised. Though now you can claim it's a shared service instead of a DB.

2. The "fat cache" database - especially common in strongly event-based systems. Basically every service decides to store whatever it needs from events to have lower latency access for common data. Sounds great but in practice leads to (undocumented) duplicate data, which theoretically should be synchronised but since those service-local mirror DBs are usually introduced without central coordination it's bound to desync at some point.

Remember this is how AWS was born, as a set of "microservices" which could start being sold to external customers, like "storage".

I think the problem is the word "micro". At my company I see a lot of projects that are run by three devs that and have 13 microservices. They are easy to develop but the maintenance overhead is enormous. And they never get shared between projects so you have 5 services that do basically the same.

And the process seems almost straightforward until you _actually try to do it_, and find out that it's actually fractally difficult - by that point you've committed your organization and your reputation to the task, and "hey, that was a mistake, oops" _after_ you've sunk that kind of organizational resources into such a project is a kind of professional suicide.

Many HN patrons are actually working where the rubber meets the road.

DNS is a poor comparison. Pretty much everything, related to your application or not, needs DNS. On the other hand, the only thing WNGMAN[0]may or may not do, is help with finding the user’s DOB.

https://m.youtube.com/watch?v=y8OnoxKotPQ

    Microservices
    
    grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too

    seem very confusing to grug

"We want to modernize how we access our FOO_TABLE for SCALE_REASONS by moving it to DynamoDB out of MySQL - unfortunately, 32 of our 59 teams are directly accessing the FOO_TABLE or directly accessing private methods on our classes. Due to competing priorities, those teams cannot do the work to move to using our FOO_SERVICE and they can't change their query method to use a sharded table. To scale our FOO_TABLE will now be a multi-quarter effort providing the ability for teams to slow roll their update. After a year or two, we should be able to retire the old method that is on fire right now. In the meanwhile, enjoy oncall."

Compare this to a microservice: Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. Users of the API report that it is now much faster.

Bologna. Choosing clear abstractions is an enabler of focus, but that doesn’t necessarily imply those abstractions are a network call away.

We have an org wide push to get more linting and more checks in place. The damage is done and now we have a multi-quarter effort to re-organize all our code.

This _can_ be enforced via well designed modules. I've just not seen that succeed. Anywhere. Microservices are a pain for smaller teams and you have to have CI and observability and your pains shift and are different. But for stepping on eachother? I've found microservices to be a super power for velocity in these cases. Can microservices be a shitshow? Absolutely, esp. when they share data stores or have circular dependencies. They also allow teams to be uncoupled assuming they don't break their API.

Well, that's different.

My experience is that "leadership" often finds quality to be expensive and unnecessary overhead.

That's one reason that I stayed at a company that took Quality seriously. It introduced many issues that folks, hereabouts would find unbearable, but they consistently shipped some of the highest-Quality (and expensive) kit in the world.

Quality is not cheap, and it is not easy. It is also not really the path to riches, so it is often actively discouraged by managers.

Really, you can't? Then I struggle to see how'll get anything else right. I've done it by using separate build scripts. That way only the interfaces and domain objects are exposed in the libraries. Now you lock down at the repository level access to each sub-project to the team working on it. There you go: modularity, boundaries, all without network hops.

With an HTTP API boundary, you don't get to reach into my code - it is literally impossible.

Presumably you would still need tooling to enforce teams not breaking their API, and also to prevent people just modifying services to expose private internals that should not be part of the public API?

I mean take your pick, anything open source be it desktop or web has a huge and deep dependency tree all the way down to libc.

Just wondering.

EDIT: Typos and desktop.

1) engineering needs a reason to embrace abstraction. Because the only thing that stops a new cowboy engineer is their peers explaining to them in what ways this isn't the Wild Wesst. One can assume if rules would benefit the team, they'd be doing them already, so why don't they? Maybe they perceive feature output velocity to be too high to risk changing method. Maybe decisionmaking power rests in the hands of one system architect who is holding the whole machine in their head so things that look complex to others seem simple to them (in that case, the team needs to spread around peer review signoff responsibilities on purpose, so one engineer can't be the decisionmaker and the architecture itself must self-describe). Maybe (this is how I see it usually happen) they were a three-person startup and complexity crept up on them like a boiling frog. Whatever the reason, if you're gonna convince them otherwise, someone's gonna have to generate hard data on how changing the abstraction could make their jobs easier.

2) If management has no idea what "prioritize quality" means (meaning no metrics by which to measure it and no real grasp of the art of software engineering), the engineers will interpret buzzwords as noise and route around them. Management needs to give actionable goals other than "release feature X by Y date" if they want to change engineering culture. That can take many forms (I've seen rewarded fixit weeks and externally-reported issue burndown charts as two good examples).

3) Engineering leadership needs time and bandwidth to do training so they can see outside their prairie-dog hole over to how other teams solve problems. Otherwise, they get locked into the solutions they know, and the only way new approaches ever enter the team is by hires.

And the key thing is: microservices may not be the tool for the job when all is said and done. This is an approach to give your engineering team ways to discover the right patterns, not to trick them into doing microservices. Your engineering team, at the end of the day, is still the best-equipped people to know the machinery of the product and how to shape it.

Instead of microservices, you could add a static analysis build step that checks if code in packages is calling private or protected interfaces in different packages. That would also help enforce service boundaries without introducing the network as the boundary.

With microservices, you deploy the updates to public facing services first, write any learnings down, pass it along to the next most vulnerable tier.

Heck on multiple occasions I've been part of projects where just updating the build system for a monolithic codebase was a year+ long effort involving dozens of engineers trying to work around commits from everyone else.

Compare this to a microservice model where you just declare all new services get the new build tools. If the new tools come with a large enough carrot (e.g. new version of typescript) teams may very well update their own build tooling to the latest stuff without anyone even asking them to!

- Performance optimizations which can't be realized without changing the interface model. For example, FOO_TABLE should actually be BAR and BAZ with different update patterns to allow for efficient caching and querying.

- Domain model updates, adding/updating/removing new properties or entities.

This kind of update will still require the 32 consumers to upgrade. The API-based approach has benefits in terms of the migration process and backwards-compatibility though (a HTTP API is much easier to version than a DB schema, although you can also do this on the DB level by only allowing queries on views).

This definitely happens but at some point someone with authority needs to show technical leadership and say "you cannot do this no matter how desperately you need those reports." If you don't have anyone in your org who can do that, you're screwed regardless.

Once you get to this point, there's no path forward. Either you have to making some breaking changes or your product is calcified at that point.

If this is a real concern then you should be asking what you can do to keep from getting into that state, and the answer is encapsulating services in defined interfaces/boundaries that are small enough that the team understands everything going on in the critical database layer.

The team that maintains the service is also responsible for how that service is represented in the data warehouse.

The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such.

If the team refactors, they also update the scripts that populate the data warehouse.

If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.

I honestly expected people used passive-aggressive corpo-slang only for work / ironically. But this reads intentionally obfuscated.

But, to answer to the substance: you for some reason assumed that whoever designed first solution was an idiot and whoever designed the second was, at least clairvoyant.

The problem you describe isn't a result of inevitable design decisions. You just described a situation where someone screwed up doing something and didn't screw up doing something else. And that led you to believe that whatever that something else is, it's easier to design.

The reality of the situation is, unfortunately, the reverse. Upgrading microservices is much harder than replacing components in monolithic systems because it's easier to discover all users of the feature. There are, in general, fewer components in monolithic systems, so less things will need to change. Deployment of a monolithic system will be much more likely to discover problems created by incorrectly implemented upgrade.

In my experience of dealing with both worlds, microservices tend to create a maze-like system where nobody can be sure if any change will not adversely affect some other part of the system due to the distributed and highly fragmented nature of such systems. So, your ideas about upgrades are uncorroborated by practice. If you want to be able to update with more ease, you should choose a smaller, more cohesive system.

There are definitely reasons why a microservice architecture could make sense for an organization, but "we don't have good application engineers/architects" should not be one of them. Going to a distributed microservice model because your teams aren't good enough to manage a modular monolith sounds like a recipe for disaster.

When I hear success stories like that, I have to ask "Is there some inherent benefit to the abstraction or did you get lucky in picking your cleave-points?"

Dry is literally "Don't Repeat Yourself" and is definitely pushed for cleaning up redundant code, so it's not unreasonable for people to think that's what about. It's only recently that people have pointed out that there's a difference between Duplicated code and Repeated code.

https://ruby-doc.org/core-3.1.0/NEWS_md.html#label-IRB+Autoc...

Isolation is not anything to scoff at, it's one of the most powerful features you can encode into your software. Isolation can improve performance, it can create fault boundaries, it can provide security boundaries, etc.

This is the same foundational concept behind the actor model - instead of two components being able to share and mutate one another's memory, you have two isolated systems (actors, microservices) that can only communicate over a defined protocol.

That is not too true at all. I've seen "microservice" setups where one microservice depends on the state within another microservice. And even cases where service A calls into service B which calls back into service A, relying on the state from the initial call being present.

Isolation is good, but microservices are neither necessary nor sufficient to enforce it.

Let me be a bit clearer on my point because I was wrong to say that you have to treat a service as being totally isolated, what I should have said that they are isolated, whether you treat them that way or not. There is a physical boundary between two computers. You can try to ignore that boundary, you can implement distributed transactions, etc, but the boundary is there - if you do the extra work to try to pretend it isn't, that's a lot of extra work to do the wrong thing.

Concretely, you can write:

    rpc_call(&mut my_state)

But shared mutable state is trivial without the process boundary. It's just... every function call. Any module can take a mutable pointer and modify it. And that's great for lots of things, of course, you give up isolation sometimes when you need to.

If your web service is generally low-state and these problems are manageable for the complexity scale you're solving for, microservices aren't really something to even consider - I mean, you basically have a microservice already, it's solving a single problem within a bounded context, give or take. It's just... one service and one context.

a) Intermittent failures force you to treat systems as if they can fail - and since every system can fail, that's a good thing. This is why chaos engineering is great.

b) Failures across a network are the best kind - they're totally isolated. As I explain elsewhere, it's impossible to share state across a network, you can only send copies of state via message passing.

These things matter more or less depending on what you're doing.

On top of this, you need to consider that most of the software you are going to write will be based on existing components. Many of these have no desire to communicate over network, and your micro- or w/e size services will have to cave in to their demands. Simple example: want to use Docker? -- say hello to UNIX sockets. Other components may require communication through shared memory, filesystem, and so on.

Finally, isolation is not a feature of microservices, especially if the emphasis is on micro. You have to be able to control the size and where you want to draw the boundary. If you committed upfront to having your units be as small as possible -- well, you might have function-level isolation, but you won't have class- or module- or program-level isolation, to put it in more understandable terms. This is where your comparison between the actors model and microservices breaks: first doesn't prescribe the size.

http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...

In no way is this throwing out performance. It's sort of like saying that Kafka is in Java so you're throwing away performance when you use it, when there are massive performance benefits if you leverage partition isolation.

> Many of these have no desire to communicate over network, and your micro- or w/e size services will have to cave in to their demands. Simple example: want to use Docker? -- say hello to UNIX sockets. Other components may require communication through shared memory, filesystem, and so on.

I'm not sure what you're referring to. Why would that matter at all? I mean, ignoring the fact that you can easily talk to Docker over a network.

> Finally, isolation is not a feature of microservices,

Isolation is a feature of any process based architecture, whether it's SoA, actors, or microservices.

> well, you might have function-level isolation, but you won't have class- or module- or program-level isolation,

You get isolation at the service layer. I don't see why that would be contentious, it's obvious. If you're saying you want more isolation, ok, you can write your code to do that if you'd like.

> first doesn't prescribe the size.

Yep, the actor model is very low level. Microservice architecture is far more prescriptive. It's one of the reasons why I think Microservice architecture has been far more successful than actor based systems.

So, here are some concerns that affect performance, they mostly come from the need of various translations done by either iptables, eBPF analogues, arpatables, DNS server(s). If you read about benchmarks of Calico and Cilius, you'll see that they concentrate on performance of eBPF code necessary to do all these translations. My claim about performance tax is based on the idea that w/o Kubernetes you wouldn't need such translations (but, of course, you could build your own version of Kubernetes networking with a lot of software-defined virtualization, in which case you'd be in the same boat).

> Kafka

Is as horrid as it sounds. It's awful performance-wise on all counts. I'm not sure what point are you trying to make. Can you choose a better example? I mean, Kafka performs very poorly in all configurations, so, to think that it can be a good example of software that wants to achieve good resource utilization is just bound to give you bad results.

> You get isolation at the service layer. I don't see why that would be contentious, it's obvious. If you're saying you want more isolation, ok, you can write your code to do that if you'd like.

You either genuinely didn't understand what this is about, or pretend to not understand something really simple. "Micro" in microservices means that your services are small. There aren't any meaningful isolation tools or approaches when it comes to microservices, because isolation at the level of service doesn't matter / is trivial to achieve by many other means / is not a problem in real-world programs.

> Yep, the actor model is very low level.

This is simply a nonsense statement. Low on what scale? Your answer reads as if it was generated by a chatbot. I.e. words from the same general domain strung together, but make no sense.

> Microservice architecture has been far more successful than actor based systems

How did you count? How do you even tell if something is a microservice-based? This is just as absurd of a claim as saying "78% of enterprises use Kubernetes" (I believe I saw this unfettered inanity on cloud-native foundation's Web site). How do you tell if it's successful? What if actor model is a more generic description which beside other things, also captures microservices?

I mean, in more simple language, this is talking out of your rear. It's not a real argument anyone should pay attention to.

You're the one who brought it up?

> My claim about performance tax is based on the idea that w/o Kubernetes you wouldn't need such translations (but, of course, you could build your own version of Kubernetes networking with a lot of software-defined virtualization, in which case you'd be in the same boat).

You don't need any of those and I don't know why you think otherwise. You can just use the host network and do whatever you want, as with any container.

> I'm not sure what point are you trying to make.

That Kafka's architecture allows you to put data into partitions and route it based on that data, which allows for "shared nothing" architectures. But whatever, you're clearly not going to get this point from this example. To be clearer, your point of "you pay a cost here so you're losing performance" ignores that you can get performance elsewhere.

> There aren't any meaningful isolation tools or approaches when it comes to microservices, because isolation at the level of service doesn't matter / is trivial to achieve by many other means / is not a problem in real-world programs.

Not true at all. Services that own a domain of work are a great place to perform isolation and security boundaries.

> Low on what scale?

As in an actor is a foundational primitive for asynchronous computation. You have to build up protocols on top of actors, hence all of OTP.

> I.e. words from the same general domain strung together, but make no sense.

I think that's because I actually know what I'm talking about and you're finding it hard to keep up?

> How did you count?

Because it's obvious? Like it's not even close. Actor based systems are exceedingly rare, microservices are not.

> I mean, in more simple language, this is talking out of your rear. It's not a real argument anyone should pay attention to.

One of us actually knows what they're talking about and I doubt we'll agree who ti is.

First, a vortex in a software system is any loop in a data flow. For example, if we send data somewhere, and then we get them back processed, or are in any way influenced by them, we have a vortex. A mutable variable is an example of a really small vortex.

Now, the single vortex principle states that there ideally should be only one vortex in the software system, or, restated, every component should know which way its vortex is going.

The rationale is when we have two vortices, and we want to compose the modules that form them into a single whole. If the vortices are correctly oriented, composition is easy. If they have opposite orientation, the composition is tricky and requires decision on how the new vortex is oriented. Therefore, it is best if all the vortices in all the modules have the same orientation, and thus form a single vortex.

This principle is a generalization of ideas such as Flux pattern, CQRS, event sourcing, and immutability.

When I look at an architecture diagram, half the battle is trying to work out what is async/sync, push/pull and the direction of travel and ordering.

Enterprise service buses and message queues are kind of solutions to the creating vortexes where there is not a matching direction on each side.

[1] https://www.youtube.com/watch?v=N7agCpAYp1Q

[2] https://en.wikipedia.org/wiki/Acyclic_dependencies_principle

This is the #1 reason we quit the microservices game. It is simply a complete waste of mental bandwidth to worry about with the kind of tooling we have in 2023 (pure cloud native / infiniscale FaaS), especially when you have a customer base (e.g. banks & financial instutitions) who will rake you over hot coals for every single 3rd party dependency you bring.

We currently operate with one monolithic .NET binary distribution which is around 250 megs (gzipped). Not even the slightest hint of cracks forming. So, if you are sitting there with a 10~100 meg SaaS distribution starting to get nervous about pedantic things like "my exe doesnt fit in L2 anymore", then rest assured - Your monolithic software journey hasn't even begun yet.

God forbid you find yourself with a need to rewrite one of these shitpiles. Wouldn't it be a hell of a lot easier if it was all in one place where each commit is globally consistent?

Microservices doesn't need to mean it's the wild west and every team can act without considering the larger org. There can and should be rules to keep a certain level of consistency across teams.

Functional decomp is left to different teams. But the libraries for logging, a&a, various utilities etc are common.

No microservices that don’t meet the stack unless they’re already developed/open source - eg open telemetry collectors.

Edit: I think the article is a path to a book written by the author. It’s more of an advertisement than an actual assessment. At least that’s my take on it.

This works well if you have knowledge redundancy in your organization, i.e., multiple teams that are experienced in each programming language. This way, if one or more developers experienced in language 'A' quit, you can easily replace them by rearranging developers from other teams.

In small companies, this flexibility of allowing multiple languages can result in a situation in which developers moving to other jobs or companies will leave a significant gap that can only be filled with recruiting (then onboarding), which takes much more time and will significantly impact the product development plan.

More often than not, the choice between Microservices and Monoliths is more of a business decision than a technical one to make.

I think that, technically you can use one or the other and make it work. However management is very different in the two cases, so I completely agree with you. I hadn't thought of the part about moving people between teams.

It's my first job but I understand why they chose microservices : 6 teams working on 6 "features/apps" can be managed (almost) fully independently of each other if you split your code base.

It takes effort to keep a monolithic application set up this way, but IMHO the effort is far less than moving to and maintaining microservices. I think a problem is that there's very popular ecosystems that don't have particularly good tooling around this approach, Python being a major example--it can be done, but it's not smooth.

To me the time when you pull the trigger on a different service+codebase should not be code complexity, because that can be best managed in one repo. It is when you need a different platform in your ecosystem (say your API is in Java and you need Python services so they can use X Y or Z packages as dependencies), or when you have enough people involved that multiple teams benefit from owning their own soup-to-nuts code-to-deployment ecosystem, or when, to your point, you have a chunk of code that the team doesn't or can't own/maintain and wants to slowly branch functionality away from.

Could you elaborate on this? I see how "separating data from logic" is a problem but what about the other two?

If you want to enforce a clean interface between an in-process client and server (i.e. a piece of code calling a library interface), then the best model is to think of it as a message passing system, where once a payload is passed from the client to the server and vice versa, the other side should not be able to witness changes to the payload. An immutable payload in this context is the same as the json that goes over the wire between a client and server.

If you really wanted an in-process application to look like a microservice you could take the added step of forcing a serialization / deserialization step on both client and server. I've seen frameworks that do this. But I think immutability, if it can be enforced, is a practical way of solving this problem and far less complex.

Inheritance is a hazier point I was making, in hindsight, because you could use inheritance for data modeling, which is quite fine in some situations... so I think it's effectively subsumed under the "separating data from logic" argument: which is that if you're passing behavioral inheritance from a "server" to a "client", then it gets harder and harder to predict how that client is going to use it and thus it's harder to reason about the functional boundary between two pieces of code. But it's a hazy point because I think the larger and more important point to make, as you point out, is that you simply shouldn't (in most cases) pass any kind of behavior between client and server--just data.

Another approach to look at (besides taking inspiration from Smalltalk) is the actor model, where it's incredibly clear that any communication between modules in an ecosystem is immutable messages. In fact a side effect of the actor model is that it gets pretty easy to move things from in-process to cross-process because most activity in the system is already performed through message passing, so you can just start channeling messages from Actor A to Actor B through a serialization/networking layer rather than in-process if you want to deploy them separately for reasons of convenience or computational needs.

"which is that if you're passing behavioral inheritance from a "server" to a "client", then it gets harder and harder to predict how that client is going to use it"

You mean a "server" returning the base class object whose methods are overridden by a successor class known only to the "server"?

Say you have Libraries A, B, and C, where B and C depend on A and not one another. You can have B call into C via a service, just as you would in a microservice. Now B and C can be versioned and deployed independency. You can also deploy updates to Library A incrementally, tying it to the B and C deployments.

If you are literally pushing different features that are in fact unrelated, you don't even need to worry about B calling into C, you can just partition your app into different modules, deploy them separately, and use a load balancer with routing rules to arbitrate between the deployments.

I like having this type of environment because you can make fairly quick and easily-resersible decisions about whether or not different parts of the codebase are deployed differently: sometimes it's compute and hardware requirements, sometimes it's because you want parts to be stable and other parts more experimental and volatile.

The microservice argument isn't addressing this type of deployment scenario: it's suggesting a shared-nothing or shared-little architecture across services.

git revert

What I mean is, if you have v2 of your software that introduces a bugfix for module A that's perfectly fine and a bugfix for module B that breaks everything, you can just roll back just module B directly with your deployment pipeline.

There's no need to go back to the code base and make a new release.

So then, not Rails apps

Though I'm seeing a lot of convergence between environments over time, which makes me think we're all headed towards a nicer future.

For example in Scala, which does a lot of type inferencing, it's typical to tell the linter to require that public methods have explicit types, even though the compiler will reify them at compile time anyhow, because that makes the interface much more robust to refactoring. Meanwhile in a more dynamic environment, in Python it's getting more typical to use type annotations, and, similarly, to especially use them on functions that define a reusable interface.

I figure that the ideal languages in the future have module-level systems where they can define strict interfaces across modules, but then get as crazy and dynamic as they want inside of the modules.