CPU reservations are limits, just implicit ones and declared as guarantees.

So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt.

Limits allow me to simulate the same behavior and plan for it by doing the right capacity planning.

They are not the only way to approach it! But they are the simplest way to so it.

The best way to utilize resources is to overcommit, and the smart way to overcommit is to, say, allow 4x overcommit with each allocation limited to 1/4th of the available resources so not individual peak can choke the system. Given varied allocations, things average out with a reasonable amount of performance variability.

Idle CPUs are wasted CPUs and money out the window.

This assumes that the scheduled workloads are created equal which isn't the case. The app owners do not have control over what else gets scheduled on the node which introduces uncontrollable variability in the performance of what should be identical replicas and environments. What helps here is .. limits. The requests-to-limits ratio allows application owners to reason about the variability risk they are willing to take in relation to the needs of the application (e.g. imagine a latency-sensitive workload on a critical path vs a BAU service vs a background job which just cares about throughput -- for each of these classes, the ratio would probably be very different). This way, you can still overcommit but not by a rule-of-thumb that is created centrally by the cluster ops team (e.g. aim for 1/4) but it's distributed across each workload owner (ie application ops) where this can be done a lot more accurately and with better results. This is what the parent post is also talking about.

> This assumes that the scheduled workloads are created equal which isn't the case.

This particular allocation technique benefits from scheduled workloads not being equal as equality would increase likelihood of peak demand collisions.

You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.

But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incidents will be created for conditions that may solve themselves because of another deployment (possibly of another team) happening.

Overcommit/bursting is an advanced cost saving feature.

Let me say it again: splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature!

The problem is that the request/limits feature is presented in the configuration spec and in the documentation in a deceptively simple way and we're tricked to think it's a basic feature.

Not all companies have ops teams that are well equipped to do more sophisticated things. My advice for those teams who cannot setup full automation around capacity management is to just not use this advanced features.

An alternative is to just use smaller dedicated nodes and (anti)affinity rules, so you always understand which pods go with which other pods. It's clunky but it's actually easier to reason about what's going to happen.

EDIT: typos

That's an interesting take. For those of us who once used microcomputers back in the 1980s like Commodore 64s, Apple IIs, and IBM PCs running MS-DOS, sure. But time-sharing systems have been around since the 1960s and Unix dates back to the 1970s, and multi-user logins and resource controls like ulimits have been part of the picture for a very long time.

We've had plenty of time to get used to multitasking and resource contention and management. Is it complicated? It can be. But if you consider that containers are basically a program lifecycle/deployment and resource control abstraction on an OS that's had these features (maybe not cgroups and namespaces, but similar ones) since its birth, it's not really all that advanced.

I would be curious to see this discussed but your article only states that people think you need limit to ensure CPU for all pods.

You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point.

> You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point.

Evicted due to node pressure - yes (but if all other pods also don't have limits it doesn't matter). For preemption QoS is not factored in the decision [0]

[0] - https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...

I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpus" allocated to the job, but upon further reflection, that probably wouldn't work for many reasons.

(There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...)

flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that).

I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.

Caveats: up to the number of cores exposed to your VM. I also believe the later versions of K8s let you do some degree of workload-core pinning and I don’t yet know how that interacts with core availability .

I know the JVM automatically uses it, and there's a popular library for Go that sets GONAXPROCS using it.

Been a bit but I do believe that dotnet does this exact behavior. Sounds like gunicorn needs a pr to mimic, if they want to replicate this.

https://github.com/dotnet/runtime/issues/8485

The true objective function I'm trying to optimize isn't just "save money" or "don't waste CPU resources", but rather "get a million different workloads to run smoothly on a large collection of resources, ensuring that revenue-critical jobs always can run, while any spare capacity is available for experimenters, up to some predefined limits determined by power capacity, staying within the overall budget, and not pissing off any really powerful users." (well, that's really just a simplified approximation)

At work at my new job we run mixed workloads in k8s and I don't really see a problem, but we also don't instrument well enough that I could say for sure. In our case it usually just makes sense to not oversubscribe machines (Google oversubscribed and then paid a cost due to preemptions and random job failures that got masked over by retries) by getting more machines.

If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%.

This isn’t the same as cpusets, which work differently.

That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo...

It's a hard time limit. It doesn't care about contention at all.

`cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.

Not at my work environment right now, don't remember the parameters I used.

    docker run --cpus 0.1 --rm -it progrium/stress --cpu 16

> Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then print the number of processors installed

https://www.gnu.org/software/coreutils/manual/html_node/npro...

https://www.flamingspork.com/blog/2020/11/25/why-you-should-...

    if [[ -e /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]] && [[ -e /sys/fs/cgroup/cpu/cpu.cfs_period_us ]]; then
        GOMAXPROCS=$(perl -e 'use POSIX; printf "%d\n", ceil($ARGV[0] / $ARGV[1])' "$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" "$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)")
    else
        GOMAXPROCS=$(nproc)
    fi
    export GOMAXPROCS

Or use https://github.com/uber-go/automaxprocs, which is very comprehensive, but is a bunch of code for what should be a simple task.

This exists: https://github.com/lxc/lxcfs

lxcfs is a FUSE filesystem that mocks /proc by inferring cgroup values in a way that makes other applications and libraries work without having to care about whether it runs in a container (to the best of its ability - there are definitely caveats).

One such example is that /proc/uptime should reflect the uptime of the container, not the host; additionally /proc/cpuinfo reflects the number of CPUs as a combination of cpu.max and cpuset.cpus (whichever the lower bound is).

As others also mentioned, inferring the number of CPUs could also be done using the sched_getaffinity syscall - this doesn't depend on /proc/cpuinfo, so depending on the library you're using you might be in a pickle.

This is new to me. What is this… behavior? What keywords should I use to find any details about it?

The only thing that rings a bell is requests/limit parameters of a pod but you can't change them on an existing pod AFAIK.

What I tried: I created a "Burstable" Pod and run `nproc` [0] on it. It returned N CPUs (N > 1).

Then I created a "Guaranteed QoS" Pod with both requests and limit set to 1 CPU. `nproc` returned N CPUs on it.

I went back to the "Burstable" Pod. It returned N.

I created a fresh "Burstable" Pod and run `nproc` on it, got N again. Please note that the "Guaranteed QoS" Pod is still running.

> Pods with Guaranteed QoS will have exactly the number of CPUs they asked for, no more or less

Well, in my case I asked for 1 CPU and got more, i.e. N CPUs.

Also, please note that Pods might ask for fractional CPUs.

[0]: coreutils `nproc` program uses `sched_getaffinity` syscall under the hood, at least on my system. I've just checked it with `strace` to be sure.

FWIW, `taskset` uses the same syscall as `nproc` (according to `strace`).

  # taskset -c -p 1
  pid 1's current affinity list: 1

  # nproc
  1

In my case it's `"cpuManagerPolicy": "none"` and I suppose you're using `"static"` policy.

Well, TIL. Thanks!

Just setting it to 16 will satisfy 99% of users.

Fundamentally, intelligent apps need to interrogate their environment to make concurrency decisions. But I agree- Go would probably work best if it just picked a standard parallelism constant like 16 and just let users know that can be tuned if they have additional context.

Most people are running on systems where the CPU capacity varies and they haven't even noticed. For example in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are faster than the others. Practically nobody even notices this effect.

Fascinating. Could you share any (all) more detail on this that you know? Is it a specific instance type, only ones that use nitro? (or only ones without?) This might be related to a problem I've seen in the wild but never tracked down...

Oh they did, they're a modern day IBM.

> Containers are a crappy abstraction

They're one of the best abstractions we have (so far) because they contain only the application and what it needs.

Delusion level: over 9900.

I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed to stay.

From Linux docs:

  - cpu.shares: The weight of each group living in the same hierarchy, that
    translates into the amount of CPU it is expected to get. Upon cgroup creation,
    each group gets assigned a default of 1024. The percentage of CPU assigned to
    the cgroup is the value of shares divided by the sum of all shares in all
    cgroups in the same level.
  - cpu.cfs_period_us: The duration in microseconds of each scheduler period, for
    bandwidth decisions. This defaults to 100000us or 100ms. Larger periods will
    improve throughput at the expense of latency, since the scheduler will be able
    to sustain a cpu-bound workload for longer. The opposite of true for smaller
    periods. Note that this only affects non-RT tasks that are scheduled by the
    CFS scheduler.
  - cpu.cfs_quota_us: The maximum time in microseconds during each cfs_period_us
    in for the current group will be allowed to run. For instance, if it is set to
    half of cpu_period_us, the cgroup will only be able to peak run for 50 % of
    the time. One should note that this represents aggregate time over all CPUs
    in the system. Therefore, in order to allow full usage of two CPUs, for
    instance, one should set this value to twice the value of cfs_period_us.

This is rather strong language without any real qualifiers. It is definitely not "mostly useless". Shares and quotas are for different use-cases, that's all. Understand your use-case and choose accordingly.

When you want to limit the max CPU time available to a container use quotas (--cpus). When you want to set relative priorities (compared to other containers/processes), use shares.

These two options can be combined, it all depends on what you need.

One caveat is that an application can detect when --cpu is used as I think it's using cpuset. When quota are used it cannot detect and more threads than necessary will likely be spawned

I'll try and clarify this. I think this is how the sympton presents but I should be clearer.

If you set request / limit to 1 core but you run on 64 cores node , then you runtime will see that which will bring performance down.

I still believe it's best to set these variables regardless of cpu limits and/or cpu shares

Other people have encountered it too hence libraries like Automaxprocs existing and issues being open with Go for it.

This aligns with what others have reported on the Go runtime issue open for this.

"When go.uber.org/automaxprocs rolled out at Uber, the effect on containerized Go services was universally positive. At least at the time, CFS imposed such heavy penalties on Go binaries exceeding their CPU allotment that properly tuning GOMAXPROCS was a significant latency and throughput improvement."

https://github.com/golang/go/issues/33803#issuecomment-14308...

Although I saw this dissonance between actual and assigned CPU causing issues, particularly CPU throttling, I struggled to find a scalable solution that would affect all Go deployments on the cluster.

Getting all devs to include that autoprocs dependency was not exactly an option for hundreds of projects. Alternatively, setting all CPU request/limit to a whole number and then assigning that to a GOMAXPROCS environment variable in a k8s manifest was also clunky and infeasible.

I ended up just using this GOMAXPROCS variable for some of our more highly multithreaded applications which yielded some improvements but I’ve yet to find a solution that is applicable to all deployments in a microservices architecture with a high variability of CPU requirements for each project.

And as a maintainer of ko[1], it was a pleasant surprised to see ko mentioned briefly, so that's for that too :)

1: https://ko.build

- "The JVM has been modified to be aware that it is running in a Docker container and will extract container specific configuration information instead of querying the operating system. The information being extracted is the number of CPUs and total memory that have been allocated to the container." https://www.oracle.com/java/technologies/javase/8u191-relnot...

- Here's a more detailed explanation and even a shared library that can be used to patch container unaware versions of Java. I wonder if the same could be done for Go?

"LD_PRELOAD=/path/to/libproccount.so java "

https://stackoverflow.com/a/64271429

https://gist.github.com/apangin/78d7e6f7402b1a5da0fa3abd9381...

-

There are more recent changes to Java container awareness as well:

https://developers.redhat.com/articles/2022/04/19/java-17-wh...

I've been looking at going in the other direction and using a Go binary with a unikernel; the machine just runs the binary and nothing else. I haven't got this working to my satisfaction yet - it works but there's still too much infrastructure, and deployment is still "fun". But I think this is way more interesting for Go deployments than using containers.

Having said that, for serious business, this setup doesn't make sense. It possibly takes more work to operate as container when the gateway runs on a dedicated instance.

Of course there's also the well-known correlation between the quality of your k8s deployment and the number of proxies it hosts. /s

[0] https://pkg.go.dev/golang.org/x/crypto/acme

Go doesn't need a format for packaging - it's one file. It's becoming common practice to embed everything else into the binary. (side note: I haven't done this with env files yet, and tend to deploy them separately, but I don't see any reason why we don't do this and produce binaries targetted at the specific deployment environments. I might give it a go).

I kinda prefer makefiles for the build stuff, or even just a script. The whole process of creating a Docker instance, pushing source files to it, trigging go build and then pulling back the binary seems redundant; there's no advantage of doing this in a container over doing it on the local machine. And it's a lot faster on the local machine.

Talking to people, it appears to be as mholt said: everyone just does everything in containers so apparently we do this too.

> OSes are still better than container runtimes at managing competing processes on the same host.

OSes and container runtimes are the same thing

For a subset of OSes

And yes, setting up a separate VM for each instance of a process is perfectly feasible. That's what all this cloud business was about in the first place.