Python is used by loads of scientists, academics, and other non software engineers. Those people need an easy way to use python.

If you want to have a platform that allow to manage Python on all machines, including allowed packages and version, integrated with ldap, with auditing capabilities, they are pretty much the only game in town.

And big companies want that.

Today I would say anaconda brings more problems than it solves (https://www.bitecode.dev/p/why-not-tell-people-to-simply-use), but at the time, we didn't have wheels for everything, and it was a life saver for anybody that wanted to use c extensions.

So anaconda became first popular because it solved a real end user problem, then it moved on to be the corporation providers because it already was well known.

It was a very good strategy.

They were never big outside Windows shops.

* Try running `python -c 'import this' | sed -n 15p`

You will notice that every package management solution from all your favourite languages will have its own set of tradeoffs.

Lock files alone are a proven piece of technology that pretty much just works across all modern package managers, and yet the best that pip can recommend to date is `pip freeze > requirements.txt`, which is strictly inferior than what is available in other package managers because there's no hash verification and no distinction between transitive dependencies and explicit dependencies.

That pip still doesn't have an answer for lock files is a sign of a deep problem in the Python package management world, not the natural result of packaging being hard.

Hopefully uv can make this the default behavior. It seems like the majority of users are not aware of its existence because it’s an optional flag.

Here's what they say about requirements:

> Requirements files are used to hold the result from pip freeze for the purpose of achieving Repeatable Installs. In this case, your requirement file contains a pinned version of everything that was installed when pip freeze was run.

Here's what they say about constraints:

> Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. ... In terms of semantics, there is one key difference: Including a package in a constraints file does not trigger installation of the package.

> ... Write a single constraints file for your organisation and use that everywhere. If the thing being installed requires “helloworld” to be installed, your fixed version specified in your constraints file will be used.

> Constraints file support was added in pip 7.1. In Changes to the pip dependency resolver in 20.3 (2020) we did a fairly comprehensive overhaul, removing several undocumented and unsupported quirks from the previous implementation, and stripped constraints files down to being purely a way to specify global (version) limits for packages.

This sounds like something vaguely similar to a lock file, but they seem to intend it to be used globally at the organization level, and they're certainly not pushing it as the answer to locking dependencies for a specific project (they specifically recommend requirements for that). Maybe you can use it that way—although this Stack Overflow answer says that the 2020 update broke it for this use case [1]—but even so it doesn't answer the fundamental need for lock files unless everyone is actually on board with using it for that, and even the pip maintainers don't seem to think you should.

[0] https://pip.pypa.io/en/latest/user_guide/#requirements-files

[1] https://stackoverflow.com/questions/34645821/pip-constraints...

ironic that pip freeze literally generates a requirements.txt with ==

constraints.txt certainly did not exist back when I was doing python.

Conversely Ruby packaging has been a solved problem for a decade, when the python community has been extremely resistant to conceptually similar solutions for the longest time on strange ideological grounds, and came around only recently.

[0] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#...

And quite a few where the tradeoffs are minor enough to be well worth some sanity & consistency.

When it comes to package managers, as you noted, the situation is even more ironic, as nicely depicted in https://xkcd.com/1987/.

    >>> from __future__ import braces
      File "", line 1
    SyntaxError: not a chance

    $ LC_ALL=C NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-13.10.tar.gz nix-shell -p python26 --run 'python -c "from __future__ import braces"'
      File "", line 1
    SyntaxError: not a chance

Early versions of go didn’t solve things well, but go has had this fixed for a long time now. Any arguments about packaging issues are likely related to older out of date problems people had.

The only real problem people have these days is around private non open source packages. You can do it, but you need to learn how to set environment variables.

> Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows.

Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".

Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim.

> uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.)

> uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions.

This is great to see though!

I can understand it being a flag on these lower level, directly invoked dependency resolution operations.

While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537

How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.

> Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".

Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution.

> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.

This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.

The living version of PEP 440 has a bit on how pre-releases are handled[1]. The basic version is that the installer shouldn't select them at all, unless the user explicitly indicates that they want a pre-release. Once opted into, they're ordered by their phase (alpha, beta, rc) and pre-release increment (e.g. `.beta.1 > `.alpha.2`).

[1]: https://packaging.python.org/en/latest/specifications/versio...

> I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.

I feel uv should provide a way to install different python versions to truly cover an end-to-end tool. The current approach of searching for existing virtualenvs of conda envs helps but I was hoping to completely remove the need for another package/dependency manager.

(Taken from the docs)

  If a --python-version is provided to pip compile (e.g., --python-version=3.7), uv will search for a Python interpreter matching that version in the following order:

  - An activated virtual environment based on the VIRTUAL_ENV environment variable.

  - An activated Conda environment based on the CONDA_PREFIX environment variable.

  - A virtual environment at .venv in the current directory, or in the nearest parent directory.

  - The Python interpreter available as, e.g., python3.7 on macOS and Linux. On Windows, uv will use the same mechanism as py --list-paths to discover all available Python interpreters, and will select the first interpreter matching the requested version.

  - The Python interpreter available as python3 on macOS and Linux, or python.exe on Windows.

By the way, the creator of mamba started his own company at https://prefix.dev/

They want to essentially leverage the conda(-forge) infrastructure to build a new cross-platform, cross-language, cargo-like package manager: pixi

[0] https://github.com/mamba-org/mamba

https://conda.org/blog/2023-11-06-conda-23-10-0-release/

After fiddling with different solutions for years and having to start fresh with a new Python install, I've been using nothing by miniconda for years and it just works

Using only „Pythons native tools“ like pip and venv simply works nowadays so good that I wonder about the purpose of many tools like poetry etc. etc.

https://github.com/python-poetry/poetry/issues/6409

pip and venv work fine, but you have to get them first; and that can be a struggle for unseasoned python devs, especially if you need a version that's not what your distro ships, and even more so on Windows and macOS.

I use micromamba [1] specifically, which is a single binary.

[1] https://mamba.readthedocs.io/en/latest/user_guide/micromamba...

https://www.bitecode.dev/p/back-to-basics-with-pip-and-venv

https://www.bitecode.dev/p/relieving-your-python-packaging-p...

pip install works, but pip's dependency management doesn't seem to (for Pytorch, specifically) which is why projects that have pip + requirements.txt as one of their installation methods will often have separate pytorch installation instructions when using that method, though if the same project supports conda installation it will be a one-stop-shop installation that way.

That’s interesting — I’ve also had difficulties with PyTorch and dependency resolution, but only on the most recent versions of Python, for some period of time after they’re released. Picking Python 3.9 as a baseline for a project, for example, has been very reliable for PyTorch and all the related tooling.

npm -> Yarn was life changing performance-wise.

I wonder what pip -> uv is.

I’m loving the pace of release with this crop of speed obsessed projects and I cannot wait for astral to tackle typing.

    Installation times (MM:SS):

    # No cache
    - uv:  01:05
    - pip: 01:56

    # Cache:
    - uv:  00:02
    - pip: 00:42

Npm has improved greatly.

Npm features follow a two-step process: (1) Get implemented in Yarn (2) Get implemented in npm.

https://news.ycombinator.com/item?id=32116649

Python packaging has gotten asymptotically worse in the last 90 days from my experience having used the language for 20 years now and working on a multi-platform triumvirate of Lin-Mac-Win that has not changed in twenty years.

I can honestly say that, given how daily failures in the packaging ecosystem such as pip failing to build egg, pkg-config, and similar problems, I'm happy to see a solution from rust, because I sent an intern into the polars underworld 2 years ago to great success and have only grown my own reliance on rust infrastructure more broadly since.

This may very well be the only packaging innovation in python that I actually look forward to trying tomorrow. I'd already be working in rust full-time if I could only get past the syntax that reminds me too much of perl after bathing in the semantic whitespace light for this long.

We need a permanent solve for python package management, governed by the python steering council.

alias pip="/usr/bin/uv pip"

> TL;DR: uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows.

  (.venv) lijok@mbp icebreaker % python -m uv pip install --upgrade uv setuptools wheel

  error: unexpected argument '--upgrade' found

  tip: to pass '--upgrade' as a value, use '-- --upgrade'

  Usage: uv pip install [OPTIONS] |--editable >

  For more information, try '--help'.

  (.venv) lijok@mbp icebreaker % python -m uv pip install -- --upgrade uv setuptools wheel

  error: Failed to parse `--upgrade`
  Caused by: Expected package name starting with an alphanumeric character, found '-'
  --upgrade
  ^

https://github.com/prefix-dev/pixi/

As conda/pip interop is just not great. And even micromamba (C++ implementation of conda) is relatively slow to resolve compared to pip.

But agreed either work together or create a type checker. I use mypy currently but it definitely slows down my editor.

    def _deep_merge(updates: dict[str, str]) -> None:
        for key, update_info in updates.items():
            if isinstance(update_info, dict) and "values" in update_info:
                value = update_info["values"]
 

    - error: Argument of type "Literal['values']" cannot be assigned to parameter "__key" of type "SupportsIndex | slice" in function "__getitem__"
        Type "Literal['values']" cannot be assigned to type "SupportsIndex | slice"
      "Literal['values']" is incompatible with protocol "SupportsIndex"
        "__index__" is not present
      "Literal['values']" is incompatible with "slice" (reportGeneralTypeIssues)

It took me a great amount of starring to figure out that changing the signature of updates to dict[str, dict] was what it was complaining about.

I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards.

(Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)

``` black @ git+https://github.com/psf/black ```

We also support editable installs for local directories, like you mentioned:

``` black @ ../black ```

The thing we don't support is using _editable_ installs for Git and direct URL dependencies, like:

``` -e git+https://github.com/psf/black ```

In my experience, these are really rare, and we didn't see a reason to support them.

Rip (https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda).

And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM.

A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools.

I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow.

As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.

Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.

Being compatible with the mess that exists is where the difficulty comes from.

The last 5 or so years I've mostly done work in rust... so I guess I'm a rust developer now.

Do the last 5 years invalidate my opinions, thoughts, or skills w.r.t. python somehow?

really curious on the reasoning behind this decision :)

Mostly it’s useful right after a new release where prebuilt wheels aren’t available for all packages yet and you have users who may care or not about this.

If you only have single requirements file, you get forced to choose which platform to support. With multiple, you can break out if needed. These breaking changes and deadlocks are rare. It’s still good to have an escape hatch.

[1]: https://hatch.pypa.io/latest/

One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.

Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed.

sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.

The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).

Sure, it’s rendered a bunch of projects obsolete, but it’s because it’s so much better than the predecessors.

That is true for every single development tool in the Python space.

Obviously it would be pretty crummy to end up in a situation like pyright where MS really have leant in to their embrace, extend, extinguish strategy.

I'm not sure what the maker of that video wants, does he want money to be poured back into the community, or for no one else to make money?

I agree that we have no idea how it'll play out. But I've seen nothing but good faith from Charlie and others, and I see no reason to preemptively condemn their attempt to build a business.

In this case they are replacing Python code with Rust which might exclude large part of Python community from being able to contribute.

(Or another framing: only the tiniest fraction of the Python community was contributing to CQA tooling in the first place. It's not clear that prioritizing the interests of the 99.9% of Python programmers who -- rightfully! -- will never modify these tools makes sense.)

Global dependencies by default (fixed by venvs). Shipping precompiled binaries that are not portable across platforms or python versions (kinda fixed by wheels, I forget the PEP number). No lock files with checksum verification by default (kinda fixed by requirements.txt with require hashes, but not really). Also, there's a bunch of weird shit that can happen if the pip version and python version disagree due setup tools.

It's got a great interface for installing stuff. But the likelihood that what it installs breaks something on your machine is pretty high the longer you use it.

Edit: the real, fundamental problem with pip is that it can't be saved. Too many dockerfiles and server provisioning scripts are relying on it - it's interface, semantics, CLI output, etc. It's not 100% broken, which is why it's still useful. But any true fix has to be done by creating a separate tool.

* Defaults to installing packages globally instead of per project.

* requirements.txt feels very much like a second class citizen when you contrast it with package.json, Cargo.toml, build.gradle, pom.xml, or most other dependency systems in other package managers.

* No native support for lockfiles short of pinning every version of every project in requirements.txt. This solution is inferior to what is available in ~every other package manager, because a pip freeze doesn't distinguish between explicit dependencies and transitive dependencies.

I'm sure there are others, but they're all along the same lines—pip is strictly inferior than basically every other package manager out there for other languages. Things that people expect to be part of a package manager have ugly hacky workarounds instead that aren't uniformly applied across projects.

I was able to build useful things with pip + virtualenv

Pip is also strongly tied to the specific Python version. If you upgraded Python interpreter on a system, congrats all virtual environments and all PyPi packages are broken now. Hopefully you saved the minimum set of requirements and they don't have some catastrophic dependency breakage. You run pip install and pray.

What? Native binaries? Building them on the fly while installing a pip package? Oh you're brave.

Of course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.

Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity.

I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.

For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.

> Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more.

A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.

For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only?

[1] https://pypi.org/project/numpy/#files

I think you're thinking of this post[1]. The code being searched for wasn't malicious, just buggy :-)

[1]: https://moyix.blogspot.com/2022/09/someones-been-messing-wit...

Heh, I forgot about the anime catgirl part.

Please keep doing what you're doing!

edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?

https://xkcd.com/1987/ https://xkcd.com/927/