No-GIL mode coming for Python
We want to be very careful with backward compatibility. We do not want another Python 3 situation, so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed). This is not Python 4. We are still considering the requirements we want to place on ABI compatibility and other details for the two builds and the effect on backward compatibility.
(Log in to post comments)
No-GIL mode coming for Python
Posted Jul 28, 2023 22:42 UTC (Fri) by mb (subscriber, #50428) [Link]
No-GIL mode coming for Python
Posted Jul 28, 2023 23:51 UTC (Fri) by tialaramex (subscriber, #21167) [Link]
"this PEP needs to specify a mechanism which ensures if you have a 2022 C library which was fine because of the GIL, and a 2022 Python program which used that C library, you cannot set everything on fire by trying to use these with a nonGIL Python. Even if you recompile the C library, even if you do some basic tinkering (adding/ removing -D parameters, fussing with file locations, that sort of thing)"
I still don't see the clarity I'd want that they understand this. They've said they don't want another Python 3, but that's acknowledging that the bad outcome was bad - it crucially is *not* a refusal to do the same thing which had that consequence. I consider this a red flag, because it communicates that they know they were expected to learn something, but they didn't and they're hoping for approval anyway.
No-GIL mode coming for Python
Posted Jul 29, 2023 1:23 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
I think that is an uncharitable read of their statement, because it immediately follows through with this:
> ...so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed).
This was a major problem with Python 3 (i.e. code written for 3 was incompatible with 2), which they are explicitly refusing to repeat. They have learned their lesson from the 3.x fiasco, and are applying that lesson to this PEP. Perhaps this is not the lesson you, personally, wanted them to learn from the 3.x fiasco, but it is disingenuous to claim that they have learned nothing.
More directly relevant to your comment, from the PEP:
> CPython builds without the GIL will not be ABI compatible with the standard CPython build or with the stable ABI due to changes to the Python object header needed to support biased reference counting. C-API extensions will need to be rebuilt specifically for this version.
The exact thing that you are asking for (GIL-less Python fails loudly if you try to load a GIL-assuming module) is already being done - the module is ABI incompatible with the new CPython, so it will fail to load. The only other question is whether they will introduce a clear error message for that situation, and it is totally unreasonable to expect a PEP to specify such fiddly details.
Also of note:
> This PEP poses new challenges for distributing Python. At least for some time, there will be two versions of Python requiring separately compiled C-API extensions. It may take some time for C-API extension authors to build --disable-gil compatible packages and upload them to PyPI. Additionally, some authors may be hesitant to support the --disable-gil mode until it has wide adoption, but adoption will likely depend on the availability of Python’s rich set of extensions.
>
> To mitigate this, the author will work with Anaconda to distribute a --disable-gil version of Python together with compatible packages from conda channels. This centralizes the challenges of building extensions, and the author believes this will enable more people to use Python without the GIL sooner than they would otherwise be able to.
So the only people who will be using this in the short term are Conda users, and they have a whole package manager which is specifically responsible for detecting and dealing with this sort of compatibility problem. I doubt they even *need* a clear error message from CPython. Conda will (should) refuse to create such an invalid installation in the first place.
No-GIL mode coming for Python
Posted Jul 29, 2023 1:33 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
Side note: Your comment also mentions the end user recompiling the module, possibly with -D flags etc., but that is wildly unreasonable. C is an unsafe language. Any time you fool around with a C compiler, without actually understanding what you are doing, you are assuming the risk of shooting yourself in the foot. If the user can figure out what flags they have to pass, they can also find and write or rewrite any hypothetical "I'm compatible with GIL-less Python" API calls that may need to exist, so this is simply impossible unless you completely deprecate the old API and start over from scratch (which might be worth doing anyway, see HPy, but that's a separate discussion).
No-GIL mode coming for Python
Posted Jul 29, 2023 5:52 UTC (Sat) by whoami (guest, #166229) [Link]
No-GIL mode coming for Python
Posted Jul 29, 2023 18:08 UTC (Sat) by tialaramex (subscriber, #21167) [Link]
I don't think that is unreasonable to assume this might happen, and as a result, in fact *because* C is an unsafe language, put in less fragile countermeasures.
No-GIL mode coming for Python
Posted Jul 29, 2023 20:13 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
Regardless, if a user is intrepid enough to recompile an extension, they're probably also intrepid enough to copy and paste arbitrary nonsense into the body of PyInit_foo(), and at that point, there's really nothing you can do to stop them from modifying the extension to explicitly declare that it supports GIL-less Python.
No-GIL mode coming for Python
Posted Jul 29, 2023 20:19 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
#ifdef ALLOW_NO_GIL
#error "GIL-less Python is unsupported."
#endif
No-GIL mode coming for Python
Posted Jul 29, 2023 20:35 UTC (Sat) by floppus (guest, #137245) [Link]
I agree with tialaramex that "GIL-less Python" needs to be API-incompatible with older Python. Merely ABI-incompatible is not enough. (That is to say, if somebody *does* follow your build instructions to the letter, and your extension isn't safe to use with the new version of Python, then it must fail to compile or fail to load.)
I'd agree with you, NYKevin, that requiring people to add a -D flag would be a sufficient barrier to stop people accidentally compiling unsafe modules. And that should be the goal - to stop people *accidentally* doing something that was previously safe and is now unsafe, not to stop people who are deliberately tinkering from shooting themselves in the foot.
But remember:
> if a user is intrepid enough to recompile an extension
"Intrepid enough to recompile an extension" can simply mean "intrepid enough to run 'pip install'".
No-GIL mode coming for Python
Posted Jul 29, 2023 20:46 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
No-GIL mode coming for Python
Posted Jul 29, 2023 21:03 UTC (Sat) by floppus (guest, #137245) [Link]
> But yes, obviously it should break if you just run pip install with an incompatible CPython
In my book, that's precisely what API-incompatible means.
(To be clear, I'm talking about using "pip install" to install a source package, what happens if there isn't a precompiled wheel package available for your Python version.)
No-GIL mode coming for Python
Posted Jul 31, 2023 7:21 UTC (Mon) by NYKevin (subscriber, #129325) [Link]
As I quoted in my top-level reply, the PEP explicitly says that they will be "ABI incompatible." In my book, "ABI incompatible" means "the symbols don't match, so the .so (or whatever your operating system calls it) file fails to load (or link, or whatever your OS calls it)." I'm not familiar with the definition of "ABI incompatible" you seem to be using. Can you elaborate on it?
No-GIL mode coming for Python
Posted Jul 31, 2023 8:35 UTC (Mon) by matthias (subscriber, #94967) [Link]
There is a difference between ABI (Application Binary Interface) and API (Application Programming Interface). The former refers to compatibility on machine code level. The latter refers to compatibility on the programming language level.
Code can be ABI incompatible but still be API compatible. In this case it suffices to just recompile the code, as the code is compatible on the programming language level. The compiler will resolve the changed symbols and produce a "working" executable. Here working means that it can be started, not that it will work correctly.
So what is requested here is API incompatibility, so that no one can accidentally compile the code against the wrong version of Python and get random crashed/errors due to the old code not being ready for No-GIL Python.
No-GIL mode coming for Python
Posted Jul 31, 2023 15:39 UTC (Mon) by NYKevin (subscriber, #129325) [Link]
No-GIL mode coming for Python
Posted Jul 29, 2023 2:03 UTC (Sat) by Subsentient (subscriber, #142918) [Link]
It would be terrific to use a GIL-less Python, but since it's an omen of the end times, I assume we'll all be dead before we can finish benchmarking the improvement.
Jokes aside, I've wanted this for many, many years. I don't care if C extensions require a little fiddling to update them for the GIL-less version. It's worth it. Threading on Python right now is just async with internal pthread handles unless you're calling into C and explicitly release the GIL.
As someone who extensively, and I mean extensively, uses multithreading in my codebases, this will open up beautiful new possibilities for me.
While they're at it, they should make an entirely new, easier, simpler C API that has consistency in the ownership of objects -- with the old API still there and supported, of couse. I shouldn't constantly have to check the C API reference to know if I need to decrement the reference count of a Python object or not.
No-GIL mode coming for Python
Posted Jul 29, 2023 7:18 UTC (Sat) by atnot (subscriber, #124910) [Link]
Even though they say that they don't want to create another python2/3 split, the fact of the matter is still that little existing Python or C code has been written with this sort of concurrency in mind, and more importantly neither the ecosystem nor the languages have any features that make getting concurrency right easy. The amount of intended and unintended breakage this will cause is immense. Especially considering how common python is as a first programming language, and how hard shared memory concurrency is to teach.
There was a real chance to make something better with the ongoing subinterpreter work plus some native lightweight communication channels ala erlang or go, but I fear the appetite for that will be significantly reduced now that this is in motion.
No-GIL mode coming for Python
Posted Jul 29, 2023 17:44 UTC (Sat) by walters (subscriber, #7396) [Link]
I understand why some people want it. I enjoyed writing Python years ago. Not having compile times is great. But as I wait for my Rust code to compile now, I just think back to those painful Python debugging sessions with problems that would have been caught at compile time in Rust. This goes 100x more with things like how Rust forces one to distinguish between mutable and immutable state and prevents data races in concurrent code.
No-GIL mode coming for Python
Posted Jul 30, 2023 7:24 UTC (Sun) by pwfxq (subscriber, #84695) [Link]
But isn't this the trade-off between a compiled & interpreted language?
One allows you to run your code quicker by removing the compile step, whereas one reduces run-time errors by catching them in a new "compile" step that's added between "edit" and "run"?
No-GIL mode coming for Python
Posted Jul 30, 2023 20:19 UTC (Sun) by khim (subscriber, #9252) [Link]
Yes, but it's somehow different with Rust. While in C++ 99% of time compile-time error means I forgot something obvious and real serious errors survive till runtime anyway in Rust about 10% of time it's “damn, this clever compiler caught something which I would have spend week or two debugging” and, more importantly, very few bugs survive till actual run time (enough that some say that Rust duplicates “if it compiles it runs” Haskell approach).
Somehow it makes tremendous difference: while most compile-time errors are still stupid and easy to fix enough of them are catching problems serious enough that all these wait times needed for complier to verify your program feel justified.
No-GIL mode coming for Python
Posted Jul 31, 2023 4:12 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
The big difference is the choice to give up when the machine can't tell if you're meeting the semantic constraints. A good number of Rust's errors could not exist in a conforming C++ compiler, because the C++ standard explicitly says that the compiler must not reject programs which it merely can't show are valid, only programs which it has proved invalid may be rejected. If you've just written a very clever program but it won't compile because rustc doesn't see how this can possibly be correct even though it is, this makes you sad - but you can probably modify it to be more obvious for the compiler. On the other hand, if you're wrong, C++ will compile your wrong program anyway, and good luck tracking down the "impossible" bugs which result.
No-GIL mode coming for Python
Posted Jul 30, 2023 2:50 UTC (Sun) by comex (subscriber, #71521) [Link]
C code, sure. But Python? Python has had multithreading since at least version 1.0 in 1994. There's huge amounts of multithreaded Python code out there, and as far as I can tell no-GIL mode won't change the user-facing semantics much if at all. The biggest fear seems to be that it will make existing race conditions more likely to reproduce, which is a valid concern but doesn't seem like the end of the world. Especially since making race conditions easier to trigger also makes them easier to diagnose and fix.
It's true that No-GIL mode may suck the air out of the room for a hypothetical new concurrency model. But on the other hand, Python already has at least three concurrency models: threads, multiprocessing, and async. (Four if you count gevent.) And "async" is really multiple incompatible async frameworks (asyncio, Trio, and others), while "threads" includes at least three generations of APIs in the standard library (_thread, threading, and concurrent.futures.thread). Do we really need yet another concurrency model?
…Well, perhaps we do. I admit I'm quite dissatisfied with Python's threading. The APIs feel clunky and lacking in useful primitives (e.g. channels, or some usable notion of cancellation). I have less experience with async, but I think it fixes a lot of these issues, yet it won't benefit from no-GIL mode, and it's clunky in its own ways.
No-GIL mode coming for Python
Posted Jul 30, 2023 10:00 UTC (Sun) by atnot (subscriber, #124910) [Link]
This is true in a narrow sense. But in a broader sense, threads in python have so far always been a rare edge case. As you said the concurrency primitives have been correspondingly threadbare, basically amounting to only a free-floating mutex. Once there is a performance advantage, people will suddenly start using 10 year old libraries from 50 threads at once and you can't really tell them to just not do that anymore. I can personally say that I've never thought about whether python code I was writing was concurrency safe and in this environment, I don't see how I'd be able to have any confidence that it was even if I tried.
There's also the extreme commonness of things like global caches and similar global objects in python code. All of these will have to adjusted to avoid massive lock contention to achieve reasonable performance without the GIL.
> But on the other hand, Python already has at least three concurrency models: threads, multiprocessing, and async. (Four if you count gevent.)
Five if you count the existing subinterpreter capabilities :)
> Do we really need yet another concurrency model?
Concurrent subinterpreters wouldn't really be a new concurrency model at all. It's the same as multiprocessing, just with lower startup and communication overhead, which are the only real big drawbacks of subprocessing. It also works much better with the modern concurrent.Futures model.
> I have less experience with async, but I think it fixes a lot of these issues, yet it won't benefit from no-GIL mode, and it's clunky in its own ways.
I think it's inevitable that people will desire a multi threaded async executor personally. It's conceptually pretty simple, instead of picking one coroutine from the queue to run in your event loop, you pick n. But considering how much of a struggle even the rust people have had at making that nice to work with, I do dread to think what that would look like for Python in a free-for-all concurrent shared memory world.
No-GIL mode coming for Python
Posted Jul 29, 2023 20:40 UTC (Sat) by gmgod (subscriber, #143864) [Link]
Sure old modules should keep working as intended so the API they see won't change much (hence why "it's not python 4" is valid) but I'm not sure it's right for native modules to check for version 3.5+ (as many do today) or 3.13+ as some will tomorrow. If some of your API consumers feel the need to use minor version numbers to make decisions, there is something wrong with your semver.
No-GIL mode coming for Python
Posted Jul 29, 2023 22:36 UTC (Sat) by randomguy3 (subscriber, #71063) [Link]
now, if you dependency is "<3.13" (or similar), then that suggests semver isn't being used correctly - and, indeed, Python versioning isn't technically semver, as we've seen standard library modules being removed on a minor version upgrade (albeit with deprecations beforehand)
No-GIL mode coming for Python
Posted Jul 30, 2023 9:00 UTC (Sun) by gmgod (subscriber, #143864) [Link]
No-GIL mode coming for Python
Posted Jul 30, 2023 9:46 UTC (Sun) by njs (guest, #40338) [Link]
No-GIL mode coming for Python
Posted Jul 31, 2023 11:32 UTC (Mon) by cyperpunks (subscriber, #39406) [Link]
No-GIL mode coming for Python
Posted Jul 31, 2023 13:08 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
I believe Go decided Go 2 shouldn't happen (at least for the foreseeable future) for the same reason, they don't actually want to give up compatibility so, they won't.
*sigh*
Posted Aug 4, 2023 23:56 UTC (Fri) by simlo (guest, #10866) [Link]
With proper non-blocking IO far the most problems could much more effectively be solved with a single-threaded program, being more deterministic and far easier to test. Use multiple processes with no shared memory instead if you really need several CPU cores.
In my daily life as software developer I rarely have found problems which couldn't be solved that way. But everyone designing languages wants to design for those exceptions, it seems.
And we don't have proper non-blocking IO available for many things, in many languages...