Python: typing.overload
typing.overload
makes it easier to type functions with behaviour that depends on input types.
Functions marked with @overload are ignored by Python and only used by the
type checker:
@overloaddefprocess(response:None)->None:...@overloaddefprocess(response:int)->tuple[int,str]:...@overloaddefprocess(response:bytes)->str:...defprocess(response):# <actual implementation>Python's multiprocessing and deadlocks
Python's multiprocessing is prone to deadlocks in a number of conditions. In my case, the running program was a standard single-process, non-threaded script, but it used complex native libraries which might have been the triggers for the deadlocks.
The suggested workaround is using set_start_method("spawn"), but when we
tried it we hit serious performance penalties.
Lesson learnt: multiprocessing is good for prototypes, and may end up being too hacky for production.
In my case, I was already generating small python scripts corresponding to worker tasks, which were useful for reproducing and debugging Magics issues, so I switched to running those as the actual workers. In the future, this may come in handy for dispatching work to HPC nodes, too.
Here's a parallel execution scheduler based on asyncio that I wrote to run them, which may always come in handy on other projects.