多进程之Pool与多线程pool 及tqdm和for 并对比pandas处理结果

    技术2023-07-01  104

    又叕又碰到这个多进程问题了,其实线程也行,下面再次进行demo测试

    召回中遇到的,存储召回的items

    For Video Recommendation in Deep learning QQ Group 277356808  

    For Visual in deep learning QQ Group 629530787  

    I'm here waiting for you  

    wanna have a date with someone in Beijing please join in the QQ Group 737813700

     

    from multiprocessing import Process,Pool

    Pool(32)与Pool(64)相比没有慢多少,一个是116s,一个是90s,因此保险考虑32为好

    遇到一个问题:多进程Pool中

    AttributeError: Can't pickle local object 'update_model.<locals>.delete_copy_items'

    下面小demo复现

    Traceback (most recent call last): File "multiprocess_Process_Pool_.py", line 99, in
    Processed: 0.010, SQL: 9