{"id":240,"date":"2020-07-24T12:59:04","date_gmt":"2020-07-24T03:59:04","guid":{"rendered":"http:\/\/localhost:8000\/?p=240"},"modified":"2021-01-16T13:06:06","modified_gmt":"2021-01-16T04:06:06","slug":"python-multiprocess","status":"publish","type":"post","link":"http:\/\/localhost:8000\/2020\/07\/python-multiprocess.html","title":{"rendered":"Python\u306b\u304a\u3051\u308b\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u3067\u306e\u72b6\u614b\u7ba1\u7406"},"content":{"rendered":"<p>Python\u3067\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u51e6\u7406\u3092\u66f8\u3044\u3066\u3044\u3066\u3001\u30b5\u30d6\u30d7\u30ed\u30bb\u30b9\u5185\u3067\u540c\u671f\u7684\u306b\u5909\u6570\u3092\u66f4\u65b0\u3059\u308b\uff08\uff1d\u72b6\u614b\u3092\u6301\u3064\uff09\u5fc5\u8981\u304c\u51fa\u3066\u304d\u307e\u3057\u305f\u306e\u3067\u8a66\u3057\u3066\u307f\u307e\u3057\u305f\u3002<br \/>\n<a href=\"https:\/\/docs.python.org\/ja\/3\/library\/multiprocessing.html#sharing-state-between-processes\">\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8<\/a>\u3092\u898b\u3066\u307f\u308b\u3068\u3001\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u3067\u72b6\u614b\u3092\u7ba1\u7406\u3059\u308b\u65b9\u6cd5\u306f\u5927\u304d\u304f\u4e8c\u3064\u3042\u308b\u3088\u3046\u3067\u3059\u3002<\/p>\n<h2>\u5171\u6709\u30e1\u30e2\u30ea\uff08Shared Memory\uff09<\/h2>\n<h3>\u7279\u5fb4<\/h3>\n<ul>\n<li>\u30e1\u30a4\u30f3\u30d7\u30ed\u30bb\u30b9\u5185\u306e\u5171\u6709\u30e1\u30e2\u30ea\u3067\u5909\u6570\u3092\u4fdd\u6301\u3059\u308b<\/li>\n<li>\u5909\u6570\u306e\u578b\uff08\u5165\u308c\u7269\uff09\u3068\u3057\u3066\u306f\u3001<code>Value<\/code>, <code>Array<\/code>\u306e\u307f\u304c\u63d0\u4f9b\u3055\u308c\u3066\u3044\u308b\n<ul>\n<li><code>Value<\/code>\u306f\u4e00\u3064\u306e\u30c7\u30fc\u30bf\u306e\u5165\u308c\u7269\u3001<code>Array<\/code>\u306f\u8907\u6570\u306e\u30c7\u30fc\u30bf\u306e\u5165\u308c\u7269<\/li>\n<li>\u4e2d\u306b\u7a81\u3063\u8fbc\u3081\u308b\u578b\u306f\u3001<a href=\"https:\/\/docs.python.org\/ja\/3\/library\/array.html#module-array\">array\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u5229\u7528\u3067\u304d\u308b\u578b<\/a>\u30fb<a href=\"https:\/\/docs.python.org\/ja\/3\/library\/ctypes.html#module-ctypes\">cypes<\/a>\u306e\u578b\u306e\u307f<\/li>\n<li>=&gt; \u578b\u306e\u5236\u7d04\u304c\u3042\u308a<\/li>\n<\/ul>\n<\/li>\n<li><code>concurrent.futures.ProcessPoolExecutor<\/code>\u3067\u306f\u5229\u7528\u3067\u304d\u306a\u3044<\/li>\n<\/ul>\n<h3>\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb<\/h3>\n<p>array\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u4f7f\u3048\u308b\u578b\uff08\u4eca\u56de\u306fint\uff09\u3092\u5229\u7528\u3059\u308b\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u306f\u3053\u3093\u306a\u611f\u3058\u3067\u3059\u3002<br \/>\n\u4e26\u5217\u51e6\u7406\u3067\u3001\u305d\u308c\u305e\u308c\u6570\u5024\u3092\u4e8c\u4e57\u3057\u3066\u5171\u6709\u30e1\u30e2\u30ea\u4e0a\u306e\u5909\u6570\u306b\u683c\u7d0d\u3059\u308b\u3088\u3046\u306a\u4f8b\u3067\u3059\u3002<\/p>\n<pre><code class=\"language-python\">from multiprocessing import Value, Array, Process\nfrom typing import List\n\ndef parallel_func(number: int, squared_nums: Array, squared_nums_total: Value):\n    squared_num: int = number * number\n    squared_nums[number] = squared_num\n    squared_nums_total.value += squared_num\n\ndef main():\n    nums = [0, 1, 2, 3, 4, 5]\n    squared_nums: Array = Array(&#039;i&#039;, len(nums))\n    squared_nums_total: Value = Value(&#039;i&#039;, 0)\n\n    processes: List[Process] = []\n    for num in nums:\n        p: Process = Process(target=parallel_func, args=(num, squared_nums, squared_nums_total))\n        processes.append(p)\n        p.start()\n\n    for p in processes:\n        p.join()\n\n    print(f&#039;squared_nums: {squared_nums[:]}&#039;)\n    print(f&#039;squared_nums_total: {squared_nums_total.value}&#039;)\n\nif __name__ == &quot;__main__&quot;:\n    main()<\/code><\/pre>\n<p>\u5b9f\u884c\u7d50\u679c\u306f\u3053\u3093\u306a\u611f\u3058\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code>squared_nums: [0, 1, 4, 9, 16, 25]\nsquared_nums_total: 55<\/code><\/pre>\n<p>\u5b9f\u88c5\u3057\u3066\u3066\u3061\u3087\u3063\u3068\u6ce8\u610f\u304c\u5fc5\u8981\u3060\u3063\u305f\u30dd\u30a4\u30f3\u30c8\u306f\u4ee5\u4e0b\u3067\u3059\u3002<\/p>\n<ul>\n<li><code>multiprocessing.Array<\/code>\u306f\u6a19\u6e96\u306e<a href=\"https:\/\/docs.python.org\/ja\/3\/library\/array.html\">array\u30e2\u30b8\u30e5\u30fc\u30eb<\/a>\u3068\u3061\u304c\u3044\u3001\u5fc5\u8981\u306a\u95a2\u6570\u304c\u307b\u3068\u3093\u3069\u63d0\u4f9b\u3055\u308c\u3066\u306a\u3044\u3002\u521d\u671f\u5316\u6642\u306b\u5fc5\u8981\u306a\u30b5\u30a4\u30ba\u3092\u78ba\u4fdd\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b<\/li>\n<li>Value\u3084Array\u306f<code>lock<\/code>\u5f15\u6570\uff08\u30c7\u30d5\u30a9\u30eb\u30c8True\uff09\u3092\u6301\u3063\u3066\u3044\u308b\u304c\u3001\u3053\u308c\u3092False\u306b\u3059\u308b\u3068\u540c\u671f\u30a2\u30af\u30bb\u30b9\u306e\u305f\u3081\u306e\u30ed\u30c3\u30af\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u304c\u4f5c\u6210\u3055\u308c\u305a\u30d7\u30ed\u30bb\u30b9\u30bb\u30fc\u30d5\u3058\u3083\u306a\u304f\u306a\u308b\u3089\u3057\u3044<\/li>\n<\/ul>\n<p>\u3061\u306a\u307f\u306b\u3001ctypes\u3092\u4f7f\u3046\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u306f<a href=\"https:\/\/docs.python.org\/ja\/3\/library\/multiprocessing.html#module-multiprocessing.sharedctypes\">\u3053\u3061\u3089<\/a>\u3092\u53c2\u7167\u304f\u3060\u3055\u3044\u3002<\/p>\n<h3>\u30c8\u30e9\u30d6\u30eb\u5bfe\u5fdc<\/h3>\n<h4>RuntimeError: SynchronizedArray objects should only be shared between processes through inheritance<\/h4>\n<p><code>concurrent.futures.ProcessPoolExecutor<\/code>\u306b\u5bfe\u3057\u3066\u3001\u5171\u6709\u30e1\u30e2\u30ea\u4e0a\u306e\u5909\u6570\u3092\u6e21\u3059\u3068\u3053\u306e\u30a8\u30e9\u30fc\u304c\u767a\u751f\u3057\u307e\u3059\u3002<\/p>\n<p>\u8272\u3005\u8abf\u3079\u305f\u306e\u3067\u3059\u304c\u3001\u5bfe\u51e6\u6cd5\u306f\u306a\u3055\u305d\u3046\u306a\u306e\u3067\u3001\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u306e\u3088\u3046\u306b<code>multiprocessing.Process<\/code>\u3092\u5229\u7528\u3059\u308b\u3053\u3068\u306b\u306a\u308b\u3068\u601d\u3044\u307e\u3059\u3002\u3053\u3061\u3089\u306fworker\u6570\u3092\u6307\u5b9a\u3067\u304d\u306a\u3044\u306e\u3067\u4e0d\u4fbf\u3067\u3059\u3002\u3002<\/p>\n<h2>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\uff08Server Process\uff09<\/h2>\n<h3>\u7279\u5fb4<\/h3>\n<ul>\n<li>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\uff08\u5225\u30d7\u30ed\u30bb\u30b9\uff09\u3092\u7ba1\u7406\u3059\u308b\u30de\u30cd\u30fc\u30b8\u30e3\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u4f5c\u6210\u3057\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u5185\u3067\u5909\u6570\uff08Python\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\uff09\u3092\u7ba1\u7406\u3059\u308b<\/li>\n<li>\u5909\u6570\u306e\u578b\uff08\u5165\u308c\u7269\uff09\u3068\u3057\u3066\u306f\u3001<code>list<\/code>, <code>dict<\/code>, <code>Namespace<\/code>, <code>Lock<\/code>, <code>RLock<\/code>, <code>Semaphore<\/code>, <code>BoundedSemaphore<\/code>, <code>Condition<\/code>, <code>Event<\/code>, <code>Barrier<\/code>, <code>Queue<\/code>, <code>Value<\/code>, <code>Array<\/code>\n<ul>\n<li>\u3069\u306e\u3088\u3046\u306a\u578b\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3067\u3082\u6271\u3048\u308b\u3089\u3057\u3044\n<ul>\n<li>\u72ec\u81ea\u30af\u30e9\u30b9\u3068\u304b\u3082OK<\/li>\n<li>C\u8a00\u8a9e\u306e\u62e1\u5f35\u30e2\u30b8\u30e5\u30fc\u30eb\u3084\u30de\u30cd\u30fc\u30b8\u30e3\u30fc\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u81ea\u8eab\u3092\u542b\u3081\u3053\u3068\u306f\u3067\u304d\u306a\u3044\uff08\u30c8\u30e9\u30d6\u30eb\u5bfe\u5fdc\u53c2\u7167\uff09<\/li>\n<li>\u3068\u306f\u3044\u3048\u3001\u5171\u6709\u30e1\u30e2\u30ea\u3088\u308a\u67d4\u8edf<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306e\u5909\u6570\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u969b\u306f\u3001\u30d7\u30ed\u30ad\u30b7\u7d4c\u7531\u3067\u30a2\u30af\u30bb\u30b9\u3059\u308b =&gt; \u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u7d4c\u7531\u306a\u306e\u3067\u5171\u6709\u30e1\u30e2\u30ea\u3088\u308a\u9045\u3044<\/li>\n<li><code>concurrent.futures.ProcessPoolExecutor<\/code>\u3067\u3082\u4f7f\u3048\u308b<\/li>\n<\/ul>\n<h3>\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2460<\/h3>\n<p><code>list<\/code>\u306b\u72ec\u81ea\u30af\u30e9\u30b9\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u7a81\u3063\u8fbc\u3093\u3067\u3001<code>ProcessPoolExecutor<\/code>\u3067\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u3092\u5b9f\u884c\u3059\u308b\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u306f\u3053\u3093\u306a\u611f\u3058\u3067\u3059\u3002List\uff08ListProxy\uff09\u306f\u5f53\u7136\u53ef\u5909\u9577\u3060\u3057\u3001\u72ec\u81ea\u30af\u30e9\u30b9\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3082\u554f\u984c\u306a\u304f\u8ffd\u52a0\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3057\u305f\u3002<\/p>\n<pre><code class=\"language-python\">from multiprocessing import Manager, Value\nfrom concurrent import futures\nfrom concurrent.futures import Future, ProcessPoolExecutor\nfrom typing import List\nimport dataclasses\n\nMAX_WORKERS = 3\n\n@dataclasses.dataclass\nclass SquaredNum:\n    num: int\n    squared_num: int\n\ndef parallel_func(num: int, squared_nums: List[SquaredNum], squared_nums_total: Value):\n    squared_num: int = num * num\n    squared_nums.append(SquaredNum(num, squared_num))\n    squared_nums_total.value += squared_num\n\ndef main():\n    nums: List[int] = [0, 1, 2, 3, 4, 5]\n\n    with Manager() as manager:\n        squared_nums: List[SquaredNum] = manager.list()\n        squared_nums_total: Value = manager.Value(&#039;i&#039;, 0)\n        future_list: List[Future] = []\n        with ProcessPoolExecutor(max_workers=MAX_WORKERS) as executor:\n            for num in nums:\n                future: Future = executor.submit(\n                    parallel_func,\n                    num=num,\n                    squared_nums=squared_nums,\n                    squared_nums_total=squared_nums_total,\n                )\n                future_list.append(future)\n            for future in futures.as_completed(fs=future_list):\n                future.result()\n        print(f&#039;squared_nums: {squared_nums}&#039;)\n        print(f&#039;squared_nums_total: {squared_nums_total.value}&#039;)\n\nif __name__ == &quot;__main__&quot;:\n    main()<\/code><\/pre>\n<p>\u5b9f\u884c\u7d50\u679c\u306f\u3053\u3093\u306a\u611f\u3058\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code>squared_nums: [SquaredNum(num=0, squared_num=0), SquaredNum(num=1, squared_num=1), SquaredNum(num=3, squared_num=9), SquaredNum(num=2, squared_num=4), SquaredNum(num=4, squared_num=16), SquaredNum(num=5, squared_num=25)]\nsquared_nums_total: 55<\/code><\/pre>\n<h3>\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461<\/h3>\n<p>\u6b21\u306b\u3001\u72ec\u81ea\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3068\u3057\u3066\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308b\u5909\u6570\uff08Python\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\uff09\u3092\u7ba1\u7406\u3059\u308b\u30b1\u30fc\u30b9\u3092\u5b9f\u88c5\u3057\u3066\u307f\u307e\u3059\u3002<br \/>\n\u5b9f\u969b\u306e\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u3067\u306f\u3001PySpark\u306e\u30af\u30e9\u30b9\u30bf\u30fc\u3092\u518d\u5229\u7528\u3059\u308b\u305f\u3081\u306e\u30af\u30e9\u30b9\u30bf\u30fc\u30d7\u30fc\u30eb\u3092\u4f5c\u3063\u305f\u306e\u3067\u3059\u304c\u305d\u308c\u3092\u8d85\u30b7\u30f3\u30d7\u30eb\u306b\u3057\u305f\u611f\u3058\u3067\u3059\u3002\u305d\u308c\u3067\u3082\u3060\u3044\u3076\u9577\u304f\u306a\u3063\u3066\u3057\u307e\u3044\u307e\u3057\u305f...<br \/>\n\u8208\u5473\u304c\u3042\u308b\u4eba\u306f\u8aad\u307f\u8fbc\u3093\u3067\u307f\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n<pre><code class=\"language-python\">from multiprocessing import Manager\nfrom multiprocessing.managers import SyncManager\nfrom concurrent import futures\nfrom concurrent.futures import Future, ProcessPoolExecutor\nfrom typing import List\nimport time\nimport os\nimport random\nimport string\n\nMAX_WORKERS = 3\n\nclass Cluster:\n    def __init__(self):\n        self.name = &#039;&#039;.join(random.choices(string.ascii_lowercase, k=5))\n\n    def submit_pyspark_job(self, num: int):\n        squared_num: int = num * num\n        time.sleep(1)  # \u672c\u5f53\u306f\u3053\u3053\u3067PySpark\u306e\u30b8\u30e7\u30d6\u3092\u5b9f\u884c\u3059\u308b\n        print(f&#039;job finished. num: {num}, squared_num: {squared_num}, pid: {os.getpid()}, cluster_name: {self.name}&#039;)\n        return num * num  # \u30b5\u30f3\u30d7\u30eb\u306a\u306e\u3067\u3001num\u306e\u4e8c\u4e57\u3092\u8fd4\u3057\u3066\u304a\u304f\n\nclass ClusterPool:\n    def __init__(self, manager: SyncManager):\n        self.clusters = manager.list()  # \u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u306b\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308bList\u3092\u4fdd\u6301\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, exc_type, exc_value, tb):\n        self.clusters[:] = []  # with\u30d6\u30ed\u30c3\u30af\u3092\u629c\u3051\u308b\u6642\u306b\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306eList\u3092\u7a7a\u306b\n\n    def pop_or_create_cluster(self) -&gt; Cluster:\n        try:\n            return self.clusters.pop()  # pop\u3057\u3066\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306eList\u304b\u3089\u524a\u9664\n        except IndexError:\n            return Cluster()\n\n    def append_cluster(self, cluster: Cluster):\n        self.clusters.append(cluster)  # \u6700\u5f8c\u306b\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306eList\u306b\u8ffd\u52a0\n\ndef parallel_func(num: int, cluster_pool: ClusterPool) -&gt; int:\n    cluster: Cluster = cluster_pool.pop_or_create_cluster()\n    result: int = cluster.submit_pyspark_job(num)\n    cluster_pool.append_cluster(cluster)\n    return result\n\ndef main():\n    nums: List[int] = [0, 1, 2, 3, 4, 5, 6, 7, 8]\n\n    with Manager() as manager:\n        with ClusterPool(manager) as cluster_pool:\n            future_list: List[Future] = []\n            squared_nums: List[int] = []\n            with ProcessPoolExecutor(max_workers=MAX_WORKERS) as executor:\n                for num in nums:\n                    future: Future = executor.submit(\n                        parallel_func,\n                        num=num,\n                        cluster_pool=cluster_pool,\n                    )\n                    future_list.append(future)\n                for future in futures.as_completed(fs=future_list):\n                    squared_nums.append(future.result())\n\n        print(f&#039;squared_nums: {squared_nums}&#039;)\n\nif __name__ == &quot;__main__&quot;:\n    main()<\/code><\/pre>\n<p>\u30dd\u30a4\u30f3\u30c8\u306f\u4ee5\u4e0b\u306e\u901a\u308a\u3067\u3059\u3002<\/p>\n<ul>\n<li>ClusterPool\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3068\u3057\u3066\u3001Cluster\u30ea\u30b9\u30c8\uff08\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308b\u5909\u6570\uff09\u3092\u4fdd\u6301\u3059\u308b<\/li>\n<li>Cluster\u3092\u5229\u7528\u3059\u308b\u5834\u5408\u306f\u3001Cluster\u30ea\u30b9\u30c8\u304b\u3089<code>pop()<\/code>\u3057\u3066\u8981\u7d20\u3092\u53d6\u5f97\u3057\u3001\u30ea\u30b9\u30c8\u304b\u3089\u306f\u524a\u9664\u3059\u308b<\/li>\n<li>Cluster\u306e\u5229\u7528\u304c\u5b8c\u4e86\u3057\u305f\u3089\u3001Cluster\u30ea\u30b9\u30c8\u306b\u8981\u7d20\u3092<code>append()<\/code>\u3059\u308b<\/li>\n<\/ul>\n<p>\u5b9f\u884c\u30ed\u30b0\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002<br \/>\n<code>14581<\/code>, <code>14582<\/code>, <code>14583<\/code>\u306e\u4e09\u3064\u306e\u30d7\u30ed\u30bb\u30b9\u304c\u7acb\u3061\u4e0a\u304c\u3063\u3066\u3044\u3066\u3001\u305d\u308c\u3089\u306e\u4e2d\u3067\u3001<code>kvyjv<\/code>, <code>uphtx<\/code>, <code>asixt<\/code>\u306e3\u3064\u306ecluster\u3092\u4f7f\u3044\u307e\u308f\u3057\u3066\u3044\u308b\u306e\u304c\u308f\u304b\u308b\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n<pre><code>job finished. num: 0, squared_num: 0, pid: 14581, cluster_name: kvyjv\njob finished. num: 1, squared_num: 1, pid: 14582, cluster_name: uphtx\njob finished. num: 2, squared_num: 4, pid: 14583, cluster_name: asixt\njob finished. num: 3, squared_num: 9, pid: 14581, cluster_name: kvyjv\njob finished. num: 4, squared_num: 16, pid: 14582, cluster_name: asixt\njob finished. num: 5, squared_num: 25, pid: 14583, cluster_name: uphtx\njob finished. num: 6, squared_num: 36, pid: 14581, cluster_name: kvyjv\njob finished. num: 7, squared_num: 49, pid: 14583, cluster_name: uphtx\njob finished. num: 8, squared_num: 64, pid: 14582, cluster_name: asixt\nsquared_nums: [0, 1, 4, 9, 16, 25, 36, 64, 49]<\/code><\/pre>\n<p><code>ProcessPoolExecutor<\/code>\u3092\u4f7f\u3063\u3066\u5225\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u95a2\u6570<code>parallel_func<\/code>\u306e\u5f15\u6570\u3068\u3057\u3066\u3001<code>ClusterPool<\/code>\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3092\u6e21\u3057\u3066\u3044\u308b\u306e\u3067\u3001<code>ClusterPool<\/code>\u5185\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570<code>clusters<\/code>\uff08\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308bPython\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3078\u306e\u53c2\u7167\uff09\u3082\u5225\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u30b3\u30d4\u30fc\u3055\u308c\u3066\u5225\u7269\u306b\u306a\u3063\u3066\u3057\u307e\u3046\u304b\u3082\u3057\u308c\u306a\u3044\u306a\u3041\u3068\u61f8\u5ff5\u3057\u3066\u3044\u305f\u306e\u3067\u3059\u304c\u3001\u5b9f\u969b\u3084\u3063\u3066\u307f\u305f\u3068\u3053\u308d\u3001\u3061\u3083\u3093\u3068\u540c\u4e00\u306e\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306e\u5909\u6570\u306b\u30a2\u30af\u30bb\u30b9\u3057\u3066\u304f\u308c\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3057\u305f\u3002<\/p>\n<h3>\u30c8\u30e9\u30d6\u30eb\u5bfe\u5fdc<\/h3>\n<h4>AttributeError: 'ForkAwareLocal' object has no attribute 'connection'<\/h4>\n<p><code>with Manager() as manger<\/code>\u306ewith\u30d6\u30ed\u30c3\u30af\u306e\u5916\u5074\u3001\u3064\u307e\u308aManager\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u9589\u3058\u305f\u5f8c\u306b\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306e\u5909\u6570\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u3068\u4e0a\u8a18\u306e\u30a8\u30e9\u30fc\u304c\u767a\u751f\u3057\u307e\u3059\u3002<\/p>\n<p>Manager\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u304c\u5b58\u5728\u3059\u308b\u9593\u306b\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306e\u5909\u6570\u3092\u53d6\u5f97\u3057\u307e\u3057\u3087\u3046\u3002<\/p>\n<h4>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306ePython\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u53d6\u5f97\u3057\u3066\u3001\u305d\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u5185\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3092\u5909\u66f4\u3057\u3066\u3082\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u4e0a\u306ePython\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306b\u306f\u53cd\u6620\u3055\u308c\u306a\u3044<\/h4>\n<pre><code class=\"language-python\">@dataclasses.dataclass\nclass Num:\n    num: int\n\ndef main():\n    with Manager() as manager:\n        nums: List[Num] = manager.list()\n        nums.append(Num(1))\n        num_before = nums[0]\n        print(f&#039;num_before: {num_before.num}&#039;)\n        num_before.num = 9\n        print(f&#039;num_changed: {num_before.num}&#039;)\n        num_after = nums[0]\n        print(f&#039;num_after: {num_after.num}&#039;)<\/code><\/pre>\n<p>\u5b9f\u884c\u30ed\u30b0\u306f\u3053\u3093\u306a\u611f\u3058\u3067\u3059\u3002<\/p>\n<pre><code>num_before: 1\nnum_changed: 9\nnum_after: 1<\/code><\/pre>\n<p>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u304b\u3089<code>Num<\/code>\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u53d6\u5f97\u3057\u3066\u3001\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u306e<code>num<\/code>\u3092\u5909\u66f4\u3057\u3066\u3082\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u304b\u3089\u518d\u5ea6\u53d6\u5f97\u3057\u305f<code>Num<\/code>\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306f\u5909\u66f4\u3055\u308c\u3066\u3044\u306a\u3044\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u304b\u3089Python\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u53d6\u5f97\u3059\u308b\u3068\u3044\u3046\u3053\u3068\u306f\u3001\u3064\u307e\u308a\u3001\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u30b3\u30d4\u30fc\u3059\u308b\u3063\u3066\u3044\u3046\u3053\u3068\u306a\u3093\u3067\u3059\u306d\u3002<\/p>\n<p>\u6700\u521d\u306f\u3001\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461\u306f\u3001<code>Cluster<\/code>\u30af\u30e9\u30b9\u306b<code>is_available<\/code>\u3068\u3044\u3046\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3092\u4f7f\u3063\u3066\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306b<code>with<\/code>\u30d6\u30ed\u30c3\u30af\u306e\u5165\u53e3\u3068\u51fa\u53e3\u3067True\/False\u3092\u66f4\u65b0\u3059\u308b\u3088\u3046\u306a\u4f5c\u308a\u306b\u3057\u3066\u3044\u305f\u306e\u3067\u3059\u304c\u3001\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3092\u5909\u66f4\u3057\u3066\u3082\u7121\u99c4\u306a\u3053\u3068\u304c\u308f\u304b\u3063\u305f\u306e\u3067\u3084\u3081\u307e\u3057\u305f\u3002<\/p>\n<pre><code class=\"language-python\">class Cluster:\n    def __enter__(self):\n        self.is_available = False\n        return self\n    def __exit__(self, exc_type, exc_value, tb):\n        self.is_available = True<\/code><\/pre>\n<p>\u3067\u3001\u5bfe\u5fdc\u65b9\u6cd5\u3068\u3057\u3066\u306f\u3001\u5927\u304d\u304f\u4e8c\u3064\u3042\u308a\u307e\u3059\u3002<\/p>\n<ol>\n<li><code>pop()<\/code>\u3067\u30b5\u30fc\u30d0\u304b\u3089\u524a\u9664\u3057\u3064\u3064\u53d6\u5f97 =&gt; \u4f55\u304b\u51e6\u7406 =&gt; <code>append()<\/code>\n<ul>\n<li>\u4e00\u65e6\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u5074\u304b\u3089\u5bfe\u8c61\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u524a\u9664\u3055\u308c\u308b\u306e\u3067\u3001\u4ed6\u306e\u30d7\u30ed\u30bb\u30b9\u304c\u3053\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u306a\u3044<\/li>\n<\/ul>\n<\/li>\n<li>Lock =&gt; \u30b5\u30fc\u30d0\u304b\u3089\u53d6\u5f97 =&gt; \u4f55\u304b\u51e6\u7406 =&gt; \u30b5\u30fc\u30d0\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u7f6e\u63db =&gt; Lock\u89e3\u9664\n<ul>\n<li>Lock\u4e2d\u306f\u3001\u4ed6\u306e\u30d7\u30ed\u30bb\u30b9\u306f\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u306a\u3044<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>\u4eca\u56de\u306e\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461\u306f\u3001<b>1.<\/b>\u306e\u65b9\u6cd5\u306b\u306a\u308a\u307e\u3059\u306e\u3067\u3001\u3053\u3053\u3067\u306f<b>2.<\/b>\u306e\u65b9\u6cd5\u306b\u3064\u3044\u3066\u3082\u5b9f\u88c5\u3060\u3051\u8cbc\u308a\u4ed8\u3051\u3066\u304a\u304d\u307e\u3059\u3002\u8a73\u7d30\u306e\u89e3\u8aac\u306f\u3057\u307e\u305b\u3093\u306e\u3067\u3001\u8208\u5473\u304c\u3042\u308b\u4eba\u306f\u30bd\u30fc\u30b9\u30b3\u30fc\u30c9\u4e0a\u306e\u30b3\u30e1\u30f3\u30c8\u3092\u53c2\u7167\u304f\u3060\u3055\u3044\u3002<\/p>\n<pre><code class=\"language-python\">from multiprocessing import Manager\nfrom multiprocessing.managers import SyncManager\nfrom concurrent import futures\nfrom concurrent.futures import Future, ProcessPoolExecutor\nfrom threading import Lock\nfrom typing import List\nimport time\nimport os\nimport random\nimport string\n\nMAX_WORKERS = 3\n\nclass Cluster:\n    def __init__(self):\n        self.name: str = &#039;&#039;.join(random.choices(string.ascii_lowercase, k=5))\n        self.is_available: bool = False\n\n    def __eq__(self, other):\n        return self.name == other.name\n\n    def submit_pyspark_job(self, num: int):\n        squared_num: int = num * num\n        time.sleep(1)  # \u672c\u5f53\u306f\u3053\u3053\u3067PySpark\u306e\u30b8\u30e7\u30d6\u3092\u5b9f\u884c\u3059\u308b\n        print(f&#039;job finished. num: {num}, squared_num: {squared_num}, pid: {os.getpid()}, cluster_name: {self.name}&#039;)\n        return num * num  # \u30b5\u30f3\u30d7\u30eb\u306a\u306e\u3067\u3001num\u306e\u4e8c\u4e57\u3092\u8fd4\u3057\u3066\u304a\u304f\n\nclass ClusterPool:\n    def __init__(self, manager: SyncManager):\n        self.clusters: List[Cluster] = manager.list()\n        self.lock: Lock = manager.Lock()  # \u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308b\u30ed\u30c3\u30af\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u751f\u6210\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, exc_type, exc_value, tb):\n        self.clusters[:] = []  # with\u30d6\u30ed\u30c3\u30af\u3092\u629c\u3051\u308b\u6642\u306b\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306eList\u3092\u7a7a\u306b\n\n    def get_or_create_cluster(self) -&gt; Cluster:\n        self.lock.acquire(blocking=True) # \u3053\u3053\u3067\u30ed\u30c3\u30af\u53d6\u5f97\u3002timeout\u306f\u6307\u5b9a\u3057\u3066\u306a\u3044\u306e\u3067\u4ed6\u306e\u30d7\u30ed\u30bb\u30b9\u306f\u7121\u9650\u306b\u5f85\u3064\n        try:\n            # \u5229\u7528\u53ef\u80fd\u306a\u65e2\u5b58\u30af\u30e9\u30b9\u30bf\u3092\u63a2\u3059\n            for i, existing_cluster in enumerate(self.clusters):\n                if existing_cluster.is_available:\n                    existing_cluster.is_available = False # \u5229\u7528\u53ef\u5426\u3092\u66f4\u65b0\n                    self.clusters[i] = existing_cluster # \u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306e\u30ea\u30b9\u30c8\u5185\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u7f6e\u304d\u63db\u3048\n                    return existing_cluster\n\n            # \u65b0\u898f\u30af\u30e9\u30b9\u30bf\u3092\u751f\u6210\u3059\u308b\n            new_cluster: Cluster = Cluster()\n            self.clusters.append(new_cluster) # \u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306e\u30ea\u30b9\u30c8\u306b\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u8ffd\u52a0\n            return new_cluster\n        finally:\n            self.lock.release()\n\n    def release_cluster(self, cluster: Cluster):\n        cluster.is_available = True\n        self.clusters[self.clusters.index(cluster)] = cluster\n\ndef parallel_func(num: int, cluster_pool: ClusterPool) -&gt; int:\n    cluster: Cluster = cluster_pool.get_or_create_cluster() # \u30af\u30e9\u30b9\u30bf\u4f5c\u6210 or \u30af\u30e9\u30b9\u30bf\u30d7\u30fc\u30eb\u304b\u3089\u53d6\u5f97\n    result: int = cluster.submit_pyspark_job(num) \n    cluster_pool.release_cluster(cluster) # \u30af\u30e9\u30b9\u30bf\u30d7\u30fc\u30eb\u306b\u30af\u30e9\u30b9\u30bf\u3092\u8fd4\u3059\n    return result\n\ndef main():\n    nums: List[int] = [0, 1, 2, 3, 4, 5, 6, 7, 8]\n\n    with Manager() as manager:\n        with ClusterPool(manager) as cluster_pool:\n            future_list: List[Future] = []\n            squared_nums: List[int] = []\n            with ProcessPoolExecutor(max_workers=MAX_WORKERS) as executor:\n                for num in nums:\n                    future: Future = executor.submit(\n                        parallel_func,\n                        num=num,\n                        cluster_pool=cluster_pool,\n                    )\n                    future_list.append(future)\n                for future in futures.as_completed(fs=future_list):\n                    squared_nums.append(future.result())\n\n        print(f&#039;squared_nums: {squared_nums}&#039;)<\/code><\/pre>\n<p>\u4e0a\u8a18\u306e\u5b9f\u88c5\u3082\u60aa\u304f\u306a\u3044\u306e\u3067\u3059\u304c\u3001Lock\u306e\u5f85\u3061\u5408\u308f\u305b\u304c\u767a\u751f\u3057\u306a\u3044\u5206\u3001\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461\u304c\u512a\u308c\u3066\u3044\u308b\u3068\u601d\u3063\u305f\u306e\u3067\u3001\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461\u3092\u63a1\u7528\u3057\u307e\u3057\u305f\u3002<\/p>\n<h4>TypeError: Pickling an AuthenticationString object is disallowed for security reasons<\/h4>\n<p>Manager\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u304c\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u95a2\u6570\u306e\u5f15\u6570\u306b\u542b\u307e\u308c\u308b\u3068\u3053\u306e\u30a8\u30e9\u30fc\u304c\u767a\u751f\u3057\u307e\u3059\u3002\u4eca\u56de\u306f\u3001<code>ClusterPool<\/code>\u306e<code>with<\/code>\u30d6\u30ed\u30c3\u30af\u3092\u629c\u3051\u308b\u6642\u306bManager\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u9589\u3058\u305f\u304b\u3063\u305f\u306e\u3067\u3059\u304c\u3001\u305d\u3053\u306f\u8ae6\u3081\u307e\u3057\u305f\u3002<\/p>\n<pre><code class=\"language-python\">class ClusterPool:\n    def __init__(self):\n        self.manager = Manager()\n        self.clusters = self.manager.list()\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, exc_type, exc_value, tb):\n        self.manager.__exit__(exc_type, exc_value, tb)  # \u3053\u3053\u3067Manager\u3092\u9589\u3058\u305f\u304b\u3063\u305f\n\ndef main():\n    with ClusterPool() as cluster_pool:\n        with ProcessPoolExecutor(max_workers=MAX_WORKERS) as executor:\n            future: Future = executor.submit(\n                parallel_func,\n                cluster_pool=cluster_pool,\n            )\n            future.result()<\/code><\/pre>\n<p>\u6700\u7d42\u7684\u306b\u306f\u3001\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u2461\u306e\u3088\u3046\u306bClusterPool\u306e\u5916\u3067Manager\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u4f5c\u6210\u3057\u3066\u3001\u6e21\u3059\u3088\u3046\u306b\u3057\u3066\u3042\u3052\u308b\u3053\u3068\u3067\u89e3\u6c7a\u3057\u307e\u3057\u305f\u3002<\/p>\n<h4>TypeError: cannot pickle 'CompiledFFI' object<\/h4>\n<p>\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u3067\u7ba1\u7406\u3059\u308bPython\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u306b<code>google.cloud.dataproc_v1.ClusterControllerClient<\/code>\u3084<code>google.cloud.storage.Client<\/code>\u306a\u3069\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u542b\u3081\u305f\u5834\u5408\u306b\u767a\u751f\u3057\u307e\u3059\u3002<br \/>\n\u3053\u308c\u306f\u3001credential\u7528\u306e\u6697\u53f7\u3092\u9ad8\u901f\u8a08\u7b97\u3059\u308b\u305f\u3081\u306ecryptography\u30e2\u30b8\u30e5\u30fc\u30eb\u304cC\u8a00\u8a9e\u306e\u62e1\u5f35\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3042\u308a\u3001\u4e0a\u8a18\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306e\u4e2d\u306b\u542b\u307e\u308c\u308b\u3088\u3046\u3067\u306a\u306e\u3067\u3059\u304c\u3001C\u8a00\u8a9e\u306e\u62e1\u5f35\u30e2\u30b8\u30e5\u30fc\u30eb\u306fpickle\u3067\u304d\u306a\u3044\u3068\u3044\u3046\u554f\u984c\u304c\u3042\u308b\u307f\u305f\u3044\u3067\u3059\u3002<\/p>\n<p>\u5bfe\u51e6\u65b9\u6cd5\u3068\u3057\u3066\u306f\u3001Client\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u4fdd\u6301\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u305d\u308c\u3092\u4f5c\u6210\u3059\u308b\u305f\u3081\u306b\u5fc5\u8981\u306a\u30af\u30ec\u30c7\u30f3\u30b7\u30e3\u30eb\u30d1\u30b9\u3084\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u540d\u306a\u3069\u3092\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u5909\u6570\u3068\u3057\u3066\u4fdd\u6301\u3057\u3066\u304a\u304d\u3001Client\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u304c\u5fc5\u8981\u306b\u306a\u3063\u305f\u3089\u6bce\u56de\u4f5c\u6210\u3059\u308b\u3088\u3046\u306b\u3057\u307e\u3057\u305f\u3002<\/p>\n<h4><code>if __name__ == &quot;__main__&quot;:<\/code>\u306e\u4e2d\u3067\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u51e6\u7406\u3092\u5b9f\u884c\u3057\u306a\u3044\u3068\u30a8\u30e9\u30fc<\/h4>\n<p>\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u9577\u6587\u306e\u30a8\u30e9\u30fc\u30e1\u30c3\u30bb\u30fc\u30b8\u304c\u8868\u793a\u3055\u308c\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-python\">RuntimeError: \n        An attempt has been made to start a new process before the\n        current process has finished its bootstrapping phase.\n\n        This probably means that you are not using fork to start your\n        child processes and you have forgotten to use the proper idiom\n        in the main module:\n\n            if __name__ == &#039;__main__&#039;:\n                freeze_support()\n                ...\n\n        The &quot;freeze_support()&quot; line can be omitted if the program\n        is not going to be frozen to produce an executable.<\/code><\/pre>\n<p>\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u611f\u3058\u3067\u3001<code>if __name__ == &#039;__main__&#039;:<\/code>\u306e\u4e2d\u3067\u5b9f\u884c\u3059\u308b\u3068\u30a8\u30e9\u30fc\u306f\u89e3\u6d88\u3055\u308c\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-python\">\ndef main():\n    # \u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u51e6\u7406\n\nif __name__ == &#039;__main__&#039;:\n    main()<\/code><\/pre>\n<h2>\u6240\u611f<\/h2>\n<p>\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u306b\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u3042\u308a\u307e\u3059\u3002<\/p>\n<blockquote>\n<p>\u30b5\u30fc\u30d0\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u30de\u30cd\u30fc\u30b8\u30e3\u30fc\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306f\u5171\u6709\u30e1\u30e2\u30ea\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3088\u308a\u3082\u67d4\u8edf\u3067\u3042\u308b\u3068\u3044\u3048\u307e\u3059\u3002\u305d\u308c\u306f\u3001\u3069\u306e\u3088\u3046\u306a\u578b\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3067\u3082\u4f7f\u3048\u308b\u304b\u3089\u3067\u3059\u3002<\/p>\n<\/blockquote>\n<p>\u306a\u306e\u3067\u3001\u30b5\u30fc\u30d0\u30d7\u30ed\u30bb\u30b9\u306e\u65b9\u306f\u306a\u3093\u306e\u5236\u7d04\u3082\u306a\u3044\u306e\u304b\u306a\u3041\u3068\u601d\u3063\u3066\u3044\u305f\u306e\u3067\u3059\u304c\u3001\u5b9f\u969b\u4f7f\u3063\u3066\u307f\u308b\u3068\u601d\u3044\u306e\u5916\u3044\u308d\u3093\u306a\u3068\u3053\u308d\u3067\u30a8\u30e9\u30fc\u304c\u767a\u751f\u3057\u3066\u5927\u5909\u3067\u3057\u305f\u3002\u3002<\/p>\n<p>\u3068\u306f\u3044\u3063\u3066\u3082\u3001\u4e38\u4e00\u65e5\u683c\u95d8\u3059\u308c\u3070\u5927\u4f53\u306e\u3053\u3068\u306f\u89e3\u6c7a\u3067\u304d\u308b\u611f\u3058\u3067\u306f\u3042\u308a\u307e\u3059\u306e\u3067\u3001\u305d\u308c\u307b\u3069\u6050\u308c\u308b\u5fc5\u8981\u306f\u306a\u3044\u304b\u306a\u3001\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Python\u3067\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u51e6\u7406\u3092\u66f8\u3044\u3066\u3044\u3066\u3001\u30b5\u30d6\u30d7\u30ed\u30bb\u30b9\u5185\u3067\u540c\u671f\u7684\u306b\u5909\u6570\u3092\u66f4\u65b0\u3059\u308b\uff08\uff1d\u72b6\u614b\u3092\u6301\u3064\uff09\u5fc5\u8981\u304c\u51fa\u3066\u304d\u307e\u3057\u305f\u306e\u3067\u8a66\u3057\u3066\u307f\u307e\u3057\u305f\u3002 \u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u3092\u898b\u3066\u307f\u308b\u3068\u3001\u30de\u30eb\u30c1\u30d7\u30ed\u30bb\u30b9\u3067\u72b6\u614b\u3092\u7ba1\u7406\u3059\u308b\u65b9\u6cd5\u306f\u5927\u304d\u304f\u4e8c\u3064\u3042\u308b\u3088\u3046\u3067\u3059\u3002 \u5171\u6709\u30e1\u30e2\u30ea\uff08Shared Memory\uff09 \u7279\u5fb4 \u30e1\u30a4\u30f3\u30d7\u30ed\u30bb\u30b9\u5185\u306e\u5171\u6709\u30e1\u30e2\u30ea\u3067\u5909\u6570\u3092\u4fdd\u6301\u3059\u308b \u5909\u6570\u306e\u578b\uff08\u5165\u308c\u7269\uff09\u3068\u3057\u3066\u306f\u3001Value, Array\u306e\u307f\u304c\u63d0\u4f9b\u3055\u308c\u3066\u3044\u308b Value\u306f\u4e00\u3064\u306e\u30c7\u30fc\u30bf\u306e\u5165\u308c\u7269\u3001Array\u306f\u8907\u6570\u306e\u30c7\u30fc\u30bf\u306e\u5165\u308c\u7269 \u4e2d\u306b\u7a81\u3063\u8fbc\u3081\u308b\u578b\u306f\u3001array\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u5229\u7528\u3067\u304d\u308b\u578b\u30fbcypes\u306e\u578b\u306e\u307f =&gt; \u578b\u306e\u5236\u7d04\u304c\u3042\u308a concurrent.futures.ProcessPoolExecutor\u3067\u306f\u5229\u7528\u3067\u304d\u306a\u3044 \u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb array\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u4f7f\u3048\u308b\u578b\uff08\u4eca\u56de\u306fint\uff09\u3092\u5229\u7528\u3059\u308b\u5b9f\u88c5\u30b5\u30f3\u30d7\u30eb\u306f\u3053\u3093\u306a\u611f\u3058\u3067\u3059\u3002 \u4e26\u5217\u51e6\u7406\u3067\u3001\u305d\u308c\u305e\u308c\u6570\u5024\u3092\u4e8c\u4e57\u3057 <a href=\"http:\/\/localhost:8000\/2020\/07\/python-multiprocess.html\" class=\"read-more button-fancy -red\"><span class=\"btn-arrow\"><\/span><span class=\"twp-read-more text\">Continue Reading<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[7],"tags":[],"_links":{"self":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/240"}],"collection":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/comments?post=240"}],"version-history":[{"count":1,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/240\/revisions"}],"predecessor-version":[{"id":241,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/240\/revisions\/241"}],"wp:attachment":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/media?parent=240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/categories?post=240"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/tags?post=240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}