I have two python scripts and I want to share a complex python object between them. The first script firstly load the contain of a large (.dat) file into the complex python object. After some time, I want to access this object from the second script without loading the file again. I attempted to use shared memory and message passing inter-process communication methods but the both methods fail due to the huge size of the object. What is the suitable method to achieve such inter-process communication in python?
https://stackoverflow.com/questions/3671666/sharing-a-complex-object-between-python-processes
https://pymotw.com/2/multiprocessing/communication.html
https://www.geeksforgeeks.org/multiprocessing-python-set-2/
Dear Cristian Ramos-Vera , thank you for your assistance. However; I tried to use multiprocessing module and the functions it provides like pipe, manner, proxy ..etc. They work well with simple objects but when I utilized them with my object they fail with different reasons (like not-supported type or it becomes unresponsive). My problem in the size of the complex object because it implicitly contains data of a file with size more than 62M.
Dear Suhaila N. Mohammed ,
I am not sure if I got your problem clear. Why you don't just pass the object which contains the data to the second script using a function? I mean import what you need from the second script into the first script, as follows:
First script:
From SecondScript import *
X = LoadData("File.dat")
DoSomething(X) // Calling a function from the second script ..
Second Script:
def DoSomething(XX):
// The code for the processing ..
Have you tried the above mentioned code?
I haven't seen any source code trying to work with huge datasets using sharing memory unless there is an important reason. I would prefer to solve the problem with easier ways than dealing with the memory directly.
Hope this helps ...
Regards ..
Dear Sinan Salih .. Thank you for your help..
Yes, I tried such method but it didn't give me the required mission..I want to load the file before its' actual need. This method will load it again when I call the function ( DoSomething) . My goal is speeding up the final project execution through brings out every job that costs time.
I agree to Mr. Sinan about calling scripts. you can load file before you use it, like; put your import MySecondScript or from MySecondScript import * code to first line, but you have to be sure MyFirstScript.py and MySecondScript.py in same file directory. And divide your complex object to sub-functions, for example LoadFile():
AnalyzeFile():.... Etc. And go MyFirstScript.py and write MyFirstScript.LoadFile() if you want load files just one time don't use this function anymore. If you use this function in a loop, it will call your file multiple times. So be sure LoadFile() is out of a loop. I hope I understood your question well. I worked with huge datasets in Python. But this is the simple way.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python