Please note: This module isn't being developed anymore. However, there are some other good options for Python IPC.
This describes the module shm (written by Vladimir Marangozov) that gives access to System V shared memory and semaphores on *nix systems as well the module shm_wrapper (written by me) which is a companion module that offers more Pythonic access. Windows users, you're out of luck here; these modules only work on platforms that support System V shared objects. Most *nixes do (including OS X) but Windows does not.
You can download shm version 1.2.2 which contains the module shm_wrapper, the module shm, installation instructions and sample code. You might also want to read about some known bugs.
There's two modules available here. shm is essential. It is a single C language file that compiles into a Python module. The module's functions and features map fairly closely to the system calls like shmctl, but with nicer names.
The module shm_wrapper offers easier-to-use, higher-lever access to shm's features. For example, shm memory objects support reading the object's permissions through the perm attribute but you must set them by calling setperm(). By contrast, shm_wrapper exposes a simple gettable/settable attribute called permissions.
shm_wrapper also allows you to ignore the messy details surrounding keys and ids for the objects. Since the fact that ftok() is broken on most (all?) modern systems makes keys not very useful, there's no reason for you to spend time messing with them if you can avoid doing so.
If you're an old (or young) Sys V hacker, you might prefer shm's "closer to the metal" feel. Otherwise, use shm_wrapper.
This module provides classes that act as handles to shared memory and semaphores as well as functions to create and destroy each. Memory segments and semaphores are distinguished from one another by a guaranteed-unique key that this module generates automatically when the object is created.
This is a handle to a piece of shared memory that allows you to read and write to the memory and manipulate its attributes. Important methods and attributes are below. Not all of the attributes are documented here; refer to the module itself for information on some of the more obscure ones.
This is a handle to a semaphore that allows you to wait on the semaphore and also read and write to its attributes. Important methods and attributes are below. Not all of the attributes are documented here; refer to the module itself for information on some of the more obscure ones.
Vladimir Marangozov's shm module is the core of the onion, so to speak. It is a single C language file that compiles into a Python module. The module's functions and features map fairly closely to the system calls like shmctl, but with nicer names. The module has the following features:
The shm module defines its own shm.error.
A Memory object is a handle to a chunk of shared memory created by shm.create_memory(). Memory objects have these attributes and methods:
A Semaphore object is a handle to a semaphore created by shm.create_semaphore(). Semaphore objects have these attributes and methods:
Many systems (although not some versions of OS X) come with ipcs and ipcrm. The former shows existing shared memory, semaphores and message queues on your system and the latter allows you to remove them.
As of version 1.2, shm should be safe to use in threaded applications. (Previous versions were not).
The tarball includes code demonstrating the use of both shm_wrapper and shm. The demo code comes in the form of two complementary applications (two apps for each demo = a total of four apps). The apps are called Mrs. Premise and Mrs. Conclusion (as in, "Four hours to bury a cat?") and they converse with one another through shared memory. Run Mrs. Premise in one terminal and Mrs. Conclusion in another.
The conversation starts with Mrs. Premise creating the shared memory and then seeding it with a random string. Mrs. Conclusion then calculates the md5 hash of this string and writes that back into the shared memory. Mrs. Premise calculates the md5 hash of that string and writes it to the shared memory, and so it goes back and forth for as many iterations as are specified. Using md5 hashes allows Mrs. Premise and Mrs. Conclusion to verify that the other process correctly read what was written because it makes the response predictable. This is important for detecting memory corruption (see below).
You can specify whether or not you want the flow of the conversation to be controlled with a semaphore. (This option as well as the number of iterations are specified in DemoConstants.py.) If you opt not to use the semaphore, memory corruption (i.e. a "simultaneous" write by both processes to the shared memory) will probably occur if you run enough iterations. On my test systems -- a G4 Powerbook and a PIII running FreeBSD -- memory corruption always happened in less than 5000 iterations. If you use the semaphore, memory corruption will not occur. When either process detects corruption, it raises an AssertionError.
In addition to demonstrating the use of shm_wrapper and shm, this demo also illustrates why one needs to be careful with shared memory programming. Consider that even without semaphores these two processes writing to the same bit of memory as fast as they can still require thousands of iterations to step on one another. Now imagine a similarly infrequent bug, but instead of one that's caused by two deliberately careless demo programs, imagine one buried in thousands of lines of your code that causes Some Random Event every two weeks or so. You don't want to have to track down a bug like that. Speaking of bugs...
Bugs? My code never has bugs! However, there is a suboptimal anomoly...
If I'm correctly interpreting what I see when I run top, the newest version of shm still leaks memory during the create/destroy cycle of both Memory and Semaphore objects. This happens despite plugging several existing memory leaks and despite the fact that Python's garbage collector doesn't report anything amiss. The leak is tiny (about 12 or 13 bytes per create/destroy cycle) and it is released when the Python process ends, so this will only be a problem if you have a long-running process that creates and destroys lots of these objects. Note that the leaks occur only when creating (or destroying) new objects. Getting a handle to an existing object doesn't cause a problem.
I created a simple program called MemoryLeakDemo.py that demonstrates the problem. Start it in one window with top running in another window and you can watch the python process eat memory.
Most sample code that you see involving the use of System V semaphores/shared memory recommends ftok() to generate an integer key that's guaranteed to be unique on that machine. The main convenience of this is that processes can get the key (and thus a handle to the shared memory or semaphore) simply by using a previously agreed-upon filename. However, most modern implementations of ftok don't guarantee that it returns a unique key, which means it creates a key that may or may not work. If it doesn't work you have to fall back on a reliable alternative method of key generation, so you might as well just use the alternative in the first place.
The operating systems affected by this include OS X, Open/Net/FreeBSD, and Linux. (See the BUGS or NOTES section of the referenced man pages.)
In my experience, ftok frequently returned duplicate keys for different files in the same directory on my G4 Powerbook. Rather than probing and documenting ftok's limitations, I decided to just avoid it entirely and rely on Python's random number generator to provide keys for me. It's not as convenient as using a previously agreed-upon filename as a key, but buggy implementations of ftok don't (reliably) provide that either. One alternative is to have the creating process generate a random key and write that to a previously agreed-upon file.
Please let me know if you find a platform for which shm doesn't compile as-is.
On some systems, you might get this error when compiling:
shmmodule.c:186: error: redefinition of `union semun'
If you see this compile error, then you need to add your platform in setup.py to the if statement just above where HAVE_UNION_SEMUN is added.
Also, I suspect that BSD users (other than FreeBSD-ers) might see something like this:
shmmodule.c:1375: error: `PAGE_SIZE' undeclared (first use in this function)
You probably need to #include <machine/param.h>. That's already done for FreeBSD thanks to an #ifdef.
Last but not least, version 1.1.4 introduced some fancy code in setup.py. The need for this comes from the ipc_perm struct in ipc.h. It contains a member that's called key, _key or __key depending on your system. Setup.py should now autodetect which version you have, but the autodetection code is new and hasn't been well exercised (or exorcised). If if doesn't work for you and you're in a rush to get shm working, you can hack shmmodule.c yourself. Find the definition of the ipc_perm structure in /include/sys/ipc.h (/include/bits/ipc.h on some systems) and note which version of key it uses in its definition. Then in shmmodule.c, replace the code instances of IPC_PERM_KEY_NAME with the variable name from ipc.h.