Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-35813: Tests and docs for shared_memory #11816

Merged
merged 44 commits into from
Feb 24, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
720a8ea
Added tests for shared_memory submodule.
applio Feb 2, 2019
29a7f80
Added tests for ShareableList.
applio Feb 3, 2019
c56e29c
Fix bug in allocationn size during creation of empty ShareableList il…
applio Feb 3, 2019
c36de70
Initial set of docs for shared_memory module.
applio Feb 7, 2019
3c89c7c
Added docs for ShareableList, added doctree entry for shared_memory s…
applio Feb 8, 2019
5f4ba8f
Added examples to SharedMemoryManager docs, for ease of documentation…
applio Feb 9, 2019
f9aaa11
Wording tweaks to docs.
applio Feb 9, 2019
2377cfd
Fix test failures on Windows.
applio Feb 9, 2019
6bfa560
Added tests around SharedMemoryManager.
applio Feb 9, 2019
eaf7888
Documentation tweaks.
applio Feb 9, 2019
e166ed9
Fix inappropriate test on Windows.
applio Feb 9, 2019
0f18511
Further documentation tweaks.
applio Feb 11, 2019
a097dbb
Fix bare exception.
applio Feb 11, 2019
7c65017
Removed __copyright__.
applio Feb 11, 2019
da7731d
Fixed typo in doc, removed comment.
applio Feb 11, 2019
242a5e9
Merge remote-tracking branch 'upstream/master' into enh-tests-shmem
applio Feb 11, 2019
7bdfbbb
Updated SharedMemoryManager preliminary tests to reflect change of no…
applio Feb 11, 2019
eec4bb1
Added Sphinx doctest run controls.
applio Feb 11, 2019
1076567
CloseHandle should be in a finally block in case MapViewOfFile fails.
applio Feb 12, 2019
0be0531
Missed opportunity to use with statement.
applio Feb 12, 2019
1e5341e
Switch to self.addCleanup to spare long try/finally blocks and save o…
applio Feb 12, 2019
a5800a9
Simplify the posixshmem extension module.
nascheme Feb 13, 2019
34f1e9a
Added to doc around size parameter of SharedMemory.
applio Feb 16, 2019
9846290
Changed PosixSharedMemory.size to use os.fstat.
applio Feb 16, 2019
1f9bbf2
Change SharedMemory.buf to a read-only property as well as NamedShare…
applio Feb 17, 2019
69dd8a9
Marked as provisional per PEP411 in docstring.
applio Feb 17, 2019
8cf9ba3
Merge branch 'enh-tests-neilsimplify-shmem' into enh-tests-shmem
applio Feb 17, 2019
594140a
Changed SharedMemoryTracker to be private.
applio Feb 17, 2019
395709b
Removed registered Proxy Objects from SharedMemoryManager.
applio Feb 17, 2019
aa4a887
Removed shareable_wrap().
applio Feb 17, 2019
885592b
Removed shareable_wrap() and dangling references to it.
applio Feb 17, 2019
9001b76
Merge remote and local branches regarding elimination of
applio Feb 17, 2019
5848ec4
For consistency added __reduce__ to key classes.
applio Feb 17, 2019
6ff8eed
Fix for potential race condition on Windows for O_CREX.
applio Feb 18, 2019
06620e2
Remove unused imports.
applio Feb 18, 2019
868b83d
Update access to kernel32 on Windows per feedback from eryksun.
applio Feb 19, 2019
9d83b06
Moved kernel32 calls to _winapi.
applio Feb 20, 2019
715ded9
Removed ShareableList.copy as redundant.
applio Feb 20, 2019
6878533
Changes to _winapi use from eryksun feedback.
applio Feb 20, 2019
0d3d06f
Adopt simpler SharedMemory API, collapsing PosixSharedMemory and Wind…
applio Feb 21, 2019
05e26dd
Fix missing docstring on class, add test for ignoring size when attac…
applio Feb 21, 2019
7a3c7e5
Moved SharedMemoryManager to managers module, tweak to fragile test.
applio Feb 21, 2019
caf0a5d
Tweak to exception in OpenFileMapping suggested by eryksun.
applio Feb 21, 2019
12c097d
Mark a few dangling bits as private as suggested by Giampaolo.
applio Feb 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Documentation tweaks.
  • Loading branch information
applio committed Feb 9, 2019
commit eaf788854b8179cb1e740d7f1278f61b8184e3e5
25 changes: 13 additions & 12 deletions Doc/library/multiprocessing.shared_memory.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This module provides a class, :class:`SharedMemory`, for the allocation
and management of shared memory to be accessed by one or more processes
on a multicore or SMP machine. To assist with the life-cycle management
of shared memory especially across distinct processes, a
:class:`multiprocessing.managers.BaseManager` subclass,
:class:`~multiprocessing.managers.BaseManager` subclass,
:class:`SharedMemoryManager`, is also provided.

In this module, shared memory refers to "System V style" shared memory blocks
Expand Down Expand Up @@ -85,7 +85,7 @@ copying of data.
inside the shared memory block after ``unlink()`` has been called may
result in memory access errors. Note: the last process relinquishing
its hold on a shared memory block may call ``unlink()`` and
``close()`` in either order.
:meth:`close()` in either order.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm... what's the point of having both close and unlink? Why not simply having close() also destroy the memory blocks?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After all instances have called close(), there are still reasons to potentially want to have a shared memory block persist. An existing technique that some services use to preserve data even when the service dies unexpectedly is to store critical data in a shared memory block with an established name -- when the service restarts, it looks for the shared memory to still be there by name, thereby reducing the startup time of the service. Other use cases likely helped motivate the mentality behind the Windows implementation of shared memory which always preserves the shared memory block until the last process with a handle on it terminates -- Windows offers no way to perform an unlink() action ahead of time. Closest thing you can do on Windows is truncate its size to near zero but even that might not trigger a partial release of memory space. Across platforms, the common mentality and conscious design is to preserve shared memory until its release is explicitly demanded.


.. attribute:: buf

Expand Down Expand Up @@ -180,16 +180,17 @@ two distinct Python shells::

.. class:: SharedMemoryManager([address[, authkey]])

A subclass of :class:`multiprocessing.managers.BaseManager` which can be
A subclass of :class:`~multiprocessing.managers.BaseManager` which can be
used for the management of shared memory blocks across processes.

Instantiation of a :class:`SharedMemoryManager` causes a new process to
be started. This new process's sole purpose is to manage the life cycle
A call to :meth:`~multiprocessing.managers.BaseManager.start` on a
:class:`SharedMemoryManager` instance causes a new process to be started.
This new process's sole purpose is to manage the life cycle
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process' ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After double-checking with a technical editor just now, I have been assured that either is correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

of all shared memory blocks created through it. To trigger the release
of all shared memory blocks managed by that process, call
:func:`multiprocessing.managers.BaseManager.shutdown()` on the instance.
This triggers a :func:`SharedMemory.unlink()` call on all of the
:class:`SharedMemory` instances managed by that process and then
:meth:`~multiprocessing.managers.BaseManager.shutdown()` on the instance.
This triggers a :meth:`SharedMemory.unlink()` call on all of the
:class:`SharedMemory` objects managed by that process and then
stops the process itself. By creating ``SharedMemory`` instances
through a ``SharedMemoryManager``, we avoid the need to manually track
and trigger the freeing of shared memory resources.
Expand Down Expand Up @@ -315,13 +316,13 @@ shared memory blocks created using that manager are all released when the

.. method:: index(value)

Returns first index position of ``value``. Raises ValueError if
Returns first index position of ``value``. Raises :exc:`ValueError` if
``value`` is not present.

.. attribute:: format

Read-only attribute containing the struct packing format used by all
currently stored values.
Read-only attribute containing the :mod:`struct` packing format used by
all currently stored values.

.. attribute:: shm

Expand All @@ -340,7 +341,7 @@ instance:
>>> a[2] = -78.5
>>> a[2]
-78.5
>>> a[2] = 'dry ice' # Changing data types is supported as well.
>>> a[2] = 'dry ice' # Changing data types is supported as well
>>> a[2]
'dry ice'
>>> a[2] = 'larger than previously allocated storage space'
Expand Down
16 changes: 16 additions & 0 deletions Lib/multiprocessing/shared_memory.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,22 @@ def shareable_wrap(
format=None,
**kwargs
):
"""Provides a fast, convenient way to encapsulate objects that support
the buffer protocol as both producer and consumer, duplicating the
original object's data in shared memory and returning a new wrapped
object that when serialized via pickle does not serialize its data.

The function has been written in a general way to potentially work with
any object supporting the buffer protocol as producer and consumer. It
is known to work well with NumPy ndarrays. Among the Python core data
types and standard library, there are a number of objects supporting
the buffer protocol as a producer but not as a consumer.

Without an example of a producer+consumer of the buffer protocol in
the Python core to demonstrate the use of this function, this function
should likely be removed from this module and potentially be made
available instead via a pip-installable package."""

augmented_kwargs = dict(kwargs)
extras = dict(shape=shape, strides=strides, dtype=dtype, format=format)
for key, value in extras.items():
Expand Down