|
Communication devices are used for interprocess communication. Currently csWMPI supports
two communication devices. We have built the interfaces so that a third party can develop
new communication devices.
If you are interested in developing your own device or if you need more information
on the subject please contact us: csWMPI-support@criticalsoftware.com
The two communication devices currently supported are:
SPECIAL NOTE to Gigabit Ethernet users: Please refer to the tcp Device section
to configure the tcp device to get the best performance.
The tcp device uses the machine's TCP/IP stack for interprocess communication.
Besides from being used to transfer csWMPI application data, it is also used in the start-up
phase and when computations are joining. Hence, a TCP/IP stack is necessary in order to run
csWMPI applications.
|
Environment variable: |
Description: |
|
csWMPI_TCP_RENDEZVOUS_START
|
Specifies the minimum message size (in bytes) of messages transferred using a rendezvous
protocol. The rendezvous protocol will synchronize both sender and receiver and the data
will be sent only when the receiver specified the buffer to receive the data, hence called
the matching receive function. This reduces the number of memory copy operations as well
as avoids allocation and de-allocation of big memory buffers.
Default value: 1048576 (1 MB).
|
|
csWMPI_TCP_RECV_BUFFER
|
Specifies the TCP socket's receive buffer in bytes.
Default value: 32768 (32 KB)
|
|
csWMPI_TCP_SEND_BUFFER
|
Specifies the TCP socket's send buffer in bytes.
Default value: 16384 (16 KB)
|
|
csWMPI_TCP_MAX_SEND_CHUNK_SIZE
|
Specifies the biggest buffer size that the system will be asked to send within a system call (Windows only).
Default value: 11680
|
|
csWMPI_TCP_DATA_BLOCK_SIZE
|
The maximum size of data sent without sending an intermediate message header. This intermediate message
header allows the device to send "request for data" headers in the middle of big messages, increasing
performance when exchanging big messages in between two peers.
Default value: 131072 (128KB)
|
|
csWMPI_TCP_RT_SIGNAL
|
The tcp device for Linux uses a realtime signal during communication.
This signal cannot be used by any other library of the process.
Default value (Linux only): SIGRTMIN+5
|
SPECIAL NOTE TO Gigabit Ethernet users: The default values of these variables were calculated to achieve good performance results
in Fast Ethernet. When using Gigabit Ethernet interconnectors, please set the following
environment variables to achieve better performance.
csWMPI_TCP_RECV_BUFFER=65536
csWMPI_TCP_SEND_BUFFER=32768
csWMPI_TCP_MAX_SEND_CHUNK_SIZE=23520
csWMPI_TCP_RENDEZVOUS_START=524288
csWMPI_TCP_DATA_BLOCK_SIZE=1048576
These variables need to be set on the starting node only. We recommend you to set them system wide, through the Properties
of "My computer" in
Windows and on a Shell resource file read at login, like .bashrc or .profile, in Linux.
The shared memory device, shmem, is used as a communication device
for processes residing on the same machine. If shared memory is used on a machine, a chunk of memory
is allocated at startup and used by all the process on that machine for their interprocess
communication. By taking advantages of the features provided by the operating system, messages can be
copied directly from the memory of one process to another. Even though this requires a rendezvous
between the sender and the receiver, the method proves efficient for messages above a certain
size.
You can fine-tune the shmem device using the following environment
variables:
|
Environment variable: |
Description: |
|
csWMPI_SHMEM_SIZE
|
Specifies the size (in bytes) of the memory region shared by processes on the
machine. The minimum and default size of this region is 16MB.
|
|
csWMPI_SHMEM_END_POINT
|
The end point of the shared memory region. The default is the bottom of the address
space. Unfortunately some other libraries might load into or otherwise use this region.
In cases where such territorial conflicts occur between libraries, you can set this variable
to some unused memory region.
|
|
csWMPI_SHMEM_UNIVERSE_SIZE
|
Specifies the maximum number of processes that can connect to the shared
memory device on a single machine.
Default value: 256
|
|
csWMPI_SHMEM_RENDEZVOUS_START
|
For big messages, a rendezvous protocol is needed to avoid flooding the share memory segment.
In MS Windows, the rendezvous protocol is implemented through a zero-copy operation, copying
directly from one process's address to another's. In Linux, the message will be transferred
in small pieces of data. Since such a protocol requires the two participating
processes to rendezvous, it is often only beneficial to use it
for messages above some size. Messages below this size are
temporarily stored in the memory region shared by the processes local to a machine.
Default value for MS Windows: 65536 (64 KB)
Default value for Linux: 131072 (128 KB)
|
|
csWMPI_SHMEM_RT_SIGNAL
|
The shmem device for Linux uses a realtime signal during communication.
This signal cannot be used by any other library of the process.
Default value (Linux only): SIGRTMIN+3
|
|