comm.hpp
This header file provides an object-oriented wrapper to MPI (Message Passing Interface) operations; utilizing MPI when available, or defaulting to a self communicator otherwise. It encapsulates common communication operations such as point-to-point communication, collective communication, data partitioning, sorting, and scatter operations.
Classes and Types
-
class Comm
Object oriented wrapper to MPI. It uses MPI when compiled with
mpicxxand the macroSCTL_HAVE_MPIis defined, otherwise, it defaults to the self communicator.
Constructors:
Comm(): Default constructor (self communicator).
explicit Comm(MPI_Comm): Construct from an existingMPI_Commhandle.
Comm(const Comm&): Copy constructor (callsMPI_Comm_dupto obtain an independent handle).
Comm(Comm&&) noexcept: Move constructor (steals theMPI_Commhandle without duplication).
operator=(copy and move): Assignment with the same semantics as the corresponding constructors.Static Methods:
MPI_Init(argc, argv): Initialize MPI.
MPI_Finalize(): Finalize MPI.
Self(): Get the self communicator.
World(): Get the world communicator.
Rank(): Get the rank of the current process.Methods:
Size(): Get the size of the communicator.
GetMPI_Comm(): Get the underlyingMPI_Commhandle.
Barrier(): Synchronize all processes.
Isend(sbuf, scount, dest, tag): Non-blocking send.
Irecv(rbuf, rcount, source, tag): Non-blocking receive.
Wait(req_ptr): Wait for a non-blocking send or receive.
Bcast(buf, count, root): Broadcast to all processes in the communicator.
Allgather(sbuf, scount, rbuf, rcount): Gather and concatenate equal-size messages from all processes.
Allgatherv(sbuf, scount, rbuf, rcounts, rdispls): Gather and concatenate messages of different lengths from all processes.
Alltoall(sbuf, scount, rbuf, rcount): Perform all-to-all operation for equal-size messages.
Ialltoallv_sparse(sbuf, scounts, sdispls, rbuf, rcounts, rdispls, tag): Sparse all-to-all communication.
Alltoallv(sbuf, scounts, sdispls, rbuf, rcounts, rdispls): All-to-all communication with varying send and receive counts and displacements.
Allreduce(sbuf, rbuf, count, op): Perform an all-reduce operation.
Scan(sbuf, rbuf, count, op): Perform a scan operation.
PartitionW(nodeList, wts_): Perform a weighted partitioning of a vector.
PartitionN(v, N): Partition a vector given the number of local elements after partitioning.
PartitionS(nodeList, splitter, comp): Perform a partitioning of a vector with a splitter element using a custom comparison function.
HyperQuickSort(arr, SortedElem, comp): Sort the elements of an array using HyperQuickSort algorithm with a custom comparison function.
SortScatterIndex(key, scatter_index, split_key): Generate scatter indices corresponding to a sorted array.
ScatterForward(data_, scatter_index): Scatter data elements forward using the provided scatter index.
ScatterReverse(data_, scatter_index_, loc_size_): Scatter data elements in reverse using the provided scatter index.Usage guide: Using the Comm class
-
enum class sctl::CommOp
Operation types for Allreduce and Scan collective operations.
Values:
-
enumerator SUM
-
enumerator MIN
-
enumerator MAX
-
enumerator SUM
#ifndef _SCTL_COMM_HPP_
#define _SCTL_COMM_HPP_
#include <functional> // for less
#include <map> // for multimap
#include <memory> // for shared_ptr
#include <utility> // for move
#include <vector> // for vector
#include "sctl/common.hpp" // for Long, Integer, sctl
#include "sctl/iterator.hpp" // for ConstIterator, Iterator
#ifdef SCTL_HAVE_MPI
#include <mpi.h>
#include <stack> // for stack
#endif
#ifdef SCTL_HAVE_PETSC
#include <petscsys.h>
#endif
namespace sctl {
template <class ValueType> class Vector;
/**
* Operation types for Allreduce and Scan collective operations.
*/
enum class CommOp {
SUM,
MIN,
MAX
};
/**
* Object oriented wrapper to MPI. It uses MPI when compiled with `mpicxx` and the macro `SCTL_HAVE_MPI`
* is defined, otherwise, it defaults to the *self* communicator.
*/
class Comm {
public:
/**
* Initialize MPI.
*/
static void MPI_Init(int* argc, char*** argv);
/**
* Finalize MPI.
*/
static void MPI_Finalize();
/**
* Default constructor, initializes to the *self* communicator.
*/
Comm();
#ifdef SCTL_HAVE_MPI
/**
* Convert MPI_Comm to Comm.
*/
explicit Comm(const MPI_Comm mpi_comm) : impl_(std::make_shared<Impl>()) { impl_->Init(mpi_comm); }
#endif
/**
* Copy constructor. Cheap reference-share of the underlying `Impl`
* (shared_ptr increment) — no `MPI_Comm_dup`. The original
* `MPI_Comm` is released only when the last copy is destroyed.
*/
Comm(const Comm& c);
/**
* Move constructor. Transfers the `Impl` pointer; leaves `c` empty so
* its destructor is a no-op.
*/
Comm(Comm&& c) noexcept;
/**
* *self* communicator.
*/
[[nodiscard]] static Comm Self();
/**
* *world* communicator.
*/
[[nodiscard]] static Comm World();
/**
* Copy assignment. Reference-shares `c`'s underlying `Impl`. Releases
* the prior `Impl` (which frees its `MPI_Comm` if this was the last ref).
*/
Comm& operator=(const Comm& c);
/**
* Move assignment. Transfers `c`'s `Impl` pointer; `c` is left empty.
*/
Comm& operator=(Comm&& c) noexcept;
/**
* Destructor.
*/
~Comm();
#ifdef SCTL_HAVE_MPI
/**
* Convert to MPI_Comm.
*/
[[nodiscard]] const MPI_Comm& GetMPI_Comm() const noexcept { return impl_->mpi_comm_; }
#endif
/**
* Split communicator.
*
* @param[in] clr identify different communicator groups.
*/
[[nodiscard]] Comm Split(Integer clr) const;
/**
* @return rank of the current process.
*/
[[nodiscard]] Integer Rank() const noexcept;
/**
* @return size of this communicator.
*/
[[nodiscard]] Integer Size() const noexcept;
/**
* Synchronize all processes.
*/
void Barrier() const;
/**
* Opaque, move-only handle for an outstanding non-blocking communication
* request returned by Isend(), Irecv(), or Ialltoallv_sparse(). The handle
* owns a pooled MPI_Request slot inside the `Comm`; that slot is released
* when the handle is passed to Wait() exactly once.
*
* Lifetime contract:
* - The handle must be passed to Wait() (via `std::move`) before it goes
* out of scope. Dropping a non-empty Request would orphan the
* underlying MPI_Request; debug builds trap in the destructor.
* - After Wait() consumes a handle, the handle is empty and must not be
* reused. Empty handles (default-constructed or moved-from) are safe
* to destroy and may also be passed to Wait() as a no-op.
* - The handle is non-copyable; ownership transfers via move only.
*/
class Request {
public:
Request() = default;
Request(const Request&) = delete;
Request& operator=(const Request&) = delete;
Request(Request&& other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
Request& operator=(Request&& other) noexcept {
SCTL_ASSERT_MSG(!ptr_, "Comm::Request: overwriting a non-empty request leaks the pending MPI_Request; Wait() must be called first");
ptr_ = other.ptr_;
other.ptr_ = nullptr;
return *this;
}
~Request() {
SCTL_ASSERT_MSG(!ptr_, "Comm::Request destroyed without Wait(); the underlying MPI_Request is leaked");
}
explicit operator bool() const noexcept { return ptr_ != nullptr; }
private:
friend class Comm;
explicit Request(void* p) noexcept : ptr_(p) {}
void* release_() noexcept { void* p = ptr_; ptr_ = nullptr; return p; }
void* ptr_ = nullptr;
};
/**
* Non-blocking send.
*
* @tparam SType type of the send-data.
*
* @param[in] sbuf const-iterator to the send buffer.
*
* @param[in] scount number of elements to send.
*
* @param[in] dest the rank of the destination process.
*
* @param[in] tag identifier tag to be matched at receive.
*
* @return a Request handle. Must be passed to Wait() (via `std::move`)
* before going out of scope; otherwise the underlying MPI_Request
* is leaked. The return value is `[[nodiscard]]` — discarding it
* is a programmer error.
*/
template <class SType> [[nodiscard]] Request Isend(ConstIterator<SType> sbuf, Long scount, Integer dest, Integer tag = 0) const;
/**
* Non-blocking synchronous send. Semantically equivalent to `Isend` except
* that completion (via `Wait`) is delayed until the matching receive has
* been posted at the destination — i.e. always uses the rendezvous protocol,
* never the eager fast-path. Intended for the post-Irecvs-then-post-sends
* pattern, where this synchronization is already satisfied by construction
* and Issend's only practical effect is to bound unexpected-message-buffer
* pressure on the receiver and to surface mismatched-receive bugs as
* immediate deadlocks rather than silent buffering at scale.
*
* @tparam SType type of the send-data.
*
* @param[in] sbuf const-iterator to the send buffer.
*
* @param[in] scount number of elements to send.
*
* @param[in] dest the rank of the destination process.
*
* @param[in] tag identifier tag to be matched at receive.
*
* @return a Request handle. Same lifetime contract as Isend(): must be
* passed to Wait() before going out of scope.
*/
template <class SType> [[nodiscard]] Request Issend(ConstIterator<SType> sbuf, Long scount, Integer dest, Integer tag = 0) const;
/**
* Non-blocking receive.
*
* @tparam RType type of the receive-data.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] rcount number of elements to receive.
*
* @param[in] source the rank of the source process.
*
* @param[in] tag identifier tag to be matched by the corresponding Isend.
*
* @return a Request handle. Same lifetime contract as Isend().
*/
template <class RType> [[nodiscard]] Request Irecv(Iterator<RType> rbuf, Long rcount, Integer source, Integer tag = 0) const;
/**
* Wait for a non-blocking send or receive. Consumes the handle by value;
* after the call, the moved-from variable in the caller is empty.
*
* @param[in] req Request handle returned by Isend(), Irecv(), or
* Ialltoallv_sparse(). May be empty (no-op).
*/
void Wait(Request req) const;
/**
* Broadcast to all processes in the communicator.
*
* @tparam Type type of the data.
*
* @param[in,out] buff send-buffer on the sending process, or the receive buffer on the receiving process.
*
* @param[in] count number of elements in the message.
*
* @param[in] root rank of the sending process.
*/
template <class Type> void Bcast(Iterator<Type> buf, Long count, Integer root) const;
/**
* Gather and concatenate equal size messages from all processes in the communicator.
*
* @tparam SType type of the send-data.
* @tparam RType type of the receive-data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[in] scount number of elements in the send buffer.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] rcount number of elements in the receive buffer. The total number of elements in the receive buffer
* should be `rcount * Size()`.
*/
template <class SType, class RType> void Allgather(ConstIterator<SType> sbuf, Long scount, Iterator<RType> rbuf, Long rcount) const;
/**
* Gather and concatenate messages of different lengths from all processes in the communicator.
*
* @tparam SType type of the send-data.
* @tparam RType type of the receive-data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[in] scount number of elements in the send buffer on each process.
*
* @param[out] rbuf iterator to the receive buffer where the gathered data is stored.
*
* @param[in] rcounts iterator to the number of elements to receive from each process.
*
* @param[in] rdispls iterator to the displacements in the receive buffer where the data from each process is stored.
*/
template <class SType, class RType> void Allgatherv(ConstIterator<SType> sbuf, Long scount, Iterator<RType> rbuf, ConstIterator<Long> rcounts, ConstIterator<Long> rdispls) const;
/**
* Perform all-to-all operation for equal size messages.
*
* @tparam SType type of the send-data.
* @tparam RType type of the receive-data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[in] scount number of elements in each send message. Size of send-buffer must be `scount * Size()`.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] rcount number of elements in each receive message. Size of receive-buffer must be `rcount * Size()`.
*/
template <class SType, class RType> void Alltoall(ConstIterator<SType> sbuf, Long scount, Iterator<RType> rbuf, Long rcount) const;
/**
* Sparse all-to-all communication.
*
* @tparam SType type of the send-data.
* @tparam RType type of the receive-data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[in] scounts iterator to the number of elements to send to each process.
*
* @param[in] sdispls iterator to the displacements in the send buffer.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] rcounts iterator to the number of elements to receive from each process.
*
* @param[in] rdispls iterator to the displacements in the receive buffer.
*
* @param[in] tag identifier tag to be matched by all processes in the communicator.
*
* @return a Request handle. Same lifetime contract as Isend(): must be
* passed to Wait() before destruction.
*/
template <class SType, class RType> [[nodiscard]] Request Ialltoallv_sparse(ConstIterator<SType> sbuf, ConstIterator<Long> scounts, ConstIterator<Long> sdispls, Iterator<RType> rbuf, ConstIterator<Long> rcounts, ConstIterator<Long> rdispls, Integer tag = 0) const;
/**
* All-to-all communication with varying send and receive counts and displacements.
*
* @tparam Type type of the data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[in] scounts iterator to the number of elements to send to each process.
*
* @param[in] sdispls iterator to the displacements in the send buffer.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] rcounts iterator to the number of elements to receive from each process.
*
* @param[in] rdispls iterator to the displacements in the receive buffer.
*/
template <class Type> void Alltoallv(ConstIterator<Type> sbuf, ConstIterator<Long> scounts, ConstIterator<Long> sdispls, Iterator<Type> rbuf, ConstIterator<Long> rcounts, ConstIterator<Long> rdispls) const;
/**
* All-to-all communication via recursive divide-and-conquer (bitonic
* split-exchange), ported from pvfmm's `par::Mpi_Alltoallv_dense`.
*
* Recursively halves the rank group and performs a single `MPI_Sendrecv`
* payload exchange between matched halves at each level, rearranging the
* combined buffer in-rank between iterations. O(log p) iterations,
* O(log p) intermediate buffers, O(log p · N_local) byte movement.
*
* Functionally equivalent to `Alltoallv`, but uses the hand-rolled
* algorithm rather than the vendor `MPI_Alltoallv`. Useful when the
* vendor implementation is known to be inferior at the relevant
* (p, message_size) shape — typically a niche case.
*
* @tparam Type trivially-copyable payload type.
*
* @param[in] sbuf iterator to the send buffer.
* @param[in] scounts per-rank send counts (length `Size()`).
* @param[in] sdispls per-rank send-buffer displacements (length `Size()`).
* @param[out] rbuf iterator to the receive buffer.
* @param[in] rcounts per-rank receive counts (length `Size()`).
* @param[in] rdispls per-rank receive-buffer displacements (length `Size()`).
*/
template <class Type> void Alltoallv_dense(ConstIterator<Type> sbuf, ConstIterator<Long> scounts, ConstIterator<Long> sdispls, Iterator<Type> rbuf, ConstIterator<Long> rcounts, ConstIterator<Long> rdispls) const;
/**
* Perform an all-reduce operation.
*
* @tparam Type type of the data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] count number of elements.
*
* @param[in] op reduction operation.
*/
template <class Type> void Allreduce(ConstIterator<Type> sbuf, Iterator<Type> rbuf, Long count, CommOp op) const;
/**
* Perform a scan operation.
*
* @tparam Type type of the data.
*
* @param[in] sbuf iterator to the send buffer.
*
* @param[out] rbuf iterator to the receive buffer.
*
* @param[in] count number of elements.
*
* @param[in] op scan operation.
*/
template <class Type> void Scan(ConstIterator<Type> sbuf, Iterator<Type> rbuf, Long count, CommOp op) const;
/**
* Perform a weighted partitioning of a vector.
*
* @tparam Type type of the vector elements.
*
* @param[in,out] nodeList vector to partition.
*
* @param[in] wts_ optional weights for weighted partitioning.
*/
template <class Type> void PartitionW(Vector<Type>& nodeList, const Vector<Long>* wts_ = nullptr) const;
/**
* Partition a vector given the number of local elements after partitioning.
*
* @tparam Type type of the vector elements.
*
* @param[in,out] v vector to partition.
*
* @param[in] N number of local elements after partitioning.
*/
template <class Type> void PartitionN(Vector<Type>& v, Long N) const;
/**
* Perform a partitioning of a vector with a splitter element.
*
* @tparam Type type of the vector elements.
* @tparam Compare comparison function type.
*
* @param[in,out] nodeList vector to partition.
*
* @param[in] splitter element to partition around.
*
* @param[in] comp comparison function for elements.
*/
template <class Type, class Compare> void PartitionS(Vector<Type>& nodeList, const Type& splitter, Compare comp) const;
/**
* Perform a partitioning of a vector with a splitter element using the default comparison function.
*
* @tparam Type type of the vector elements.
*
* @param[in,out] nodeList vector to partition.
*
* @param[in] splitter element to partition around.
*/
template <class Type> void PartitionS(Vector<Type>& nodeList, const Type& splitter) const {
PartitionS(nodeList, splitter, std::less<Type>());
}
/**
* Sorts the elements of an array using HyperQuickSort algorithm with a custom comparison function.
*
* @tparam Type type of the elements in the array.
* @tparam Compare comparison function type.
*
* @param[in] arr input array to be sorted.
*
* @param[out] SortedElem sorted array.
*
* @param[in] comp comparison function for elements.
*/
template <class Type, class Compare> void HyperQuickSort(const Vector<Type>& arr, Vector<Type>& SortedElem, Compare comp, bool partition = true) const;
/**
* Sorts the elements of an array using HyperQuickSort algorithm with the default comparison function.
*
* @tparam Type type of the elements in the array.
*
* @param[in] arr input array to be sorted.
*
* @param[out] SortedElem sorted array.
*/
template <class Type> void HyperQuickSort(const Vector<Type>& arr, Vector<Type>& SortedElem) const {
HyperQuickSort(arr, SortedElem, std::less<Type>());
}
/**
* Sorts the elements of a distributed array using a single-pass sample sort (regular
* sampling -> one Alltoallv -> k-way merge). Scales better than HyperQuickSort for large
* arrays by avoiding its O(log p) merge/comm-split rounds.
*
* @tparam Type type of the elements in the array.
* @tparam Compare comparison functor type.
*
* @param[in] arr input array to be sorted.
* @param[out] SortedElem sorted array.
* @param[in] comp comparison function for elements.
* @param[in] partition if true (default) the output is equally partitioned across ranks
* (via PartitionW), matching HyperQuickSort; if false the output is globally sorted but
* only approximately balanced (skip when a PartitionS/repartition follows).
*/
template <class Type, class Compare> void SampleSort(const Vector<Type>& arr, Vector<Type>& SortedElem, Compare comp, bool partition = true) const;
/**
* Sorts the elements of a distributed array using a single-pass sample sort with the
* default comparison function.
*/
template <class Type> void SampleSort(const Vector<Type>& arr, Vector<Type>& SortedElem) const {
SampleSort(arr, SortedElem, std::less<Type>());
}
/**
* Distributed sort using a caller-provided per-rank splitter (like PartitionS): each rank
* supplies its own lower boundary, gathered internally, and rank r ends up with the
* globally-sorted elements in [splitter_r, splitter_{r+1}). Skips splitter selection and
* PartitionW, sorting and partitioning in one pass when the boundaries are already known.
*
* @tparam Type type of the elements in the array.
* @tparam Compare comparison functor type.
*
* @param[in] arr input array to be sorted.
* @param[out] SortedElem sorted, splitter-partitioned array.
* @param[in] splitter this rank's lower boundary key (splitters must be ascending by rank).
* @param[in] comp comparison function for elements.
*/
template <class Type, class Compare> void SampleSort(const Vector<Type>& arr, Vector<Type>& SortedElem, const Type& splitter, Compare comp) const;
/**
* Distributed sort using a per-rank splitter and the default comparison function.
*/
template <class Type> void SampleSort(const Vector<Type>& arr, Vector<Type>& SortedElem, const Type& splitter) const {
SampleSort(arr, SortedElem, splitter, std::less<Type>());
}
/**
* Generates scatter indices corresponding to a sorted array.
*
* @tparam Type type of the elements in the array.
*
* @param[in] key array of keys to be sorted.
*
* @param[out] scatter_index array of indices giving the original global-position of each element in the sorted array.
*
* @param[in] split_key optional key to determine the partitioning of the sorted array between processes.
*/
template <class Type> void SortScatterIndex(const Vector<Type>& key, Vector<Long>& scatter_index, const Type* split_key = nullptr) const;
/**
* Scatter data elements forward (i.e. sorted to unsorted order) using the provided scatter index.
*
* @tparam Type type of the data elements.
*
* @param[in,out] data_ data elements to be scattered.
*
* @param[in] scatter_index array of indices giving the original global-position of each element in the sorted array.
*/
template <class Type> void ScatterForward(Vector<Type>& data_, const Vector<Long>& scatter_index) const;
/**
* Scatter data elements in reverse (i.e. unsorted to sorted order) using the provided scatter index.
*
* @tparam Type type of the data elements.
*
* @param[in,out] data_ data elements to be scattered.
*
* @param[in] scatter_index array of indices giving the original global-position of each element in the sorted array.
*
* @param[in] loc_size_ number of local element after rearrangement.
*/
template <class Type> void ScatterReverse(Vector<Type>& data_, const Vector<Long>& scatter_index_, Long loc_size_ = 0) const;
private:
/**
* Structure to hold a pair of elements for sorting.
*/
template <typename A, typename B> struct SortPair {
int operator<(const SortPair<A, B>& p1) const { return key < p1.key; }
A key;
B data;
};
/**
* Core of SampleSort: given a locally-sorted array `loc` and this rank's lower boundary
* `splitter` (one value per rank, gathered internally so the split is always consistent),
* redistribute (one Alltoallv) and parallel-merge so rank r ends up with the globally-sorted
* elements in [splitter_r, splitter_{r+1}).
*/
template <class Type, class Compare> void DistributeAndMerge(const Vector<Type>& loc, const Type& splitter, Vector<Type>& SortedElem, Compare comp) const;
/**
* Determine this process's lower-boundary splitter for a load-balanced distributed sort via
* iterative exact-rank histogramming. Generic (uses only `comp` and actual elements), O(npes)
* communication per round; the resulting balance is independent of the data distribution.
*
* @param[in] loc locally-sorted elements on this process.
* @param[in] totSize total element count across all processes.
* @param[in] comp comparison functor.
* @return the value at global rank Rank()*totSize/npes (rank 0's return value is unused).
*/
template <class Type, class Compare> Type DetermineSplitter(const Vector<Type>& loc, Long totSize, Compare comp) const;
#ifdef SCTL_HAVE_MPI
/**
* Internal reference-counted state for the duplicated `MPI_Comm` and
* its per-comm `MPI_Request` pool. Held via `shared_ptr<Impl>` so that
* `Comm` copy/assignment are O(1) and the underlying `MPI_Comm_free`
* fires exactly once when the last `Comm` referencing it is destroyed.
*/
struct Impl {
int mpi_rank_;
int mpi_size_;
int mpi_tag_ub_;
MPI_Comm mpi_comm_;
mutable std::stack<void*> req;
Impl();
~Impl();
Impl(const Impl&) = delete;
Impl(Impl&&) = delete;
Impl& operator=(const Impl&) = delete;
Impl& operator=(Impl&&) = delete;
/**
* Initialize the impl by duplicating the given MPI_Comm.
* Collective on the input communicator.
*/
void Init(MPI_Comm mpi_comm);
};
template <class Type> static MPI_Op GetMPIOp(CommOp op);
static void RegisterDatatype(MPI_Datatype datatype);
static void RegisterOp(MPI_Op op);
static void FreeRegisteredHandles();
static std::vector<MPI_Datatype>& DatatypeRegistry();
static std::vector<MPI_Op>& OpRegistry();
Vector<MPI_Request>& NewReq(Long request_count) const;
void DelReq(Vector<MPI_Request>* req_ptr) const;
std::shared_ptr<Impl> impl_;
template <class Type> class CommDatatype;
#else
mutable std::multimap<Integer, ConstIterator<char>> send_req;
mutable std::multimap<Integer, Iterator<char>> recv_req;
#endif
};
} // end namespace
#endif // _SCTL_COMM_HPP_