rdmapp::qp Class Reference

This class is an abstraction of an Infiniband Queue Pair. More...

#include <rdmapp/qp.h>

Inheritance diagram for rdmapp::qp:

Collaboration diagram for rdmapp::qp:

Classes
class	recv_awaitable
class	send_awaitable

Public Member Functions
	qp (const uint16_t remote_lid, const uint32_t remote_qpn, const uint32_t remote_psn, const union ibv_gid remote_gid, std::shared_ptr< pd > pd, std::shared_ptr< cq > cq, std::shared_ptr< srq > srq=nullptr)
	Construct a new qp object. The Queue Pair will be created with the given remote Queue Pair parameters. Once constructed, the Queue Pair will be in the RTS state.
	qp (const uint16_t remote_lid, const uint32_t remote_qpn, const uint32_t remote_psn, const union ibv_gid remote_gid, std::shared_ptr< pd > pd, std::shared_ptr< cq > recv_cq, std::shared_ptr< cq > send_cq, std::shared_ptr< srq > srq=nullptr)
	Construct a new qp object. The Queue Pair will be created with the given remote Queue Pair parameters. Once constructed, the Queue Pair will be in the RTS state.
	qp (std::shared_ptr< pd > pd, std::shared_ptr< cq > cq, std::shared_ptr< srq > srq=nullptr)
	Construct a new qp object. The constructed Queue Pair will be in INIT state.
	qp (std::shared_ptr< pd > pd, std::shared_ptr< cq > recv_cq, std::shared_ptr< cq > send_cq, std::shared_ptr< srq > srq=nullptr)
	Construct a new qp object. The constructed Queue Pair will be in INIT state.
void	post_send (struct ibv_send_wr const &send_wr, struct ibv_send_wr *&bad_send_wr)
	This function is used to post a send work request to the Queue Pair.
void	post_recv (struct ibv_recv_wr const &recv_wr, struct ibv_recv_wr *&bad_recv_wr) const
	This function is used to post a recv work request to the Queue Pair. It will be posted to either RQ or SRQ depending on whether or not SRQ is set.
send_awaitable	send (void *buffer, size_t length)
	This method sends local buffer to remote. The address will be registered as a memory region first and then deregistered upon completion.
send_awaitable	write (remote_mr const &remote_mr, void *buffer, size_t length)
	This method writes local buffer to a remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.
send_awaitable	write_with_imm (remote_mr const &remote_mr, void *buffer, size_t length, uint32_t imm)
	This method writes local buffer to a remote memory region with an immediate value. The local buffer will be registered as a memory region first and then deregistered upon completion.
send_awaitable	read (remote_mr const &remote_mr, void *buffer, size_t length)
	This method reads to local buffer from a remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.
send_awaitable	fetch_and_add (remote_mr const &remote_mr, void *buffer, size_t length, uint64_t add)
	This method performs an atomic fetch-and-add operation on the given remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.
send_awaitable	compare_and_swap (remote_mr const &remote_mr, void *buffer, size_t length, uint64_t compare, uint64_t swap)
	This method performs an atomic compare-and-swap operation on the given remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.
recv_awaitable	recv (void *buffer, size_t length)
	This method posts a recv request on the queue pair. The buffer will be filled with data received. The local buffer will be registered as a memory region first and then deregistered upon completion.
send_awaitable	send (std::shared_ptr< local_mr > local_mr)
	This function sends a registered local memory region to remote.
send_awaitable	write (remote_mr const &remote_mr, std::shared_ptr< local_mr > local_mr)
	This function writes a registered local memory region to remote.
send_awaitable	write_with_imm (remote_mr const &remote_mr, std::shared_ptr< local_mr > local_mr, uint32_t imm)
	This function writes a registered local memory region to remote with an immediate value.
send_awaitable	read (remote_mr const &remote_mr, std::shared_ptr< local_mr > local_mr)
	This function reads to local memory region from remote.
send_awaitable	fetch_and_add (remote_mr const &remote_mr, std::shared_ptr< local_mr > local_mr, uint64_t add)
	This function performs an atomic fetch-and-add operation on the given remote memory region.
send_awaitable	compare_and_swap (remote_mr const &remote_mr, std::shared_ptr< local_mr > local_mr, uint64_t compare, uint64_t swap)
	This function performs an atomic compare-and-swap operation on the given remote memory region.
recv_awaitable	recv (std::shared_ptr< local_mr > local_mr)
	This function posts a recv request on the queue pair. The buffer will be filled with data received.
std::vector< uint8_t >	serialize () const
	This function serializes a Queue Pair prepared to be sent to a buffer.
std::vector< uint8_t > &	user_data ()
	This function provides access to the extra user data of the Queue Pair.
std::shared_ptr< pd >	pd_ptr () const
	This function provides access to the Protection Domain of the Queue Pair.
void	rtr (uint16_t remote_lid, uint32_t remote_qpn, uint32_t remote_psn, union ibv_gid remote_gid)
	This function transitions the Queue Pair to the RTR state.
void	rts ()
	This function transitions the Queue Pair to the RTS state.
Public Member Functions inherited from rdmapp::noncopyable
	noncopyable (noncopyable &&)=default
	noncopyable (noncopyable const &)=delete
noncopyable &	operator= (noncopyable const &)=delete
noncopyable &	operator= (noncopyable &&)=default

Detailed Description

This class is an abstraction of an Infiniband Queue Pair.

Constructor & Destructor Documentation

qp() [1/4]

rdmapp::qp::qp	(	const uint16_t	remote_lid,
		const uint32_t	remote_qpn,
		const uint32_t	remote_psn,
		const union ibv_gid	remote_gid,
		std::shared_ptr< pd >	pd,
		std::shared_ptr< cq >	cq,
		std::shared_ptr< srq >	srq = nullptr
	)

Construct a new qp object. The Queue Pair will be created with the given remote Queue Pair parameters. Once constructed, the Queue Pair will be in the RTS state.

Parameters

remote_lid	The LID of the remote Queue Pair.
remote_qpn	The QPN of the remote Queue Pair.
remote_psn	The PSN of the remote Queue Pair.
pd	The protection domain of the new Queue Pair.
cq	The completion queue of both send and recv work completions.
srq	(Optional) If set, all recv work requests will be posted to this SRQ.

qp() [2/4]

rdmapp::qp::qp	(	const uint16_t	remote_lid,
		const uint32_t	remote_qpn,
		const uint32_t	remote_psn,
		const union ibv_gid	remote_gid,
		std::shared_ptr< pd >	pd,
		std::shared_ptr< cq >	recv_cq,
		std::shared_ptr< cq >	send_cq,
		std::shared_ptr< srq >	srq = nullptr
	)

Construct a new qp object. The Queue Pair will be created with the given remote Queue Pair parameters. Once constructed, the Queue Pair will be in the RTS state.

Parameters

remote_lid	The LID of the remote Queue Pair.
remote_qpn	The QPN of the remote Queue Pair.
remote_psn	The PSN of the remote Queue Pair.
pd	The protection domain of the new Queue Pair.
recv_cq	The completion queue of recv work completions.
send_cq	The completion queue of send work completions.
srq	(Optional) If set, all recv work requests will be posted to this SRQ.

qp() [3/4]

rdmapp::qp::qp	(	std::shared_ptr< pd >	pd,
		std::shared_ptr< cq >	cq,
		std::shared_ptr< srq >	srq = nullptr
	)

Construct a new qp object. The constructed Queue Pair will be in INIT state.

Parameters

pd	The protection domain of the new Queue Pair.
cq	The completion queue of both send and recv work completions.
srq	(Optional) If set, all recv work requests will be posted to this SRQ.

qp() [4/4]

rdmapp::qp::qp	(	std::shared_ptr< pd >	pd,
		std::shared_ptr< cq >	recv_cq,
		std::shared_ptr< cq >	send_cq,
		std::shared_ptr< srq >	srq = nullptr
	)

Construct a new qp object. The constructed Queue Pair will be in INIT state.

Parameters

pd	The protection domain of the new Queue Pair.
recv_cq	The completion queue of recv work completions.
send_cq	The completion queue of send work completions.
srq	(Optional) If set, all recv work requests will be posted to this SRQ.

Member Function Documentation

compare_and_swap() [1/2]

qp::send_awaitable rdmapp::qp::compare_and_swap	(	remote_mr const &	remote_mr,
		std::shared_ptr< local_mr >	local_mr,
		uint64_t	compare,
		uint64_t	swap
	)

This function performs an atomic compare-and-swap operation on the given remote memory region.

Parameters

remote_mr	Remote memory region handle.
local_mr	Registered local memory region, whose lifetime is controlled by a smart pointer.
compare	The expected old value.
swap	The desired new value.

Returns

send_awaitable A coroutine returning length of the data sent.

compare_and_swap() [2/2]

qp::send_awaitable rdmapp::qp::compare_and_swap	(	remote_mr const &	remote_mr,
		void *	buffer,
		size_t	length,
		uint64_t	compare,
		uint64_t	swap
	)

This method performs an atomic compare-and-swap operation on the given remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

remote_mr	Remote memory region handle.
buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.
compare	The expected old value.
swap	The desired new value.

Returns

send_awaitable A coroutine returning length of the data sent.

fetch_and_add() [1/2]

qp::send_awaitable rdmapp::qp::fetch_and_add	(	remote_mr const &	remote_mr,
		std::shared_ptr< local_mr >	local_mr,
		uint64_t	add
	)

This function performs an atomic fetch-and-add operation on the given remote memory region.

Parameters

remote_mr	Remote memory region handle.
local_mr	Registered local memory region, whose lifetime is controlled by a smart pointer.
add	The delta.

Returns

send_awaitable A coroutine returning length of the data sent.

fetch_and_add() [2/2]

qp::send_awaitable rdmapp::qp::fetch_and_add	(	remote_mr const &	remote_mr,
		void *	buffer,
		size_t	length,
		uint64_t	add
	)

This method performs an atomic fetch-and-add operation on the given remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

remote_mr	Remote memory region handle.
buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.
add	The delta.

Returns

send_awaitable A coroutine returning length of the data sent.

pd_ptr()

std::shared_ptr< pd > rdmapp::qp::pd_ptr

(

)

const

This function provides access to the Protection Domain of the Queue Pair.

Returns

std::shared_ptr<pd> Pointer to the PD.

post_recv()

void rdmapp::qp::post_recv	(	struct ibv_recv_wr const &	recv_wr,
		struct ibv_recv_wr *&	bad_recv_wr
	)		const

This function is used to post a recv work request to the Queue Pair. It will be posted to either RQ or SRQ depending on whether or not SRQ is set.

Parameters

recv_wr	The work request to post.
bad_recv_wr	A pointer to a work request that will be set to the first work request that failed to post.

post_send()

void rdmapp::qp::post_send	(	struct ibv_send_wr const &	send_wr,
		struct ibv_send_wr *&	bad_send_wr
	)

This function is used to post a send work request to the Queue Pair.

Parameters

recv_wr	The work request to post.
bad_recv_wr	A pointer to a work request that will be set to the first work request that failed to post.

read() [1/2]

qp::send_awaitable rdmapp::qp::read	(	remote_mr const &	remote_mr,
		std::shared_ptr< local_mr >	local_mr
	)

This function reads to local memory region from remote.

Parameters

remote_mr	Remote memory region handle.
local_mr	Registered local memory region, whose lifetime is controlled by a smart pointer.

Returns

send_awaitable A coroutine returning length of the data read.

read() [2/2]

qp::send_awaitable rdmapp::qp::read	(	remote_mr const &	remote_mr,
		void *	buffer,
		size_t	length
	)

This method reads to local buffer from a remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

remote_mr	Remote memory region handle.
buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.

Returns

send_awaitable A coroutine returning length of the data read.

recv() [1/2]

qp::recv_awaitable rdmapp::qp::recv

(

std::shared_ptr< local_mr >

local_mr

)

This function posts a recv request on the queue pair. The buffer will be filled with data received.

Parameters

local_mr

Registered local memory region, whose lifetime is controlled by a smart pointer.

Returns

recv_awaitable A coroutine returning std::pair<uint32_t, std::optional<uint32_t>>, with first indicating the length of received data, and second indicating the immediate value if any.

recv() [2/2]

qp::recv_awaitable rdmapp::qp::recv	(	void *	buffer,
		size_t	length
	)

This method posts a recv request on the queue pair. The buffer will be filled with data received. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.

Returns

recv_awaitable A coroutine returning std::pair<uint32_t, std::optional<uint32_t>>, with first indicating the length of received data, and second indicating the immediate value if any.

rtr()

void rdmapp::qp::rtr	(	uint16_t	remote_lid,
		uint32_t	remote_qpn,
		uint32_t	remote_psn,
		union ibv_gid	remote_gid
	)

This function transitions the Queue Pair to the RTR state.

Parameters

remote_lid	The remote LID.
remote_qpn	The remote QPN.
remote_psn	The remote PSN.
remote_gid	The remote GID.

send() [1/2]

qp::send_awaitable rdmapp::qp::send

(

std::shared_ptr< local_mr >

local_mr

)

This function sends a registered local memory region to remote.

Parameters

local_mr

Registered local memory region, whose lifetime is controlled by a smart pointer.

Returns

send_awaitable A coroutine returning length of the data sent.

send() [2/2]

qp::send_awaitable rdmapp::qp::send	(	void *	buffer,
		size_t	length
	)

This method sends local buffer to remote. The address will be registered as a memory region first and then deregistered upon completion.

Parameters

buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.

Returns

send_awaitable A coroutine returning length of the data sent.

serialize()

std::vector< uint8_t > rdmapp::qp::serialize

(

)

const

This function serializes a Queue Pair prepared to be sent to a buffer.

Returns

std::vector<uint8_t> The serialized QP.

user_data()

std::vector< uint8_t > & rdmapp::qp::user_data

(

)

This function provides access to the extra user data of the Queue Pair.

Returns

std::vector<uint8_t>& The extra user data.

write() [1/2]

qp::send_awaitable rdmapp::qp::write	(	remote_mr const &	remote_mr,
		std::shared_ptr< local_mr >	local_mr
	)

This function writes a registered local memory region to remote.

Parameters

remote_mr	Remote memory region handle.
local_mr	Registered local memory region, whose lifetime is controlled by a smart pointer.

Returns

send_awaitable A coroutine returning length of the data written.

write() [2/2]

qp::send_awaitable rdmapp::qp::write	(	remote_mr const &	remote_mr,
		void *	buffer,
		size_t	length
	)

This method writes local buffer to a remote memory region. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

remote_mr	Remote memory region handle.
buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.

Returns

send_awaitable A coroutine returning length of the data written.

write_with_imm() [1/2]

qp::send_awaitable rdmapp::qp::write_with_imm	(	remote_mr const &	remote_mr,
		std::shared_ptr< local_mr >	local_mr,
		uint32_t	imm
	)

This function writes a registered local memory region to remote with an immediate value.

Parameters

remote_mr	Remote memory region handle.
local_mr	Registered local memory region, whose lifetime is controlled by a smart pointer.
imm	The immediate value.

Returns

send_awaitable A coroutine returning length of the data sent.

write_with_imm() [2/2]

qp::send_awaitable rdmapp::qp::write_with_imm	(	remote_mr const &	remote_mr,
		void *	buffer,
		size_t	length,
		uint32_t	imm
	)

This method writes local buffer to a remote memory region with an immediate value. The local buffer will be registered as a memory region first and then deregistered upon completion.

Parameters

remote_mr	Remote memory region handle.
buffer	Pointer to local buffer. It should be valid until completion.
length	The length of the local buffer.
imm	The immediate value.

Returns

send_awaitable A coroutine returning length of the data written.

---

The documentation for this class was generated from the following files:

• include/rdmapp/qp.h
• src/qp.cc