Overview of SHM Support Options and Primitives
This content provides an overview of SHM (Shared Memory) support options and primitives, detailing regions, commands, responses, addresses, mappings, utilities, providers, and protocols used in SHM communication. It covers the adaptation of primitives for local communication, the use of native providers, synchronization processes, and memory access control. Additionally, it delves into specific SHM primitives like smr_region, smr_cmd, message structures, buffer sizes, and peer memory region management.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
OFI Shared Memory OFIWG
Overview SHM Support Options SHM Primitives Region / cmd / resp / addr / map SHM Utilities Initialization / mapping SHM Provider Requirements / status Message protocols: inline / inject / iov Address exchange protocol 2 www.openfabrics.org
Overview SHM Support Options SHM Primitives Region / cmd / resp / addr / map SHM Utilities Initialization / mapping SHM Provider Requirements / status Message protocols: inline / inject / iov Address exchange protocol 3 www.openfabrics.org
SHM Support Options SHM support SHM primitives provided in utility code without protocol Provider adapts use of primitives for shm local communication using own protocol SHM provider Native provider using SHM primitives Assumed all local communication Provider SHM Utilities SHM Primitives 4 www.openfabrics.org
Overview SHM Support Options SHM Primitives Region / cmd / resp / addr / map SHM Utilities Initialization / mapping SHM Provider Requirements / status Message protocols: inline / inject / iov Address exchange protocol 5 www.openfabrics.org
SHM Primitives: smr_region smr_region version flags pid lock map total_size Synchronize shared memory access Stores peer addresses and pointers to peer smr_regions Control cmd_queue_offset resp_queue_offset inject_pool_offset peer_addr_offet cmd_queue Shared access resp_queue inject_pool peer_addr 6 www.openfabrics.org
SHM Primitives: smr_cmd smr_cmd hdr { } msg / tagged / rma / atomic op { } msg_id addr op op_src size data smr_src_inline / smr_src_inject / smr_src_iov Used for smr_src_inject (inject offset) and smr_src_iov (response offset) union data { msg iov rma_iov rma_ioc } 7 www.openfabrics.org
SHM Primitives: Other smr_resp msg_id status Used by rx side to signal completion for large CMA (smr_src_iov) operations 4096 byte buffer for medium (smr_src_inject) message transfers smr_inject_buf data[SMR_INJECT_SIZE] smr_addr name addr Used to exchange remote endpoint information smr_map smr_peers { smr_addr smr_region } peers [SMR_MAX_PEERS] List of all peers and pointers to peer memory regions 8 www.openfabrics.org
Overview SHM Support Options SHM Primitives Region / cmd / resp / addr / map SHM Utilities Initialization / mapping SHM Provider Requirements / status Message protocols: inline / inject / iov Address exchange protocol 9 www.openfabrics.org
SHM Utilities: init int smr_create(const struct fi_provider *prov, struct smr_map *map, const struct smr_attr *attr, struct smr_region **smr) Create smr_region and initializes all components int smr_map_create(const struct fi_provider *prov, int peer_count, struct smr_map **map) Create smr_map to store peer addresses and pointers to peer regions 10 www.openfabrics.org
SHM Utilities: mapping int smr_map_add(const struct fi_provider *prov, struct smr_map *map, const char *name, int id) Add a peer by name into map with specific id Note: adding to the map does not ensure peer region is mapped and accessible int smr_map_to_region(const struct fi_provider *prov, struct smr_peer *peer_buf) Try to map to the peer region Returns FI_EAGAIN if peer s region is not initialized yet void smr_map_to_endpoint(struct smr_region *region, int index) Exchange addressing information with a specific peer Find this region in the peer to see if it has been mapped yet and update addresses 11 www.openfabrics.org
Overview SHM Support Options SHM Primitives Region / cmd / resp / addr / map SHM Utilities Initialization / mapping SHM Provider Requirements / status Message protocols: inline / inject / iov Address exchange protocol 12 www.openfabrics.org
SHM Provider Endpoint address defaults to info->src_addr if provided or default endpoint name format if no src_addr given, requiring out-of-band endpoint name exchange EP address = pid:domain_idx:endpoint_idx Only messaging and tagged support Currently only DGRAM Inherently supports RDM RMA and atomic implementation in progress Aim to get full support working for MPI 13 www.openfabrics.org
Small Message Example (smr_src_inline) Tx side writes smr_cmd into peer s cmd_queue Only very small messages that can fit inline into smr_cmd (128 bytes) Rx side decodes header and processes msg Data is retrieved directly from cmd Tx CMD Rx Command Queue 14 www.openfabrics.org
Medium Message Example (smr_src_inject) Tx side writes data into Rx side inject buffer Tx side writes msg header to Rx cmd Header includes inject buffer offset Rx side decodes header and processes msg Data is retrieved from Rx inject buffer Tx CMD Rx Command Queue Inject Buffer 15 www.openfabrics.org
Large Message Example (smr_src_iov) Tx side writes msg header to Rx cmd Header includes smr_resp offset (for ACK) Rx side decodes header and processes msg from Tx process using CMA Rx side writes ACK msg back to Tx side Tx Rx Command Queue CMD Tx RESP CMA Buffer 16 www.openfabrics.org
Completion Handling For small to medium sized messages Tx completes immediately after send For large messages delivery complete semantics Tx does not complete until it has processed an ACK from the Rx side 17 www.openfabrics.org
Portability SHM is disabled on non-linux platforms (no support for CMA) SHM can be extended later to avoid using CMA Add bounce buffering Make SMR_INJECT_SIZE environment variable Max message size = SMR_INJECT_SIZE 18 www.openfabrics.org
Address / Name Exchange Endpoint name: 11111:0:0 Endpoint name: 22222:0:0 . . . . . . Endpoint name: 44444:0:0 Endpoint name: 33333:0:0 . . . . . . 19 www.openfabrics.org
Address / Name Exchange Endpoint name: 11111:0:0 Endpoint name: 22222:0:0 fi_av_insert . . . . . . 0 22222 UNSPEC 1 33333 UNSPEC 2 44444 UNSPEC Endpoint name: 44444:0:0 Endpoint name: 33333:0:0 . . . . . . 20 www.openfabrics.org
Address / Name Exchange Endpoint name: 11111:0:0 Endpoint name: 22222:0:0 fi_av_insert fi_av_insert . . . . . . 0 22222 UNSPEC 1 33333 UNSPEC 2 44444 0 Endpoint name: 44444:0:0 Endpoint name: 33333:0:0 . . . . . . 0 11111 2 1 22222 UNSPEC 2 33333 UNSPEC 21 www.openfabrics.org
Address / Name Exchange Endpoint name: 11111:0:0 Endpoint name: 22222:0:0 fi_av_insert fi_av_insert . . . . . . 0 22222 UNSPEC 1 33333 0 2 44444 0 fi_av_insert Endpoint name: 44444:0:0 Endpoint name: 33333:0:0 . . . . . . 0 11111 1 1 22222 UNSPEC 2 44444 2 0 11111 2 1 22222 UNSPEC 2 33333 2 22 www.openfabrics.org
Address / Name Exchange Endpoint name: 11111:0:0 Endpoint name: 22222:0:0 fi_av_insert fi_av_insert . . . . . . 0 22222 0 1 33333 0 2 44444 0 0 11111 0 1 33333 1 2 44444 1 fi_av_insert fi_av_insert Endpoint name: 44444:0:0 Endpoint name: 33333:0:0 . . . . . . 0 11111 1 1 22222 1 2 44444 2 0 11111 2 1 22222 2 2 33333 2 23 www.openfabrics.org
Summary Shared memory support available through primitives, utilities, and shm provider 3 types of messages: inline, inject, iov Currently EP_DGRAM, but RMA and atomics in progress to support EP_RDM Focusing on provider and integrating into MPI 24 www.openfabrics.org
