Upgrading to ATS 7.x: Pitfalls and Changes

This presentation covers the pitfalls and changes when upgrading to Apache Traffic Server 7.x. Topics include logging format conversions, new commands usage, plugin adjustments, and the removal of traffic_line. Guidance is provided on adapting cacheurl to cachekey, creating compatible Cachekeys, and recommended testing with ATS xdebug plugin. Enhancements in ATS are highlighted alongside tips for traffic_ctl usage.

• Apache Traffic Server
• ATS 7.x
• Upgrade
• Logging Changes
• Cachekey

mever Follow

Uploaded on Feb 23, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Apache Traffic Server @ Comcast Evan Zelkowitz

2. Upgrading to ATS 7.x

3. Pitfalls Logging changes (convert to lua format) Traffic_line to traffic_ctl (use new command) Cacheurl to cachekey (use new plugin) Changed Parameters

4. Logging Config Changes Uses existing logs_xml.config conventions TO now understands logging.config . This triggers TO to output a LUA formatted config file using the same parameters Can use the same parameter values, assign to logging.config file

5. traffic_line to traffic_ctl Traffic_line officially removed in 7.x Ort will now use traffic_ctl. This exists in 6/5.x as well This makes ort not backwards compatible past 5.x

6. Cacheurl to Cachekey Cacheurl is deprecated in 6.x, removed in 7.x Cachekey in 6.x is usable though some patches are needed to support everything used in cacheurl (included in migration doc) Change from single regex to parameter usages Can create compatible Cachekeys that match Cacheurl regex keys (Thanks to Gancho Tenev)

7. Cachekey examples http://example.net/(.*) http://example-cdn.net/$1 @plugin=cachekey.so @pparam=--separator= @pparam=--static-prefix=http://example- cdn.net/ Simple host replacement @plugin=cachekey.so @pparam=--separator= @pparam=--remove-all-params=true @pparam=-- remove-path=true @pparam=--capture-prefix- uri=/https?:\/\/([^?]*)/http:\/\/example- cdn.net\/$1/ http://([^?]+)(?:?|$) http://example-cdn.net/$1 Special case that includes the original host in the key, i.e. http://example-cdn.net/example.net/...(path).. Also note the s?, both http/https will use the same key http://example.net/([^?]+)(?:\?|$) http://example- cdn.net/$1 @plugin=cachekey.so @pparam=--remove-all- params=true @pparam=--separator= @pparam=-- static-prefix=http://example-cdn.net/ Strips off params but leaves the path

8. Highly recommended to test with the ATS xdebug plugin to ensure you are matching your new cachekeys to your old cacheurl values -H 'X-Debug: X-Cache-Key

9. ATS Enhancements

10. Traffic_ctl parent usage (John Rushford and Vijay Mamidi Next Hop) Parent caches are defined in parent.config by delivery service Parent state (up/down) is per line or delivery service in parent.config The http state machine marks down a parent due to connection errors or timeouts. Parent selection will return a parent for retry once the retry window has elapsed and the parent is marked up if a retry is successful. There is no way to manually control the state of parents. Today parents are removed from parent.config when maintenance must occur.

11. # traffic_ctl host status odol-atsmid-den-07.denver.comcast.net host_status.odol-atsmid-den-07.denver.comcast.net 0 # traffic_ctl host up odol-atsmid-den-07.denver.comcast.net # traffic_ctl host status odol-atsmid-den-07.denver.comcast.net host_status.odol-atsmid-den-07.denver.comcast.net 1 # traffic_ctl host down odol-atsmid-den-07.denver.comcast.net There can be a slight delay between changing states as it propagates through ATS Using traffic_ctl to mark down a parent is global across all remaps. Parent selection will not choose that parent again until you mark it back up States appear in the stats_over_http. Lists all parents up/down states Also available from traffic_ctl, /opt/trafficserver/bin/traffic_ctl metric match host_status Can do multiple hosts on a single command as well On master branch of ATS but pick-able back to 7.x

12. Jemalloc (Thanks to Kit Chan) Current tools are slow (10x to 20x degradation). Not really possible to use in production Some need recompilation Hard to debug on large scale, can take days-weeks for a problem to show up

13. Simple Script to do a preload with the jemalloc library: #!/bin/sh prefix=/usr exec_prefix=/usr libdir=/usr/lib64 MALLOC_CONF="prof:true,prof_prefix:/tmp/jeprof.out,lg_prof_interval:33,lg_prof_sample:20" LD_PRELOAD=${libdir}/libjemalloc.so.2 export LD_PRELOAD export MALLOC_CONF /opt/trafficserver/bin/traffic_server $@ This samples every 2^20 1mb, Does file dumps ~2^32 4GB Change proxy.config.proxy_binary to match the script name We setup a cronjob to clean up any dumps older than 2 weeks Most dumps ~150-200k, we usually see them every few hours during normal usage Can use dot to generate a pdf

14. Jemalloc memory tracking traffic_server at 2018-03-01 05:07:42.202743637 +0000 Total MB: 38402.7 Focusing on: 38402.7 Dropped nodes with <= 192.0 abs(MB) Dropped edges with <= 38.4 MB traffic_server at 2018-03-01 22:21:54.294587434 +0000 Total MB: 42577.5 Focusing on: 42577.5 Dropped nodes with <= 212.9 abs(MB) Dropped edges with <= 42.6 MB Http1ClientSession new_connection proxy/http/Http1ClientSession.cc:214 __clone ??:? 0.0 (0.0%) of 37754.0 (98.3%) 00002af6d9e0c6f7 __clone ??:? 0.0 (0.0%) of 41928.8 (98.5%) 0.0 (0.0%) of 1785.3 (4.6%) 0.0 (0.0%) of 1969.2 (4.6%) 37754.0 41928.8 1969.2 start_thread pthread_create.c:? 0.0 (0.0%) of 37754.0 (98.3%) 00002af6d9e3708f start_thread pthread_create.c:? 0.0 (0.0%) of 41928.8 (98.5%) 1977.6 (4.6%) 37752.2 41927.1 spawn_thread_internal iocore/eventsystem/Thread.cc:84 0.0 (0.0%) of 37752.2 (98.3%) spawn_thread_internal iocore/eventsystem/Thread.cc:84 0.0 (0.0%) 25663.4 8033.3 4055.5 of 41927.1 (98.5%) 30078.7 6309.6 5456.8 EThread execute_regular EThread execute_regular iocore/eventsystem/UnixEThread.cc:248 iocore/eventsystem/UnixEThread.cc:207 0.0 (0.0%) 0.0 (0.0%) of 25663.4 (66.8%) of 8033.3 (20.9%) EThread execute_regular EThread execute_regular iocore/eventsystem/UnixEThread.cc:248 iocore/eventsystem/UnixEThread.cc:207 0.0 (0.0%) 0.0 (0.0%) of 30078.7 (70.6%) of 6309.6 (14.8%) EThread execute_regular 25648.8 8033.3 iocore/eventsystem/UnixEThread.cc:221 0.0 (0.0%) of 4055.5 (10.6%) EThread execute_regular 30058.0 6309.6 iocore/eventsystem/UnixEThread.cc:221 0.0 (0.0%) of 5456.8 (12.8%) NetHandler waitForActivity iocore/net/UnixNet.cc:498 EThread process_queue 4055.5 iocore/eventsystem/UnixEThread.cc:175 0.0 (0.0%) 0.0 (0.0%) NetHandler waitForActivity iocore/net/UnixNet.cc:498 EThread process_queue of 25648.8 (66.8%) of 8033.3 (20.9%) 5456.8 iocore/eventsystem/UnixEThread.cc:175 0.0 (0.0%) 0.0 (0.0%) 6613.4 18965.1 8033.3 of 30058.0 (70.6%) of 6309.6 (14.8%) 3870.8 26099.7 6309.6 SSLNetVConnection net_read_io iocore/net/SSLNetVConnection.cc:596 EThread process_event iocore/eventsystem/UnixEThread.cc:140 0.0 (0.0%) 0.0 (0.0%) SSLNetVConnection net_read_io iocore/net/SSLNetVConnection.cc:596 EThread process_event of 18965.1 (49.4%) of 12088.8 (31.5%) iocore/eventsystem/UnixEThread.cc:140 0.0 (0.0%) 0.0 (0.0%) of 26099.7 (61.3%) of 11766.4 (27.6%) read_from_net iocore/net/UnixNetVConnection.cc:384 18965.1 0.0 (0.0%) of 6613.4 (17.2%) read_from_net iocore/net/UnixNetVConnection.cc:384 26099.7 0.0 (0.0%) of 3870.8 (9.1%) UnixNetVConnection readSignalAndUpdate iocore/net/UnixNetVConnection.cc:1092 6613.4 0.0 (0.0%) UnixNetVConnection readSignalAndUpdate iocore/net/UnixNetVConnection.cc:1092 of 18965.1 (49.4%) 3870.8 0.0 (0.0%) of 26099.7 (61.3%) 18965.1 12088.8 26099.7 11766.4 read_signal_and_update (inline) iocore/net/UnixNetVConnection.cc:144 0.0 (0.0%) read_signal_and_update (inline) iocore/net/UnixNetVConnection.cc:144 of 25582.5 (66.6%) 0.0 (0.0%) of 29978.6 (70.4%) 25582.5 29978.6 Continuation handleEvent (inline) iocore/eventsystem/I_Continuation.h:153 Continuation handleEvent (inline) 0.0 (0.0%) of 37678.8 (98.1%) iocore/eventsystem/I_Continuation.h:153 0.0 (0.0%) of 41834.4 (98.3%) 20765.6 2916.0 5647.8 7207.6 2058.8 4500.5 4476.8 4353.5 4353.5 2134.3 1786.3 25414.2 2536.6 3511.2 2058.8 5589.9 2502.5 2502.5 2497.2 2494.3 1904.8 HttpSM main_handler Http1ClientSession state_keep_alive proxy/http/Http1ClientSession.cc:420 Http1ClientSession release proxy/http/Http1ClientSession.cc:466 CacheVC CacheVC handleReadDone iocore/cache/Cache.cc:2463 AIOCallbackInternal io_complete iocore/aio/P_AIO.h:117 ConfigUpdateContinuation update mgmt/ProxyConfig.h:103 ProtocolProbeTrampoline ioCompletionEvent proxy/ProtocolProbeSessionAccept.cc:107 proxy/http/HttpSM.cc:2693 (discriminator 3) openReadStartHead iocore/cache/CacheRead.cc:1169 HttpSM main_handler Http1ClientSession state_keep_alive proxy/http/Http1ClientSession.cc:420 Http1ClientSession release proxy/http/Http1ClientSession.cc:466 ConfigUpdateContinuation update mgmt/ProxyConfig.h:103 AIOCallbackInternal io_complete iocore/aio/P_AIO.h:117 CacheVC handleReadDone iocore/cache/Cache.cc:2463 ProtocolProbeTrampoline ioCompletionEvent proxy/ProtocolProbeSessionAccept.cc:107 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) proxy/http/HttpSM.cc:2693 (discriminator 3) of 20765.6 (54.1%) of 1785.3 (4.6%) 0.0 (0.0%) of 7384.2 (19.2%) of 4500.5 (11.7%) of 4353.5 (11.3%) of 2134.3 (5.6%) of 1786.3 (4.7%) 2189.4 of 25583.5 (66.6%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 25414.2 (59.7%) of 1903.8 (4.5%) 0.0 (0.0%) of 5589.9 (13.1%) of 2502.5 (5.9%) of 2497.2 (5.9%) of 1904.8 (4.5%) of 29972.5 (70.4%) CacheVC openReadMain CacheDisk openDone 20765.6 1785.3 25568.5 2634.3 1992.3 iocore/cache/CacheRead.cc:740 iocore/cache/CacheDisk.cc:210 CacheVC CacheDisk openDone 0.0 (0.0%) 0.0 (0.0%) openReadStartHead iocore/cache/CacheRead.cc:1169 of 5831.3 (15.2%) of 2058.8 (5.4%) 25414.2 1903.8 29946.4 5427.8 1903.8 iocore/cache/CacheDisk.cc:210 0.0 (0.0%) 0.0 (0.0%) of 3572.0 (8.4%) of 2058.8 (4.8%) Http1ClientSession new_transaction (inline) proxy/http/Http1ClientSession.cc:502 ParentConfig reconfigure (inline) HttpSM CacheVC state_read_client_request_header proxy/http/HttpSM.cc:822 openReadStartEarliest iocore/cache/CacheRead.cc:909 2851.2 2058.8 Http1ClientSession new_transaction (inline) proxy/http/Http1ClientSession.cc:502 ParentConfig reconfigure (inline) proxy/ParentSelection.cc:274 HttpSM HttpSessionAccept accept proxy/http/HttpSessionAccept.cc:72 0.0 (0.0%) 0.0 (0.0%) state_read_client_request_header proxy/http/HttpSM.cc:822 0.0 (0.0%) 0.0 (0.0%) of 25568.5 (66.6%) of 2634.3 (6.9%) 2058.8 of 22550.9 (58.7%) of 1992.3 (5.2%) proxy/ParentSelection.cc:274 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 29946.4 (70.3%) of 1903.8 (4.5%) of 27318.1 (64.2%) of 5427.8 (12.7%) CacheProcessor diskInitialized iocore/cache/Cache.cc:865 22550.9 25568.5 2630.2 1992.3 CacheVC openReadMain CacheProcessor diskInitialized iocore/cache/Cache.cc:865 0.0 (0.0%) 27318.1 29946.4 5427.8 1903.8 of 2058.8 (5.4%) iocore/cache/CacheRead.cc:740 0.0 (0.0%) 0.0 (0.0%) of 2276.1 (5.3%) of 2058.8 (4.8%) HttpSM CacheVC ControlMatcher ControlMatcher proxy/ControlMatcher.cc:728 call_transact_and_set_next_state proxy/http/HttpSM.cc:7269 openReadStartEarliest iocore/cache/CacheRead.cc:914 1928.4 HttpSM ControlMatcher ControlMatcher proxy/ControlMatcher.cc:728 Http1ClientSession new_connection proxy/http/Http1ClientSession.cc:214 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) call_transact_and_set_next_state proxy/http/HttpSM.cc:7269 1928.4 of 25578.5 (66.6%) of 2852.2 (7.4%) of 2175.3 (5.7%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 29963.5 (70.4%) of 5610.8 (13.2%) of 1903.8 (4.5%) Cache open ProxyClientTransaction new_transaction proxy/ProxyClientTransaction.cc:60 iocore/cache/Cache.cc:2187 (discriminator 4) Cache open 75985.3 24840.4 24840.4 24840.4 2851.2 2175.3 ProxyClientTransaction new_transaction proxy/ProxyClientTransaction.cc:60 0.0 (0.0%) iocore/cache/Cache.cc:2187 (discriminator 4) 88300.2 28382.3 28382.3 28382.3 5610.8 1903.8 of 22550.9 (58.7%) 0.0 (0.0%) 0.0 (0.0%) of 1928.4 (5.0%) of 27318.1 (64.2%) 0.0 (0.0%) of 1928.4 (4.5%) HttpSM set_next_state proxy/http/HttpSM.cc:7302 HttpSM set_next_state proxy/http/HttpSM.cc:7451 HttpSM set_next_state proxy/http/HttpSM.cc:7344 CacheVC callcont ControlMatcher BuildTable proxy/ControlMatcher.cc:976 22550.9 3275.2 HttpSM set_next_state proxy/http/HttpSM.cc:7302 HttpSM set_next_state proxy/http/HttpSM.cc:7344 HttpSM set_next_state proxy/http/HttpSM.cc:7451 ControlMatcher BuildTable proxy/ControlMatcher.cc:976 ProxyClientSession do_api_callout proxy/ProxyClientSession.cc:182 iocore/cache/P_CacheInternal.h:658 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 27317.0 of 25575.5 (66.6%) of 24840.4 (64.7%) of 24840.4 (64.7%) of 2851.2 (7.4%) of 2175.3 (5.7%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 29959.5 (70.4%) of 28382.3 (66.7%) of 28382.3 (66.7%) of 5610.8 (13.2%) of 1903.8 (4.5%) HttpSM attach_client_session proxy/http/HttpSM.cc:593 50416.9 24840.4 4748.9 2175.3 HttpSM attach_client_session proxy/http/HttpSM.cc:593 0.0 (0.0%) 58346.8 28382.3 5608.8 of 22550.9 (58.7%) 0.0 (0.0%) of 27317.0 (64.2%) HttpSM HttpSM ControlMatcher BuildTableFromString proxy/ControlMatcher.cc:926 state_api_callout proxy/http/HttpSM.cc:1477 do_cache_lookup_and_read proxy/http/HttpSM.cc:4596 22550.9 5831.3 HttpSM HttpSM ControlMatcher BuildTableFromString proxy/ControlMatcher.cc:926 state_api_callout proxy/http/HttpSM.cc:1477 do_cache_lookup_and_read proxy/http/HttpSM.cc:4596 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 27317.0 2610.9 of 25581.9 (66.6%) of 24840.4 (64.7%) of 2175.3 (5.7%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 29967.9 (70.4%) of 28382.3 (66.7%) of 5608.8 (13.2%) HttpSM state_add_to_list proxy/http/HttpSM.cc:464 HttpSM 75994.8 24840.4 1922.2 2175.3 state_add_to_list proxy/http/HttpSM.cc:464 88310.6 28382.3 2276.1 1922.2 5608.8 0.0 (0.0%) of 22550.9 (58.7%) 0.0 (0.0%) of 27317.0 (64.2%) INKContInternal handle_event proxy/InkAPI.cc:1050 HttpCacheSM open_read HostMatcher NewEntry INKContInternal handle_event proxy/InkAPI.cc:1050 HttpCacheSM open_read HostMatcher NewEntry 22550.9 22550.9 proxy/http/HttpCacheSM.cc:287 proxy/ControlMatcher.cc:219 27317.0 27317.0 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) proxy/http/HttpCacheSM.cc:287 proxy/ControlMatcher.cc:219 of 25581.9 (66.6%) of 24840.4 (64.7%) of 2175.3 (5.7%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 29967.9 (70.4%) of 28382.3 (66.7%) of 5608.8 (13.2%) HttpSM do_api_callout (inline) proxy/http/HttpSM.cc:436 HttpSM do_api_callout (inline) proxy/http/HttpSM.cc:436 24840.4 25568.5 25560.4 24840.4 2135.0 29933.4 28382.3 29946.4 28382.3 5509.1 0.0 (0.0%) of 22555.0 (58.7%) 0.0 (0.0%) of 27325.1 (64.2%) HttpCacheSM do_cache_open_read (inline) proxy/http/HttpCacheSM.cc:254 ParentRecord Init handle_post_remap astats_origin handle_read_req_hdr HttpCacheSM do_cache_open_read (inline) proxy/http/HttpCacheSM.cc:254 ParentRecord Init plugins/experimental/remap_stats/remap_stats.c:152 plugins/astats_over_http/astats_over_http.c:467 plugins/experimental/remap_stats/remap_stats.c:131 handle_read_req_hdr handle_post_remap astats_origin 25568.5 25568.5 75978.3 proxy/ParentSelection.cc:698 plugins/experimental/remap_stats/remap_stats.c:131 plugins/experimental/remap_stats/remap_stats.c:152 plugins/astats_over_http/astats_over_http.c:467 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 29953.5 29946.4 88287.2 proxy/ParentSelection.cc:698 1.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 24840.4 (64.7%) of 25568.5 (66.6%) of 25560.4 (66.6%) 0.0 (0.0%) 1.0 (0.0%) of 2135.0 (5.6%) of 29933.4 (70.3%) of 28382.3 (66.7%) of 29946.4 (70.3%) 0.0 (0.0%) of 24840.4 (64.7%) of 5509.1 (12.9%) of 28382.3 (66.7%) Vol init Vol init 24840.4 25568.5 25560.4 21523.8 3275.2 2134.0 iocore/cache/Cache.cc:1385 29933.4 28382.3 29946.4 27056.4 5508.1 iocore/cache/Cache.cc:1385 0.0 (0.0%) 0.0 (0.0%) of 1922.2 (5.0%) of 1922.2 (4.5%) Cache open_read Cache open_read ParentConsistentHash ParentConsistentHash proxy/ParentConsistentHash.cc:38 TSHttpTxnReenable proxy/InkAPI.cc:5900 Cache open_read ParentConsistentHash ParentConsistentHash proxy/ParentConsistentHash.cc:38 TSHttpTxnReenable proxy/InkAPI.cc:5900 22554.0 iocore/cache/CacheRead.cc:137 iocore/cache/CacheRead.cc:158 27322.1 0.0 (0.0%) iocore/cache/CacheRead.cc:137 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 25576.5 (66.6%) 0.0 (0.0%) 0.0 (0.0%) of 21523.8 (56.0%) of 3275.2 (8.5%) of 2134.0 (5.6%) of 29962.5 (70.4%) of 27056.4 (63.5%) of 5508.1 (12.9%) CacheVC CacheVC openReadStartEarliest iocore/cache/CacheRead.cc:841 75981.3 21523.8 2134.0 openReadStartEarliest iocore/cache/CacheRead.cc:841 88292.2 27056.4 5508.1 0.0 (0.0%) 0.0 (0.0%) of 4900.1 (12.8%) of 2633.1 (6.2%) CacheVC do_read_call (inline) CacheVC do_read_call (inline) HttpSM ATSConsistentHash insert lib/ts/ConsistentHash.cc:65 HttpSM ATSConsistentHash insert lib/ts/ConsistentHash.cc:65 state_api_callback proxy/http/HttpSM.cc:1355 state_api_callback proxy/http/HttpSM.cc:1355 4803.6 2615.7 iocore/cache/P_CacheInternal.h:669 iocore/cache/P_CacheInternal.h:669 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 25576.5 (66.6%) of 2134.0 (5.6%) of 29962.5 (70.4%) of 5508.1 (12.9%) of 27361.5 (71.2%) of 29335.3 (68.9%) 50412.9 27172.8 2134.0 58345.8 29287.5 5508.1 HttpSM CacheVC handleRead insert (inline) HttpSM CacheVC handleRead insert (inline) state_api_callout proxy/http/HttpSM.cc:1556 state_api_callout proxy/http/HttpSM.cc:1556 iocore/cache/Cache.cc:2504 /usr/include/c++/4.8.2/bits/stl_map.h:603 iocore/cache/Cache.cc:2504 /usr/include/c++/4.8.2/bits/stl_map.h:603 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 25578.5 (66.6%) of 32289.4 (84.1%) of 2134.0 (5.6%) of 29965.5 (70.4%) of 32297.6 (75.9%) of 5508.1 (12.9%) 22550.9 75984.3 32289.4 2134.0 27317.0 88300.2 32297.6 5508.1 IOBufferData_tracker operator (inline) iocore/eventsystem/I_IOBuffer.h:1364 std IOBufferData_tracker operator (inline) iocore/eventsystem/I_IOBuffer.h:1364 std HttpSM HttpSM HttpSM HttpSM _Rb_tree _Rb_tree handle_api_return proxy/http/HttpSM.cc:1599 handle_api_return proxy/http/HttpSM.cc:1621 handle_api_return proxy/http/HttpSM.cc:1599 handle_api_return proxy/http/HttpSM.cc:1621 _M_insert_unique _M_insert_unique /usr/include/c++/4.8.2/bits/stl_tree.h:1382 /usr/include/c++/4.8.2/bits/stl_tree.h:1382 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 27317.0 (64.2%) of 29955.5 (70.4%) of 22550.9 (58.7%) of 25572.5 (66.6%) of 32390.9 (76.1%) of 5508.1 (12.9%) of 32605.7 (84.9%) of 2134.0 (5.6%) 32390.9 5508.1 32605.7 2134.0 new_IOBufferData_internal (inline) iocore/eventsystem/P_IOBuffer.h:264 _M_insert_ (inline) new_IOBufferData_internal (inline) iocore/eventsystem/P_IOBuffer.h:264 _M_insert_ (inline) 1922.2 1922.2 /usr/include/c++/4.8.2/bits/stl_tree.h:1023 /usr/include/c++/4.8.2/bits/stl_tree.h:1023 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 32460.5 (76.2%) of 5508.1 (12.9%) of 32639.0 (85.0%) of 2134.0 (5.6%) 32365.9 5508.1 32581.7 2134.0 IOBufferData alloc _M_create_node (inline) IOBufferData alloc _M_create_node (inline) iocore/eventsystem/P_IOBuffer.h:293 /usr/include/c++/4.8.2/bits/stl_tree.h:403 iocore/eventsystem/P_IOBuffer.h:293 /usr/include/c++/4.8.2/bits/stl_tree.h:403 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) 0.0 (0.0%) of 32365.9 (76.0%) of 5508.1 (12.9%) of 32581.7 (84.8%) of 2134.0 (5.6%) 32365.9 5508.1 32581.7 2134.0 std Allocator alloc_void (inline) std _Rb_tree _M_get_node (inline) Allocator alloc_void (inline) _Rb_tree _M_get_node (inline) iocore/hostdb/../../lib/ts/Allocator.h:63 /usr/include/c++/4.8.2/bits/stl_tree.h:370 iocore/hostdb/../../lib/ts/Allocator.h:63 0.0 (0.0%) /usr/include/c++/4.8.2/bits/stl_tree.h:370 0.0 (0.0%) 0.0 (0.0%) of 32365.9 (76.0%) 0.0 (0.0%) of 5508.1 (12.9%) of 32581.7 (84.8%) of 2134.0 (5.6%) 32365.9 5508.1 32581.7 2134.0 __gnu_cxx new_allocator allocate (inline) __gnu_cxx new_allocator allocate (inline) ink_freelist_new lib/ts/ink_queue.cc:183 ink_freelist_new lib/ts/ink_queue.cc:183 0.0 (0.0%) of 32694.0 (85.1%) 0.0 (0.0%) of 32535.6 (76.4%) /usr/include/c++/4.8.2/ext/new_allocator.h:104 /usr/include/c++/4.8.2/ext/new_allocator.h:104 2134.0 (5.6%) 5508.1 (12.9%) 32694.0 32535.6 malloc_new lib/ts/ink_queue.cc:260 0.0 (0.0%) of 32694.0 (85.1%) malloc_new lib/ts/ink_queue.cc:260 0.0 (0.0%) of 32535.6 (76.4%) 32660.0 32482.5 ats_memalign lib/ts/ink_memory.cc:102 34828.5 (90.7%) ats_memalign lib/ts/ink_memory.cc:102 34651.1 (81.4%)

15. Very helpful in finding plugin memory issues Can help in other areas as shown Freelist can be problematic. Jemalloc does not understand the freelist so you can see it marks all the memalign as lost memory when its just being held in the freelist Similar syntax to gperftools

16. Jemalloc ATS presentation: https://cwiki.apache.org/confluence/download/attachments/70255385/ATSSummit_jemall oc.pptx?version=1&modificationDate=1508884895000&api=v2 Traffic Control Migration Doc: http://traffic-control- cdn.readthedocs.io/en/latest/admin/traffic_ops/migration_from_20_to_22.html ATS Cachekey Documentation: https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/cachekey.en.html ATS v7 Upgrade Notes: https://cwiki.apache.org/confluence/display/TS/Upgrading+to+v7.0

17. Questions?