Optimizing Performance with Parallelism and SIMD Instructions

Behind the scenes perspective: Into the abyss of profiling for performance Part II Servesh Muralidharan & David Smith IT-DI WLCG-UP CERN 29 Aug 18 1

Sources of Parallelism in Modern Architectures 1. Instruction level parallelism (ILP) 2. Pipelining 3. Vector Operations 4. Hardware Threads 5. Multicore 6. Multi Socket 7. Cluster 8. Grid Architecture handles this with good coding practice Can be implicitly generated by compiler understandable code or explicitly with intrinsic Explicitly programmed using shared-memory parallel programming model Explicitly programmed using distributed-memory parallel programming model 2

Flynns taxonomy: Can be a programmers guideline SISD MISD Proposed in 1966 Instruction streams single (SI) or multiple (MI) Data streams single (SD) or multiple (MD) Types SISD SIMD MISD MIMD Instruction Pool Instruction Pool Data Pool Data Pool Processing Unit Processing Unit Processing Unit MIMD SIMD Instruction Pool Instruction Pool Processing Unit Processing Unit Processing Unit Processing Unit Processing Unit Data Pool Processing Unit Data Pool Processing Unit Processing Unit Processing Unit Processing Unit Processing Unit Processing Unit 3

Using SIMD Choose between code manageability and portability and speed: Use different levels of abstraction Assembly Intrinsic Wrapper functions or classes in C or C++ (using intrinsics) Custom languages like Cilk/Cilk++ Autovectorization 4

SIMD: SSE, AVX & FMA The use of SIMD instructions in vectorized code can give good performance gains (S)SSE 1,2,3 SSE 4.1, 4.2 AVX-512 AVX AVX2 + FMA From 1999 Width 128b From 2007 Width 128b From 2011 Width 256b From 2013 Width 256b From 2016 Width 512b 256b -> 8 floats or 4 doubles (possibility of 4 or 8 speedup) 5

Fused Multiply Add These are instructions which can do calculations of the form: A <- A*C + B or A <- B*C + A Can bring gain because one instructions replaces a multiply and add instruction (reduces throughput cycles and latency) Is also a SIMD instruction 6

FMA and floating point FMA is specified to round only once Therefore FMA is a change which can affect the result of a calculation compared to using separate multiply and add operations Enable with: GCC: Will use FMA if it is compiling for an architecture that has it May be explicitly set with mfma or mno-fma 7

Autovectorization Depends on compiler and version Advantages Can get great speedups (x2 or more) with little change of the source code Compiler can generate a report to help Source code remains architecture independent Disadvantages Can be delicate. A small change of the source can stop the compiler autovectorizing, causing a large change in performance Lots of compiler options that affect the autovectorization Usually you could have bigger gains using intrinsics or assembly Some compiler options which help autovectorization can change the way FP operations are done: you have to be aware when that may be important 8

Autovectorization Two main places where automatic vectorization can be done In loops compiler tries to do several iterations of the loop at once using SIMD May unroll a number of the loops so to fill pipeline and improve ILP May peel loops to allow aligned access to data Combining similar independent instructions into vector instructions Known as SLP vectorizer 9

Autovectorization difficulties for(i=0;i<*p;++i) { A[i] = B[i] * C[i]; sum += A[i]; } (example from slide by Georg Zitzlsberger, Intel) Possible to vectorize? Concerns may be: o Is the loop range invariant during the loop o Is A[] aliased with the other arrays or with sum, is sum aliased with B[] or C[] o Is the + operator associative? o Is the vectorized version expected to be faster? 10

Using autovectorization GCC: Switch on using ftree-vectorize Off by default Information on autovecotization analysis using ftree-vectorizer-verbose=X (X=0-7, 7 is most information) Or examine generated code with gdb May be necessary to use ffast-math Often with reductions (e.g. summations) This will cause the compiler to relax certain some constrains, for example allow it to assume associative properties ICC: On by default, modify with x and -ax Would switch off by using no-vec ICC defaults to ignoring parentheses to specify floating point associativity and generally can make more aggressive optimizations on floating point calculations Controlled by -fprotect-parens and fp-model Information on autovectorization analysis -qopt-report=2 qopt-report-phase=vec -opt-report-help and opt-report-phase={hpo,ipo} Or examine with gdb 11

FP caution E.g. summing a number of values: One technique to reduce rounding error over long sums is Kahan summation double sum = 0.0, C = 0.0, Y, T; size_t i; /* Kahan summation of values in A[i] */ for (i=0; i<length; i++) { Y = a[i] - C; T = sum + Y; C = (T-sum) - Y; sum = T; } If the compiler reasons C=0 the correction is lost. Perhaps pairwise summation could be used instead. 12

SoA vs AoS It is better to load the contents of a vector register from a contiguous piece of memory rather than gather the values Can arrange that the data layout fits this access pattern. e.g. use of Structure of Arrays instead of Array of Structures. e.g. struct points { double x[1000]; double y[1000]; double z[1000]; } locations; struct point { double x,y,z; } location[1000]; 13

data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABQODxIPDRQSEBIXFRQYHjIhHhwcHj0sLiQySUBMS0dARkVQWnNiUFVtVkVGZIhlbXd7gYKBTmCNl4x9lnN+gXz/2wBDARUXFx4aHjshITt8U0ZTfHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHz/wAARCAElAMEDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwBpMsk85a4uBiaQALO4AAYgAAGnqjH/AJeLr/wIf/GkQfvJ/wDrvL/6GamUV0pKy0Odt3eo1YSes91/4EP/AI08QD/ntdf+BMn+NSKKkUUrLsO7IhbL/wA9rr/wJk/xp4tU/wCet1/4Eyf41KBUgFKyKuyAWkf/AD1uv/AmT/GnCzi/56XX/gTJ/jU4FPApWQ9St9ii/wCel1/4Eyf40v2GH/npdf8AgTJ/8VVkCnAUrIepV+wQ/wDPS6/8CZP/AIqj7DD/AM9Lr/wJk/8AiqtYpcUtA1Kn2CH/AJ6XP/gTJ/8AFUfYIf8Anpdf+BMn/wAVVvFGKNA1Kn2GH/npdf8AgTJ/8VSfYYv+el1/4Eyf41bxSEU7INSmbKL/AJ6XX/gTJ/jSGzj/AOet1/4Eyf41cIppFFkLUpm0j/563X/gTJ/jTTar/wA9br/wJk/xq2RVe7OyBiJPK77/AEobUVdgk27IhNuB/wAtrr/wJk/xphg9J7r/AMCH/wAasRqShLNtiVd26Q5YgdelQ/aI2bKBmjzjcflz7j2qFWpvZfgU6cyIxEf8t7r/AMCH/wAar3fmR20rpc3QZUJB+0PwcfWrZljPVW/Cq18QbK4wCP3bdfpWqafQzaa6nTYHoPyooormNznoh80//XeX/wBDNTqKjhHM3/XeX/0M1KzpEm+Q4H6mutPRHO1qSKKkUVUGoRD/AJZv+lWraeO5B8skMOqt1FTzJjs0SgU8CqbalDHI6GKQlCVJGO1C6vb55ilA9cDj9anmRVmXgKcBSeZEIfPMi+SBu3+1UjrVsvSGYj14Gf1pNjsaGKXFVp9Rht4oJGjdxMu5QuPl+v51D/blsDjyJs/h/jSuOxoYpcVFaXkF6rGEkMv3kYYIqS4lS1t3nmJ2L2HVj2AouAuKMVVtNUt7uYQqjxORld+MMfT61JfXkenpG0yM4kJAC44ouFiXFIRVH+3Lf/nhN+n+NPi1aCaOZ1ikAhUMQcZPOOKLhYskUhFQ2l/FeymONHjIG7L4wah/taAzeV5UgbfsycYznGadxWLJFQzxCWPYQDyDg9D60t5eR2bqsiO+8EgrSxTRz2/nfcTBJ3dgKJJSTiCuncpyBnhmwPkdsPn+4O34k80RwgruYfQUk99Z+VuVWLk8fLg4p0N3DMRGNyP23d6zopRvYqo3K1xSoHQVUvx/oNx/1zb+VXmFVNQH+gXP/XJv5V0mHU36KKK5DpMO3H+u/wCu8v8A6G1R6gMeT+P9KmthxL/13l/9Dao9R48n/gX9K6H8JiviGQRq9rIxAyqkg4punki7UjriprUf6DN/uNUOn/8AH0v0qOxfcZcf8fkv/XU/zqzqUaRrAyKF3bs4GPSq1z/x+TevmHH51LOby52LLF9zO3Ax19aQx2T/AGNtz8vm9Kk0iFZ2kVkVuR1GcUTwmDSQp5O8ZPvzVa2tHuwwSTYV/WgC7rcIgFuq9Pm49OlJZwLJpM7lFLIrENjnim6qHSCzjk6xqVz64xVeKzea1e4WTCoCSn0pAO0gkXykHBKH+lP1i6a5vPKXiOA7QPVu5p+iOpujGygbhuD9+O1Vb3jUbj2lP86AH31obTymRz83IPcMOpFbVlLHqVojTRozjghlBwfUVkajfRXqQrCrqYs7t3fOK0NAH7j/AIEaAMe6UJfToowqykAe2a17u2jXTZ5UVUbH8Ix3rJvf+Qjc/wDXZv51uXw/4k9x9P6ihAZWj83f/Af61VnGZ5h/tt/Orejf8fn/AAH+tVycX7H/AKbH+dHQAnuDcRwh+XjUqT6+hqwv/IHf8P8A0Ko9RthbTgr/AKuUblH931FSr/yBX+o/9CpoCGwjSSVg6hhx1GadJYzRzZ+UruyMHoM0ljvBlMYzIFyo9TThqNyZFWTYPmAYbeR60K1tRO99C6MlAT1qpqA/4l91/wBcm/lV44YZHSqmoj/iXXX/AFyb+VbGZs0UUVym5kWo+WX/AK7y/wDoxqbqMTvFHIgLBM7gPQ96ktB+7l/67y/+jGq0g6noB1Jro3RjszHiuvKt3iCbt4I3Z6ZqTTY2M4fBCjgZ7mtHNoTkzRf99CrEUSYDx4cHoQciosXc5+5P+nTD/pqf510cKKYwSM00vazMq+fEz9AAwyak3wW+EkmSM9QHYA4o2Ag1O3efTmSBcurBwo/ix1FYtlffYSzeUXJ7E4xiunQhwGRgykZDA8Y9ahaWxZiWuYc9/nFIZla5Os8NjKBjepYr1xnHFP08/wDEluj/ANM3rYiSIxExMrxt1ZTkVHHNYxtuS6h/77FIDn9FOb9f90/0qK9b/iZXA5/1x/nXStJYM2TdQ5J/vipp1hChrh0jUcAsQKAMDWoo4UtmjRV37s7Rj0q54e5gH+8a0YpLaRTDDNHIcZ2qwJxSH7NbOPMmjiZhwGYDIoA5a+b/AImVzwf9c38637/jRbg+39RVwrFcoHRlkToGU5FN822tj5b3ESMOqs4BFAHP6Ic3p/3f6iqrH/T26/64/wDoVdT5ETYkXBB5DDoaWWJZDlh2pgYWtnD2v+4f50+2ha50h44uXIyo9cHOK2PLXy/Lx8tMWJYySvemI5y1uTaSljES3dTwRUe/fcb8Y3vux6ZNdJJEjnLDmomt4/SiwXI1H7sVV1If8S66/wCuLfyq7tCjA6CqmpD/AIlt3/1xf+VWQatFFFc5sZdmP3Un/XeX/wBGNUWrMRHDGDhWJLD1xjH86nsv9VJ/13l/9GNTdTt5JoY3iUsYycqOpB//AFVu/hM1uU7eyFxbNIgYyAHCjvU+kCe1uyrxskbj5gemaqxXFzbjELPGP93p+daWlXxuJjDdBS+MrIBjPsahWKMq1G29iK8EScVb10bryMtyfJH8zVRc214VlUhon+Ze9S6ldpdzJIisoVAmG6k5P+NICe7kcaLawhiEf7wHcDtTLHTUu7aSQbvMXIVV74GRVmaymn0qERqWlg5ZB1II5/GqMFxeWoIg8yIZyfk6fmKALGlR3VnekSQvHFIpWQHgex+tN1PS4dOSIws7K5x85FXNK1OS4uPs96FfcMpIBg59DUniUfuLb/fP8qBlDTtMivo97Fg6vgAYxxg1FrDvJqUvnHcY8Ko7AYrT8O/6tv8AfP8AIVmarg6rdZ6bh/IUAathpK2863FuzZxg7j2NZWoy/btTkZfmG7y4/wAOP55Naul3l/8AahHdxhINpwdmOe1YtnxexY/56UAWbC9k0maeJlLLyCno/Y/Q1UMUkqSXUvzhnw7nux/pV/xAALyIgYJhGffk1Myj/hG2bpgL/wChUATaBcb7Y27HJiPy/Q8j+tahFc/oP/HzIfYV0JFAhhFNIp5ppqhEZFRsKlIpjUxELCqepD/iWXf/AFxf+VXmFU9TH/EsvP8Ari/8jVCNGikornNTPsR+5k/67zf+jGpNQuprNYmhC4fIYsM89v606x/1En/Xeb/0Y1WmjSaMxyKGQ9Qa26GfUz1voZrCYXDKtyysAApwfSqulg/bB7Lz7Vof2PCTwpx/vGrNvYJBwoCjOeO9TYZHraKdPMpUb1KgNjnFUdFRXll3AE/Lgkcjr0rbmto7uEwzKWjJzgHHIqG10xbSQtEMA9ckmgZHqtzNplvC9qFw77XLDPbiq8GqxTWM4vXRbhgyoFU4Ixx+ua2XiSeNopUDow5Uis5tAtgeEOO3zmkMxtJB+3R8fdUk+1aniX/j3tf94/yq7baZHbcIoUHk9yfxqa8sYL5FWdC2w5XDEYP4UAZnhz/VN/vn+QrM1gEatcqRgkgj3GBXR2OnrZEiLhCckZzzT73T7e+wZowXXgMDgj8RSAhsLyzuGEUcoeUDONp6VzVn/wAf0X/XT+tdFbaRHaTCaAbXxjJYnik/sO2jmWSBCrKcjLk80wMzxD/x+Q/9cR/M1M3/ACLLfRf/AEKtK70qC82vMpMqrtBDEcUqadH9ka0lXdAcYUE9jnrQBi6D/wAfMn0FdGapW2lJaSl4Rtz1+Ymr5U+lAiI00ipCM9KaVPpTAiIphFSmmGmIhYVT1P8A5Bd5/wBcX/kavMKpan/yC7z/AK4v/KqEXqKKKxNCjYf6iT/rvN/6Mak1K7ktIkWHAkkJ+YjO0D/9dOsP+PeT/rvN/wCjGqtrf/Lt/wAC/pWr2M+pAkmoPA0y3L+WuSfn54q7o+oyzzG3uW8wEZRz1HsfWjR1WS2ZHG5GyGHqKuQabBFOHhQIfbPSpKMibUry6uikEphQttjVePzNJLdanZSqHuWL43AE7lIqvbf8fsX/AF0/rV3X/wDj6g/64j+ZpAXdRvnl0WK5t2aF3Zc7TyPUVFoWoSNcvBdSvI0gzGXOenUf59Khm/5F2L/fX+tZ6Rv5LXEZP7lhkjqvofzoGWbS+vPt8Qe5lZPMwVLcEZ6Vf1zU7iGdLe2fyhsDOwHJz2FZNpzfQH1cVZ17/kI/9sl/rSASR9UhtluXu38psYIfJ56Vp6Xq0ktjcvcgPJbqWyON4xn8+KnsbeK70yKK4QOm1Tg+uKbPp8NnYXbQqE3RNkDvwaAMaO81O/mIS5KOQWCqdqgVcuJL620XE07i4Eo+cNng9s1nafcpaXAmlDFdpGF61qapex3+iCaFWVfNAww54oApWr6pebvIu3JXGdz461Jpmq3K3SwzymWOQ7ct1VuxzUnh6WOKSYyyJGDtxuYDPWs2z/4/of8ArpQBauf7WsVX7Tdv8xwCr5pIG1S6XfDduVDYOXxV/wARf8elr/vn+VHh3/Vv/vn+QpgV9W1W4Ny0Nu5gjTrtHzMar3Eup2YR5btwGOFw+eaZrCldVuFf5dxDD3GKt/2zA8YjmsfMUdywJH0yKAL2m3pvrYNIAJVO1iOh96tNUFmls8Sz2hHlt2Axg+49asGqRJE1UtT/AOQXef8AXF/5VeaqWqf8gu9/64P/ACpiLlFFFZGhT0//AI93/wCu83/oxqra5/y7fRv6VZ0//j3f/rvN/wCjGqS8s0vYQjMUdDlGHb2rR7GfUraSxisJJ9u4RhmI9cVa07V0vLkReQYvlLbi2fwrM/sq6HAlAHoM4NXNO09reTeTuc98YAHtSKMm1/4/Yv8Arp/Wr3iAYu4Af+eI/mamvNEYztLayBVc7ipH3T7GoBo1w5HmzEj15J/WkA6b/kXIv99f60/w8od50YAqwAIPcYNXbjTXn00W0RWLDAjdyMCmaRYy2Mz+YQwcjG3tQMw7MYvoQOgkwPzq1r3/ACEf+2S/1qdNEube6SQyo4V92ADV/UtJ+3iOZH8udV28jIYehpANju/7O0aK4MZkBVBtBx1FNXVBqWm3/wC6MXlxHq2c5B/wqj/Yl5jaZwV/u8kflWpp2mrbxPGw3q4IkLcbsjGPpQBiaNGst6EdA6lDwRkVqa5bx22l7YlCgyLwBgd6rS+H54nP2e4+T+EnIYD04qddMuZdL+xPKDIJN/mNkjHpQBm6dYR6h5gkZlKYxtxznP8AhWvaaKltJvGWfpuY5wPam6TYTWMr+YQ28jBHbGf8a2TQBh+JRttrYDs5/lSeHeYn/wB8/wAhVzWbCTUIohE6oY23HcOvFRaRaS2QZJCGJYtkUwIdT1LTTK9vd28kjRHG4KOPoc1W1mxhsoYWhUAuxBIz6Vc1TRvtUxuYHCSEYdWGQ3vWedHujw8+R75OKALOgnELgdDIf5CtQ1WsLT7KgUZwO56k+tWjTQiM1S1T/kFXn/XB/wCRq6apap/yCr3/AK4P/I1RJbooorIsyLbVbW2WWGXzN6zy52rkffY1ONdsB/z2/wC+P/r1z1x/x93X/XxJ/wChGrGl26XFzJ5qh1iiaTYzbQxGBgnsOa61CPLdnJ7WXPyo3B4g0/8A6bf98f8A16cPEWnj/nt/3x/9eqM+lW/keUAwMTSESjHIDKAG454PFRf2Tbv9r8syqIGdVZnB3FBk8Af4VNoF81Q1B4j04dpv++P/AK9L/wAJLp/pN/3xWfdaZBJeO8aMiBn3orBVAUDkccDn0NNk0e2i3hmmYqJGyGA4RlGOncN19qVqYc1Q0v8AhJtP9Jv++P8A69L/AMJPp/pN/wB8f/XrEtrSBdTu7aRDKsWRGWViAcgAtt5/GrL2McVq+YkLgHjAbbiYDg4yePWm4wQKc2aX/CT6f6Tf98f/AF6P+En0/wBJv++P/r1nzaRbPdu/7yNG8w+WDjG1wvGAeMHpioF0y0Dxo7TksJWz0GEYqAeCV7ZPaly0w5qhr/8ACT2HpN/3x/8AXo/4Sew9Jv8Avj/69Y0umxW+PknkbzHAIwVQKVxu9Qc9c/hVifSYJb0sgdYmaTcqsFCbWAyOOBz0waOWmHNUNH/hJ7D0m/74/wDr1Naa/Z3l0lvEJd75xuXA4Gf6VjtplqPJjMUxCmbzZkbnCHqRg/TAx1qpoHOuWxxjluv+6afJBptB7SaaTO3pKWkrnOgKaadTTTASmmnU00CGGmGnmmGmAxqpap/yCr3/AK4P/I1daqWqf8gq9/64P/I1RJaooorIs5KZoRc3IeJmbz5MkPj+M+1Pto1uXdIIypVCzkzbRtHXJxUc0Ez3VyyRSMpnkwQpI++amsmnsmnf7LI5khaNcpkAnHJBHPSuuy5dHr6nC377uKbe43GNrW6P7wIT5pK7s+uMdcc0kkM0RmDW1ziNsSsJDjPucc8H9atG/uns1jkt5mnBOZNgw4LbueMj6CnPfSGOZEs7hTI7spKg/fxkE4z69Pap/rf/AIJXu9yosMrXAgaGeObBf95NtwMcnkegolhmjMoe3uGEXMjLNuUAjOcgdxg1KtxKNXkvWtZ9sgYbQvzLkY+lTNeszXDf2fNvmBAcjJwU2+nHrxim/wCtf+CCt3KawziVStndiVwWDBzuYd+3uPzogheaFpoo3EaErzcAEnGWA49OavT6lNMZWFnOvmxupAUDDMoGeBk9O9VbV2hsXtZ7GWYbzIvyZGSu3kEduuRR0/4P/BC0b7jYoJpHDmO4iPltIsjykZUDJAOKbFE5YHybiHYCQxdvl43Y4HGRzV37c3kiIafOqEEMFHPKbTg45Pfmmi+uPLVWspc7T5pCn5m2bAR6DFL+t/8Agj93uUZA8CYlt7iJJDnDSEBj69OTTPPTOds2eefO9evbvVy+nmvIjGLSZMy+ZkqT/CFx+lUfs1x/zwl/74NXFRa1/Myk2noPW4CZ2eeuTk4nIyfXpVvQ2iOtW22Jlb5uS+f4T7VR+zXH/PCX/vg1e0OCZNZtmeJ1UbskqQPumiShZ2f4hBy5lc7SkpaSuQ9AKQ0tRTzw26q080cQJwC7AZPpzQA400isy41pSzRadE11OvJ4wmO/Pf8ACoY7jUJCoN5GBcDdGRH90/3R/wDX9DWcq0I7sLNmsaYazk1eSGRVvo1MT/dliH3OxBHXr3HrV9JEmiWaFg8TjKsO9aQnGauhCNVLVP8AkFXv/XB/5GrrVS1T/kFXv/XB/wCRrQktUUUVkWcdcf8AH3df9d5P/QjVrSYVnuJw0YlZIGeNWBILDGOB160p0u7uZbiaJFMbTy4JcD+MinDQ9QJGEQHsfMFdi5XBI4XGSm3Ysf2ZB50aSoRLI0pKxyHYNnRRxk9frxQ2lWwW4KiZgm/aScbMIGAIx6k/lUH9g6if4Ez/ANdBUsOkatAkixRwjzRhmLKWx3APapaXctX/AJSO2tfN0YzRwxO+JDJI+crtxjB7HnoetWZdHgiK/LKT5bkxh8klWUZzgdmJ6VWHh7UhwI0APbzBzT4dD1SCRZEhiZl6B2DAfgaHbowSfWJLJpFsNyhpFILYfdwAJAvT6Gqer2cVlNGkG8bt24NnscAgkDqKmk0HVZZHkkRGdySx8wcmm/8ACPan3jQ/WQU48qerFJNqyiZtJ+JrT/4R7Uv+eSf9/BR/wj2pf88o/wDv4K05oGXs59jM/E0v4mtL/hHtS/55J/38FH/CPal/zyT/AL+CjmgHs59jM/E1oaD/AMhu2/4F/wCgmn/8I9qX/PJP+/gq3pOjX9rqkE80aCNN24hweqkVMpQ5XYuEJqSujqaSlqnqd4LKykl/jPyxj1Y9K472O4ydV1e5lu3s9LIxHxLMBuwfQdjjv/8AWqvbaYgwz/v1AwWlYsufr1WpdOtfKjDzs4ckszAbWdj1JB+9+FWrm8+zoWBUNjBIOAw9T6V59Sq5O3QuMerAQQou12KFSMf34z+H3lpwbcrIF2ByTkDhHHRh7H/PWo7RleIPHIsqHoqAgY9sU1Yglyd8khB+4+QoP+yc9/8ACsW30RpZdSdvKuFKsNjOQQCPulhj8wQDVe2D2N55ef8ARpwSq/3W4P4df0qy8MuwnasoAztPU/Q//rqtL5M8LxoxwRnB48s8j+ZqoVHTkm0S4p7Gi1UtU/5BV7/1wf8AkaltZzNAC/Djgj6cVDqn/IKvP+uD/wAjXrp3VznLdFFFZllLTz/o7/8AXeb/ANGNVxTVOw/493/67zf+jGq2vStehHUlU1IDUQNPFIZIDTwTUYp4pDH5NKCabVfUrlrPTpp4xlwAF+pOB/OkMdc6laWkixzzASN0RQWP5DpT7a9t7rIglDMOSvQj8DzXGyGaOMSRgyMR5rFhl2bOMfXnJpBeSCaCW3YR3SEgg9CMdz0xxj3paXt1DW1zuqCwXqQPqazG1qxkjGJ2jbOGTad49vb61BFd2VztKJIodtoeVGALdMbumfxqZOS2QGwssbEhZEJHUAjilZ1UEswAHUk1mvYJ/FbofoBUJsrbvFjnulZudRfZHbzLl1qcUPyQ4nmPREOQPdj2FZblvtYuL+RS54jQcgD2HQ/XrVpYUjG2Mlf9xAP1606K0LMSIxk9SahxqVNHog0RQMVzdkbnNvEDkBeSPoO34/lUi6Pal/Me2Nw3dpmLZ/p+la8dsqct8xqYegraNOMVoDk2cs7R6JdSmW3cwXOBGyDiPH8I59STipLi6tnhjZBO8AcAMmHMbdRkHkfjVnUcag8LPH5kQLCKINjnp5je3oPem6dp8lqpKyeZI3LEj5foB6VjOkqjvE0U0ty1Y3kc+0PIGbp5g4z7N2/z2qK4J+1nYAzqcMD3U8fpwQfqKhfba3eJpBE8qkqxX923qG9PxqVJ3I8sQwoneRJgY8nj03ZPpWFSErcriP3b3QtkCgIbqcn9aNU/5Bd5/wBcH/kafGpFy45+Tg571Hqn/ILvf+uD/wAq9GmmoJPexzy3LlFFFSWUbD/USf8AXeb/ANGNVtap2P8AqZP+u83/AKMaramtuhn1JRUgqJakWkMkFOFRg08VIyQVl6pctcWM620LPHGQzSHgfKQTjPXpU+oyslq2wkc4AHV/8B+FZjyukGNpNpCud0gIjbsFUZ3MxPqfzpAZF2skcsk8UoKhd+xjkMMgYH59PaoYbSWdg6ToDOpHlzHYMZxx1yM/yNaKQyQtFBIscsjxFWVydpBGSCQPYflVe60mSO0860aS4tmO8bDukhY/xDH3lPeo5vfaGrWNDw1BdWc11bXSsjIUQHOcjk8HuMEV0Zt0OQ2WBGCD0qnpT/aLWF5VKuFBweCPwq3dkiznIOD5bcj6VQIlVdoxnNKQD1ArlNMs5r+3L/bJkZZAmN55HU498Z/KnyWc5CiO5ZAqys0jSMdyo2OR2P0qOd2vY19mr2udPsH90UYPauWNnLHFO818+6IpgK7YYN0/PpRqVtLaPLJDcObdZBHt8xtynAPOaXO7bB7NN2udQVNRTyLGoVm2l89OoHc1zn2K4eK18q4uDJcLu3lj5aDGcZ654qVoXhQS204OYUkd3YkElsbsEZ7DjiqUr7icPM04rcSSHagRe+PTsPyq3sKjAGBXOf2fJC9zDDO0jJIiK5kYFd3YjoaT7DdbsLe7l2u2/c2BsOGFLnt0D2afU1tWsPt0CKzFCrj5gOxOD/j+FVFtY7eOVHwY4nw7YxwehP41nyG5sJxmcyEpuRgxIII4OD3pqardIrq0aPvGGJH3hx1H4fqaOdPfQfsn0N7LmQIyHftyJOzj0PvVbUwTpN4e3kP/ACNYcEs0eCJpNy9CWzUV08xtrgvK53I2efahVVtYHQfc62in0VVjK5l2X+pk/wCu83/oxquLVKy/1Un/AF3l/wDRjVbU1stjPqSqakBqJTUgNIZKOTxVOXVLRJJFkuAkcWNxAJZyewxzj3ovryKzWBZn2LO+0nGeACSPqcAfjWXDbNcN58ihEDZVGHI929TSSuDdix/aK3IP2K2Z3zw0uFUe/XOB6VLDEWlSW5cyyx8jcMKvuo6fj1oMAiZWI+QngHtSXxcwbc/eBAqrJEXbILOJb7U3KMDFESZcnnnOMH61qaerQv5ZIOM7SPTJx+NVNLutmLWe3SKUR4R06OB2Pv3/ADq5ZgtKWPauTV1/KxurcpcWNEYlRgnrSXCl7WZEGWZGAHqcU7NLW4jmrO21K1gWP7AzYlWXPmKOnbrT7qS8it5HurPyYRHIrOZR8odgc/geMd66FnCKWdsKOprO1+MXNlFaPnFxMqHB6Drn9Kz5LdTR1L62KDrfz+diwLQzKm0iQD7v3W/H0ouodSuopYzYFBLKJSfMU4+ULj9K2rUbFIXhF+VB6AdKm3H1p8l+oKpboYEceoxm2b+zi0lupQHzQAykYxjPX3oeLUWhaJNOKKYliH7wHAU5B963iT600sfWjk8xe08jFYalvkkTTiryukjZkBGV/wAaG+35+TTSibZF2mUE5c5Jz/StgsfWmkmn7PzD2nkYF1a39y8bmzKeXGI8bwc479ah/s+9/wCfdv8Avof410RY0xmNL2SfUft5Loc/9gvP+fc/mP8AGoLyzuVsp2aEhRGxJyOOK6NiapamT/Zl3/1xf+Rp+xj3F7eXY1Mj1optFFyLGXZn93L/ANd5f/RjVaU1TtD8kv8A13l/9GNVpTW62MupMDUi/XA7k9qiWoJHF0fKVwLc/eYfx4PI+lId7BNKL11UDNojckj77dj9KtIkbtt3ZI6ZOTTfNDJ5eQoHGMcY96kEkJVQzpjs2RgUXsRuDxl/kJwgPp1qBQpvSuCyxpk5OACenP4GrCyRsRh0YDsDnNZ8cpg1qW3uAGhvwGictgKVGMY7nOPzqXKxUVdioytdtlcEIdp9DkcflmtW1TZFk9TWX8sWpCNmABTC7jy2DWurK2ArqfYGk1rctbD6Ws7V9RewiSO2QTXspxHH6Dux9hT3vHjQRW+bmYDmR8AD3OMDH5UhkWqzzS3EWmWyf68bppT/AMs0/wAf/retN3G8vjdknyrclIFH8TEcn8v6e9NjUfvAkm95OZZj3Ht6D+f0q3axKgURrtiTO0HuTyTQBZjG2MA9e9LmkJpM0wFzTSaCaaTQAE00mlJxTCaYhCajY04moyaYhGNUtSP/ABLbv/ri/wDKrTGqepH/AIlt3/1xf+VUSa1FFFc5sY9qfll/67y/+htVgyKmAclj0UdTVW2yRPtONs8u5sZ2/O3buanWNBEfOIQfxNnJLeg75/l2rpWxg9xwM0udzJGi/eBPBH171etomdQyAqrDOcYB/Pn9KpoXZRNNEI7UEYV+rEdOO4z0HtTbrUrv7PLNGdnGEUAE5PArGblb3RxS3ZbZfs+fPAIY9R0qGWSJxsKZHYVTtJb24iK3kvmMTgDaBj8qka2mBwAPrmuXkqSV2U1bYUwwNw0Q/AVHc2TyCOS2uCHt8tEJPmUHHv8AQfiBUSyvIyiINg9HP3T7irIIIMRO5iPnx6VKvzKwK5Bb+KNMnEQug0cj5LFVJVPQZ/r9elX/ADNNun2pcxsW5AUZwMe34UW+mWyLujtoY8+kYzVlbWMDHb0FdthlVI2jlPkFXB43GPDH/wCtT47VwxIBBY5PuauoqoMKAKduoGQR25438KOcep96s9sDpSZpM0AOzSE03NJmmApNITSE0hNAgJphNBNNJpiEJqNjTiajY1QhrGqepH/iXXX/AFyb+VWmNU9RP/Evuv8Ark38qZJtUUUVzG5kWyMwkSBN87zykKDgAeY3zNU32m1tZC8YF3dE/wCsHCqMYwvpx6VnpJK32iCM+XEbiUvt4Lneep/TFWIkVBwOa6FG61MG0mSs013IHuG4HRQMAVJIN7RQoCSTnAHb/JpqmlkTzBlWZJAMBlOOPSpqRbg1EafcW2AW4dSQCCeCakvmP2ZolP72X5QvfBPJ/LNUWt5JHzKFfgD7vXHTNWrRWjldXZmkZS0bOc89D/SueUpwp6ovRkLl95C43Yxx0FWLS3wB3HUk/wARqO3jzKynseavgjGBVUaajFPqxEuaXNRg0ua2GSZopmaXNIB1GabmkzQA7NITSZpCaYC5ppNITTSaBCk0wmgmmE0xAxqNjSk1GxqkIRjVPUD/AMS+5/65N/KrLGqmoH/QLn/rk38qfQnqb9FLRXKdBzaSqkk4J58+Xt/tmpRcxjufyqo3+uuP+u8v/oZorZTdjPkReF3CP4j+VPF7AP4j/wB81nUUudhyo1Bf24/jb/vk04X9u0kW1iWDjqMdeD/OsnFGBkZGR6VMnzRaY+U15Z47O43SFhHJ0YDPPSnf2paf32/74NUW1IS+Yl1Buiboo7VX+0hgyuHAx8ucMPpjjA+lYU5ThFRlHYppN3NgapZ/32/74NL/AGrZ/wDPRv8Avg1h7g5JCbAew6CjFdHMxWNz+1bP/no3/fBpf7Ws/wDno3/fBrCxRii4WN3+1rL/AJ6P/wB8Gnw6hbXEoihdi5GQCpFc/irWlcagn+6f5UJisbpNITTSaQmrEOJppNITTCaYhSaYTQTTCaYgY1GxpSajY00SIxqpfn/Qbj/rm38qsMaqXx/0K4/65t/KqJudNRRRXGdRjPokxlkdLtFWR2cAwkkbiTjO73pP7DuP+f2P/vwf/iqKKYg/sO4/5/Y/+/B/+Ko/sO4/5/Y/+/B/+KoooAP7DuP+f2P/AL8H/wCKo/sO4/5/Y/8Avwf/AIqiigA/sO4/5/I/+/B/+Ko/sOf/AJ/I/wDvwf8A4qiimAf2Hcf8/kf/AH4P/wAVR/Ydx/z+x/8Afg//ABVFFIA/sO4/5/Y/+/B/+Ko/sO4/5/Y/+/B/+KoooAP7DuP+f2P/AL8H/wCKqSDSrq2lEqXkRYAjBtzj/wBDoopgWfKvv+fm2/8AAc//ABdJ5N9/z9W//gOf/i6KKOZhZCeRff8AP1b/APgOf/i6Ps17/wA/Vv8A+A5/+Looo5mFkJ9lvT/y9W//AIDn/wCLpDZXp/5e4P8AwHP/AMXRRRzMXKhDYXh/5e4P/Ac//F006ddn/l7h/wDAc/8AxdFFPnl3DlQh0u6P/L5D/wCA5/8Ai6jl0S4mjaNr2MBwVJEB7/8AAqKKOeXcOSPY1/MH939aKKKko//Zdata:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABQODxIPDRQSEBIXFRQYHjIhHhwcHj0sLiQySUBMS0dARkVQWnNiUFVtVkVGZIhlbXd7gYKBTmCNl4x9lnN+gXz/2wBDARUXFx4aHjshITt8U0ZTfHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHz/wAARCAElAMEDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwBpMsk85a4uBiaQALO4AAYgAAGnqjH/AJeLr/wIf/GkQfvJ/wDrvL/6GamUV0pKy0Odt3eo1YSes91/4EP/AI08QD/ntdf+BMn+NSKKkUUrLsO7IhbL/wA9rr/wJk/xp4tU/wCet1/4Eyf41KBUgFKyKuyAWkf/AD1uv/AmT/GnCzi/56XX/gTJ/jU4FPApWQ9St9ii/wCel1/4Eyf40v2GH/npdf8AgTJ/8VVkCnAUrIepV+wQ/wDPS6/8CZP/AIqj7DD/AM9Lr/wJk/8AiqtYpcUtA1Kn2CH/AJ6XP/gTJ/8AFUfYIf8Anpdf+BMn/wAVVvFGKNA1Kn2GH/npdf8AgTJ/8VSfYYv+el1/4Eyf41bxSEU7INSmbKL/AJ6XX/gTJ/jSGzj/AOet1/4Eyf41cIppFFkLUpm0j/563X/gTJ/jTTar/wA9br/wJk/xq2RVe7OyBiJPK77/AEobUVdgk27IhNuB/wAtrr/wJk/xphg9J7r/AMCH/wAasRqShLNtiVd26Q5YgdelQ/aI2bKBmjzjcflz7j2qFWpvZfgU6cyIxEf8t7r/AMCH/wAar3fmR20rpc3QZUJB+0PwcfWrZljPVW/Cq18QbK4wCP3bdfpWqafQzaa6nTYHoPyooormNznoh80//XeX/wBDNTqKjhHM3/XeX/0M1KzpEm+Q4H6mutPRHO1qSKKkUVUGoRD/AJZv+lWraeO5B8skMOqt1FTzJjs0SgU8CqbalDHI6GKQlCVJGO1C6vb55ilA9cDj9anmRVmXgKcBSeZEIfPMi+SBu3+1UjrVsvSGYj14Gf1pNjsaGKXFVp9Rht4oJGjdxMu5QuPl+v51D/blsDjyJs/h/jSuOxoYpcVFaXkF6rGEkMv3kYYIqS4lS1t3nmJ2L2HVj2AouAuKMVVtNUt7uYQqjxORld+MMfT61JfXkenpG0yM4kJAC44ouFiXFIRVH+3Lf/nhN+n+NPi1aCaOZ1ikAhUMQcZPOOKLhYskUhFQ2l/FeymONHjIG7L4wah/taAzeV5UgbfsycYznGadxWLJFQzxCWPYQDyDg9D60t5eR2bqsiO+8EgrSxTRz2/nfcTBJ3dgKJJSTiCuncpyBnhmwPkdsPn+4O34k80RwgruYfQUk99Z+VuVWLk8fLg4p0N3DMRGNyP23d6zopRvYqo3K1xSoHQVUvx/oNx/1zb+VXmFVNQH+gXP/XJv5V0mHU36KKK5DpMO3H+u/wCu8v8A6G1R6gMeT+P9KmthxL/13l/9Dao9R48n/gX9K6H8JiviGQRq9rIxAyqkg4punki7UjriprUf6DN/uNUOn/8AH0v0qOxfcZcf8fkv/XU/zqzqUaRrAyKF3bs4GPSq1z/x+TevmHH51LOby52LLF9zO3Ax19aQx2T/AGNtz8vm9Kk0iFZ2kVkVuR1GcUTwmDSQp5O8ZPvzVa2tHuwwSTYV/WgC7rcIgFuq9Pm49OlJZwLJpM7lFLIrENjnim6qHSCzjk6xqVz64xVeKzea1e4WTCoCSn0pAO0gkXykHBKH+lP1i6a5vPKXiOA7QPVu5p+iOpujGygbhuD9+O1Vb3jUbj2lP86AH31obTymRz83IPcMOpFbVlLHqVojTRozjghlBwfUVkajfRXqQrCrqYs7t3fOK0NAH7j/AIEaAMe6UJfToowqykAe2a17u2jXTZ5UVUbH8Ix3rJvf+Qjc/wDXZv51uXw/4k9x9P6ihAZWj83f/Af61VnGZ5h/tt/Orejf8fn/AAH+tVycX7H/AKbH+dHQAnuDcRwh+XjUqT6+hqwv/IHf8P8A0Ko9RthbTgr/AKuUblH931FSr/yBX+o/9CpoCGwjSSVg6hhx1GadJYzRzZ+UruyMHoM0ljvBlMYzIFyo9TThqNyZFWTYPmAYbeR60K1tRO99C6MlAT1qpqA/4l91/wBcm/lV44YZHSqmoj/iXXX/AFyb+VbGZs0UUVym5kWo+WX/AK7y/wDoxqbqMTvFHIgLBM7gPQ96ktB+7l/67y/+jGq0g6noB1Jro3RjszHiuvKt3iCbt4I3Z6ZqTTY2M4fBCjgZ7mtHNoTkzRf99CrEUSYDx4cHoQciosXc5+5P+nTD/pqf510cKKYwSM00vazMq+fEz9AAwyak3wW+EkmSM9QHYA4o2Ag1O3efTmSBcurBwo/ix1FYtlffYSzeUXJ7E4xiunQhwGRgykZDA8Y9ahaWxZiWuYc9/nFIZla5Os8NjKBjepYr1xnHFP08/wDEluj/ANM3rYiSIxExMrxt1ZTkVHHNYxtuS6h/77FIDn9FOb9f90/0qK9b/iZXA5/1x/nXStJYM2TdQ5J/vipp1hChrh0jUcAsQKAMDWoo4UtmjRV37s7Rj0q54e5gH+8a0YpLaRTDDNHIcZ2qwJxSH7NbOPMmjiZhwGYDIoA5a+b/AImVzwf9c38637/jRbg+39RVwrFcoHRlkToGU5FN822tj5b3ESMOqs4BFAHP6Ic3p/3f6iqrH/T26/64/wDoVdT5ETYkXBB5DDoaWWJZDlh2pgYWtnD2v+4f50+2ha50h44uXIyo9cHOK2PLXy/Lx8tMWJYySvemI5y1uTaSljES3dTwRUe/fcb8Y3vux6ZNdJJEjnLDmomt4/SiwXI1H7sVV1If8S66/wCuLfyq7tCjA6CqmpD/AIlt3/1xf+VWQatFFFc5sZdmP3Un/XeX/wBGNUWrMRHDGDhWJLD1xjH86nsv9VJ/13l/9GNTdTt5JoY3iUsYycqOpB//AFVu/hM1uU7eyFxbNIgYyAHCjvU+kCe1uyrxskbj5gemaqxXFzbjELPGP93p+daWlXxuJjDdBS+MrIBjPsahWKMq1G29iK8EScVb10bryMtyfJH8zVRc214VlUhon+Ze9S6ldpdzJIisoVAmG6k5P+NICe7kcaLawhiEf7wHcDtTLHTUu7aSQbvMXIVV74GRVmaymn0qERqWlg5ZB1II5/GqMFxeWoIg8yIZyfk6fmKALGlR3VnekSQvHFIpWQHgex+tN1PS4dOSIws7K5x85FXNK1OS4uPs96FfcMpIBg59DUniUfuLb/fP8qBlDTtMivo97Fg6vgAYxxg1FrDvJqUvnHcY8Ko7AYrT8O/6tv8AfP8AIVmarg6rdZ6bh/IUAathpK2863FuzZxg7j2NZWoy/btTkZfmG7y4/wAOP55Naul3l/8AahHdxhINpwdmOe1YtnxexY/56UAWbC9k0maeJlLLyCno/Y/Q1UMUkqSXUvzhnw7nux/pV/xAALyIgYJhGffk1Myj/hG2bpgL/wChUATaBcb7Y27HJiPy/Q8j+tahFc/oP/HzIfYV0JFAhhFNIp5ppqhEZFRsKlIpjUxELCqepD/iWXf/AFxf+VXmFU9TH/EsvP8Ari/8jVCNGikornNTPsR+5k/67zf+jGpNQuprNYmhC4fIYsM89v606x/1En/Xeb/0Y1WmjSaMxyKGQ9Qa26GfUz1voZrCYXDKtyysAApwfSqulg/bB7Lz7Vof2PCTwpx/vGrNvYJBwoCjOeO9TYZHraKdPMpUb1KgNjnFUdFRXll3AE/Lgkcjr0rbmto7uEwzKWjJzgHHIqG10xbSQtEMA9ckmgZHqtzNplvC9qFw77XLDPbiq8GqxTWM4vXRbhgyoFU4Ixx+ua2XiSeNopUDow5Uis5tAtgeEOO3zmkMxtJB+3R8fdUk+1aniX/j3tf94/yq7baZHbcIoUHk9yfxqa8sYL5FWdC2w5XDEYP4UAZnhz/VN/vn+QrM1gEatcqRgkgj3GBXR2OnrZEiLhCckZzzT73T7e+wZowXXgMDgj8RSAhsLyzuGEUcoeUDONp6VzVn/wAf0X/XT+tdFbaRHaTCaAbXxjJYnik/sO2jmWSBCrKcjLk80wMzxD/x+Q/9cR/M1M3/ACLLfRf/AEKtK70qC82vMpMqrtBDEcUqadH9ka0lXdAcYUE9jnrQBi6D/wAfMn0FdGapW2lJaSl4Rtz1+Ymr5U+lAiI00ipCM9KaVPpTAiIphFSmmGmIhYVT1P8A5Bd5/wBcX/kavMKpan/yC7z/AK4v/KqEXqKKKxNCjYf6iT/rvN/6Mak1K7ktIkWHAkkJ+YjO0D/9dOsP+PeT/rvN/wCjGqtrf/Lt/wAC/pWr2M+pAkmoPA0y3L+WuSfn54q7o+oyzzG3uW8wEZRz1HsfWjR1WS2ZHG5GyGHqKuQabBFOHhQIfbPSpKMibUry6uikEphQttjVePzNJLdanZSqHuWL43AE7lIqvbf8fsX/AF0/rV3X/wDj6g/64j+ZpAXdRvnl0WK5t2aF3Zc7TyPUVFoWoSNcvBdSvI0gzGXOenUf59Khm/5F2L/fX+tZ6Rv5LXEZP7lhkjqvofzoGWbS+vPt8Qe5lZPMwVLcEZ6Vf1zU7iGdLe2fyhsDOwHJz2FZNpzfQH1cVZ17/kI/9sl/rSASR9UhtluXu38psYIfJ56Vp6Xq0ktjcvcgPJbqWyON4xn8+KnsbeK70yKK4QOm1Tg+uKbPp8NnYXbQqE3RNkDvwaAMaO81O/mIS5KOQWCqdqgVcuJL620XE07i4Eo+cNng9s1nafcpaXAmlDFdpGF61qapex3+iCaFWVfNAww54oApWr6pebvIu3JXGdz461Jpmq3K3SwzymWOQ7ct1VuxzUnh6WOKSYyyJGDtxuYDPWs2z/4/of8ArpQBauf7WsVX7Tdv8xwCr5pIG1S6XfDduVDYOXxV/wARf8elr/vn+VHh3/Vv/vn+QpgV9W1W4Ny0Nu5gjTrtHzMar3Eup2YR5btwGOFw+eaZrCldVuFf5dxDD3GKt/2zA8YjmsfMUdywJH0yKAL2m3pvrYNIAJVO1iOh96tNUFmls8Sz2hHlt2Axg+49asGqRJE1UtT/AOQXef8AXF/5VeaqWqf8gu9/64P/ACpiLlFFFZGhT0//AI93/wCu83/oxqra5/y7fRv6VZ0//j3f/rvN/wCjGqS8s0vYQjMUdDlGHb2rR7GfUraSxisJJ9u4RhmI9cVa07V0vLkReQYvlLbi2fwrM/sq6HAlAHoM4NXNO09reTeTuc98YAHtSKMm1/4/Yv8Arp/Wr3iAYu4Af+eI/mamvNEYztLayBVc7ipH3T7GoBo1w5HmzEj15J/WkA6b/kXIv99f60/w8od50YAqwAIPcYNXbjTXn00W0RWLDAjdyMCmaRYy2Mz+YQwcjG3tQMw7MYvoQOgkwPzq1r3/ACEf+2S/1qdNEube6SQyo4V92ADV/UtJ+3iOZH8udV28jIYehpANju/7O0aK4MZkBVBtBx1FNXVBqWm3/wC6MXlxHq2c5B/wqj/Yl5jaZwV/u8kflWpp2mrbxPGw3q4IkLcbsjGPpQBiaNGst6EdA6lDwRkVqa5bx22l7YlCgyLwBgd6rS+H54nP2e4+T+EnIYD04qddMuZdL+xPKDIJN/mNkjHpQBm6dYR6h5gkZlKYxtxznP8AhWvaaKltJvGWfpuY5wPam6TYTWMr+YQ28jBHbGf8a2TQBh+JRttrYDs5/lSeHeYn/wB8/wAhVzWbCTUIohE6oY23HcOvFRaRaS2QZJCGJYtkUwIdT1LTTK9vd28kjRHG4KOPoc1W1mxhsoYWhUAuxBIz6Vc1TRvtUxuYHCSEYdWGQ3vWedHujw8+R75OKALOgnELgdDIf5CtQ1WsLT7KgUZwO56k+tWjTQiM1S1T/kFXn/XB/wCRq6apap/yCr3/AK4P/I1RJbooorIsyLbVbW2WWGXzN6zy52rkffY1ONdsB/z2/wC+P/r1z1x/x93X/XxJ/wChGrGl26XFzJ5qh1iiaTYzbQxGBgnsOa61CPLdnJ7WXPyo3B4g0/8A6bf98f8A16cPEWnj/nt/3x/9eqM+lW/keUAwMTSESjHIDKAG454PFRf2Tbv9r8syqIGdVZnB3FBk8Af4VNoF81Q1B4j04dpv++P/AK9L/wAJLp/pN/3xWfdaZBJeO8aMiBn3orBVAUDkccDn0NNk0e2i3hmmYqJGyGA4RlGOncN19qVqYc1Q0v8AhJtP9Jv++P8A69L/AMJPp/pN/wB8f/XrEtrSBdTu7aRDKsWRGWViAcgAtt5/GrL2McVq+YkLgHjAbbiYDg4yePWm4wQKc2aX/CT6f6Tf98f/AF6P+En0/wBJv++P/r1nzaRbPdu/7yNG8w+WDjG1wvGAeMHpioF0y0Dxo7TksJWz0GEYqAeCV7ZPaly0w5qhr/8ACT2HpN/3x/8AXo/4Sew9Jv8Avj/69Y0umxW+PknkbzHAIwVQKVxu9Qc9c/hVifSYJb0sgdYmaTcqsFCbWAyOOBz0waOWmHNUNH/hJ7D0m/74/wDr1Naa/Z3l0lvEJd75xuXA4Gf6VjtplqPJjMUxCmbzZkbnCHqRg/TAx1qpoHOuWxxjluv+6afJBptB7SaaTO3pKWkrnOgKaadTTTASmmnU00CGGmGnmmGmAxqpap/yCr3/AK4P/I1daqWqf8gq9/64P/I1RJaooorIs5KZoRc3IeJmbz5MkPj+M+1Pto1uXdIIypVCzkzbRtHXJxUc0Ez3VyyRSMpnkwQpI++amsmnsmnf7LI5khaNcpkAnHJBHPSuuy5dHr6nC377uKbe43GNrW6P7wIT5pK7s+uMdcc0kkM0RmDW1ziNsSsJDjPucc8H9atG/uns1jkt5mnBOZNgw4LbueMj6CnPfSGOZEs7hTI7spKg/fxkE4z69Pap/rf/AIJXu9yosMrXAgaGeObBf95NtwMcnkegolhmjMoe3uGEXMjLNuUAjOcgdxg1KtxKNXkvWtZ9sgYbQvzLkY+lTNeszXDf2fNvmBAcjJwU2+nHrxim/wCtf+CCt3KawziVStndiVwWDBzuYd+3uPzogheaFpoo3EaErzcAEnGWA49OavT6lNMZWFnOvmxupAUDDMoGeBk9O9VbV2hsXtZ7GWYbzIvyZGSu3kEduuRR0/4P/BC0b7jYoJpHDmO4iPltIsjykZUDJAOKbFE5YHybiHYCQxdvl43Y4HGRzV37c3kiIafOqEEMFHPKbTg45Pfmmi+uPLVWspc7T5pCn5m2bAR6DFL+t/8Agj93uUZA8CYlt7iJJDnDSEBj69OTTPPTOds2eefO9evbvVy+nmvIjGLSZMy+ZkqT/CFx+lUfs1x/zwl/74NXFRa1/Myk2noPW4CZ2eeuTk4nIyfXpVvQ2iOtW22Jlb5uS+f4T7VR+zXH/PCX/vg1e0OCZNZtmeJ1UbskqQPumiShZ2f4hBy5lc7SkpaSuQ9AKQ0tRTzw26q080cQJwC7AZPpzQA400isy41pSzRadE11OvJ4wmO/Pf8ACoY7jUJCoN5GBcDdGRH90/3R/wDX9DWcq0I7sLNmsaYazk1eSGRVvo1MT/dliH3OxBHXr3HrV9JEmiWaFg8TjKsO9aQnGauhCNVLVP8AkFXv/XB/5GrrVS1T/kFXv/XB/wCRrQktUUUVkWcdcf8AH3df9d5P/QjVrSYVnuJw0YlZIGeNWBILDGOB160p0u7uZbiaJFMbTy4JcD+MinDQ9QJGEQHsfMFdi5XBI4XGSm3Ysf2ZB50aSoRLI0pKxyHYNnRRxk9frxQ2lWwW4KiZgm/aScbMIGAIx6k/lUH9g6if4Ez/ANdBUsOkatAkixRwjzRhmLKWx3APapaXctX/AJSO2tfN0YzRwxO+JDJI+crtxjB7HnoetWZdHgiK/LKT5bkxh8klWUZzgdmJ6VWHh7UhwI0APbzBzT4dD1SCRZEhiZl6B2DAfgaHbowSfWJLJpFsNyhpFILYfdwAJAvT6Gqer2cVlNGkG8bt24NnscAgkDqKmk0HVZZHkkRGdySx8wcmm/8ACPan3jQ/WQU48qerFJNqyiZtJ+JrT/4R7Uv+eSf9/BR/wj2pf88o/wDv4K05oGXs59jM/E0v4mtL/hHtS/55J/38FH/CPal/zyT/AL+CjmgHs59jM/E1oaD/AMhu2/4F/wCgmn/8I9qX/PJP+/gq3pOjX9rqkE80aCNN24hweqkVMpQ5XYuEJqSujqaSlqnqd4LKykl/jPyxj1Y9K472O4ydV1e5lu3s9LIxHxLMBuwfQdjjv/8AWqvbaYgwz/v1AwWlYsufr1WpdOtfKjDzs4ckszAbWdj1JB+9+FWrm8+zoWBUNjBIOAw9T6V59Sq5O3QuMerAQQou12KFSMf34z+H3lpwbcrIF2ByTkDhHHRh7H/PWo7RleIPHIsqHoqAgY9sU1Yglyd8khB+4+QoP+yc9/8ACsW30RpZdSdvKuFKsNjOQQCPulhj8wQDVe2D2N55ef8ARpwSq/3W4P4df0qy8MuwnasoAztPU/Q//rqtL5M8LxoxwRnB48s8j+ZqoVHTkm0S4p7Gi1UtU/5BV7/1wf8AkaltZzNAC/Djgj6cVDqn/IKvP+uD/wAjXrp3VznLdFFFZllLTz/o7/8AXeb/ANGNVxTVOw/493/67zf+jGq2vStehHUlU1IDUQNPFIZIDTwTUYp4pDH5NKCabVfUrlrPTpp4xlwAF+pOB/OkMdc6laWkixzzASN0RQWP5DpT7a9t7rIglDMOSvQj8DzXGyGaOMSRgyMR5rFhl2bOMfXnJpBeSCaCW3YR3SEgg9CMdz0xxj3paXt1DW1zuqCwXqQPqazG1qxkjGJ2jbOGTad49vb61BFd2VztKJIodtoeVGALdMbumfxqZOS2QGwssbEhZEJHUAjilZ1UEswAHUk1mvYJ/FbofoBUJsrbvFjnulZudRfZHbzLl1qcUPyQ4nmPREOQPdj2FZblvtYuL+RS54jQcgD2HQ/XrVpYUjG2Mlf9xAP1606K0LMSIxk9SahxqVNHog0RQMVzdkbnNvEDkBeSPoO34/lUi6Pal/Me2Nw3dpmLZ/p+la8dsqct8xqYegraNOMVoDk2cs7R6JdSmW3cwXOBGyDiPH8I59STipLi6tnhjZBO8AcAMmHMbdRkHkfjVnUcag8LPH5kQLCKINjnp5je3oPem6dp8lqpKyeZI3LEj5foB6VjOkqjvE0U0ty1Y3kc+0PIGbp5g4z7N2/z2qK4J+1nYAzqcMD3U8fpwQfqKhfba3eJpBE8qkqxX923qG9PxqVJ3I8sQwoneRJgY8nj03ZPpWFSErcriP3b3QtkCgIbqcn9aNU/5Bd5/wBcH/kafGpFy45+Tg571Hqn/ILvf+uD/wAq9GmmoJPexzy3LlFFFSWUbD/USf8AXeb/ANGNVtap2P8AqZP+u83/AKMaramtuhn1JRUgqJakWkMkFOFRg08VIyQVl6pctcWM620LPHGQzSHgfKQTjPXpU+oyslq2wkc4AHV/8B+FZjyukGNpNpCud0gIjbsFUZ3MxPqfzpAZF2skcsk8UoKhd+xjkMMgYH59PaoYbSWdg6ToDOpHlzHYMZxx1yM/yNaKQyQtFBIscsjxFWVydpBGSCQPYflVe60mSO0860aS4tmO8bDukhY/xDH3lPeo5vfaGrWNDw1BdWc11bXSsjIUQHOcjk8HuMEV0Zt0OQ2WBGCD0qnpT/aLWF5VKuFBweCPwq3dkiznIOD5bcj6VQIlVdoxnNKQD1ArlNMs5r+3L/bJkZZAmN55HU498Z/KnyWc5CiO5ZAqys0jSMdyo2OR2P0qOd2vY19mr2udPsH90UYPauWNnLHFO818+6IpgK7YYN0/PpRqVtLaPLJDcObdZBHt8xtynAPOaXO7bB7NN2udQVNRTyLGoVm2l89OoHc1zn2K4eK18q4uDJcLu3lj5aDGcZ654qVoXhQS204OYUkd3YkElsbsEZ7DjiqUr7icPM04rcSSHagRe+PTsPyq3sKjAGBXOf2fJC9zDDO0jJIiK5kYFd3YjoaT7DdbsLe7l2u2/c2BsOGFLnt0D2afU1tWsPt0CKzFCrj5gOxOD/j+FVFtY7eOVHwY4nw7YxwehP41nyG5sJxmcyEpuRgxIII4OD3pqardIrq0aPvGGJH3hx1H4fqaOdPfQfsn0N7LmQIyHftyJOzj0PvVbUwTpN4e3kP/ACNYcEs0eCJpNy9CWzUV08xtrgvK53I2efahVVtYHQfc62in0VVjK5l2X+pk/wCu83/oxquLVKy/1Un/AF3l/wDRjVbU1stjPqSqakBqJTUgNIZKOTxVOXVLRJJFkuAkcWNxAJZyewxzj3ovryKzWBZn2LO+0nGeACSPqcAfjWXDbNcN58ihEDZVGHI929TSSuDdix/aK3IP2K2Z3zw0uFUe/XOB6VLDEWlSW5cyyx8jcMKvuo6fj1oMAiZWI+QngHtSXxcwbc/eBAqrJEXbILOJb7U3KMDFESZcnnnOMH61qaerQv5ZIOM7SPTJx+NVNLutmLWe3SKUR4R06OB2Pv3/ADq5ZgtKWPauTV1/KxurcpcWNEYlRgnrSXCl7WZEGWZGAHqcU7NLW4jmrO21K1gWP7AzYlWXPmKOnbrT7qS8it5HurPyYRHIrOZR8odgc/geMd66FnCKWdsKOprO1+MXNlFaPnFxMqHB6Drn9Kz5LdTR1L62KDrfz+diwLQzKm0iQD7v3W/H0ouodSuopYzYFBLKJSfMU4+ULj9K2rUbFIXhF+VB6AdKm3H1p8l+oKpboYEceoxm2b+zi0lupQHzQAykYxjPX3oeLUWhaJNOKKYliH7wHAU5B963iT600sfWjk8xe08jFYalvkkTTiryukjZkBGV/wAaG+35+TTSibZF2mUE5c5Jz/StgsfWmkmn7PzD2nkYF1a39y8bmzKeXGI8bwc479ah/s+9/wCfdv8Avof410RY0xmNL2SfUft5Loc/9gvP+fc/mP8AGoLyzuVsp2aEhRGxJyOOK6NiapamT/Zl3/1xf+Rp+xj3F7eXY1Mj1optFFyLGXZn93L/ANd5f/RjVaU1TtD8kv8A13l/9GNVpTW62MupMDUi/XA7k9qiWoJHF0fKVwLc/eYfx4PI+lId7BNKL11UDNojckj77dj9KtIkbtt3ZI6ZOTTfNDJ5eQoHGMcY96kEkJVQzpjs2RgUXsRuDxl/kJwgPp1qBQpvSuCyxpk5OACenP4GrCyRsRh0YDsDnNZ8cpg1qW3uAGhvwGictgKVGMY7nOPzqXKxUVdioytdtlcEIdp9DkcflmtW1TZFk9TWX8sWpCNmABTC7jy2DWurK2ArqfYGk1rctbD6Ws7V9RewiSO2QTXspxHH6Dux9hT3vHjQRW+bmYDmR8AD3OMDH5UhkWqzzS3EWmWyf68bppT/AMs0/wAf/retN3G8vjdknyrclIFH8TEcn8v6e9NjUfvAkm95OZZj3Ht6D+f0q3axKgURrtiTO0HuTyTQBZjG2MA9e9LmkJpM0wFzTSaCaaTQAE00mlJxTCaYhCajY04moyaYhGNUtSP/ABLbv/ri/wDKrTGqepH/AIlt3/1xf+VUSa1FFFc5sY9qfll/67y/+htVgyKmAclj0UdTVW2yRPtONs8u5sZ2/O3buanWNBEfOIQfxNnJLeg75/l2rpWxg9xwM0udzJGi/eBPBH171etomdQyAqrDOcYB/Pn9KpoXZRNNEI7UEYV+rEdOO4z0HtTbrUrv7PLNGdnGEUAE5PArGblb3RxS3ZbZfs+fPAIY9R0qGWSJxsKZHYVTtJb24iK3kvmMTgDaBj8qka2mBwAPrmuXkqSV2U1bYUwwNw0Q/AVHc2TyCOS2uCHt8tEJPmUHHv8AQfiBUSyvIyiINg9HP3T7irIIIMRO5iPnx6VKvzKwK5Bb+KNMnEQug0cj5LFVJVPQZ/r9elX/ADNNun2pcxsW5AUZwMe34UW+mWyLujtoY8+kYzVlbWMDHb0FdthlVI2jlPkFXB43GPDH/wCtT47VwxIBBY5PuauoqoMKAKduoGQR25438KOcep96s9sDpSZpM0AOzSE03NJmmApNITSE0hNAgJphNBNNJpiEJqNjTiajY1QhrGqepH/iXXX/AFyb+VWmNU9RP/Evuv8Ark38qZJtUUUVzG5kWyMwkSBN87zykKDgAeY3zNU32m1tZC8YF3dE/wCsHCqMYwvpx6VnpJK32iCM+XEbiUvt4Lneep/TFWIkVBwOa6FG61MG0mSs013IHuG4HRQMAVJIN7RQoCSTnAHb/JpqmlkTzBlWZJAMBlOOPSpqRbg1EafcW2AW4dSQCCeCakvmP2ZolP72X5QvfBPJ/LNUWt5JHzKFfgD7vXHTNWrRWjldXZmkZS0bOc89D/SueUpwp6ovRkLl95C43Yxx0FWLS3wB3HUk/wARqO3jzKynseavgjGBVUaajFPqxEuaXNRg0ua2GSZopmaXNIB1GabmkzQA7NITSZpCaYC5ppNITTSaBCk0wmgmmE0xAxqNjSk1GxqkIRjVPUD/AMS+5/65N/KrLGqmoH/QLn/rk38qfQnqb9FLRXKdBzaSqkk4J58+Xt/tmpRcxjufyqo3+uuP+u8v/oZorZTdjPkReF3CP4j+VPF7AP4j/wB81nUUudhyo1Bf24/jb/vk04X9u0kW1iWDjqMdeD/OsnFGBkZGR6VMnzRaY+U15Z47O43SFhHJ0YDPPSnf2paf32/74NUW1IS+Yl1Buiboo7VX+0hgyuHAx8ucMPpjjA+lYU5ThFRlHYppN3NgapZ/32/74NL/AGrZ/wDPRv8Avg1h7g5JCbAew6CjFdHMxWNz+1bP/no3/fBpf7Ws/wDno3/fBrCxRii4WN3+1rL/AJ6P/wB8Gnw6hbXEoihdi5GQCpFc/irWlcagn+6f5UJisbpNITTSaQmrEOJppNITTCaYhSaYTQTTCaYgY1GxpSajY00SIxqpfn/Qbj/rm38qsMaqXx/0K4/65t/KqJudNRRRXGdRjPokxlkdLtFWR2cAwkkbiTjO73pP7DuP+f2P/vwf/iqKKYg/sO4/5/Y/+/B/+Ko/sO4/5/Y/+/B/+KoooAP7DuP+f2P/AL8H/wCKo/sO4/5/Y/8Avwf/AIqiigA/sO4/5/I/+/B/+Ko/sOf/AJ/I/wDvwf8A4qiimAf2Hcf8/kf/AH4P/wAVR/Ydx/z+x/8Afg//ABVFFIA/sO4/5/Y/+/B/+Ko/sO4/5/Y/+/B/+KoooAP7DuP+f2P/AL8H/wCKqSDSrq2lEqXkRYAjBtzj/wBDoopgWfKvv+fm2/8AAc//ABdJ5N9/z9W//gOf/i6KKOZhZCeRff8AP1b/APgOf/i6Ps17/wA/Vv8A+A5/+Looo5mFkJ9lvT/y9W//AIDn/wCLpDZXp/5e4P8AwHP/AMXRRRzMXKhDYXh/5e4P/Ac//F006ddn/l7h/wDAc/8AxdFFPnl3DlQh0u6P/L5D/wCA5/8Ai6jl0S4mjaNr2MBwVJEB7/8AAqKKOeXcOSPY1/MH939aKKKko//Z Software processes & threads Process (OS process): Encapsulated entity of a program running in its own private address space It consists of a private copy of program code and data along with file descriptors and permissions It has a dedicated heap and stack space from which data is accessed and modified. Thread Lightweight execution context that runs under a process They share address space, program code and operating system resource of their parent process Can be created and destroyed with low overhead in comparison to that of a process Consists of a small amount of thread local storage space 14

Processes & threads Process Thread 1 Registers Registers Registers Thread 2 Time Program Counter Program Counter Program Counter Thread 3 Stack Stack Stack Data File Descr Program Counter Registers Stack Code Data File Descr Code 15

Parallel computing Performing certain computations simultaneously using multiple resources Amdahl's law & Gustafson's law Speedup only comes from the parallelizable part of the code i.e. Serial part of the code will impact or limit the achievable performance Amdahl's law Gustafson's law Where, Slatency is the theoretical overall speedup s is the speedup in the parallel part p is the percentage of the execution time of serial part 16

Parallel Programming Is a large topic. Many tools and techniques, a few: pthreads is a standard API for managing threads Fundamental API for threading in Linux Cilk Plus Language support by compiler extensions: appears as C/C++ with extensions TBB (threading building blocks from Intel) C++ large use of templates commercial binary distribution with support or open source C++11 threads CUDA and OpenCL GPU or CPU/GPU unified programming models OpenMP http://concurrency.web.cern.ch/GaudiHive A framework from the HEP community 17

OpenMP A specification: See https://www.openmp.org Are compiler directives, routines and variables that can be used to specify high-level parallelism in C, C++ and Fortran GCC 4.4 OpenMP 3.0 4.9 OpenMP 4.0 6.1 OpenMP 4.5 Clang 3.7 OpenMP 3.1 Intel 12 OpenMP 3.1 16 OpenMP 4.0 17 OpenMP 4.5 18

OpenMP Code looks similar to a serial version #pragma are used to indicate handling of parallel parts Usually uses a fork-join model Sequential Sequential Parallel Region Parallel Region master thread 19

OpenMP May need to compile with fopenmp (check your compiler) Most OpenMP features are used through pragmas #pragma omp construct [clause [clause] ] You can change the number of threads via environment or an API or specify it in the pragma export OMP_NUM_THREADS=16 20

Parallel regions Threads (up to the number configured) are created, if needed, when the pragma is crossed Threads execute the parallel region, the sequential part continues once all the threads have come to the end of the region Data is shared, but stack variables declared in the parallel region are private #pragma omp parallel { function_called_in_parallel(); } function_sequential(); 21

Parallel for-loops Loop iterations become threads Data is shared between threads (i.e. iterations), except loop index Threads wait at the end of the for loop The pragma is specified directly before the loop #pragma omp parallel { #pragma omp for for (i=0;i < N; i++) { function(i); } } The two pragmas above are equivalent to #pragma omp parallel for 22

Sharing control Consider double x, y; #pragma omp parallel for for(i=0;i<N;i++) { x = a[i]*4; y = b[i] * b[i]; b[i] = x/y; } This will probably not give the intended result: x and y are shared between the threads of the parallel for loop 23

The private clause Used to give each thread a private copy of a variable which was already declared outside The variable is uninitialized double x, y; #pragma omp parallel for private(x,y) for(i=0;i<N;i++) { x = a[i]*4; y = b[i] * b[i]; b[i] = x/y; } 24

Variations on sharing control As well as private: firstprivate: initializes each private copy to the value from the master thread lastprivate: copies the value from the thread, which executed the last iteration of the loop, to the master thread shared: is the default, but for documentation or if the default is changed you can uses this clause Plus others, e.g. those which concern threadprivate variables 25

Reductions Reductions will implicitly produce a local copy of the reduction variable in each thread Each thread updates its copy At the end of the construction the reduction operation merges sub-results into a single value and puts it in the reduction variable Operations: + * - ^ & | && || min max #pragma omp for reduction(op:var) 26

Reduction example double dotprod=0; #pragma omp parallel for reduction(+:dotprod) for(i=0;i<N;i++) { dotprod += a[i] * b[i]; } 27

Exercise 4 Use SIMD in the matrix multiplication Starting with the blocked version of the matrix multiplication see if autovectorization has an effect The Makefile already has the matrixmul-simd target. Autovectorization may work or might need a small change Compare the execution time to a similar multiplication not using autovectorization See if you have SIMD instructions (you may use Intel SDE, see next slide) Check the compiler report if it didn t work 28

Intel SDE Is the software development emulator In this case we can use it to count and classify different types of instructions export PATH=/home/gss2018/exercises/sde-external-8.16.0-2018-01-30-lin:$PATH sde -iform 1 -omix test.out -top_blocks 5000 -- ./my_executable Look in test.out for lines ending in _1, _2 or _4 representing scalar or packed operations, e.g. cat test.out | egrep '^\*.*_[124]' 29

Exercise 5 - OpenMP Use OpenMP to make the matrix multiplication from the previous exercise use multiple threads Set OMP_NUM_THREADS=6, 12, 24, 36 and then 48 Run a multiplication and use top to look at the running process. Note that %CPU should be >100%. Compare the runtime each time and Use perf to measure the instructions and cycles; vary the number of threads and note how each changes 30