Profiling Tool for Trusted Execution Environments

TEE-Perf A Profiler for Trusted Execution Environments Maurice Bailleu, Pramod Bhatotia Donald Dragoti, Christof Fetzer Code available: https://github.com/mbailleu/tee-perf

Trusted Execution Environments Address space Security in untrusted infrastructure: How to establish trust in the untrusted computing infrastructure? Secure memory region (or enclave) Trusted Execution Environment (TEE): Hardware extension to provide secure memory region Protects application code and data against a powerful adversary (e.g. malicious OS/VMs) Trusted application Trusted Execution Environment

Trusted Execution Environments Different implementations: Different ISAs Different OSs A wide range of TEEs available that are supported by different platforms

Performance problems inside TEEs TEE implementation details: Memory encryption overhead Switches between un-/trusted environments Syscalls (I/O operations) are prohibited Different characteristics for different TEEs Code running inside an TEE has surprisingly different performance characteristics

Research gap: Profiling for TEEs TEE environment: No HW counter No I/O OS cannot inspect processor state Architecture or platform dependent Makes it hard to adapt existing profiling tools

Our contribution TEE-perf: An architecture and platform-independent tool to measure performance on function level for application running inside a TEE Properties: Generality Transparency Accuracy Architecture- and platform-independent Unmodified multi-threaded application Easy-to-use interface Accurate method-level profiling No instruction sampling

Outline Motivation Challenges Design Evaluation

Challenge #1: HW counter unavailability Hardware counters Not available inside TEEs Architecture dependent Acquire from the untrusted host Requires switch between un-/trusted environments Mapping a counter into the secure memory

Challenge #2: Application inspection Sampling by interrupting periodically Interrupts are expensive TEEs prevent observing the CPU Use function instrumentation to measure the code while executing

Challenge #3: Getting measurements data Communication over channels Trusted enclave Channels require to leave TEEs I/O call TEE exit operations are expensive since they require TLB flushing, security checks, etc. Exit enclave to issue the syscall Introducing a shared-memory log in the host memory

Challenge #4: Log format Measurement information are not human readable Tools do not understand the format Designed an offline analyzer that allows queries on the measurements and export data to other tools

Outline Motivation Challenges Design Evaluation

System overview #1 #2 #3 #4 Compiler Recorder Analyzer Visualizer

Stage 1: Compiler Compiler takes unmodified code and produces a binary for measurements Inject code Function instrumentation Call/Ret Map code Communication Recorder

Stage 2: Recorder Recorder uses the instrumented binary to measure the execution and writes the profiled info to the shared-memory log Host memory Enclave Fn(A) Fn(B) Recorder Write log Call B Call B Software Counter Write log Ret Ret Log Header Record 1 Record 2

Stage 2: Log format Append-only log allows lock-free appends, and small entries reduce log size Call/ Ret Call/ Ret Counter value Counter value Instruction address Instruction address Thread ID Thread ID Log header Log entry #1 Log entry #2

Stage 3: Analyzer Analyzer takes the log and presents retrieved information to the user Call stack for each thread Calculates time spent per method Human readable Declarative query interface

Stage 4: Visualizer Takes an Analyzer run and produces a Flamegraph An example flame graph produced by TEE-perf

Outline Motivation Challenges Design Evaluation

Evaluation Questions: 1. What are the profiling overheads of TEE-Perf? 2. Does TEE-Perf detect performance optimization opportunities? See the paper for more results Experimental setup: Intel Xeon E3-1270 v5 (3.60 GHz, 4 cores, 8 hyper-threads) -- Skylake w/ SGX 64GiB RAM

Q1 : Overhead of TEE-perf 6 Relative Overhead 5 4 3 2 1 0 matrix_mul word_count linear_reg hist string_mat pca mean TEE-Perf has an mean overhead of 1.9x compared to perf

Q2: Detecting optimization opportunities Case study porting SPDK to Intel SGX: 14.4x slowdown of naively ported version TEE-perf showed that: 72% of the time was spent in getpid syscall 20% of the time was spent in getting a timestamp After optimization SPDK performance is on par with native TEE-perf is able to detect performance critical sections

Summary TEE-perf: An architecture and platform independent profiling tool for trusted execution environments (TEEs) Our tool is General: architecture and platform independent Transparent: supports unmodified multi-threaded applications Accurate: provides method-level profile w/o instruction sampling Code available: https://github.com/mbailleu/tee-perf