QoE Evaluation and Enforcement Framework for Internet Services

ITU Workshop on Performance, QoS and QoE of Emerging Networks and Services Athens, Greece, September 2015 QoE Evaluation and Enforcement Framework for Internet Services Marcus Eckert, Thomas M. Knoll Research Assistant, TU-Chemnitz marcus.eckert@etit.tu-chemnitz.de

Contents Motivation Architecture Overview Architecture Components QMON QoE Monitoring QRULE QoE Policy and Rules QEN QoE Enforcement Results Summary Page 2

Motivation Experienced quality of Internet services is crucial for customer satisfaction QoE monitoring and enforcement is thus required for business success Aim: application specific differentiated handling of traffic flows for major Internet services 3GPP standard based procedures using dedicated bearers are hardly used today Default bearer differentiated flow handling is missing Improved QoE measurement and enforcement framework required ISAAR Framework (ISAAR = Internet Service quality Assessment and Automatic Reaction) ISAAR augments existing QoS functions by flow based network centric QoE monitoring and enforcement functions Page 3

Architecture Overview Modular service specific QoE management architecture 3 functional components: QoE Monitoring (QMON) flow detection and measurement, QoE Rules (QRULE) policy rules and permission checking and QoE Enforcement (QEN) respective flow manipulation Interworking with existing QoS mechanisms 3GPP PCC Priority marking (DiffServ, Ethernet prio, MPLS prio) Proprietary router QoS support (queueing, scheduling, shaping) SDN based QoS support (e.g. through OpenFlow Action Sets) Page 4

Architecture Overview Page 5

Architecture Components QMON QMON operation Flow classification With and without DPI Centralized / distributed With SDN match and action rules Flow capturing SDN support to tee out flows Flow Monitoring Application specific KPI calculation Standardization: G.102y QMON output Flow Information and QoE estimation currently implemented: Video QoE estimation Page 6

Measurement Procedure Measurement at end device Most precise Firmware / device specific User involvement Tampering possible Measurement within operator network User and device independent End device (buffer) model required estimation Reliable results at scale Challenges: constant changes in video streaming (encoding + media container formats) MPEG DASH Page 7

Measurement Procedure Flow Detection and Classification Deep Packet Inspection (DPI) Built-in or external using 3GPP PCC Gx interface Video Quality Measurement TCP = reliable transport no longer fine grained pixel and block structure errors Video stall events, duration and inter-stall timing is important Buffer fill level estimation as measurement result Page 8

Measurement Procedure Buffer fill level estimation (exact method) Difference between TCP segment timestamp and playout time encoded in video data within the segment Each segment is traced and processed Use TCP ACKs to increase precision (segment loss, RTT measurement) Buffer model required for fill level estimation (initial buffering, re-buffering and play-out thresholds) Buffer depletion / stall events, re-buffering times and inter-stall timing as measurement results Page 9

Measurement Improvements (speed & accuracy) Buffer fill level estimation (estimation method) Speed-up by chunk based throughput like measurement to avoid decoding every packet ( jump through the stream method) Full header decoding needed and limited to suitable formats e.g. MP4 MP4 Header Chunk 1 Chunk 2 Chunk 3 ... Chunk i Chunk i+1 Variable look-up interval trade off between processing speed-up and accuracy Loss of fill level estimation precision especially when high delays or even losses occur due to congestion Page 10

Measurement Improvements (speed & accuracy) Buffer fill level estimation (combined method) Automatic switching between exact and estimation mode of operation to gain the speed-up during good times and to keep the precision during bad networking conditions. There is hardly any difference between the exact and the combined method result. However, the processing load increases (speed-up decreases) for bad case video streaming conditions Page 11

MOS calculation for Video QoE Mean Opinion Score (MOS) derived from P.862 (ITU-T: Perceptual evaluation of speech quality = pesq) 0 = worst quality / 4.5 = highest quality Assumption: initial 4.5 decreased by negative impact (NI) factor Initial buffering (e.g. 10s) does not raise NI Each following stall decreases the MOS; follows an e-function (exponential function) 5 4.5 4 3.5 3 MOS 2.5 2 1.5 1 0.5 ??? = ? ? 0 5+1,5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 stalling events where x = # of stall events Page 12

MOS calculation for Video QoE 5 4.5 4 3.5 3 MOS 2.5 2 1.5 1 0.5 0 105 112 119 126 133 140 147 0 7 14 21 28 35 42 49 56 63 70 77 84 91 98 time Example video with 5 stalling events Resulting in a bad MOS value Page 13

MOS calculation for Video QoE Memory effect also influences the negative impact of each stall (D1) as well and dampens the impact the longer the play-out has run smoothly before ??? = ? ? ? = ?????? ?? ?????? ? = ???? ????? ???? ????? ? = ?????? ????????? 5+1,5 ? ?? 5 4.5 4 3.5 3 MOS 2.5 2 1.5 1 0.5 0 102 108 114 120 126 132 138 144 150 0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 time Page 14

Architecture Components QRULE QRULE input Flow information and corresponding QoE estimation from QMON QRULE operation Mapping input flow to service flow classes Check whether QoE enhancement is allowed by general operator policy Determine Per Flow Behaviour (PFB) based on the Enforcement Database of QEN QRULE output PFB specific commands for QEN for 3GPP PCC triggering and/or marking, shaping, dropping and even (SDN/LSP) path selection Page 15

Architecture Components QRULE / PFB Determination Example: PFB commands for marking rules and SDN flow based path selection in high contention situations Page 16

Architecture Components QRULE / PFB Example LTE Test 720p 70 60 50 BE/LE marking buffered video time in s low priority 40 normal priority 30 CS5 marking 20 Buffer Level Threshold 1 Threshold 2 high priority 10 EF or equivalent class marking 0 0 50 100 150 200 packet time in s Page 17

Architecture Components QEN QEN input Flow information and QRULE action command set QEN operation Register enforcement capabilities Execute flow manipulation via: 3GPP PCC (PCRF / PCEF) IETF & IEEE priority marking Automated router configuration with vendor specific QoS capabilities and settings SDN capabilities for marking and Traffic Engineering (TE) Granularity: per-flow or per-class PFB & class PHB / flow & class TE Page 18

Architecture Components QEN QEN flow manipulation options 3GPP PCC (PCRF / PCEF) QCI marking and/or dedicated bearer setup IETF & IEEE priority marking IP Diffserv, Ethernet priority, MPLS traffic class priority without the need to change the configuration of network elements Synchronized inside/outside GTP tunnel & IPSec tunnel marking Automated router configuration with vendor specific capabilities and settings Cisco / Juniper specific router configuration with flow-specific rules for scheduling, shaping, dropping as well as path (LSP) selection SDN capabilities for marking and TE marking via OpenFlow switch action list configuration flow-specific traffic engineering (LSP selection or flow-specific forwarding paths) Page 19

Results Demonstrator setup additionally to the field trials Field trials using packet traces at SGi interface of an operator Demonstrator Lab setup using 2 Laptops Online Buffer fill level estimation and MOS calculation Page 20

estimation interval stepping re- Results Results for exact vs. estimation vs. combined method of operation processing time Both algorithms - good case video - 6 s 10 packets 3 s 50 packets 3 s 100 packets 3 s 150 packets 3 s 250 packets 3 s Estimation algorithm - bad case video Human - Exact 12 s 10 packets 6 s 50 packets 6 s 100 packets 6 s 150 packets 6 s 250 packets 5 s Combined algorithm - bad case video Human - exact 12 s 10 packets 8 s 50 packets 8 s 100 packets 8 s 150 packets 8 s 250 packets 7 s # re-buffering events buffering time Human Exact 0 0 0 0 0 0 0 0 s 0 s 0 s 0 s 0 s 0 s 0 s 10 10 10 10 10 9 6 58 s 56,6 s 56,0 s 54,4 s 53,7 s 51,3 s 49,1 s 10 10 10 10 10 10 10 58 s 56,6 s 56,6 s 56,6 s 56,6 s 56,6 s 56,6 s Page 21

Results Results for exact vs. combined method of operation Good case video Bad case video Page 22

Results Results for buffer fill level to MOS QoE calculation Each stall is sharply impacting the experienced quality (MOS score) Re-buffering times gradually impact the MOS score Periods of smooth play-out lead to slight MOS recovery Full recovery is possible if only a few stalls occur and a long smooth play-out follows 5 4.5 4 3.5 3 MOS 2.5 2 1.5 1 0.5 0 105 112 119 126 133 140 147 0 7 14 21 28 35 42 49 56 63 70 77 84 91 98 time Page 23

Summary ISAAR addresses QoE management for Internet based services 3 components QMON, QRULE, QEN to monitor and manipulate flows Location aware service flow observation and steering Automated network based QoE estimation is feasible and produces accurate results in terms of MOS score calculations and underlying buffer fill level estimations. Aware of 3GPP standardized PCC Interworking with PCRF/PCEF 3GPP interfaces are supported (Sd, UD/Sp, Rx, Gx/Gxx) ISAAR is also able to work independently of 3GPP QoS functionality Page 24