JavaScript Behavior in Web Applications with JSMeter

JSMeter: Characterizing the Behavior of JavaScript Web Applications Paruj Ratanaworabhan, Ben Livshits, and Ben Zorn In collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1

Why Measure JavaScript? Standardized, de facto language for the web Support in every browser, much existing code Major Microsoft investment Web Application Companion based on Script# Benchmarks are driving browser marketing: browser wars Are benchmarks representative? Limited understanding of its behavior in real sites Who cares? Users, web site designers, JavaScript engine developers 2

ZDNet 29 May 2008 ghacks.net Dec. 2008 3

Artificial Benchmarks versus Real World Sites JSMeter 7 V8 programs: 8 SunSpider programs: 11 real sites richards 3-draytrace deltablue access-nbody crypto bitops-nsieve raytrace controlflow Maps Maps Goals of this Talk earley- boyer crypto-aes date-xparb regexp Show behavior of JavaScript benchmarks versus real sites math-cordic splay string-tagcloud Consider how benchmarks can mislead design decisions 4

Instrumentation Framework Source-level instrumentation \ie\jscript\*.cpp custom jscript.dll website visits custom trace files 100% 90% Offline analyzers 80% 70% 60% 50% Constant 40% Other Str Ops 30% Concat Op 20% 10% 0% custom trace files 5

Visiting the Real Sites Attempted to use each site in normal way: amazon bing bingmap cnn ebay economist facebook gmail google googlemap Search for a direction from one city to another hotmail Search a book, add to shopping cart, sign in, and sign out Type in a search query and also look for images and news Search for a direction from one city to another Read front page news Search for a notebook, bid, sing in, and sign out Read front page news, view comments Log in, visit a friend pages, browse through photos and comments Sign in, check inbox, delete a mail, and sign out Type in a search query and also look for images and news Sign in, check inbox, delete a mail, and sign out 6

Understanding JavaScript Behavior Code JavaScript Events Objects 7

Code Behavior Function size Instructions/call Code locality Code Instruction mix JavaScript Events Objects 8

Code|Objects|Events Total Bytes of JavaScript Source 2500 2000 Source size (kilobytes) Source size (kilobytes) 1500 1000 500 0 amazon earley cnn facebook richards string-tagcloud access-nbody bing gmail googlemap regexp splay date-xparb 3d-raytrace crypto-aes bitops-nsieve google hotmail deltablue crypto bingmap ebay controlflow regexp-dna math-cordic economist raytrace 9 Real Sites V8 SunSpider

Code|Objects|Events Static Unique Functions Executed 4,000 3,500 3,000 Unique Functions Executed Unique Functions Executed 2,500 2,000 1,500 1,000 500 0 amazon bingmap crypto facebook access-nbody gmail bing ebay regexp googlemap deltablue date-xparb regexp-dna cnn raytrace splay string-tagcloud bitops-nsieve crypto-aes richards 3d-raytrace economist hotmail earley google controlflow math-cordic Real Sites V8 SunSpider 11

Code|Objects|Events Total Bytecodes Executed 90395 30000 (thousands) Excuted (thousands) 25000 20000 Bytecodes Excuted 15000 Total Bytecodes 10000 Total 5000 0 amazon bing earley access-nbody cnn facebook richards splay string-tagcloud gmail 3d-raytrace googlemap date-xparb regexp bitops-nsieve crypto google hotmail deltablue bingmap ebay controlflow economist crypto-aes regexp-dna math-cordic raytrace Real Sites V8 SunSpider 12

Code|Objects|Events Bytecodes / Call 300.00 function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) 250.00 a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a 200.00 Bytecodes / Call Bytecodes / Call } 150.00 100.00 50.00 0.00 amazon earley cnn facebook access-nbody bing gmail googlemap splay date-xparb string-tagcloud richards regexp 3d-raytrace crypto-aes bitops-nsieve google hotmail crypto ebay controlflow deltablue bingmap Real Sites V8 SunSpider regexp-dna economist raytrace math-cordic 14

Code|Objects|Events Fraction of Code Executed 100.00% 90.00% 80.00% Fraction of Code Executed Fraction of Code Executed Most code not executed 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% amazon cnn richards access-nbody facebook earley splay string-tagcloud bing gmail googlemap 3d-raytrace date-xparb regexp bitops-nsieve hotmail deltablue crypto bingmap ebay controlflow economist google crypto-aes regexp-dna math-cordic raytrace Real Sites V8 SunSpider 15

Code|Objects|Events Hot Function Distribution 80% of time in 100+ functions 80% of time in < 10 functions 80% 80% Real Sites V8 Benchmarks 16

Code|Objects|Events Opcode Distribution Outliers Real Apps Green = SunSpider | Blue= Real Web Apps | Red = V8 17

Object Allocation Behavior Allocation by types Lifetime distribution Code Live heap composition JavaScript Events Objects 18

Code|Objects|Events Total Bytes Allocated 45000 40000 Total heap data (kilobytes) Total heap data (kilobytes) 35000 30000 25000 20000 15000 10000 5000 0 amazon earley access-nbody cnn facebook richards splay string-tagcloud 3d-raytrace bing gmail googlemap date-xparb regexp bitops-nsieve crypto bingmap google hotmail deltablue ebay controlflow economist crypto-aes regexp-dna math-cordic raytrace Real Sites V8 SunSpider 19

Code|Objects|Events Few benchmarks allocate much data Heap Data by Type 1 0.9 0.8 Heap Data by Type 0.7 Objects Enumarator 0.6 RegExp 0.5 Math 0.4 Error Date 0.3 Native Func 0.2 String 0.1 Arrays 0 Script Func splay cnn gmail amazon hotmail richards V8 aggragate math-cordic googlemap bitops-nsieve aggregate ebay google bing facebook regexp-dna string-tagcloud deltablue crypto raytrace earley crypto-aes regexp economist access-nbody controlflow date-xparb bingmap 3d-raytrace Many functions Rest are strings Real Sites V8 SunSpider 20

Code|Objects|Events Object Type Distribution Real Apps economist is an outlier (arrays) 21

Code|Objects|Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 22

Code|Objects|Events Live Heap over Time (ebay) Heaps repeatedly created, discarded Heap contains mostly functions Heap drops to 0 on page load 23

Code|Objects|Events Two Search Websites Bing Google (Web 1.0) (Web 2.0) 24

Code|Objects|Events Two Mapping Websites Google Maps Bing Maps 25

Code|Objects|Events Object Lifetimes (V8) Lives forever Lives until program ends Dies immediately splay earley 26

Code|Objects|Events Object Lifetimes (gmail) Many objects live a major fraction of page lifetime Functions are longest lived This is as old as an object can be Strings are shortest lived 27

Event Handlers in JavaScript Number of events Sizes of handlers Code JavaScript Events Objects 28

Code|Objects|Events Event-driven Programming Model Single-threaded, non-preemptive event handlers Example of some standard event-handler attributes: onabort, onclick, and onmouseover Very different from batch processing of benchmarks Handler responsiveness critical to user experience 29

Code|Objects|Events Total Events Handled 7,000 6,000 5,000 Total Events Handled Total Events Handled 4,000 3,000 Almost no events 2,000 1,000 0 Real Sites V8 30

Code|Objects|Events Median Bytecodes / Event Handled 500 450 Median Bytecodes / Event Handled Median Bytecodes / Event Handled 400 350 300 250 200 150 100 50 0 32

Code|Objects|Events Maximum Bytecodes / Event Handled 1,200,000 Maximum Bytecodes / Event Handled Maximum Bytecodes / Event Handled 1,000,000 800,000 600,000 400,000 200,000 0 33

Code|Objects|Events Distribution of Time in Handlers 35

A Simple Experiment Observation Real web apps have lots of code (much of it cold) Benchmarks do not Question: What happens if the benchmarks have more code? We added extra, unused to code to 6 SunSpider benchmarks We measured the impact on the benchmark performance 36

Performance Impact of Cold Code Without cold code, Chrome12x faster than IE8 With 2M cold code, Chrome 4.7x faster than IE8 Which comparison is more meaningful? 900 900 800 800 700 700 600 600 Time (msec) Time (msec) 0K 0K 500 500 200K 200K 400 400 400K 400K 300 300 800K 800K 1M 1M 200 200 2M 2M 100 100 0 0 Chrome IE 8 Cold code makes SunSpider on Chrome up to 4.5x slower Cold code has non-uniform impact on execution time 37

Impact of Benchmarks What gets emphasis Making tight loops fast Optimizing small amounts of code Important issues ignored Garbage collection (especially of strings) Managing large amounts of code Optimizing event handling Considering JavaScript context between page loads 38

Conclusions JSmeter is an instrumentation framework Used to measure and compare JavaScript applications High-level views of behavior promote understanding Behavior in benchmarks significantly differs from real sites Misleads designers, skews implementations Next steps Develop and promote better benchmarks Design and evaluate better JavaScript runtimes Promote better performance tools for JavaScript developers 39

Additional Resources Project http://research.microsoft.com/en-us/projects/jsmeter/ Papers "JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages) "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages) An Analysis of the Dynamic Behavior of JavaScript Programs , Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010) 40