Exploring JavaScript Web Application Behavior

JSMeter: Characterizing the Behavior of JavaScript Web Applications Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research, Redmond in collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1

One Word: Standard for scripting web applications Fast JITs widely available JavaScript Lots of code present in all major web sites Support in every browser 2

ZDNet 29 May 2008 Browser Wars! ghacks.net Dec. 2008 4

Artificial Benchmarks versus Real World Sites 7 V8 programs: 8 SunSpider programs: 11 real sites: JSMeter richards 3-draytrace deltablue access-nbody crypto bitops-nsieve raytrace controlflow Maps Maps Goals of this Talk earley- boyer Show behavior of JavaScript benchmarks versus real sites crypto-aes date-xparb regexp math-cordic splay Consider how benchmarks can mislead design decisions string-tagcloud 5

Instrumentation Framework Source-level instrumentation \ie\jscript\*.cpp custom jscript.dll website visits custom trace files 100% 90% Offline analyzers 80% 70% 60% 50% Constant 40% Other Str Ops 30% Concat Op 20% 10% 0% custom trace files 6

Visiting the Real Sites Getting past page load performance Attempted to use each site in normal way: amazon bing bingmap cnn ebay economist facebook gmail google googlemap Search for a direction from one city to another hotmail Search a book, add to shopping cart, sign in, and sign out Type in a search query and also look for images and news Search for a direction from one city to another Read front page news Search for a notebook, bid, sing in, and sign out Read front page news, view comments Log in, visit a friend pages, browse through photos and comments Sign in, check inbox, delete a mail, and sign out Type in a search query and also look for images and news Sign in, check inbox, delete a mail, and sign out 7

Understanding JavaScript Behavior Code JavaScript Events Objects 8

Code Behavior Function size Instructions/call Code locality Code Instruction mix JavaScript Events Objects 9

Code|Objects|Events Total Bytes of JavaScript Source 2500 2000 Source size (kilobytes) Source size (kilobytes) 1500 1000 500 0 amazon earley cnn facebook richards string-tagcloud access-nbody bing gmail googlemap regexp splay date-xparb 3d-raytrace crypto-aes bitops-nsieve google hotmail deltablue crypto bingmap ebay controlflow regexp-dna math-cordic economist raytrace 10 Real Sites V8 SunSpider

Code|Objects|Events Static Unique Functions Executed 4,000 3,500 3,000 Unique Functions Executed Unique Functions Executed 2,500 2,000 1,500 1,000 500 0 amazon bingmap crypto facebook access-nbody gmail bing ebay regexp googlemap deltablue date-xparb regexp-dna cnn raytrace splay string-tagcloud bitops-nsieve crypto-aes richards 3d-raytrace economist hotmail earley google controlflow math-cordic Real Sites V8 SunSpider 12

Code|Objects|Events Bytecodes / Call 300.00 function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) 250.00 a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a 200.00 Bytecodes / Call Bytecodes / Call } 150.00 100.00 50.00 0.00 amazon earley cnn facebook access-nbody bing gmail googlemap splay date-xparb string-tagcloud richards regexp 3d-raytrace crypto-aes bitops-nsieve google hotmail crypto ebay controlflow deltablue bingmap Real Sites V8 SunSpider regexp-dna economist raytrace math-cordic 15

Code|Objects|Events Fraction of Code Executed 100.00% 90.00% 80.00% Fraction of Code Executed Fraction of Code Executed Most code not executed 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% amazon cnn richards access-nbody facebook earley splay string-tagcloud bing gmail googlemap 3d-raytrace date-xparb regexp bitops-nsieve hotmail deltablue crypto bingmap ebay controlflow economist google crypto-aes regexp-dna math-cordic raytrace Real Sites V8 SunSpider 16

Code|Objects|Events Hot Function Distribution 80% of time in 100+ functions 80% of time in < 10 functions 80% 80% Real Sites V8 Benchmarks 17

Code|Objects|Events Opcode Distribution Outliers Real Apps Green = SunSpider | Blue= Real Web Apps | Red = V8 18

Object Allocation Behavior Allocation by types Lifetime distribution Code Live heap composition JavaScript Events Objects 19

Code|Objects|Events Total Bytes Allocated 45000 40000 Total heap data (kilobytes) Total heap data (kilobytes) 35000 30000 25000 20000 15000 10000 5000 0 amazon earley access-nbody cnn facebook richards splay string-tagcloud 3d-raytrace bing gmail googlemap date-xparb regexp bitops-nsieve crypto bingmap google hotmail deltablue ebay controlflow economist crypto-aes regexp-dna math-cordic raytrace Real Sites V8 SunSpider 20

Code|Objects|Events Few benchmarks allocate much data Heap Data by Type 1 0.9 0.8 Heap Data by Type 0.7 Objects Enumarator 0.6 RegExp 0.5 Math 0.4 Error Date 0.3 Native Func 0.2 String 0.1 Arrays 0 Script Func splay cnn gmail amazon hotmail richards V8 aggragate math-cordic googlemap bitops-nsieve aggregate ebay google bing facebook regexp-dna string-tagcloud deltablue crypto raytrace earley crypto-aes regexp economist access-nbody controlflow date-xparb bingmap 3d-raytrace Many functions Rest are strings Real Sites V8 SunSpider 21

Code|Objects|Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 23

Code|Objects|Events Live Heap over Time (ebay) Heaps repeatedly created, discarded Heap contains mostly functions Heap drops to 0 on page load 24

Code|Objects|Events 2 Search Websites, 2 Architectures You stay on the same page during your entire visit Code loaded once Heap is bigger Every transition loads a new page Code loaded repeatedly Heap is smaller Bing Google (Web 1.0) (Web 2.0) 25

Event Handlers in JavaScript Number of events Sizes of handlers Code JavaScript Events Objects 29

Code|Objects|Events Event-driven Programming Model Single-threaded, non-preemptive event handlers Example handlers: onabort, onclick, etc. Very different from batch processing of benchmarks Handler responsiveness critical to user experience 30

Code|Objects|Events Total Events Handled 7,000 6,000 5,000 Total Events Handled Total Events Handled 4,000 3,000 Almost no events 2,000 1,000 0 Real Sites V8 31

Code|Objects|Events Median Bytecodes / Event Handled 2137 506 500 450 Median Bytecodes / Event Handled Median Bytecodes / Event Handled 400 350 300 250 200 150 100 50 0 33

Cold-code Experiment Observation Real web apps have lots of code (much of it cold) Benchmarks do not Question: What happens if the benchmarks have more code? We added extra, unused to code to 6 SunSpider benchmarks We measured the impact on the benchmark performance 37

Performance Impact of Cold Code Without cold code, Chrome12x faster than IE8 With 2M cold code, Chrome 4.7x faster than IE8 Which comparison is more meaningful? 900 900 800 800 700 700 600 600 Time (msec) Time (msec) 0K 0K 500 500 200K 200K 400 400 400K 400K 300 300 800K 800K 1M 1M 200 200 2M 2M 100 100 0 0 Chrome IE 8 Cold code makes SunSpider on Chrome up to 4.5x slower Cold code has non-uniform impact on execution time 38

Impact of Benchmarks What gets emphasis Making tight loops fast Optimizing small amounts of code Important issues ignored Garbage collection (especially of strings) Managing large amounts of code Optimizing event handling Considering JavaScript context between page loads 39

Conclusions JSmeter is an instrumentation framework Used to measure and compare JavaScript applications High-level views of behavior promote understanding Benchmarks differ significantly from real sites Misleads designers, skews implementations Next steps Develop and promote better benchmarks Design and evaluate better JavaScript runtimes Promote better performance tools for JavaScript developers 40

Conclusions JSmeter is an instrumentation framework Used to measure and compare JavaScript applications High-level views of behavior promote understanding Behavior in benchmarks significantly differs from real sites Misleads designers, skews implementations Next steps Develop and promote better benchmarks Design and evaluate better JavaScript runtimes Promote better performance tools for JavaScript developers 41

Additional Resources Project: http://research.microsoft.com/en-us/projects/jsmeter/ Video: Project JSMeter: JavaScript Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn Papers: JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications , Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps 10), June 2010. "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009. 42

Additional Slides 43

Related Work JavaScript An Analysis of the Dynamic Behavior of JavaScript Programs , Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek, PLDI 2010, June 2010/ C# and Java Doufour C++ Calder et al. 44