Where's the HTML5?
This is all HTML/CSS/JavaScript.
Many mobile apps are also hybrid native / HTML5.
Never: Uptime of 30+ days
Expected to be self-healing (auto-restart on failure)
HTML & JS hosted locally on device
Native plug-ins for many operations
Network operations relegated to background
Web engine built into the app / device (typically WebKit)
More control
Able to customize
Robust & mature test strategy
Targeting 1,000,000+ devices
Failure not an option
Web sites
Mobile apps
The question isn't whether or not you need to harden your site / apps...
Practices - foreign to most web developers
Platforms - maturing but not quite there yet
Libraries - completely clueless
Tools - focused on spot debugging
A robust test discipline is critical.
“Long term stability” testing
“Stress” testing
“Soak” testing
1 in 10,000 scenarios
Performance degradation (Inconsistency, Sluggishness)
Abnormal behavior
Unhandled exceptions
Data handling
Lost functionality
Reboots / browser resets
Corrupt state
Ensure clean environment before each test
Reset cookies and local storage
Restart browser and device
Test result database: By device, By build
Statistics matter for performance measurements
7+ samples (min, max, mean)
Comparable data sets
Comparable run-time environments
Comparable server-side environment
Comparable network environment
Purpose: Simulate normal usage over an extended period of time over a large number of instances
# of units: as many as possible (50+ units)
Methodology: Normal usage scenarios run continuously over a period of 1-2 days
Target: No noticeable degradation
Purpose: Simulate a long run-time by accelerating the usage time
# of units: as many as possible (50+)
Methodology: Automated scripts with a key/click rate as fast as possible --> 1 day test can simulate 15-30 days of usage
Target: No noticeable degradation
Purpose: Find latent bugs that only show up over days or weeks
# of units: small(er) number (10-20)
Methodology: Well-defined set of tests run once or twice a day for as many days as possible
Target: 30 days uptime with no degradation in functionality or performance
Test environments
Test automation tools
Know your browser
Brute-force debugging techniques
Racks of physical hardware
Amazon AWS
Multiple browser instances
Proprietary
IR blasters
Help me out - what do you use?
WebKit Web Inspector
Chrome Debugger
GC collector stats
Memory stats
DOM tree stats
Firebug
Console logs
Etc.
Challenge: Figure out how to automate data collection across large # of instances!
Garbage collector memory pressure
Hard upper limit? GC bogs down
No upper limit? Unexpected memory growth
Cache memory pressure
Graphics cache
Fix them (duh) - Brute force
You’ll miss some bugs (duh) - Resilient application architecture(s)
DOM leaks
Timer storms
Dangling callbacks
Exceptions
Beware of your libraries!
DOM leaks - Walk DOM tree, dump # to console
Timer storms - Wrap all timer calls with helpers you can instrument later
Dangling callbacks - Wrapper for all callbacks
Exceptions - window.onerror(), automatic iframe reset
Break large apps (50K lines of code) into a number of small micro-apps
Run each micro app only when needed
Pick a sandboxing architecture (native, HTML5)
Separate web context with web intents for communication
Automatically kills an app that is misbehaving
CPU hog
Memory hog
Automatically restart an app that is mission-critical
App prioritization
Sand-boxing
Process separation
Storage separation (cookies, IndexDB, web storage)
Not always available
No help for problems in big apps (173K lines of JavaScript)
Designed to protect you from others...but why not from yourself?
Negligible memory impact (Chrome about:memory)
Also used for secure data sandboxing: Devdatta Akhawe
Okay to restart whole app
Native app framework does not support automatic restart
Small amount of management code in root page
Whole app wrapped in iframe
Delete / re-create iframe on a schedule (daily), during idle time, or when app misbehavior detected
<script> function createFrame() { var f = document.createElement("iframe"); f.name = "myapp"; f.id = "myapp"; f.sandbox = "allow-scripts allow-forms allow-same-origin"; f.src = "myapp.html"; var d = document.getElementById("app-div"); d.appendChild(f); } function removeFrame(frame) { var node = document.getElementById(frame); node.parentNode.removeChild(node); } </script>
<div id="app-div"></div>
Not okay to restart whole app
Very large apps
High data / network usage on start-up
More resilient user experience
Seamless recovery from bugs with only minor user disruption
More complex
More restrictions
Core management logic in root page
Segment different parts of your app into "micro-apps"
Each micro-app wrapped in iframe
Communication between root page and micro-apps via postMessage()
Most micro-apps started / stopped on-demand
Some micro-apps run longer – restart when misbehavior detected
Duplicate files loaded into each iframe
Extra work to proxy communication across iframe boundaries
With a little work your libraries can be iframe sandbox friendly!
PureMVC ExampleCredits: Aaron Gascoigne
Test the heck out of your apps
Watch out for the key bugs
Break large apps into micro-apps
Manage app / micro-app lifetime
Use native app sandboxes and/or iframe sandboxes
Modify your frameworks to be iframe sandbox friendly
Code on GitHub