Breakpad/socorro symbol usage

WTF SYMBOLS

• Map addresses to functions:
• xul.dll@0xABC123 -> gfxContext::PushClipsToDT(...)
• Stack unwind info:
• "how do I get from PushClipsToDT to its caller"
• Text files, format documented on Breakpad wiki
  

Symbol storage

• Flat files on disk
• Special directory structure:

•  <debug filename>/<debug identifier>/<debug filename>.sym

• e.g.,
  

   firefox.pdb/C48C38C9E37E4A689D57C7509EEB017A2/firefox.sym

• 4TB NetApp volume, accessed via NFS

•  Filesystem            Size  Used Avail Use% Mounted on
  x.x.x.x:/symbols   5.5T  4.3T  1.3T  78% /mnt/netapp/breakpad
  
  

SYMBOL UPLOAD

• SCP to symbolpush.mozilla.org (externally accessible)
• Unzip zip file full of symbols + symbols.txt index file
• Build machines
• one set of symbols per Nightly (per branch, platform), release
• Third parties
• Linux distros
• B2G partners
• Adobe (Flash player symbols)
• Windows system symbols
• In-progress: symbol upload REST API
  

Windows Symbols

• Symbols for Windows system libraries
• Retrieved from Microsoft's symbol server
  
• fetch-win32-symbols
• Runs on symbolfetch1.dmz.phx1
• Fed by modulelist cron currently
• https://crash-analysis.mozilla.com/crash_analysis/modulelist/
  

Symbol Retention

• Python cleanup script
• Keep 30 nightly builds per branch
• Currently keep releases forever
• Horrible, scrapes symbols.txt files
• Storage usage is an ongoing problem
  

FUTURE WORK

• Saner symbol storage
  
• object store?
• God anything please get us off of NetApp
• Make everything use symbol upload API
  
• Symbol caching
• Only 106GB of symbols used per day
  
• Switch to binary-format symbols
• supported in upstream Breakpad
• much faster processing
• Missing symbol alerting