Setup
Architecture
Meet The Team
Engineering
Build Or Not
Conclusions
We serve mobile advertising
Match bid offers with campaigns
Real time bidding (OpenRTB)
Make bid and compose ad
In near real time (~80 ms)
We help keep the app ecosystem alive
50+ countries
Peaks of 800 krps
Average of 500 krps
45 billion requests per day
35 million daily impressions
Filter out 90% of traffic
Decides which requests are less attractive
Written in Erlang (!)
Rewritten in Go (!!)
Saved about 80% of costs
Holds the grunt of the 300 krps with about 10 servers
It's an operational miracle
Decides the price to pay for each bid
Intensive computations
Predictive algorithm:
estimates CTR of bid offer
Written in Go (!)
Handles queries in ~500 µs (2 ms)
Discard traffic fast
Watch what you do online
Take work off hot paths
Optimize hot paths
Never commit to anything!
Keep your options open
Isolate external accesses
Maintain interfaces
Colleagues all around the world
Constant contact on Slack
Frequent team meetings
Screen sharing essential
Priorities on Mondays
Improve code
Disseminate knowledge
Increase sense of ownership
Build your team
Educate
Brave to try new things
Reward initiatives
Mistakes happen
Blame-free
Warn operations and sales
Warn customers ASAP
Explain everything, blame nobody
Immediately and thoroughly
Issue recommendations (and follow through)
Amateurs talk about tactics,
but professionals study logistics.
Gen. Robert H. Barrow, USMC
Paraphrasing:
Amateurs worry about programming,
but professionals care about deployment.
Probably the most important practice in DevOps
Continuous Integration: tested on integration env
Continuous Delivery: tested on several envs
Continuous Deployment: go all the way and deploy!
A simple bash script to start
Built for (and with) Node.js packages
Held together with some custom Node.js scripts
Recently
migrated to StriderCD
Now with a Graphical Interface!
Starting a green field project can be exhilarating
Helps attract and keep elusive talent
Someone had to build first!
Why let them have all the fun?
Companion article:
"Not Invented Here":
Distrust products offered by third parties
If it is a core business function
do it yourself, no matter what.
As easy as identifying core business functions...
which is easy, right?
Banks: insourcing IT
Google: self-driving cars
Facebook: providing Internet access
Apple: music distribution, watches
There is not a universal path for all destinations
Consider:
Previously using Amazon's Elastic Load Balancer
Eating up to 20% of our monthly bill
Replaced by an Nginx balancer
DNS balancer in Node.js
Custom Lua logging
You will not get it right the first time
Make sure you can improve it later
Devise strategies to migrate
Avoid dead-ends
Not just install a Jenkins instance somewhere
Merge development and sysadmining
Write code to do fun stuff
Build upon your work
If it gives you a competitive advantage
build it yourself, no matter what.