chugging flask
Gyaan from building REST APIs for our Android App
PYCON 2013
Note: Some slides navigate up/down as well for details & resources
Aravind Krishnaswamy
Co-Founder, Levitum
@twitortat
WHAT WE BUILT
REST APIs for Android App
~200,000 active users
Backend for Singing Game
NOW IN INVITE ONLY BETA
Our Stack
Nginx, Gunicorn, Gevent, Flask,
CouchDB, Redis,
NewRelic, Sentry, Loggly, Fabric, Jenkins
WHAT THIS PRESENTATION ISNT ABOUT
- Using Flask
- assumes familiarity with similar platforms
-
REST API
- Building REST API with Flask
- plenty of online material, refer to Flask-Restless
what this presentation is about
Open dialog about best practices.
Sharing battle hardened experience.
Architectural and design choices made.
Tips and tricks on what worked well and didn't.
One size doesn't fit all, your mileage may vary.
Yes, it's mostly lots of random gyaan.
BEGIN wITH TDD/BDD
Use Flask-Testing
Setup continuos builds with Jenkins
Wire CI test results to Jenkins
Test CRUD API separately
Use Blitz.io to test performance of views
Build meaningful test data sets
P.S: Hit the down arrow for more resources
sample blitz usage
def callback(result):
for point in result.timeline:
print("[")
print("total:"+ str(point.total))
print(", errors: " + str(point.errors))
print(", hits: " + str(point.hits))
print(", steps: " + str(len(point.steps)))
for step in point.steps:
print("\tstep duration: " + str(step.duration))
print("]\n");
options = {
'steps':[
{'url': "http://api.singr.in/spam"},
{'url': "http://api.singr.in/eggs"}
],
'pattern': { 'intervals': [{'start':1, 'end':50, 'duration':30}]}}
r = Rush("your@account.com", "aqbcdge-sjfkgurti-sjdhgft-skdiues")
r.execute(options, callback)
Simply wire this up to JenkinsAnd you'll be running continuos performance tests
USE HEROKU
- Flask setup on Heroku is a breeze
- Consider Heroku for all dev & test instances
-
Automate instance setup (single dynos are free)
- Custom buildpacks extend instances (eg ffmpeg)
-
Fork postgres datasets, fork instances to EU zone.
- No APAC zone yet, so latencies are high.
- No websockets support yet.
- Use NewRelic for monitoring
- Helps avoid dyno idling on dev servers
- Purchase dynos for prod instances
P.S: Hit the down arrow for more resources
Resources
Env var config for Heroku:
small is big
- Break the app into little decoupled services
- Let services work over HTTP, exchange JSON
- Easier to refactor independent services if needed
- Or rewrite in a new language -
- for instance, nodejs for async stuff
- Use Blueprints where patterns emerge
- Werkzeug Middlewares are useful
- Use delayed jobs queues with Redis & RQ
- Considered Celery, but RQ met needs
P.S: Hit the down arrow for more resources
basic EXAMPLE 1 - GAME Backend
- Authentication & Authorization (Flask + Flask-OAuth)
- Real time Pub-Sub Channel (Nodejs + Express)
- Can be rewritten in Flask + Gevent-SocketIO
- Quiz Engine (Nodejs + Express)
- Audio Processor (Built with Flask + pydub)
- Can be scaled and tuned separately for long running requests
basic EXAMPLE 2 - Saas shopfront
Create a 'blueprint' for the shopfront APIs
/about
/store
'Register' the blueprint with each shop URL
/about -> /maiyas/about
/store -> /maiyas/about
/about -> /mtr/about
/store -> /mtr/about
USE Redis
Flask + Redis is awesome
Use it instead of memcache*
Use it for server side sessions
Use it for delayed jobs
Use it for pub sub
Use its powerful data structures
Use it for 'instant' search
You got it. Use Redis as much as you can.
* Contrary to frequent misconceptions, Redis can persist data and is not just in memory.
P.S: Hit the down arrow for more resources
Leverage decorators
Have all your API views return objects
Use a decorator to spit out views (JSON/HTML/XML)
Minimal boilerplate
Incredibly convenient with Flask
Use a decorator for etags, jsonp, etc.
P.S: Hit the down arrow for more resources
example - etags
def support_etags(f):
""" Decorator for handling etags """
@wraps(f)
def decorated(*args, **kwargs):
if_none_match = request.headers.get('If-None-Match', False)
foo = f(*args, **kwargs)
try:
spit(foo)
hash_value = crypto.hash(str(foo.data))
except (TypeError, AttributeError):
# Bails if the returned data is not a string type do to an exception
return foo
# Ok, now that we have a hash, lets see if it matches up fine
if if_none_match and crypto.match(if_none_match, hash_value):
# And return a not modified code if it hasnt
return not_modified()
# And if it was, stamp in the hash value
foo.headers['etag'] = str(crypto.hash(str(foo.data)))
return foo
return decorated
GRequests
Gevent + Requests
Makes wiring multi API mashups a breeze
Leverages your decoupled little services
Parallelism made easy.
For instance, if you need a mashup like:
User Profile API = User API + User Likes + User Comments + User Assets + User's Friends
P.S: Hit the down arrow for more resources
EXAMPLE
(Source: https://github.com/kennethreitz/grequests)
import grequests
urls = [
'http://www.heroku.com',
'http://python-tablib.org',
'http://httpbin.org',
'http://python-requests.org',
'http://kennethreitz.com'
]
>>> rs = (grequests.get(u) for u in urls)
>>> grequests.map(rs)
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
Flask + COUCHDB
Nothing like talking http to your DB
Flask-CouchDB is good
Flask-CouchDBKit is awesome
G-Spot lies between large docs & lots of relations
Separate concerns, use a Couchapp
Preload test data sets for new instances by replication
Both Cloudant and IrisCouch are excellent
Logging & Debugging
Pipe *everything* to Loggly, archive to S3.
App, web, system logs included.
Configure nginx to log API service times.
Compare with NewRelic data.
Sentry > Exceptional for exception tracking.
Use SendGrid for worker job notifications.
Flask extensions
- Flask-WTF great for form validation
- Flask-Cache cache invalidation is not solid
- Flask-CouchDBKit is awesome
- Flask-Heroku handy heroku config presets
- Flask-SSLify https everything
- Flask-Testing light extn for tests with nose
- Flask-Split basic A/B testing
- Flask-Uploads
- Flask-SQLAlchemy
- Flask-Script more on this in a bit
FLASK-SCRIPT
- Primarily intended for management commands
- Has unintended team management benefits too
- One group handles REST CRUD
- Another builds out view mashups
- Done in isolation outside of request context.
- Can be used as delayed jobs
- Or wired up with view wrappers
QUICK RECAP & tips
Flask docs are terrific. RTFM.
Lots of help on #pocoo.
Not all popular extensions are listed.
Not all listed extensions are stable.
Write small decoupled services.
Never hesitate to just rewrite them.
Dont optimise early.
Automate early, often.
cue & eh
Aravind Krishnaswamy
@twitortat
Co-Founder, Levitum
We're hiring! Awesome pythonistas & mobile devs
P.S: Thanks to Vinu, Jay, Kiran, and Noufal for their help with the slides.