How to send gzipped requests
with boto3

Luciano Mammino (@loige)

AWS UserGroup Dublin
2021-07-01

Get these slides!

Let me introduce myself first...

I'm Luciano (🇮🇹🍕🍝) 👋

Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨‍💻

Co-Author of Node.js Design Patterns  👉

Connect with me:
 

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

We are business focused technologists that deliver.


Accelerated Serverless | AI as a Service | Platform Modernisation

We are hiring: do you want to work with us?

In the previous episodes...

SLIC WATCH:              fth.link/slic-watch

SLIDES:      fth.link/o11y-no-pain

⚠️   ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)

Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).

@wake_me_up_bot

This guy was failing... a lot!

for event in payload['logEvents']:
  # ...
  cw_client.put_metric_data(...)

Sending 1 metric per log line... 🙈

[
    {
        'MetricName': 'SomeMetric1',
        'Dimensions': [
            {
                'Name': 'Dimension1Name',
                'Value': 'Dimension1Value'
            },
            # Up to other 9 dimensions here ...
        ],
        'Unit': 'Count',
        'Values': [217, 220, 221], # Up to 150 items here ...
        'Timestamp': 1624290910000
    },
    # Up to other 19 metric items here ...
]

⚠️ Payload size
limit: 40 KB

🤔

Maybe boto3 gzips the data automatically! 😏

import boto3
import gzip

endpoint_url = "http://localhost:8000/"

cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)

cw_client.put_metric_data(
    MetricData = [
        {
            'MetricName': 'TEST_BOTO',
            'Dimensions': [
                {
                    'Name': 'APP_VERSION',
                    'Value': '1.0'
                },
            ],
            'Unit': 'None',
            'Value': 17
        },
    ],
    Namespace='BotoTest'
)

... and there is no magic flag like GzipPayload=True! 😭

Can we extend boto3 somehow? 🤔

boto3 has an event system! 🤩

import boto3

lambda_client = boto3.client('lambda')

# our event handler
def add_xtrace_header(request, **kwargs):
    request.headers.add_header('x-trace-id', 'trace-trace')

# get the event system for the lambda_client
event_system = lambda_client.meta.events

# attach an event handler to the client for 
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)

# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')

Event naming convention

<event-type>.<service-name>.<operation-name>
provide-client-params
request-created
before-sign
before-send
response-received
s3
cloudwatch
lambda
...
ListBuckets
PutMetricData
Invoke
...

You can be a 🌟 too!

* (every event)

after-call.*.* (all responses)

after-call.lambda.* (all responses for lambda)

 

OK, now we know enough to write our own event handler for gzip compression! 🤓

import boto3
import gzip

cw_client = boto3.client('cloudwatch')
event_system = cw_client.meta.events

# Gzip handler
def gzip_request_body(request, **kwargs):
    request.headers.add_header('Content-Encoding', 'gzip')
    gzipped_body = gzip.compress(request.body)
    request.data = gzipped_body

# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData', 
                      gzip_request_body)

cw_client.put_metric_data(...)

Other potential use cases

  • Log data before/after it is sent to AWS
  • Add additional headers to AWS requests (tracing)
  • Additional input validation (your custom rules)
  • Payload enrichment (e.g. make sure certain tags exist)

What if you don't use Python?

What did we learn?

  • Always have alarms for your lambdas!
    (have you checked SLIC Watch already? 😉)
  • You can use Gzip compression to send big payloads to AWS
  • Boto3 is extensible through its event system!

Do you think boto3 should make it easier to send gzipped payloads?

👍 UPVOTE this issue: github.com/boto/botocore/issues/2425

Do you want more details?

Cover picturea by Moritz Mentges on Unsplash

Thanks doggie by Stefanie Shank

How to send gzipped requests with boto3

By Luciano Mammino

How to send gzipped requests with boto3

Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximise the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!

  • 3,125