Reiterating JavaScript & Node.js iterators

Luciano Mammino (@loige)

loige.link/iter

RomaJS - June 16, 2021

@loige

loige.link/iter

How many ways do you know to "do iteration" in JavaScript & Node.js? 😰

@loige

for i / for...in / for...of

@loige

while / do while

@loige

Array.forEach / Array.map / Array.flatMap / Array.reduce / Array.reduceRight  / Array.filter / Array.find / Array.findIndex / Array.entries / Array.values / Array.every / Array.some

@loige

Object.keys / Object.values / Object.entries

@loige

Iterators / Generators

@loige

 Spread operator

[...iterable]

@loige

Events / Streams

@loige

Async iterators / Async generators

@loige

for await...of

@loige

That was 28 different concepts! 😳

@loige

📝 AGENDA

  • Iteration protocols... What? Why?
  • Syntax review
  • Iteration protocols
    • Iterator protocol
    • Iterable protocol
    • Generator functions
    • Async iterator protocol
    • Async iterable protocol
  • Tips & tricks

@loige

Let me introduce myself...

I'm Luciano (🇮🇹🍕🍝) 👋

👨‍💻  Senior Architect

nodejsdp.link

Co-Author of Node.js Design Patterns  👉

Connect with me:
 

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

@loige

We are business focused technologists that deliver.


Accelerated Serverless | AI as a Service | Platform Modernisation

WE ARE HIRING: Do you want to work with us?

@loige

Iteration protocols

what? why? 🤔

@loige

  • An attempt at standardizing "iteration" behaviors providing a consistent and interoperable API
    • Use for...of, for await...of and spread operator.
  • You can also create lazy iterators
  • You can also deal with async iteration
  • You can create your own custom iterators/iterables

Syntax Review 🧐

@loige

const judokas = [
  'Driulis Gonzalez Morales',
  'Ilias Iliadis',
  'Tadahiro Nomura',
  'Anton Geesink',
  'Teddy Riner',
  'Ryoko Tani'
]

for (const judoka of judokas) {
  console.log(judoka)
}

@loige

for...of

Driulis Gonzalez Morales
Ilias Iliadis
Tadahiro Nomura
Anton Geesink
Teddy Riner
Ryoko Tani
OUTPUT
const judoka = 'Ryoko Tani'

for (const char of judoka) {
  console.log(char)
}

@loige

for...of (with strings)

R
y
o
k
o 

T
a
n
i
OUTPUT
const medals = new Set([
  'gold',
  'silver',
  'bronze'
])

for (const medal of medals) {
  console.log(medal)
}

@loige

for...of (with Set)

gold
silver
bronze

 

OUTPUT
const medallists = new Map([
  ['Teddy Riner', 33],
  ['Driulis Gonzalez Morales', 16],
  ['Ryoko Tani', 16],
  ['Ilias Iliadis', 15]
])

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

@loige

for...of (with Map)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals

OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of Object.entries(medallists)) {
  console.log(`${judoka} has won ${medals} medals`)
}

@loige

for...of (with object & Object.entries)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals

OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

@loige

for...of (with object literals)

for (const [judoka, medals] of medallists) {
                              ^
TypeError: medallists is not iterable
  at Object. (.../05-for-of-object.js:8:32)

ERROR
const countdown = [3, 2, 1, 0]

// spread into array
const from5to0 = [5, 4, ...countdown]
console.log(from5to0) // [ 5, 4, 3, 2, 1, 0 ]

// spread function arguments
console.log('countdown data:', ...countdown)
// countdown data: 3 2 1 0

@loige

Spread operator

import {
  DynamoDBClient,
  paginateListTables
} from '@aws-sdk/client-dynamodb'

const client = new DynamoDBClient({});

for await (const page of paginateListTables({ client }, {})) {
  // page.TableNames is an array of table names
  for (const tableName of page.TableNames) {
    console.log(tableName)
  }
}

@loige

for await...of (async iterable)

Iteration Protocols 😵‍💫

@loige

Iterator protocol

In JavaScript, an object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

@loige

function createCountdown (start) {
  let nextVal = start
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }
      return {
        done: false,
        value: nextVal--
      }
    }
  }
}

@loige

Countdown iterator

const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Iterable protocol

An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.

 

* Symbol.iterator

@loige

function createCountdown (start) {
  let nextVal = start
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }
        return {
          done: false, 
          value: nextVal--
        }
      }
    })
  }
}

@loige

Countdown iterable

const countdown = createCountdown(3)
for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

Can an object be both an iterator and an iterable?! 🤨

@loige

const iterableIterator = {
  next() {
    return { done: false, value: "hello" }
  },
  [Symbol.iterator]() {
    return this
  }
}

@loige

Iterator + Iterable

iterableIterator.next()
// { done: false, value: "hello" }

for (const value of iterableIterator) {
 console.log(value)
}

// hello
// hello
// hello
// ...

Generators

A generator function "produces" an object that is both an iterator and an iterable! 🤯

@loige

function * createCountdown (start) {
  for (let i = start; i >= 0; i--) {
    yield i
  }
}

@loige

// As iterable
const countdown = createCountdown(3)
for (const value of countdown) {
 console.log(value)
}
// 3
// 2
// 1
// 0
// As iterator
const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Well, what about async iteration? 🤌

@loige

Async Iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

@loige

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  let nextVal = start
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }
      return { done: false, value: nextVal-- }
    }
  }
}

@loige

const countdown = createAsyncCountdown(3)
console.log(await countdown.next()) // { done: false, value: 3 }
console.log(await countdown.next()) // { done: false, value: 2 }
console.log(await countdown.next()) // { done: false, value: 1 }
console.log(await countdown.next()) // { done: false, value: 0 }
console.log(await countdown.next()) // { done: true }

@loige

Async Iterable protocol

An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.


* Symbol.asyncIterator

@loige

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  return {
    [Symbol.asyncIterator]: function () {
      let nextVal = start
      return {
        async next () {
          await setTimeout(delay)
          if (nextVal < 0) {
            return { done: true }
          }
          return { done: false, value: nextVal-- }
        }
      }
    }
  }
}

@loige

const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}
import { setTimeout } from 'timers/promises'

async function * createAsyncCountdown (start, delay = 1000) {
  for (let i = start; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

@loige

const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}

@loige

When to use async iterators

@loige

  • Sequential iteration pattern

  • Data arriving in order over time

  • You need to complete processing the current “chunk” before you can request the next one

  • Examples

    • paginated iteration

    • consuming tasks from a remote queue

Tips & pitfalls ☠️

@loige

import { createReadStream } from 'fs'

const sourceStream = createReadStream('bigdata.csv')

let bytes = 0
for await (const chunk of sourceStream) {
  bytes += chunk.length
}

console.log(`bigdata.csv: ${bytes} bytes`)

@loige

Node.js readable streams are async iterators

What about
backpressure? 😩

@loige

import { createReadStream } from 'fs'
import { once } from 'events'

const sourceStream = createReadStream('bigdata.csv')
const destStream = new SlowTransform()

for await (const chunk of sourceStream) {
  const canContinue = destStream.write(chunk)
  if (!canContinue) {
    // backpressure, now we stop and we need to wait for drain
    await once(destStream, 'drain')
    // ok now it's safe to resume writing
  }
}

@loige

But if you are dealing with streaming pipelines it's probably easier to use pipeline().

@loige

import { pipeline } from 'stream/promises'
import { createReadStream, createWriteStream } from 'fs'
import { createBrotliCompress } from 'zlib'

const sourceStream = createReadStream('bigdata.csv')
const compress = createBrotliCompress()
const destStream = createWriteStream('bigdata.csv.br')

await pipeline(
  sourceStream,
  compress,
  destStream
)

@loige

In Node.js we can convert any Event Emitter to an Async Iterator! 😱

@loige

import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

@loige

import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

// ⚠️  DANGER, DANGER (high voltage ⚡️): We'll never get here!
console.log('ALL DONE! :)')

@loige

import { on } from 'events'
import glob from 'glob'

const matcher = glob('**/*.js')
const ac = new global.AbortController()

matcher.once('end', () => ac.abort())

try {
  for await (const [filePath] of on(matcher, 'match', { signal: ac.signal })) {
    console.log(`./${filePath}`)
  }
} catch (err) {
  if (!ac.signal.aborted) {
    console.error(err)
    process.exit(1)
  }
  // we ignore the AbortError
}

console.log('NOW WE GETTING HERE! :)') // YAY! 😻

@loige

NOTE:
If you know ahead of time how many events you need to process you can also use a break in the for...await loop.

@loige

LAST TIP:

Can we use async iterators to handle web requests a-la-Deno? 🦕

@loige

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  res.end('hello dear friend')
}

@loige

EASY PEASY LEMON SQUEEZY! 🍋

 

But... wait, aren't we processing all requests in series, now? 😱

@loige

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  // ... AS LONG AS WE DON'T USE await HERE, WE ARE FINE!
}

@loige

import { createServer } from 'http'
import { on } from 'events'
import { setTimeout } from 'timers/promises'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  await setTimeout(1000)
  res.end('hello dear friend')
}

@loige

@loige

Let's stick to the basics... 😅

@loige

import { createServer } from 'http'
import { setTimeout } from 'timers/promises'

const server = createServer(async function (req, res) {
  await setTimeout(1000)
  res.end('hello dear friend')
})

server.listen(8000)

@loige

@loige

Conclusion 🤓

@loige

  • Iterable protocols are a way to standardize iteration in JavaScript and Node.js
  • Async iterators are ergonomic tools for sequential asynchronous iteration
  • But don't use them for everything!
    • Consuming data from paginated APIs or reading messages from a queue are good examples!
    • Handling web requests or events from an emitter might not be the best use cases!

Want to learn more? 👩‍🏫

@loige

If you enjoyed this talk, you might also enjoy nodejsdp.link 😛

@loige

Let's connect:

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

loige.link/iter