Luciano Mammino PRO
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com
Did you know
JavaScript has iterators?
Luciano Mammino (@loige)
A new adventure awaits you!
𝕏 loige
You just joined a new STARTUP.
This is DAY 1!
𝕏 loige
CTO: Can you help me with troubleshooting something?
𝕏 loige
CTO: I am debugging a PRODUCTION issue.
𝕏 loige
CTO: We need to count how many "ERR_SYS_FCKD" there are per customer
𝕏 loige
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5KGHRV3SNJ71C4E920872","error":"ERR_SYS_FCKD"}
{"requestId":"3e3e49b5...","level":"ERROR","timestamp":"2023-05-26T23:18:50.277Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_CLIENT_TIMEOUT"}
{"requestId":"f28abf98...","level":"ERROR","timestamp":"2023-05-26T23:18:56.575Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_BUSY"}
{"requestId":"608e579f...","level":"ERROR","timestamp":"2023-05-26T23:19:04.529Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_NOT_FOUND"}
{"requestId":"3125588e...","level":"ERROR","timestamp":"2023-05-26T23:19:11.514Z","customerId":"01H1D5MJ4XNJ580DMM8Y7T1HRW","error":"ERR_SERVER_TIMEOUT"}
{"requestId":"40fb0329...","level":"INFO","timestamp":"2023-05-26T23:19:13.536Z","customerId":"01H1D5N6HS2EMA900A2V6NQQ2Q","message":"user logged in"}
{"requestId":"564afe31...","level":"ERROR","timestamp":"2023-05-26T23:19:21.708Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_CLIENT_TIMEOUT"}𝕏 loige
Let's do it!
𝕏 loige
import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)𝕏 loige
Errors by customer:
{
'01H1D5NGA5D689HD57CN5BZSG3': 109,
'01H1D5MZAVPSXSXPTYADF4BDND': 118,
'01H1D5KGHRV3SNJ71C4E920872': 118,
'01H1D5KSG0K81CCAGTSTBXHFA6': 95,
'01H1D5MJ4XNJ580DMM8Y7T1HRW': 103,
'01H1D5N6HS2EMA900A2V6NQQ2Q': 96,
'01H1D5M5RTKKK5SC4ADWAPK7Q0': 130,
'01H1D5K2JDVD941R6H67YEY0G8': 115,
'01H1D5KZ7GV7HK213XE3WV5GQX': 113
}𝕏 loige
CTO: Great job! Exactly what I needed!
𝕏 loige
DAY 2
𝕏 loige
CTO: that script it's now giving "RangeError: Invalid string length"!
𝕏 loige
import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)𝕏 loige
import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)We are loading everything into memory...
If the file is too big this will fail!
𝕏 loige
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
//{
}𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
}OK
Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}NOPE
Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}OK
Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}OK
Ideal approach
Lazy processing, line by line
𝕏 loige
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 2,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
𝕏 loige
We can do this using Iterators!
𝕏 loige
Player 1
I'm Luciano
Senior Architect @ fourTheorem (Dublin)
Co-Author of Node.js Design Patterns
Always re-imagining
We are a pioneering technology consultancy focused on Cloud, AWS & serverless
We can help with:
Cloud Migrations
Training & Cloud enablement
Building serverless applications
Cutting cloud costs
𝕏 loige
Why Iterators?
-
Lazy abstraction to represent repetition
-
You can consume the collection 1 item at the time
-
Great for large (or endless) datasets
𝕏 loige
The concept of Iterators exists already in JavaScript...
Good news...
... since ES2015!
𝕏 loige
const array = ['foo', 'bar', 'baz']
for (const item of array) {
console.log(item)
}
Prints all the items in the array!
Does it need to be an array? 🤔
Output:
foo
bar
baz
𝕏 loige
const str = 'foo'
for (const item of str) {
console.log(item)
}Output:
f
o
o
𝕏 loige
const obj = {
foo: 'bar',
baz: 'qux'
}
for (const item of obj) {
console.log(item)
}Output: ⛔️ Uncaught TypeError: obj is not iterable
OMG `for ... of`
does not work with plain objects! 😱
𝕏 loige
𝕏 loige
const array = ['foo', 'bar', 'baz']
console.log(...array)Output:
foo bar baz
spread syntax!
𝕏 loige
𝕏 loige
𝕏 loige
Generators
𝕏 loige
Generator fn & obj
function * myGenerator () {
// generator body
yield 'someValue'
// ... do more stuff
}const genObj = myGenerator()
genObj.next() // -> { done: false, value: 'someValue' }𝕏 loige
function * fruitGen () {
yield '🍑'
yield '🍉'
yield '🍋'
yield '🥭'
}
const fruitGenObj = fruitGen()
console.log(fruitGenObj.next()) // { value: '🍑', done: false }
console.log(fruitGenObj.next()) // { value: '🍉', done: false }
console.log(fruitGenObj.next()) // { value: '🍋', done: false }
console.log(fruitGenObj.next()) // { value: '🥭', done: false }
console.log(fruitGenObj.next()) // { value: undefined, done: true }𝕏 loige
function * fruitGen () {
yield '🍑'
yield '🍉'
yield '🍋'
yield '🥭'
}
const fruitGenObj = fruitGen()
// generator objects are iterable!
for (const fruit of fruitGenObj) {
console.log(fruit)
}
// 🍑
// 🍉
// 🍋
// 🥭𝕏 loige
Iterators & Iterables
𝕏 loige
Iterator Obj
An object that acts like a cursor to iterate over blocks of data sequentially
𝕏 loige
Iterable Obj
An object that provides data that can be iterated over sequentially
𝕏 loige
The iterator protocol
Iterator object:
next() method returns:
-
done (boolean)
-
value (any)
const iterator = {
next () {
return {
done: false,
value: "someValue"
}
}
}𝕏 loige
function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}A factory function that creates an iterator
returns an object
... which has a next() method
... which returns an object with
done & value
𝕏 loige
const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true }𝕏 loige
function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}Generator objects implement the iterator protocol
function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}Equivalent code using generators
𝕏 loige
The iterable protocol
Iterable object:
Symbol.iterator() method returns:
-
an iterator
const iterable = {
[Symbol.iterator] () {
return {
// iterator
next () {
return {
done: false,
value: "someValue"
}
}
}
}
}𝕏 loige
function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}𝕏 loige
[...createCountdown(3)] // [3,2,1,0]
for (const value of createCountdown(3)) {
console.log(value)
}
// 3
// 2
// 1
// 0We can use spread and for ... of with iterable objects!
𝕏 loige
function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
})
}
}Generator objects implement the iterable protocol
function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}Equivalent code using generators
YES, it's the same code as the iterator example!
𝕏 loige
const iterableIterator = {
next () {
return { done: false, value: 'hello' }
},
[Symbol.iterator] () {
return this
}
}An object can be both an iterator and an iterable!
Iterator protocol
Iterable protocol
𝕏 loige
ASYNC
Iterators & Iterables
𝕏 loige
The async iterator protocol
Async Iterator object:
next() method returns a
Promise that resolves to:
-
done (boolean)
-
value (any)
const asyncIterator = {
async next () {
return {
done: false,
value: "someValue"
}
}
}𝕏 loige
The async iterable protocol
Async Iterable object:
Symbol.asyncIterator() method returns:
-
an async iterator
const asyncIterable = {
[Symbol.asyncIterator] () {
return {
// iterator
async next () {
return {
done: false,
value: "someValue"
}
}
}
}
}𝕏 loige
import { setTimeout } from 'node:timers/promises'
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}Async Generator objects implement the async iterator & async iterable protocols
𝕏 loige
const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value)
}
We can use for await ... of with async iterable objects!
𝕏 loige
Let's rewrite our log analyser to use iterators!
𝕏 loige
import { createReadStream } from 'node:fs'
const readable = createReadStream(
'logs.jsonl',
{ encoding: 'utf-8' }
)
// a readable stream is an Async Iterable!
for await (const chunk of readable) {
// do something with a chunk of data
}We can read a file incrementally using Node.js streams!
𝕏 loige
import { createReadStream } from 'node:fs'
const readable = createReadStream(
'logs.jsonl',
{ encoding: 'utf-8' }
)
// a readable stream is an Async Iterable!
for await (const chunk of readable) {
// do something with a chunk of data
}We can read a file incrementally using Node.js streams!
A chunk is an arbitrary amount of data (not necessarily a line)
𝕏 loige
// utils/byline.js
export async function * byLine (asyncIterable) {
let remainder = ''
for await (const chunk of asyncIterable) {
const lines = (remainder + chunk).split(/\n+/)
remainder = lines.pop()
yield * lines
}
if (remainder.length > 0) {
yield remainder
}
}
𝕏 loige
import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'
const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })
const errorsByCustomer = {}
for await (const line of byLine(readable)) {
const message = line === '' ? {} : JSON.parse(line)
if (message.error === 'ERR_SYS_FCKD') {
if (!errorsByCustomer[message.customerId]) {
errorsByCustomer[message.customerId] = 0
}
errorsByCustomer[message.customerId]++
}
}
console.log('Errors by customer:')
console.log(errorsByCustomer)
𝕏 loige
import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'
const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })
const errorsByCustomer = {}
for await (const line of byLine(readable)) {
const message = line === '' ? {} : JSON.parse(line)
if (message.error === 'ERR_SYS_FCKD') {
if (!errorsByCustomer[message.customerId]) {
errorsByCustomer[message.customerId] = 0
}
errorsByCustomer[message.customerId]++
}
}
console.log('Errors by customer:')
console.log(errorsByCustomer)
CTO: It works with
any file now!
𝕏 loige
Can't we use .map(), .filter(), .reduce()?
-
you can wait for Iterator Helpers & Async Iterator Helpers TC39 proposals
-
... or you can use core-js to polyfill these features today!
𝕏 loige
𝕏 loige
if you need to read a file line by line, there's another option:
BONUS Material
SLIDES: loige.link/iter-garda
Code: loige.link/iter-garda-code
𝕏 loige
Original Front cover photo by Jörg Angeli on Unsplash
Original Background photo by Michael Behrens on Unsplash
Grazie!
𝕏 loige
Did you know JavaScript has iterators? - Node.js Conference Italy
By Luciano Mammino
Did you know JavaScript has iterators? - Node.js Conference Italy
How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for…in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!
