Lazy Looping: The Next Iteration

In this talk we’ll learn about the properties of iterators, learn how to create our own iterators with generators, and take a look at how iterators and generators allow us to write our looping code in a fundamentally different way.

Embracing Python’s iterators and generators often allow us to drastically restructure our code for the better. They allow for writing code more like a recipe of descriptive steps, which often improves code readability.

We’ll start off this talk by reviewing iterables, iterators, and generators. Then we’ll take a detailed look at a number of interesting and useful leverage Python’s fancy iterators and generators for good.

We’ll learn about iterators and generators and discuss making your own. We’ll also discuss best practices around using and creating iterable helper functions that are iterator-aware.

Attendees will walk away from this session with specific actionable recommendations for refactoring their own code as well as a wealth of new topics to look deeper into after the session.

Outline

Intro & Review (4 minutes)

• Iterables are anything you can loop over
• Iterables are powered by iterators
• Iterators are also iterables
• The iterator protocol boiled down

Objectives (1 minute)

• Create iterables
• Create iterators
• Use iterables and iterators to clean up our code

Creating iterables and iterators (6 minutes)

• Creating iterables: first attempt
  • Create an object with a __iter__
  • The __iter__ must return an iterator… how do we do that?
• Creating iterators
  • Classes with a __iter__ and a __next__
  • The __iter__ returns self and __next__ returns the next item
• Creating iterables: revisited: create an iterator class and then an iterable class
• Iterator properties
  • Iterators “consume” items as they loop
  • Iterators get “exhausted”: Iterators cannot be looped over more than once

But wait… there’s a better way! (5 minutes)

• Generator functions are an easy way to create an iterator
• Generator functions are bizarre: yield magically turns a function into a generator function
• Generator expressions are a short cut for making a generator object that does a simple map and/or filter
• If you understand how iterators work from the outside, you understand how iterators work

Working with iterators (2 minutes)

• itertools: there are some lovely lazy iteration tools in here!
• Other libraries of alternatives
  • boltons
  • more-itertools

Creating our own iteration helpers (3 minutes)

• Looking at itertools recipes and at the source code for boltons and more-itertools can help us write better iteration tools
• Common techniques
  • Grab an iterator from an iterable and work with that for the rest of your function
  • Use next to grab just a few things (or use islice)
  • Use a couple loops over the same iterator (or use takewhile/dropwhile/etc.)
• Creating iterators
  • Classes with a __iter__ and a __next__
  • The __iter__ returns self and __next__ returns the next item
• Creating iterables: revisited: create an iterator class and then an iterable class

Laziness: it’s not just about iterators (2 minutes)

• You can make iterables that compute lazily but are not iterators
• Example: range objects are not iterators
• Try not to assume something is an iterator (or a non-iterator)

Using iteration helpers to write more elegant code (4 minutes)

• Using with_previous/with_next/window/pairwise-like helpers to remove “side cars”
• Using any/all instead of early break/return
• Using dropwhile/takewhile instead of early break
• Using next instead of an early return/break
• Using islice instead of manual next
• Iterators allow for more descriptive code

Conclusion (1 minute)

• Iterables give iterators and iterators give each item individually
• Iterators are the backbone of looping in Python
• Iterators also empower lazy iteration and working directly with iterators can allow you to write more flexible code
• Iterators allow you to write your code in a different way

Why Me?

I teach iterators, iterables, and lazy looping concepts very frequently. I’ve written articles and given talks on Python’s iterators and I’d like to now do a talk that showcases why how to leverage lazy looping to improve code quality.