Video summary
This video provides a comprehensive summary of iterators, contrasting them primarily with materialized list objects through practical examples. The speaker illustrates that while a standard Python list stores every integer element individually in memory—creating an object proportional to its size—a `range` object acts as a specialized iterator known here as "memoryless." Unlike lists which consume increasing amounts of RAM based on the number of elements, a range object only holds start and stop parameters. When iterating over a list, Python retrieves references to existing numbers already in memory, whereas looping through a range generates each number sequentially one by one, effectively reusing the same variable space for every iteration without growing its footprint.
The tutorial further explores how these concepts apply to built-in functions like `sort`, `sorted`, and `reversed`. Calling `.sort()` on a list modifies it in place because lists are mutable, while the `sorted()` function always returns a new materialized list regardless of whether the input is a standard list or an infinite range. A critical distinction highlighted between iterators (like ranges) and generators is that generators can be exhausted after being iterated once; they cannot restart from the beginning without re-creation. In contrast, range objects are unique because they never exhaust in the traditional sense—if you loop over them multiple times, each iteration starts fresh from the initial value to the end, making them ideal for scenarios requiring repeated access or infinite sequences that fit within a defined step pattern.
The video concludes by demonstrating other powerful iterator tools such as `zip`, which combines two iterables into tuples of corresponding elements until the shortest one is exhausted, and `enumerate`, which adds index numbers to an iterable's items starting from a default value of zero. The speaker emphasizes that Python utilizes iterators extensively throughout its language, often implicitly in functions like map or filter, but understanding them explicitly prepares developers for handling large datasets and streaming data where memory constraints are significant. Ultimately, mastering these iterator patterns allows programmers to work efficiently with big data without needing to load entire datasets into computer memory at once, a crucial skill for advanced applications in data science and software development.
Read the full video transcript
in this video i'm going to summarize a
couple of things we have seen throughout
this course
in the context of the iterator
discussion
so let's go ahead and
call the file simply iterator examples
so let's um first of all
create a list of numbers as we always
use for example purposes
it's going to be the list 7 11
8 5 and 3 and then 12
2 and 6 9 10
and 1 4.
so that is a list of numbers so in terms
of memory consumption
this is going to create a list object
and 12 integer objects and the list has
12 references to these 12 different
integer objects so
that is what i would call a materialized
object so that is not a
technical term that you will find in
literature but i use it often
to make apparent the difference between
something that does not occupy memory
versus something that does
so a list object occupies memory
on the contrary if i want to model the
same numbers
with a different order let's say in a
sorted order what we could do instead is
we could introduce let's call it a
variable let's call it memoryless
because it should indicate that the
following object has basically no memory
and let's simply use a range object and
let's go from
1 through 13 which is modeling the same
numbers
so both numbers and memory less
basically models the same numbers of
course in different order and sometimes
order is important but i'm disregarding
the order here
so my claim is that the range object
is also something is a special
um it is a special kind of an iterator
so actually range is a very special
object there is also a link in the notes
wire range is super special a super
special case but for now let's treat it
as if it were an iterator
so let's open python tutor
and compare the two in terms of memory
so let's first create a numbers list
and then second let's create this
variable called memoryless
which is a range object so let's go
ahead
and first create numbers and then second
we create memoryless
so i call it memoryless because no
matter how much
how big numbers or how big of a sequence
of numbers we are modeling
let's say this is not one but let's say
one billion and
or let's say this is one and this is 13
billion so we have let's say
modeling many many different numbers the
range object in memory is not going to
grow
the only thing that is different is the
parameters that is stored on top of the
range object however if i want to model
let's say 10 billion numbers using a
list object all of them
have to be in memory okay so therefore a
list object really
grows in terms of memory consumption
proportionally to the number of elements
it has
the range object does not grow so
the using the range object we can model
any sequence of numbers as long as of
course we can express it as a sequence
using a step size you all know
you all remember how the range building
works but let's assume we can model a
series of numbers using the range
building
then using the range build in basically
takes no memory because we don't have to
store any numbers
so what is going to happen if we loop
over both of them
so if i loop for example by saying for
number in numbers
let's say print number and
let's print that again on one line
what happens if i do that in memory and
also what happens
if i do the very same thing here but i'm
going to use instead of numbers i'm
going to use
memoryless
so if we go ahead and let's go ahead and
put
the for loop right here and let's
put the seconds for loop
down here what we are going to see
is simply that when we iterate over the
list object we
obtain a reference to the list to the
numbers that are already there
okay but we don't have a second number
here actually but really we get back a
reference
to one of the numbers in every iteration
of the for loop
okay and now we get to the end of the
first for loop
and now comes the second object called
memoryless and what happens in memory
when i loop over it well
it pulls out the numbers on a one by one
basis as we've seen it overwrites the
the number singular variable
so it number is going to be one two
three and so on
and we simply see how we are going to
print that out but the important idea is
that the range object also produces the
numbers on a one by one basis
therefore i think of it as a general as
a iterator or kind of like a generator
it generates a number of numbers on a
one by one basis
so a range object whenever you can use
range object instead of a list with the
actual numbers in it
well you should prefer the range object
because it models the same numbers
but at all points in time it only has
one number in memory
simultaneously and then it pulls out the
next number and overwrites
the space from the first number from the
previous number
okay so in terms of looping both the
list object
and the range object work exactly the
same
however the range object does not need
to have all the numbers in memory
simultaneously
okay so let's look at a couple of
further built-ins we have seen
in throughout this course so let's
review the topic
of sorting so you all remember
in the discussion on list methods there
the list
every list object has a method called
sort which could sort
the which could sort the elements
in place so let's go ahead and do that
so i'm going to say numbers.sort
and remember i remember that
the list object is mutable and this is
indicated by the fact that we don't see
a return value here
so i just executed the code cell but we
don't see anything below the cell
if we look at numbers we see now numbers
is sorted right
okay so this is one way of sorting a
list
and this happens in place so this does
not create a second list object
so what are other ways of sorting well
remember we have
in the in the python documentation
let's go here under built in functions a
function called
sorted right here and it says it takes
any iterable
and it's going to return a new list with
that is sorted okay so let's do that
let's
use the sorted function and it's passed
to its numbers
and now we see that we have output below
the code cell indicating that we get
back a new cell
and a new list object in this situation
with
the same same order because we
previously sorted the list in place
but now we get back a second list object
where the numbers are also sorted
if we wanted to reverse it we could do
so by passing a flag called reverse
equals true
and we get back a list object with the
numbers
in reverse sort order and now if we look
at numbers
numbers is of course still in the
previously sorted order because
the sorted function does not touch the
underlying numbers list
so what happens if i use the sorted
built-in
on the range object so what if i use it
on the memoryless range object
well i get back a list object okay and
that is
something that we should expect from
reading the documentation the
documentation says
return a new sorted list so no matter
what we pass it in
we always get back a new list object so
therefore
what this basically does here this
basically materializes the list so this
is equivalent
of calling the list constructor
and passing it memory less this would
result in the same
outcome okay of course if you want to
reverse it
you can also do that right here using
the reverse flag
and now we get back a new list where the
numbers are in reverse
sort order okay however note that
memoryless
is still only a range object without so
it's still a rule that can generate
the numbers one by one so one thing that
i want to mention that is so special
about the range
built in is that we have seen in
previous videos
that a generator will be exhausted
after some time so let's go ahead and
compare that to generators so comparison
with generators
let's first create a generator let's
simply call it squares
and let's derive it with a generator
expression so using the parentheses
notation
and let's say n squared
or n in numbers
okay and i mistyped squares so it should
be
u first of course
so if i look at squares it is a
generator object
and if i pull out let's say using the
next built-in function
one number i get back the number one and
now if i pull out all the remaining
numbers
all the remaining squares for example
using their list constructor
what's going to happen is i get all the
numbers except the first because
the first number was already pulled out
and you remember that
a generator cannot go back it can only
go forward
however if we use the range built in so
let's say if i write
memoryless again here
and let's say i want to materialize the
numbers in memory less
i get it back if i want to do that a
second time
it works i can pull out the same numbers
as often as i want
if however i go ahead and
try to pull out one further number from
the squares
i will get back an empty list why
because the squares generator has
already been exhausted
so that is a big difference between uh
that is a thing that makes the range of
that
object so special the range object is
never exhausted
so whenever you loop over it you loop
over it from the beginning to the end
and then it simply when when it reaches
the end
the loop stops but when you loop over
the same range object the second time
it starts at the beginning again so the
range object is kind of special
okay but uh no need to really care about
that too much
that is just something i want to mention
here
okay so what else is there so
now we saw sorting and
we had did a little comparison so let's
go ahead and also
look at the built-in function called
reversed
so reversed
okay so let's go to the documentation
one more time
and let's look what the documentation
says for reversed it says
the argument has to be a sequence so it
has to fulfill all the four properties
that we uh want to remember for
sequences especially the one that it has
in order
uh without an order it doesn't make
sense to do that and also a sequence has
to have an end so there has to be a
finite number of things otherwise you
couldn't reverse it if
if it's an infinite stream of data you
couldn't reverse it
and then it says return a reverse
iterator
okay so in previous videos when we
talked about the four behaviors of a
sequence
the fourth property of them was the
reversible property and i said
to you that whenever an object supports
being passed to
the reverse built in just like numbers
so for example the numbers list
then um the object is considered
reversible
so ordered in in easy terms so what do
we get back here when we use the reverse
built-in
well we get back an iterator okay and in
earlier videos when we
use the reverse built-in you could use
it
so the way we used it all the time was
with the
with a for loop so i would say for
number in reversed numbers
print number
this is what you saw before so this is
how we used reverse before in this
course
but now we now understand after chapter
eight
that what an iterator is so what in
other words what the reverse built in
gives us back
is an iterator that is a rule in memory
that simply goes
in backward order okay so just like um
in the previous video
where i compared the iterators versus
the intervals in general
we saw how i can get a list iterator
using the iterator built in function
from an from a list that is a rule that
goes in forward fashion
now i can simply go in backwards fashion
so maybe let's compare the two
so it's a call of the one thing here
forward iterator
forward iterator can be constructed by
using the iter built in
and we pass it in iterable so numbers
for example
and this here is going to be the reverse
iterator and now what we can do is
we can go ahead and we can use the next
function
on the forward iterator and we get the
first number
if i use the next function on the
reverse iterator
i get the last number and so on if i do
it a second time i get the number two
and if i do it the second time here i
get the number eleven right
so this is also something that is um not
so trivial actually to understand
so note how when i switch between this
this cell here and i execute that and
this cell
what i'm really doing is i'm looping
over the same numbers list in parallel
twice okay so i'm looping in forward and
backward fashion at the same time
and this doesn't matter because the
numbers list is not the thing that does
the iteration
the thing that does the iteration is the
iterator that manages the iteration
and by using the build and enter
function we get a forward iterator
by using the built-in but by using the
reverse built in
we are going the backward way okay so
that's it but other than that it's the
same concept
okay so now you know what reversed
really does so now you're not confused
anymore
to see um the return value of simply
calling the reverse build in just like
that
because before chapter eight this really
didn't make sense what you're seeing
here but now it does
so this is just a rule that knows how to
do something without having it done yet
okay so um that is
um basically um yeah all of the the
things i want to to summarize here
so this video um is more of a summary um
probably
um that compares a couple of ideas that
you have seen throughout this course
and um puts them together into a like
kind of a bigger picture
okay so note how uh before this chapter
you always have to materialize all the
data in your computer's memory
that is okay to do when you're learning
when you're a beginner
but at some point when you go into data
science and you want to work with big
amounts of data
that maybe not that hardly fit into your
computer's memory working memory
then you need to come up with some
strategies to deal with that
and the way to do that is by first
understanding the concept of an iterator
how are different kinds of iterators so
understanding map filter generator and a
couple of others
and yeah there are actually maybe i can
show you a couple of more iterators that
are built in
so one other thing that we have seen
before
is the zipper so let's say i have a list
called numbers right and let's say i
have a second list called names
and let's put in the list the names
let's make it easy let's say ahem
bathhold again and of course also cesar
and let's say you want to loop over the
numbers list and the names list
simultaneously in parallel then we saw
the sip built in right
so if you give discipled in numbers as
the first argument
and names as the second argument what
you get back is a sip object
so here it does not say iterator but the
sip object really is an iterator
so maybe let's call that or store that
in a variable called zipper
and with the zipper object we can also
pass it to next
and this will give us back a tuple as we
see a tuple of a number
and a name right and if i do that a
second time it's i get back two and
better if i do it the third time i get
back three and ceasar
and if i do it one more time what will i
see well yes of course a stop iteration
exception
because the generator or the iterator is
at its end
it is exhausted okay another example we
have seen before
in this course is the enumerate
or enumerating intervals that's how i
would call it
so let's assume we have an iterable like
names and i want to enumerate it so
enumerate
basically means i want to give every
every element in this interval an index
number how can i do that
well i'm going to use the enumerate
built in
and i simply give it names and enumerate
gives me back as we see
some object of type enumerate so what is
that well let's call it
enumerate tor and
enumerator if i pass it to the next
function
i get back zero and ahim okay next time
i get back one and bat hold and then two
in caesar
but if i want to mimic the same as above
what i should do
is i should provide the enumerate build
in a start value of let's say start
equals one
and then i get back one ahem two beth
hold three caesar and then stop
iteration exception so what we learned
from that is
python uses iterators all over the place
okay
and oftentimes you use them without
knowing
and if you want to use them
explicitly probably the most important
one to know
is the generator expression and also
it's uh
it's um yeah let's say it's close cousin
uh the generator function
and um which creates a generator object
and a generator object is probably
the most flexible iterator that we have
but
a python uses iterators all over the
place um
in in its language everywhere okay so
that is a summary video
so um i hope now with all these examples
you understand the difference between an
iterator and interval
and iterator really prepares you to work
with big uh
big amounts of data with streaming data
with data that is infinite
okay and that is really what you want to
do if you want to found the next unicorn
the next super startup
then you must be ready to work with big
data
so i will see you in the next video