Video summary
In this video, the presenter revisits a previous task involving a list of numbers where each number must be transformed by squaring it and adding one, followed by filtering out odd results and summing the remainder. The primary goal is to implement this workflow in a memory-efficient manner without creating intermediate lists that store all transformed or filtered data at once. To achieve this, the presenter introduces Python's built-in `map` function, which accepts a transformation rule defined as a helper function and an iterable of numbers. Instead of immediately calculating every result, `map` returns a `map` object that acts as an iterator; it essentially stores a "rule in memory" that knows how to calculate the next value on demand rather than holding all values simultaneously.
The concept extends similarly to the filtering step using Python's built-in `filter` function. A helper function named `is_even` is defined to check if a number is divisible by two, and `filter` uses this rule to create another iterator object. When chained together, these iterators form a pipeline where the `filter` object requests the next item from the `map` object, which in turn fetches the next raw number from the original list. This process ensures that data is pulled through the transformation and filtering stages one by one, preventing the creation of large temporary lists that would consume significant memory if the input dataset were massive.
To finalize the calculation, the presenter uses Python's `sum` function to reduce the stream of filtered numbers into a single total. The `sum` function internally iterates over the `filter` object, requesting the next valid number until the iterator is exhausted, at which point it stops and returns the running total. A visualization using Python Tutor demonstrates that during this execution, only the original list and a few small function objects exist in memory; no secondary lists are created to hold intermediate results. This approach highlights how iterators allow programmers to model computational rules rather than just data structures, enabling efficient processing of large datasets by materializing values only when absolutely necessary for the final reduction step.
Read the full video transcript
in this video we're going to revisit the
example of the previous video
and we are going to implement a memory
efficient solution
so let's go ahead and create a new file
let's call it mfr revisited
so the example is the same as the
previous video so we are given a list of
numbers
and we are going to use the same numbers
that we always use in this course so 7
11 8 5 3
12 2 6 9
10 1 and 4.
so hopefully i didn't mess up the
numbers because otherwise will not get
the same result of 292
as we did in the previous video so let's
see if i got it correct
so let's also briefly revisit the task
so the task is simply to first
transform all the numbers
and the transformation rule is simply
going to be the new
number it's going to be the old number
raised to the power of two plus one
so i will write that using a
mathematical notation y will be the new
y the mapped number
will be x squared plus one the old
number
then we are going to filter out the odd
ones
and last we are going to
sum up all the remaining numbers
so that is a task so previously we
solved the task using a temporary list
object or two temporary list objects
and in this video we are going to avoid
that we want to be memory efficient
so let's go ahead and see how python can
help us do that
so at first we are still going to do the
mapping step and mapping step is all the
operations
in our calculation that take some number
or some element
in our original data set and map it to
some other element
that is of course step number one in the
task so we are going to map
every number to each square plus 1.
in order to do that in this video we are
going to define a helper function and
simply call it transformed for now
the function takes one element as an
argument so let's
simply say element the argument
and now it is going to return
the element raised to the power of two
plus one and you will see in a bit why i
have to define a function here
okay so that is the mapping step so
given some element we just raise it we
just square it and add one
so now comes the big idea now we are
going to use a built-in into python
called the map built-in and the name map
is
of course derived from the mapping step
right from the map filter reduce
paradigm
there is the mapping step and in python
there is a built-in function called
map and this function implements or
helps us implement the mapping step in a
memory efficient way
so let's do that so what does the map
function
do so the map function takes
as input first a function so maybe let's
go to the python documentation
so that you can also read that up
simultaneously
so the map built-in is right here and we
see that
the map building takes as its first
argument a function
and then it takes further intervals
so one interval and then the dot simply
says
there can be further arguments also
going to be intervals and we have seen
in a previous video of how we can
implement a function ourselves
that takes an any number of arguments
using the star notation for packing and
unpacking
if you remember but for now we only have
one interval which is our numbers list
and one function
and the function that map takes is going
to be a function
that has to take one argument itself
and it has to provide back the element
that is uh the thing that is being
mapped to
okay so what we are going to do is we
are going to
give the map build in a reference
to the transform function so note how
once the function is defined
i can simply reference the function by
its name okay
so a transform is just a normal variable
that i can refer
that i can use to reference the function
object
and now i'm going to pass a reference to
the function object as the first
argument to the map built-in
the second argument is going to be our
numbers list
and now let's see what happens if i
execute that i get back
a map add some memory address okay
so what can we do with that so first of
all
let's go ahead and store that in a
variable let's call it transformer
okay so there's a transform function and
now there is a transformer
and the transformer is the thing that
transforms the elements
that's why i call it transformer so the
type of transformer
is going to be map okay it's a map so
sometimes you hear programmers say that
it's a map
you know mapping elements from this set
of things to this set of things
so but map is a type as we see it's a
type
a concrete data type built into python
and now what can a map do
well now comes the thing it can only do
one thing
it can only using the function next the
built-in function called next
it can give you the next element in line
so by saying
next and we give it transformer
as the argument i get the number 50. the
number 50
happens to be seven to the power of two
plus one
okay let's see if i execute next
transformer
a second time now i get back 122.
that is simply 11 to the power of 2 plus
1.
now i do it one more time i get by 65
and you get the idea it's 8 to the power
of 2 which is 64 plus 1.
so what is the map object
so there's another concept that i will
talk extensively about in a future video
which is called the iterator so iterator
is an abstract data type
and just like we saw before a sequence
is an abstract data type and a list is a
concrete data type that implements the
sequence
idea the map data type is a concrete
data type
and it implements the idea behind an
iterator
in an iterator formally speaking if any
object if any object in python
that can do one thing and one thing only
it can provide you
then the next element in line okay so in
other words
the transformer object here it cannot
even go backwards
right so if i go ahead and let's say
there is no
previous function there is only a next
function and also if i go ahead and i
ask python hey what is the length
of transformer
i will get an error message because the
transformer object
does not even know how long it is okay
so it is not a sized object you remember
the term
size that's also an abstract property
that sequences have for example
but the transformer object is an
iterator and iterators are not sized so
in other words
how i think of that casually speaking is
transformer is really a rule in memory
that knows how to calculate something
but has not yet calculated that
okay it's like a postponed calculation
it's like a rule
it's a a box in memory and object memory
that models a rule how to calculate
something without having it done yet
okay and how can we make the rule
calculate the elements well one way
is the way that i just showed you by
simply using the next function
the next function goes to the rule and
says hey rule can you give me the next
element that you can calculate
and it cannot the rule cannot go
backward it can only go forward
instead of using the next function we
could also
go ahead and use the list constructor
and i could give the list constructor
the transformer object and now i get
back a list of all the transformed
numbers however note one thing
this list is shorter than the numbers
list why is it shorter well
let's see what is the first number we
get here the first number is the number
10.
so why is the first number 10 well the
last time i called the next function
with transformer as the argument i got
back 26.
so 26 is obviously um
the the mapped object that is mapped to
the number 5 because 5
to the power of 2 plus 1 is 26 so 3
squared plus 1 gives me 10 right and 12
squared plus 1 gives me 145.
so therefore the list constructor now
basically behind the scenes we we
remember that the list constructor takes
any iterable
and an iterator is also an iterable no
that is
already uh maybe a confusing thing that
an iterator and an iterable they are two
different concepts
but we will talk about that in a future
video in detail so
for now let's simply view it this way
the next function will simply get us
the next object and the list constructor
will simply
go ahead and call next next next as long
as there is no next
okay so the list constructor is like
maybe let's write it down
automating next okay it automatically
calls next as often as until there is
until the end is uh reached right
and the object itself as we just saw it
does not even know when the end is
so the only thing the transformer knows
is well hey i don't have any more
any more element to give you so let's go
ahead
and see what would happen now if we went
ahead and recalled
next on transformer again
now i get a so called stop iteration
exception
so i get a red error message and this
error message is really not an error
this is just a signal from python to you
that
the iterator so the transformer object
is now out of elements
so now from now on as long as i call
that
i get back nothing and let's say one
more thing
if i go ahead and call the list
constructor one more time
with transformer as the object
i simply get back an empty list because
the transformer object does not know how
to produce anything
anymore okay so what can we do now with
transformer
well with the transformer object you
cannot do anything anymore it is
basically exhausted that's the technical
term it is exhausted
cannot do anything anymore so let's
remove all these cells
um again and let's create a new a new
transform object by
calling the built-in map function one
more time
and this basically gives us a new
transformer
and now if i call next again i will get
the first
the first mapped element okay
however i don't want to do that now i
want to postpone that
that's the whole idea of being memory
efficient i want to postpone
all the calculations until i really have
to do them
so let's continue with the second step
the filtering step and the filtering
step for that i will also write a
function
and i will call the function let's say
if even
and the function also takes one element
as its argument
and now it is going to do the following
it will check if element
modular divided by two is zero
then the function is going to return
true
otherwise so else the function is going
to return
false okay so this function is given an
element
any num a number here and it tells me
yes or no if the number is even or not
so now let's make this function a bit
shorter so the first way to make it
shorter is get rid of the else clause
and unindent the return false this is
called the early exit pattern
and now there's even a nicer way to do
that so note how this function gives us
back
either true or false one of the two so
this expression here the condition
element divided by two double equals
zero itself
also evaluates into either true or false
we can try that out
by simply going ahead and say let's say
if i had two
modeling divided by two double equals
zero i get back a true
if i divide 3 by 2 i get back of 4. so
we see that
the condition itself generates already
booleans
and this is going to be true when the
number is even so in other words
what i could do i could get back but i
could get rid of both the return
statements here
i can get rid of the if clause the if
statement
and i can simply return the result
of this expression that's it okay
the result this is going to return
either true or false
and true indicates the number is even so
that is our is even function
and now similarly to how we use the map
built-in
we are now going to use a built-in that
is called the filter built-in right here
and the filter built-in also takes a
function and one iterable as an argument
and basically how this works is as
follows
if i go ahead and i create a filter
and i give it as its function the it's
even function
and let's say i for now give it the
numbers list
i get back a filter object so what is a
filter object
so filter that's
maybe what is a good name for that so
let's simply call it evens
so even there's now another list but
evens is now of type
filter so filter is another concrete
data type
that abstractly speaking is also an
iterator
okay so what do the map and the filter
data types have in common
well they have in common that they only
are good for one operation
getting the next element in line so if i
now go ahead and say next filter
and not not next filter but next even of
course
i get back the number eight why did
number eight well
if we go through this list here the
original list of numbers
it says seven eleven they are both odd
and eight is the first even number
so next events gives me um eight so what
would be the next number it would give
me back it would be the number 12
obviously
so let's try if it works and indeed it
does
so the uh filter object is good for one
thing it will give us the
according to some rule in memory it will
ask it will give us back the next
element in line
and the rule is going to be the next
element must be even
that's it so now if i go ahead and
execute the cell a couple of times i get
like all the even numbers
one by one and at some point i get a
stop iteration exception because
the events iterator is exhausted
okay so same idea as above so now we are
going to do
uh one thing different here instead of
instead of drawing the numbers from the
numbers list what we are going to do is
we are going to draw the numbers from
the transformer object
okay so note how
here i created um let's do it over again
so that we can be sure that
the object is fresh it is it is not
exhausted
not used so far the transformer object
is now only a rule
in memory that knows how to calculate
something
okay and the rule is simply take some
number from numbers
and transform it to this according to
this rule and give me the next number
according to this rule
now the filter object down here
as its source so to say is not using the
numbers list but the transformer
so we are building a rule that as a
source
has another rule so in other words we
are going to ask the filter object hey
give me your next thing
and the filter object says well i'm
going to ask the transformer object
to give me the next thing and then the
transformer object
goes into the numbers list and the
numbers list will give will give us the
next number
and then the number first goes through
the transformer to be transformed
and then the transform number is going
to be filtered right after each other
okay so in other words if i now go ahead
and i say next evens i am not going to
get the number 8 anymore
but i'm possibly getting the number 7
squared plus 1 which is 50.
so let's see that and indeed i get back
the number 50.
okay so i'm doing one by one calculation
so to say
okay this is like some people call that
a pipeline so whenever i
enter or whenever i execute this cell
here
next evens the filter object is going to
get its next number
and it's going to ask transformer and
transformer is going to ask for
its next number which gets the next
number from numbers so we are pulling
out
in other words we are pulling out the
numbers one by one we are transforming
them
filtering them and then we have to do
something else with them
okay and the something else we want to
do with them
is the following this is the easiest
part in the whole chain here
so let's get rid of the next here the
easiest part in the whole chain is
simply the reduction step
as before so this is not no different
from before
and the reduction step we can simply use
the sum function
so the sum function according to
the documentation takes
as we see any iterable anything we can
loop over
and now we are going to loop over the
events
object and even the filter object right
so now
this is giving me the same answer as in
the previous video luckily so the
numbers above are correct
292. however we are doing that in a
memory efficient way
okay so let's quickly go ahead and
copy paste that over to python tutor so
that we see the difference
also in a memory diagram
so here are the numbers now i have to
copy paste a little bit more
and in the next video we are going to
see how we can even get rid of the
functions that i now need to copy paste
so let's go ahead
and copy paste over this here
let's also copy paste over the filter
function and let's maybe put
the functions on top so the filter
function goes on top here
then we create the evens
object and then last but not least we
are calculating the result as
the sum of events okay
so let's go ahead and visualize that so
first we are going to create the numbers
list in the global scope that is exactly
the same as before
and we must do that because somehow we
have to be given some raw data
now we are going to create two objects
which model the rules they are just
a plain ordinary function objects
nothing special about them
and when it comes to memory consumption
we can assume that
they are not consuming a lot of memory
okay so we only care about the big list
that possibly has
i don't know 10 billion numbers in it
but we don't really care about the size
of the function objects here
now next we get a transformer object
which is um
it says here a map instance so it's an
object of type map
which is simply a rule that uses the
functions above to pull out numbers one
by one
then we create the evens filter object
and last but not least we are going to
run all of them
through the uh into or putting all of
them into the
sum built-in and this is simply going to
calculate as we see on a one by one uh
this is actually a very nice
visualization so now we are going to run
the sum function
and the sum function first calls the
transform function
with an element of seven so the first
number
this is going to return 50 which is a
square plus one
now the is even function is given the
number 50 and it's going to return
true because the number 50 is even and
now
we go over all the numbers one by one
as we see so it's a couple of steps i
have to click through i will make that a
bit faster
but we see that there is no second list
object or no third list object
so all the numbers are processed one by
one and what the sum
function does is the sum function
internally
works just like the very first python
example in this course it's
it's basically calculating running total
okay
that is usually how pro programmer does
um
addition for more than two numbers by
doing a running total
and at the end we see a result of 292
and there is no other list object
and again those function object and map
and filter objects here we can disregard
them they don't really have a
you know they don't really have big
memory consumption
okay so that is the first step that you
have to get used to if you want to work
with big amounts of data
using objects that model
rules that know how to calculate the
next object in line
without having it without calculating it
yet
and then in this case the the reduction
step is the thing that drives all the
elements
the reduction step is the thing that
makes the rules um
materialize into real numbers but we're
doing that on a one by one basis
and then in this case the sum function
simply takes the transformed and
filtered numbers on a one by one basis
and simply adds them on top of a running
total and then calculates the result
so super memory efficient okay so i know
that
for beginner this may be a bit hard to
swallow but i think it's not too hard
i would suggest compare it with the
previous video again
to see the differences in memory we
don't have a
second list here as we see and in the
next video we are going to talk about
the same example one more time
and we are going to make it um even more
concise by
getting rid of the function objects that
we have to define here
okay so i will see you in the next video