Video summary
This video revisits a classic data processing example involving a list of numbers, where the goal is to transform each number by squaring it and adding one, filter out the odd results, and finally sum the remaining even values. The primary focus is on rewriting this logic using Python's built-in `map` and `filter` functions with lambda expressions to achieve a more memory-efficient solution. Instead of defining named functions with the `def` statement, which are typically used for code reuse, the presenter introduces lambda expressions as anonymous functions. These allow developers to pass transformation logic directly into `map` and filtering logic directly into `filter` without creating separate function objects that sit in memory unnecessarily.
The tutorial explains the syntax of lambda expressions, noting that they consist of a parameter list followed by a colon and a single expression that serves as the return value. A key constraint highlighted is that lambda expressions can only contain one line of logic; if a task requires multiple steps or complex conditions like an `if-else` block, a standard function definition is preferred. To handle conditional filtering, such as checking for even numbers, the video demonstrates the "early exit" pattern where the `else` clause is removed, and the condition itself returns a boolean value directly. This ensures the lambda remains a single expression compatible with the built-in functions.
The execution flow is described as a pipeline where data moves through stages sequentially: first, the `map` function applies the transformation rule to each item in the original list one by one; second, the resulting iterator is passed to `filter`, which checks each transformed item against the condition and yields only those that match; and finally, the `sum` function consumes the filtered stream to calculate the total. The presenter uses Python Tutor to visualize this process, showing how values are processed individually without creating intermediate lists in memory at any stage. This approach contrasts sharply with earlier methods that generated temporary lists after each step, emphasizing that lambda expressions allow functions to be created "on the fly" and passed directly as arguments, minimizing memory overhead.
The video concludes by reinforcing the importance of understanding iterators and the map-filter-reduce paradigm for working with large datasets in data science. While Python abstracts away much of the manual memory management, knowing how to design calculations that avoid unnecessary intermediate objects is crucial for handling big data efficiently. The presenter suggests that chaining these built-in functions into a single expression, read from the inside out, is both elegant and performant. Ultimately, mastering these tools prepares learners to tackle real-world problems where memory efficiency is not just an optimization but a necessity for processing substantial amounts of information without exhausting system resources.
Read the full video transcript
in this video we are going to review
the same example of the two previous
videos and we are going to rewrite
the memory efficient version using the
map and filter built-ins
in a nicer way without having to define
a function so we are going to
create a new file and let's call it
mfr for map filter reduce revisited
again and
let's go ahead and create the same
example one more time
so it's a list of numbers and it's going
to be the list
seven eleven eight five three
twelve two six
9 10 and 1 4
and i'm not going to write down the task
or maybe i write a short version of the
task so the task is first
transformation so transform so maybe
let's write it down the task is to first
transform all the numbers according to
the rule
y is x squared
plus 1. just like that
then we are going to filter out
the odds and last but not least we are
thumb up
we're going to sum up the remaining
numbers
okay so same example as before
and now let's do that without having to
write a function
so previously i wrote a function to do
the transformation step
and it looked like that def transform
function took an element as its argument
and then we simply wrote
return element raised to the power of 2
plus 1. however i'm not going to define
the function you see i'm not going to
execute the cell we're not going to use
that function this is just to compare
so how can we model a function without
defining a function so maybe you ask
yourself why would i want to do that why
why do i not just you know stick with a
function
so remember that in order to use the
map built in we first have to
pass some function that does the
transformation and then we have to pass
in this case an iterable the numbers
list that
contains all the elements that are to be
transformed
so in other words for syntactical
reasons
what we need is we need to have a
function right here
was in technical reasons however what is
the main purpose of defining functions
using a def statement
well the main purpose of doing so is to
reuse the function
so are we going to reuse the function in
this example
and the answer is not really we're only
going to use the function exactly once
and we're going to define the function
we have to define the function so far
because
the map built in needs to have a
function object as its first argument
so wouldn't it be nice if you could
somehow pass
in the logic the transformation logic
without defining a function and we can
do that
using a so-called lambda expression so
in chapter 2
there is a video where i talk about how
to define anonymous functions so
functions that do not have a name
and we did that using lambda expressions
and maybe back then
you were wondering why would i ever need
that why would i want to define a
function that has no name
well one situation where this is quite
useful
is in a place like here in the map built
in and also later in the filter building
they need a function for syntactical
reasons but we are not going to reuse
the function anyways so we don't want to
give the function name because we are
not going to reuse it
so what we do instead is we will go and
define a lambda expression
and i will briefly review how lambda
expressions work
so lambda expressions they are written
lambda
that is where the name comes from num
the expression
then we are going to mention all the
parameters the function should take
so for now let's call it element i'm
going to use exactly
the same names as above then we are
going to write colon so there's no
parentheses here
okay the real function definition has a
parameter list up here
using parentheses the lambda expression
has no parentheses
we have a colon and then after the colon
we are going to uh to define one
expression
that is also going to be the return
value however we don't need the return
statement
so we are simply going ahead and we will
say element
to the power of two plus one
this if i execute that as we see below
the cell
will give me back some function object
so it's code
in memory that can be executed
which is nothing but a function object
so really what i'm doing here is i'm
creating a box in memory an object
that is of type function and it contains
all the function code
however it does not have a variable
pointing to it
okay there is no name in the global
scope that can reference this function
okay but other than that the object that
contains the function's body
is pretty much the same it's exactly the
same actually except for
a couple of technicalities but it is
really the same
so now what we often do is when we use a
lambda expression lambda expressions
they are a simplified version of a
function so
note how we can only have one expression
that is automatically going to be the
return value so if you have a function
that has to take more than one more than
one line in order to calculate something
it
needs like two or three lines to
calculate some result then probably
you you're still better off using a
function definition using the def
statement
okay only functions that have basically
one line that we can immediately return
these are um yeah functions that are
good enough
to be transformed into a lambda
expression okay
and then what we do is to keep it more
concise instead of calling the parameter
element here we're simply going to call
it x
so we're going to say lambda is given um
is given um oh there was a fly here so
lambda is given
um some x as the input and it's going to
return x squared plus one
okay so that's it so that's the
transformation it's also basically the
transformation we see right up here
is x squared plus one okay but the
overall result would be
um not a here would not be y here but
it's simply the return value of that
so now you may wonder how can you call
such a function well in the previous
video in chapter two where i talked
about anonymous functions
what i did is i put the lambda
expression in parentheses we have to do
that for grouping purposes
and then i'm going to call the function
using the call operator which happens to
be also parentheses
and let's say i give it two and i get
back five here okay so the two is
transformed
it's squared and then we add one to it
okay
however if i don't want to call the
function there is no need
for putting parentheses here so this is
a function object
so this is an expression resulting in a
function object that could then be
called
and now what we are going to do is we
are going ahead and we will copy paste
that
and we will place it right here where it
says function
okay let's also get rid of the cell that
the with the def statement which we
never executed
and now we um execute the map cell here
and this is going to give us back a
transformer object
so let's call it transformer as before
and the transformer object what can we
do with it well
we can of course call the next function
with it
and we see we get the next number 50. so
7 to the power of 2 plus 1 gives
50. we could also as we saw in the
previous video use the list constructor
and we get the list of all the numbers
transformed except for the first one
here
why not well the first one was already
pulled out of the transformer object and
we remember from last video
that the transformer object can only
give us the next element in line it can
never go back
okay that's an important idea so if i
execute this cell again
i will get back an empty list because
the transformer object is what we say
exhausted okay so what what do we do
with an exhausted object well there's
nothing you can do with it you can
basically throw it away
so let's get rid of this code cell and
let's get ourselves a new transform
object okay we have to create a new
transformer object
and now we're doing the same thing here
now for
the filtering step so for the filtering
step remember that
i wrote a function is even which also
took an element
and the function has an if statement in
it in its original version
and it says if element modulo divided by
2 has
no rest then return
true otherwise so else
return false and then we made this
function a bit shorter why do i need to
make this function shorter
well as i just mentioned a lambda
expression
can only consist of one expression that
becomes the return value
so therefore we have a statement here so
the if statement is as we can
guess by the name it's a statement it's
not an expression
so therefore we have to make this
function
such that it consists of only one
expression
so what we can do is first of all we can
get rid of the else
and unindent you return false this is
what we call the early exit pattern
and then what we also mentioned in the
last video was
the condition of the if statement itself
returns either true or false so this
condition
is exactly true when we want to return
true
so therefore what we did is we got rid
of the bow of the two lines
that say return we get rid of the colon
at the end and also instead of the if
statement
we're simply going to say return so
we're going to return the result
of this one expression here which is
either true or false
okay and now in order to transform that
into a lambda expression
what we are going to do is the following
i'm going to write lambda as above
i'm going to write element for now and
we simply go ahead
and copy paste this expression okay
so that's how we can transform this
function from its longer version
into a single lumbar expression also
let's get rid of this function we don't
need it
i told you that i don't want to use any
function objects here
and also let's get rid of the element
here and let's call it x
and every function has its internal
scope
and because lambda expressions also
create function objects
these function objects also have the
internal local scope
therefore the variable the parameter x
here and the parameter x here
they are different parameters they live
in different scopes when we execute
these functions
so therefore we don't have to worry that
we are reusing x here
so if you want to call this function we
can do our trick and we can
wrap everything in a pair of parentheses
for grouping and then we can add another
pair of parentheses for
calling the function and let's say i
give it as
a value the number two and it returns
true y true well because the number two
is even right
if i give it a number three which is an
odd number i get back false
okay so we see that this function will
always give us back true or false
so let's get rid of the calling operator
here
and get rid of the grouping parentheses
and now let's go ahead
and put that lumper expression
as the first argument inside the
filter built in and now as the second
parameter or the second argument that we
passed to the filter building
we are going to simply pass as in the
previous video the transformer
object okay and we get a filter object
and now what we are going to do is we
are simply going ahead and we will put
that in the same code cell
and we will call that evens just like
that okay
so let's get rid of all the other code
cells
let's go back into this code cell and
now we want to do
we want to finish up the example we want
to sum up all remaining numbers
and we will simply do that by saying sum
up the events
and if i execute that it says 292
right so here i don't have any
any function object that has a name so
what we can do here
is we can do the pi can go to python
tutor
and one more time for this example copy
paste
over the code that solves the problem
and you can then compare that with the
previous two videos
in how that is different in in terms of
memory usage
so now we are going to create first our
numbers list
and then we are going ahead and we will
create a transformer object
which is basically a rule in memory that
knows how to calculate the next object
in line
without that is important without having
calculated it yet
then we create an evens object which is
a filter object
which is also a rule that knows how to
calculate the next object in line
without having calculated it yet and
then last but not least
we are going to execute the sum function
and the sum function is the reduction
step
and the reduction step is basically on a
one by one basis
asking the events object hey can you
give me your next
uh your next element your next number
and then the evens object itself
is will go back to the transformer
object and we'll ask it hey can you give
me your next number
and the transformer object will go back
to the original list of numbers
and it will pull out the next number in
line okay
so therefore what we see here the
numbers in the list here
they will individually go through the
first
transformer then to the filter okay so
each number
here will one on a one by one basis go
through this we call it a pipeline
okay and only the even numbers after
after the transformation will come out
of this pipeline
so if uh for example the number eight
the number eight squared plus one gives
me 65
and the number 65 is obviously odd so
therefore the number eight will go
through the pipeline but it will not
come out here
okay and then down here we can imagine
so to say
the sum function waiting for all the uh
for all the transformed and even numbers
that come out of this pipeline
and then it sums up all the remaining
numbers
onto a running total so internally the
sum function
works with a running total just as we
did in the very first
python example of this course so let's
execute that
and we see that now it says lumper here
so python twitter shows that we are
executing a lambda expression
and we see that the numbers are
transformed on a one by one basis and
filtered on a one by one basis
and of course it takes some time so we
still have to go through all the numbers
and transform them and so on
however we see at no point in time a
temporary list object
and at the end the end result will
simply be result is 292
okay no second list no third list as in
the very first version of this example
okay and in comparison to the previous
the second version
of this example we don't even have any
function object here
okay the function object is created what
we say on the fly so to say
so it is created once and then put right
into the
map and filter objects as the argument
okay so what else
is there to be said well what you could
do is
if you still want to save yourself these
two temporary variables here this is
something you would not need to do
because otherwise the
whenever you create a variable in python
this is a very cheap operation it does
basically
take no memory but let's assume you want
to do that you could rewrite that
just like that we write the sum function
and the sum function takes as its
argument
the filter object and the filter object
here which the transformer takes as its
argument
another map expression here another map
a map called a call to the map function
okay and now if you want to make that
a little bit nicer to read what we could
do is
we could go ahead and go ahead and break
a couple of lines here
so that we see the internal structure of
how
this will be constructed
and now we see that this expression here
in this cell will be evaluated from the
inside out
so first we take the numbers and we
transform them on a one by one basis
that is important this is still this is
still the same as above
just using no variables for transformer
and evens
after the transformation we filter that
and then at the end very end we are
summing that up so you have to read this
expression
kind of from the inside out first
mapping then filtering then reduction
with a sum so let's do that and we get
back the same result 292.
okay so um so that third video now um
basically concludes the discussion on
the map filter paradigm
uh which is kind of an introduction to
the more topics follow in in chapter
eight um
in the book materials and um yeah so
make sure you understand the math field
or reduce paradigm
the first video in the last in in this
video in the last two videos
shows you um how you could um solve the
problem without any without learning
anything from chapter eight
so with everything you learned before
chapter eight and then the last video
and this video
show you how you can use python
built-ins in particular the map and
filter built-ins
to do the same exercise again in a
memory efficient way and this is where
you want to aim for if you want to go
into the field of data science because
eventually you want to solve interesting
problems and that means you want to work
on big data sets
and therefore you have to know how you
can design your calculations
in a memory efficient way okay so that
is the the main reason
why chapter 8 is in the materials of
this course because this course
wants to prepare you to become a data
scientist for real life and work
to work with real life data big data
sets and
therefore we should concern ourselves
with
how the memory works in detail even
though python takes away a lot of the
load from managing the memory but some
details are very worthwhile to know
and knowing about map and filter i think
is very much worthwhile
so in the next couple of videos we will
continue discussion of
what is called iterators so iterators is
the abstract concept
that is behind the map and the filter
objects so iterators are all objects in
python
that function like rules like rules that
can give me the next thing without
having
or that know how to calculate the next
object without having calculated it yet
okay so that is the big paradigm or the
big the big topic of chapter eight
so i will see you in the next video