Video summary
This video tutorial introduces viewers to an efficient method for cleaning datasets in R by removing specific rows using the `subset` command. The presenter, Monica from Death Wench Professional Services, utilizes a real-world dataset called NHANES, which is often used in public health data science, to demonstrate how to filter data based on criteria. She explains that while R can handle various file formats, this specific demonstration focuses on removing rows where the gender variable equals one, effectively deleting all male participants from the dataset to leave only female records. The goal is to show how a single line of code can drastically reduce the size of a dataset, transforming an initial collection of approximately 15,000 rows down to roughly 7,800 rows in just a few steps.
The process begins with importing the data file, which is named `pcor_demo.xpt`, into the R GUI environment and assigning it a shorter variable name for easier typing. Before applying any filters, the presenter runs an `nrow` command to establish the baseline count of rows in the original dataframe. She then constructs a new dataframe using the `subset` function, specifying the condition that the gender variable must equal two, which corresponds to women according to the data dictionary provided by NHANES. By executing this command, she creates a filtered version of the dataset while keeping the original intact, a practice she recommends for safety in case errors occur during analysis. Running the `nrow` command again on the new dataframe confirms that the filtering was successful and that only the desired subset of the population remains.
Beyond the primary demonstration of row removal, the video offers valuable advice on handling data imported from foreign software formats like SAS or SPSS, which often come in obscure file types such as `.xpt`. Monica recommends using the `foreign` package to successfully read these non-standard files into R, highlighting that this package supports various external data formats beyond just SAS. She walks through loading the necessary library and using the `read.xport` function to bring the data into the R environment seamlessly. The tutorial concludes with a call to action for viewers to share their own favorite downloadable datasets in the comments, like the video to support the channel, and subscribe to stay updated on future content regarding data science techniques and public health analytics.
Read the full video transcript
hey there did you know that in r with
one line of code you can remove
thousands of rows from any data set R
can read even data sets that are in
foreign formats like SAS and SPSS if you
are learning R use this video to
challenge yourself I'll walk you through
how to use one simple command the subset
command to delete rows by criteria in R
using a real world data set and if you
stick around until the end I'll show you
my trick for importing data from forign
form
hi I'm Monica of death wench
Professional Services and I teach Public
Health Data science if you are new to
the channel welcome and if you are
returning thank you for coming back I
try to post a new video every Friday at
10: a.m. so don't forget to be a regular
visitor so you can be the first to check
out my new video every
week today I'm going to demonstrate how
to remove rows from a real world data
set called en haanes using our R GUI if
you want to learn more about nanes read
my blog post about anhes the link is in
the
description as you'll learn in my blog
post the nanes data and documentation
come from this online portal the nanes
is one surveillance effort but for
whatever reason they split up the data
into different data sets for this
demonstration we will be using the
demographics data I'll click on it we
only have one choice of data set as you
can see you can read the document mation
under the doc file heading and download
the data from clicking under the data
file heading let's click on the
documentation here we get to the data
dictionary for this demonstration we are
going to use the Ria gender variable
which means gender note that it is coded
as 1 equals men and two equals women got
that all right let's go over to our guey
okay here we are in our GUI if you want
this code I'm using just click on the
link in the video description to go to
my GitHub folder make sure you download
the code named R1
0310 delete rows as I showed you before
we started with the nanes demographic
data set which is called pcore demo. xpt
I imported that into our GUI and named
the data frame dgor a so it has a
shorter name that is easier to type now
let's focus on the code for deleting
rows by criteria
as you can see here I first run an N row
command to count the number of rows in
the data frame dgor a actually let's run
this code and see how many rows are in
dgor a okay we see in the console that
we start out with about 15,000 rows in
dgor a let's look at the next line where
I use the subset command we are going to
subset the data frame dgor a meaning we
are going to remove some rows from it
and copy it into dataframe
dgb I use these naming conventions so I
can do a roll back if anything goes
wrong hey if you ever want advice about
public health or data science schedule a
free 30-minute Zoom appointment with me
using the link in the description and
don't forget to follow me and the death
wench Professional Services Company page
on LinkedIn as you can see we are
subsetting dgor a by the criteria Ria
gender equals 2 remember how two means
women so we are basically keeping all
the women and removing the men from the
data set and then we do another n row to
count the rows in dgor B all right let's
run this code and see what happens okay
we kicked out all the men as you can see
in the console we have about 7,800 women
left in the data set so that's how you
use the subset command in our GUI to
remove thousands of rows from a data set
with one line of
coat hey I don't just use this enhan to
demonstrate how to use R I also help my
customers analyze this data set for
scientific papers and portfolio projects
do you have a favorite downloadable data
set you like to use if so please put it
in the comments to this video and add a
link if you like this content please be
sure to hit the like button because then
my videos will come up in your feed more
often or better yet hit subscribe it's
free and it really helps me out thank
you okay here's the tip I was going to
give you for importing foreign data into
R you might notice that the enhan data
are stored in a very obscure format
which is xpt this is a SAS format from a
foreign software so what if you are
trying to read a SAS data set or an SPSS
data set into R for this I recommend you
use the foreign package as you can see
here I call up the library foreign then
use the read. exort command to import it
into the r guy environment the forign
package also handles other nonr data
formats check it out on
cray want to continue your data science
Journey with me and experienced
professional at your side take my online
course application Basics and learn
business application terminology
demonstrated through unique case studies
so you are empowered to Divine novel
insights from analyzing application data
better yet register for the workshop
click on the link in the description to
access the course and
workshop and don't forget to follow me
on the socials links to all my social
media accounts are in the description
let's stay connected because you never
know where your data science Journey
will lead have a data emergency or just
need advice remember to let death wench
do your data