Data Science Pipeline

First, let’s look at the data science pipeline.

The data science pipeline starts with defining what major questions one wants to answer and subsequently acquiring and importing the relevant data to be analyzed.

Then, the data is viewed and data tidying must occur; where a rectangular data structure model is assumed and three requirements must be met. Each observation (called an entity) forms a row, each variable (called an attribute) forms a column and each observational unit (type of entity) forms a table.

Leading to the exploratory data analysis process, where the data is transformed and visualized. Data cleaning may be necessary for missing data. When handling missing data, the missing data may be removed, encoded or imputation (replace missing values with the mean of non-missing values) of a numeric variable may be necessary.

Hypothesis testing and machine learning (ML) modeling are the final steps before the data and its results can be communicated.

Image of Pipeline

Source: https://r4ds.had.co.nz/explore-intro.html

Economy & Vacations

The rise of companies like Airbnb has given rise to The Sharing Economy. The sharing economy is a model defined as the facilitation of goods and services on a peer-to-peer level usually through online community platforms. This new model has made it possible for a great deal of people to gain another source of income and for you to have an affordable vacation.

As more sharing economy companies have opened, like Airbnb and Uber, the way we vacation has changed. This change has been documented and open data on it is available.

DataSet Used

The data we will be using in this tutorial is New York City Airbnb Open Data from Kaggle. We will use this data to look at the relationships between types of housing and location.

Loading Data

CSV files are files that include data which are “comma-separated values”, meaning that data values are literally separated by commas.

After we’ve downloaded our CSV file from Kaggle into our working directory, we can use the read_csv function to load the CSV file data into our program’s data frame, which is a table of the data.

# create a dataframe from our CSV file
airbnb_tab <- read.csv("AB_NYC_2019.csv", header=TRUE)

There are some attributes that we don’t need for our purposes, like host_id, host_name, minimum_nights, number_of_reviews, last_review, reviews_per_month, and calculated_host_listings_count. So, let’s remove these from our data frame:

# a vector called "to_remove" that has the names of the attributes we don't want
to_remove <- c('host_id', 
              'host_name', 
               'minimum_nights', 
               'number_of_reviews', 
               'last_review', 
               'reviews_per_month', 
               'calculated_host_listings_count')

# removing attributes from data frame using "to_remove"
airbnb_tab = airbnb_tab[ , !(names(airbnb_tab) %in% to_remove)]

Viewing Data

Here, we see the first 10 rows in our dataset:

knitr::kable(head(airbnb_tab, n=10))

Some Notes:

• knitr::kable() is used to make the table “pretty” and easier to read

• head(df, n=10) is used to view the dataframe with a specific number of rows (head() is not always necessary, you can just list the data frame for it to render)

• the first argument is where the dataframe goes, in this case, airbnb_tab

• n = determines the number of rows visible, in this case, 10

The following is a list of descriptions for the attributes of our data set:

Tidying Data

Tidying Data entails the elements listed in the list below.

Elements of a tidy dataset:

1. Each observation/entity forms a row
2. Each variable/attribute forms a column
3. Each observational unit (type of entity) forms a column (i.e. not dependent on one another)

Our dataset is already tidy and meets the criteria above. Each entity is a row and each attribute is a column, where no entity is dependent on another.

However, if your data set is untidy, below is an example on a different small dataset, to show you what to do.

airline_schedule <- read.csv("messy-airlines.csv", header=TRUE) # read downloaded CSV file into a dataframe (array of data)
knitr::kable(airline_schedule)

airline_schedule = rename(airline_schedule, c("airline"="X", "status"="X.1"))
knitr::kable(airline_schedule)

tidy_data <- gather(airline_schedule, "destination", "count", 3:7) %>% # Step 1
  spread(status, count) %>% # Step 2
  mutate(destination=gsub("\\.", " ", destination)) %>% # Step 3
  arrange(destination, airline) # Step 4

knitr::kable(tidy_data)

Handling Missing Data

Recall that the attribute availability_365 tells us how many days in the year that this particular listing is available for people to book.

Notice that 0 is a value for some of the entities (Airbnb listings). It doesn’t make much sense for us to look at entities that aren’t available at all during the year. In fact, more than 17000 entities are listed at being available for 0 days out of the year! That’s about 1/3 of our dataset.

We’ll call this “missing data”, and remove these entities from our dataset:

airbnb_tab <- airbnb_tab %>%
  filter(availability_365 > 0) # filter() is used to filter the dataframe via specific conditions

knitr::kable(head(airbnb_tab, n=10))

Note that a way to handle missing data, as mentioned in the data science pipline section (data cleaning), is removing missing data altogether. Having 0 as a value for availability_365 is a form of missing data.

Data Visualizations

What are data visualizations? Data visualizations are representations of data and/or information in visual format (like a graph or chart). These visualizations allow for patterns and larger data to be presented and communicated straightforwardly. Data is used to gain insight and is valuable so the way it’s presented is important. Humans are visual, it’s how the brain works!

There are many different forms of data visualization, each of which have their own advantages.

The types of data to be analyzed (categorical and/or numeric) is an important factor in deciding what data visualization is applicable and pertinent.

Want to know more about picking the best data visualization for your message, click here.

# Download necessary library to integrate and control maps
library(leaflet)

# Creating NYC Map
nyc_map <- leaflet(airbnb_tab) %>%
  addTiles() %>%
  setView(
    lat=40.730610, #set latitude of NYC
    lng=-73.935242, #set longitude of NYC
    zoom=11)

nyc_map #outputs the map

leaflet(airbnb_tab) %>% #pass in 
  addTiles() %>%
    addAwesomeMarkers(
      #pass in given longitude of entity
      lng = ~longitude,
      
      #pass in given latitude of entity
      lat = ~latitude,
      
      # Setting up icon for entity on map
      icon = awesomeIcons(
              icon = 'ios-close',
              iconColor = 'black',
              library = 'ion',
              # Determines color of the icon on map with nested if else statements
              markerColor = ~ifelse(room_type == 'Entire home/apt', "green", 
                                    ifelse(room_type =='Private room', "orange", 
                                           "red"
                                    )
                            )
            ),
      # Price Label
      label = ~paste("$", as.character(price), "per night"),
    
      # Clustering for identifying arrest density
      clusterOptions = markerClusterOptions()
    ) %>%
  addLegend(
    position = 'bottomright', 
    # Color keys correspond to values, respectively
    colors= c("green", "orange", "red"), labels=c("Entire Home/Apt", "Private Room", "Shared Room"), 
    title='Types of Rentals', 
  )

# Download subsequent libraries
library(ggplot2)  #data visualization package for the statistical programming
library(ggthemes) #package for themes

# Filter out all the Enitre home/appt listings into new dataframe
airbnb_home <- airbnb_tab %>%
  filter(room_type == 'Entire home/apt')

# View new table
knitr::kable(head(airbnb_home, n=10))

airbnb_home %>%   #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  geom_boxplot()+   #creating a boxplot
  coord_flip() +    #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "Entire Homes & Appts. Price By Neighborhood in 2019",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

airbnb_home %>%   #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  
  geom_boxplot()+   #creating a boxplot
  
  # Limiting the y-axis to get a better view of data
  scale_y_continuous(limits = c(0, 1500)) +
  
  coord_flip() +  #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "2019 NYC Homes & Appts. Prices (Up to $1500/night)",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

# Filter out all the Private room listings into new dataframe
airbnb_room <- airbnb_tab %>%
  filter(room_type == 'Private room')

# View new table
knitr::kable(head(airbnb_room, n=10))

airbnb_room %>%   #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  geom_boxplot()+   #creating a boxplot
  coord_flip() +    #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "Private Room Price By Neighborhood in 2019",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

airbnb_room %>%   #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  
  geom_boxplot()+   #creating a boxplot
  
  # Limiting the y-axis to get a better view of data
  scale_y_continuous(limits = c(0, 500)) +
  
  coord_flip() +    #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "2019 NYC Private Room Prices (Up to $500/night)",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

# Filter out all the Shared room listings into new dataframe
airbnb_sroom <- airbnb_tab %>%
  filter(room_type == 'Shared room')

# View new table
knitr::kable(head(airbnb_sroom, n=10))

airbnb_sroom %>%   #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  geom_boxplot()+   #creating a boxplot
  coord_flip() +    #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "Shared Room Price By Neighborhood in 2019",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

airbnb_sroom %>%  #send in dataframe
  # Setting up what data is used from the dataframe
  ggplot(aes(x = neighbourhood_group, y = price)) +
  
  geom_boxplot()+   #creating a boxplot
  
  # Limiting the y-axis to get a better view of data
  scale_y_continuous(limits = c(0, 200)) +
  
  coord_flip() +   #flipping the coordinates to have horizontal view
  
  # Themes for graph
  theme_economist() + 
  scale_fill_economist() +
  
  # Setting up title and axis labels
  labs(title = "2019 NYC Shared Room Prices (Up to $200/night)",
       x = "Major Neighborhood Groups",
       y = "Price(USD)")

ML & Hypothesis Testing Walk Through

With datasets that are large, it can be very useful to generate a linear regression, or a line of “best fit”, for an easier interpretation of the data. This data analysis technique is also an effective way to learn about general trends of our data set and lets us construct confidence intervals and do hypothesis testing, which analyzes and tests for relationships between variables.

We want to look at the relationship between price and distance away from Times Square in New York City, one of the largest populated cities in New York. We are looking at Time Square since it is a major commercial intersection, tourist destination, entertainment center, and neighborhood in the Midtown Manhattan section of NYC (Wikipedia).

For these reasons, we would like to see if Airbnb listings would increase as their distance to Times Square (latitude 40.757, longitude -73.986) decreases, and vice versa. We will be using functions from the geosphere library to calculate distance between coordinates.

First, let’s add an attribute called distToTimesSquare in our dataset. This will contain the distance (in miles) between each listing and Times Square.

coordsTimeSquare <- c(-73.986, 40.757) # vector of Times Square coordinates, first longitude, second latitude

airbnb_tab <- airbnb_tab %>%
  mutate(distToTimesSquare = by(airbnb_tab, 1:nrow(airbnb_tab), # calculate distance using distHaversine function
                                function(row) { 
                                  distHaversine(c(row$longitude, row$latitude), coordsTimeSquare)
                                }) / 1609) # divide by 1609 to convert meters to miles

knitr::kable(head(airbnb_tab))

Second, let’s split our current airbnb_tab data frame into two data frames, one with room_type == "Entire home/apt" and one with room_type == "Private room". This is because prices are much more expensive for “Entire home/apt” listings, so we don’t want to get confused when regressing against distance. We only want to see the relation between distance and prices, not between prices and size of the space being listed!

# create new dataframe of listings where room_type=="Entire home/apt"
entire_tab <- airbnb_tab %>%
  filter(room_type == "Entire home/apt")

# create new dataframe of listings where room_type=="Private room"
private_tab <- airbnb_tab %>%
  filter(room_type == "Private room")

# create new dataframe of listings where room_type=="Shared room"
shared_tab <- airbnb_tab %>%
  filter(room_type == "Shared room")

knitr::kable(head(entire_tab))

knitr::kable(head(private_tab))

knitr::kable(head(shared_tab))

Third, we want to create a scatter plot of the prices of listings against their distance to Times Square. We’ll also add a regression line to this scatter plot to the general increasing or decreasing trend in our data! Let’s do this three times, once for each room_type we are interested in.

entire_tab %>%
    ggplot(aes(x=entire_tab$distToTimesSquare,y=entire_tab$price)) +
    geom_point() + # plot points for scatter plot
    geom_smooth(method=lm) + # plot linear regression line or line of best fit
    ylim(0, 1500) + # set the upper limit of prices to $1500
    labs(title="Homes & Appts. Prices vs Distance to Times Square", x="Distance to Times Square (miles)", y="Price (USD)")

private_tab %>%
    ggplot(aes(x=private_tab$distToTimesSquare,y=private_tab$price)) +
    geom_point() + # plot points for scatter plot
    geom_smooth(method=lm) + # plot linear regression line or line of best fit
    ylim(0, 500) + # set the upper limit of prices to $500
    labs(title="Private Room Prices vs Distance to Times Square", x="Distance to Times Square (miles)", y="Price (USD)")

shared_tab %>%
    ggplot(aes(x=shared_tab$distToTimesSquare,y=shared_tab$price)) +
    geom_point() + # plot points for scatter plot
    geom_smooth(method=lm) + # plot linear regression line or line of best fit
    ylim(0, 200) + # set the upper limit of prices to $200
    labs(title="Shared Room Prices vs Distance to Times Square", x="Distance to Times Square (miles)", y="Price (USD)")

Lastly, let’s analyze the resulting models quantitatively using broom::tidy.

entire_fit <- lm(distToTimesSquare~price, data=entire_tab) # create the simple regression model
entire_fit %>%
  tidy() # turn model into a tibble with information about the model

## # A tibble: 2 x 5
##   term        estimate std.error statistic  p.value
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
## 1 (Intercept)  4.43    0.0284        156.  0.      
## 2 price       -0.00149 0.0000760     -19.6 6.45e-85

private_fit <- lm(distToTimesSquare~price, data=private_tab) # create the simple regression model
private_fit %>%
  tidy() # turn model into a tibble with information about the model

## # A tibble: 2 x 5
##   term        estimate std.error statistic  p.value
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
## 1 (Intercept)  5.44     0.0277       197.  0.      
## 2 price       -0.00237  0.000141     -16.8 7.26e-63

shared_fit <- lm(distToTimesSquare~price, data=shared_tab) # create the simple regression model
shared_fit %>%
  tidy() # turn model into a tibble with information about the model

## # A tibble: 2 x 5
##   term        estimate std.error statistic   p.value
##   <chr>          <dbl>     <dbl>     <dbl>     <dbl>
## 1 (Intercept)  5.41      0.128       42.3  2.95e-212
## 2 price       -0.00555   0.00109     -5.11 3.93e-  7

As we can see in all three of these linear regression plots, the prices of all the types of listing decreases slowly as the location of the listing gets further away from Times Square. From the models, it is clear that prices of Airbnb listings decrease by 0.00149 (homes and apts), 0.00237 (private rooms), and 0.00555 (shared rooms) on average each mile further away from Times Square.

Even though we can clearly see a trend in our linear regressions, it is best to conduct hypothesis testing in order to determine if our results are valid and there is a significantly meaningful relationship between Airbnb prices and their distance away from high traffic locations, such as Times Square in New York City (Statistics How To).

Let’s ask the question: Do we reject the null hypothesis of no relationship between price and distance from Times Square?

Our answer: Yes, we reject the null hypothesis since the p-values for all three linear regressions are significantly smaller than 0.05. A p-value less than or equal to 0.05 means that the results for our data holds, that our data is repeatable, and that our results didn’t just happen by chance (Statistics How To).

You can read more about Linear Regression at CMSC 320 Linear Regression Lecture Notes by Professor Hector Corrada Bravo.