Taxi Fare Pricing
2023 — A model that predicts how much a taxi ride will cost
01 Problem
A rider who sees the fare before the ride starts can decide whether to take it, and a price that matches what the meter would have charged builds the trust that brings them back.
The goal is a model that predicts the fare of a New York taxi ride when it is booked, from where it starts and ends, how many people ride, and when.
02 Data
The dataset holds 200,000 New York City taxi trips from 2009 to 2015, sampled from the Kaggle New York City Taxi Fare Prediction competition, which is built on records from the Taxi and Limousine Commission.
It is often shared as the Uber Fares dataset, but its trips start in 2009, two years before Uber arrived in New York, so the fares are taxi meter fares.
03 Cleaning
The data has no duplicated rows and a single trip without a dropoff location, which is removed. The row ID and the key, which repeats the pickup time, are dropped.
The summary exposes impossible records, such as negative fares, a $499 fare, trips with no passengers or 208 of them, and one trip of 16,409 miles. Fares under $1, empty trips, trips shorter than 0.1 mile, and outliers in fare, passengers, and distance are removed, which leaves rides of up to 6.5 miles and $22.10.
taxi <- taxi %>%
filter(complete.cases(.)) %>%
select(-X,
-key)
taxi <- taxi %>%
filter(fare_amount >= 1) %>%
killOutliers(fare_amount)
taxi <- taxi %>%
filter(passenger_count >= 1,
distance >= 0.1) %>%
killOutliers(passenger_count) %>%
killOutliers(distance)04 Features
The pickup and dropoff coordinates become a single straight-line distance in miles, measured with the haversine formula across the curve of the earth.
Three features come out of the pickup time, which are the hour, whether the trip starts on a Saturday or Sunday, and the year.
taxi <- taxi %>%
mutate(distance = getHaversine(pickup_latitude, pickup_longitude, dropoff_latitude, dropoff_longitude)) %>%
select(-pickup_latitude,
-pickup_longitude,
-dropoff_latitude,
-dropoff_longitude)
taxi <- taxi %>%
mutate(pickup_datetime = ymd_hms(pickup_datetime),
hour = as.factor(hour(pickup_datetime)),
is_weekend = as.factor(ifelse(wday(pickup_datetime, week_start = 1) > 5, 1, 0)),
year = as.factor(year(pickup_datetime))) %>%
select(-pickup_datetime)05 Fare
After cleaning, 149,102 trips remain. Most rides are short, and the most common fare is about $6.
06 Rate increase
New York raised its taxi meter rate in September 2012. A mile that cost a median of $3.71 in 2009 cost $4.53 in 2015, so a model that ignores the year prices old and new trips the same and misses on both.
fare_per_mile <- taxi %>%
group_by(year) %>%
summarise(fare_per_mile = median(fare_amount / distance))07 Split
The trips are split 80/20 into 119,281 for training and 29,821 for testing. Every score below is measured on trips the models have never seen.
set.seed(1)
indices <- sample(nrow(taxi), nrow(taxi) * 0.8)
train_data <- taxi[indices, ]
test_data <- taxi[-indices, ]08 Linear regression
A linear regression explains 69% of the variation in fares and misses by $1.47 on average. Each mile adds $2.12 to the fare, and every other feature adds or removes a fixed amount that can be read straight off the model.
For a 1.9-mile weekday ride in 2015, the model charges $7.69 at 06:00 and $9.38 at 12:00.
model_lr <- lm(fare_amount ~ ., train_data)
pred_lr <- predict(model_lr, X_test)09 Neural network
The hour, weekend, and year are one-hot encoded for a feed-forward network with three hidden layers of 16, 8, and 4 ReLU units and a single linear output, trained with Adam on mean squared error.
The first epoch starts from random weights at an error of 14.4. From the second epoch on, the training and validation errors stay side by side, so the network generalizes rather than memorizes, and it misses by $1.40 on average.
model_fnn <- keras_model_sequential() %>%
layer_dense(units = 16, activation = "relu", input_shape = ncol(X_train), name = "hidden_1") %>%
layer_dense(units = 8, activation = "relu", name = "hidden_2") %>%
layer_dense(units = 4, activation = "relu", name = "hidden_3") %>%
layer_dense(units = 1, activation = "linear", name = "output")
model_fnn %>% compile(
loss = "mean_squared_error",
optimizer = optimizer_adam(learning_rate = 0.001),
metrics = "mean_absolute_error")
history <- model_fnn %>% fit(
X_train, y_train,
epochs = 25,
batch_size = 128,
validation_split = 0.2)10 Nearest neighbours
A k-nearest-neighbour regressor averages the fares of the 345 most similar training trips. The test trips are scaled with the mean and spread of the training trips, so nothing about them leaks into the model.
It misses by $1.97, the worst of the three. After scaling, every one-hot hour and year column weighs as much as distance when measuring similarity, so neighbours are picked by when a trip happened as much as by how far it went.
X_train_scaled <- X_train %>% scale()
X_test_scaled <- X_test %>% scale(center = attr(X_train_scaled, "scaled:center"),
scale = attr(X_train_scaled, "scaled:scale"))
pred_knn <- knn.reg(train = X_train_scaled,
test = X_test_scaled,
y = y_train,
k = round(sqrt(nrow(X_train))))$pred11 Selection
The neural network is the most accurate, at $1.40 against $1.47 for the linear regression. That is 7 cents a ride.
The linear regression wins anyway. A fare is a promise to the rider, and the regression can show where every dollar of it comes from, so much per mile, so much for the hour, and so much for the year. The network cannot, and 7 cents is not worth a price nobody can explain.
12 Limits
Removing outliers keeps only rides of up to 6.5 miles and $22.10, so the model prices short city trips and knows nothing about airport runs or flat fares.
Distance is a straight line, not the road the taxi takes, and traffic or weather are not in the data at all, which leaves room for the next version.
Taxi Fare Pricing
Prices a ride within $1.47 of the real fare on average