Hazardous Asteroids
2023 — A model that flags asteroids that could threaten Earth
01 Problem
Thousands of asteroids pass close to Earth, and planetary defense has limited telescope time to follow them. A model that names the dangerous ones tells astronomers where to look first.
The goal is a classifier that flags potentially hazardous asteroids from its orbit, its size, and how it passes Earth, all of which are already measured for each one.
02 Cleaning
NASA's near-Earth object data holds 4,687 close approaches with no duplicated rows and no missing values. The ID, name, orbit ID, and date columns are dropped since they label a row rather than describe the asteroid, and the equinox and orbiting body are dropped because every row has the same value.
That leaves 32 features, and the label becomes 1 for hazardous and 0 for harmless.
asteroids <- read.csv("datasets/nasa.csv", stringsAsFactors = T)
asteroids$Hazardous <- ifelse(asteroids$Hazardous == "True", 1, 0) %>% as.factor()
asteroid_ids <- asteroids$Neo.Reference.ID
asteroids <- asteroids %>%
select(-Close.Approach.Date,
-Orbit.Determination.Date,
-Neo.Reference.ID,
-Name,
-Orbit.ID)
asteroids <- asteroids %>% select(-Equinox,
-Orbiting.Body)03 Repeats
The 4,687 rows describe only 3,692 asteroids. Of those, 751 passed Earth more than once, up to 7 times, and each pass is its own row with the same orbit and the same label.
Split by row, the same asteroid would land in both training and testing, and the model would be graded on an asteroid it had already seen. So the asteroid IDs are kept aside for the split.
asteroids$Neo.Reference.ID %>% duplicated() %>% sum()04 Imbalance
Only 16.1% of close approaches are hazardous. A model that calls every asteroid harmless would already score 83.0% accuracy on the test set, so accuracy is not the metric that matters.
Sensitivity is the share of hazardous close approaches the model catches. A hazardous asteroid called harmless is the one mistake planetary defense cannot afford.
05 Split
The asteroids, not the rows, are split 85/15, stratified by the label, into 3,139 asteroids for training and 553 for testing. Every pass of an asteroid lands on the same side, and the test set keeps the real hazard rate.
Only after the split are the hazardous training rows upsampled, to 6,680 balanced rows. Upsampling first would copy the same asteroids into both sets.
asteroid_labels <- data.frame(id = asteroid_ids, Hazardous = asteroids$Hazardous) %>% distinct()
set.seed(1)
indices <- createDataPartition(y = asteroid_labels$Hazardous,
p = 0.85,
list = F)
train_ids <- asteroid_labels$id[indices]
is_train <- asteroid_ids %in% train_ids
train_data <- asteroids[is_train, ]
test_data <- asteroids[!is_train, ]
train_groups <- asteroid_ids[is_train]
up_train <- upSample(x = train_data %>% select(-Hazardous),
y = train_data$Hazardous,
yname = "Hazardous")06 Baselines
Naive Bayes catches 91.7% of hazardous close approaches and logistic regression 96.7%. Both pay with false alarms, and only 65.3% and 75.0% of the close approaches they flag are actually hazardous, and every false alarm costs telescope time to rule out.
model_nb <- naiveBayes(x = X_train,
y = y_train,
laplace = 1)
model_lgr <- glm(Hazardous ~ ., "binomial", up_train)
pred_lgr <- ifelse(predict(model_lgr, X_test, type = "response") > 0.5, 1, 0) %>% as.factor()
confusionMatrix(pred_lgr, y_test, positive = "1")07 Random forest
Danger depends on sharp thresholds in orbit and size, which a forest of 101 trees can learn and a single straight line cannot. It is tuned with 3-fold cross validation whose folds are split by asteroid too, with upsampling inside each fold, and 17 features per split wins.
It catches the most, 97.5% of hazardous close approaches, and 99.2% of its alarms are real, against 75.0% for logistic regression.
set.seed(1)
ctrl <- trainControl(method = "cv",
index = groupKFold(train_groups, k = 3),
sampling = "up")
model_rf <- train(Hazardous ~ .,
data = train_data,
method = "rf",
trControl = ctrl,
ntree = 101)08 Evaluation
On asteroids it has never seen, the forest catches 118 of the 121 hazardous close approaches and raises 1 false alarm among 592 harmless ones, a specificity of 99.8%.
pred_rf <- predict(model_rf, X_test)
confusionMatrix(pred_rf, y_test, positive = "1")09 ROC
The AUC rounds to 1.000, which means the forest ranks almost every hazardous close approach above every harmless one. Lowering the threshold would catch all 121 for 1 false alarm.
pred_rf_raw <- predict(model_rf, X_test, type = "prob")
roc_rf <- pROC::roc(response = y_test,
predictor = pred_rf_raw[, "1"],
levels = c("0", "1"),
direction = "<")
pROC::auc(roc_rf)10 Drivers
How close the orbit comes to Earth's, the minimum orbit intersection, is by far the strongest signal. The rest of the top 10 are all about the size of the asteroid, meaning its absolute magnitude and the diameter estimates NASA computes from it in kilometers, meters, miles, and feet.
That is NASA's own definition. An asteroid is potentially hazardous when its orbit comes within 0.05 AU of Earth's and its absolute magnitude is 22 or lower, which means it is big enough to do real damage. On the same test asteroids that rule alone scores 99.9% accuracy and catches 121 hazardous close approaches where the forest catches 118, so the forest is learning the definition back rather than finding a hidden pattern, and the rule is the better tool whenever the orbit is known.
varImp(model_rf)$importance %>%
arrange(desc(Overall)) %>%
head(10)
rule_moid <- 0.05
rule_magnitude <- 22
pred_rule <- ifelse(test_data$Minimum.Orbit.Intersection <= rule_moid &
test_data$Absolute.Magnitude <= rule_magnitude, 1, 0) %>% as.factor()
confusionMatrix(pred_rule, y_test, positive = "1")Hazardous Asteroids
Catches 97.5% of hazardous close approaches — AUC 1.00