PDRI/MLP/USAID-DRG Policy Brief 3: Forecasting High-Level Travel Advisories with Machine Learning for Peace Data

This policy brief was prepared by Mahda Soltani, Jeremy Springman, and Erik Wibbels from the University of Pennsylvania. It highlights findings from a study conducted under the Machine Learning for Peace (MLP) project, funded by USAID’s Center for Democracy, Human Rights, and Governance (DRG).

Key Takeaways:

• Forecasting Travel Warnings: The MLP team has developed models that accurately predict high-level travel advisories (levels 3 and 4) issued by the U.S. Department of State (DOS) up to six months in advance.
• Advanced Warning Benefits: Early warnings improve the ability of policymakers to prepare for crises that threaten U.S. citizens abroad, reducing the costs and risks associated with last-minute responses.
• High Predictive Performance: Using a combination of machine learning tools and high-frequency civic space data, the models achieved an ROC-AUC of 0.87, reflecting strong accuracy.
• Key Predictors: Events such as states of emergency, protests, election activity, and disasters emerged as the most significant indicators of future travel warnings.

Introduction:

The U.S. Department of State (DOS) issues travel advisories in response to crises like civil unrest, natural disasters, or armed conflict, often with little warning. These warnings impose significant operational and financial burdens, particularly in severe cases requiring embassy closures or evacuations. This study evaluates whether MLP’s civic space data can enhance DOS’s capacity to predict such crises and issue timely warnings.

The Challenge:

Predicting travel advisories is a rare event problem, complicated by limited lead time and data availability. Serious advisories disrupt DOS operations, emphasizing the need for proactive forecasting to mitigate impacts on U.S. citizens and government resources.

Approach:

The research focused on:

• Collecting monthly DOS travel advisory data for 60 developing countries (2012–2023).
• Combining this data with MLP’s tracking of 20 political event categories (e.g., protests, election irregularities, martial law) sourced through machine learning-based analysis of domestic media outlets in 40 languages.
• Building predictive models using LightGBM (a machine learning algorithm) and temporal cross-validation to ensure robust forecasting of advisory onsets (new warnings).

Two metrics evaluated model performance:

• ROC-AUC: Measures the model’s ability to distinguish between onset and non-onset months (achieved 0.87).
• AUPRC: Reflects precision and recall for rare events (achieved 0.31 for monthly predictions and 0.57 for quarterly forecasts).

Findings:

1. High Accuracy: The model correctly predicted 87% of onset months (ROC-AUC = 0.87) and significantly outperformed random guessing.
2. Prediction Trade-Offs: Adjusting model thresholds allows policymakers to prioritize precision (avoiding false alarms) or recall (capturing more true warnings).
3. Predictor Variables: Political events like martial law declarations, election activity, and disasters were most predictive of future advisories.

Policy Implications:

1. Early Warning Systems: The integration of MLP forecasting tools into DOS operations can enhance crisis preparedness and improve the timeliness of advisories.
2. Customizable Models: Policymakers can tailor thresholds to their tolerance for false positives and negatives, balancing trust and actionability.
3. Global Monitoring: High-frequency civic space data provides actionable insights into underlying political conditions that precipitate advisories.

Acknowledgments:

This brief was made possible through support from USAID’s Center for Democracy, Human Rights, and Governance. The authors acknowledge the contributions of the MLP team and its research collaborators.