Measures of Model Fit

We assessed the specification of our model using a posterior predictive check, which is common in Bayesian estimation. The posterior predictive check compares the distribution of predicted response times for each MCMC draw, where each draw represents one guess of what the model coefficients could be, to the actual distribution of response times in the data. Below, we have the posterior predictive check results for the model including neighborhood effects. Light blue lines represent the distributions of response times predicted by the model, while dark blue lines represent the actual response times. The predictions follow the model closely, giving strong support to the strength of this model.

In our case, the posterior predictive check did not immediately favor one model over the other. To compare our two models, we can check against information criterion. These summaries indicate a model’s ability to predict data not included in the sample the model was trained on. Strong performance on this check suggests that a model represents the data generation process well.

We assessed our models using the leave-one-out information criterion, which measures how well a model predicts each data point if that data point were removed from the data the model was trained on. In the following table, the model with a 0 indicates the best fit. Large negative numbers indicate that model performed much worse than the best model. Thus, while both models produced a reasonable distribution of predicted response times, the multilevel model was superior in predicting individual response times based on the specified predictors.

Spatial Autocorrelation

Because our data varies over space, we must attempt to account for spatial autocorrelation, or the general tendency for data points near each other to have similar values. Many models assume that no spatial autocorrelation exists, so its presence can produce misleading results. One way to assess this is to examine the model’s prediction accuracy across space. If we see that nearby areas have similar prediction errors, we have reason to be concerned about spatial autocorrelation. We therefore plotted the residuals from the neighborhood-level model across the EMS response region.

Visual inspection suggests some minor clustering in our model errors. An empirical measure for the magnitude of this effect is the local Moran’s I statistic. Large or small values of this statistic suggest the presence of autocorrelation in that area.

Since few areas have extreme values for the local Moran’s I statistic, we did not feel it necessary to generate a more complex model to address spatial autocorrelation. Our initial linear model suffered much more from spatial autocorrelation, which was our primary reason for abandoning it in favor of the neighborhood-level model. Evidently, the inclusion of neighborhoods was sufficient in accounting for much of the spatial autocorrelation present in the data.

Model Coefficients

Having selected the neighborhood level model as the appropriate model, we move on to interpretations of the resulting coefficients. The following plot displays the effects of each variable included in the model when compared to a reference patient (in this case, the reference is a black woman transported by ambulance with abuse of substance symptoms before COVID-19). The reference response time for this case is 9.9 minutes. As an example of coefficient interpretation, if the same woman instead had cardiovascular symptoms, but all other variables remained the same, we would expect an EMS unit to arrive in about 8.4 minutes, since the coefficient associated with cardiovascular symptoms is approximately 0.84.

From this, we can gather that there is no significant difference in how long service takes across different races and genders. While this is not the only area of EMS services where equity issues can occur, it does not appear that there is systematic inequity in response times.

However, we do see large differences in the response times for different types of symptoms. Some of these make sense, as a cardiovascular event is likely considered more time-sensitive than an alcohol or drug-related incident. We do see some variation in the response times to some symptoms across the beginning of the COVID-19 pandemic. Based on the plot above, we can be fairly confident that GI/GU cases take longer to serve during the COVID-19 era than before. Unfortunately, due to the relatively few incidents after March 15th, our estimates are not highly precise. For most of these symptoms, we cannot say for sure whether response times have increased or decreased. These results still serve to point out possible trends worth monitoring as the pandemic continues.

Because this is a neighborhood-level model, the expected change in response time due to being in a particular neighborhood is also interpretable (the interpretation of units is the same as explained above). When plotting response times across the region, we see that rural areas are expected to wait longer for emergency medical services than their urban counterparts. However, these results should be interpreted with caution because of our inability directly control for distance traveled. As neighborhood or census tract regions get larger, this deficiency in our model becomes more and more meaningful. Therefore, comparing smaller regions within Charlottesville to those in Albemarle is suspect. Still, differences within similarly-sized regions may be meaningful.