A Semi-Dynamic Bus Routing Infrastructure based on MBTA Bus Data

As traffic congestion continues growing in urban areas, more and more cities have realized that investment priority should be given to public transport modes, such as bus transit systems (BRT) instead of personal vehicles. Simply put, in congested cities, public transport modes are more efficient than personal vehicles in terms of carrying and moving people around. As city populations grow and as their economic bases shift and evolve, their housing sector adjusts, even more vehicles are expected to enter the roads each day, creating more traffic congestion. The 2012 Urban Mobility Report states that, the lack of public transportation services would have cost commuters an additional 865 million hours of delay. With growing urban population numbers, this number undoubtedly stands higher today (National Express). On average, expanding and optimizing transit services produced an economic benefit of roughly $45 million a year by connecting urban areas in the US. There is no doubt that expanding public transportation use is key to reducing traffic congestion. One of the effective solutions to decreasing the pressure on city streets and highways is maintaining a robust and efficient public transportation system.

Public transportation in American cities has been poorly funded and provided sketchy service. According to the Washington Post[1], in the past two years, nearly every US city, with a few notable exceptions, have reported a decline in their transit ridership rates. Data has shown that overall transit ridership rates hit an all-time low in 2017 since 2005, and bus ridership alone fell 5 percent. This steady decline has been attributed mostly to reliability issues and the rise of convenient alternatives such as rideshare. According to American Prospects Magazine [2], in New York City, there are approximately 68,000 Uber and Lyft cars, about five times as many as yellow taxis, which has caused average speeds during business hours in Manhattan’s core to drop to a crawl—about 5 to 6 miles per hour, 15 percent to 23 percent slower than in 2010, before Uber’s emergence. The convenience of alternatives has contributed to an increase in competition, leading to the upsurge of ridesharing companies such as Uber and Lyft and further increasing urban traffic congestion to a very high level. These rideshare companies are using modern technologies driven by sophisticated data analytics to gain significant market share. While most of the public transport system is still at the level of basic GPS tracking and either lack implementation of more sophisticated technologies or are very slow in adapting and implementing them. According to an annual overview of public transit usage, transit ridership fell in 31 of 35 major metropolitan areas in the US last year, including the seven cities that serve the majority of riders, with losses largely stemming from buses.

In order to compete in our dynamically evolving society, there is the need for BRT to incorporate data analytics into their overall mode of operation by improving efficiency and effectiveness to optimally gain public confidence and essentially compete in the urban transportation business. In view of these problems, there is the need to find the middle-ground between the highly dynamic ride sharing technology and the static bus routing system.

The use of big data in public bus transport system is still at its infancy. According to Welch and Widita[3], public transportation studies using big data began to emerge around 2013. The sources of data used in these studies are mainly GPS points and traces, smart card data, automated data such as automated passenger counts (APC), automated fare control (AFC), automated vehicle location (AVL), sensor data, mobile phone data, web data, and social media data. This type of data could produce an abundance of data daily and can be used to study anything from the behavior of individual passengers to the functioning of a large public transit system. There exist many ambiguities related to what constitutes big data, the ethical implications of big data collection and application, and how to best utilize the emerging data sets. However, within the transportation literature, there is a growing emphasis on developing sources of commonly collected public transportation data into more powerful analytical tools. A commonly held belief is that application of big data to transportation problems will yield new insights previously unattainable through traditional transportation data sets.

Most of the previous studies have incorporated purely dynamic rideshare application models or GPS based fixed-route model. There has been limited research in the area of data-driven, semi-dynamic public transportation modeling. The lack of strategic planning and evaluation in the semi-flexible system, coupled with the lack of extensive literature review has created a huge gap in research in the public transportation industry. To bridge this gap, this research proposes to use data analytical techniques, based on existing data of a Boston bus line, to create a semi-dynamic routing framework which can improve the efficiency of that bus system whilst optimally increasing passenger satisfaction by way of increasing passenger pick-up effectiveness.

This will be achieved by developing necessary metrics and heuristics from the data in order to achieve a responsive routing framework, which reflects the trade-off between time efficiency and boarding effectiveness, that can in turn be tuned by the bus authority. We hope to demonstrate how rudimentary data collection and analytics, which is rather non-invasive, can establish a semi-dynamic routing mechanism that is much smarter and effective than the existing static ones.

We hope also with this to contribute to the ever-growing repository of research in transportation analytics and hopefully encourage such data collection initiatives from bus transportation authorities. A data-driven method of bus schedule planning and bus allocation is demonstrated in the work.

According to a 2009 USDOT-sponsored report, only 53% of agencies monitor bus route performance on a monthly basis, although the survey did not distinguish between monitoring for the purpose of performance reporting and monitoring for service evaluation[4]. This makes it very difficult to obtain quality data to perform insightful analysis. For decades, transportation planning analysis has consistently relied on manually collected data obtained largely through active solicitation, particularly for understanding transportation users’ behavior, e.g. household travel survey[5]. This particular type of data, which is usually small scale and collected fairly infrequently (i.e. every 5–20 years), tends to be developed deliberately and intentionally to conduct transportation planning, evaluate specific transportation policy, and address relevant research purposes but not to improve efficiency and effectiveness. The lack of appropriate data was one of the major hurdles encountered during the data collection process for this study.

In all, we first seek to assess the latest literature and work in this field, related to such transportation analytics, to gauge the progress in this growing field. Second, we analyze the data wrangling methods and algorithms used to utilize the raw event data from the bus system. We then carefully assess some importance metrics that we calculated and used from the data such as station stopping probability, which were elemental in our routing framework. Further, we present how artificial passenger data was generate and then we dissect the routing infrastructure fueled by the respective data. Afterwards, we present a comprehensive evaluation composed of three critical components: framework responsiveness, ”dry run” comparisons, and bus allocation optimization. Finally, we discuss the implications of our results and the pathway for future work and research.

The advent of open data from city governments to public transportation has enabled researchers and enthusiasts alike to utilize such valuable data conceive and analyze better applications and systems. Our research, likewise, has benefited from this openness as MBTA (Massachusetts Bay Transportation Authority) has been diligently publishing transportation event data from their API for developers[21]. We were interested in the bus data from the MBTA and while metro data is more abundant, the MBTA had recently begun collecting and publishing bus metric data. The only available bus data regarded two bus lines (712 and 713) as shown in Figure 1, whose routes take place in Winthrop. While limited in scope, the benefit of this bus line stemmed from its geographic simplicity, which was ideal an initial infrastructure.

The event data that we pinged from the public API gave us the necessary granularity in order to proceed with our research. Namely, we needed to know at what times the bus was arriving at the respective stations and a sense of what particular trip it belongs to. The relevant features that we used and their descriptions can be seen in Table 1.

Schedule times (in order to determine lateness) was also obtained by scraping the schedules times on the respective website. Scraping was efficient, however, schedule times are erased quite quickly (we assume the schedule times are not static for a long period of time and rather may differ or change at any day).

A central part to route optimizations is applying shortcuts that can cut down on trip time given the bypassed stations are to be skipped. The process of conceiving possible shortcuts demands that we be cognizant of the streets and traffic patterns around the bus route. In our case, the geography of Winthrop yields not many opportunities for shortcuts, however, several shortcuts still exist throughout the entire bus route.

In determining which shortcuts can be taken we had to ensure that 1) the route does indeed decrease the trip time compared to the original route 2) there are no street restrictions (e.g. one-way). Second, we had to determine how long those shortcuts would take. We believed the most accurate way was to extrapolate known trip times from similar length routes in our system and apply them for the shortcuts.

In the data wrangling process, linking events together is a careful and critical process to stitch data together rendering it for use. We had to apply event linking (described in Algorithm 1) to our processes in order to obtain all the data components that we used for our algorithms. This linking is necessitated especially when there was missing data or errors; in essence, a sanity check for the data that we were wrangling together. It was especially useful for discarding outlier events such as if a bus broke down, which we would not want to include in our historical data aggregation.

From a passenger point of view, bus lateness is the primary indicator of bus route performance and efficiency. This indicator was not readily available in the raw MBTA event data; thus, some work had to be done to construct this important metric. Firstly, the schedules time had to be web-scraped from the MBTA website. This was not so difficult and involved routine web-scraping techniques with the BeautifulSoup python package. With that, the station departures had to be linked with the scheduled departures - the event linking algorithm helped link these times together. After the linkage, lateness was as expected: simply the difference between the actual departure and the scheduled departure. Figure 2 shows the lateness for departures throughout a particular weekday for one of the stops on the bus route.

A routine part of bus trips is the idle time of the bus in the respective station. In other words, the difference between the departure of the bus and its arrival at a given station. In fact, idle time can be rather correlative of passenger boarding and disembarking quantities at stations, which is something not collected by the bus. Firstly though, the idle time is needed in order to incorporate into our proposed route times. We do this at a granular level of idle time per station per hour. For example, a given station may have higher idle times at rush hour when many people are boarding. The event linking algorithm is utilized in this process to link respective events together and/or exclude data errors. In the end, we are able to use the median Idle Time for a given station for a given hour from the aggregated, linked data for our routing purposes.

To accurately calculate our proposed route times, it is imperative to know trip times between all stations from available historical data. This computation is straight-forward as the trip time is simply the difference between the given station’s arrival and the departure from the preceding station. However, missing data and data errors can hamper data linkage (e.g. the arrival at a given station belongs to a different trip coupled with the departure from its preceding station). Thus, the event linking algorithm was applied to account for such possible errors.

The second issue that had to be addressed was completely missing data. Given our granularity of having the trip time per station pair per hour, there were some station pairs that had either no trips at certain hours or even no trips at all. The imputation method chosen was to impute the station pair’s overall median trip time for hours during which there was no trip data for that station pair. For a few station pairs that had no data data at all (stations towards the end of the route), the median trip time was imputed for all their hours based on the trip time of station pairs with similar distances.

The key probability that we needed to calculate from the historical event data collected was the probability that a given bus will stop at a given station at a given hour (example shown in figure 3). This would optimize our routing by providing a quantitative value for a station’s ”importance” given past trips. The probability is calculated with the following equation:

One important heuristic is applied to determine the time at which the bus skipped a given station:

This metric further introduces a new hyper-parameter to represent the probability threshold above which a bus on a given proposed route will stop at stations. Let us refer to this hyper-parameter as $t$. While during testing, we are able to choose different values for our $t$ as part of paramatarization, for a rudimentary baseline we can have $t$ to be set as the 25th percentile of the stopping probabilities for every given hour; thus, having a different $t$ for each hour. We can further constrain our paramatarization to be the percentile value under which stations will be skipped. We can refer to this more constrained parameter as $t_{p}$.

Let us keep in mind that this probability will very likely correlate with passenger boarding at each station, which is data not available from our source.

While our steps highlight ways that can make the route more time efficient based on shortcuts and efficient ways of station skipping; till now, we have not taken into account passenger boarding. Firstly, data on passenger boarding at each station during each trip would have been extremely useful for our work, however, such data was not available. Data that was available was the average number of boardings for each departure time (for which there is scheduled bus trip). Using this aggregate data coupled with our stopping probabilities and population density, we were able to smartly generate artificial passenger boarding data for the stations of the route at a given time.

Our generation works as such:

The generated passenger data gives us a very important dimension to our problem, since now we have the added optimization criteria of effectiveness. In reality, boarding data is obviously never known before a route, however, this data gives us an important evaluation metric for how effective our route is. The trade-off between this effectiveness and time efficiency is what we need to balance in our routing framework.

In practical application, before a route starts, the passengers waiting at stations will obviously not be known. However, historical data of passenger boardings at each station for the particular trip departure time can inform the routing. In order to replicate this historical data, we run a number of simulations (e.g. 100) of the passenger data generation and then come up with a final aggregation of the numbers to produce a ”passenger pickup percentage” for each station. With this we can introduce a new parameter: Minimum Passenger Aggregation ($PA_{min}$), which denotes the minimum percentage aggregate of passenger pickup that you want to accumulate during a trip. In short, this parameter helps to optimize the route for boarding effectiveness. Figure 4 demonstrates this aggregate percentage for a given departure time after 100 simulations, which in practice would be 100 days of historical data for trips departing at that time.

It is important to assess how our routing framework reacts to differing parameter values for our two main parameters $t_{p}$ and $PA_{min}$ for a weekday in October. For population generation 100 simulations are run. The figures (Fig. 6, Fig. 7, Fig. 8, Fig. 9) for two selected departing times show the effect.

A way to fairly evaluate our semi-dynamic routing system to the existing static routing of the bus lines is to perform a ’dry run’ for a few departure times. Given that it is not feasible to test this framework in the real-world, we must rely on simulations. Please note that we opt not to compare total trip times, simply because it would not be a fair comparison. We focus on passenger pickup as our evaluation metric. Our dry run has the following conditions:

• •

  Total passenger number inferred from available average of total passengers for given time.

• •

  100 simulations run with passenger distr. generated in each simulation.

• •

  Each route ”picks up” passengers for the generated population at that time and iteration.

• •

  Pick-up percentages is averaged over the 100 simulations.

• •

  Number of stops taken is recorded.

• •

  Semi-Dynamic parameters may be changed. This is a natural algorithmic advantage of the semi-dynamic routing, however, the fairness of the test is not affected.

• •

  Number of stops (Num_Stops) excludes the origin station and the terminus station because those stops are mandatory (either way this doesn’t affect our comparison).

The bus authority may very well want to allocate its buses in an optimal manner. A critical metric that can reflect upon a bus system’s performance is passenger waiting time at stations. Naturally, the authority will want to minimize this time to an acceptable level, which henceforth will mean that the timing of the bus allocation needs to be enhanced. A useful result from our framework has been the capability for the second bus’s departure to be determined based on the set maximum median passenger waiting time. Usually, there is a direct relationship between the set waiting time and how many minutes after the bus will depart, however, because of the semi-dynamic nature of our scheduling; this can differ based on how our routing for the particular time. In short, the process that calculates this can be summarized as such and example results are displayed in Table 3.

• •

  Record bus departure times for each station for trip A

• •

  Propose trip B start 1 minute after trip A start (route semi-dynamically calculated for that start time).

• •

  See if proposed trip B arrivals times at its stations would violate the inputted median passenger waiting time.

  $$\frac{tripB\_arrival-tripA\_departure}{2}$$

• •

  If violated: optimal departure for trip B is set to 1 minute before the start time that resulted in the violation.

• •

  If not violated: Continue loop with 1 minute increments of trip B start time until there is a violation.

Our results show the large potential that such a framework can have for making such bus routes more data driven; a welcome and much needed departure from the relatively static nature of how bus routing and scheduling is done today. First, in Fig. 6, Fig. 7, Fig. 8, and Fig. 9), we demonstrate how two important parameters ($t_{p}$ and $PA_{min}$) can be tuned based on the needs of the bus authority and what affect the parameter choices can have on the number of stops taken by the trip and the full trip times. For example, take note of how a high passenger pick-up minimum increases trip time, while a very high percentile value produces less trip times because the higher value signifies we are seeking only to stop at the most ”important” stops thus reducing the stops. This trade-off interplay is what we wanted in our framework. We see a fully responsive framework that can be used to aid routing decisions based on the constraints and needs of the particular trip. With this research, we want to recognize the fact that every route can have its unique constraints and challenges, which can be most efficiently addressed with such a semi-dynamic framework.

While we focused on one particular line and direction for our research, in reality, we believe we have produced a framework general enough to apply to other routes given the rudimentary data is available. MBTA has recently begun publishing such granular bus event data and it was only for two bus lines (712 and 713). We hope as bus authorities increase the amount of data collection; it can enhance the effectiveness of such dynamic approaches and improve frameworks such as ours.

Furthermore, it must be said that it is not feasible to test our framework in the real-world given the necessary resources and permissions needed. That being said, it is important to at least have some form of evaluation to compare how our routing would do with the existing static routing. Thus, we conceived the ”dry run” comparisons as the fairest way possible to conduct simulated comparisons between the two systems. While the results expectedly show our system to produce more efficient and effective routes (by pick up % and number of stops), only real-world testing can give conclusive confidence for our system. We believe however that the dry runs provide evidence of the strength and potential of our framework and it should be highlighted that each of the runs were from 100 simulations. One metric that would have been interesting to compare would have been trip times, however, without testing this is not possible and in general, we did not use trip time as an evaluation metric given there was no way to assess how much in reality trip times would be affected. Moreover, we were using the historical trip time data from the existing data. The number of stops would be the closest metric to that.

Finally, we present the bus allocation feature as a result of our produced framework. It shows the potential that the tool can have for bus authorities that want to more efficiently allocate buses based on a given metric; in this case, passenger waiting time. A framework that actively measures and updates the metrics that we have produced can help bus authorities make more informed decisions and have a direct view of the ”health” of the given routes. Certain changes such as removing stations, increasing number of buses, moving stations, and so forth, are ones that the bus authority would make based on such data. Our framework will respond to any such change and thus inform about how effective in reality those changes were.

The granularity of our framework also affords the changes that occur throughout the year based on season and holiday. For example, maybe passenger use during the cold month of December differs from passenger use from the hotter months. Such trends can be absorbed by our algorithms; if of course, historical data is used based on month.

In all, the results from our work have shown how the coupling of more granular bus event collection and straight-forward algorithmic analysis of that data can produce a much smarter infrastructure. Our work doesn’t utilize complicated machine learning approaches, rather produces very rudimentary and essential metrics for bus routing and then creates a framework to optimize and give the authority the tools to optimize the routing so that a semi-dynamic routing can be achieved.