What Is Big Data?
And What Does it Mean for Trucking?
What’s the weather going to be tomorrow?
I read an article in Time magazine this week that helped me understand the concept of “big data.” The Internet gives this definition: “extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions.” The topic is relevant today because digital tools are dramatically increasing our ability to collect and then analyze such data. Time helped me understand it through an article on weather forecasting. Think about it. Our planet’s weather is a hideously complex phenomenon beset with violent extremes and subject to many, many interactions. And yet, Time pointed out that every ten years we get one day better in forecasting the weather. So a six-day forecast today is as good as a five-day forecast in 2009 and a three-day forecast in 1989. This progress is the result of an explosion in weather data collection and the invention of supercomputers to analyze that data. That’s big data at its best: gaining understanding of large complex systems when such understanding is IMPORTANT! Used to be hurricanes were surprises – no more.
What’s the trucking connection?
There is a direct parallel here to trucking. Our market is the sum of two to three million matching decisions a day and the trillions of daily decisions that move those trucks along the highways. Understanding such behaviors has profound implications for safety, supply chain efficiency, and profit. As with the weather, new data and new analysis are improving our ability to forecast. Truckstop.com is a good example. Last week, for instance, it collected information from its customers regarding 1.7 million loads and trucks. Moreover, 97 million customers searched its databases, revealing all the more about spot market behavior. As a result of Truckstop.com’s data, we now have weekly data on spot market conditions, something we only guessed at ten years ago.
But does it help forecasting?
In 2011, using old-fashioned methods, I forecast a dramatic tightening of market conditions sometime in 2017 or 2018, depending on when FMCSA would roll out the ELD mandate. Once they set the date for December 2017, we knew roughly when this important event would occur, subject to the usual run-up and run-down processes. We also knew that human nature would cause a series of over-reactions that would make the problem worse.
The market is adjusting to one such over-reaction as I write this. The industry will take delivery of something around 300,000 new Class 8 vehicles this year, far more than are needed now that the crisis is over. What we didn’t know prior to 2017 was the degree of the crisis. We knew it could be very tight, but the relatively mild conditions of the 2014 tightness left us with considerable doubt. Would contract price increases be the relatively benign 5-7% or the more dangerous 9-11%? Of course, spot prices increases would have much bigger multiples. Then, in 2017 Truckstop.com’s big data gave us early warning of a bad hurricane when its second and third quarter spot numbers came in at 14% and 21% higher than year-earlier numbers. In the old days, we would get warning signs nine to twelve months later. So a three-quarter-out 2019 trucking price forecast is now as good as a 2009 one-quarter-out forecast was.
Do people use the new forecasts?
Here lies the nub of the problem. In September of 2017, well within the new forecasting window, I presented my (still very conservative) contract price forecast of 6-8% to a group of very skeptical shippers who still had average increases of 2-4% on their books. I talked about the possibility of much worse and received a “nice theory, Noël, but we will wait and see” response from the shippers, shippers about to receive the largest pricing and service shock of their careers.
Of course, part of the problem was my inability to show the new data in a compelling fashion (spot rates were already up 20% or more). People, raised in a climate of great forecasting uncertainty, remained skeptical. Remember how we used to scoff at the weather forecasts? TV weathermen were often cast as lovable clowns. Now the 10:00 news leads with dramatic pictures of weather radar. I suggest that our industry is moving the same way. People who saw the recent crisis coming early than their competitors clearly benefited. So did the people who saw the draw-back from that tightness. The same big-data insight allowed me to give plenty of warning for the fall in spot prices, and the delayed-but-inevitable fall in contract prices.
Where are we headed with this?
There are two clear messages among the many possibilities. First, the explosion of data will continue. Consider, for instance, the growing requirement for real-time location information on each truck. Owner-operators hate it, but the customer is always right – and the customers want to know where their shipments are. Think of the possibilities that would come from collecting those billions of location data points. We would know to the minute what the trucking volume is on every highway segment in the country. The Staties would love it. So would we transportation economists. Some fleets are already using similar data to continually monitor their drivers’ performances, discovering bad habits before they turn into accidents. Supply chains will eventually integrate such movement data into their fulfillment algorithms, progressively shaving time and waste from their systems. My new car already has a system to sense if I am falling asleep at the wheel. Big data on me and millions of other drivers will soon be making such analysis (and warning) a standard feature on trucks – and cars. Perhaps, the ultimate expression of big data will come in blockchain applications that store and make available to all of our business partners a multitude of business behaviors. “You claim you are reliable? Let’s look at the data and see.”
What do I do?
So here’s what I take from the weather example. Big data is real. It is here and progressively adding value in an important arena. It will also increasingly be part of our every-day trucking lives; as with weather, it already is in helping me to plan my gardening this afternoon, and my golf four days out.
The survivors in this time of rapid change for supply chains will have learned how to acquire and use the insights from big data. Sure, there will be a list of success characteristics, but using data will be close to the top. It is time, at a minimum, to acquire regular sources of pricing and capacity data and the experts who know how to use the data. Finally, the Time article points out that, as big data demonstrates its value, it will become something that people will collect, hoard, and charge for. You will be wise to favor “open source” suppliers like Truckstop.com who offer their data to all comers over suppliers that used their data to force you to subordinate your operations to their services.
As you consider such issues, remember that the data you provide to vendors of any flavor is valuable. As those vendors consolidate and manipulate your data into big data sets, they should recognize your value in their pricing and services. Ascertain, also, that such business partners are protecting your business privacy. Their summing up of many private inputs into market overviews has great value, as long as no one else can see your individual data. Yeah, this is just one more complexity to deal with. But don’t you want to know when the next hurricane is going to strike?