Evaluating Predictions of AI Use and Actual Use
Evaluating the accuracy of predictions about artificial intelligence (AI) use and outcomes is important given their potential influence on policy and decision-making. A new U.S. Bureau of Economic Analysis working paper studies how closely businesses' expectations about future AI use translated into their desired outcomes. This new paper, by Tina Highfill and Jon D. Samuels, analyzes survey responses spanning multiple years and approaches the analysis in two ways.
The first approach looks at surveys showing industry expectations of AI use 6 months into the future and compares those predictions to actual use 6 months later to see how well businesses anticipated their use of AI. The paper finds that, on average, businesses predicted their future use of AI within 2 percentage points of actual use, though prediction accuracy varied greatly across industries. The findings from this approach imply that AI adoption initially happened slower than expected, was followed by a short period of growth that was faster than expected, and more recently has been close to expected rates.
The second approach looks at the motivations businesses reported for adopting AI, such as automating tasks performed by labor or expanding the range of goods or services, and compares those motivations to observed economic measures years later to see if these motivations translated into the desired outcomes. The paper finds that the motivations of early AI adopters were sometimes realized, though oftentimes the authors did not find clear linkages between AI adoption and the original motivation.
Their results indicate that businesses' expectations about their use of AI are not always realized in future outcomes. Businesses may be getting better at anticipating their future use of AI, but a longer time series of data is needed to confirm this trend. The paper thus concludes that the uncertainty and inconsistency of business expectations about AI use should be taken into consideration when forecasting the economic impacts of AI.
The authors also note it will be important for future research to continue to develop relevant data for measuring the ongoing economic impact of AI, pointing to the murky conceptual link they find between stated motivations for using AI and economic outcomes. It is possible early adopters of AI did achieve some of their original goals, but the authors are unable to isolate the effects given the coarseness of the available data.
