
01/01/2026
7 mins
Why today's transformers struggle with the two things finance and resources care about most: time and space:
I didn't set out to criticize transformers. I set out to find the best forecasting engine I could.
That search led me down a rabbit hole of every predictive system I could get my hands on. I studied competitors in commodity forecasting and noticed the same pattern again and again: their models were respectable over days and weeks, and then fell apart. Ask where a spot price would be a year or two out, and the forecasts became confident-looking noise.
I've always been drawn to game strategy and to modeling probabilistic, stochastic systems, the kind where randomness isn't a nuisance to filter out but part of the structure. That interest deepened after I moved to Silicon Valley. I became friends with frontier researchers working on everything from single-agent to multi-agent AI systems, and on interpretable architectures and training methods meant to align AI with human and economic values.
The turning point came at a virtual cell training event at an AGI House hackathon, connected to the Arc Institute's Virtual Cell Challenge. The challenge is an open, recurring competition to build models that predict how cells respond to genetic or chemical perturbations they've never seen. Watching that work, I became interested in whether transfer learning could help uncover the underlying structure of complex systems, biological or financial.
Working on this with friends, and alone, I kept running into the same gap. The transformer, the architecture behind nearly every AI headline of the last few years, struggles with two kinds of data that matter enormously in finance and natural resources: space and time.
A model built for meaning, applied to motion
Transformers were designed for language. Their core mechanism, attention, asks: which other tokens in this sequence are relevant to this one? That works well for words, because a word carries meaning on its own.
A single price tick doesn't. A number like 74.12 means nothing until you know what came before it, how fast it got there, and what regime the market is in.
In 2023, a team of researchers made this argument directly in a paper called "Are Transformers Effective for Time Series Forecasting?" They argued that attention is built to find semantic relationships, while time series require temporal relationships across an ordered set of continuous points, and that attention inevitably loses some of that ordering information. To test this, they compared elaborate transformer forecasters against a deliberately simple one-layer linear model. The simple model won on all nine real-world datasets they tested, often by a large margin.
To be fair, the story didn't end there. A follow-up architecture called PatchTST grouped time series into chunks, or "patches," before feeding them to the transformer, and its authors concluded that transformers do work for time series after all. That's an important lesson in itself. The architecture isn't hopeless, but its success depends heavily on how you represent time, and representing time was never what it was built for.
The problem underneath the architecture
Here's where I had to be honest with myself. Even a perfect time series architecture runs into a wall with financial data, because the data itself is hostile to forecasting.
This year, a benchmark tested the leading pretrained time series foundation models, including Google's TimesFM, Amazon's Chronos, and Salesforce's Moirai, on stock returns. The authors described financial returns as a difficult test case because of low signal-to-noise ratios, structural breaks, heavy tails, and weak persistence. Their conclusion: these models are useful starting points that lower development costs, but they are not universal engines for reliable alpha in realistic deployment.
A separate study found that off-the-shelf foundation models performed poorly on global market data, while models pre-trained on financial data delivered substantial forecasting and economic gains. Bigger general-purpose models didn't solve it. Data that fit the problem did.
Finance has a particular property that makes this worse: it is non-stationary. The rules that generated last decade's prices are not the rules generating today's. A language model can assume grammar is roughly stable over time. A market model can't assume anything of the kind. Interest rate regimes change, supply chains reorganize, new players arrive, and the patterns a model learned quietly stop applying.
The long-horizon wall
This brings me back to what I saw in commodity forecasting. I call it temporal dilation: the further out you need to see, the more each small uncertainty compounds, until predicting time t+n becomes fundamentally different from predicting t+1. A model optimized for the next step is not a model of the distant future.
The evidence suggests this wall isn't specific to AI. An IMF study of 10 commodity futures at horizons up to two years found that, out of sample, the forecast embedded in futures prices is hard to beat. Federal Reserve researchers found that futures generally outperform a naive random-walk forecast, but not by a large margin. In other words, decades of economists, traders, and models have struggled to say much more than "tomorrow's price is roughly today's price" at long horizons.
I want to be precise about what this means. No model, transformer or otherwise, will predict the exact price of copper in 2035. The uncertainty grows with the horizon until the honest forecast is a wide distribution. Even the researchers who critiqued transformers noted that not every time series is predictable, and that long-term forecasting only works for series with a relatively clear trend and periodicity.
So the real opportunity isn't a better crystal ball. It's models that:
Know when they don't know. Calibrated uncertainty that widens honestly with the horizon, instead of false precision.
Detect regime changes rather than assuming the past continues.
Model structure, not just sequence. Supply, demand, inventory, and the strategic behavior of the actors involved, so that long-horizon forecasts come from mechanisms rather than extrapolated patterns.
That last point is where my interest in game theory and stochastic systems comes in. Markets are not physics. They are games played by agents who adapt to each other, including to the models other players are running.
The spatial blind spot
The second gap is space. Transformers process the world as a one-dimensional sequence of tokens. Images, maps, and geological surveys get flattened into that sequence.
This year at ICLR, researchers found that modern vision-language models remain brittle at basic 2D spatial relationships such as relative position, layout, and counting, and argued this is linked to design choices like flattening images into 1D token sequences with 1D positional encoding. Another study found that when multimodal models failed multi-step questions, around 91% of the failures involved incorrect spatial reasoning steps.
This matters for anyone trying to find critical minerals. Mineral exploration is spatial reasoning at continental scale: how rock formations, faults, geochemistry, and geophysical signals relate to each other in three dimensions.
There is real progress here. Researchers have built geospatial foundation models specifically for mineral prospectivity. But they face a familiar obstacle: huge amounts of multi-modal geospatial data, and very few historical records of where deposits were actually found. Once again, the bottleneck is as much about data and structure as it is about model size.
What I think comes next
My takeaways so far:
Represent the data the way the world is shaped. Time isn't a sentence and a map isn't a paragraph. Architectures need inductive biases that respect order, continuity, and geometry.
Domain data beats general scale. Models trained on finance-specific or geology-specific data consistently outperform general ones applied off the shelf.
Forecast distributions, not points. At long horizons, honesty about uncertainty is the product.
Model the mechanism. Transfer learning may help uncover shared structure across complex systems, from cells to markets. Efforts like Arc's Virtual Cell Challenge show what it looks like to test that kind of idea rigorously, with shared data and fair evaluation. I'm treating this as a hypothesis to test, not a result.
Always beat the boring baseline. If your model can't beat a random walk or the futures curve out of sample, it hasn't learned anything useful yet.
Transformers changed what AI can do with language. But markets and the earth aren't language. The next generation of forecasting systems will need to take time and space as seriously as today's models take words.
That's what I'm building toward. If you're working on similar problems, I'd love to hear from you.
————————————————————————————————————
References: Arc Institute, Virtual Cell Challenge (2025); Zeng et al., "Are Transformers Effective for Time Series Forecasting?" (AAAI 2023); Nie et al., "A Time Series is Worth 64 Words" (ICLR 2023); Noguer i Alonso & Pereira Franklin, "Pretrained Time-Series Foundation Models for Financial Return Forecasting" (2026); Rahimikia, "Re(Visiting) Time Series Foundation Models in Finance" (2025); Reichsfeld & Roache, "Do Commodity Futures Help Forecast Spot Prices?" (IMF, 2011); Reeve & Vigfusson, Federal Reserve IFDP 1025 (2011); "Spatial Reasoning is Not a Free Lunch" (ICLR 2026); Daruna et al., "GFM4MPM" (SIGSPATIAL 2024).
