Google just released TorchTPU, an open-source way to run your PyTorch code on its TPUs, a day after Amazon shipped its own TorchNeuron backend for Trainium chips. Both aim to let you move off Nvidia GPUs without rewriting your models. Neither ends Nvidia’s lead overnight, but it’s the clearest push yet.
Here are the quick facts before we get into the details.
- TorchTPU was announced on October 20, 2026, and is open source, according to TechCrunch.
- TorchNeuron targets AWS Trainium2 and Trainium3 and uses the Apache 2.0 license.
- Its public beta starts October 20, 2026, per the PyTorch blog.
- Both promise easy porting of existing PyTorch apps, with Google calling it a “simple configuration change.”
The point: if you train or serve models, you may soon have real alternatives to Nvidia without a rewrite.
Why TorchTPU and TorchNeuron matter now
PyTorch runs most AI apps, and it grew up around Nvidia hardware. That made Nvidia the default choice, even when other chips were cheaper or easier to get.
Google and Amazon want to break that habit. Instead of asking you to learn a new framework, they’re meeting you inside PyTorch. You keep your code, and they swap what runs underneath.
That also fits a wider trend of AI labs hunting for compute wherever they can find it. We saw a similar angle in our look at Anthropic’s Nova Scotia data center lease.
What TorchTPU does for Google Cloud users
TorchTPU lets existing PyTorch apps run on Google’s Tensor Processing Units. Google says migration needs only a configuration change, not a rewrite.
Google also says it worked with Meta, Hugging Face, Inferact (the team behind vLLM), and RadixArk (SGLang) so popular tools keep working. It added a TPU agent suite that helps you port apps and fix errors, and updated LiteRT for on-device AI with PyTorch apps.
What we don’t know yet: the license details, version numbers, pricing, and independent benchmarks. Treat “easy migration” as a claim to test, not a promise.
How TorchNeuron works on AWS Trainium
TorchNeuron registers Trainium as a normal PyTorch device. In practice, you move a model with .to('neuron') and leave the rest of your training loop alone.
For production, it supports torch.compile with a “neuron” backend. It also covers distributed setups such as DDP, FSDP2, tensor parallelism, and pipeline parallelism. You install it with pip install torch-neuronx.
Amazon is also sending code upstream so PyTorch depends less on Nvidia-specific features. The first release has gaps, though: sparse tensors, NestedTensor, JVP, and dynamic shapes are not supported yet.
On results, Deepgram reports about 60% more audio seconds processed per dollar than its GPU baseline. That’s a customer figure, not an independent test.
TorchTPU vs TorchNeuron: side-by-side
| Detail | TorchTPU (Google) | TorchNeuron (Amazon) |
|---|---|---|
| Hardware | Google TPUs on Google Cloud | AWS Trainium2 and Trainium3 |
| Announced | October 20, 2026 | Public beta from October 20, 2026 |
| License | Open source (terms not yet stated) | Apache 2.0 |
| Porting claim | “Simple configuration change” | One-line .to('neuron') |
| Known gaps | Benchmarks and pricing not published | No sparse tensors, NestedTensor, JVP, dynamic shapes |
This table shows both bets are similar, and both are early. It’s worth running your own workload before you move anything.
Will this really loosen Nvidia’s grip?
Not quickly. TechCrunch notes this won’t make developers abandon Nvidia en masse. Years of tooling, custom kernels, and habits don’t vanish with a new backend.
The pressure is still real, because supply is tight. According to Reuters via The Edge Malaysia, ByteDance, Alibaba, and Tencent placed about $35B of orders for Nvidia’s export-compliant RTX PRO 5500 cards, and Nvidia reportedly told partners it can make no more than 3 million in 2027. When chips are scarce, alternatives look a lot better.
Open inference tooling is moving the same way. Our post on DeepSeek’s open-source DeepInfer engine covers a similar push to run on more than one kind of chip.
What to check before you switch chips
Start with your model’s operators. TorchNeuron’s own notes say operator coverage gaps remain, and unsupported ops fall back to the CPU, which can erase any speed or cost gains.
Next, compare cost per result, not cost per hour. Deepgram’s “more audio per dollar” number is the kind of metric that matters, but only your own workload can confirm it for you.
Finally, check your custom kernels. If you wrote CUDA code, it won’t run as is. Trainium uses its own NKI kernels, and that’s where porting can still take real work.
Frequently Asked Questions
What is TorchTPU?
It’s Google’s open-source PyTorch support for its TPUs. It lets existing PyTorch apps run on Google Cloud TPUs with little change.
What is TorchNeuron?
It’s Amazon’s open-source PyTorch backend for Trainium2 and Trainium3 chips. It uses the Apache 2.0 license and is in public beta.
Do I need to rewrite my PyTorch code?
Both companies say no or almost no. TorchNeuron needs a device change like .to('neuron'), and some features aren’t supported yet, so test first.
Is this only for big AI labs?
No. Anyone training or serving models on Google Cloud or AWS can try them. Smaller teams may gain the most if cloud GPUs are pricey or hard to get.
Our take: if your bills or wait times are driven by GPU supply, put a small test workload on TPU or Trainium this month. Don’t migrate production until independent benchmarks show up, but do stop treating Nvidia as your only option.
Leave a Reply