|
| 1 | +--- |
| 2 | +layout: blog_detail |
| 3 | +title: 'How IBM Research Uses PyTorch and TerraTorch to Make Geospatial Computer Vision Accessible for Everyone' |
| 4 | +hidden: true |
| 5 | +--- |
| 6 | + |
| 7 | +Earth Observation-based analytics are becoming essential for understanding our planet — from monitoring deforestation to tracking urban development and analyzing the impacts of climate change. However, the coding and deep learning skills for applying AI models to satellite imagery and earth observation data has traditionally been a major barrier for many practitioners. |
| 8 | + |
| 9 | +By IBM Research’s launch of TerraTorch 1.0, a PyTorch domain library for fine-tuning of Geospatial Computer Vision Foundation Models, we make geospatial AI not only more accessible but also more practical for the wider PyTorch community. Our goal: simplify the process so that any data scientist, researcher, or enthusiast can build powerful geospatial models with ease and low GPU and data processing requirements. |
| 10 | + |
| 11 | +{:style="width:100%"} |
| 12 | + |
| 13 | + |
| 14 | +**The power of foundation models, even with 75-95% of the input data removed, the models do a fantastic job in reconstruction of the input data - therefore learning the underlying physics of our planet in a deep, latent space** |
| 15 | + |
| 16 | +## The Business Challenge |
| 17 | + |
| 18 | +Our goal was to remove the technical barriers that prevent people from working with satellite imagery, weather and climate data at scale. Together with NASA, we’ve developed the Prithvi family of foundation models. Integrating the latest innovations of AI research using the clean API PyTorch provides has facilitated the job. |
| 19 | + |
| 20 | +We wanted to create a framework that anyone can use to go from raw data to inference ready models in just a few steps. |
| 21 | + |
| 22 | + |
| 23 | +{:style="width:100%"} |
| 24 | + |
| 25 | + |
| 26 | +**How a weather and climate foundation model created and fine-tuned on PyTorch is used for weather forecasts** |
| 27 | + |
| 28 | +## How IBM Research Used PyTorch |
| 29 | + |
| 30 | +We’ve built TerraTorch on top of PyTorch, leveraging its dynamic ecosystem to integrate: |
| 31 | + |
| 32 | + |
| 33 | + |
| 34 | +* PyTorch Lightning for clean, scalable training loops |
| 35 | +* TorchGeo for geospatial data handling and transformations (PyTorch transforms) |
| 36 | +* For foundation models like the leading generative multimodal foundation model ['Terramind'](https://research.ibm.com/blog/terramind-esa-earth-observation-model), co-developed by IBM and ESA, and [the ‘Prithvi’ family](https://huggingface.co/ibm-nasa-geospatial), co-developed by IBM and NASA, TerraTorch has been used to fine-tune all of the downstream geospatial models for satellite imagery, weather and climate data. It includes the family of fine-tuned models that IBM has released as part of [Granite](https://huggingface.co/collections/ibm-granite/granite-geospatial-models-667dacfed21bdcf60a8bc982). In addition, other interesting foundation models and ecosystem components like Clay, SatMAE, Satlas, DeCur and DOFA are included in TerraTorch. |
| 37 | +* Powerful and state-of-the-art vision transformers to experiment with modern neural network architectures |
| 38 | +* TerraTorch-Iterate build on top of PyTorch, Optuna, MLFlow and Ray Tune for Hyperparameter Optimization (HPO), Neural Architecture Search (NAS) and Foundation Model Benchmarking (GeoBench), where TerraTorch became the reference implementation |
| 39 | + |
| 40 | + |
| 41 | +{:style="width:100%"} |
| 42 | + |
| 43 | +**The fine-tuning and inference process is completely described in a single YAML config file. There, the architectural building blocks of the model (backbone, neck, decoder, head) are defined. The Model Factory assembles the model using the build-in and custom registries. In addition, the Optimizer and Data Modules are created as defined in the config. Finally, everything is passed to the Lightning Trainer, who executes the task.** |
| 44 | + |
| 45 | + |
| 46 | +With PyTorch’s flexibility, we were able to prototype quickly, iterate on model architectures, and deploy pipelines for a range of geospatial applications — from flood and biomass detection to increasing resolution of climate data, where some of our our work became part of the [IBM Granite Geospatial Model Family](https://huggingface.co/collections/ibm-granite/granite-geospatial-models-667dacfed21bdcf60a8bc982). |
| 47 | + |
| 48 | + |
| 49 | +{:style="width:100%"} |
| 50 | + |
| 51 | + |
| 52 | +**Architecture of the Prithvi-EO-2.0-600M foundation model which IBM Research developed together with NASA** |
| 53 | + |
| 54 | +## Solving AI Challenges with PyTorch |
| 55 | + |
| 56 | +PyTorch helped us to tackle three major challenges: |
| 57 | + |
| 58 | +* Ease of experimentation: Dynamic computation graphs, automatic differentiation, full abstraction of CUDA and rich visualization tools made it simple to test different models and training strategies. |
| 59 | +* Scalability: With DDP, FSDP, PyTorch Lightning and TorchGeo, we could train models on large-scale datasets without worrying about infrastructure. |
| 60 | +* Community support: PyTorch - the de-facto standard in AI research - with its active community and excellent documentation made it easy to overcome hurdles and stay up to date with the latest advancements in AI research. |
| 61 | + |
| 62 | +## A Word from IBM Research |
| 63 | + |
| 64 | +*"PyTorch gave me the power to turn complex linear algebra and optimization problems into accessible, shareable solutions for the community. It feels empowering that we’re building and fine-tuning models for anyone curious about understanding our planet through AI."* |
| 65 | + |
| 66 | +— Romeo Kienzler, AI Research Engineer at IBM Research Zurich, Rueschlikon |
| 67 | + |
| 68 | + |
| 69 | +{:style="width:100%"} |
| 70 | + |
| 71 | + |
| 72 | +## The Benefits of Using PyTorch |
| 73 | + |
| 74 | +Using PyTorch allowed us to: |
| 75 | + |
| 76 | + |
| 77 | + |
| 78 | +* Build a reproducible, open-source framework for fine-tuning geospatial foundation models |
| 79 | +* Share our work with the community through easy-to-follow notebooks, TerraTorch configuration files, tutorials and model checkpoints on HuggingFace |
| 80 | +* Rapidly iterate over foundation model architectures and deploy fine-tuned models for inference, from research to real-world client products |
| 81 | + |
| 82 | +## Learn More |
| 83 | + |
| 84 | +For more information about this project and to explore the code, visit: |
| 85 | + |
| 86 | +* [GitHub Repository](https://github.com/IBM/terratorch) |
| 87 | +* [IBM Research: Simplifying Geospatial AI with TerraTorch 1.0](https://research.ibm.com/blog/simplifying-geospatial-ai-with-terra-torch-1-0) |
| 88 | +* [TerraTorch PrithviEOv2 example notebooks](https://github.com/IBM/terratorch/tree/main/examples/tutorials/PrithviEOv2) |
| 89 | +* [TerraMind example notebooks](https://github.com/IBM/terramind/tree/main/notebooks) |
| 90 | +* [Run TerraMind using TerraTorch on Colab](https://colab.research.google.com/github/IBM/terramind/blob/main/notebooks/terramind_v1_base_sen1floods11.ipynb) |
0 commit comments