Fused is excited to be featured in a TED Talk today by Peter Wilczynski, CPO at Maxar, which explored a multi-source site monitoring scenario of Vancouver's greening initiatives over the past five years powered by the Fused platform.

Site monitoring scenarios are at their most effective when data from disparate sources is combined—from satellite imagery and map data to municipal and demographic data. Analytics today requires bringing all these different sources of data together: from raw pixels to contextual intelligence — in real-time, seamlessly, and at scale.

This is an iterative process, each dataset being in its own format & hosted on different cloud providers, updated independently and on different schedules.

Fused is built by data scientists, engineers & open source contributors who know the reality of building pipelines with breaking changes in libraries, new file formats that require rethinking how data is ingested and business requirements that constantly change.

We firmly believe we need tools that help us bring as many datasets together as possible, especially in a world where LLMs and AI can increasingly talk directly to data.

If you’d like to try out an instance of the Fused workbench with pre-loaded data from Maxar later this month, which Peter demoed on screen, sign up for the waitlist on fused.maxar.com to get access and see for yourself this is more than a pre-recorded tech demo!

Give LLMs the ability the talk to your data & code​

We're launching Fused AI Builder, a simple way for you to give an LLM access to any UDF you want

Fused UDFs meets MCP Server​

MCP (Modal Context Protocol) Servers allow Large Language Models like Claude or ChatGPT to directly make HTTP queries. This means they can now access APIs and make all sorts of calls.

But a lot of the internet still doesn't have APIs, be it a new dataset you just discovered that's hosted on an FTP server or data on an S3 bucket that needs some processing before it's useful.

That's where Fused UDFs come into play. UDFs allow you to write Python code that can be executed from anywhere, and more importantly called from anywhere.

By using UDFs with MCP Server you can now easily write any Python code and allow a LLM like Antrophic's Claude to execute it & use its output.

Data, Python & LLMs all in one​

You can now use LLM + Fused UDFs to do a whole host of tasks:

A New UDF Editor in Workbench​

Introducing Collections: A simple way to organize your UDFs​

Collections now allows you to organize your UDFs together as you work on different projects, save them together and keep your editor focused on the project at hand.

Redesigned UDF Editor​

UDF Editor now comes with a host of new features & redesigned UI:

• Adding a new Full Screen Map View
• The Visualize Tab is now under the UDF expanded parameters, allowing you to hide your Code Editor and just focus on tweaking the visualization of your data
• Split screen "Editor" & "Module" on top of each other: Keeping your code clean in the main "Editor" Tab is now easier by moving functions under the "Module" tab.

A New Share Page​

We've moved all the tools & information you need to easily share your UDFs into a dedicated page (and button). You can easily:

• Create a token to share your UDFs
• Edit the Description, Tags & Image preview of your UDFs

Changes to fused-py​

Our Python library fused-py is getting some updates to make processing data in Python simpler, from a small one time task to processing huge datasets

Simplifying how Fused handles geometries​

• Moving away from Fused having many different fused.types to only having 2 simple types: fused.types.Bounds and fused.types.Tile.
• Replacing bbox object with bounds: a more generic term to pass geometries to UDFs

fused.submit() for simple, multi job runs​

We're introducing fused.submit() as a simple way to run many UDFs all at once in parallel

This can significantly speed up parallel tasks like:

• Fetching lots of individual data points from an API
• Scaling small processing steps to lots of data points

Improved caching​

Under the hood we've significantly improved how Fused caches results of recurring UDFs & cached functions + we've introduced new tools for developers to control caching:

• A default 90 days cache time for all UDFs
• Editing the cache duration with the new cache_max_age argument

Read more about Caching

Full Fused 2.0 Changelog​

Read our Changelog to see every change happening with Fused 2.0

The Global Entity Reference System (GERS) is a framework that structures, encodes, and matches map data to a shared universal reference within Overture. GERS helps organizations identify and reference their own datasets with standard identifiers to Overture data to help unify datasets.

In this blog post we show how to create simple endpoints with Fused UDFs to enrich a dataset with GERS IDs. We'll use the Overture Building footprints to first enrich a polygon with GERS IDs then another one to look-up metadata for a specified GERS ID.

The benefit of GERS​

When third-party dataset is spatially matched to an Overture feature it's "enriched" with that feature's GERS ID and becomes "GERS-enabled". This makes it easy to associate it by ID to any other GERS-enabled dataset.

                            gers_id             buliding_name
0  08b2a1072534cfff020018b8a6efde22  James A. Farley Building
1  08b2a100d2cb6fff02000821de8bdff1      Pennsylvania Station

For example, a municipal government with a dataset of building footprints for local offices, coffee shops, and museums could match those entities to a GERS ID. This would enable the government to easily join its data other "GERS-enabled" datasets to enrich them with additional information such as insurance data, historical property values, restaurant reviews, fire risk, or rooftop solar potential.

Create a UDF to enrich a polygon with GERS IDs​

We can create a UDF that takes in a polygon and returns a GERS ID. This UDF will spatially match the polygon to Overture Buildings and return the GERS ID of the building that intersects the polygon. This is useful for enriching a dataset with GERS IDs.

Users can design and preview the workflow interactively, allowing them to test assumptions and visualize their effect. Parameters can be adjusted, and the output can be previewed before running the UDF on the entire dataset.

import json

aoi = json.dumps({"type":"FeatureCollection","features":[{"type":"Feature","properties":{},"geometry":{"coordinates":[[[-73.9945443706057,40.751205998161026],[-73.9945443706057,40.749882643934455],[-73.99222917395902,40.749882643934455],[-73.99222917395902,40.751205998161026],[-73.9945443706057,40.751205998161026]]],"type":"Polygon"}}]})

@fused.udf
def udf(bbox: fused.types.TileGDF=aoi):
    import geopandas as gpd
    utils = fused.load("https://github.com/fusedio/udfs/tree/e1fefb7/public/Overture_Maps_Example/").utils

    # 1. Convert bbox to GeoDataFrame
    if isinstance(bbox, str):
        bbox = gpd.GeoDataFrame.from_features(json.loads(bbox))

    # 2. Load Overture Buildings that intersect the given bbox centroid
    gdf = utils.get_overture(bbox=bbox.geometry.centroid, overture_type='building', min_zoom=10)

    # How many Overture buildings fall within the bbox centroid?
    print("Buildings in centroid: ", len(gdf))

    # 3. Rule to set only one GERS on the input polygon
    bbox['id'] = gdf.id.values[0]

    return bbox

Create an HTTP endpoint​

With Fused, it's easy to turn your UDF into an HTTP endpoint. This enables you to run the UDF it programmatically via HTTP requests to integrate the functionality into various workflows and applications.

This endpoint runs a public UDF with the code above. You can call it with a geojson of a single polygon in the bbox query parameter and it will return a geojson with the polygon and an assigned GERS ID.

https://www.fused.io/server/v1/realtime-shared/UDF_Overture_Buildings_GERS_Enrich/run/file?dtype_out_vector=geojson&bbox={"type":"FeatureCollection","features":[{"type":"Feature","properties":{},"geometry":{"coordinates":[[[-73.9945443706057,40.751205998161026],[-73.9945443706057,40.749882643934455],[-73.99222917395902,40.749882643934455],[-73.99222917395902,40.751205998161026],[-73.9945443706057,40.751205998161026]]],"type":"Polygon"}}]}

Create a UDF to look-up metadata for a GERS IDs​

We can also create a sample UDF to do a reverse operation: look-up a Building and its attributes by passing a GERS id. A user will be able to pass a GERS id and the UDF will look-up the building and return its geometry along with attributes about the building.

To do this, we create a UDF that takes in a gers_id parameter. Because the first 16 digits of GERS correspond to an H3 cell, we can use the ID to create a polygon to spatially filter the dataset. It'll bring up any buildings that intersect the H3 cell. Once we have the building, we can easily work with its geometry object and attributes using GeoPandas.

@fused.udf
def udf(gers_id: str='08b2a100d2cb6fff02000821de8bdff1'):
    import h3
    from shapely.geometry import Polygon
    import geopandas as gpd
    import pandas as pd

    utils = fused.load("https://github.com/fusedio/udfs/tree/e1fefb7/public/Overture_Maps_Example/").utils

    # 1. H3 from GERS
    h3_index = gers_id[:16]
    print('h3_index', h3_index)

    # 2. Polygon from H3
    bounds = Polygon([coord[::-1] for coord in h3.cell_to_boundary(h3_index)])
    bbox = gpd.GeoDataFrame({'h3_index': [h3_index], 'geometry': [bounds]})

    # 3. Load Overture Buildings
    gdf = utils.get_overture(bbox=bbox, overture_type='building', min_zoom=10)

    # 4. Subselect building
    gdf = gdf[gdf['id'] == gers_id]

    # 5. De-struct the names column
    normalized_df = pd.json_normalize(gdf['names'])
    gdf = gdf.reset_index(drop=True).join(normalized_df)

    return gdf[['id', 'primary', 'subtype', 'class', 'geometry']]

Create an HTTP endpoint​

Here's how you can create and use an HTTP endpoint for your GERS building lookup UDF.

This endpoint returns a CSV table of the building's GERS ID, primary name, subtype, class, and geometry. You can use this endpoint to enrich your dataset with GERS IDs by calling it with a GERS ID query parameter.

https://www.fused.io/server/v1/realtime-shared/UDF_Overture_Buildings_GERS_Lookup/run/file?08b2a100d2cb6fff02000821de8bdff1&dtype_out_vector=csv

For example, you could call this endpoint from a Google Sheet (docs) to enrich a dataset with GERS IDs. This sample Google Sheet returns the enriches the “primary” name column for any given Building GERS. Just drag the formula to apply it to any row below. It works by calling the "GERS lookup" endpoint with a GERS ID query parameter.

Fused has been working with the team at The Overture Maps Foundation to enable direct access to their data through Fused UDFs. We are excited to share that the Overture docs now show examples on how to see how to integrate any Overture data into your workflows using Fused.

Overture Maps aims to provide foundational building blocks of data across various themes designed to be broadly applicable across industries. Our goal at Fused is to make it easy to work with Overture data and adopt standards (such as GERS). To this end, we are creating easy abstractions to access data, tools to perform foundational operations such as conflation and enrichment, and example workflows to inspire and help you understand how to leverage this data.

One of the key usecases for Overture + Fused is enriching datasets with Overture Maps data. This tutorial showcases 2 simple Python workflows that do this by performing a spatial join with Overture Buildings. This example of a simple enrichment operation will help you understand how to work with Overture data in your own workflows.

Overview​

The Overture Buildings dataset is dividen into themes. Two key themes are:

• Buildings is composed of building footprints represented as polygons
• Places is composed of business establishment locations and associated metadata, represented with point coordinates.

We'll first show how you can load Overture data by reusing an existing Fused UDF, then write a User Defined Function (UDF) with custom logic to perform enrichment with a spatial join. You'll be able to run the resulting UDF for any custom area of interest (AOI).

Step 1: Load data with the Overture Maps UDF​

Fused has a catalog of pre-made UDFs you can easily copy and repurpose for your own data analysis workflows. In the catalog, you'll find the Fused Overture UDF which enables you to quickly load Overture data from any of the themes for an area of interest (AOI). You can run the UDF with fused.run and specify an AOI to load data for using the bbox parameter. You may also pass optional parameters to select between Overture releases, themes, and columns - that way you can fetch only the data you need. In this example, we can specify the 'building' theme by setting the overture_type parameter.

import fused
import geopandas as gpd
import shapely

bbox = gpd.GeoDataFrame(
    geometry=[shapely.box(-73.9847, 40.7666, -73.9810, 40.7694)],
    crs=4326
)

fused.run('UDF_Overture_Maps_Example', bbox=bbox, overture_type='building')

The output should look like this:

        id	                                geometry	                                        class   ...
24134	08b2a100d65a6fff0200b45ce7e2b99b	POLYGON ((-73.98552 40.76736, -73.98557 40.767...	apartments  ...
24135	08b2a1008b259fff02007917db1c32d3	POLYGON ((-73.98441 40.76703, -73.98431 40.767...	apartments  ...
24178	08b2a100d6516fff0200ded2bf849c8a	POLYGON ((-73.98375 40.76693, -73.98381 40.766...	apartments  ...
24179	08b2a100d6516fff02006dc174022a7e	POLYGON ((-73.98346 40.76623, -73.98327 40.766...	commercial  ...
24180	08b2a1008b248fff0200315983940aa8	POLYGON ((-73.98407 40.76749, -73.98402 40.767...	None    ...

By browsing the UDF's catalog page, you can see its code and even copy it to run it interactively on the Fused Workbench. You'll notice that the UDF uses the get_overture helper function to read from spatially partitioned parquets of the overture data releases, hosted in a Source Cooperative S3 bucket. The source code of the helper function is fully open and hosted on GitHub here.

Here's a simplified version to show the core of what's going on in get_overture. It constructs the table path on S3 and then uses the table_to_tile helper function from Fused to load data that falls within the specified bounding box. This approach allows you to efficiently perform spatial queries on a large dataset and load only the records within the given area.

# Structure the table path with input parameters
table_path = f"s3://us-west-2.opendata.source.coop/fused/overture/{release}/theme={theme}/type={overture_type}"

# Load the data within the bounding box
df = utils.table_to_tile(bbox, table=part_path)

Step 2: Write a Custom UDF to join Places with Buildings​

The example above shows how to run an existing UDF to load Overture data, but it's likely you want to write your own data transformations. You can borrow get_overture to load data into your own UDFs. As an example, here is a UDF to load Overture Buildings polygons and Overture Places points. The UDF perform a spatial join between them to determine which points fall within each building.

@fused.udf
def udf(bbox: fused.types.TileGDF = None):

    utils = fused.load("https://github.com/fusedio/udfs/tree/e1fefb7/public/Overture_Maps_Example/").utils

    # 1. Load Buildings
    gdf_buildings = utils.get_overture(bbox=bbox, theme='buildings')

    # 2. Load Places
    gdf_places = utils.get_overture(bbox=bbox, theme='places')

    # 3. Create a column with the Buliding Name
    gdf_buildings['primary_name'] = gdf_buildings['names'].apply(lambda x: x.get('primary') if isinstance(x, dict) else None)

    # 4. Spatial join between Places and Buildings
    gdf_joined = gdf_places.sjoin(gdf_buildings[['geometry', 'primary_name']])[['id', 'names', 'primary_name', 'geometry']]

    return gdf_joined

To run this UDF, you simply call it with your AOI. Fused will execute the code with the given parameter then return the UDF's output.

import fused
import geopandas as gpd
import shapely

bbox = gpd.GeoDataFrame(geometry=[shapely.box(-73.9847, 40.7666, -73.9810, 40.7694)], crs=4326)

fused.run(udf, bbox=bbox)

This will return a GeoDataFrame with the geometry of the Place, the primary_name of the building it falls within, and other attributes as defined in the UDF's return statement.

The output should look like this:

        id	                                names                                                   primary_name	    geometry
3883	08f2a100d65160860308e7269804dcb7	{'common': None, 'primary': 'Alamo Rent A Car'...	The Sheffield 57    POINT (-73.98404 40.76661)
3884	08f2a100d65160860306e05f63f3cbaf	{'common': None, 'primary': 'National Car Rent...	The Sheffield 57    POINT (-73.98403 40.76661)
3896	08f2a100d6516b110387be8074670bfa	{'common': None, 'primary': 'Quality Fashion',...	Hearst Tower	    POINT (-73.98370 40.76660)
3898	08f2a100d65147290359c345582a702e	{'common': None, 'primary': 'House Beautiful M...	Hearst Tower	    POINT (-73.98338 40.76664)

Step 3: Write a Custom UDF to join Buildings with the NSI dataset​

As a second example, you can also enrich the Overture Buildings dataset with metadata data from the National Structure Inventory (NSI) API. The NSI API offers point data on buildings in the U.S. that is relevant to hazard analyses.

This UDF loads the Overture and NSI datasets, performs a spatial join to enrich the building polygons with hazard metadata, and returns the enriched GeoDataFrame. It can be used within a larger analysis workflow to enrich building polygons to calculate risk indices. You can read more about performing a point in polygon operation to enrich Overture Buildings with NSI here.

@fused.udf
def udf(bbox: fused.types.TileGDF = None):
    import geopandas as gpd
    import pandas as pd
    import requests

    utils = fused.load("https://github.com/fusedio/udfs/tree/e1fefb7/public/Overture_Maps_Example/").utils

    # 1. Load Overture Buildings
    gdf_overture = utils.get_overture(bbox=bbox)

    # 2. Load NSI from API
    response = requests.post(
        url="https://nsi.sec.usace.army.mil/nsiapi/structures?fmt=fc",
        json=bbox.__geo_interface__,
    )

    # 3. Create NSI gdf
    gdf = gpd.GeoDataFrame.from_features(response.json()["features"])

    # 4. Join Overture and NSI
    cols = ["id","geometry","metric","ground_elv_m","height","num_floors","num_story"]
    join = gdf_overture.sjoin(gdf, how='left')
    join["metric"] = join.apply(lambda row: row.height if pd.notnull(row.height) else row.num_story*3, axis=1)
    return join[cols]

The output should look like this:

	    id	                                geometry	                                        val_struct      med_yr_blt  ...
24178	08b2a100d6516fff0200ded2bf849c8a	POLYGON ((-73.98375 40.76693, -73.98381 40.766...	378633.733	1939	...
24178	08b2a100d6516fff0200ded2bf849c8a	POLYGON ((-73.98375 40.76693, -73.98381 40.766...	348190.820	1939	...
24178	08b2a100d6516fff0200ded2bf849c8a	POLYGON ((-73.98375 40.76693, -73.98381 40.766...	378633.733	1939	...

Conclusion​

In this short tutorial, we outlined how you can integrate Overture data into your workflows using Fused. We saw how you can use Fused to load the data, write a custom Python workflow, and run it for a custom AOI. We hope the these foundational pieces help you see how you can unlock the full potential of Overture Maps and start creating your own workflows to enrich your own datasets.

Pilot Fiber is a commercial Internet Service Provider primarily in New York City. Our primary value proposition in competing with national-scale ISPs is our commitment to customer experience–we are fast and flexible in responding to customer needs in all aspects of the business:

• During the sales process, we aim to answer customer questions quickly and accurately, including technical details and routing.
• When designing our deployment into new buildings, we equip our engineers with as much detail as possible before they arrive on-site to maximize the efficiency of time spent with building engineers.
• When an incident interrupts a service we will immediately jump into action to address the cause and restore connectivity.

We use Fused to support all of these areas, and this post focuses on the last one: incident management.

• What happens when a service interruption occurs?
• How do we identify the likely location of an issue and get to a solution as quickly as possible?

The Problem: Why Speed Matters​

One of the most common ways a customer's service can be impacted is through damage to the physical fiber cables connecting them back to a data center and the internet. Almost all fiber optic cables in Manhattan run through a shared manhole-and-duct system beneath the streets. As such, road construction or work by other providers in a manhole has the potential to damage the equipment of multiple providers. When that damage occurs, it is first come, first served to get your network repaired and customers back online. Field teams from multiple providers can't work in the same manhole simultaneously, so being onsite first and ready to repair can mean a difference of hours in customer downtime.

Because of this, Pilot uses an active fiber monitoring system across our network. Sophisticated devices in our data centers are constantly shooting light down the fibers in our network looking for potential damage. Those devices return a reflectance signature from the fiber and compare it with a reference “snapshot” created when that fiber was initially installed in a building.

When an anomaly is registered, it immediately fires an alert giving a fiber route and distance to the potential problem (i.e. “There is unexpected light loss on the fiber serving 1234 5th Avenue at a distance of 2.351 kilometers from the data center.”). When this happens, our engineering and support teams analyze the data within minutes to determine the issue's exact location and, if necessary, get crews headed to the site to begin repairs.

The Process: Fused As The Glue-Layer​

Historically, this analysis has required the attention of an Outside Plant engineer with access to specialized software and network knowledge, regardless of the time of day or day of the week. This bottleneck is not ideal when time is of the essence, even at 3 a.m. So today, we are creating a more sophisticated future using Fused to make this information accessible to more support team members and make our response times even faster.

Using Fused as a back-end glue layer, we built a web app allowing users to select a route and distance and calculate where the system has registered the fault. We also created a simple user interface that provides the user a view of nearby network infrastructure and automatically generates the reports field crews would need to complete repairs based on that nearby infrastructure.

The workflow requires a series of calls to UDFs that act as intermediaries to a Postgres/PostGIS database, which in turn is sourcing data from other internal sources. This structure allows us to easily keep the business logic organized at the UDF layer while limiting the scope of data access and security via Postgres and internal processes maintaining the sync.

The basic process is seen below:

Workflow diagram.

Two separate flows are initiated when the user inputs a route and a distance to process. One retrieves the selected route to load onto the Mapbox-based map within the app, while the second kicks off a processing chain to analyze the fault information. This chain utilizes UDFs that assist in isolating the location of the fault and relevant nearby infrastructure and adding elements to the map display to assist the user in visualizing what may be occurring.

UDF to find fault location.

If you consider the cables you see strung along utility poles, they are not perfectly straight: they can sag, bend, go up and down, and have coils of extra cable along the way. The same is true of our cables under the streets. All of those variations add distance to the run, which needs to be considered when determining where a fault is likely to be located.

Taking those variables into account, we use Fused to apply GeoPandas and PostGIS spatial functionality to assess where the fault is most likely to be located. After calculating that location, the tool loads splice cases along that route that point to where problems are most likely to have occurred, and slack loops built into the route to make the user aware of nearby capacity that could enable faster repair of more significant damage. We next determine which splice cases are closest to the likely damage point and if any of those are within 150m of the automated distance calculation. These manholes would be the first locations our field teams would be sent to investigate.

Once we have determined the relevant nearby splice cases, we use another UDF to build a CSV that reproduces what a splice report export looks like from our primary fiber mapping software. This report details what cables enter the case, what fibers on which cables are spliced to which fibers on other cables, and what building and/or customer circuits are being carried along which fibers. Fused then integrates with a separate process we previously built to process that CSV into the final file that goes to the field teams. This gives them the confidence to quickly address issues while minimizing the risk of damaging other circuits, as well as giving them the information needed to monitor the status of other services in the case while they work.

The Impact of Fused​

The impact of Fused across this process is many-fold:

• The ability to easily work with data across several systems.
• Centralizing business logic to the UDFs involved eliminates the tendency for this logic to be spread across client-side processes, server-side processes, and possibly the database itself.
• Modularization of operations into UDFs. For example, the UDF that generates the splice reports for this process can easily be reused in any other method that requires the same functionality.
• The effortless ability for the same UDF to simultaneously serve as a modular processing unit where needed in one workflow and a map service for display in another.
• Using Python and standard libraries enables developers who may not be spatial data experts to read, understand, and modify UDFs as necessary.

Conclusion​

Pilot's success in the market is largely based on our flexibility and responsiveness to customer needs, which is never more important than when physical damage to the network is impacting their service. Within this scenario, Fused is providing a critical layer enabling us to offer more non-technical users access to data in multiple systems through a simple UI that will result in repair teams moving to restore service to our customers as quickly as possible.

Fused is an ideal product for Pilot Fiber in that we can increasingly make highly technical information available in a usable format to additional teams throughout the company in support of our drive to be fast, flexible, and accurate in delivering service to our customers in all aspects of the business.

BlackPrint Technologies began its journey as a satellite mapping service designed to help municipalities modernize their property registries. Recognizing a greater opportunity, we transitioned into the private sector to address a significant gap in the commercial real estate market: the need for accessible, actionable data. Today, our platform empowers professionals with precise acquisition intelligence, transforming how decisions are made across Mexico and, soon, all of Latin America.

In this blog post, we will explore how BlackPrint built the backend of its intelligence platform with Fused to provide a comprehensive view of the commercial real estate market in Mexico.

The Problem: Challenges in Real Estate Data​

Real estate professionals face fragmented data sources and manual processes. This complexity limits access to critical metrics such as zoning regulations, demographic patterns, and traffic trends, making site selection decisions both high-stakes and prone to errors. BlackPrint recognized this challenge and set out to create an all-in-one platform, transforming disjointed data into actionable intelligence to empower professionals across the commercial real estate landscape in Latin America. Our Solution: The BlackPrint Approach

BlackPrint's platform offers a comprehensive suite of tools and insights, including property and zoning data, demographic and socioeconomic analytics, and detailed foot and vehicle traffic analysis. Underpinning this solution is a staggering volume of geospatial data sourced from diverse datasets such as cadastral records, demographic studies, and traffic sensors. This vast amount of information is meticulously analyzed to deliver actionable insights, all while being presented through an intuitive and user-friendly interface. By simplifying complex data into accessible formats, BlackPrint empowers users—regardless of their technical expertise—to navigate and leverage insights with ease. Building such a robust platform required not only expertise but also the right tools. That's where Fused came in.

Leveraging Fused: A Partnership for Efficiency​

Fused became an indispensable partner in turning our vision for BlackPrint into reality. Its end-to-end cloud platform allowed us to move from concept to MVP significantly faster by simplifying our data processing and data delivery workflows. Before using Fused, we faced daunting challenges, such as processing and visually inspecting terabytes of traffic data and massive datasets from the Overture Maps Foundation for points of interest.

These tasks, which previously required extensive time and resources, became streamlined and efficient with Fused. Its ability to process geospatial datasets via HTTP-accessible Python functions made the development process seamless, enabling us to focus on creating an intuitive, high-performance platform. With Fused, we could deliver real-time insights and user-friendly visualizations at a scale that was unimaginable only a few years ago.

Empowering Professionals with Unprecedented Insights​

BlackPrint's platform is transforming real estate decision-making by delivering unprecedented insights to professionals. For example, a retailer in Mexico City utilized our tools to identify a high-traffic site with optimal customer reach, saving weeks of manual analysis and significantly improving accuracy.

Our platform's effectiveness is measured through key metrics like time saved, improved precision, and enhanced ROI for our clients. Users have reported up to a 30% increase in efficiency when planning site expansions or evaluating investments.

Looking ahead, BlackPrint's vision extends beyond Mexico, aiming to revolutionize acquisition intelligence across Latin America. With partners like Fused, we continue to innovate, making geospatial analytics scalable, intuitive, and accessible to professionals at every level.

Conclusion​

BlackPrint Technologies is on a mission to redefine real estate intelligence by delivering actionable insights that drive smarter, faster decisions. This journey has been significantly accelerated thanks to Fused, whose serverless data delivery empowered us to process and serve complex geospatial data at an unprecedented scale. Together, we are shaping a future where real estate professionals can access intuitive, data-driven tools that simplify their workflows and enhance their outcomes. Join us in transforming the industry—visit blackprint.ai to see how we are revolutionizing real estate decision-making.

In 2020 while working in Silicon Valley for the county of Santa Clara Valley, I became obsessed with improving monitoring and prevention efforts surrounding Arundo donax. The search and mitigation process this invasive plant species, Arundo donax, was a Sisophisian struggle that had been subject to the same procedures year in and year out, with no progress on beating the spread. To improve the efficacy and efficiency in battling against this notorious weed, I decided to build a tool that would identify the key areas for mitigation - both for the frequency of propagation (occurrence) and for spread potential.

In this blog post I show how I used Fused to create a map of key hotspots where Arundo donax is likely to spread based on built-environment factors derived from Overture Maps data.

Introduction​

In 2020, the problem with managing Arundo was that many agency employees considered this to be a hopeless pursuit, given the exorbitant amount of time cleaning, layering, and calculating the weighted analysis would take. Despite our vast ArcGIS library of tools, engineers, hydrologists, and GIS managers all warned me that I was going to drive myself crazy trying to get the enterprise software to successfully run my analysis. My only regret in building the tool back then was that I didn't have a tool like Fused to expedite the data pulling, processing, and calculating - as it would have saved me from the very lunacy I was warned about.

The obsession with the grass species, Arundo donax, began with an insight to the positive feedback loops created by increased flooding and the spread of invasive species due to climate change. Arundo donax is one of the most invasive plant species worldwide. In addition to destroying biodiversity and disrupting habitats for native species, this large grass also contributes to significant flooding patterns. As weather events become more severe and biodiversity declines, these changes create compounded consequences in our changing climate.

Challenges with Hot-spot Analysis​

Hot-spot analysis tools using weighted sums can significantly increase accuracy in targeting key areas for prevention. To build a hot-spot analysis for Arundo donax, the following variables need to be scored according to their degree of influence: distance to nutrient loading sources (such as a golf course), distance to a riparian buffer (creek or river), distance to a water-flow disruptor (such as a bridge), and size of the stream.

The key pain points of running a weighted sum on traditional GIS software include:

• Slow calculations: “State-of-the-art” software like ArcGIS can take several hours to calculate weighted sums. Furthermore, running weighted sums on large datasets/geographic areas can be nearly impossible given exhaustive RAM and GPU demands, so analyses over 1000 sq miles often require a user to split analyses into different geographic regions.
• Program crashes: beyond the significant wait time required for typical weighted sum calculations, users of prominent GIS software often experience output delays as a result of runtime errors and other issues spurring a program crash.
• Data transformations, cleaning, and standardization: most users of traditional GIS software will find they need to start from scratch when compiling data for a hot-spot analysis or weighted sum. Sometimes, standard base layers like slope and aspect will be searchable on the local software basis. Still, often, lengthy transformations are required to make the layers compatible with the final overlay calculation.

How Fused Changed My Workflow​

Using cloud-based systems like Fused can significantly increase calculation speeds, program resilience, and access to public Cloud Native datasets.

After encountering the Fused and learning about how I could improve the speed and geographic spread of site suitability analyses, I wanted to put it to the test by expanding on a previous analysis I did in 2021 using ArcGIS. The original 2021 analysis took several months of data collection, interviews with other local agencies, and extensive data cleaning, standardization, and transformations. I experienced all the aforementioned pain points of hot-spot analysis/weighted sum calculations and more.

Flash forward to today, where I am compiling global data sets, layering them, and deriving statistical insights within 1% of the time that it took me using ArcGIS. Running buffer analyses, weighting variables, and procuring data has taken a fraction of the time for GLOBAL data - and if you remember from before, the previous analysis from 2021 took months for a county-wide calculation and final product.

The most satisfying aspect of my new hot-spot analysis application wasn't just the expansive end product; the process was more seamless and engaging than I had imagined.

Workflow Design​

I created a UDF with a simple model to identify hotspots susceptible to arundo. The model uses a weighted sum of several base Overture data classes:

• Golf Courses
• Bridges
• Water bodies (rivers, streams, etc.)

The UDF performs the following steps:

1. Create GeoDataFrames from the Overture maps dataset using get_overture
2. Generate an H3 score based on buffers around each feature
3. Aggregate the H3 scores to create a weighted sum

Key Takeaways​

Given the complexity of procuring, layering, and interweaving data along with significant wait times and resource consumption, many governmental agencies, non-profits, and even private corporations struggle to run spatial analyses such as site-suitability and hot-spot tools. Insights and tools can be created by improving the speed and efficacy of operations such as weighted sums and fuzzy analysis across spatial layers with UDFs.

Site-suitability and hot-spot analysis go beyond species detection. By simplifying the approach to these types of tools, we can more quickly and accurately detect climate-vulnerable zones, prioritize habitat restoration, and create models to build resilient communities. Industries such as real estate development, retail, and logistics can more quickly understand the variables that affect their businesses by using cloud-based systems like Fused, which can easily manage large datasets.

As communities continue to rebuild and recover from the devastation caused by natural disasters such as wildfires, the question remains: How can we quantify what was lost, especially in the built environment? With the ability to analyze detailed building footprints and overlay fire boundaries, we can begin to answer this by providing a rough estimate of the damage and identifying which structures were impacted by the flames.

In this blog post, Chris Amico shows how by leveraging data such as Overture Building footprints and fire progression maps, we can gain insight into the extent of fire risk. This enables news agencies to derive insights such as count of shops or homes exposed or even assess the capacity of highway routes for residents to evacuate before a prospect fire reaches them.

Introduction​

Fused simplifies the process of spatially joining datasets with Overture Maps data, such as integrating with fire-related datasets.

For this example, we use the Inter Agency Fire Perimeter Historical dataset (published by the National Interagency Fire Center (NIFC)) which includes historical fire perimeters up to 2024. Joining fire perimeters with Overture Buildings and Places data enables us to highlight service gaps or identify regions that may require immediate response.

This demo will first select buildings within a buffer zone to determine which fire perimeter they fall within. Then, it will perform an H3 stats roll-up of business categories from Overture Places, counting the number and types of businesses that fall within each distance range. This will involve rolling up Overture Places business categories into H3 hexagons.

The Workflow​

These UDFs return Overture Buildings and Places within a buffer distance from a fire. They offers a simple way to determine the scope of possible fire damage and quickly assess the number of businesses, homes, and other significant structures within the affected area. By adjusting the buffer or selecting fire extent based on dates, users can fine-tune their analysis to gain deeper insights into how far the fire's reach extends and what establishments were most at risk.

a. Fire Proximity Buffer​

First, we create a buffer around the fire perimeter to define the area of interest. This buffer helps us understand the extent of the fire's impact and provides a clear boundary for further analysis.

• In Historic Perimeter: The building/point is within a historic perimeter
• Near Historic Perimeter: The building/point is within 1km of a historic perimeter
• Outside Perimeter: The building/point is within 10km of a historic perimeter

b. Fire Proximity Building Score​

Next, we load the Overture Buildings dataset and spatially join it with the fire buffer zones. This workflow categorizes buildings based on their proximity to the fire, helping us assess which structures are most at risk.

1. Load the NIFC fire perimeter data
2. Create buffer zones around the fire perimeters
3. Load Overture Buildings
4. Spatially join buildings within the buffer zones to categorize them by proximity to the fire

c. Overture Places Rollup by H3​

Finally, we perform a spatial aggregation by calculating the H3 index for the centroids of Overture Places within the fire buffer. This allows us to roll up business categories into H3 hexagons, enabling a holistic overview of business distribution in relation to the fire perimeter.

1. Load the NIFC fire perimeter data
2. Load Overture Places
3. Determine the H3 for the centroid of each building
4. Normalize the 'categories' column into individual columns
5. Roll-up categories by H3, create categories primary set

Conclusion and Next Steps​

This kind of analysis helps understand not only the immediate impact but also in planning for future fire preparedness and recovery efforts. In this post, we saw how Overture and Fire-extent data can help us estimate the extent of the damage, from individual buildings to entire neighborhoods.

Organizations looking to integrate these types of perspectives into their workflows could create apps or API services that deliver derivative products, such as GERS lookups to categorize "fire risk" based on buffer proximity.

They could also use Fused HTTP endpoints from the UDF to return a CSV with the GERS and "Fire Risk" score for buildings within a defined bounding box, as specified by a query parameter. Additionally, they could also use the HTTP endpoints to automatically enrich the "Fire Risk" column of an arbitrary dataset for any given Building GERS. Users could apply it to any row of their table, with the functionality powered by a "GERS lookup" endpoint using a GERS ID query parameter.

Maribel Hernandez is a computer scientist and researcher at CINVESTAV, a multidisciplinary academic institution in Mexico. She specializes in graph theory in the field of computational genomics and complex networks. In this blog post, Maribel shows how she characterizes cities based on the distribution of businesses by rolling-up business categories by H3.

As someone who works with urban networks, Maribel's focus is on exploring how cities function and how their design impacts inclusivity. The core question driving this analysis is: Does the city have an even distribution of business services? Or are there shortages such as food deserts or unequal access to health facilities?

Introduction​

Consider these scenarios:

• How connected are areas dominated by upper-class establishments to the broader city fabric?
• Are health services distributed with equal access from low-income neighborhoods?
• Do certain metropolis have better accessibility and service availability compared to others?

Take, for example, a comparison between 3 key cities in Mexico: Mexico City, León, and Mérida. Are neighborhoods in these cities equally served by essential services such as healthcare, food, and transportation? Could some areas be considered food deserts, while others enjoy easy access to all services?

Creating Embeddings of Overture Places​

To answer these questions, I use a method that can scale across cities and ensure consistency. By grouping business categories in Overture Places at the H3 grid level, we can analyze the distribution of services and their accessibility. This approach provides a quantifiable way to compare cities globally, shedding light on the urban inclusivity of different regions. Workflow

The UDF I created follows this workflow:

1. Load Overture Places records for a given H3
2. Create business category column
3. Create an embedding to represent business categories
4. Run UDF for all H3 spanning an area of interest (polyfill)
5. Calculate Kmeans clustering based on cosine similarity.
6. Describe each cluster

Preview of UDF on Workbench.

Descriptive analysis​

When analyzing the urban fabric of cities like Mexico City, Leon, and Merida, we can obsere distinct patterns of service distribution that manifest as rings around the city center.

Mexico City.

a. The heart of the city​

Especially in Leon and Merida, the central area tends to form a cohesive, dense cluster of services. The core zone houses a variety of businesses including health services, retail, and food outlets, which are generally well-connected and easily accessible. Intuitively, the central cluster can be thought of as the “heart” of the city, serving as the primary hub for commerce, social interaction, and access to essential services.

Leon.

b. Islands of Services​

Beyond the center, we notice smaller clusters of services scattered across the city, often in the form of “islands.” These islands represent pockets of neighborhoods that, while not part of the dense city center, offer an array of unique services. These can serve to highlight the phenomenon of emergent neighborhoods within a greater whole.

Merida.

c. Peripheral ring​

The periphery of these cities forms another distinct pattern. These outer regions also tend to cluster together in similar service categories, forming a ring that surrounds the central core. It’s possible services in these peripheral areas are often more limited in scope and may reflect a focus on residential and less commercial needs, or reflect lower-income neighborhoods with scarcer access to key services.

Conclusion​

This ring-like pattern of service distribution suggests a common trend where the core of the city is highly accessible, while the periphery often lacks the same diversity and depth of business services. The islands of services in between can be seen as attempts to bridge this gap, but they are not always as effective in meeting the needs of the population on the periphery.

These spatial patterns offer valuable insights into the inclusivity of urban networks, highlighting areas that may need attention to ensure strategic business placement and guarantee equitable access to essential services.

Future work​

• Create Network: Create a bipartite network between place categories and H3 indices, using weights as counts for each category.
• Tie in Demographic Data: Integrate INEGI sociodemographic data at the census block group level to understand how services align with population needs.
• Access by Neighborhood: Use origin-destination (OD) movement networks to evaluate service accessibility by neighborhood of residence.
• Segregation Analysis: Identify zones with low visit rates or difficult accessibility. These zones may represent areas of segregation or neglect in the urban network.

In the parcel delivery business, geospatial analyses are crucial to answer questions about daily operations. Do delivery drivers visit the same regions each day, letting them know their areas intimately? Or is there a high volatility of the regions? And of course, how do we optimize the routes of multiple drivers servicing the same region?

Those are the questions Antonius is working on at GLS Studio, an innovation lab by GLS (General Logistics Systems) which is an international parcel delivery service provider.

In this blog post, Antonius highlights how he uses Fused to create stable delivery areas for single-day and multi-day aggregates.

The Challenge Designing Delivery Areas​

While the planned areas of driver tours to delivery packages can be rather well-defined, an evaluation of the areas actually served by drivers is equally important and might not be as easy. It can be used for guiding delivery drivers to become more efficient while also enabling managers to identify planned areas that are suboptimal due to built environment features like rivers or big highways.

Creating delivery areas out of single geospatial data points is challenging. A naïve approach would be to use convex polygons, but this causes multiple issues:

• A single data point can have a high influence on the shape and area of the polygon
• A tour consisting of multiple not connected sub-areas is hard to detect and correctly display
• This limits the area to what is covered by historic data which leaves a gap in new target regions
• Calculations using polygons are computationally expensive, hindering ad-hoc changes to the selected time span or the calculation parameters

Building new and sometimes experimental features means that the database is often not optimized for the use case. Data needs to be joined between multiple tables and even between multiple data sources. Therefore, fetching all data can be slow, highly limiting the usefulness of having an experimental feature to play around with.

Solving the Challenge with H3 and Fused UDFs​

Dynamic Delivery Areas with H3​

The solution to the first problem came by using H3 cells instead of polygons. By assigning cells to the drivers based on their historic deliveries to the cell, driver areas result automatically. Using H3 cells across different resolutions also allows us to represent the differences between urban and rural areas which see different parcel volumes. While there exists one "base resolution" to ensure non-overlapping and complete areas, the logical hierarchy among H3 cells can be used to calculate on lower resolutions for rural areas that see fewer deliveries, speeding up the computation and ensuring a broader coverage of those areas beyond the historical data points.

On the other hand, disputed H3 cells can be broken down to a higher resolution and assigned to different drivers or, when the "base resolution" has been reached, assigned to the driver delivering most parcels to the cell. As H3 cells have clearly defined neighborhoods, areas can easily be extended beyond their historical limits when desired, covering the empty space around to include a new parcel that falls outside of historically served area boundaries.

Fused app to show dynamic delivery areas at different H3 resolutions.

Streamlining workflows with Fused UDFs and Caching​

Fused UDFs helped us solve problems around the latency of querying and calculations. When a user looks at an area for a day, they are probably interested in the same area on some of the previous and following days as well, right? So why not pre-calculate that already?

Using Fused, it is simply a matter of fire-and-forget to trigger the UDF with parameters for some previous and following days which are then already running to cache. So when the user views an adjacent day, the data will already be there. And more broadly, when it is possible to limit the number of parameter combinations in an experimental feature to a manageable amount, this fire-and-have-it-cached approach is not limited to caching data from previous and following days, but can also be used for a range of other cases.

Sample workflow with Fused UDFs.

Conclusion​

When developing new features that are not yet supported by the current data infrastructure, Fused UDFs enabled us to easily test things without having to change the underlying infrastructure in advance. The UDFs are easily shareable and adjustable, allowing testing by multiple people without having to run code locally while automatically making sure everyone is using the same code that is hosted in the UDF. And because we can easily call UDFs with HTTP endpoints, when we have verified the feasibility of a feature, it’s easy to integrate into our product.

Hello friends, thanks for following my journey so far. To catch you up, I'm trying to solve the problem of farmers and traders relying on weak and untimely predictions of corn yield. Weak because they are at the national level and untimely because the predictions come once a month.

So here's the deal: farmers and traders have been relying on national-level corn yield predictions that are not only weak but also painfully slow, arriving just once a month. Imagine making critical decisions based on a single data point each month.

Not ideal, right? That's exactly the issue we're tackling in this blog post series.

Let's say you are a corn trader and it's halfway through the month, a major flood devastates key corn-growing areas. If you only get yield estimates monthly, you won't know until it's too late—prices will have already spiked, and you've missed your chance to act. But with updates twice a month or more, you'd catch the disaster early, adjust your trades, and stay ahead of the market. In a $21 billion corn commodities market, better, faster insights can make all the difference. Advanced geospatial modeling can bring those efficiencies, turning weak data into powerful decisions.

Motivations​

Let's find some efficiencies using advanced geospatial modeling and the latest research.

I previously showed how Solar Induced Flourescence (SIF) is a direct measurement of photosynthesis and therefore a better indicator of crop yield than NVDI indices.

But there's a catch—this data is weak because it's at a 25 km² resolution, which any geospatial machine learning scientist would find, well, less than ideal. To make matters worse, our actual yield data (our target variable) is at the county level. Double yikes!

But fear not. This problem can be solved!

Thanks to the scaling capabilities for geospatial modeling in Fused I am able to solve this data by increasing the amount of data and therefore the statistical power of it.

Diving into the Data Details​

Remember from the previous post how we transformed one data point per county into 20,000 by taking only SIF measurements for corn? Let's visualize that for the first collection in May (we get two collections per month -a and b):

Now, suppose we want to do this where corn is grown in the midwest US. Here is what the states that grew corn in 2018:

Within these states, we have 1,333 counties. Assuming each is similar in size to my home county of Lyon County, we can calculate:

1,333 counties × 20,000 data points = 26 million data points

That's 20,000 times the statistical power! 🎉

Let me say that louder for the people in the back: 26,000,000 vs. 1,333 data points

And that's just for one time period. If we run a model on 2–4 time periods, we're looking at nearly 100 million data points. Now, building a model on 100 million data points isn't trivial, but at Fused, this process becomes a breeze.

Building the Training Dataset​

I'm aiming to predict corn yield based on my SIF readings from early May, late May, early June, and late June. So, we need to build out a table with this structure:

To quickly validate this table against a map, I'll build out my workflow in Fused using Python and query the table with SQL. In the past, working with these two languages would have required complex tooling, storing data in a warehouse, and roughly five hours to run. With Fused, I can simply reference the a GeoDataFrame object and query in SQL with DuckDB all within the same UDF—taking just five seconds!

Here's what it looks like:

Splitting the Data​

But we're not stopping there! To ensure our model is both robust and unbiased, we need to carefully split our data. Enter Walk-Forward Cross-Validation — a game-changer for time series data. Think of it like planting seeds each season and harvesting them before the next planting. By always training on past data and testing on future data, we respect the natural flow of time. This method is perfect for corn yields because, just like how last year's harvest influences this year's, our model benefits from understanding those temporal dependencies. Plus, it prevents any sneaky data leakage, ensuring our predictions are based solely on what's known up to that point.

Conclusion and Next Steps​

Keep going or end here?

By progressively expanding our training set, each fold builds on the last, capturing more nuanced patterns and trends. To bring this to life, I'm leveraging TimeSeriesSplit from sklearn, seamlessly integrating it into our workflow. This tool simplifies the process, allowing us to focus on what truly matters—unlocking the full potential of our 100 million data points for actionable predictions.

And there you have it! We've taken weak, infrequent data and transformed it into a powerhouse dataset ready to drive smarter decisions in the $21 billion corn commodities market. Stay tuned for the next part of our journey, where we'll dive into building and fine-tuning our predictive models. Until then, keep cultivating those insights! 🌽

VIDA uses dozens of the latest generation of climate models to have the most up-to-date climate information, collectively part of the Coupled Model Intercomparison Project 6 (CMIP6). These models provide a range of possible futures, under different emission scenarios, as well as differences in how each model does its forecast. Using this information, we can create ensembles from the models, and have higher confidence in the risks and hazards we derive from the models, and which we present to our customers.

In this blog post, I show how I created a UDF to pre-process and visually inspect the Zarrs we generate as the output from our climate risk models.

As the Senior Machine Learning Engineer at VIDA, I work daily with large, multidimensional geospatial datasets to determine the future risks to infrastructure from climate change globally. These risks and hazards help organizations that invest in infrastructure to accurately assess investments, plan for climate resilient adaption where needed, and comply with new climate regulations such as the EU Taxonomy.

The engineering challenge to create climate models​

At VIDA, we derive dozens of risk metrics from CMIP6 and derive possible hazards from their outputs such as from flooding risk, drought, tornados, and many more. We write datasets to either Zarr or Cloud-Optimized GeoTiff so that we can efficiently load the data that we need. One of the best ways to sanity-check our outputs and the data we give our customers is through visualizing it.

Prior to using Fused, much of this visualization was ad-hoc, using tools such as matplotlib or xarray to generate plots locally and then sharing them as image files for each client site. While this does work, it builds a lot of friction into the feedback cycle, and to focus on a single site or area, we'd have to rerun our visualization pipeline.

Creating the UDF​

With Fused, we've been able to easily interactively pre-process and visually inspect the Zarrs that we generate as the output from our models. Being able to just share a link to a UDF to view the data in an interactive map has greatly sped up exploring the model outputs, and sanity check the data that we are using.

Conclusion​

Fused made it a lot easier for me to load, preprocess, and visualize the large datasets I work with and share those across my team. We create a lot of Zarrs which we put on the cloud, so our internal workflow of creating datasets, saving them as Zarrs, and sharing the outputs among the team significantly improved.

We now save engineering hours and maintain more simple workflows because Fused makes it easier to work with data and collaborate compared to sharing static PNGs with colleagues. Fused streamlines our internal workflows by enabling us to create Zarr datasets, upload them to Google Cloud, and efficiently build, run, and share climate models along with their outputs.

Dr. Qiusheng Wu is an Associate Professor of Geography and Sustainability at the University of Tennessee and a Founding Editorial Board Member at the Cloud-Native Geospatial Forum (CNG). As part of his commitment to making open-source geospatial analysis and visualization more accessible, he has developed several widely used open-source packages, including geemap, leafmap, and segment-geospatial.

In this Notebook Qiusheng shows a few examples of how Cloud Native Geospatial datasets help you easily load data into a Jupyter Notebook environment using leafmap. His practical examples showcase how you can call the Overture Maps UDF and Foursquare Places UDF to load data into a custom area of interest and render it in a leaflet map.

Calling Fused UDFs to load data​

You first use leafmap to create a bounding box over an area of interest (AOI) user_aoi and create a GeoDataFrame gdf_aoi with it. Then, you can run the Overture Maps UDF, passing the AOI as a parameter to define the area to fetch data for.

Behind the scenes, the the Overture UDF reads data directly from the Overture Maps dataset hosted in Source Cooperative.

import fused
import geopandas as gpd
from shapely.geometry import shape

# Define AOI, which can be done with leafmap drawing tools
user_aoi = {'type': 'Feature','properties': {},'geometry': {'type': 'Polygon','coordinates': [[[-74.025621, 40.699967],[-74.025621, 40.730283],[-73.966055, 40.730283],[-73.966055, 40.699967],[-74.025621, 40.699967]]]}}

# Convert drawing to GeoDataFrame
gdf_aoi = gpd.GeoDataFrame([user_aoi['properties']], geometry=[shape(user_aoi['geometry'])])

# Fetch Overture Bulidings GDF with Fused UDF
gdf = fused.run("UDF_Overture_Maps_Example", bbox=aoi)

Naty is a Senior Software Engineer and a contributor to Ibis, the portable Python dataframe library. One of her main contributions was enabling the DuckDB spatial extension for Ibis in 2023.

In this blog post, she shows us how to leverage the spatial extension in DuckDB with Ibis to query Overture data. Ibis works by compiling Python expressions into SQL, you write Python dataframe-like code, and Ibis takes care of the SQL. Thanks to Ibis integration with Pandas and GeoPandas, you only need to do to_pandas() to get your expression as a GeoDataFrame.

Motivations​

Overture Maps is emerging as a global backbone for linked spatial data. The Overture Maps Foundation, founded in 2022, is an open data initiative aiming to introduce the foundation for next-generation map services and applications. Overture offers reliable, easy-to-use, and interoperable open map data. Its structured schema enables consistent data access and seamless integration with other spatial data using unique identifiers.

Overture Maps has multiple datasets you can explore and query. Check their documentation to find out more about them. In this blog, we will explore the base theme and some of its infrastructure features.

Have you ever been on a walk as a tourist and wondered where you could get public drinking water or a bench to sit, or maybe you need to make a restroom stop, but you don't know where to find any of these?

We will find out how you can easily get this information from Overture data and visualize it, using Ibis and Fused. You'll be able to insert an address and see this key information.

Getting the data​

Overture distributes its datasets as GeoParquet, an optimized cloud-native format that enables you to easily get chunks of data from the official S3 bucket where the Overture dataset is published. The release of DuckDB 1.1.1 marked a significant milestone by adding support for GeoParquet files. This update enables users to query rich geospatial datasets like Overture Maps. Thanks to the support for DuckDB geospatial operations in Ibis, we can query the datasets from Overture Maps using Python with the performance of DuckDB without having to write SQL. Together, these tools offer an accessible and powerful solution for unlocking insights from global geospatial data.

Exploring Overture Maps​

Using Ibis with DuckDB as the backend, we can access the Overture Maps data directly from the cloud. These queries will only fetch the attributes and rows that meet specified conditions, minimizing how much data is processed locally. You can learn more about it in this DuckDB blog.

Creating a custom UDF​

I created a UDF to read an Overture maps GeoParquet file stored on S3 and filter it based on a given bounding box and column values using Ibis. You can access and run it here.

We first establish a connection to a DuckDB database, in this particular case we have an in-memory connection. Then, we do read_parquet and we receive a table expression. Since our result, t, is a table expression, we can now run queries against the file using Ibis expressions. In this example, to start, we filter by some infrastructure subtypes (pedestrian, and water), select only a few columns, and limit our “search” to a bounding box bbox. Notice that this bbox is the Fused bounding box, not the overture maps one.

We then rename the column class to avoid conflicts with the deferred operator, and finally filter the expression by specific infrastructure classes like toilets, ATMs, drinking water, and other useful information. Up to this point, we only have a table expression, Ibis has a deferred execution model. It builds up expressions based on what you ask it to do and then executes those expressions on request.

To show an example of an aggregate, we executed and printed the value_counts() as a Pandas DataFrame. Ibis can execute the table against the DuckDB backend, and return it as a Pandas DataFrame or a GeoPandas GeoDataFrame (if geometry column is present), by only doing to_pandas().

Conclusion​

The synergy between Ibis, DuckDB, and Fused has redefined the ease of querying and visualizing geospatial data. These frameworks provide an intuitive and powerful toolkit, enabling users to express geospatial queries, perform efficient transformations, and access high-performance analytics with minimal setup.

By leveraging this stack, interacting with vast geospatial datasets like Overture Maps becomes straightforward, efficient, and accessible.

Resources​

If you want to learn more about Ibis geospatial capabilities, check some of the geospatial blog posts here.

You might also find these resources useful as you dive into Ibis, DuckDB, and Overture:

• Overture Maps Data Repo
• Ibis Docs
• DuckDB spatial extension
• DuckDB spatial functions docs
• Ibis Zulip Chat

On December 2023, I visited the Institute for Metropolitan Planning (IMP) in Lima. The director had invited me to share some of my geospatial analysis projects from my master's studies and explore potential collaborations. Around that time, Lima's mayor had announced a bold infrastructure initiative: building 60 flyover bridges to ease traffic congestion in one of the most gridlocked cities in Latin America.

When I asked how the city was simulating the impact of new network designs on urban mobility, the answer was: "We are not simulating anything, our budget is constrained, and there is no political will to solve this problem." I couldn't think of anything else after this meeting. I started thinking about how I could create an easy-to-use tool to simulate urban mobility using open-source data, tools with no subscriptions or licenses, and without data privacy concerns.

My first attempt with FastAPI and React came to an unfortunate halt. Fused allowed me to revisit the idea and easily create an API endpoint and lightweight app I could easily share with anyone.

Challenges Building With FastAPI & React​

I took on IMP's challenge and, with references like Eclipse's Simulation of Urban MObility (SUMO) app as my guide, I rolled up my sleeves and started building a simulation tool. The synergy between GeoPandas and OSMnx allowed me to create a simple trip simulation based on real-world data and mobility behavior assumptions within days.

However, building the simulation was one thing. I also wanted to share the results interactively in a user-friendly format.

Before learning about Fused, I had set up a backend server with FastAPI and hosted a single generate_trips endpoint that would return GeoJSON data. The problem arose when trying to visualize the output. Downloading GeoJSON files and manually uploading them into tools like Kepler.gl was time-consuming and clunky. To address this I built a React application with Deck.gl, leveraging its TripsLayer to fetch and display streaming data from my API. It worked beautifully on localhost, but deployment costs were a blocker as neither I, as a student, nor the IMP had the budget to host the app. After two months of effort, I ended up with a half-finished tool that I couldn't effectively share. It was time to move on. Or so I thought…

How a Fused UDF Simplified Everything​

With Fused, all I had to do was write a Python function and my parallelized streaming service was ready to integrate with a Streamlit application via the App Builder. Here's how I redesigned the workflow as a User Defined Function (UDF) using lessons from my previous project:

1. Extracting the Road Network: Using OSMnx, I extracted road networks within a defined Area of Interest (AOI).
2. Enriching Data: Each road segment was assigned speed and travel time values.
3. Defining Population Data: A 1km² grid with population density and zoning data was loaded into a GeoPandas GeoDataFrame.
4. Setting Simulation Parameters:
   • Population size: Derived from density data.
   • Trips per person: Assumed at 2 trips/day (commute to and from work).
   • Origins and Destinations: Residential zones were assigned as homes and commercial zones as workplaces.
   • Trip Schedules: Normal probability distributions were used for departure (6-8 AM) and return times (5-7 PM).

With these parameters, the simulation sampled "home" and "work" nodes, calculated start times, and determined the shortest paths between origins and destinations. Async UDF calls made the process parallelized and efficient. The final output was a GeoDataFrame with:

• Start Time (Unix timestamp)
• Trip Type ("home" or "work")
• Path (list of coordinates)
• Timestamps (for each coordinate)

Future Plans​

This project is far from over. Here are the features I aim to add to make it a valuable tool for urban planners, especially in resource-constrained settings like Lima:

1. Larger AOI Support: Handle bigger datasets and simulate more trips.
2. Multimodal Routing: Incorporate walking, biking, driving, and public transit options, akin to OSRM profiles.
3. Custom Infrastructure: Allow users to model new transit infrastructure within the OSM road network.
4. Mobility Metrics: Provide detailed metrics (e.g., travel times, congestion levels) for each simulation.

With these enhancements, this tool could empower city stakeholders to make data-driven decisions on critical urban interventions—whether it's building flyovers or optimizing public transit routes. The ultimate goal? Improving mobility for over 11 million residents in Lima and beyond.

As a research specialist focused on remote sensing applications in semi-arid rangelands, I'm constantly seeking tools that can enhance our ability to process and analyze large-scale geospatial data. The excitement of discovering new platforms that streamline complex workflows never gets old, especially when dealing with the massive datasets typical in remote sensing research.

My journey with Fused began unexpectedly through the "Minds Behind Maps" podcast, where host Maxime Lenormand interviewed Sina Kashuk, Co-Founder and CEO of Fused (see episode). The conversation sparked my curiosity, leading me to explore Fused's community examples and documentation. After joining their waitlist and receiving access, I knew exactly how I wanted to test it: an interactive tool for exploring NEON's Airborne Observation Platform (AOP) data.

Making Hyperspectral Data Accessible​

For those unfamiliar with NEON AOP, it's an NSF-funded initiative revolutionizing ecological observation. Using advanced imaging spectrometers, NEON collects hyperspectral data across 426 distinct wavelength bands at 1-meter resolution. Imagine having 426 different perspectives of the same landscape, each revealing unique insights about vegetation, soil composition, and ecosystem health.

Source: NEON Imaging Spectrometer.

The real challenge, however, isn't collecting this rich data - it's making it accessible and actionable for researchers. This is where Fused enters the picture. Diving into their documentation and gallery of click-and-run examples, I found myself inspired by the platform's potential. By combining elements from various examples, I began building my first User Defined Function (UDF), eventually discovering the App Builder - a feature that would prove crucial in creating an interactive interface for hyperspectral data exploration. Having worked extensively with Google Earth Engine (GEE) and NEON data, the recent announcement of NEON AOP's availability through GEE presented the perfect opportunity to test Fused's capabilities. My goal was simple yet powerful: create a user-friendly application that could tap into this wealth of hyperspectral data and make it instantly accessible to researchers.

Building the Fused App​

The resulting application streamlines what traditionally would be a complex data acquisition process into a few simple clicks. Users can select any NEON site and survey year from a dynamically populated list - a feature that automatically stays current with GEE's data catalog, as NEON sites are surveyed at different frequencies. Once a site is selected, users can click anywhere within the highlighted area to instantly view all 426 bands of spectral data in an interactive plot.

Under the hood, each click triggers a carefully crafted UDF that connects to GEE's data catalog, extracts the hyperspectral values for that specific location, and transforms them into a clear, interactive visualization. What makes this particularly exciting is how Fused handles all the infrastructure complexity, letting me focus solely on the analytical workflow and user experience.

Looking Ahead​

Looking ahead, I'm already planning the next phase of this app. I just want to add functionality that tracks sampling locations and allows users to export their collected data as CSV files, complete with all 426 bands and additional metadata. This will enable data users to easily build training datasets for Machine Learning applications, develop new spectral indices, or conduct detailed statistical analyses.

The experience of building this demo app has reinforced my belief in the importance of platforms like Fused in the geospatial community. They serve as crucial bridges between massive datasets and practical applications, eliminating infrastructure headaches and letting researchers focus on what matters most - the science.

For those interested in exploring NEON's hyperspectral data or learning more about this application, feel free to connect with me or try the app for yourself here.

The future of remote sensing analysis lies in making powerful data more accessible, and I'm excited to be working in this field.

Sameer, co-founder of DigitalTwinSim, leads the development of advanced geospatial analysis tools to support the telecom industry in strategic network planning. DigitalTwinSim specializes in using high-resolution data to optimize the placement of network towers ensuring reliable, high-speed connectivity.

In this blog post, Sameer shares how he leverages Ibis with a DuckDB backend, and Fused to model wireless networks at high resolution. This approach enables him to quickly generate network coverage models for his clients. He explains and shares a Fused UDF that processes data in an H3 grid to evaluate optimal locations for network towers.

Introduction​

Broadband Equity Access and Deployment Program provides funding to expand high-speed internet access by funding planning, infrastructure deployment and adoption programs across US. BEAD prioritizes unserved and underserved locations, with a mandate to guarantee 100/20 Mbps service.

Terrestrial fixed wireless technology, utilizing a hybrid of licensed and unlicensed spectrum, is one of the approved technologies for regions where fiber deployment is cost-prohibitive. The Tarana Wireless G1 Platform exemplifies an FWA technology that meets program requirements at a significantly lower cost than fiber.

Since BEAD requires 100/20Mbps service with coverage guarantees, it is critical to model networks accurately to ensure compliance with program standards. We use 7.5 cm resolution data from the Vexcel Data Program to model networks and help our clients meet these requirements.

Modeling networks across large, sparsely populated geographies targeting every building generates massive datasets. Traditional tools like QGIS struggled with interactive filtering of such large datasets. In contrast, Fused allows us to filter, visualize, and share data interactively with clients.

Modeling Methodology​

We model wireless networks using an H3 grid at resolution 15, which translates to approximately 0.895m² per cell. This results in about 111 Million H3 cells per 10 km² city area.

Identifying the optimal serving site for each H3 index requires processing 11 million groups, each containing 5-10 rows. Previously, this was done with Pandas and Dask, facing limitations in CPU and memory capacity.

Integrating DuckDB into our workflow has significantly eased these constraints, enabling complex group_by operations on large datasets effortlessly. For example, the following command operates on a parquet folder with hundreds of files representing candidate site locations, helping us identify the best serving site for each H3 cell.

ibis.read_parquet('parquet_folder').agg(by=['h15'], Rx_dBm=_.Rx.dBm.max())

This efficiency is maintained regardless of dataset size or available memory.

Laurens Kuiper's blog post "No Memory? No Problem. External Aggregation in DuckDB", explains DuckDB's approach to scaling up grouped aggregations—an approach that directly supports our work in high-resolution wireless network modeling. By utilizing Fused with DuckDB, we at DigitalTwinSim have harnessed its capabilities to efficiently manage extensive data aggregations, enabling us to scale up projects in both resolution and candidate site counts.

Easy Integrations

Traditionally, we've used QGIS with GeoParquet as our file storage for visualizations. However, the sheer amount of data generated from each site at H3 resolution 15 has made it difficult to filter outputs interactively in QGIS. Fortunately, Fused HTTP endpoints make it easy to dynamically integrate UDFs with QGIS.

This image shows the output of the UDF for five network nodes rendered in QGIS.

Fused for Interactive Processing With Instant Visualization

Here, tools like Fused have become essential. Fused allows us to filter and visualize raw output data in a more interactive way, which we can also share with clients to illustrate network design and coverage areas.

To set up the UDF in Fused, we uploaded our data as a Hive-partitioned Parquet folder and created a UDF in Ibis to generate visualizations on demand based on zoom level and area of interest. At higher zoom levels, we compute the parent H3 index and aggregate data to show broader coverage areas; at lower zoom levels, we display individual H3 indices. The H3 polygons are generated and colored dynamically based on the data in the Parquet folder, allowing us to interactively filter data and share visualizations with clients.

Click here to launch the UDF in Fused Workbench.

Conclusion

As network demands grow and requirements for high-speed internet access become more stringent, accurate, high-resolution modeling is essential for effective planning and deployment.

DigitalTwinSim's integration of tools like DuckDB and Fused, alongside Ibis and H3 grids, enables us to tackle the challenges of processing, analyzing, and visualizing massive datasets. By leveraging DuckDB's powerful data aggregation capabilities, we can manage and analyze high-resolution data efficiently, irrespective of memory constraints. Meanwhile, Fused empowers us to deliver interactive, client-ready visualizations, allowing stakeholders to better understand network coverage and performance.

Foursquare just released an open dataset of over 100M global places of interest.

We at Fused have partitioned these points into easily accessible GeoParquet files, and hosted them on Source Cooperative

On top of that, we've build a publicly available User Defined Function (UDF) that anyone can use to easily look at & download to GeoJSON, all from the browser

Try it out for yourself!

You don't need to login or create an account to easily access the Foursquare POI points

• Join our waitlist to get full access to UDFs that you can edit in your browser
• Get familiar with Fused by checkout out our Quickstart docs
• Follow us on LinkedIn to keep up with updates
• Read this more in-depth look at the whole dataset from our community member Mark Litwintschik

I am a self taught developer and data enthusiast. I first came across the spatial data community when I saw a Matt Forrest video on LinkedIn where he demonstrated how to visualize buildings from the Vida Combined Building Footprints dataset with DuckDB. Immediately I thought, what if you could see all the buildings in a country, say, Egypt? I set out to do just that and made this map with DuckDB and Datashader.

Buildings in Egypt.

Starting with Fused​

The day after I posted that image on LinkedIn, in April, 2024, I had a call with Plinio Guzman of Fused. I told him what I had been up to, and he was enthusiastic and confident that Fused would fit my needs. One key feature he mentioned was the live development. While I was developing that Egypt map, I had to start the ETL to the final product over and over until I got it looking the way I wanted.

So I got started right away. I found Fused User Defined Functions (UDFs) like Overture Maps and the S2 Explorer and traveled all over the world looking for stunning images. It was thrilling to fly from New York to Tokyo and see the results render instantly.

Exploring the world with Sentinel 2.

I then began to change the components of these UDFs to see different Overture types, but at this point I was hesitant to build my own UDF from scratch.

Instant UDFs​

That was until Fused launched its File Explorer. With one click, it was now possible for me to create a UDF from providers like Source Cooperative and visualize with numerous presets like DuckDB or GeoPandas. With this new feature, I recreated my Egypt map with the same Vida dataset, this time using DuckDB with the H3 extension. It was liberating, I came to realize the components were simpler than I thought.

Local Tests​

I used DuckDB with the H3 extension without Fused to query Overture Maps for countries and continents all locally in a Jupyter notebook. The benefit with the H3 extension is that if you set up the query right you can aggregate larger than in memory datasets at ease from your notebook.

Road Density in Africa.

And made this Egypt building map with H3, how does it compare with the Datashader version up top?

Egypt Building Density with H3.

Fused and Overture Maps​

In August, Fused announced a tighter partnership with Overture Maps Foundation and that came with even more Overture features. Like with Source Cooperative I could now instantly generate UDF of buildings, places, land use, or roads, etc by joining parquet files (and more). I proceeded to use the framework of that UDF to join all kinds of data.

Proximity analysis between Road Networks and Hospitals in Paris.

Joining H3 with GeoJSON​

One day I was looking at the DuckDB_H3_Example, and I was struck — what if I joined those cells with Overture Buildings? I learned how to use the DuckDB H3 extension from all of the example UDFs on Fused. So I called that UDF in an Overture UDF and used GeoPandas to join the two. The result is the map below. The color of the buildings comes from the count of corresponding Yellow Cab pickups. There are millions of points in this TLC parquet file, and H3 helped me to aggregate to thousands for an easier spatial join.

Overture Buildings joined with H3 Yellow Cab pickups in New York City.

I made this particular map with Kepler.gl, with two clicks from the Workbench. I could have also exported the data to tools like Felt and Mapbox. You can find the code I used to recreate this map here.

App Builder​

I just started working on the Fused App Builder, and made a dashboard to view and interact with NYC’s 311 call data as a 3D H3 heatmap. Anyone using it can set the date range and resolution to change the display. Very fast and easy to use.

Community​

There's so much exciting data science happening on Fused. Check out Kevin Lacaille's post on ML-less global vegetation segmentation at scale. And Christopher Kyed's Analyzing traffic speeds from 100 billion drive records, that is the kind of project I would love to work on.

I continuously find inspiration as I browse community UDFs. Here's a join of H3 heatmaps with Overture types. This is a heatmap of connectors (intersections) joined with segments (roads) in London. The darker colors have more intersections. I am looking to incorporate traffic counts.

Road density in London.

Conclusion​

I have been using Fused for several months but it feels like I am just getting started. It seems like the only real limit here is what I can dream up.

This is a cross-post of Stephen Kent's Medium Article published October 14, 2024.