Is it possible to enrich a 3D model of the city using image recognition? That was the central research question of a pilot project Spotr executed with the municipality of Rotterdam. The answer? Yes, it’s possible. In this interview, Jane Hermans-van Ree and Tjits Tuinhof from the Municipality of Rotterdam explain more about the trailblazing project.
This pilot was part of the ‘Total Three Dimensional’ program, an initiative of the municipalities of The Hague, Rotterdam and Amsterdam, in collaboration with the Association of Dutch Municipalities. Aim of the program is to create a breakthrough in the field of three-dimensional information. The municipalities are investigating, among other things, how three-dimensional information that is already available can be included in their registrations. This way, they’re able to gain ánd provide more insights with the enriched 3D models.
Experimenting and collecting pieces of the puzzle
“We are truly in the midst of a transition period in the Netherlands,” says Jane Hermans-van Ree, program manager 3D at the Municipality of Rotterdam. “The transition from analog to digital was huge, but the transition from 2D to 3D has at least as much impact. This transition is going one step at a time. We are now in a trial phase, collecting pieces of the puzzle and gaining experience. The program and this pilot are part of that phase.”
Detecting windows and doors from images
Tjits Tuinhof, advisor at the Municipality of Rotterdam, explains which problem the municipality specifically had in mind for this pilot.
“3D models have different so-called ‘Levels of Detail’. In Level 1, the building is only a square block, in Level 2 it already has some roof shapes, and in Level 3 it’s a complete building with windows and doors. We wanted to know: is it possible to get from Level 2 to 3 by detecting windows and doors from existing images? What is needed to get there and how do we include this in our registration? When housing association Woonstad heard about our problem, they suggested to contact Spotr.”
The value of an enriched 3D model
Spotr develops image recognition models to map all aspects of the exterior of a building using artificial intelligence.
“This is what we were looking for,” says Tjits. “We sat down with Spotr to brainstorm about what we could use the 3D model for. We didn't have any specific use cases in mind at the time. We were mainly motivated by the idea that the use of such a model is stimulated if people are actually able to recognize more details. If it's less abstract. But during the brainstorm with Spotr, all kinds of ideas arose.”
“For example: if we can see where doors are located in the 3D model and how high the sills are, we can map out the risks of flooding. And: if we can see where windows are located, we can show what developing a new residential tower does to the light and shadows on the windows of nearby houses.”
“The enriched 3D model can also be extremely valuable for the police and fire department. They are better prepared when knowing in advance where the windows and doors are located in a certain building. Comprehensive information about each building also helps to plan maintenance and estimate costs. And by adding even more details to the 3D model, we can see, for example, how many roofs are already equipped with solar panels and where there are still opportunities in the city to installing new panels.”
Capturing the entire process
Conclusion of the brainstorm: let's get to work. The first step was to collect and process images.
“There are certain conditions to collecting and processing images,” says Tjits. “Because the visual data must be provided with precise information for a proper interpretation. Such as: the exact spot and angle from which the photo was taken. We make sure to continuously write down these kinds of learnings, so other municipalities can learn from them in the future.”
The precise location and angle of the image is important because it determines the location of the element in the 3D model. If there’s a window or door in the photo, the computer program cuts a hole in the block that is the 3D model, and pastes a window or door in that exact spot. The model has to remain accurate during this practice, because this impacts other analyses as well.
“This process of projecting windows and doors onto the 3D model was a puzzle. The main reasons for this were that the 3D model sometimes deviates from the real situation and Spotr uses a 2D basis in the current process, which made positioning windows and doors on staggered facades a major challenge. In addition, it’s important that the technical structure, the semantics of the model, remain the same and that windows and doors are also added in a semantically correct way. All of these challenges were a big part of the main research question.”
Continuously better results
Jane: “We’re still figuring out a lot along the way. Some things we already do well, other things still need to be further explored. That is why it was great that we worked Agile with Spotr, in sprints of 2 weeks. This helped us improve every step of the way. You can really see the progress every time. We are impressed by what is possible, also by how the algorithm continuously can be improved. The results from the third and fourth sprint were significantly better than the results from the first sprint.”
3D model enriched 300 homes
In total, the project ran for 3 months, during which windows and doors were added to the 3D model of 300 homes. Tjits: “The completeness of the results is amazing, the image recognition has worked very well.” So in short, the answer to the research question – ‘Is it possible to enrich 3D models of buildings with information about where windows and doors are located based on existing photo material, using artificial intelligence?’, is: yes. Yes, that's possible.
Jane: “Together with Spotr, we achieved a really good result. And in a way of working that’s important to us. There was continuous interaction, there were ongoing conversations about how the work was developing and the progress was very transparent. Spotr is definitely a great partner to work with. Enthusiastic, knowledgeable, dynamic.”
Blueprint for other municipalities
What about their next step? Jane: “Our next step is, among other things, finding an answer to the question: how do we organize the management of the 3D model? Because it is of course crucial that the 3D model continues to match reality. If a dormer is placed somewhere, this should also be implemented in the 3D model. That's one of the things we're working on now."
The results of the pilot are also intended to inspire other municipalities. “We have written everything down, so all the tools municipalities need to include extra information in the registration based on visual material, are there. From what is needed as input and what comes out, to how to include all of this in a 3D registration. But most of all we included information about what you can do with it. This way, we'll move step by step towards capturing the city in 3D together, and experiencing the city in 3D the same as we experience it when we actually walk outside.”
Up-to-date and reliable data in one place
Meanwhile, Spotr is not standing still. By continuously improving their models, results are becoming more and more accurate. And this is not limited to windows and doors. Spotr detects a total of 80 objects on the facade, roof, surroundings of the house and interior. Spotr's AI also recognizes defects in homes with algorithms. All this information automatically comes together in one accessible platform.
.png)
