Haly.Id-lens
This incubator presents the new application, Haly.Id-lens, that has been developped as a byproduct of the research in Haly.ID project. The purpose of Haly.id-lens is to disseminate among the public the results of our ongoing research.
The haly.ID project is focused on the monitoring of an invasive emergent pest, called Halyomorpha halis (Hh), and commonly known as the Brown Marmorated Stink Bug (BMSB). This pest is well known to the farmers for the extensive damages that leads to orchards (especially, orchards of pears, peaches, and hazelnuts). BMSM causes annoyance in urban environments, expecially in fall, when it seeks refuge in cracks for overwintering.
Haly-id lens is a citizen science activity. It is an application that educates citizens to recognize BMSB, testing and stregthening their ability to scout BMSM. In the same time, the researchers collect the images taken by the citizens to enhance their BMSB detection models.
The Haly.id-lens is an application with a multi-clients / server architecture. Once installed in any device, the app prompts the user to upload an image. The image is sent to the server which after few seconds returns the same image with a prediction box around each BMSB specimen found (possibly none). Each prediction box is augmented with the confidence level , an index that measures the trust of the model in its evaluation.
At the server side, each georeferenced received image is saved for further use in the BMSB dataset. Such a dataset will be made available to the scientific community at the end of the Haly.id project. The image is processed using three (medium, large, extralarge) models trained ad-hoc for urban images. Other models, like those trained by the main HALY-ID project, can be included as well. The prediction boxes with the highest confidence are returned to the client that uploaded the image.
One of the goals of the Haly.iD project is to replace field inspections of phytosanitary personnel with remote inspections of drones. The drones take photos of trees and pass them to computer vision algorithms able to scout BMSB. To analyze the images, computer vision algorithms trained on BMSB detection are needed. During these two years of the project, we learn that there is no computer vision model suitable for any kind of image taken. So it is not only necessary to have hundreds of images of the insect, but it is required to have good quality similar images, especially images with the same field of view and similar backgrounds. During the first year of the project, we trained our models with images taken in first person of view mode, with the drone flying inside the orchard and staying at few meters of distance from the trees. During the second year of the project, we trained our models with images taken while the drone autonomously flies above the trees (at an average distance of 10 meters).
In addition to the above activities all planned into the project, we added a new goal: to raise the citizens awareness of this invasive insect. We wanted help citizen to scout BMSB. This would have highlighted the impact of our project and put the experience gained in Haly.id to good use for everyone. Therefore, we planned to realize an app that can be used by any user to test the BMSB presence in their gardens or houses.
We propose a multi-clients / server architecture to let citizens scout BMSB. Once installed in any device, the app prompts the user to upload an image. The image is sent to the server which applies YOLO algorithms, some from the Haly-Id project and some ad hoc for citizen images, to create prediction boxes around BMSB specimen found. At the server side, the images uploaded by the citizens will be also saved for further use by the researchers.
The idea of an app that is used by citizens to recognize objects is not new, but at the moment, at the best of our knowledge, there are no available apps to identify this insect. Moreover, the app could have an impact on the citizens behaviour. If citizens are in habit of using the app, the idea can easily be extended to other invasive pests. The app could become an alarm system under the supervision of experienced entomologists that analyze the collected images, scouting new insects not yet registered in Europe (e.g., the spotted lanternfly -- Lycorma delicatula -- who is nowadays an invasive pest in USA).
