Calvin Hung and Salah Sukkarieh
Australian Centre for Field Robotics, The University of Sydney, Camperdown, New South Wales 2006, Australia.
Hung, C. and Sukkarieh, S. (2015) Using robotic aircraft and intelligent surveillance systems for orange hawkweed detection. Plant Protection Quarterly. 30(3), 100-102.
Remote sensing using robotic aircraft (either Unmanned Aerial Vehicles (UAVs) or remotely piloted) can be an efficient and cost-effective way of collecting geo-spatial information over large areas. Furthermore, the large amounts of data collected by these systems can be handled by intelligent software systems that are able to autonomously classify the data. The Australian Centre for Field Robotics (ACFR) has been using aerial robots for environmental monitoring and has developed algorithms to support weed detection and mapping projects. The algorithm research at the ACFR has led to various intelligent detection and mapping software systems for accurate terrain mapping, vegetation segmentation and detection of different invasive species. The ACFR recently completed detecting and mapping of orange hawkweed (Hieracium aurantiacum subsp. Nägeli & Peter) at two different sites in Kosciuszko National Park, New South Wales. This paper presents an update of recent developments of the autonomous weed detection research program for orange hawkweed and a discussion on the current operational constraints and algorithm limitations to be addressed to support future large scale implementation.
The technological advancement of robotic aircraft (Unmanned Aerial Vehicles (UAVs) or remotely piloted) along with light-weight imaging sensors over the past decade has seen an increase in their usage for remote sensing applications. Robotic aircraft have the advantage of potentially being more cost efficient and can be operated from remote locations where there is reduced infrastructure. In addition, robotic aircraft have the potential to operate at lower altitudes, allowing collection of high resolution image data, and hence an ideal tool for weed management applications (Dehaan et al. 2012, Clements et al. 2014).
Over the last decade the Australian Centre for Field Robotics (ACFR) at the University of Sydney has developed robotic air vehicles and intelligent software systems for environmental monitoring with particular emphasis on weed detection and eradication (Goktogan et al. 2009, Hung et al. 2012, Hung and Sukkarieh 2013, Hung et al. 2014). ACFR is now working with land managers to develop an intelligent detection system for orange hawkweed (Hieracium aurantiacum subsp. Nägeli & Peter), a non-native, invasive weed that is the target of an eradication program in New South Wales, Australia (Cousens et al. 2012, Caldwell and Wright 2013, Hamilton et al. 2015). Continuing the work presented in Hung and Sukkarieh (2013), this paper details progress towards the aerial imaging of orange hawkweed using an UAV and developing algorithms to automate the detection process.
The objective of this study was to collect image data of orange hawkweed flowers at different altitude settings, ranging between 5 to 40 metres, and to develop and test orange hawkweed flower detection algorithms.
The standard methodology defined in Hung et al (2014) is followed here. The methodology is decomposed into three steps: (1) data collection using the UAV which includes site trial selection and setting; (2) image pre-processing which includes data handling; and (3) weed detection that uses algorithms for the classification of the vegetation.
Site surveys and data collection
The surveys were carried out 15–17 January 2015 to coincide with the flowering season of orange hawkweed. This allowed discrete detection of orange hawkweed flowers as the flower has a distinct orange colour and no other flowering species in this area have this colour.
Field trials were performed at two different sites (Ogilvies Creek and Fifteen Mile Ridge) 12 km from Tooma Reservoir in Kosciuszko National Park, New South Wales. Originally five sites were proposed for the survey, however only two sites were flowering in mid-January and were the only sites surveyed.
The first orange hawkweed site (36º2’10.07”S, 148º19’14.00”E, Figure 1) is shown with the trajectory of the UAV (Figure 2). The coordinates of the first site were slightly different from what were originally proposed and were outside of the defined region of the ground station (the location of UAV take-off and where the ground-computing and the communication system was located). The robotic aircraft was controlled manually by a pilot to survey this site. The pilot used visual feedback to position the platform over the orange hawkweed flowers. The robotic aircraft moved mostly in the vertical direction (Figure 2). An area of approximately 40 by 60 metres was surveyed at site one.
Site two (36º1’12.81”S, 148º23’35.69”E, Figure 3) is shown with the trajectory of the robotic aircraft (Figure 4). The coordinates of site 2 were within the original proposed region of interest and the survey was performed both manually around the flowers and autonomously following a pre-defined trajectory to cover the entire area. The trajectory shows the vertical movements near the flowers performed by a human pilot and the raster scan performed autonomously (Figure 4). An area of approximately 80 by 120 metres was surveyed in site two.
Aerial images of orange hawkweed were collected using the AscTec™ Falcon 8 (Figure 5). The vehicle was equipped with a downward pointing camera (Sony Nex 7™) for data collection. The technical specifications of the vehicle and camera have been summarised (Table 1 and 2).
During the survey the UAV was flown over the sites of interest at various altitudes between 5 to 40 metres and captured an image approximately every 5 seconds between 10 am to 2 pm. Four flights were performed per site to ensure full coverage of site one and two.
Image stitching and geo-registration
Mosaicking (where images are stitched together to provide a unified map of the area) and geo-registration (referencing this map to GPS coordinates) of the images into a single ortho-photo were conducted using off-the-shelf software, Agisoft PhotoScan (Agisoft LLC, St. Petersburg, Russia). In this study, markers were laid out around the weed/s of interest, such that there was typically at least one marker visible in each image frame. The image stitching and geo-registration required information from: (1) the image global positioning system (GPS) locations logged by the UAV; and (2) the feature points extracted from the images.
Orange hawkweed detection using colour
A classification algorithm was then developed to detect the weeds in an image. The development of a classification algorithm typically starts from using simple lower dimension features (such as colour, texture and shape of the target and background). The complexity of the algorithm and dimensionality of the features can be increased throughout the development cycle if the classification performance is not sufficient. This approach is taken because simple algorithms require less training data and have faster processing speeds.
In this study, the colour-based classifier was sufficient due to the unique orange colour of the orange hawkweed flower. The colour of the flower was the most distinctive feature of orange hawkweed during the time of the survey. Using the colour of the flower as the main predictor allowed for the easy construction of a simple and robust classifier that can run in real-time. Images from a standard colour camera are normally captured in the Red, Green and Blue (RGB) colour space. This colour space is not optimal for training a detector because the channels are highly correlated. In this study, the images were transformed into Hue, Saturation, Value (HSV), YCbCr and Lab colour spaces for evaluation (Figure 6). Ultimately, the Lab colour space was selected because it was robust against changes in lighting conditions.
We next identified the orange hawkweed flower spectral ranges in the Lab colour space. A subset of images collected during the trial (training images) representative of the scene, containing different backgrounds, objects and other flowers were selected for the study (see, for example, Figure 7). These were used to manual tune the algorithm for parameter selection. The optimal values (the upper and lower bound of three colour channels) and the final output containing only the orange hawkweed flower is shown (Figure 8). This parameter setting was then applied to all the images collected from the trial to identify all the orange hawkweed flowers over the whole trial.
The two sites with flowering orange hawkweed was successfully surveyed. The weather condition was ideal (sunny, no wind) and the survey was completed within four hours.
An area of site one with a high density of orange hawkweed flowers is shown (Figure 9). The top figure is from the mosaic map and the bottom figure shows the detection results. The bright spots in the detection results indicate locations with high probability of orange hawkweed flowers. The close-up views of the four high probability regions (labelled 1 to 4 on the detection results) are shown (Figure 10).
The colour-based detector correctly identified the orange hawkweed flowers in the survey sites, however false positives did occur in the second site because orange ribbons with the same colour as the orange hawkweed flowers were used to label the weeds (middle image, Figure 11). To remove false positives, we implemented a more complex algorithm which incorporated additional features such as the geometric properties of the flower. (We could have manually ignored the ribbons but took this opportunity to improve the algorithm in case there were other features of similar colour). The final detection contained only the orange hawkweed flowers without the ribbons (shown on the bottom image in Figure 11 highlighted by the arrow and detection square).
The relationship between light altitude and image resolution
Given that the diameter of the orange hawkweed flower is around 15 mm, this placed an operational constraint on the UAV flight altitude. The relationship between flight altitude and pixel size is shown (Table 3). When the UAV operates 5 m above the ground, the flower has a diameter of 15 pixels, whereas at 30 m above the ground the flower is made up by only 2 pixels.
The image patches of the flowers extracted from images taken at different altitudes are shown (Figure 12). At lower altitudes, the structure of the flowers could be clearly resolved, and the resolution decreased with increasing altitude. At 30 m, the flowers were represented by only a few pixels and there was significant pixel mixing (where a flower pixel mixes with the background). This can reduce the performance of the detection algorithm.
This paper outlined the aerial image collection using UAV and data analysis of orange hawkweed at different altitudes from two sites in Kosciuszko National Park during the flowering season in mid-January. This study showed that the detection of orange hawkweed flowers was achievable. In addition, due to the unique orange colour of the flowers, a relatively simple and fast colour based detection algorithm can provide robust results.
This study also highlighted the operation limits using the same UAV and camera specifications. These will inform future trials. Due to the small size of the flowers, the UAV should fly at altitudes lower than 15 m to maximise the image resolution and should travel relatively slowly to minimise motion blur. Alternatively, a higher flying UAV with higher resolution sensing and a faster frame rate will also achieve the same task, but at greater expense.
The advancement of robotic aircraft and light-weight sensor technology has increased their use and potential for remote sensing applications. These systems can be used to collect images covering a large area as well as access difficult terrains in order to assist weed management programs. This paper introduced a vision based image analysis algorithm to detect orange hawkweed, allowing the timely process of large image datasets for future missions. The platform used has an operating time of approximately 30 minutes. Currently the ACFR is trialing a fixed wing robotic platform that has an operating time of 5 hours thus vastly increasing the amount of area covered.
Future work will focus on the development of detection algorithms using the geometry of plants and leaves in addition to the flowers. This will allow detection of orange hawkweed outside the 4–6 weeks flowering season, providing land managers ability to detect new infestations during the non-flowering period.
This work was supported, in part, by the National Landcare Programme Innovation Grants (INNOV-084), the Australian Government Department of Agriculture, the Northern Tablelands Local Land Services, and the Australian Centre for Field Robotics at the University of Sydney.
Caldwell, J. and Wright, G. (2013). Orange hawkweed control program report 2012/13. (Office of Environment and Heritage, New South Wales National Parks and Wildlife Service, Hurstville).
Clements, D., Dugdale, T., Hunt, T., Fitch, R., Hung, C., Sukkarieh, S. and Xu, Z. (2014). Detection of alligator weed using an unmanned aerial vehicle. Plant Protection Quarterly 29, 84-9.
Cousens, R., Heydel, F., Giljohann, K., Tackenberg, O., Mesgaran, M. and Williams, N. (2012). Predicting the dispersal of hawkweeds (Hieracium aurantiacum and H. preaealtum) in the Australian Alps. Proceedings of the 18th Australasian Weeds Conference, ed. V Eldershaw, pp. 5-8. (Weed Society of Victoria, Melbourne).
Dehaan, R.L., Weston, L.A. and Rumbachs, R. (2012). The design and the development of a hyperspectral unmanned aircraft mapping system for the detection of invasive plants. Proceedings of the 18th Australasian Weeds Conference, ed. V. Eldershaw, pp. 103-7. (Weed Society of Victoria, Melbourne).
Goktogan, A., Sukkarieh, S., Bryson, M.T., Randle, J.A., Lupton, T. and Hung, C. (2009). A rotary-wing unmanned air vehicle for aquatic weed surveillance and management. Journal of Intelligent and Robotic Systems: theory and applications 57 (1-4), 467-84.
Hamilton, M.A., Cherry, H. and Turner, P.J. (2015). Hawkweed eradication from NSW: Could this be ‘the first’? Plant Protection Quarterly 30, 110-5.
Hung, C., Bryson, M. and Sukkarieh, S. (2012). Multi-class predictive template for tree crown detection. ISPRS (International Society for Photogrammetry and Remote Sensing) Journal of Photogrammetry and Remote Sensing 68, 170-83.
Hung, C. and Sukkarieh, S. (2013). Robotic aircraft and intelligent surveillance systems for weed detection. Plant Protection Quarterly 28, 78-80.
Hung, C., Xu, Z. and Sukkarieh, S. (2014). Feature learning based approach for weed classification using high resolution aerial images from a digital camera mounted on a UAV. Remote Sensing 6, 12037-54.