Hi everyone, thank you for joining us for today's webinar on land classification with multispectral imagery. Please feel free to ask your questions in the questions box throughout the presentation, and we'll do our best to answer them as we go. We'll also have a Q&A session at the end of the presentation.
CanDrone is Canada's leading consultancy in aerial and ground-based remote sensing solutions. We are a full-service drone agency, offering everything from hardware and software sales, rentals, training, and piloting services, to custom drone engineering solutions. We offer drone-based solutions and land cover classification using multispectral image analysis. We offer this service in industries such as forestry, agriculture, and urban development.
We bring you this webinar to provide an overview of this technology and parts of the workflow. My name is Cody Wildgust, Strategic Sales Expert, and this is Ian Perry, Solutions Specialist. On today's agenda, we will introduce the topic with some technology, talk about multispectral imagery, describe the major components of land cover classification, and briefly highlight some uses of this powerful technology. So with that, I'll turn things over to my colleague Ian to explain for us what is multispectral imagery.
Yeah thanks Cody. So multispectral imagery captures data from light in various bands in the electromagnetic spectrum. So light can be thought of as a wave and the waves range in length across the electromagnetic spectrum.
Our eyes detect red, green, and blue light waves in the visible subsection of the spectrum. And our brains generate a true color image when we look at the world. And this visible spectrum consists of waves ranging between 400 and 700 nanometers in length.
So when we see different colors, are those actually just different wavelength variations of the same electromagnetic wave? Those colors are all wavelengths of... just that a slightly different length and so they say you know blue green they scatter differently they behave differently with different objects on the planet that's why you see water as blue sometimes and why you see plants is green etc awesome the multi-spectral cameras are engineered to see light traveling in waves that our eyes cannot see and these cameras capture each wavelength in a frequency called a band and often have a lens wholly devoted to producing an image in bands like infrared and red edge. I should say near infrared to be more specific and red edge. In addition to blue, green and red data and unlike color cameras like the one in your phone, multi-spectral cameras present us with a separate photograph for each band.
This means that we can isolate the reflectance of individual light waves such as blue off of objects such as trees and bare earth and buildings. So here you see the Micasense Altum which consists of five different lenses for red, green, blue, red edge and near infrared. Mounting this camera on a drone like the DJI M300 we could survey an area and capture photographs from each of these bands. Processing the data in a photogrammetry software suite such as Pix4D give us a separate orthomosaic in a TIFF format for each band.
Great. Yeah, and Mark had a question here. He was asking, can you do multispectral with an RGB camera? Well, yes, you can.
You can isolate red, green, and blue bands and create a separate image for each. And then you can put together some index that uses one or two or three of those bands. So yeah, you can actually absolutely pull apart a regular color photograph and manipulate it to accentuate things that we're looking for.
We'll get into that. So with an RGB camera, though, you'd likely be missing those red edge and near infrared bands that we get from a multi-spectral camera, right? Correct. Okay, great. So each of the five orthomosaics from an Altum camera can be brought into a GIS environment such as QGIS or ArcGIS as a separate layer.
And if you add the usual red, green and blue layers into a project and overlay them, you produce a true color composite, just as you see in the JPEG coming off of your phone. With a multispectral camera, we have those two additional bands, red edge and near infrared. covering a region outside the visible spectrum and which can produce spatial information that is not available by examining a true color composite. Okay, and red edge is that region of rapid change in the light spectrum between red and near infrared, is that right?
Yeah, exactly. And what that essentially offers is the sort of the nuances in vegetation. So when you want to see variability in a tightly packed canopy or variability within a plant that is sort of multi-layered, red edge is an emerging technology that's really taking the...
the market by storm which a lot of agriculturalists prefer over traditional NDVI. We will get into that a little bit later on in the presentation. Awesome. So I pose a question why do we care about isolating a certain band from the electromagnetic spectrum to produce separate ortho images? The answer is because we can measure how the various light waves interact with objects to get that object's spectral signature.
So in analyzing the spectral signature, we can easily identify objects or their qualities and make quantitative assessments about their condition. So for instance, healthy plants reflect a lot of near infrared light by contrast, the chlorophyll in healthy plants absorbs a lot of red light. And you can see this represented pictorially on the left, but then graphically on the right. And you can see there's a trough in band three, which is red for vegetation and a spike in band four for near infrared with vegetation. So isolating near infrared and red signals and then recombining them will give us an idea of where a landscape contains healthy plants and where it contains not so healthy plants.
Okay. So does that mean that we could use multi-spectral imagery to target problem areas of a crop more precisely? Yeah, absolutely.
So it's not always obvious where stressed vegetation is, though it may be obvious to the observer in color where dead plants are. Do you think you could tell a stressed cherry tree from a very healthy one from 60, 80 meters up? I definitely know that I couldn't.
Might be challenging right? Yeah so that could really make things a lot more efficient out in the field in terms of resources used and the amount of time that you're spending on things. Exactly so instead of walking throughout an area to just sort of gauge health empirically on foot you're doing so over an area rapidly. This is a much more efficient system. Yeah.
Okay so Let's say you want to quantify plant health in the ways that we've been discussing in an image. After you fly your field with your drone and your multi-spectral camera, you process drone images using photogrammetric software like Pix4D. You get a geo-tiff for each band image and you want to use the near infrared and red pixel values to represent plant health. So we want to take you under the hood and show you how this works. So pixel values for red and near infrared need to be normalized for ease of interpretation.
So we create a ratio of near infrared signal to red signal because those light waves are at the high and low end of what gets reflected by healthy vegetation. And we normalize the pixel values to a scale that's convenient. So in this case it's minus one to plus one with values near plus one representing healthy plants and and minus one representing pretty well dead vegetation. We can also represent non-living entities like water or gravel.
Okay. When we perform this calculation for every pixel across two images and we render a brand new raster. So we had originally a red one we had a near infrared and then when we're finished with this band math or raster math we are left with a brand new layer in our GIS.
And this is a new TIFF file that provides a normalized ratio of near infrared to red pixel values. And this is called NDVI or normalized difference vegetation index. And it is a great indicator of plant health.
And you can see the arithmetic on the slide there and what the values might come out to in healthy versus stressed plants. Yeah, so I get the idea here that lighting is becoming a very important factor in this data collection. So do varying light conditions affect the data collected and if so how do you go about mitigating that during emissions?
Yeah, you definitely want to make sure you have enough light to render your multispectral images properly. And a camera like the Micasense Ultem or the Sequoia, it has a sunshine sensor that would be, in the case of the DJI M300, placed on top of that drone. And that sort of uses the sunshine or lack. thereof to calibrate the camera before it takes images right in in conjunction with the calibration target so it's very important that you do calibrate your your camera i have read that you want to calibrate before and after you fly in case the lighting conditions changed a little bit throughout your flight but i think what you want to avoid would be really really sunny and shadowy conditions and you'd also want to avoid flying when it's quite dark and overcast, such as a winter day here in Vancouver, where you get big thick clouds, et cetera.
So for lighting conditions in an ideal world, we're looking for very much the same requirements of what we would have for photogrammetry missions, right? Where steady overcast with constant brightness and not a lot of changing variables, no breaking up clouds or anything like that. Yeah, flying around solar noon, high overcast lighting conditions, better.
Better without a lot of moisture. All right well here we have a forest and meadow system in Banff National Park and this data was captured by a drone at about 110 meters flying height and the image on the left is that of a color camera on the right is that of the Micasense Sequoia a multi spectral camera with near infrared and red edge image capability. So using QGIS to do the raster math.
We put together an NDVI and placed it in the green channel. So the greater the chlorophyll content, the more brilliant the green color is in this image. The black areas indicate non-living materials.
So you can spot some of the snags and deadfall or non-living trees along with cured grass and gravel areas with the color image on the left. With the NDVI, you can see that the dead and other non-living surfaces really pop or stand out in the black areas of the image. So this simple multispectral raster could reveal lots in a forest management setting such as where insect outbreak or winter kill have occurred in a stand or surviving trees remain after a fire.
So it's a simple very powerful way to tell the living apart from the non-living. So can you tailor an Indice to target data for answering specific questions? For example, let's say we want to know if we're over or under watering an area in the crop.
Yeah exactly, with you know the proper sort of project goals defined, you can do your research, figure out the Indice that is best for your needs, and figure out exactly how to apply it and how to get the most. out of it. And on that note, it might be time to talk about some land class.
Okay, fair enough. So yeah, we've now seen how to create an image that emphasizes certain features, such as healthy vegetation, while providing a sharp contrast to dead vegetation. This is the essence of the NDVI, or Normal Normalized Difference Vegetation Index.
How do we go a step further and differentiate living and dead vegetation with statistical certainty? All right, that's a great question Cody and for that we need to classify an image. Okay, so differentiating a living coniferous tree and a dead one by simply examining the photo and making or marking it up manually, it would take a long time. And what about other objects in the image like water and grass?
If we need to differentiate several classes of objects rapidly across the entire landscape. we must use a machine learning algorithm to do so efficiently. There are two steps that come first, including selecting training and testing data and segmentation, which I will go through briefly in the following slides. But guys, keep in mind that I have tried to boil this down to absolute basics to save time and not get too technical. There are numerous resources online which delve further into this process and I'd be happy to provide them if you put a request into the question box.
Okay, great. But in step one of a land classification, we want to select a bunch of samples of the classes that we eventually want to depict in our mapping output. So as you can see here, there are points representing conifers, there are points representing snags and deadfall, water, grass, etc. And the pixels representing these objects will be used to train and test the machine learning algorithm.
So in this image a sample of living conifers, snags, grasses, shrubs, water, etc. were all classified manually and marked using a point vector layer in QGIS. This is done by scrolling around the image and grabbing a bunch of samples of each class pretty much at random. You want to spread it out make sure you have good coverage of your entire orthomosaic.
because about half the points are going to be used for training and the other half will be used for testing. So you want to have ample. So after we create our training and testing points, we need to generate some statistics for pixels representing our classes.
So rather than generate statistics for every pixel in the image, we want to group pixels based on objects they represent in the image and generate statistics based on each group. This is a technique known as image segmentation, and you can see an example of it to the right. So by creating segments of pixels, we reduce the file size of the raster and reduce the computational load for the machine learning algorithm. And this basically increases the processing speed, and that means that we can classify an image much quicker. This could be very important for mapping very large areas, thousands of squared kilometers.
drone. Right, yeah doing that manually would take a long time I imagine. Absolutely and doing it by pixels would be problematic. So segmenting pixels it smooths out noise.
This is the extreme pixel values that often come in a highly complex environment with high resolution imagery such as what you would get flying the M300 and the Altum. We don't want to overfit. our machine learning algorithm.
So segmentation really smooths out the range of values that are coming from our pixels. We want to maximize the classification results from our machine learning algorithm. This is a good way to do that.
So the segments are known as vectors in the GIS world and are made by applying a segmentation algorithm. to the entire image. We combine pixels into segments by their likeness using a segmentation algorithm in QGIS such as mean shift and we can alter the parameters of the segments by playing around with the menu options for mean shift in QGIS.
The function can be accessed through the Orfeo toolbox or OTB for short. And here is a sample of our color orthomosaics segmented in QGIS. And you see that this complex landscape, the segments are far from perfect, right?
Getting every tree sectioned off exactly right is difficult. They're actually over segmented with multiple vectors encapsulating one tree. But those vectors are going to bear similar pixel statistics. And we'll provide our...
our algorithm with a good set of training data. So this is a good start for us. Yeah, and Mark had a good question too, just covering some of these previous steps.
He just wanted to make sure, is all this done within QGIS then? Yes, the answer to that is yes. And I will review some of the workflow in a little bit. Perfect.
So just to move on to our final step, we talked about grabbing training and testing data, we talked about segmentation. In our final step we bring it all together and we build an algorithm using the training points from step one and statistics from segments in step two. In this example I'm going to talk about a random forest machine learning algorithm but you can use other algorithms to do this.
Essentially a random forest consists of a group of decision trees and hence the name random forest. So training points from step one that fall into pixel segments from step two produce what we call training segments. These training segments build our random forest algorithm and is easiest to think about it in tabular form first. So on the left, each row of our table represents a training segment. And remember, this is a cluster of pixels for which we derive statistical information.
The statistics in each heading come from the image segmentation algorithm. In this case, mean shift that we accessed in QGIS and they include but are not limited to the statistics mean variance skewness and kurtosis. So to begin building a decision tree we randomly select one of the training data one row of training data rather and randomly select one statistic as the first node in the decision tree.
We keep randomly selecting nodes until each of those statistics makes up a branch in the tree or becomes a variable in our machine learning algorithm. Once built we can run another sample segment from the testing set through this decision tree and mark whether that segment was classified correctly or not. So we build numerous decision trees out of many training segments and we run test data through those decision trees, we come up with a confusion matrix to gauge our accuracy. And once we're happy with the accuracy of the random forest, we use it to classify all of the segments in our raster image.
Okay, so with those initial steps, we were essentially creating a baseline for the algorithm to learn from. And then it follows that as it automatically classifies the rest of the area based on what we input or taught to it. That is correct. Great. So to summarize these three steps, we first select a set of points for each land cover class we want to establish.
This can be done by interpreting a color orthophoto like the one we have here in the slide, or we could also go to the field and directly identify samples of each land cover type. Yeah, and this could come in handy where we cannot easily locate objects by simply looking at them in the photo. In this crude example, I was able to just, you know, hand... select water, shrubs, conifers, just with this ortho photo on my computer. But you might actually want to use a high-accuracy GPS, like a Trimble or Leica unit with RTK capabilities, to mark a point in the field and import those points later on into your GIS in order to delineate training and testing data.
Again, I can't stress enough the importance of taking lots of sample points because you are going to split them into two groups, right? Training and testing. Right. Okay. So then the next step, we run the segmentation algorithm?
Yeah. Okay. And to Mark's question, this can be found in toolboxes and GIS environments like Q or ArcGIS.
The algorithm. The algorithm produces a set of vector files that segment objects based on similar pixel characteristics and these pixel characteristics are captured with statistics. We look at stats like the average reflectance, the shape of the distribution of reflectance known as kurtosis, the skewness of the distribution and the variance or how spread out those reflectance values are. Using statistics to summarize pixel values helps smooth out extreme pixel values or noise in the data.
And in the final step, we assign those statistics to our training points and build a set of decision trees. Great. So with the random forest built and the segments classified, we can assign image segments into object classes.
And this is a screen capture of seven classes of. land cover type as it appears in QGIS. Now I spent the first few slides talking about multi-spectral indices. So to tie back into that here, you could classify images of some spectral index instead of just true color. And this may be helpful where you want to classify objects which are only certain indexes or indices will render, such as dead or stressed vegetation using an NDVI.
Right. You could use the classification steps we use here to generate classes based on varying levels of plant health, say in agricultural settings where some areas are super stressed while others only moderately stressed and so on. So more on those applications in a minute. First, I do want to review some software and processing options. So this entire workflow can be done using free and readily available software.
QGIS is an open source geographic information system or GIS platform where users can analyze and edit spatial data. QGIS is available for Mac. Linux, Unix, and Windows operating systems. In this image here you can see a capture of the processing tool box containing Orfeo tools and the segmentation tool mean shift that we used in our example a few slides back. Okay and how about the random forest machine learning algorithm?
Is that something that's publicly available for people to use? Yeah absolutely random forest support vector machines these learning algorithms can be accessed through the GIS or in a coding language and that leads me into discussing a little bit about Python. If you're more like me and you want to see how things are done under the hood, this entire process can be managed with minimal usage of a GIS, a graphical user interface. It can be instead executed using the Python programming language. Now you still may need to use a GIS to pick training data for your image, but the segmentation and machine learning components that we covered can be performed entirely in a script.
And this may be optimal for larger projects where image classification must be repeated for hundreds or even thousands of GeoTIFF files. And it's also optimal where repeat drone flights produce data over several epochs, and where image data needs to be processed exactly the same way throughout a time series. So in short, scripting this will be rapid and will be precise.
Great. I know personally in projects I've worked on where staff turnover or students in a grad program in a lab. turnover and so you're trying to process data the way your predecessors did and if it's all scripted lives in a file somewhere you can just run your new raster image through your script and you know you're comparing apples to apples yeah and making sure that you have consistency throughout all those data sets would be very beneficial for helping to validate things yeah exactly so i want to address some of the variations to ndvi um with a normalized difference vegetation index. So there are situations where NDVI has to be adapted to compensate for factors in the landscape. So for instance where vegetation is sparse or where chlorophyll is limited and exposed soil overpowers the spectral response from vegetation, NDVI is modified to become the soil adjusted vegetation index seen here.
And this will be more effective for depicting healthy vegetation in settings where pioneering plants have been recently established, such as reclaimed mining sites, oil and gas sites, etc. This index is also handy in arid environments, dry grassland environments, where, again, you want to gauge plant health without getting the signal really washed out or overpowered by exposed soil. bare earth. Okay. So another variation to NDVI is called the normalized difference red edge and the red edge portion of the electromagnetic spectrum as we discussed a bit earlier is the interface of red and near infrared and this band is increasingly used in place of NDVI where complex multi-layered canopies are present so your image will not be totally saturated by the uppermost layer of the canopy as it is with NDVI.
Instead your Indicee using the red edge instead of red will provide a little bit more nuance in a complex overstory understory type of landscape and this again always comes back to agriculture where don't necessarily want to gauge plants health based on the uppermost layer of the canopy. There are stems and leaves below that might need tending to and red edge is increasingly the tool to see a little bit more variability in your canopy or even within one plant or plant cluster. Right.
Okay, so stepping away from vegetation health, we can use multispectral indices to gauge things like fire effects. An even or an ever increasing problem as summers are getting hotter and fires are raging for longer and devastating more forest lands. So the normalized Burn ratio has been used to gauge the effects of fire based on how near infrared light behaves with ash on the ground after a fire. And this is a large landscape output, map output using satellite multispectral imagery. This is a useful output for mapping fire extent over a large region such as the Central Valley in California.
But we can actually use indices adapted to drones to map fire severity on a much finer scale. And this is on the scale of, say, property lines, on the scale of the wildland urban interface, as so many of our smaller cities, especially in British Columbia, are sort of edging ever closer to wild forest lands that perhaps have become. dried out and fire oppressed and could burn and provide problems in residential areas.
Right, yeah we certainly saw a lot of that this past summer here in BC so anything we can do to help mitigate those disasters is definitely going to be beneficial. Exactly, I mean in the days that follow a fire using the right indices with your drone captured orthomosaic you could quickly identify and isolate the hardest hit areas after a fire event. And you could mitigate for things like flash flooding, because with intense fire, you have sort of a loss of soil biomass.
And then if rains follow that intense fire event, the lack of stability in the ground can cause flooding and a huge degree of erosion. So drone-based solutions with fire severity indices could be of great use all throughout the western part of North America. Absolutely. So we've seen there are many industries where color and multispectral drone imagery are classified to make meaningful map outputs. So if you have an area of land or an asset you're hoping to classify and you want to do so with multispectral or true color imagery, the applications are virtually endless.
If you have a use case you would like to explore, CanDrone can help consult you on the best approach and help you realize your project goals from planning to collection and processing your deliverables. At this point, I'd like to thank everyone for joining us today. If you're interested in multispectral services or you're looking to start collecting data yourself for your own projects or service offerings, get in touch with us today and we would be happy to help. Additionally, if there are any specific topics you'd like to see us cover in our future webinars, be sure to mention them in the chat and we will... base our upcoming content on what you'd like to see.
So for now I think it's a good time to open things up for some Q&A and we can answer any more of those burning questions you've been holding on to. So if you do have any more questions just feel free to pop them in the questions box or you can also enter them into the chat and we'll take some time now and we can get those answered. Okay, so we have another question here.
This person says, what I want to achieve is using an RGB camera and NDVI to create better classifications. Is that something that is possible to do using just an RGB and then pairing that with an NDVI? You definitely need to have your near infrared band.
So an RGB camera is not going to cover it. But if you have a camera like the Ultem, you can render a color photo, true color composite. You can perform image classification using that and then have a coincident multispectral in the sea. and perform land cover classifications using that.
Right. And overlaying those two rasters is going to give you the ability to turn layers on and off and to compare classifications between your true color composite and your NDVI. And so you could do this and have many different image layers, each with their strength for telling different qualities in vegetation, for identifying different objects on the landscape, and sort of use a lot of these different indices in concert. Right, okay, great.
Let's see what else we have for a question here. How many pictures a second can I take with this new model? I assume they're referring to the Altum PT for this one when they say new model.
Recently, Micasense updated the Altum series, and there's now the new version of the Altum PT. I can't remember the exact images per second that it can take, but we do have everyone's contact information here, so this is something we can definitely make sure somebody follows up with and we get that answered for you. What else do we have for questions? Have you had any experience with Gulf Corps surveys and their plant health?
We have. Yeah, we've definitely done a number of work on Gulf Corps surveys. I don't know how much of this we could always share, but it's definitely something that CanDron has experience in, and we'd be happy to consult with you on following best practices for that. So with an RGB image, you can't separate that into individual bands.
I think Ian kind of touched briefly on this earlier. You can definitely separate those individual colors, but what you're going to be missing is that red edge and those near-infrared bands. which are going to be really crucial for you to have to get a lot more quantitative data out of your outputs.
So missing those extra bands that you would get with the multispectral payload, you can still get some work done with it, but you're definitely going to be limited in the amount of applications that you can apply your equipment to. Yeah, and as far as using a color image that just has a single lens and a single image capture, can we split those up into the respective red green and blue and I think if you have your GIS and you can turn layers off you can turn sorry different channels off I want to double check that but the question I think I'm hearing is can we do just create indices using just red green and blue bands Using a camera that doesn't actually take separate images for all those bands, I do want to make sure that I find the truest answer to that question for you, because it is a good one. You don't necessarily want to have to buy a whole different camera to do multi-spectral imaging if you don't need to.
For sure. And then Kevin had a good question here. Are multi-spectral satellite images available?
Yes, satellite is where this all began and Landsat images are available online for free. You can access them fairly easily. It's just a matter of resolution.
So it's going to be very comparable with photogrammetry there as well, right? Where what we see in Google Maps from satellite imagery, you can get a good overall picture of everything. But once you zoom in. an area where you might need more detail, you're going to find yourself to be lacking in that area. Is that right?
Yeah, the one advantage among many that I can talk about with satellite imagery is that it has a short wave infrared band, which not many drone-based sensors have. And so with that short wave infrared band, you can do things with the fire severity gauging that I was talking about near the end. But again, the trade-off is much, much lower resolution.
Right. You don't have to buy anything if you want to do image classification with satellite imagery. You could do this entire process with just a computer and an internet connection. You wouldn't even need a drone and a sensor. So it depends what your project objectives are.
But mapping on the county level, the provincial or state level. It's certainly a big industry, and that's where satellite multispectral comes into the equation. I don't see any more new questions coming in here right now. If anybody does have any more questions, please pop them into the chat or you can always feel free to reach out to us here at CanDron as well if you're not able to get it entered in in time there.
We'll just give a few more seconds just in case there's any last minute questions here. There was one more question that just trickled in here. Someone's asking if we can redirect them to a resource for separating our RGB into bands.
I think that's definitely something we could probably share. Maybe when we upload the webinar video, what we can do is we'll share a link in the description there for some resources for handling some of that content. Absolutely.
Great. I think that about wraps it up there for questions. Thank you again for everyone for joining in today. Again, please feel free to pop some suggestions into the chat for future content that you want to see in the webinars here.
We want to make sure that we're covering stuff that is engaging and captivating for people and that's going to be helpful for you and your workflows. Thank you guys. Have a great afternoon.