{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Masking?\n",
    "\n",
    "In processing EO data, _masking_ is a very common operation to mark certain pixel locations as _null_ values. For more discussion, see the [RasterFrames masking page](https://rasterframes.io//masking.html).\n",
    "\n",
    "## MODIS MCD43A4\n",
    "\n",
    "\n",
    "Let’s demonstrate the masking procedure of [MODIS MCD43A4](https://lpdaac.usgs.gov/products/mcd43a4v006/) data on the red, blue and green bands. Each MCD43A4 band has quality information stored in a separate QA band.\n",
    "\n",
    "The specifications for the QA band are as follows:\n",
    "\n",
    "```\n",
    "Data conversions:\n",
    "       Mandatory QA    0 = processed, good quality (full BRDF inversions)\n",
    "                       1 = processed, see other QA (magnitude BRDF inversions)\n",
    "                       255 = Fill Value\n",
    "```\n",
    "\n",
    "So in order to mask out poor quality pixels, we need to ensure that the QA value for that pixel is 0.\n",
    "\n",
    "Using EOD, let's query some MODIS imagery. We will query the 3 measurement bands along with their QA bands. First, let's use Earth OnDemand to see what image collections are available to us.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "init",
     "option_string": "name = \"init\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "from earthai.init import *\n",
    "\n",
    "from shapely.geometry import Point\n",
    "from pyspark.sql.functions import col, lit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": false,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": null,
     "option_string": "evaluate=False",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "earth_ondemand.collections()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In the above DataFrame note that the collection we want has an id of `mcd43a4`. Let's see which band names correspond to red, green, and blue.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "bands",
     "option_string": "name = \"bands\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "earth_ondemand.item_assets('mcd43a4').sort_values('asset_name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The bands for red, green, and blue are `B01`, `B04`, and `B03` respectively. We also see each band has a QA band named `B0Xqa`.\n",
    "\n",
    "Now that we know the appropriate collection id and band names, let's perform our query. We will create a Shapely point geometry near Charlottesville, Virginia, USA to create an Earth OnDemand catalog. Then we will read the bands we want into a Spark DataFrame named `df`.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "read",
     "option_string": "name = \"read\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "catalog = spark.read.earth_ondemand_catalog(\n",
    "                                               geo=Point(-78.461530, 38.039243),\n",
    "                                               start_datetime='20190901',\n",
    "                                               end_datetime='20190905',\n",
    "                                               collections='mcd43a4',\n",
    "                                               max_cloud_cover=10\n",
    "                                           )\n",
    "band_names = ['B01', 'B04', 'B03',\n",
    "              'B01qa', 'B04qa', 'B03qa']\n",
    "\n",
    "df = spark.read.raster(catalog, catalog_col_names=band_names)\n",
    "\n",
    "#keep only tile columns and rename\n",
    "df = df.select(\n",
    "            col('B01').alias('red'),\n",
    "            col('B01qa').alias('red_qa'),\n",
    "            col('B04').alias('green'),\n",
    "            col('B04qa').alias('green_qa'),\n",
    "            col('B03').alias('blue'),\n",
    "            col('B03qa').alias('blue_qa'),\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "MCD43A4 measurement bands already have some NoData defined as it is the output of a time composited model of surface reflectance. We can inspect this as shown below. The cell type of `int16ud32767` indicates that the value of 32767 will be interpreted as NoData. Having a NoData value defined allows us to do the masking.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "cell_type",
     "option_string": "name = \"cell_type\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "df.select(rf_cell_type('red'), rf_cell_type('green'), rf_cell_type('blue'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Let's now use `rf_mask_by_value` to set all pixel locations where the QA band is 1 to NoData in the measurement band.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "mask",
     "option_string": "name = \"mask\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "masked = df.withColumn('red_masked',   rf_mask_by_value('red',   'red_qa',   1)) \\\n",
    "           .withColumn('green_masked', rf_mask_by_value('green', 'green_qa', 1)) \\\n",
    "           .withColumn('blue_masked',  rf_mask_by_value('blue',  'blue_qa',  1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Inspect tiles with many data pixels masked out using the filter below.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto",
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "options": {
     "caption": false,
     "complete": true,
     "display_data": true,
     "display_stream": true,
     "dpi": 200,
     "echo": true,
     "evaluate": true,
     "f_env": null,
     "f_pos": "htpb",
     "f_size": [
      6,
      4
     ],
     "f_spines": true,
     "fig": true,
     "include": true,
     "name": "inspect",
     "option_string": "name = \"inspect\"",
     "results": "verbatim",
     "term": false,
     "wrap": "output"
    }
   },
   "outputs": [],
   "source": [
    "masked.select('red_qa', 'red', 'red_masked', 'blue_qa', 'blue', 'blue_masked') \\\n",
    "      .filter(rf_no_data_cells('red_masked') > 2500) \\\n",
    "      .filter(rf_no_data_cells('blue_masked') > 2500)"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "echo"
  },
  "kernelspec": {
   "display_name": "EarthAI Environment",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "zendesk": {
   "draft": true,
   "section_id": 360009044691,
   "title": "Masking MODIS MCD43A4 (NBAR) Using RasterFrames"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}