Sumangala Nooji

If you want python code drawing, I leave an example below.

import numpy as np

import matplotlib.pyplot as plt

# Example data: Hits, Misses, False Alarms, and Wrong Hits for each class

# Replace this with your own confusion matrix calculations

class_A = {'Hits': 50, 'Misses': 15, 'False Alarms': 15, 'Wrong Hits': 5}

class_B = {'Hits': 40, 'Misses': 10, 'False Alarms': 10, 'Wrong Hits': 2}

class_C = {'Hits': 45, 'Misses': 10, 'False Alarms': 12, 'Wrong Hits': 3}

# Function to calculate cumulative Hits and False Alarms for the TOC curve

def calculate_toc_curve(hits, false_alarms, total_instances):

cumulative_hits = np.cumsum(hits)

cumulative_false_alarms = np.cumsum(false_alarms)

return cumulative_hits, cumulative_false_alarms

# Function to plot the TOC curve for a given class

def plot_toc_curve(class_data, class_name):

hits = np.array([class_data['Hits']])

false_alarms = np.array([class_data['False Alarms']])

total_instances = hits + false_alarms + class_data['Misses'] + class_data['Wrong Hits']

cumulative_hits, cumulative_false_alarms = calculate_toc_curve(hits, false_alarms, total_instances)

plt.plot(cumulative_false_alarms, cumulative_hits, label=f'{class_name} TOC Curve')

plt.xlabel('Cumulative False Alarms')

plt.ylabel('Cumulative Hits')

plt.title('Total Operating Characteristic (TOC) Curve')

plt.legend()

plt.grid(True)

# Plot TOC curves for each class

plot_toc_curve(class_A, 'Class A')

plot_toc_curve(class_B, 'Class B')

plot_toc_curve(class_C, 'Class C')

plt.show()

You're on the right track with using the confusion matrices at different thresholds and calculating Hits, Misses, False Alarms, and Wrong Hits. Here's a breakdown of how to plot a Total Operating Characteristic (TOC) curve for multi-class LULC classification, along with explanations and considerations:

1. Understanding TOC vs. ROC

• ROC (Receiver Operating Characteristic): Traditionally used for binary classification. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various thresholds. For multi-class problems, you can do a "one-vs-all" ROC for each class, but it doesn't fully capture the multi-class performance in a single plot.
• TOC (Total Operating Characteristic): Designed for multi-class problems. It considers all categories simultaneously, providing a more comprehensive view of performance across different thresholds. It accounts for "Wrong Hits" in addition to Hits, Misses, and False Alarms.

2. Key Concepts and Calculations (for each threshold)

You've already got the confusion matrices. Let's define the terms with respect to a specific class, c, and a specific threshold, t:

• True Positives (TPc,t) or Hits: Samples correctly classified as class c with a predicted probability for class c above threshold t. This is the diagonal element of the confusion matrix for class c at threshold t.
• False Negatives (FNc,t) or Misses: Samples that belong to class c but are incorrectly classified as any other class (or below the threshold t for class c). This is the sum of the row for class c in the confusion matrix excluding the TP value (the diagonal element).
• False Positives (FPc,t) or False Alarms: Samples that do not belong to class c but are incorrectly classified as class c with a probability above threshold t. This is the sum of the column for class c in the confusion matrix excluding the TP value.
• Wrong Hits (WHc,t): This is the crucial addition for multi-class TOC. These are samples that are incorrectly classified as class c, but actually belong to a different class (c'). This is a subset of the False Positives, and calculated with this procedure: For samples predicted as Class C, check samples that are not actually Class C. Crucial Point: The key distinction is that a False Alarm could theoretically be any class other than c. A Wrong Hit is another specific class (not c).
• True Negatives (TNc,t): Samples that do not belong to class c and are correctly classified as not class c (either as another class or with a probability for class c below threshold t). This is everything else in the confusion matrix except the row and column for class c. TN is often less important in TOC than the other measures.
• Total number of Pixel: The total number of pixel is the sum of TP, FP, FN and TN.

3. TOC Curve Construction

The TOC curve isn't a simple 2D plot like ROC. It's a series of points in a multi-dimensional space, but we typically visualize it by plotting the area associated with each class at each threshold. Here's how to build it: