Identifying Toxic Content

Overview

Protecto is designed to identify and score content based on the presence of toxic language, such as inflammatory remarks or discriminatory statements. This guide provides an understanding of how Protecto scores content and the types of toxic content it identifies.

Scoring System

Protecto assigns a score to content indicating the severity of toxic content:

• Score Range: 0 (not present) → 1 (severe toxic content present)

Customers can utilize these scores to establish thresholds for blocking or issuing warnings about potentially harmful content.

Types of Toxic Content

Protecto classifies toxic content into several categories:

1. Obscene
   • Definition: Use of profanity or sharing sexually suggestive content.
   • Example: Explicit language or graphic sexual references.
2. Threat
   • Definition: Directly or indirectly threatening violence or harm.
   • Example: Statements implying intent to physically harm someone.
3. Insult
   • Definition: Use of derogatory language or making personal attacks.
   • Example: Name-calling, slurs, or any language intended to demean an individual.

Example Analysis

Below is an example of how Protecto analyzes and scores a piece of content:

"value": "You're really dumb, aren't you?",
"token_value": "You're really dumb, aren't you?",
"toxicity_analysis": {
    "toxicity": 0.9648129,
    "severe_toxicity": 0.011925396,
    "obscene": 0.39630863,
    "threat": 0.0010725626,
    "insult": 0.9019991,
}

Toxicity Score Examples

The table below provides examples of content with their corresponding toxicity scores and categorizations: