share
case study

Blogs Text moderation using..

Text moderation using Google Cloud

5 months ago

8 mins read

The internet has become an integral part of our day-to-day life. It is where we connect with people, share thoughts, and access vast information. However, the internet's open nature offers issues, particularly in terms of maintaining a secure and welcoming online environment. Businesses and communities must come together to keep internet platforms free of dangerous and inappropriate information.

This is the scenario where the importance of text moderation arises. Tech companies are turning to cloud-based technologies that can automatically analyze and filter language to maintain their platforms secure and in compliance with community norms. There are tools and services from Google Cloud to do that.

Let’s discuss text moderation steps with the Python client library (google-cloud-language).

Categories of moderation

Thinking of moderation categories comes with the usual items like, ‘violent’, ‘sexual’, etc. However, Google Cloud’s API categorizes about 16 moderation classes. The predictions are powered by Google Cloud’s next-generation language model PaLM 2. The model has improved multilingual, reasoning, and coding capabilities.

 

 

The following is a comprehensive list of the categories returned by the moderate_text method:

  • Toxic: Rude, impolite, or unreasonable content.
  • Insult: Insulting, provocative, or unfavourable remark directed towards a person or group of people.
  • Public Safety: Services and organizations that give aid and protect the public.
  • War & Conflict: War, military conflicts, and large-scale physical battles involving huge populations. Discusses military services, even if they are not directly tied to a war or battle.
  • Derogatory: Comments that are negative or harmful to identity and/or protected traits.
  • Profanity: Illicit language such as cursing
  • Health: Medical conditions, diseases, and problems Medical therapy, medicine, vaccines, medical practices, and therapeutic resources, such as support groups.
  • Finance: Consumer and business financial services.
  • Violent: Describes scenarios portraying violence against a person or group, as well as basic gore descriptions.
  • Death, Harm & Tragedy: Human fatalities, calamities, accidents, disasters, and self-harm.
  • Religion & Belief: Possibility of supernatural laws and creatures. Astrology and the occult are included.
  • Politics: Political news and media; social, governmental, and public policy discussion
  • Sexual: Content related to sexual acts.
  • Firearms & Weapons: Information mentioning knives, firearms, personal weapons, and accessories like ammo, holsters, and so on.
  • Illicit Drugs: Illicit and recreational drugs
  • Legal: Content with law-related information.

Text moderation

  • An example function is provided below that uses moderate_text to associate a confidence score that ranges from 0.0 to 1.0 with the safety attributes.
from google.cloud import language

def moderate_text(txt: str) -> language.ModerateTextResponse:
    model= language.LanguageServiceClient()
    document = language.Document(
        content=txt,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return model.moderate_text(document=document)

A sample response of the program can be viewed as

{
  "moderationCategories": [
    {
      "name": "Toxic",
      "confidence": 0.10
    },
    {
      "name": "Insult",
      "confidence": 0.12
    },
    {
      "name": "Profanity",
      "confidence": 0.07
    },
    {
      "name": "Derogatory",
      "confidence": 0.04
    },
    {
      "name": "Sexual",
      "confidence": 0.00
    },
    {
      "name": "Death, Harm & Tragedy",
      "confidence": 0.00
    },
    {
      "name": "Violent",
      "confidence": 0.00
    },
    {
      "name": "Firearms & Weapons",
      "confidence": 0.00
    },
    {
      "name": "Public Safety",
      "confidence": 0.01
    },
    {
      "name": "Health",
      "confidence": 0.01
    },
    {
      "name": "Religion & Belief",
      "confidence": 0.00
    },
    {
      "name": "Illicit Drugs",
      "confidence": 0.01
    },
    {
      "name": "War & Conflict",
      "confidence": 0.02
    },
    {
      "name": "Politics",
      "confidence": 0.01
    },
    {
      "name": "Finance",
      "confidence": 0.00
    },
    {
      "name": "Legal",
      "confidence": 0.00
    }
  ]
}

We can also simply send a POST request to the documents:moderateText REST method with the appropriate parameters. An cURL sample of the request is given below:

curl -X POST \
    -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
    -H "Content-Type: application/json; charset=utf-8" \
    --data "{
  'document':{
    'type':'PLAIN_TEXT',
    'content':'Shut up!'
  }
}" "https://language.googleapis.com/v1/documents:moderateText"

The challenges of text moderation may evolve further, but with Google Cloud's innovative solutions, we can confidently navigate the ever-changing landscape of online content, ensuring that the internet remains a place where people can connect, share, and engage without fear or hesitation.

Let's Connect: Reach Out to Us for Expert Guidance and
Collaborative Opportunities

We're just a click away! Contact us today to embark on a journey of digital transformation and unlock new possibilities for your business