Panoptic Segmentation with Convex Object Representation (2024)

Article Navigation

Volume 67 Issue 6 June 2024
  • Next >

Journal Article

Get access

,

Zhicheng Yao

Institute of Computing Technology, Chinese Academy of Sciences

,

No. 6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190

,

China

University of Chinese Academy of Sciences

,

No. 1 Yanqihu East Rd, Huairou District, Beijing 101408

,

China

Search for other works by this author on:

Oxford Academic

,

Sa Wang

Institute of Computing Technology, Chinese Academy of Sciences

,

No. 6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190

,

China

University of Chinese Academy of Sciences

,

No. 1 Yanqihu East Rd, Huairou District, Beijing 101408

,

China

Corresponding author: wangsa@ict.ac.cn

Search for other works by this author on:

Oxford Academic

,

Jinbin Zhu

Institute of Computing Technology, Chinese Academy of Sciences

,

No. 6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190

,

China

Search for other works by this author on:

Yungang Bao

Institute of Computing Technology, Chinese Academy of Sciences

,

No. 6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190

,

China

University of Chinese Academy of Sciences

,

No. 1 Yanqihu East Rd, Huairou District, Beijing 101408

,

China

Search for other works by this author on:

Oxford Academic

The Computer Journal, Volume 67, Issue 6, June 2024, Pages 2009–2019, https://doi.org/10.1093/comjnl/bxad119

Published:

20 December 2023

Article history

Received:

06 March 2023

Revision received:

22 November 2023

Accepted:

27 November 2023

Published:

20 December 2023

Search

Close

Search

Advanced Search

Search Menu

Abstract

The accurate representation of objects holds pivotal significance in the realm of panoptic segmentation. Presently, prevalent object representation methodologies, including box-based, keypoint-based and query-based techniques, encounter a challenge known as the ‘representation confusion’ issue in specific scenarios, often resulting in the mislabeling of instances. In response, this paper introduces Convex Object Representation (COR), a straightforward yet highly effective approach to address this problem. COR leverages a CNN-based Euclidean Distance Transform to convert the target instance into a convex heatmap. Simultaneously, it offers a parallel embedding method for encoding the object. Subsequently, COR characterizes objects based on the distinctive embedding vectors of their convex vertices. This paper seamlessly integrates COR into a state-of-the-art query-based panoptic segmentation framework. Experimental findings validate that COR successfully mitigates the representation confusion predicament, enhancing segmentation accuracy. The COR-augmented methods exhibit notable improvements of +1.3 and +0.7 points in PQ on the Cityscapes validation and MS COCO panoptic 2017 validation datasets, respectively.

© The British Computer Society 2023. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/pages/standard-publication-reuse-rights)

You do not currently have access to this article.

Download all slides

Sign in

Get help with access

Personal account

  • Sign in with email/username & password
  • Get email alerts
  • Save searches
  • Purchase content
  • Activate your purchase/trial code
  • Add your ORCID iD

Sign in Register

Institutional access

    Sign in through your institution

    Sign in through your institution

  1. Sign in with a library card
  2. Sign in with username/password
  3. Recommend to your librarian

Institutional account management

Sign in as administrator

Get help with access

Institutional access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Sign in through your institution

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

  1. Click Sign in through your institution.
  2. Select your institution from the list provided, which will take you to your institution's website to sign in.
  3. When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
  4. Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Sign in with a library card

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

  1. Click Sign in through society site.
  2. When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.
  3. Following successful sign in, you will be returned to Oxford Academic.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

Personal account

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

  • View your signed in personal account and access account management features.
  • View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

Institutional account management

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Purchase

Subscription prices and ordering for this journal

Purchasing options for books and journals across Oxford Academic

Short-term Access

To purchase short-term access, please sign in to your personal account above.

Don't already have a personal account? Register

Panoptic Segmentation with Convex Object Representation - 24 Hours access

EUR €15.00

GBP £13.00

USD $16.00

Rental

Panoptic Segmentation with Convex Object Representation (3)

This article is also available for rental through DeepDyve.

Advertisem*nt

Citations

Views

69

Altmetric

More metrics information

Metrics

Total Views 69

44 Pageviews

25 PDF Downloads

Since 12/1/2023

Month: Total Views:
December 2023 9
January 2024 26
February 2024 2
March 2024 2
April 2024 7
June 2024 18
July 2024 5

Citations

Powered by Dimensions

Altmetrics

×

Email alerts

Article activity alert

Advance article alerts

New issue alert

Receive exclusive offers and updates from Oxford Academic

Citing articles via

Google Scholar

  • Latest

  • Most Read

  • Most Cited

COVID-19 virus mutation prediction with LSTM and attention mechanisms
Security analysis of the MAP-T IPv6 transition technology
TPRT: a trajectory publishing scheme for the Internet of Vehicles based on radix tree
IPOD2: an irrecoverable and verifiable deletion scheme for outsourced data
An improved DNN model for WLAN intrusion detection

More from Oxford Academic

Computer Science

Science and Mathematics

Books

Journals

Advertisem*nt

Panoptic Segmentation with Convex Object Representation (2024)

FAQs

What is an example of panoptic segmentation? ›

In panoptic segmentation, an instance can either represent a distinct thing or a region of stuff. Things are countable objects such as pedestrians, animals, or cars, while stuff represents uncountable amorphous regions such as the sky or grass.

What is the difference between instance segmentation and panoptic segmentation? ›

While semantic segmentation classifies each pixel into a category, instance segmentation identifies individual object instances. On the other hand, panoptic segmentation does both: it classifies every pixel and assigns a unique instance ID to distinguishable objects.

What is the panoptic segmentation stuff? ›

Stuff: Stuff in panoptic image segmentation refers to amorphous or uncountable regions in an image, such as sky, road, grass, walls, etc. These regions do not have well-defined boundaries and are typically treated as a single continuous segment without individual instances.

What is the output format of panoptic segmentation? ›

The task format we adopt for panoptic segmentation is simple: each pixel of an image must be assigned a semantic label and an instance id. Pixels with the same label and id belong to the same object; for stuff labels the instance id is ignored. See Figure 1d for a visualization.

Why do we need panoptic segmentation? ›

Surveillance and security: Panoptic segmentation is used in video surveillance systems to identify and track objects of interest within packed scenes, improving security and threat detection.

What is an example of panopticism in our daily life today? ›

A few examples will illustrate how widespread panoptical observation has become. Cameras placed in highly visible locations, or simply signs saying that there are cameras, serve to deter criminals. Presumably, the cameras and signs have a deterrent effect even if the cameras are not operative.

Which dataset for panoptic segmentation? ›

Benchmarks Add a Result
DatasetBest Model
COCO minivalOneFormer (InternImage-H, emb_dim=1024, single-scale)
ADE20K valOneFormer (InternImage-H, emb_dim=256, single-scale, 896x896)
Mapillary valOneFormer (DiNAT-L, single-scale)
Cityscapes testOneFormer (ConvNeXt-L, single-scale, Mapillary Vistas-Pretrained)
21 more rows

Is segmentation better than object detection? ›

Segmentation vs Detection: When to Choose Each

Both approaches have their strengths and specific use cases, making it important to understand when to choose each one. Segmentation is the preferred choice when the task requires a comprehensive understanding of object boundaries and extracting fine-grained information.

What is the best instance segmentation model? ›

  • Segment Anything Model (SAM) Segment Anything (SAM) is an image segmentation model developed by Meta Research, capable of doing zero-shot segmentation. ...
  • YOLOv8 Instance Segmentation. ...
  • YOLOv5 Instance Segmentation. ...
  • Mask RCNN. ...
  • Grounded SAM. ...
  • YOLOv7 Instance Segmentation. ...
  • FastSAM. ...
  • YOLACT.

How does panoptic work? ›

Bentham argued in The "Panopticon" that the perfect prison would be structured in a such a way that cells would be open to a central tower. In the model, individuals in the cells do not interact with each other and are constantly confronted by the panoptic tower (pan=all; optic=seeing).

What is video panoptic segmentation? ›

Video Panoptic Segmentation is a computer vision task that extends panoptic segmentation by incorporating temporal dimension. That is, given a video sequence, the goal is to predict the semantic class of each pixel while consistently tracking object instances.

What are the three types of segmentation in image processing? ›

Broadly speaking, image segmentation is used for three types of tasks: semantic segmentation, instance segmentation and panoptic segmentation. The difference between each type of image segmentation task lies in how they treat semantic classes: the specific categories a given pixel might be determined to belong to.

What is the difference between panoptic and instance segmentation? ›

Panoptic segmentation unifies the typically disjoint tasks of semantic segmentation (identifying and classifying objects in an image) and instance segmentation (segmenting individual instances of each object), offering a more holistic and precise tooth and oral tissue segmentation strategy [13, 14] .

What is the formula for panoptic quality? ›

Panoptic Quality (PQ)

We use the standard PQ Kirillov et al., which is defined as ∑{1(p,g) IoU(p,g)} / (|TP|+ 0.5|FP|+ 0.5|FN|). The set of true positives (TP), false positives (FP), and false negatives (FN), are computed by matching prediction p to ground-truth g based on the IoU scores.

What is the difference between semantic and instance segmentation? ›

Semantic segmentation classifies pixels based on their semantic meaning, treating all objects within a category as one entity. Instance segmentation, on the other hand, distinguishes between different instances of the same class, allowing for more precise object identification and differentiation.

What is a simple example of semantic segmentation? ›

A simple example of semantic segmentation is separating the images into two classes. For example, in Figure 1, an image showing a person at the beach is paired with a version showing the image's pixels segmented into two separate classes: person and background. Figure 1: Image and labeled pixels.

What is an example of segmentation in digital image processing? ›

Groups of image segmentation

Semantic segmentation is an approach detecting, for every pixel, the belonging class. For example, in a figure with many people, all the pixels belonging to persons will have the same class id and the pixels in the background will be classified as background.

References

Top Articles
Latest Posts
Article information

Author: Jamar Nader

Last Updated:

Views: 5855

Rating: 4.4 / 5 (75 voted)

Reviews: 90% of readers found this page helpful

Author information

Name: Jamar Nader

Birthday: 1995-02-28

Address: Apt. 536 6162 Reichel Greens, Port Zackaryside, CT 22682-9804

Phone: +9958384818317

Job: IT Representative

Hobby: Scrapbooking, Hiking, Hunting, Kite flying, Blacksmithing, Video gaming, Foraging

Introduction: My name is Jamar Nader, I am a fine, shiny, colorful, bright, nice, perfect, curious person who loves writing and wants to share my knowledge and understanding with you.