Liwaiwai Liwaiwai
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
    • Architecture
    • Design
    • Software
    • Hybrid Cloud
    • Data
  • About
Liwaiwai Liwaiwai
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
    • Architecture
    • Design
    • Software
    • Hybrid Cloud
    • Data
  • About
  • Artificial Intelligence
  • Data

Building AI That Can Generate Images Of Things It Has Never Seen Before

  • September 21, 2021
  • relay

Generative adversarial networks (GANs) are a well-established AI method to create images, whether photorealistic pictures or abstract collages. However, to date these models have had an important limitation: They can typically only generate images of objects or scenes that are closely related to the training data set.

A traditional GAN trained on images of cars shows impressive results when asked to generate other images of cars, for example, but will likely fail if asked to generate images of flowers or other objects outside of its automotive data set.

Facebook AI has made great strides in solving this problem with Instance-Conditioned GAN (IC-GAN), a new and simple image generation model that creates high-quality, diverse images — even if its input image doesn’t appear in the training set. Unlike previous methods, IC-GANs can generate realistic, unforeseen image combinations, such as camels surrounded by snow or zebras in a city. Our approach exhibits exceptional transfer capabilities across different types of objects. Researchers can use IC-GANs off the shelf with previously unseen data sets and still generate realistic-looking images, without requiring labeled data.

With these new capabilities, IC-GANs could be used to create new visual examples to augment data sets to include diverse objects and scenes; help artists and creators with more expansive, creative AI-generated content; and advance research in high-quality image generation.

Training and using the IC-GAN

Standard methods, called class conditional GANs, focus on conditioning on class labels, effectively partitioning the data into groups corresponding to those labels. This enables them to generate higher-quality samples than their unconditional counterparts. And rather than creating only random images, these GANs are also able to create images that fit a particular label, such as “clothing” or “car.” However, they rely on labeled data which may be unavailable or unfeasible to obtain.

Read More  A Neural Network That Keeps Seeing Art Where We See Mundane Objects

Previous label-free learning approaches (using no labeled data) to image generation have been promising, but their output is typically of poor quality when trained to model complex data sets, such as ImageNet. They either use coarse, nonoverlapping data partitions (resulting in very large clusters, each of which contains images of very different objects and thus won’t be semantically similar to the picture the model is trying to create). Or they use fine partitions that tend to deteriorate results because the clusters contain too few data points.

Our new approach, the IC-GAN, can be used with both labeled and unlabeled data sets. It extends the GAN framework to model a mixture of local and overlapping data clusters. It can take a single image (or “instance”) and then generate images that are similar to the instance’s closest neighbors in the data set. We use neighbors as an input to the discriminator, to force the generator to create samples that are similar to the neighborhood samples of each instance. This avoids the problem of partitioning data into small clusters, since so much of the data is overlapping, so the model can use data sets more efficiently.

Once the model is trained, we then test it on images it has never seen before. Using a single image, the model can generate visually rich images that are similar to the closest neighbors in the data set.

The IC-GAN used the instance on the left to generate the corresponding images shown on the right. In this case, no class label was provided.

For both class-conditional settings (where the training set includes labeled images) and also where there are no labels at all, the IC-GAN can be transferred to other data sets not seen during training. In the case of IC-GAN, we do this by swapping out the conditioning instances at inference time. In the case of a class-conditional IC-GAN, we can swap either the instance conditioning or the class label. By appropriately combining instances and the class labels, the class-conditional IC-GAN can create unusual scenes that either aren’t present or are very rare in current data sets. For example, given an image of a snowplow surrounded by snow and a class label “camel,” which doesn’t appear in the instance conditioning, we can generate camels surrounded by snow, bypassing the bias that camels live only in the desert.

Read More  Harmful Content Can Evolve Quickly. Facebook's New AI System Adapts To Tackle It.
When conditioned on the image shown on the left along with the class label, the IC-GAN generated the images shown on the right.

Applying IC-GANs to future work

IC-GAN can be used to augment data and include items or objects that are not commonly found in the training data. Moreover, since it works across different domains, our approach can generate more diverse training data for object recognition models. Traditional GAN models, for instance, would not be able to generate images of zebras standing in urban areas, since its training data would likely only contain images of zebras in grasslands. The IC-GAN model can be used to augment data and include items or objects that are not commonly found in standard data sets. We’ve shown that we can use controlled semantics to generate unusual image combinations, like cows in the sand, for example.

In the future, we hope to explore ways to bring even more control to this model. It will no longer be just about the background and the object at the center. We want to explore how more objects can be placed in the background and determine where the items are placed, creating complex, picture-perfect scenes.

The IC-GAN was able to use the graphic of a house shown here to create the more realistic-looking buildings shown below.

By releasing our pretrained models into the open source community, along with code to reproduce the results from the paper, we hope this research will lead to AI models that generate images with more flexibility, accuracy, and efficiency than ever before.

Source Facebook AI Research.

relay

Related Topics
  • Facebook AI
  • GANs
  • IC-GAN
  • Images
You May Also Like
View Post
  • Data
  • Platforms
  • Technology

How Osmo Is Digitizing Smell With Google Cloud AI Technology

  • March 20, 2023
View Post
  • Data
  • Engineering
  • Tools

Built With BigQuery: How Sift Delivers Fraud Detection Workflow Backtesting At Scale

  • March 20, 2023
View Post
  • Data

Understand And Trust Data With Dataplex Data Lineage

  • March 17, 2023
View Post
  • Artificial Intelligence
  • Technology

Limits To Computing: A Computer Scientist Explains Why Even In The Age Of AI, Some Problems Are Just Too Difficult

  • March 17, 2023
View Post
  • Big Data
  • Data

The Benefits And Core Processes Of Data Wrangling

  • March 17, 2023
View Post
  • Artificial Intelligence
  • Machine Learning
  • Platforms
  • Technology

Using ML To Predict The Weather And Climate Risk

  • March 16, 2023
View Post
  • Artificial Intelligence
  • Platforms
  • Technology

Google Is A Leader In The 2023 Gartner® Magic Quadrant™ For Enterprise Conversational AI Platforms

  • March 16, 2023
View Post
  • Artificial Intelligence
  • Technology

The Future Of AI Is Promising Yet Turbulent

  • March 16, 2023

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay Connected!
LATEST
  • 1
    How Osmo Is Digitizing Smell With Google Cloud AI Technology
    • March 20, 2023
  • 2
    Built With BigQuery: How Sift Delivers Fraud Detection Workflow Backtesting At Scale
    • March 20, 2023
  • 3
    Building The Most Open And Innovative AI Ecosystem
    • March 20, 2023
  • 4
    Understand And Trust Data With Dataplex Data Lineage
    • March 17, 2023
  • 5
    Limits To Computing: A Computer Scientist Explains Why Even In The Age Of AI, Some Problems Are Just Too Difficult
    • March 17, 2023
  • 6
    The Benefits And Core Processes Of Data Wrangling
    • March 17, 2023
  • 7
    We Cannot Even Agree On Dates…
    • March 17, 2023
  • 8
    Financial Crisis: It’s A Game & We’re All Being Played
    • March 17, 2023
  • 9
    Using ML To Predict The Weather And Climate Risk
    • March 16, 2023
  • 10
    Google Is A Leader In The 2023 Gartner® Magic Quadrant™ For Enterprise Conversational AI Platforms
    • March 16, 2023

about
About
Hello World!

We are liwaiwai.com. Created by programmers for programmers.

Our site aims to provide materials, guides, programming how-tos, and resources relating to artificial intelligence, machine learning and the likes.

We would like to hear from you.

If you have any questions, enquiries or would like to sponsor content, kindly reach out to us at:

[email protected]

Live long & prosper!
Most Popular
  • 1
    The Future Of AI Is Promising Yet Turbulent
    • March 16, 2023
  • 2
    ChatGPT: How To Prevent It Becoming A Nightmare For Professional Writers
    • March 16, 2023
  • 3
    Midjourney Selects Google Cloud To Power AI-Generated Creative Platform
    • March 8, 2023
  • 4
    A Guide To Managing Your Agile Engineering Team
    • March 15, 2023
  • 5
    10 Ways Wikimedia Does Developer Advocacy
    • March 15, 2023
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
  • About

Input your search keywords and press Enter.