Liwaiwai Liwaiwai
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
    • Architecture
    • Design
    • Software
    • Hybrid Cloud
    • Data
  • About
Liwaiwai Liwaiwai
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
    • Architecture
    • Design
    • Software
    • Hybrid Cloud
    • Data
  • About
  • Machine Learning

Using Integrated ML To Deliver Low-latency Mobile VR Graphics

  • March 10, 2020
  • relay

What it is:

A new low-latency, power-efficient framework for running machine learning in the rendering pipeline for standalone VR devices that use mobile chipsets. This architecture makes it possible to use ML on these devices to significantly improve image quality and video rendering.

We have created a sample application under this framework to reconstruct higher-resolution rendering (known as super-resolution) to improve VR graphics fidelity on a mobile chipset with minimal compute resources. This new framework can also be used to perform compression artifact removal for streaming content, frame prediction, feature analysis, and feedback for guided foveated rendering.

How it works:

In a typical mobile VR rendering system, the application engine retrieves movement-tracking data at the beginning of each frame and uses this information to generate images for each eye. To work effectively for VR applications, processing time for the whole graphics pipeline is typically constrained tightly, with, for example, a budget of 11 milliseconds rendering time for both eye buffers in order to achieve a 90 Hz refresh rate.

To overcome these constraints, our new architecture offloads model execution so it is asynchronized on specialized processors. In this design, the digital signal processor (DSP) or neural processing unit (NPU) is pipelined with the graphics processing unit (GPU) and takes either portions of the rendered buffers or the entire rendered buffers for further processing. The processed content is picked up asynchronously by a GPU warping thread for latency compensation before sending to display.

This graphic shows how we parallelize machine learning model execution on DSP with other processors in the graphics display pipeline.

To improve performance, we modify the graphics memory allocation system in the OS to use the specialized allocator for the GPU-DSP shared memory. This is more efficient than direct mapping, because the graphics framebuffer is often optimized for GPU-only access (and performs poorly on CPU) and because a special memory registration process is needed to avoid copying with remote calls at runtime.

Read More  Financial Services Companies Are Starting To Use The Cloud For Big Data And AI Processing

We tested this pipeline with a sample application that applies deep learning to improve image quality in the central region, but uses more efficient, lower-resolution rendering for other parts of the scene. The super-resolved content is blended with the surrounding regions in asynchronized timewarp. If we render at around 70 percent lower resolution in each direction, we save approximately 40 percent of GPU time, and developers can use those resources to generate better content. To achieve temporally coherent and visually pleasing results in VR, we developed recurrent networks trained with a specially designed temporal loss function.

Why it matters

Creating next-gen VR and AR experiences will require finding new, more efficient ways to render high-quality, low-latency graphics. Traditional rendering and super-resolution techniques may not be acceptable on low-persistence displays used in VR headsets, since temporal artifacts are more perceptible. This method provides a new way to use AI to address this challenge on devices powered by mobile chipsets.

In addition to its AR/VR applications, we believe this new framework can open the door for innovations in mobile computational graphics, by removing constraints on memory and enabling other new innovations in image quality enhancement, artifacts removal, and frame extrapolations.

Read the full paper

https://dl.acm.org/doi/10.1145/3355088.3365154

 

Behnam Bastani, Haomiao Jiang, Rohit Rao Padebettu, Kazuki Sakamoto

Source: Facebook AI

relay

Related Topics
  • Facebook AI
  • Gaming
  • Virtual Reality
You May Also Like
View Post
  • Artificial Intelligence
  • Data
  • Data Science
  • Machine Learning
  • Technology

Google Data Cloud & AI Summit : In Less Than 12 Hours From Now

  • March 29, 2023
View Post
  • Artificial Intelligence
  • Machine Learning
  • Technology

ChatGPT 4.0 Finally Gets A Joke

  • March 27, 2023
View Post
  • Artificial Intelligence
  • Machine Learning
  • Technology

Mr. Cooper Is Improving The Home-buyer Experience With AI And ML

  • March 24, 2023
View Post
  • Artificial Intelligence
  • Machine Learning
  • Technology

GPT-4 : The Latest Milestone From OpenAI

  • March 24, 2023
View Post
  • Engineering
  • Machine Learning

Peacock: Tackling ML Challenges By Accelerating Skills

  • March 23, 2023
View Post
  • Data
  • Machine Learning
  • Platforms

Coop Reduces Food Waste By Forecasting With Google’s AI And Data Cloud

  • March 23, 2023
View Post
  • Artificial Intelligence
  • Machine Learning
  • Robotics

Gods In The Machine? The Rise Of Artificial Intelligence May Result In New Religions

  • March 23, 2023
View Post
  • Artificial Intelligence
  • Machine Learning

6 ways Google AI Is Helping You Sleep Better

  • March 21, 2023

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay Connected!
LATEST
  • 1
    Unlocking The Secrets Of ChatGPT: Tips And Tricks For Optimizing Your AI Prompts
    • March 29, 2023
  • 2
    Try Bard And Share Your Feedback
    • March 29, 2023
  • 3
    Google Data Cloud & AI Summit : In Less Than 12 Hours From Now
    • March 29, 2023
  • 4
    Talking Cars: The Role Of Conversational AI In Shaping The Future Of Automobiles
    • March 28, 2023
  • 5
    Document AI Introduces Powerful New Custom Document Classifier To Automate Document Processing
    • March 28, 2023
  • 6
    How AI Can Improve Digital Security
    • March 27, 2023
  • 7
    ChatGPT 4.0 Finally Gets A Joke
    • March 27, 2023
  • 8
    Mr. Cooper Is Improving The Home-buyer Experience With AI And ML
    • March 24, 2023
  • 9
    My First Pull Request At Age 14
    • March 24, 2023
  • 10
    The 5 Podcasts To Check If You Want To Get Up To Speed On AI
    • March 24, 2023

about
About
Hello World!

We are liwaiwai.com. Created by programmers for programmers.

Our site aims to provide materials, guides, programming how-tos, and resources relating to artificial intelligence, machine learning and the likes.

We would like to hear from you.

If you have any questions, enquiries or would like to sponsor content, kindly reach out to us at:

[email protected]

Live long & prosper!
Most Popular
  • 1
    GPT-4 : The Latest Milestone From OpenAI
    • March 24, 2023
  • 2
    Ditching Google: The 3 Search Engines That Use AI To Give Results That Are Meaningful
    • March 23, 2023
  • 3
    Peacock: Tackling ML Challenges By Accelerating Skills
    • March 23, 2023
  • 4
    Coop Reduces Food Waste By Forecasting With Google’s AI And Data Cloud
    • March 23, 2023
  • 5
    Gods In The Machine? The Rise Of Artificial Intelligence May Result In New Religions
    • March 23, 2023
  • /
  • Artificial Intelligence
  • Machine Learning
  • Robotics
  • Engineering
  • About

Input your search keywords and press Enter.