Offre de stage

Super-résolution pour le rendu rapide de jeux vidéo

Stage 6 mois Niveau Master 2 ou 3A cursus ingénieur —Ubisoft La Forge

PDF : Télécharger

English version below

Apply here: https://jobs.smartrecruiters.com/Ubisoft2/743999784399849-stagiaire-ingenieur-de-recherche-super-resolution-pour-le-rendu-rapide-de-jeux-video-f-h-nb-?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic

Résumé. L’objet de ce stage est d’étudier et de mettre en œuvre une ou plusieurs méthodes de super-résolution dans le cadre de l’accélération de rendu de jeu vidéo. L’idée est d’économiser de la puissance de calcul en effectuant le rendu à basse résolution puis d’utiliser un procédé de super-résolution pour afficher l’image finale.

Localisation et encadrement. La.le stagiaire intégrera le centre de recherche Ubisoft La Forge France, situé à Bordeaux sous ma responsabilité.

Contexte et détails

Avec l’augmentation des standards, les jeux vidéo requièrent la combinaison d’une qualité d’image élevée avec une haute fréquence d’images. Les standards actuels sont une image ultra haute-définition 4K (4096×2160) avec une fréquence de 60 images par secondes (FPS). Passer d’une résolution HD (1920×1080) à 30 FPS à une image 4K à 60 FPS nécessite de multiplier la puissance de calcul par 8. Or, l’augmentation de la puissance du matériel commence à plafonner et les utilisateurs ne possèdent pas tous du matériel de dernière génération. Dans ce contexte, des solutions fondées sur la super-résolution ont été proposées pour afficher à moindre coût des images ultra HD générée à partir d’images de plus faible résolution :

  1. La technologie FSR proposée par AMR, qui est une méthode de super-résolution image par image utilisant des outils de traitement du signal dit classiques ;
  2. La technologie DLSS proposée par Nvidia, qui est une méthode de super-résolution par apprentissage profond utilisant la donnée de plusieurs frames et d’une donnée de mouvement.

Ces deux méthodes permettent d’augmenter les performances jusqu’à 200% pour leur mode performance (cf. documentations officielles). Cependant, les résultats visuels sont difficiles à évaluer et peuvent varier en fonction des données d’entrée.

D’autre part, le problème général de la super-résolution est un sujet largement étudié dans la littérature scientifique. Cependant, les approches scientifiques ne répondent pas toujours aux problématiques propres au rendu de jeu vidéo (temps réel, coût mémoire, etc.).

Ce stage vise à explorer des pistes de recherche pour innover dans ce domaine et améliorer la qualité graphique de nos jeux. En partant des deux approches (DLSS et FSR) proposées par l’industrie et de la littérature scientifique sur le sujet, nous essayerons de proposer des pistes d’amélioration.


Super-sampling for fast rendering of video games

Abstract. The goal of this intership is to study and implement super-sampling methods for high resolution rendering of video games.

Supervision. The intern will be supervised by aresearch engineer from Ubisoft La Forge located in Bordeaux.

Subject

Nowadays, video games require both high image quality and high frame rate. Today’s standards are ultra high-definition 4K (4096×2160) with a frame rate of 60 FPS. Going from HD resolution (1920×1080) at 30 FPS to 4K at 60 FPS requires at least 8 times more ressources. In another hand, the Moore’s law is not true anymore and many users don’t have latest hardware. In this context, solutions based on super-resolution have been proposed to display at lower cost ultra HD images generated from lower resolution images:

  1. The FSR technology from AMR which is a super-sampling method using signal processing tools ;
  2. The DLSS technology from Nvidia which is a super-sampling method using deep learning.

Both methods increase performances up to 200%. However, visual results can be hard to evaluate and can be very different depending on the game.

On the other hand, super-sampling is a hot topic from the scientific literature, although these approaches are not always suited for real-time or memory constraints that arise from the video game rendering.

This internship aim at exploring the literature in order to bring inovation for enhancing game rendering with super-sampling.

Pytorch implementation of HDMI denoising method

I just released a pytorch implementation of the HDMI denoising method that can be used on GPU: github.com/ahoudard/HDMI

This version allows to run the EM algorithm and the denoising in about half a minute!

python run_HDMI_denoising.py alley.png --stdv 0.1 --n_iter 40 --verbose --gpu
selected device: cuda
run EM algorithm...
denoise patches...
total runtime: 34s
PSNR: 29.8940dB

WGenPaTex code

With the publication in the SSVM 2021 proceeding of our paper Wasserstein Generative Models for Patch-based Texture Synthesis we just released the corresponding Python code available on github !

github.com/ahoudard/wgenpatex

Example on my local laptop (core i5 CPU) :

python run_optim_synthesis.py radishes.png --save
cpu
iteration 0 - elapsed 2s - loss = 3.3926462531089783
iteration 50 - elapsed 131s - loss = 0.5145381316542625
iteration 100 - elapsed 260s - loss = 0.23772023245692253
iteration 150 - elapsed 390s - loss = 0.20723320543766022
iteration 200 - elapsed 522s - loss = 0.20013415440917015
iteration 250 - elapsed 652s - loss = 0.19069230556488037
iteration 300 - elapsed 781s - loss = 0.18442947790026665
iteration 350 - elapsed 912s - loss = 0.18047476187348366
iteration 400 - elapsed 1042s - loss = 0.17874254286289215
iteration 450 - elapsed 1162s - loss = 0.17782559990882874
DONE - total time is 1292s

On the Existence of Optimal Transport Gradient for Learning Generative Models

New preprint available! [HAL] [PDF][ArXiv]

Abstract : The use of optimal transport cost for learning generative models has become popular with Wasserstein Generative Adversarial Networks (WGAN). Training of WGAN relies on a theoretical background: the calculation of the gradient of the optimal transport cost with respect to the generative model parameters. We first demonstrate that such gradient may not be defined, which can result in numerical instabilities during gradient-based optimization. We address this issue by stating a valid differentiation theorem in the case of entropic regularized transport and specify conditions under which existence is ensured. By exploiting the discrete nature of empirical data, we formulate the gradient in a semi-discrete setting and propose an algorithm for the optimization of the generative model parameters. Finally, we illustrate numerically the advantage of the proposed framework.

Learning local regularization for variational image restoration

New preprint available! [HAL] [PDF][ArXiv]

Joint work with Jean Prost, Andrés Almansa and Nicolas Papadakis

Abstract: In this work, we propose a framework to learn a local regularization model for solving general image restoration problems. This regularizer is defined with a fully convolutional neural network that sees the image through a receptive field corresponding to small image patches. The regularizer is then learned as a critic between unpaired distributions of clean and degraded patches using a Wasserstein generative adversarial networks based energy. This yields a regularization function that can be incorporated in any image restoration problem. The efficiency of the framework is finally shown on denoising and deblurring applications.

Semi-discrete OT loss for image generation

Some new application of our recent work Wasserstein Generative Models for Patch-based Texture Synthesis [arXiv] [HAL] that proposed a loss with semi-dual formulation of OT.

  •  Style transfert using semi-dual formulation for minimizing the OT distance between VGG-19 features:

       

   

   

  

  • Textures barycenters using our texture generation algorithm (Alg.1 from this work).

The key idea is to generate a new texture such that its patch distributions at various scales are Wasserstein barycenters of the patch distribution of the two inputs:


input 1                       W-barycenter                    input 2


input 1                       W-barycenter                    input 2

 

Wasserstein Generative Models for Patch-based Texture Synthesis

Wasserstein Generative Models for Patch-based Texture Synthesis [arXiv] [HAL]

NEW! Published at SSVM 2021 [link]

joint work with Arthur Leclaire, Nicolas Papadakis and Julien Rabin

Abstract: In this paper, we propose a framework to train a generative model for texture imagesynthesis from a single example. To do so, we exploit the local representationof images via the space of patches, that is, square sub-images of fixed size (e.g. 4×4). Our main contribution is to consider optimal transport to enforce themultiscale patch distribution of generated images, which leads to two differentformulations. First, a pixel-based optimization method is proposed, relying ondiscrete optimal transport. We show that it is related to a well-known textureoptimization framework based on iterated patch nearest-neighbor projections, whileavoiding some of its shortcomings. Second, in a semi-discrete setting, we exploitthe differential properties of Wasserstein distances to learn a fully convolutionalnetwork for texture generation. Once estimated, this network produces realisticand arbitrarily large texture samples in real time. The two formulations result innon-convex concave problems that can be optimized efficiently with convergenceproperties and improved stability compared to adversarial approaches, withoutrelying on any regularization. By directly dealing with the patch distribution ofsynthesized images, we also overcome limitations of state-of-the art techniques,such as patch aggregation issues that usually lead to low frequency artifacts (e.g. blurring) in traditional patch-based approaches, or statistical inconsistencies (e.g. color or patterns) in learning approaches.