Project 4K05

Discussion in 'Revenge of the Sith' started by DarthPyro52, May 30, 2019.

  1. DarthPyro52

    DarthPyro52 Jedi Knight

    For the past 2 months I've been experimenting with AI to upscale Revenge of the Sith. The results have been, surprisingly, pretty good. Not perfect, but still good. I've done a few different tests using built-in DNR, but tend to prefer keeping them unaltered, or everything starts to look like silly putty.

    Below are a few tests I did last month (without DNR). I've attempted to upscale an entire scene using this method, but it tends to not only take a ton of time, but also a ton of disk space. The first duel between Obi-Wan and Grievous, for example, took about a week to finish, and took up about 200+GB of space, not including the final After Effects render, since each frame needed to be upscaled separately.

    https://ibb.co/dPcX6jk
    https://ibb.co/Jq5Xjz9
    https://ibb.co/Mnh57zw
    https://ibb.co/ymjjqL5
    https://ibb.co/6m6LCbX
     
  2. InitAbsolute

    InitAbsolute Padawan

    I'm curious: what algorithm are you using to upscale these images?
     
  3. DarthPyro52

    DarthPyro52 Jedi Knight

    For now I just use a program called Topaz Gigapixel AI, which takes care of all the specifics automatically.
     
    Last edited: Jun 2, 2019
    thomas947 and sade1212 like this.
  4. Moonstrider

    Moonstrider Jedi Knight

    My concern with using topaz as I have looked into it myself is that its AI adds what it perceives to detail to any image it upscales resulting in a "faux" image, it may be usable as an overlay but it will most certainly create some artifacts that would need to be removed but none the less you have my attention and I am very interested as this is the first sw film I saw in theaters, I wouldn't call it a bad film just misexecuted. I considered using it for my BATB wip restoration and Sword in the stone restoration of which I would do a lot of overlay and clean up
     
  5. InitAbsolute

    InitAbsolute Padawan

    Any AI-based algorithm will add its own perceived detail to the image which can improve the quality of the image. However, there can be too much "perceptual" detail added to an image, in which case artifacts are introduced to the image and it stops looking natural. Part of the difficulty behind developing a good AI algorithm is finding the right balance: too much "perceptual detail" will create unwanted artifacts; too little will make the images perceived as blurry and take very little advantage of the increased resolution.
     
    DarthPyro52 likes this.
  6. DarthPyro52

    DarthPyro52 Jedi Knight

    That's why it's an experiment. As I said, it's not perfect, but it's a good start. Any AI you use will have a similar effect. I did notice however that a lot of the "perceived detail" can be avoided slightly by not touching any of the noise removal settings. Having to sort of mix the upscaling with the original footage to remove any added detail makes it sort of a patchwork job, but it's not a bad place to begin. Unfortunately using AI will be the only way a 4K version of AOTC or ROTS will ever be possible, given the circumstances in which they were filmed. The AI will only get better with time, perhaps one day yielding a near-perfect result. We are now only experiencing the infancy of this technology.
     
    thomas947 and sade1212 like this.
  7. zxthehedgehog

    zxthehedgehog Jedi Master

    This might not be a bad use of it - Gigapixel really shines when it comes to upscaling CGI, so I'd really like to see a video sample.
     
  8. DarthPyro52

    DarthPyro52 Jedi Knight

    Unfortunately my no-DNR render had some issues, but I'm uploading my first DNR attempt, which unfortunately I had to compress quite a bit in order to upload. Should be done soon. I haven't had time to test this one with a 4K monitor yet, but I assume it turned out okay.
     
    thomas947 likes this.
  9. LuketheNerd

    LuketheNerd Jedi Master

    I'm 100% sure that within the next 5 years upscaling will be able to create incredible detail in images from machine learning. If you give a neural-network a million 8k videos of trees, and train it for a year, it will be able to upscale 100x100 videos of trees like it's nobody's business.

    The potential of upscaling is so incredible. With the computing power that is being rolled out in the near future, we're going to be getting a lot more indispensable plug-ins and programs.
     
    SkyDude, sade1212 and InitAbsolute like this.
  10. InitAbsolute

    InitAbsolute Padawan

    I've been frequently going through research papers on this topic for the past year or so, and the progress that's been made in machine learning-based super-resolution (SR) even a year ago is truly unbelievable. The constant amount of activity around the research behind these algorithms are incredibly active; in fact roughly half of the computer science papers released on the open access repository arXiv are machine learning papers. And there's a new video/image enhancement technique published what almost seems as frequently as a week, sometimes even more than that at once (even as frequent as twice a day in some instances).

    And one thing about image super-resolution: It doesn't appear that Gigapixel is capable of upscaling videos itself, but please correct me if I'm wrong. Applying image-based super-resolution to videos isn't a bad way to go for upscaling (definitely a big leap over even edge-based interpolation), however because of the system's inability to analyze the motion in the image sequence, you may have some temporal inconsistencies (detail inconsistencies between individual frames) in the final results. (This might not be that big of a problem for a x2 upscale, but x4 and higher it may be apparent.) There are indeed AI algorithms in the research labs that are designed to take advantage of neighboring frames in a video.

    Thus, if you decide to look for a new algorithm to toy with, I suggest you look for an algorithm that is specifically designed for video. Right now I'm toying around with a video SR algorithm called TecoGAN, and the software itself is definitely research code and not very friendly for end users; however I can try and package it up into a more friendly program if you'd like. I still have yet to test the code on some material and see its results though.
     
  11. DarthPyro52

    DarthPyro52 Jedi Knight

    Interesting, I haven't heard of that one before. Being able to upscale a full video as opposed to upscaling the individual frames and compiling them seems like it would be a lot simpler... and much less time-consuming. I'd love to take a look at a more "user friendly" version!
     
    thomas947 and sade1212 like this.
  12. LuketheNerd

    LuketheNerd Jedi Master

    Upscaling a video by upscaling independent frames is like trying to form a friendship with a different person every day and wondering why the friendship doesn't evolve.
     
    sade1212 likes this.
  13. DarthPyro52

    DarthPyro52 Jedi Knight

    Well, it works. It takes a really, really, really long time, but it works.

    My video sample won't upload to Google Drive for some reason. Had it going all night and it's stuck at 1%. I'll have to find another site to upload it.
     
    thomas947, sade1212 and Roobyoo like this.
  14. Darth Robin

    Darth Robin Padawan

    Why would you need DNR for a film that was shot on digital anyway?
     
  15. InitAbsolute

    InitAbsolute Padawan

    Digital cameras' electronics aren't perfect, and the produced videos can still create noise too. Digital noise is different than film grain, however.
     
    sade1212 and DarthPyro52 like this.
  16. DarthPyro52

    DarthPyro52 Jedi Knight

    ^What he said. ROTS has a somewhat surprising amount of noise when upscaled, especially the live-action elements.

    Here's one example. This is not upscaled; this is a frame from the official Blu-Ray release:
    https://ibb.co/VjgmXTt
     
    Last edited: May 31, 2019
    thomas947 and sade1212 like this.
  17. theMaestro

    theMaestro Jedi Knight

    I could be wrong, but I heard that they actually overlayed a grain plate for ROTS.
     
    sade1212 and camroncamera like this.
  18. DarthPyro52

    DarthPyro52 Jedi Knight

    I wouldn't be surprised if that was done for the CGI. The noise in the filmed shots are at least mostly real.
     
    thomas947 and sade1212 like this.
  19. LuketheNerd

    LuketheNerd Jedi Master

    In my brief but comprehensive research I couldn't find anything that says this, do you remember where you heard that information?
     
  20. theMaestro

    theMaestro Jedi Knight

    I remember someone mentioning it on the TFN forums. But I just did my own search and have thus far not found anything official, so it's probably just a theory that some fans have. Though if I do find something official, I'll post it here.
     
    LuketheNerd likes this.

Share This Page