The Official Blog of Alpha Chromatica
The Official Blog of Alpha Chromatica
In Which the Author Explores Machine Learning for Green Screen Keying and Relighting - Part 3

In Which the Author Explores Machine Learning for Green Screen Keying and Relighting - Part 3

Tags
Machine LearningNormal MapsCopycat
Author
Ganz
Date
October 21, 2024
Post No.

Chroma 0324

URL
https://www.linkedin.com/posts/heyganz_chroma-0324-in-which-the-author-explores-activity-7255291979440279552-s_8w?utm_source=share&utm_medium=member_desktop
image

There’s a certain alchemy to this process. You tinker, adjust, and hold your breath, waiting for the magic or catastrophe to surface.

📊 Experiment Overview: The Frame Spacing Challenge

To pinpoint the optimal interval for temporal stability, here’s what I did:

1️⃣ Generated 125 individual frames of normals using ComfyUI as my starting dataset.

2️⃣ Tested three frame intervals as ground truth inputs, using the normals generated by ComfyUI, for Nuke’s Copycat node :

  • Every 4th frame (32 images total)
  • Every 8th frame (17 images total)
  • Every 24th frame (7 images total)

3️⃣ Trained models across 40,000 epochs, letting the model learn every nuance in light shifts, color values, and motion. This is the only setting I changed in the Copycat node.

🎯 The Sweet Spot: Every 8th Frame

The breakthrough came with the 8-frame interval.

It struck the perfect balance between:

1️⃣Detail retention: Preserving high visual fidelity

2️⃣AI generalization: Giving the model room to predict somewhat accurately without overfitting

3️⃣Temporal stability: Smoother transitions between frames, less flickering

This setup worked because 17 frames were just enough for the AI to capture the essence of the shot without overwhelming it with unnecessary data.

💡 Technical Adjustments for Clarity

1️⃣Premultiplied full-frame normals with AI-generated alpha, hence the terrible buzzing on the edges

2️⃣Gamma adjustments were applied to deepen colours for visualization

🔍 Insights

Using more spaced-out frames for simpler shots, which don’t require as much data. It’s a smart optimization that I’ll continue to explore in future tests.

Thanks Nicolas Erba for taking the time to explain this to me.

🚀 What’s Next?

The results are promising, but the journey isn’t over.

Every experiment opens the door to new questions.

While the 8-frame interval worked well here, every shot has its nuances, and what works for one may not for another.

For future experiments, the 8-frame interval will be my starting point, a solid foundation to build on while refining the process further.

Missed the previous posts?

Catch up on Part 1: https://lnkd.in/dMiRKuwN

Catch up on Part 2: https://lnkd.in/dPKcVMNz