MonetGAN: Translating Real-World Photos into Monet-Style Paintings using CycleGAN

Objective

This project aimed to generate Monet-style images from real-world photographs using a CycleGAN model. The final deliverable was a set of 7,000–10,000 translated images suitable for submission to the Kaggle “GANs Getting Started” competition, evaluated using the MiFID (Memorization-informed Fréchet Inception Distance) metric.

Data

Downloaded on kaggle here: https://www.kaggle.com/competitions/gan-getting-started/data

Source Domain: 7,028 natural photos (photo_jpg/)

Target Domain: 300 Monet paintings (monet_jpg/)

All images were 256x256 in resolution, formatted in JPEG.

Model Architecture

We implemented a CycleGAN using TensorFlow/Keras, consisting of:

Two Generators (G: Photo → Monet, F: Monet → Photo)

Two Discriminators (D_Y: Monet-real vs Monet-fake, D_X: Photo-real vs Photo-fake)

Generators used residual blocks and transposed convolutions for image translation. Discriminators were patch-based CNNs (PatchGAN) for localized realism evaluation.

Loss Functions

Adversarial Loss (for realism): Ensures generated images resemble real Monet paintings.

Cycle Consistency Loss: Encourages that an image translated to the other domain and back returns to the original.

Identity Loss: Helps preserve color and content fidelity when applicable.

Training

Trained on paired batches from both domains for 25 epochs.

Periodic image generation allowed qualitative monitoring of training progress.

Used tf.data pipelines with caching, shuffling, and parallel loading for efficiency.

Optimization was performed using the Adam optimizer with learning rate decay.

Evaluation

Visual inspection confirmed the model learned artistic texture transfer, especially brushstroke simulation and color palette adaptation.

The translated images retained general structure but with Monet-esque textures and hues.

A set of 10,000 Monet-style images was generated using the trained generator and exported for submission.

Conclusion

The CycleGAN successfully performed unpaired image-to-image translation from photos to Monet-style paintings. Despite a small dataset of Monet paintings, the model learned to generate visually coherent and stylistically consistent images suitable for artistic domain adaptation, demonstrating the power of adversarial learning in creative AI applications.

data preprocessing and file loading

import tensorflow as tf
import os
import glob

IMG_HEIGHT = 256
IMG_WIDTH = 256

def load_image(filename):
    image = tf.io.read_file(filename)
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.resize(image, [IMG_HEIGHT, IMG_WIDTH])
    image = (image / 127.5) - 1  # Normalize to [-1, 1]
    return image

def image_dataset(file_paths):
    dataset = tf.data.Dataset.from_tensor_slices(file_paths)
    dataset = dataset.map(load_image, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.cache().shuffle(1024).batch(1)

monet_paths = glob.glob('monet_jpg/*.jpg')
photo_paths = glob.glob('photo_jpg/*.jpg')

monet_ds = image_dataset(monet_paths)
photo_ds = image_dataset(photo_paths)

defining generator and discriminator models

from tensorflow.keras import layers
from tensorflow.keras.models import Model

def downsample(filters, size, apply_batchnorm=True):
    initializer = tf.random_normal_initializer(0., 0.02)
    result = tf.keras.Sequential()
    result.add(layers.Conv2D(filters, size, strides=2, padding='same',
                             kernel_initializer=initializer, use_bias=False))
    if apply_batchnorm:
        result.add(layers.BatchNormalization())
    result.add(layers.LeakyReLU())
    return result

def upsample(filters, size, apply_dropout=False):
    initializer = tf.random_normal_initializer(0., 0.02)
    result = tf.keras.Sequential()
    result.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same',
                                      kernel_initializer=initializer, use_bias=False))
    result.add(layers.BatchNormalization())
    if apply_dropout:
        result.add(layers.Dropout(0.5))
    result.add(layers.ReLU())
    return result

def Generator():
    inputs = layers.Input(shape=[256, 256, 3])

    down_stack = [
        downsample(64, 4, apply_batchnorm=False),
        downsample(128, 4),
        downsample(256, 4),
        downsample(512, 4),
        downsample(512, 4),
        downsample(512, 4),
    ]

    up_stack = [
        upsample(512, 4, apply_dropout=True),
        upsample(512, 4, apply_dropout=True),
        upsample(256, 4),
        upsample(128, 4),
        upsample(64, 4),
    ]

    initializer = tf.random_normal_initializer(0., 0.02)
    last = layers.Conv2DTranspose(3, 4, strides=2, padding='same',
                                  kernel_initializer=initializer, activation='tanh')

    x = inputs
    skips = []
    for down in down_stack:
        x = down(x)
        skips.append(x)
    skips = reversed(skips[:-1])

    for up, skip in zip(up_stack, skips):
        x = up(x)
        x = layers.Concatenate()([x, skip])

    x = last(x)
    return Model(inputs=inputs, outputs=x)

def build_discriminator():
    initializer = tf.random_normal_initializer(0., 0.02)
    inp = layers.Input(shape=[256, 256, 3], name='input_image')
    x = downsample(64, 4, False)(inp)
    x = downsample(128, 4)(x)
    x = downsample(256, 4)(x)
    x = layers.ZeroPadding2D()(x)
    x = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(x)
    x = layers.BatchNormalization()(x)
    x = layers.LeakyReLU()(x)
    x = layers.ZeroPadding2D()(x)
    x = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(x)
    return Model(inputs=inp, outputs=x)

instantiate models

generator_g = Generator()  # Monet-style (Photo → Monet)
generator_f = Generator()  # Reverse (Monet → Photo)

discriminator_x = build_discriminator()  # Monet discriminator
discriminator_y = build_discriminator()  # Photo discriminator

define loss functions and optimizers

loss_obj = tf.keras.losses.BinaryCrossentropy(from_logits=True)

def discriminator_loss(real, generated):
    real_loss = loss_obj(tf.ones_like(real), real)
    generated_loss = loss_obj(tf.zeros_like(generated), generated)
    return (real_loss + generated_loss) * 0.5

def generator_loss(generated):
    return loss_obj(tf.ones_like(generated), generated)

LAMBDA = 10

def cycle_loss(real_image, cycled_image):
    return LAMBDA * tf.reduce_mean(tf.abs(real_image - cycled_image))

def identity_loss(real_image, same_image):
    return LAMBDA * 0.5 * tf.reduce_mean(tf.abs(real_image - same_image))

g_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)
f_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)
dx_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)
dy_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)

training step

@tf.function
def train_step(real_x, real_y):
    with tf.GradientTape(persistent=True) as tape:
        fake_y = generator_g(real_x, training=True)
        cycled_x = generator_f(fake_y, training=True)

        fake_x = generator_f(real_y, training=True)
        cycled_y = generator_g(fake_x, training=True)

        same_x = generator_f(real_x, training=True)
        same_y = generator_g(real_y, training=True)

        disc_real_x = discriminator_x(real_x, training=True)
        disc_real_y = discriminator_y(real_y, training=True)

        disc_fake_x = discriminator_x(fake_x, training=True)
        disc_fake_y = discriminator_y(fake_y, training=True)

        gen_g_loss = generator_loss(disc_fake_y)
        gen_f_loss = generator_loss(disc_fake_x)

        total_cycle_loss = cycle_loss(real_x, cycled_x) + cycle_loss(real_y, cycled_y)

        total_gen_g_loss = gen_g_loss + total_cycle_loss + identity_loss(real_y, same_y)
        total_gen_f_loss = gen_f_loss + total_cycle_loss + identity_loss(real_x, same_x)

        disc_x_loss = discriminator_loss(disc_real_x, disc_fake_x)
        disc_y_loss = discriminator_loss(disc_real_y, disc_fake_y)

    g_gradients = tape.gradient(total_gen_g_loss, generator_g.trainable_variables)
    f_gradients = tape.gradient(total_gen_f_loss, generator_f.trainable_variables)

    dx_gradients = tape.gradient(disc_x_loss, discriminator_x.trainable_variables)
    dy_gradients = tape.gradient(disc_y_loss, discriminator_y.trainable_variables)

    g_optimizer.apply_gradients(zip(g_gradients, generator_g.trainable_variables))
    f_optimizer.apply_gradients(zip(f_gradients, generator_f.trainable_variables))

    dx_optimizer.apply_gradients(zip(dx_gradients, discriminator_x.trainable_variables))
    dy_optimizer.apply_gradients(zip(dy_gradients, discriminator_y.trainable_variables))

train the cycleGAN over multiple epochs

import time
import matplotlib.pyplot as plt

EPOCHS = 25  # adjust as needed

def generate_images(model, test_input):
    prediction = model(test_input, training=True)
    plt.figure(figsize=(12, 6))

    display_list = [test_input[0], prediction[0]]
    title = ['Input Image', 'Translated Image']

    for i in range(2):
        plt.subplot(1, 2, i+1)
        plt.title(title[i])
        # Rescale to [0,1] for display
        plt.imshow((display_list[i] * 0.5 + 0.5))
        plt.axis('off')
    plt.show()

sample_photo = next(iter(photo_ds))

for epoch in range(EPOCHS):
    start = time.time()

    for real_x, real_y in tf.data.Dataset.zip((photo_ds, monet_ds)):
        train_step(real_x, real_y)

    print(f'Time taken for epoch {epoch+1} is {time.time()-start:.2f} sec')

    # Show sample translated image every 5 epochs
    if (epoch + 1) % 5 == 0:
        generate_images(generator_g, sample_photo)

Time taken for epoch 1 is 466.60 sec
Time taken for epoch 2 is 454.50 sec
Time taken for epoch 3 is 454.38 sec
Time taken for epoch 4 is 469.10 sec
Time taken for epoch 5 is 464.59 sec

Time taken for epoch 6 is 463.76 sec
Time taken for epoch 7 is 468.58 sec
Time taken for epoch 8 is 463.48 sec
Time taken for epoch 9 is 481.19 sec
Time taken for epoch 10 is 553.29 sec

Time taken for epoch 11 is 577.82 sec
Time taken for epoch 12 is 584.46 sec
Time taken for epoch 13 is 589.33 sec
Time taken for epoch 14 is 640.25 sec
Time taken for epoch 15 is 636.20 sec

Time taken for epoch 16 is 689.75 sec
Time taken for epoch 17 is 701.06 sec
Time taken for epoch 18 is 701.70 sec
Time taken for epoch 19 is 693.82 sec
Time taken for epoch 20 is 689.02 sec

Time taken for epoch 21 is 679.28 sec
Time taken for epoch 22 is 709.25 sec
Time taken for epoch 23 is 724.06 sec
Time taken for epoch 24 is 716.77 sec
Time taken for epoch 25 is 764.50 sec

Create images

import os
from PIL import Image
from tqdm import tqdm
import zipfile
import numpy as np 

# Output directory for generated Monet-style images
output_dir = "generated_images"
os.makedirs(output_dir, exist_ok=True)

photo_dir= 'photo_jpg'
# Reload photo dataset without batching to save individual images
photo_paths = tf.io.gfile.glob(str(photo_dir + '/*.jpg'))
photo_ds = tf.data.Dataset.from_tensor_slices(photo_paths)
photo_ds = photo_ds.map(load_image, num_parallel_calls=tf.data.AUTOTUNE).batch(1)

# Generate and save images
for i, img in enumerate(tqdm(photo_ds)):
    monet_img = generator_g(img, training=False)[0]
    monet_img = (monet_img * 127.5 + 127.5).numpy().astype(np.uint8)  # Denormalize

    img_pil = Image.fromarray(monet_img)
    img_pil.save(os.path.join(output_dir, f"monet_{i:04d}.jpg"))

    if i >= 9999:  # Limit to 10,000 images
        break

100%|██████████| 7038/7038 [11:51<00:00,  9.89it/s]

# Create images.zip for submission
zip_path = "images.zip"
with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
    for fname in os.listdir(output_dir):
        zipf.write(os.path.join(output_dir, fname), arcname=fname)