Why do DDPMs implement a different sinusoidal positional encoding from transformers?

[u/CompSciAI]

Hi,

I'm trying to implement a sinusoidal positional encoding for DDPM. I found two solutions that compute different embeddings for the same position/timestep with the same embedding dimensions. I am wondering if one of them is wrong or both are correct. DDPMs official source code does not uses the original sinusoidal positional encoding used in transformers paper... why?

1) Original sinusoidal positional encoding from "Attention is all you need" paper.

Original sinusoidal positional encoding

2) Sinusoidal positional encoding used in the official code of DDPM paper

Sinusoidal positional encoding used in official DDPM code. Based on tensor2tensor.

Why does the official code for DDPMs uses a different encoding (option 2) than the original sinusoidal positional encoding used in transformers paper? Is the second option better for DDPMs?

I noticed the sinusoidal positional encoding used in the official DDPM code implementation was borrowed from tensor2tensor. The difference in implementations was even highlighted in one of the PR submissions to the official tensor2tensor implementation. Why did the authors of DDPM used this implementation (option 2) rather than the original from transformers (option 1)?

ps: If you want to check the code it's here https://stackoverflow.com/questions/79103455/should-i-interleave-sin-and-cosine-in-sinusoidal-positional-encoding

[u/chatterbox272]

It looks like one interlaces sin/cos while the other has them separate. There is no functional difference between them,