Well, I'm exaggerating a little, but this is one of those Facebook features that shows an algorithm still can't really figure out the complexity of human relationships.
To a human it's quite obvious that you really don't want to see photos of your ex's smiling face popping up on your Facebook homepage two weeks after you've broken up, but to the Facebook software, well it's not.
Clearly the Photo Memories pictures are ones that have been selected, like the News Feed, because you used to show an interest in them. Your friends can grasp things have changed, but the algorithms can't, unfortunately.
Problems have come up before with Facebook's little widgets - especially around the Reconnect With feature which has suggested people reconnect with dead friends, with exs or perhaps most worryingly, with their current partners.
This modern life eh.
[illustration from XKCD, via @Super_James , thank you]