I have no answers.
If I did, I'd be a lot more successful than I am. But just asking the questions can lead us to greater insight. (Or at least, that's what I tell myself.)
But back up a little bit: exactly what am I asking here?
See, the book I'm reading has a protagonist who is a cold-blooded murderer, turned professional assassin. One who shows no sign of remorse so far (though is certainly headed for a "redemptive" arc) and similarly no sign of personality other than a dedication to duty, to death. And I just can't get on board.
I've always differed from some commentators in that I thought you didn't need to work so hard to win over an audience. I figured that the very fact that you have chosen to invest in this book means that you will by default side with the protagonist.
This book is changing my viewpoint.
So what is the key here? Must we rely on cliched "pet the dog" moments to make our heroes and heroines approachable? I hope not.
My favorite novel is Vladimir Nabokov's "Lolita". Humbert Humbert is a pedophile and a child-molester, and yet we feel enough "sympathy" to go along with him as our lead (even as we despise everything he does and stands for). Why?
Probably just because of the beauty and poetry of Nabokov's prose, attributed in-world to narrator Humbert Humbert. We are entranced by his way with words, his wit, his playfulness. We enjoy the experience of reading his point of view, without having to enjoy him as a person at all.
Of course, the book also starts with something of a pre-apology from Humbert as he narrates the childhood experience that "trapped" him into liking young girls. Yet this is not the thing that binds us to him, as it is always clear that he is manipulating the reader, presenting this as an explanation for his actions (even while claiming contrition).
No, this "explaining away" of his nature is not what draws in the audience; we are not fooled by Humbert Humbert's tactics. So apart from the sheer delight of reading English written so well, why do we enjoy the experience of seeing the world through his eyes?
No idea. Sorry.
Similarly, Van Veen in the same author's "Ada, or Ardor: A Family Chronicle" is a hateful slimeball, but the sheer cleverness of the prose makes me delight in the book even while despising the protagonist.
So it's not about relatability. Not exactly. Not about having us feel as though we are necessarily like the character. But we do (perhaps) need to feel as though we understand the character's experience.
Change media here: let's look at film.
For the first 20-odd minutes of Alfred Hitchcock's Psycho, we are watching Janet Leigh's character Marion act as lead. Then, when she is killed (ostensibly by motel owner Mrs Bates) the lead role is taken by the killer['s son?] Norman, as he disposes of the body and tries to evade certain parties interested in the murder.
Why on Earth would we switch our allegiance to this handsome but odd-and-slightly-creepy character who is getting rid of our heroine's corpse?
Norman Bates hides the body in his car, then pushes it into the swamp. As it starts to sink, it catches--for just a moment, it looks as though the vehicle will not submerge. We are tense; our hearts catch in our throats.
Then: relief. It sinks. Phew!
Just that simple and relatable moment ("Oh no! The car won't sink!") transfers our sympathy to this strange young man instead. We instantly understand this tension and relate to it--feel it ourselves. Which immediately puts us on his side, whatever else we may think of the man.
So what does this mean for our writing? To be honest, I'm not particularly sure.
What I do know is that making sure my main character is all sunshine and rainbows and dog-petting is unnecessary. Giving him/her a situation to which the reader can instantly relate may just give the audience all the opportunity it needs to latch onto the protagonist after all.