Vernon married a lithe, respectable woman named Petunia, and they had robust baby boy called Dudley, and for one glorious moment it seemed like all was well.
But Petunia's rotten sister Lily had fallen into the wrong crowd when they were only kids, and Lily's loser husband only worsened her situation.
The Potters were drug addicts, and they ended up getting killed in some feud by some big bad overlord, a real shady fellow. Vernon pictured him as their drug lord, come to collect his dues, but the whole story was vague. No one had ever bothered to give the Dursleys the facts. All he knew was that they'd been on the run from their killer for months, like cowards.
Vernon had no pity for them. If you get mixed up in that kind of filth, you have to know it's going to end badly.

Irresponsibly, they also had had a kid during all that mess, but then that was the Potters summed up, wasn't it? Irresponsible.
The boy, Harry, was dropped on Vernon's door, just a baby in a blanket, already scarred. The Potters had never wanted anything to do with the Dursleys when they were off their heads high- thought they were better than the common decent folk who worked for what they got- but when things fell apart, it was the Dursleys who took the boy in. None of Potters' druggie buddies were clamouring to mind the kid now, were they? Too much responsibility for scatter-brained scum like that.

The Potter boy was born with a weakness, a predisposition to the drugs. Vernon and Petunia did absolutely everything they could to quash the weakness in him- to wean him off the habit he'd probably learnt at his mother's tit- and shelter their own son from the madness.
Those drugs ruined lives - just ask the Potters- and Harry was drawing it into their house, exposing Dudley to those dangers. If they treated Harry harshly, then it was for the boy's own good; anything, absolutely anything that would stamp the addiction out of him.

Vernon and Petunia failed. They failed Harry. At the age of eleven- eleven, still only an impressionable child- he got mixed back up in his parents' crowd. After that, it was all over. First a terrifying man attacked Dudley, then Harry attacked Vernon's sister, sabotaged Vernon's own chance of promotion, had his lot damage the Dursley's house and attack Dudley again, then Harry dragged Dudley's near-comatose body into the house and swore he hadn't hurt him, and then finally Harry, a grown man at this stage, forced them to abandon their home and go into hiding because of the feud he was caught up in; he pleaded that he was afraid that his enemies would go after them.
After all of the damage he'd caused, they never denied him refuge in their home, and now he was forcing them to leave that same home. Selfish, that's what he was, endangering them like that. Irresponsible.

Vernon never saw the Potter boy again after that last encounter. He and Petunia reared Harry and he left them with a threat over their heads as thanks. They sold the house and started again, and like his parents before him, Harry's name was never spoken, just another failure that was best forgotten.