When the house I grew up in, was being finished in the early 1960's, someone lost the keys to the front door. We always came and went through the kitchen door, that was on the side of the house. My parents owned that house about 20 years, before selling/giving it to my sister, and they never had keys for the front door. It was rare, very rare for anyone to enter through the front door, the car insurance salesman. The police the night my middle brother totaled a car a couple of miles up the road (and walked away, got a ticket for leaving the scene of an accident.)
We never really thought much about it. There was a back and forth with the builder and the people who did the final finish, but when that yielded no key, nothing was ever done.
The house was locked when we were away, we had keys to the kitchen door, but a copy of that key was hidden in a vent pipe in the garage wall. Friends knew where the key was. That is just way it was. The garage was seldom locked.
Was there a door to your childhood home that no one had a key to? Did you come and go through the front door?