He was such an escape artist that I should have named him Houdini. He did get out a few times and I worried because I was also in an area not safe for him to roam. Fortunately, he came back unharmed, but I understand your concern.
The only thing I can suggest is eternal vigilance on your part and some precautions. To prevent him from slipping out when I opened the door to leave or on returning, I put him in a closed off room with toys, a litter pan, food and water if I was going to be gone long.
For short "trips" like picking up the mail, or even just answering the door when someone knocked, I put him in the bathroom or bedroom and closed the door. The problem with that, of course, is that he learned to avoid being picked up when he heard the door knocks or noticed me heading for the door. So, I only opened the door a crack and kept a foot in the doorway to block him. No, I did not kick him, but I got pretty good at nudging him away and maneuvering one foot while talking to whoever was at the door.
If it is any consolation, he did mellow on the escape efforts once he was around 6 years old.