I live in the city, where we have 40' wide lots. According to the 40' diameter rule, it would be "safe" to launch shells, cakes and any aerials in my back yard. Would I? No. Would I feel safe if one of my next door neighbors started shooting aerials in their backyard? No. I have shot fountains and ground items in my back yard and I believe that is safe enough for most S&S and ground items. Though my yard is also surrounded by a thick 10'-15' high hedge as well. I also brace or barricade the fountains when possible.
For aerials, I want to have at least 75' for tubes 1" and under and 70' per inch for tubes larger than 1". I will take more distance if I can get it. This is basically the NFPA 1123 standard.
Whenever I am shooting any fireworks, I use the "family rule" for the spectators. If one of my family members (wife, son, daughter-in-laws, brothers, sisters, etc.) was in the audience, where would I want that family member to be or how far away would I want that family member to be away from the fireworks, if something would go wrong.