I actually wouldn't have a problem with this at all. Women's handball had huge problems with exactly this issue between 1996 and 2004, when the competition was limited to 10 teams (or only 8, like in Atlanta!) and one of those teams were the hosts USA, Australia and Greece, all of which are VERY far from the top level and got thrashed by everyone. Honestly, it made the entire tournament look like a bit of a joke. IMO, the hosts should show they are able to compete on an at least a somewhat acceptable level if they're going to get a direct entry.
That’s one of the perks of being the host. You can’t expect the host nation to be good at every sport.