A while back, I mentioned in passing that some chemical structures were ugly ones from a med-chem perspective. That prompted a reader to ask, very appropriately, what makes a structure ugly. The quick answer is easy, and a bit embarassing: an ugly drug structure is one that looks too different from other things that we already know are drugs.
Hah! There's the "Rule of Five" in a nutshell for you. But there's something to that approach, although I don't approve of it being used religiously. If a compound deviates wildly from the kinds of structures that we already know work, you do run some risks. Those deviations can be in sheer size (molecular weight), where we know from bitter experience that many synthetic compounds up in the high hundreds of daltons seem to have trouble being absorbed and transported around the body. Not all of them - but more and more as you go up.
Or you can deviate in things like the number of polar functional groups. Way too few of them, we also know, and your compound is so greasy that it also can have absorption problems, and the liver will tend to rip it to pieces even if it gets that far. Way too many polar groups, though, and the compounds seem to have problems getting through cell membranes. Again, these aren't written in stone, but your chances of success decrease the further away that you get and your research efforts should be planned with those odds in mind.
Beyond these parameters, there are specific chemical reasons to dislike a structure. If it has strongly reactive functional groups in it, there had better be a good reason for them. After all, there are a lot of things in the body to react with, almost all of which you'd want to avoid. Some chemotherapy agents fall into this category, which goes a long way toward explaining their low toxicity threshold. To give you an idea, a structure with an acid chloride in it will be dismissed out of hand, and an epoxide will be regarded as guilty until proven innocent.
Reactive groups can also include things that are chemically stable, but will fall apart under physiological conditions. Very acid-labile groups, for example, are probably not going to make it through the hydrochloric acid bath of the stomach (unless you coat the tablet to make it through into the intestine). And something like a simple methyl ester probably isn't going to survive, either, because there's an awful lot of acylase/hydrolase enzyme activity floating around in the intestines and in the blood. (You can use these effects to unmask your active drug once it's in the body - a "prodrug" - but that's another story).
And, finally, we reserve the right to call a structure ugly if it looks just plain too hard to make. After all, we generally have to make hundreds (sometimes thousands) of related analogs during the early development of a lead structure, and if the shortest route into the series is fifteen steps long, that's just not going to happen. The odds of success go down as the difficulty of the chemistry goes up, so all things considered, we'd rather work on something that we can deal with. There are plenty of compounds that are reasonable by all other criteria except this one, I'm sad to say, which means that we pay attention any time an interesting new synthetic method comes along. It might lead us into chemical spaces that no one's had a chance to explore (or claim!)