I put this question to Uncle Junkal's player, who, in addition to playing a Bard in-game, is quite the mythology and folklore buff in real life. He replied:
I don't have a straight myth studies answer for this nor do I think one is available. From my experiences with various permutations of what I call postmodern mythology, I can comment, though I don't mean to be definitive. Pomo myth comes in a spectrum of fidelity to its source material. Sometimes an author re-interprets with great fidelity a mythological phenomenon in a new time period or universe. This is complicated as well as made more interesting by the fact that myths themselves already exist in a variety of permutations. Other authors use their awareness of a myth as a starting point and feel free to make any changes as fits their purposes and whims. This spectrum even occurs in the same piece of work sometimes, so in a large universe like D&D I would expect it to be present in all of its forms.
After doing some research, I'm sticking to my initial hunch that the Otyugh is heavily influenced by the Star Wars garbage monster. The Otyugh first appeared closely after Star Wars came out, so it's hard to nail down a definitely causal chronology, but the resemblance is striking and designs for the garbage monster existed well in advance of Gygax's publication:
I feel like Gygax was operating in a mode of some other monster creators in the pomo myth spectrum and creating a world in which every monster he's ever found interesting is included if possible regardless of if it came from myth or legend.
I would say the garbage monster and the Otyugh are descended from a few monsters of myth and legend. The first is the Kraken. Tolkien's inclusion of a Kraken-like Watcher in the Water in The Fellowship of the Ring in particular resembles the mode of the Otyugh and the Star Wars monster. Another that comes to mind is the Hydra for its swamp residing tendencies. The important differences are that monsters in classical/Greek myth are usually godlike/imposing or descended from gods. They are too significant to do things like eat garbage. The one abject classical monster that comes the closest is the Harpy who did tend to be disgusting as well as eat as a scavenger, though was often wasteful, while the Otyugh seems quite eco-friendly.
I think the Star Wars connection, while not demonstrably causal, is spot-on in terms of historical coincidence. Star Wars was released on May 25, 1977, and as my friend suggests, the design of the trash compactor creature was complete long before the film's actual release date. The Monster Manual, in which Gygax's Otyugh first appears, was published no earlier than September 27, 1977, the date given at the end of Mike Carr's Foreword to the book. So unless evidence exists that proves Gygax invented the Otyugh much earlier than its preparation for publication in the MM, it is indeed possible that Lucas influenced Gygax in this -- or at least that they were simultaneously creating creatures along the same general lines.
The trash compactor monster Uncle Junkal's player refers to is called a Dianoga and is described on Wookieepedia, the Star Wars wiki, as follows:
Dianoga were large cephalopods, capable of growing up to ten meters in length, though most specimens grew to approximately five or six meters. They had a single eyestalk protruding from their mollusk-like bodies, and seven suckered tentacles surrounding a fanged maw.
Yet, as Uncle Junkal's player points out, both Gygax's and Lucas' creatures share DNA with the mythological Kraken as well as the tentacle-creature encountered outside Moria gate in Tolkien's Fellowship of the Ring, first published in 1954.
Thinking that some data on the etymology of the Otyugh's name might help my friend track down other possible historical connections, I checked its entry on wikipedia and found that it has another name as well: Gulguthra. This is the name for the general group of creatures to which Otyughs and Neo-Otyughs belong, and is the name used in Ed Greenwood's Dragon Magazine article on the ecology of the creature (in Dragon #96). I asked Uncle Junkal's player if the name "Gulguthra" rang any ancient-mythological bells, and he answered:
I came across Gulguthra too in my research. In Google if you put Otyugh with a word that may be related to its origin, like "star wars" you can come up with some pretty extensive discussions on the matter. After doing a bit more research, I feel like Gulguthra is equally an off the cuff naming. I believe the phonetic combination may be a random permutation of the biblical Golgotha perhaps combined with Mothra (though not necessarily consciously or purposefully on either count), and the words gullet, gut -- which would be relevant to the monster.
Nicely said! My friend's concluding general remarks on the naming of the Otyugh (and other D&D creatures) were characteristically thought-provoking, so I'll give him the last word:
An interesting but likely uber-fanboy question is how Gygax named the Otyugh. This is where the spectrum of pomo myth comes into play again. Etymologies are rendered very fanboy knowledge especially when a creator is in an off the cuff mode like Lucas with Mon Calamari and even some of his less random inclusions of yiddish further down the spectrum. I suspect Gygax may have been operating in this off the cuff mode when he came up with the name. Conversely, when a creator is putting a lot of effort into the myth they are homaging, name etymologies begin to fall more closely to a scholarly and not fanboy pursuit -- though the binary between fanboy and scholar is mostly a false one.
There may be an essay in there about the nature of low art creators in the days when the low art and high art divide was clear and distinct. Would Lucas for example have named something Calamari, if he knew how "high art" his work would eventually be regarded by the mainstream? When Alan Moore or Neil Gaiman put lots of thought into how naming characters and monsters will signify, is it because of how they regard themselves and their work as important in ways Gygax didn't see his work?