A music AI needs to have a utility function which optimizes for beautiful music. So we need a computable function which says what music is beautiful.
The first problem with this is that beauty feels inherently mysterious. When I introspect on “beauty,” it feels like this magical, irreducible thing which couldn’t possibly be explained in logic. This is related to the problem of qualia; since beauty is, after all, a quale.
If materialism is true, then that intuition is wrong, and beauty is not inherently mysterious. If materialism is false, then that intuition may as well be correct. Let us for the moment assume materialism, and see where it gets us.
There is some circuitry in the brain which corresponds to our perception of beauty. There is some analogy to the beauty function in the human brain.
But it is a truism of truisms that “beauty is in the eye of the beholder.” There is no fact of the matter about how beautiful something is. So there cannot be a single correct beauty function. This makes sense. Natural selection created the beauty function, so it is probably slightly different in everybody.
Also, it is probably affected by culture as well as genes. There could be an underlying circuitry for beauty perception, relatively constant between people, which needs to be “fed” by cultural data to develop fully. To make an analogy, there is relatively constant circuitry which says what is grammatical language, but what the circuitry labels as grammatical depends on what languages you know.
In any case, we are interested in writing a function which does something like the brain’s beauty circuitry. There cannot be a single correct beauty function, because the beauty circuitry varies between people. Furthermore, we probably cannot untangle all of the nuances of the brain’s beauty circuitry; so we can only write an approximation which captures some of the most important and salient features.
Now suppose that materialism is false, and beauty is inherently mysterious and irreducible. We can still approximate musical beauty in a function, because beautiful music follows patterns, and those patterns are correlated with how beautiful the music is. But the limitations preventing us from writing a perfect beauty function are more fundamental. We could speculate that in this case there are an infinite series of progressively better beauty functions, which approach beauty without ever reaching it.