From that experience of living in those days with people her age, her fans, and in that culture, the exact translation would be, “Everybody is down on me,” or “Everybody is upset with me.” As

an adult “Really? My response to them would have made her feel down (sick)?

Had she done something while drunk or drugged, that she was now ashamed of, and was being punished by others (through shunning, curt remarks, cancelled appointments, angry fan mail), she would feel ‘down’, and would say ‘they are down on me’, bringing her own feelings of ‘down’ into their feelings about her. It was an expression created by those expressing feelings while high.