Is it rude to speak in Spanish when not everyone understands?

I have been dating a man from Colombia. He's very nice and seems interested in me. However, he sinks back into speaking nothing but Spanish when he's around his friends. His friends all understand and speak English. I felt so left out over dinner while they chattered in Spanish, leaving me out of the conversation -- even when I later learned the topic moved to questions about me! I can understand that it's tempting to speak in your native language, but my friend voluntarily immigrated to the United States where English is the predominant language. Should I bring up being shut out of the conversation and ask him to speak in English?