As a native English speaker from America. I do not interpret this use of "silly" as insulting.
In this context I interpret it more as teasing. Almost as calling the thought of switching factions being foolish.
Now that sounded bad. lol More like calling you a big tease.
If this conversation was meant sincerely. It seems like a very gentlemanly thing to do.
The end comment merely meant friendly while letting you know that he is not interested in changing factions nor considers his faction the wrong one.
It's so easy to misinterpret a word from another language.
*Note to self "Don't use the word silly with non-native speakers. :)"
---
The Clan