So it seems your entire connection gets lost. Something like this actually happened to me before, except it was with a router. I can't really suggest anything to fix it, but you can probably reinstate your internet connection without restarting the computer. What operating system are you using?

I think you'll need to contact your ISP about this one, unless someone else who has DSL knows how to fix this problem or at least has some suggestions.

EDIT: I just noticed that you said this happens when people join, or not. Does this happen when people don't come? Because I know DSL kind of works like a regular modem in some cases (though faster), so your ISP might have some sort of idle disconnect of some sort. But then again if you had that, you wouldn't get disconnected when people were in the server.