TECHNICAL SUPPORT / WEB APPs BUGS

Please fix the networking.

I love the skills system.
I like the world and the environment.
I love the friendliness of the other players.
But the absolutely awful networking is a game breaker.

I am looking for a game to play with my wife, and I am happy to pay to play - I paid WoW subscriptions for years - but no way am i going to pay, or even play, a game that gives me "Please Wait" dialogues every 10 minutes or so and "FS lost" at least twice a week. And FS lost means that I can't play anymore. I have to quit the game and then, because there's no connection when I log off, the login server refuses to let me log in again because it thinks I already am, and refuses, absolutely refuses, to change its mind.

Loads and loads of other games can get it right, why can't Ryzom?

Support pointed me at a post that basically blamed it on ISPs using bandwidth shaping. Well my ISP swears blind, and are subject to huge fines if caught lying, that they do no bandwidth shaping of any sort at all on anyone.

Since Ryzom is open source I thought I would look at the source code and see if I could spot anything obvious that might help, but I stopped when the documentation stated that TCP development was underway and UDP was planned for the future, when I have been told, and can tell, that the game uses UDP, and if the networking were still under development, no-one could play.

Here are some suggestions, from a software engineer with 17 years experience including a lot of network software experience:

1) When you try and log in, the login server should tell the game to log off anyone already logged in and should then allow the login. This means that people who get disconnected can reconnect rather than getting locked out until the login server reboots/crashes/decides to change its mind.
2) Use TCP. For the love of software, why would anyone write a protocol that runs across the internet and uses UDP. You aren't even numbering the damn packets, are you? That's why you get rubberbanding. Packets 1 and 3 get through, so you run forward nicely, and then packet 2 arrives a bit late, saying that you're actually 5 yards back, so then your position gets reset. Not necessarily the reason or even the only reason, but very likely. With TCP, if the connection gets laggy then you hang a bit, but after a few seconds all the packets arrive all at once (in effect) and you get a brief bit of fast forward and then all is well.
3) Don't give up unless you really really cannot reconnect, and in that case log the character out! The "FS lost" message in the network window is a sure sign that the game client is likely to hang if you aren't careful, and it never ever recovers, so why not simply log the character out and put them back to the character selection screen, and if you can't retrieve a list of clients then push them back to the login screen. Don't simply hang. And retry the connection at least 3 times, with a reasonably short timeout on each try, say 2 seconds.
4) Inform the user. In game ping is 200, actual ping is 30. What on earth does "FS lost" mean anyway? Why must I "Please Wait". None of these messages means anything.
5) Do NOT buffer data. According to the, probably incorrect and certainly out of date, documentation on the networking, data is buffered at both client and server, and sends are suppressed until a certain time has passed or a certain amount of data is available to send. This is meant to be a realtime game! i'd even consider disabling Nagle's algorithm on TCP connections! And Nagle's algorithm is more sophisticated than what is suggested by the doucmentation.
6) Sort out the design of having a separate login server. It shouldn't be hard to come up with a good design.

I appreciate that running a server with hundreds of clients can be tricky on the bandwidth and other such things, but even a relatively simple, straightforward, server would perform better than this and form the basis for additional tuning and tweaks as issues were found. Reading the documentation it sounds like someone came up with an unnecessarily complicated design with unnecessary "sophistication" that just wrecks the game for me, and probably for a lot of others too. Keep it simple. TCP, asynchronous (non-blocking) sockets, possibly use one thread to transfer data between read/write buffers and sockets and other thread(s) to process the data out of the buffers (I always used one thread to do the whole lot, but with multi-core processors these days, multi-threading makes a lot more sense).

I'm a busy man with a busy family life, but I am happy to offer what assistance and advice I can. Somewhere I even have a little C++ code that handles writing C++ message objects to a socket, written low level (direct Berkeley sockets APIs rather than C++ sockets classes) so it's easy to get direct and detailed control if you later need to tune the performance. I wrote it for a P2P framework I was working on a while back. I'd be happy to share it.

This game has a lot of potential, but a real time online game HAS to have perfect networking and not glitch all the time under normal operating conditions.

sadly, you assume the game makes money.
as far as i know, there are no paid staff working on ryzom and the server is a bit on the cheap side... to say the least. 
i reckon if you are willing to offer your services voluntarily then you'l be snapped up in a jiffy.
i know there are a bunch of other volunteers working on ryzom in different areas.

Ryzom sends constantly small packets to-from server. I believe the aim is one packet every 10ms (game tick). You can't (could not) do that with TCP.
If you miss too many packets, then you enter resync phase (please wait) and those packets are replayed. So it's pretty good for UDP.

Ryzom does try to reconnect as long as possible. If anything, then it does not know when to give up (ie, FS lost, no recovery)

Also, if you try to login it tells you are already logged, it should kick you out in 30sec and allow relog.

You can send packets as often as you like with TCP. And they are guaranteed with an automatic resend mechanism that maintains packet ordering (to avoid rubberbanding). Actual send rate depends on bandwidth and Nagle's algorithm - unless you disable Nagle's algorithm.

The only real advantage that I can see to UDP is multi-casting. Yes, if you structure your data correctly you can become resilient to data loss and improve performance, and maybe that's what they tried to do here. For example if a single positional message is lost, then it doesn't really matter because the information is obsolete almost as soon as it is sent and the next packet, with updated data, replaces it.

However here the networking is clearly inadequate. My wife and I used to play WoW simultaneously, both with an active TeamSpeak connection too, and we almost never had network issues at all, and when we did it was because the server was having problems. Sure Blizzard undoubtedly had top of the range servers etc., but then if the server available cannot handle the load then it should have a login limit - and anyway should have a better mechanism to prevent the please wait happening. Perhaps the current mechanism is great for high bandwidth and performance server end, but not scalable down to low bandwidth and/or poor performance. Maybe it just needs tuning or something.

How about, instead of limiting levels to 125 etc. etc. the game has two servers, a cheap server for trial/free to play, and a second, better, server that is only accessible to subscription members. As more players pay subscription, a yet better server, or a second server, plus better bandwidth, can be purchased.

As for volunteering to help, I already offered what help and assistance I can. The dev team has my email, they are free to use it!

Both server and client sources are available at ryzomcore repo.

Really? I'm guessing you don't have much experience with real time network applications.

Short answer: because TCP and UDP are not designed for the same usage. TCP is used when data integrity is critical and UDP when the throughput is critical and the data integrity isn't the main priority. That's why Ryzom, just like many real-time media applications uses both TCP and UDP depending on the nature of the data which is being transferred. I guess you're quite skeptical, therefore I'll have to name some well-known applications and protocols which relies on UDP (and also TCP, depending on the situation) like Skype, Teamspeak, the Real Time Streaming Protocol and the Real-time Transport Protocol.

If you don't trust me for a reason or another, let me quote Glenn Fiedler: "Using TCP is the worst possible mistake you can make when developing a networked game!" (source). This guy has an impressive résumé in the game programming industry, I think he knows what he's talking about and he explained it well in the article.

+10

Did you mean to link to Frozen?

And how about making this sound more like a discussion than a defensive reaction to the new homins suggestions, remember your manners! *Ponders giving Markanjio time-out in Demon's Lap*

(This seems to link here as well)

And Marceline is right - we are run on a very restricted budget - so any cheap alternatives or adjustments are good *cheap meaning free*

The question still is:

The vast majority of players don't care that you have no money, and that you're operating out of a broom closet. Deliver or go home. Consider that most people just quit playing in a quiet way, without taking the time to write about it; if the issue made it to the forums it must be really bad.

For instance, this very sane and normal suggestion -

could not be implemented because a lot of clients crash when going to the character list screen. A bug as old as dirt, never fixed, that prevents other improvements.

i'm guessing it was a rather poor attempt at sarcastic humour... if the link was intended of course. if not, then mr markanjio is a homin with poor taste in cinema and a knack for condescension.


p.s. i'm aware of the irony here.

Yes, the situation in this clip is similar to the current situation: Olaf claiming he'll be happy in summer but doesn't know it will kill him is just like Halimaw claiming TCP will fix Rysom issues… and of course using TCP everywhere will create huge issues, breaking everything. And, of course, I adapted what Kristoff said to Olaf.

You don't get it, it's absolutely not a "defensive reaction to the new homins suggestions". My attitude was simply a reaction to Halimaw's own bad attitude. This guy came here bragging about his superiority ("software engineer with 17 years experience") and how Ryzom developers did stupid choices ("why would anyone write a protocol that runs across the internet and uses UDP"). Even if he was right, which is absolutely not the case, it wouldn't have been a nice way to explain the problem. The fact is he was terribly wrong, and I explained him why using as much politeness as he did to expose his terribly bad idea. I would have been very kind if he wasn't harsh in the first place.

We have a catch 22 here. If people do not subscribe because of the current lag situation it won't improve. but get worse as income might not even high enough for the current used low power server.

"I would play but the player base is to small, thats why I do not play" . A death spiral many mmos games face.

I suggest a prepaid system instead (or on top) of the current pay monthly subscription plan. With perks the more money you deposit. The only reason I wait until I hit level 125 with my main skill, is the outdated subscription model. You always pay for the full month but might only play a few days, or do not play full time for a certain time period (and feel forced to play as you pay for something) .

That way winchgate can work with the money players deposit, and do not struggle as soon as peopel unsub for a break or whatever ) summertime for example

Sorry for my english ;)


PS: Yesterday a GM hop on and notified everyone connected during that time that the lags should be fixed, and since then they only happen here and there .

(this is straying offtopic, but..)

The vibe I've gotten from a lot of the current players is that any form of paying money for game features, aside from an unlocking of skill levels and a few perks, is literally the Worst Thing That Could Ever Happen. People have suggested cash shops before ... and they were met with a very forceful NO from players.

Implementing some cash-grab features, along with some developer work put in to fix the glaring bugs (i.e. Zora crash), would definitely drag in more players. At the same time it would drive out the old-timers, some of whom would be unable to cope with the influx of changes. So the move would effectively throw out a portion of the game's history that lives with those players (especially the RP histories, etc). As the landscape and game atmosphere change, more and more veterans will quit.

This is a rather known phenomenon, mass adoption always drives out the weird customs that a small community develops. Along with all the good stuff like a peaceful environment, etc.

From the company perspective, it would seem both your death spiral scenario and mine are equally scary.

I think the subscription is pretty cheap in reality - and my old guild leader paid for 3 accounts for a very long time after he "hung up his gaming boots" as a way of supporting the game that he had enjoyed.

Hello, the "please wait" problem has been fixed. We apologize for the inconvenience.