Long time no speak! I'm sorry that I have been MIA for the past week or so, but this post will detail a timeline of what's been going on with the server for the past few days.
Friday, January 6th @ 2am CST
The server encountered massive errors, causing it to crash. This crash revealed (and most likely caused) massive chunk errors on a player's land. Their builds and progress was completely reset and the land looked as if it had never been touched.
We started rifling around, looking for errors and trying to find some sort of solution to this problem. We were unsuccessful. The affected area spanned tens of thousands of chunks. We knew as a whole that we couldn't leave the map in this condition, as it wasn't healthy.
I (Bannah) regioned off the affected area to the best of my ability and kept players from entering until we resolved the issue.
We had options at this time. We could try and fix the errors or put in a backup of the map made the previous Monday. If we put in the backup made of the map, all players would've lost progress made for around five days. Myself, Bri and Rayne were being faced with a tough decision after we discovered that we weren't going to be able to fix the errors in the map on our own.
Sunday, January 8th
I left for vacation and while en route, Rayne messaged me, telling me that the server had been fixed, by updating a few files. I was relieved alongside everybody else who was working on the issue. We thought all was well.
Monday, January 9th - 12th
The server went down again, logging over 32,000 errors. We were at a loss as to what was going on, as we were sure we had fixed the issue the previous day. Now, instead of just a portion of the map being affected by the regeneration error, the ENTIRETY of the map was now affected and the server could not be played on.
I can only speak for myself, but i'd be lying if I said that I wasn't dejected and frustrated. We had no clue where to go from here or how to fix this situation. I was out of the state in an area with no internet connection, supposed to be enjoying a vacation. All the while, Bri and Rayne are back home working on the server, trying to give me updates and just figure out why this was happening.
This process went on for the next 3 days. The server was down and we were all clueless as to why the map files were so unhealthy. We were now faced with the prospect of having to scrap the entire map and start anew. We even considered moving the server on to new hardware in an effort to solve this problem. We weren't sure that putting in the backup made the previous week would solve our problem. We didn't know where the problem even began let alone where it was.
We were all ready to give up, to be completely honest. The map wasn't playable and we knew that players would be upset and possibly leave if we had to reverse progress or reset the map all together.
Thankfully, in the end, we were able to solve our issues, only losing minimal amounts of progress.
THE SERVER IS NOW LIVE, BUT:
- Map progress has been reversed back about a week and a half.
(this was unavoidable, and was the only way other than a full reset that we could get the server back up again)
- Some plugins have been removed. We aren't sure that we will ever be able to put them back in. Some or all of these plugins caused massive errors and are possibly the cause of our entire ordeal. We cannot promise if these plugins will ever be put back onto the server.
( Lift, ExoticGarden, and MobArena )
- If you notice any significant changes to anything OTHER THAN your map progress, i.e. ranks or other purchased items. PLEASE contact us IMMEDIATELY.
I would personally love to thank all the the administrative team who worked endlessly to figure this issue out and resolve our problems in the most efficient and effective way possible. From the bottom of my heart, you guys mean the world to me and I could not have done this without you and would never want to see a day that you guys aren't working alongside me on this server. Specifically Bri and Rayne, you guys are the heart and soul of this place and I am forever indebted to you both. Thank you for your commitment and coming along with me when I proposed this crazy scheme over a year ago.