@Keener-Trooptrack@AaronStorey I am double posting this to be sure it get’s attention. TroopTrack appears to be down again. I am getting the “We’re sorry but something went wrong” message. It’s not my first choice to communicate my frustrations with TroopTrack in a public forum, but since my calls and emails haven’t resulted in a response I have little choice.
Over the past month or so TroopTrack has been consistently problematic. On 3/29 I opened ticket 2278 for an issue that made TroopTrack unusable for our Troop. Multiple voicemails and responses to the ticket email thread went unanswered, for weeks, until I posted in this forum. In the meantime I devised my own work-arounds so that we could at least use TroopTrack in limp mode, without any support from you or your team. The issue wasn’t resolved until 4/22, three and half weeks later.
Since that time there have been several significant outages, including emails being stuck for an extended period of time, multiple cases of TroopTrack being exceedingly slow, and multiple cases of it being down like it is now.
I am resisting internal pressure in my Troop to ditch TroopTrack and move to Scoutbook. I have a strong preference to stick with TroopTrack because of it’s superior communication (email) and scheduling functionality. However, the poor uptime and customer service are making it more and more difficult to defend my stance. I hope you can help me out by improving on both of those fronts.
Just as an FYI, I am a user just like you. I have just been using TT for many years and am very familiar with it so try and help people in the forums as much as I can. I tagged other TT admins in your other post so hopefully they will communicate soon. It can be very frustrating when things like this happen, but please bear in mind TT is a small company with very few employees and other than the problems over the last month or so it has been very solid for the last several years. Unfortunately in technology when one thing goes wrong, everything goes wrong.
Thanks @AaronStorey. You have always been so responsive and helpful that I just assumed you were on staff! Thanks so much for your help! FWIW I am the Director of IT for my company and have a pretty long history in the IT world, ranging from being helpdesk support to managing global corporate infrastructure and support teams. So, I like to think I have some idea of the challenges involved. Still, my complaints stand… I hope the TroopTrack staff can figure out the root issues and improve on their customer service so I can continue to promote its use in our Troop.
I definitely agree that they can do better in their timely responsiveness, overall they had actually improved quite a bit in that realm with adding Tyler and then David onto staff, monitoring the Community, and support e-mails however they have fallen off over the last month or so with these issues.
I understand your concerns with the servers and slow downs. I experience these issues myself as I try and help people navigate TroopTrack and do demos.
I was just now able to talk with our tech team about these issues. They’ve had a lot on their plate recently with bug fixes and server issues in combination. They’re currently focused on resolving the server issues and improving the server performance (this is why it may seem like other bugs are building up). We did have two servers that showed they were working, but one of them had some issue where it actually wasn’t loading jobs but it said it was. This server has been replaced (just now). Our tech team has also been working on upgrading to a faster system as well.
All this is in hopes that these server slowdowns occur less frequently and less often. I realize that having slowdowns and hiccups is never fun, but TroopTrack is working on the issue and this is a relatively new problem that has started to occur in these past couple weeks. All in all they’re working to make sure that TroopTrack continues to stay up, and continue to support our growing customer base.
I appreciate your transparency in letting us know here in a forum regarding the issues. However, a basic email communication to end-users letting them know to expect outages or maintenance, let them know they are being acknowledged or even a reliability site URL where users can see what issues and bugs are being worked on/pending or what particular pieces of the application are being worked on. For instance, without releasing any private or intellectual information, disclosing the type of server would be beneficial as an end-user guaging reliability. For instance, has there been discussion of maybe virtualizing your platform to go to AWS or an Azure environment? Based on the response above, it sounds like 100% on premise is still the way things are currently functioning (or trying to stay functioning). Has a hybrid solution been proposed with an on-site replication with a cloud backup? The amount of slowdowns has been extremely stressful for troop management here as well. I love that the community is growing but, as a former systems administrator that’s worked with other applications, I’ve seen way too many die off because they didn’t build or plan systems for future growth and I would hate to see that happen with Troop Track. Thanks for listening.
TroopTrack is 100% hosted on AWS and has been for over ten years. Our database is backed up in real-time and is totally recoverable. We run multiple application servers behind a load balancer. The specific problem we’ve had lately is that one of these servers was refusing to connect to the load balancer. We’ve since replaced that server and the new server is balancing correctly.
Even with servers in the cloud, they still sometimes have problems. In the case of TroopTrack, PDF generation is a very popular feature but it is memory and CPU intensive and is not scaling well. So we are contemplating ways of improving that. Right now it is causing some performance problems and occasionally causing server reboots.
I hope this is helpful information. I’m sorry that the way we talk about our servers created the impression we were on-premises.
That is great to hear and know. I appreciate the info, David. This makes completed sense and it’s very unfortunate this happened to you. And, yes, I know all about problems in the cloud as well. PDF generation is a beast so I empathize with you on that front. Have you all given thought to rendering on a different server with dedicated resources? We had to do this with another application that solely ran reports because we ran into the same issue where trying to run those processed on the same system was killing the front-end.
We currently generate some PDFs in the background on a different server. I am considering expanding that. I am also thinking of reducing the number of PDFs we offer and instead just rendering HTML pages that are printable. We will probably end up with a mix of both.
I like this idea! Html printables with maybe some custom editing options as a pro feature (with all the crazy stuff disabled. Don’t want to generate animated sparkly reports. Ok, maybe one out two sparkles with flash gifs )
@dave Thanks for your detailed response. It’s interesting to hear about the specific challenges being faced. I have always held the belief that IT Customer Service starts with the perspective that the system belongs to the users, not the IT department, and as such transparency about the issues being faced creates a lot of good will. Obviously the users often need protected from themselves! One of the more frustrating aspects of my recent experience was the lack of communication, so I greatly appreciate your efforts in that regard. FWIW in a previous life I worked for a company that focused on document management and dealt quit a bit with PDF conversions. Each engine has it’s own idiosyncrasies of course, but if memory serves, TIFF conversions were usually more reliable and less resource intensive. Not sure if that helps any, but I thought I would throw it out there. In the case of PDFs it was pretty common practice to have an independent server or servers that did all of the conversion/generation work because it was so resource intensive, so it seems like you are on the right track. Good luck sorting it all out and thanks for building such a great product!
So, 6 days ago, the server was replaced and you’re working on upgrading. When should we expect to be reliably able to use TroopTrack again? Throughout the day, I’ve tried doing some needed updates to get us ready for our year-end stuff. One minute, it works. Then suddenly I get a timeout failure, or “We’re sorry but something went wrong.” So I walk away, come back later, it’s working again, until it’s suddenly not.
The server slowdowns that were occuring have been fixed. There shouldn’t be any more random error messages when navigating from one page to another in TroopTrack. You should be able to use TroopTrack normally again. The upgrades are simply to make TT even faster in the future,
What specific things are you trying to do and you get the error message for?
FWIW I got the error once yesterday logging in from the mobile app on my iPhone. I tried a second time, then went to my computer, and it worked fine. Seemed to be transient.
well, that was the weird part. it wasn’t consistent. For example, I remember one specific thing I did was click the gear at the top, then Manage Active Achievements. The “loading bar” slowly inched along the top of the screen, then I got one of the error messages. So I walked away. Tried again 5 minutes later. Worked fine.
I don’t remember any other specifics because it was happening SO often, I didn’t try to keep track. But if I just walked away and did something else, eventually when I came back and tried the same thing, it would work. Attendance/calendar tasks. Service hour tasks. Bulk achievements. I think it was all affected. There was no pattern that I could discern. Two or three clicks would work, then the next would time out or tell me sorry. I’d walk away, come back in 5 minutes, it would often work then.
Just now I clicked around a bit and everything was fast and perfect…
Yes, yesterday, may 11.
Based on my history, here is when I was on TT;
11:18-11:21am (manage users, participation and achievement)
11:25-11:27am (calendar)
11:46-11:49am (calendar, edit and attendance)
7:08-7:12pm (achievement, progress)
7:32pm (dashboard)
7:55-8:00pm (messages)
8:54pm (dashboard)
8:59-9:09pm (achieve, agenda, presentation)
10:51-10:53pm (user achievement; calendar)
I don’t know if I encountered issues EVERY time, but I feel like I did. And it would explain why most of my sessions were 5 minutes or less.
Hope this helps pinpoint what the problem was. (Today i was on for about 15 minutes without any issues at all.)
It’s down again now. I was just using the Incomplete Badge book. Then clicked to go to Calendar and got a “We’re sorry, but something went wrong.” Tried in another window to go to the dashboard and got a gateway time out.
)