Are you still getting ERROR 500 now?

[Andrea B.]

Please let me know if you have seen ERROR 500 within the last hour or so.

Current time:  22:18 UTC Sunday 16 July 2023

 

And let me know if you see it in the future.

 

I have tried a few things to fix it. 

[Andrea B.]

Also note that I have temporarily throttled searches by "Guests" to 1 per hour. 

This is just a test. I will probably remove that throttle later.

Meanwhile if you want to make repeated searches, then you must log in, OK?

[Andrea B.]

Unique Visitor
A unique visitor is a person or computer (host) that has made at least 1 hit on 1 page of your web site during the current period shown by the report. If this user makes several visits during this period, it is counted only once. Visitors are tracked by IP address, so if multiple users are accessing your site from the same IP (such as a home or office network), they will be counted as a single unique visitor.

 

Visits

The number of visits made by all visitors. Think "session" here: for example, a unique IP accesses a page and then requests three other pages within an hour. All 4 of those pages are included in the one visit. So you should expect multiple pages per visit and multiple visits per unique visitor.

 

Pages

The number of pages viewed by visitors. Pages are usually HTML, PHP, JS or ASP files, not images or other files requested as a result of loading a page.

 

HITS

The number of files requested from the server such as web pages css docs, graphics, images, buttons, links. The loading of one web page can amount to several hits depending on what elements a page contains.

 

Jan 277593 friendly hits from 4240 unique visitors for 8053 visits (1.89v/v).

Jan 114169 bot hits

 

Feb 591156 friendly hits from 8302 unique visitors for 20197 visits (2.43v/v).

Feb 264350 bot hits

 

Mar 616307 friendly hits from 9436 unique visitors for 20541 visits (2.17v/v).

Mar 259407 bot hits

 

Apr 605348 friendly hits from 8821 unique visitors for 19249 visits (2.18v/v).

Apr 279361 bot hits

 

May 517936 friendly hits from 7348 unique visitors for 15087 visits (2.05v/v).

May 402490 bot hits

 

Jun 562792 friendly hits from 8858 unique visitors for 17454 visits (1.97v/v).

Jun 450392 bot hits

 

And so far in July

Jul 284078 friendly hits from 4611 unique visitors for 8213 visits (1.78v/v).

Jul 1405118 bot hits in the first half of the month !!!

[ulf]

It looks like the problem is solved.

I have not seen the error for several hours and many accesses.

Both reading and posting seams to work well again.

 

Thank you Andrea for solving this.

[Andrea B.]

I hope that I actually did solve the ERROR 500 problem! I will be keeping a close eye on things for awhile.

---

 

 

Admin Note:  I'm placing some ERROR 500 notes here for future reference.

 

After I created a Trouble Ticket, the Site5 tech discovered a Quality-of-Service error ("too many concurrent requests") for UVP in the server log of our shared server. Site5 included an excerpt from the log (see above), but made no suggestions for dealing with the problem.

 

I ran a WhoIs on one of the IP addresses associated with the QoS error and learned that the requests originated from Singapore.

 

That led me to ask: were we being hit on or scraped by a bad crawler?

In countries where the internet is tightly controlled, a search engine will sometimes attempt to reproduce a foreign website (for "local" access via that search engine) by collecting and caching all the pages and page elements. I suppose we could consider this a theft of a website, but I like to think the search engine team simply wanted to provide a nice website for folks who could not ordinarily access it. 

 

UVP has instructions in its robots.txt that instruct such bots and crawlers to go slowly. But obeying robots.txt is not mandatory. Bots misbehave all the time. If a bot is creating too many simultaneous requests for different pages and page elements, the server and the software cannot cope with the dense traffic.

 

Next I looked at our website statistics (cPanel>Metrics>Awstats), where it was clear that there was indeed some unusual Robot/Spider traffic beginning in May and dramatically increasing in the first half of July. This is documented above. 

 

Then I looked in our UVP raw access logs (cPanel>Metrics>Raw Access) in order to attempt to find who was hitting on UVP excessively. The request string IP address in our raw access log matching the Singapore IP address belonged to Bytespider.

 

Bytespider was blocked in .htaccess. UVP's web server log (cPanel>Metrics>Errors) lists several current denials to Bytespider.

 

---

 

 

The Site5 tech handling the Trouble Ticket made no suggestions about how to solve the problem. I asked them again on chat and was told that I should check with the software people (Invision). I posted a query on the Invision customer forum and was first told that I needed to contact the server people (Site5). Bit of a circular handoff there, geez.

 

The second Invision response was that I needed a larger server hosting plan. Clearly this is not the case because UVP's monthly Unique Visitors never reaches the 10,000 limit of our contract. 

 

---

 

 

 

 

 

 

 

[Andrea B.]

BTW, FWIW, I had absolutely no idea at first what was going on. I should have looked into the ERROR 500 problem sooner. But I was thinking that it was just the usual internet hassle.  

 

Oh well. We live & learn. Or sometimes learn. I hope I've learned this time!! I think so.