Say you have a shiny working WordPress based website withtout any monitoring set for years but suddenly, you open the site and you get the terrifying error:
There Has Been a Critical Error on Your Website
That is quite of a stress for sure. As in the first few minutes you don't understand how this has happened since, you did not touched the perfeclty working site for a very, very long time.
Then you start to debug into the apache / nginx access.log, error.log and mysql mysql.err etc. franticly trying to figure it out the normal ideas pop-up immediately into mind, whether you have a recent backup for the website's database. If you have pair of high availability webservers service or backup databases that serve the traffic via a separate standby instance of the service, you might try to switch off the official service and see whether the standby Webserver / SQL server instance would serve the website fine.
However, if this is not an option and you have no standby backup service as a recovery Plan B option already set. Your only option is to continue to debug what is wrong.
Then the next thing to do is to check whether you don't have a Web Caching or Proxy in front of your webservers that are preventing you to see a recent version of the website and give you some old cache or you don't have an ISP proxy that is giving you some unreal results. That is easily seenable from the Webserver logs. If this is neither the case the next thing is to:
Enable WordPress (wp-config.php) Debug mode
By default for Security reasons the WordPress PHP execution debug mode is switched off inside wp-config.php.
When there are odd pages with the WordPress based blog or site however this can easily be changed by modifying the WP_DEBUG true|false value.
To do so edit with a text editor such as vim / nano / mcedit wp-config.php or if no SSH access to the remote machine, use SFTP / FTP transfer protocol copy the file to your desktop and inspect it and make sure the WP_DEBUG / WP_DEBUG_DISPLAY / WP_DEBUG_LOG has following values:
define( 'WP_DEBUG', true );
define( 'WP_DEBUG_DISPLAY', false );
define( 'WP_DEBUG_LOG', true );
Reloading the Browser window tab with There is a critical error on Your website, you should get some Errors or Warnings like:
Warning: Illegal string offset 'parent_slug' in /var/www/websitecom/wp-content/plugins/photo-gallery/booster/main.php on line 180
Warning: Illegal string offset 'slug' in /var/www/websitecom/wp-content/plugins/photo-gallery/booster/main.php on line 180
Then you can temporary disable the problematic problem in that case for example the photo-gallery and recheck the website, and then restore from backup snapshot the respective plugin files version from a moment, when the website was working.
If this doesn't solve it and more plugins are crashing and you can't find an easy way to work-around it you miss a backup, you might try to
Disable all WordPress active plugins
Disable your plugins from the dashboard, visit Plugins > Installed Plugins and tick the checkbox at the top of the list to select them all.
Then click Bulk Actions -> Deactivate, which should be enough to disable any conflicts and restore your site.
You can do essentially the same thing through SSH / FTP session.
Step 1: Log in to your site with SSH / FTP.
Step 2: Open the wp-content folder to find your plugins.
Step 3: Rename the plugins folder to plugins_old and verify that your site is working again via SSH run commands:
# cd path_to/plugins; mv plugins plugins_old
or rename via FTP client
Step 4: Rename the folder back to “plugins”. The plugins should be disabled still, so you should be able to log in to your dashboard and activate them one by one. If
the plugins reactivate automatically, rename individual plugin folders with _old until your site is restored.
Raise the PHP Memory Limit
Sometimes, a low PHP limitation causes critical errors on WP based blogs and sites, if necessery raise up the memory limitation via:
define( 'WP_MEMORY_LIMIT', '128M' );
Change Max Upload File Size and Text Processing function limits
To increase the max upload file size, add this code to wp-config.php:
ini_set('upload_max_size' , '256M' );
ini_set('post_max_size','256M');
And to fix the breaking of large pages on your site, add this code:
ini_set('pcre.recursion_limit',20000000);
ini_set('pcre.backtrack_limit',10000000);
Clear up any caches
If you use some session caching of the website on the machine such as memcached / ncache / redis / varnish or an haproxy or any proxy in front of the webserver to do some kind of High availability could produce strange unexpected Critical errors on Your Website, thus restarting such services or cleaning up any cache would be advisable if you have such.
What Causes "There Has Been a Critical Error on Your Website" error?
The reason could be practically anything as WP is a kind of multi-comonent free and a bit of bloatware. The general ones could be from a missing database table / table fields to a messed up plugin after update a disappeared critical plugin or essential wordpress PHP file, but in my specific case the reason was simple the Plugins Auto-update, which I have had the stupidity to enable.
The WordPress Automatic Updates, though saving you effort and Protecting your website in most cases against recent bugs and Exploits and increasing the WP security level, often causes issues and from my personal experience it is not recommended so better avoid it. Again next time you implement any automation to your server make sure you put some kind of monitoring.
Even if you decide to enable it make sure you do it the right way and not like me, by enabling some Monitoring to the WordPress site via Zabbix / Nagios / Cacti / monit etc to be sure you get notified immediately if the WordPress based site is down.
Report haproxy node switch script useful for Zabbix or other monitoring
Tuesday, June 9th, 2020For those who administer corosync clustered haproxy and needs to build monitoring in case if the main configured Haproxy node in the cluster is changed, I've developed a small script to be integrated with zabbix-agent installed to report to a central zabbix server via a zabbix proxy.
The script is very simple it assumed DC1 variable is the default used haproxy node and DC2 and DC3 are 2 backup nodes. The script is made to use crm_mon which is not installed by default on each server by default so if you'll be using it you'll have to install it first, but anyways the script can easily be adapted to use pcs cmd instead.
Below is the bash shell script:
To configure it with zabbix monitoring it can be configured via UserParameterScript.
The way I configured it in Zabbix is as so:
1. Create the userpameter_active_node.conf
Below script is 3 nodes Haproxy cluster
Once pasted to save the file press CTRL + D
The version of the script with 2 nodes slightly improved is like so:
The haproxy_active_DC_zabbix.sh script with a bit of more comments as explanations is available here
2. Configure access for /usr/sbin/crm_mon for zabbix user in sudoers
3. Configure in Zabbix for active.dc key Trigger and Item
Tags: access, ALL, and, Anyways, are, available, awk, bash shell, bash shell script, Below, bit, case, cat, Central, check, Cluster, cmd, Comments, conf, configure
Posted in Linux, Monitoring, Zabbix | No Comments »