Last active
July 19, 2021 09:34
-
-
Save jczaplew/5799404 to your computer and use it in GitHub Desktop.
502 Proxy Error with Apache, Node, and MySQL
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Setup: | |
Node running on :8080 | |
Apache running on :80 | |
Using Apache mod_proxy to forward application requests to :8080 | |
Symptoms | |
On client: | |
1. Proxy Error | |
The proxy server received an invalid response from an upstream server. | |
The proxy server could not handle the request GET /route. | |
Reason: Error reading from remote server | |
(with 502 header) | |
2. Hanging page load (request sends, no response received) | |
3. Routes without complex/any queries work fine after restarting Node | |
In Apache error_log: | |
1. [error] [client xx.xx.xxx.10] proxy: Error reading from remote server returned by /route | |
2. [error] [client xx.xx.xxx.10] (70014)End of file found: proxy: error reading status line from remote server localhost:8080 | |
In Node error log: | |
1. Nothing | |
On server: | |
1. Simple SELECT queries on database work fine | |
2. Complex queries hang indefinitely | |
3. Running "SHOW PROCESSLIST" produces a list of queries with the status "Waiting for table-level lock" | |
Temporary fix | |
1. In MySQL, run "SHOW PROCESSLIST" | |
2. For each process, run "KILL [unique id]" | |
3. Restart Apache | |
4. Restart Node | |
After doing this, CPU usage dropped from ~12% to ~0.5% and performance went back to normal | |
A slightly better fix | |
1. Bash script to kill all at once per this answer: http://serverfault.com/questions/11357/bulk-or-mass-killing-misbehaving-mysql-queries | |
Ideal fix | |
1. Figure out what is causing certain queries to obtain the status of "Waiting for table-level lock". Could have something to do with the database storage engine being InnoDB instead of MyISAM (http://stackoverflow.com/questions/6543624/what-is-waiting-for-table-level-lock) | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment