Ticket #641 (closed maintenance: fixed)
Enable dynamic Munin graphs
| Reported by: | chris | Owned by: | chris |
|---|---|---|---|
| Priority: | critical | Milestone: | Maintenance |
| Component: | Dev server | Keywords: | |
| Cc: | jim, ed | Estimated Number of Hours: | 0.0 |
| Add Hours to Ticket: | 0 | Billable?: | yes |
| Total Hours: | 1.8 |
Description
If we only generate munin graphs on the fly we will also get the zoom function working, this example config looks good: http://uname.pingveno.net/blog/index.php/post/2013/08/25/Configure-Munin-graphs-with-Nginx-and-Debian-7
Attachments
Change History
comment:1 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 1.0
- Status changed from new to closed
- Resolution set to fixed
- Total Hours changed from 0.0 to 1.0
comment:2 Changed 3 years ago by chris
- Priority changed from major to critical
- Status changed from closed to reopened
- Resolution fixed deleted
I have just noticed that we haven't had any updated graphs from Munin since just after noon yeaterday, something isn't working, reopening this ticket.
Changed 3 years ago by chris
- Attachment penguin-cpu-2013-12-07.png added
Craph of Penguin CPU usage showing the effect of switching to dynamic Munin graphs
comment:3 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 0.7
- Total Hours changed from 1.0 to 1.7
I can't see anything significant in /var/log/munin/*
I'm still getting Munin emails.
This line in /etc/init.d/munin-fastcgi was wrong:
FCGI_SPAWN_HTML=/usr/lib/munin/cgi/munin-cgi-graph
So it was changed to:
FCGI_SPAWN_HTML=/usr/lib/munin/cgi/munin-cgi-html
Also in /etc/nginx/sites-available/penguin this was added to the end of the location /munin/ section, following http://munin-monitoring.org/wiki/MuninConfigurationMasterCGI
fastcgi_split_path_info ^(/munin)(.*);
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_pass unix:/var/run/munin/fastcgi-munin-html.sock;
include fastcgi_params;
And now it's all working.
I have posted a comment to http://uname.pingveno.net/blog/index.php/post/2013/08/25/Configure-Munin-graphs-with-Nginx-and-Debian-7 noting the mistake in the script.
This graph illustrates the effect on CPU usage from the switch (orange):
This is the final Ngnix config file, minus comments:
server {
listen 80 default_server;
server_name penguin.transitionnetwork.org penguin.webarch.net 81.95.52.111;
access_log /var/log/nginx/penguin.access.log;
error_log /var/log/nginx/penguin.error.log crit;
root "/web/penguin.transitionnetwork.org/www";
index index.html;
location ~ /\. {
access_log off;
log_not_found off;
deny all;
}
location ~ .~$ {
return 403;
}
location = /apc_info.php {
fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_intercept_errors on;
include fastcgi_params;
access_log off;
allow 127.0.0.1;
allow 81.95.52.78;
allow 81.95.52.29;
deny all;
}
location /nginx_status {
stub_status on;
access_log off;
allow 127.0.0.1;
allow 81.95.52.78;
allow 81.95.52.29;
deny all;
}
location ~ ^/(status|ping)$ {
fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_intercept_errors on;
include fastcgi_params;
access_log off;
allow 127.0.0.1;
deny all;
}
location ^~ /munin {
rewrite ^/(.*)$ https://penguin.transitionnetwork.org/$1? permanent;
}
location ^~ /webalizer {
rewrite ^/(.*)$ https://penguin.transitionnetwork.org/$1? permanent;
}
location ^~ /phpmyadmin {
rewrite ^/(.*)$ https://penguin.transitionnetwork.org/$1? permanent;
}
location ^~ /info {
rewrite ^/(.*)$ https://penguin.transitionnetwork.org/$1? permanent;
}
include gzip;
}
server {
listen 443 default_server;
server_name penguin.transitionnetwork.org penguin.webarch.net 81.95.52.111;
access_log /var/log/nginx/penguin.ssl_access.log;
error_log /var/log/nginx/penguin.ssl_error.log notice;
ssl on;
ssl_certificate /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;
ssl_certificate_key /etc/ssl/transitionnetwork.org/transitionnetwork.org.key;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
root "/web/penguin.transitionnetwork.org/www";
index index.html;
location /munin/ {
if ($uri ~ /munin/([^/]*)) { set $path $1; }
fastcgi_split_path_info ^(/munin)(.*);
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_pass unix:/var/run/munin/fastcgi-munin-html.sock;
include fastcgi_params;
}
location ^~ /munin-cgi/munin-cgi-graph/ {
if ($uri ~ /munin-cgi/munin-cgi-graph/([^/]*)) { set $path $1; }
fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*);
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_pass unix:/var/run/munin/fastcgi-munin-graph.sock;
include fastcgi_params;
}
location ^~ /munin-cgi/munin-cgi-html/ {
if ($uri ~ /munin-cgi/munin-cgi-html/([^/]*)) { set $path $1; }
fastcgi_split_path_info ^(/munin-cgi/munin-cgi-html)(.*);
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_pass unix:/var/run/munin/fastcgi-munin-html.sock;
include fastcgi_params;
}
location /munin/static/ {
alias /etc/munin/static/;
}
location /webalizer/ {
alias /web/penguin.transitionnetwork.org/www/webalizer/;
autoindex on;
satisfy any;
deny all;
auth_basic "Web stats";
auth_basic_user_file /web/tech.transitionnetwork.org/.htpasswd;
}
location = /info/ {
alias /web/penguin.transitionnetwork.org/www/info/;
autoindex on;
satisfy any;
deny all;
auth_basic "PHP Info";
auth_basic_user_file /etc/phpmyadmin/htpasswd;
}
location ~ /info/(apc|php-info)\.php$ {
fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
fastcgi_intercept_errors on;
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_param HTTPS on;
include fastcgi_params;
satisfy any;
deny all;
auth_basic "PHP Info";
auth_basic_user_file /etc/phpmyadmin/htpasswd;
}
location /phpmyadmin/libraries {
return 403;
}
location /phpmyadmin {
root "/usr/share";
index index.php;
autoindex off;
satisfy any;
deny all;
auth_basic "PHPMyAdmin";
auth_basic_user_file /etc/phpmyadmin/htpasswd;
}
location ~ ^/phpmyadmin.*\.php$ {
root "/usr/share";
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_param HTTPS on;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_intercept_errors on;
include fastcgi_params;
fastcgi_pass unix:/var/run/php5-fpm/phpfpm.sock;
satisfy any;
deny all;
auth_basic "PHPMyAdmin";
auth_basic_user_file /etc/phpmyadmin/htpasswd;
}
location ~ /\. {
access_log off;
log_not_found off;
deny all;
}
}
comment:4 Changed 3 years ago by chris
- Add Hours to Ticket changed from 0.0 to 0.1
- Status changed from reopened to closed
- Resolution set to fixed
- Total Hours changed from 1.7 to 1.8
Closing the ticket again, all Munin HTML and images are now generated dynamically on request, see the date in the footer of pages and dates on graph images. You might need to do Ctrl-r in your browser to refresh pages.
Note: See
TracTickets for help on using
tickets.
On wiki:PenguinServer following http://uname.pingveno.net/blog/index.php/post/2013/08/25/Configure-Munin-graphs-with-Nginx-and-Debian-7
Created /etc/init.d/munin-fastcgi containing:
#! /bin/sh ### BEGIN INIT INFO # Provides: munin-fastcgi # Required-Start: $remote_fs $network # Required-Stop: $remote_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: starts munin-fastcgi # Description: Spawn Munin FCGI sockets for Web access ### END INIT INFO # # munin-fastcgi Startup script for Munin CGI services # # chkconfig: - 84 15 # description: Loading Munin CGI services using spawn-cgi # HTML files and CGI. # # Author: Ryan Norbauer <ryan.norbauer@gmail.com> # Modified: Geoffrey Grosenbach http://topfunky.com # Modified: David Krmpotic http://davidhq.com # Modified: Kun Xi http://kunxi.org # Modified: http://drumcoder.co.uk/ # Modified: http://uname.pingveno.net/ DAEMON=/usr/bin/spawn-fcgi FCGI_GRAPH_SOCK=/var/run/munin/fastcgi-munin-graph.sock FCGI_HTML_SOCK=/var/run/munin/fastcgi-munin-html.sock WWW_USER=www-data FCGI_USER=munin FCGI_GROUP=munin FCGI_SPAWN_GRAPH=/usr/lib/munin/cgi/munin-cgi-graph FCGI_SPAWN_HTML=/usr/lib/munin/cgi/munin-cgi-graph PIDFILE_GRAPH=/var/run/munin/fastcgi-munin-graph.pid PIDFILE_HTML=/var/run/munin/fastcgi-munin-html.pid DESC="Munin FCGI for Graph an HTML" # Gracefully exit if the package has been removed. test -x $DAEMON || exit 0 test -x $FCGI_SPAWN_GRAPH || exit 0 test -x $FCGI_SPAWN_HTML || exit 0 start() { $DAEMON -s $FCGI_GRAPH_SOCK -U $WWW_USER -u $FCGI_USER -g $FCGI_GROUP -P $PIDFILE_GRAPH $FCGI_SPAWN_GRAPH 2> /dev/null || echo "Graph Already running" $DAEMON -s $FCGI_HTML_SOCK -U $WWW_USER -u $FCGI_USER -g $FCGI_GROUP -P $PIDFILE_HTML $FCGI_SPAWN_HTML 2> /dev/null || echo "HTML Already running" } stop() { kill -QUIT `cat $PIDFILE_GRAPH` || echo "Graph not running" kill -QUIT `cat $PIDFILE_HTML` || echo "HTML Not running" } restart() { kill -HUP `cat $PIDFILE_GRAPH` || echo "Can't reload Graph" kill -HUP `cat $PIDFILE_HTML` || echo "Can't reload HTML" } case "$1" in start) echo "Starting $DESC: " start ;; stop) echo "Stopping $DESC: " stop ;; restart|reload) echo "Restarting $DESC: " stop # One second might not be time enough for a daemon to stop, # if this happens, d_start will fail (and dpkg will break if # the package is being upgraded). Change the timeout if needed # be, or change d_stop to have start-stop-daemon use --retry. # Notice that using --retry slows down the shutdown process somewhat. sleep 1 start ;; *) echo "Usage: $SCRIPTNAME {start|stop|restart|reload}" >&2 exit 3 ;; esac exit $?Make it executable and start it and chown log files:
Edit /etc/logrotate.d/munin in vim and run:
Edit /etc/nginx/sites-available/penguin and comment out and insert
# If the most specific prefix location has the "^~" prefix then regular expressions are not checked. # http://nginx.org/en/docs/http/ngx_http_core_module.html#location #location /munin { # http://munin.readthedocs.org/en/latest/example/webserver/ # alias /var/cache/munin/www; # expires modified +310s; #} location /munin/ { alias /var/cache/munin/www/; # not authenticated => no rewrite (back to auth) if ($remote_user ~ ^$) { break; } # is on the right subtree ? set $ok "no"; # admin can see it all if ($remote_user = 'admin') { set $ok "yes"; } # only allow given path if ($uri ~ /munin/([^/]*)) { set $path $1; } if ($path = $remote_user) { set $ok "yes"; } # not allowed here ? redirect them where they should land if ($ok != "yes") { # redirect to where they should be rewrite / /munin/$remote_user/ redirect; } } location ^~ /munin-cgi/munin-cgi-graph/ { # not authenticated => no rewrite (back to auth) if ($remote_user ~ ^$) { break; } # is on the right subtree ? set $ok "no"; # admin can see it all if ($remote_user = 'admin') { set $ok "yes"; } # only allow given path if ($uri ~ /munin-cgi/munin-cgi-graph/([^/]*)) { set $path $1; } if ($path = $remote_user) { set $ok "yes"; } # not allowed here ? redirect them where they should land if ($ok != "yes") { # redirect to where they should be rewrite / /munin-cgi/munin-cgi-graph/$remote_user/ redirect; } fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*); fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_pass unix:/var/run/munin/fastcgi-munin-graph.sock; include fastcgi_params; } location ^~ /munin-cgi/munin-cgi-html/ { # not authenticated => no rewrite (back to auth) if ($remote_user ~ ^$) { break; } # is on the right subtree ? set $ok "no"; # admin can see it all if ($remote_user = 'admin') { set $ok "yes"; } # only allow given path if ($uri ~ /munin-cgi/munin-cgi-html/([^/]*)) { set $path $1; } if ($path = $remote_user) { set $ok "yes"; } # not allowed here ? redirect them where they should land if ($ok != "yes") { # redirect to where they should be rewrite / /munin-cgi/munin-cgi-html/$remote_user/ redirect; } fastcgi_split_path_info ^(/munin-cgi/munin-cgi-html)(.*); fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_pass unix:/var/run/munin/fastcgi-munin-html.sock; include fastcgi_params; } location /munin/static/ { alias /etc/munin/static/; }Nginx was restarted but it's not working, we have a Bad Gateway here:
This appears to be the error:
After restarting munin-fastcgi we now have a text file served containing:
And we now have a 500 error in the Nginx logs:
I removed the authentication checks in the nginx config but that didn't help.
After digging some I found that this fixed it:
So zooming now works and this is a good graph to test it on:
This shows the increased bandwidth during the copying on the movie site:
One question, do you want these graphs to be protected with HTTP Authentication to reduce the load that bots will generate or is the adding of /munin/ to https://penguin.transitionnetwork.org/robots.txt enough?
The following changes were made following http://munin-monitoring.org/wiki/MuninConfigurationMasterCGI
In /etc/munin/munin.conf
Everything seems to be working, I have updated the docs at wiki:PenguinServer#MuninStats
I think this ticket is good to close.