一、说明
因为电脑只能开四个虚拟机,于是使用4台虚拟机模拟12台主机。
如下图所示:
图解:
1、四台虚拟机均使用 CentOS 6.5 系统;
2、前端使用 keepalived给haproxy作高可用,keepalived为双主模型,前端两主机互为主从,两虚拟IP为:172.16.36.100、172.16.36.200;
3、前端haproxy给后端的web服务器作负载均衡;
4、前端DNS用于keepalived两虚拟IP作轮询解析域名,域名为:www.wubin.com;
5、事实上,web服务器分为三组,一组用于存储图片,第二组用于存储静态文本,第三组用于存储动态程序;此处只使用两台web服务器实现,使用基于端口的各虚拟主机模拟各web服务器;
6、于是web服务器1中有五个虚拟主机,webserver_img是用于存储图片的虚拟主机,webserver_txt是用于存储静态文件的虚拟主机,webserver_dynamic是用于存储动态程序的虚拟主机;web服务器2中有一个虚拟主机webserver_dynamic;各同类别的虚拟主机互相作负载均衡集群;
7、PHP使用fast-cgi方式安装,两web服务器各安装一个;
8、mysql数据作主从复制,使用mysql-proxy对mysql数据库作读写分离;
各软件版本:
1、keepalived:keepalived-1.2.7-3.el6.x86_64,yum 安装
2、haproxy: haproxy-1.4.24-2.el6.x86_64,yum安装
3、bind:bind-9.8.2-0.17.rc1.el6_4.6.x86_64,yum安装
4、nginx:nginx-1.4.7,编译安装
5、PHP:php-5.4.26,编译安装fast-cgi方式
6、mysql:mariadb-10.0.10-linux-x86_64,二进制版本
7、mysql-proxy:mysql-proxy-0.8.3-linux-glibc2.3-x86-64bit,编译安装
二、实现
1、操作前提条件:
1)、使用任务计划将各主机的时间同步至ntp服务器
# vim /etc/crontab
*/3 * * * * /usr/sbin/ntpdate 172.16.0.1 &> /dev/null
2)、各主机配置IP与主机名
前端一: ha1.wubin.com 172.16.36.10
前端二: ha2.wubin.com 172.16.36.20
web1服务器: web1.wubin.com 172.16.36.30
web2服务器:web2.wubin.com 172.16.36.40
3)、给各虚拟机安装软件
2、实现步骤
1)、配置web服务
配置web1服务器上的虚拟主机, 配置/etc/nginx/nginx.conf文件,配置5台虚拟主机,对应图片中的名字为: webserver_img*2、 webserver_txt *2 、 webserver_dynamic ;
#user nobody;worker_processes 1;#error_log logs/error.log;#error_log logs/error.log notice;#error_log logs/error.log info;#pid logs/nginx.pid;events { worker_connections 1024;}http { include mime.types;default_type application/octet-stream;#log_format main '$remote_addr - $remote_user [$time_local] "$request" 'log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '# '$status $body_bytes_sent "$http_referer" '# '"$http_user_agent" "$http_x_forwarded_for"';#access_log logs/access.log main;sendfile on;#tcp_nopush on;#keepalive_timeout 0;keepalive_timeout 65;gzip on;server { listen 10010;server_name img1.wubin.com;#charset koi8-r;access_log /var/web/log/img1.wubin.com.access.log main;location / { root /var/web/htdocs/img.wubin.com;index index.html index.htm;}error_page 404/404.html;# redirect server error pages to the static page /50x.html#error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/img.wubin.com;}location =/404.html { root /var/web/htdocs/img.wubin.com;}# proxy the PHP scripts to Apache listening on 127.0.0.1:80##location ~ \.php$ { # proxy_pass http://127.0.0.1;#}# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000##location ~ \.php$ { # root html;# fastcgi_pass 127.0.0.1:9000;# fastcgi_index index.php;# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;# include fastcgi_params;#}# deny access to .htaccess files, if Apache's document root# concurs with nginx's one##location ~ /\.ht { # deny all;#}}server { listen 10020;server_name img2.wubin.com;#charset koi8-r;access_log /var/web/log/img2.wubin.com.access.log main;location / { root /var/web/htdocs/img.wubin.com;index index.html index.htm;}error_page 404/404.html;# redirect server error pages to the static page /50x.html#error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/img.wubin.com;}location =/404.html { root /var/web/htdocs/img.wubin.com;}# proxy the PHP scripts to Apache listening on 127.0.0.1:80##location ~ \.php$ { # proxy_pass http://127.0.0.1;#}# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000##location ~ \.php$ { # root html;# fastcgi_pass 127.0.0.1:9000;# fastcgi_index index.php;# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;# include fastcgi_params;#}# deny access to .htaccess files, if Apache's document root# concurs with nginx's one##location ~ /\.ht { # deny all;#}}server { listen 10030;server_name txt1.wubin.com;access_log /var/web/log/txt1.wubin.com.access.log main;location / { root /var/web/htdocs/txt.wubin.com;index index.html index.htm;}error_page 404/404.html;error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/txt.wubin.com;}location =/404.html { root /var/web/htdocs/txt.wubin.com;}}server { listen 10040;server_name txt2.wubin.com;access_log /var/web/log/txt2.wubin.com.access.log main;location / { root /var/web/htdocs/txt.wubin.com;index index.html index.htm;}error_page 404/404.html;error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/txt.wubin.com;}location =/404.html { root /var/web/htdocs/txt.wubin.com;}}server { listen 10050;server_name dynamic.wubin.com;access_log /var/web/log/dynamic.wubin.com.access.log main;location / { root /var/web/htdocs/dynamic.wubin.com;index index.php index.html index.htm;}location ~ \.php$ { root /var/web/htdocs/dynamic.wubin.com;fastcgi_pass 127.0.0.1:9000;fastcgi_index index.php;fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;include fastcgi_params;}error_page 404/404.html;error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/dynamic.wubin.com;}location =/404.html { root /var/web/htdocs/dynamic.wubin.com;}}}
配置web2服务器上的web服务,一台虚拟主机 webserver_dynamic;
#user nobody;worker_processes 1;#error_log logs/error.log;#error_log logs/error.log notice;#error_log logs/error.log info;#pid logs/nginx.pid;events { worker_connections 1024;}http { include mime.types;default_type application/octet-stream;#log_format main '$remote_addr - $remote_user [$time_local] "$request" 'log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '# '$status $body_bytes_sent "$http_referer" '# '"$http_user_agent" "$http_x_forwarded_for"';#access_log logs/access.log main;sendfile on;#tcp_nopush on;#keepalive_timeout 0;keepalive_timeout 65;#gzip on;server { listen 80;server_name localhost;#charset koi8-r;#access_log logs/host.access.log main;location / { root html;index index.html index.htm;}#error_page 404 /404.html;# redirect server error pages to the static page /50x.html#error_page 500502503504/50x.html;location =/50x.html { root html;}# proxy the PHP scripts to Apache listening on 127.0.0.1:80##location ~ \.php$ { # proxy_pass http://127.0.0.1;#}# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000##location ~ \.php$ { # root html;# fastcgi_pass 127.0.0.1:9000;# fastcgi_index index.php;# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;# include fastcgi_params;#}# deny access to .htaccess files, if Apache's document root# concurs with nginx's one##location ~ /\.ht { # deny all;#}}server { listen 10050;server_name dynamic.wubin.com;access_log /var/web/log/dynamic.wubin.com.access.log main;location / { root /var/web/htdocs/dynamic.wubin.com;index index.php index.html index.htm;}location ~ \.php$ { root /var/web/htdocs/dynamic.wubin.com;fastcgi_pass 127.0.0.1:9000;fastcgi_index index.php;fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;include fastcgi_params;}error_page 404/404.html;error_page 500502503504/50x.html;location =/50x.html { root /var/web/htdocs/dynamic.wubin.com;}location =/404.html { root /var/web/htdocs/dynamic.wubin.com;}}}
2)、配置PHP
使用基于 fast-cgi方式安装PHP,安装完成后, 编辑/etc/nginx/fastcgi_params,将其内容更改为如下内容:
fastcgi_param GATEWAY_INTERFACE CGI/1.1;fastcgi_param SERVER_SOFTWARE nginx;fastcgi_param QUERY_STRING $query_string;fastcgi_param REQUEST_METHOD $request_method;fastcgi_param CONTENT_TYPE $content_type;fastcgi_param CONTENT_LENGTH $content_length;fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;fastcgi_param SCRIPT_NAME $fastcgi_script_name;fastcgi_param REQUEST_URI $request_uri;fastcgi_param DOCUMENT_URI $document_uri;fastcgi_param DOCUMENT_ROOT $document_root;fastcgi_param SERVER_PROTOCOL $server_protocol;fastcgi_param REMOTE_ADDR $remote_addr;fastcgi_param REMOTE_PORT $remote_port;fastcgi_param SERVER_ADDR $server_addr;fastcgi_param SERVER_PORT $server_port;fastcgi_param SERVER_NAME $server_name;
而后重新载入nginx的配置文件:
# service nginx reload
3)、配置mysql主从;
主服务器配置,服务器IP为172.16.36.30:
# service mysqld stop# vim /etc/my.cnf log-bin=/mydata/binlog/master-bin //开启二进制日志 binlog_format=mixed server-id =1# service mysqld start# mysqlmysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repluser'@'172.16.%.%' IDENTIFIED BY 'replpass';Query OK, 0 rows affected (0.00 sec) //创建复制帐号mysql> FLUSH PRIVILEGES;Query OK, 0 rows affected (0.00 sec)
从服务器配置 ,服务器IP为172.16.36.40 :
# service mysqld stop# vim /etc/my.cnf#log-bin=/mydata/binlog/master-bin //关闭二进制日志#binlog_format=mixedserver-id =11//设定 server-idrelay-log =/mydata/relaylogs/slave-bin //开启中继日志# mkdir /mydata/relaylogs# chown -R mysql.mysql /mydata/relaylogs# service mysqld start# mysqlmysql> show global variables like '%relay%'; relay_log |/mydata/relaylogs/slave-binmysql> change master to master_host='172.16.36.30',master_user='repluser',master_password='replpass';Query OK, 0 rows affected (0.07 sec)mysql> show slave status\G //查看从服务器状态 Slave_IO_Running: No //表示IO线程没有启动Slave_SQL_Running: No //表示SQL线程没有启动mysql> start slave; //手动启动两个线程Query OK, 0 rows affected (0.00 sec)mysql> show slave status\G
4)、配置mysql-proxy
编译安装完成mysql-proxy后, 为服务脚本提供配置文件/etc/sysconfig/mysql-proxy,内容如下所示:
# Options for mysql-proxy ADMIN_USER="admin"ADMIN_PASSWORD="admin"ADMIN_ADDRESS=""ADMIN_LUA_SCRIPT="/usr/local/mysql-proxy/share/doc/mysql-proxy/admin.lua"PROXY_ADDRESS=""PROXY_USER="mysql-proxy"PROXY_OPTIONS="--daemon --log-level=info --log-use-syslog --plugins=proxy --plugins=admin --proxy-backend-addresses=172.16.36.30:3309 --proxy-read-only-backend-addresses=172.16.36.40:3306 --proxy-lua-script=/usr/local/mysql-proxy/share/doc/mysql-proxy/rw-splitting.lua"
复制如下内容建立admin.lua文件,将其保存至/usr/local/mysql-proxy/share/doc/mysql-proxy/目录中。
--[[ $%BEGINLICENSE%$ Copyright (c) 2007, 2012, Oracle and/or its affiliates. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; ifnot, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA02110-1301 USA $%ENDLICENSE%$ --]]function set_error(errmsg) proxy.response = { type = proxy.MYSQLD_PACKET_ERR,errmsg = errmsg or"error"}endfunction read_query(packet)if packet:byte() ~= proxy.COM_QUERY thenset_error("[admin] we only handle text-based queries (COM_QUERY)")return proxy.PROXY_SEND_RESULTendlocal query = packet:sub(2)local rows = { }local fields = { }if query:lower() =="select * from backends"thenfields = { { name ="backend_ndx", type = proxy.MYSQL_TYPE_LONG },{ name ="address", type = proxy.MYSQL_TYPE_STRING },{ name ="state", type = proxy.MYSQL_TYPE_STRING },{ name ="type", type = proxy.MYSQL_TYPE_STRING },{ name ="uuid", type = proxy.MYSQL_TYPE_STRING },{ name ="connected_clients", type = proxy.MYSQL_TYPE_LONG },}for i =1, #proxy.global.backends dolocal states = { "unknown","up","down"}local types = { "unknown","rw","ro"}local b = proxy.global.backends[i]rows[#rows + 1] = { i,b.dst.name, -- configured backend addressstates[b.state +1], -- the C-id is pushed down starting at 0types[b.type +1], -- the C-id is pushed down starting at 0b.uuid, -- the MySQL Server's UUID if it is managedb.connected_clients -- currently connected clients}endelseif query:lower() == "select * from help" thenfields = { { name = "command", type = proxy.MYSQL_TYPE_STRING },{ name = "description", type = proxy.MYSQL_TYPE_STRING },}rows[#rows + 1] = { "SELECT * FROM help", "shows this help" }rows[#rows + 1] = { "SELECT * FROM backends", "lists the backends and their state" }elseset_error("use 'SELECT * FROM help' to see the supported commands")return proxy.PROXY_SEND_RESULTendproxy.response = { type = proxy.MYSQLD_PACKET_OK,resultset = { fields = fields,rows = rows}}return proxy.PROXY_SEND_RESULTend
启动服务
[root@web1 ~]# service mysql-proxy start
查看mysql
[root@web1 ~]# mysql -uadmin -padmin -h172.16.36.30 --port=4041Welcome to the MariaDB monitor. Commands end with ; or \g.Your MySQL connection id is 1Server version:5.0.99-agent-adminCopyright (c) 2000, 2014, Oracle, SkySQL Ab and others.Type 'help;'or'\h'for help. Type '\c' to clear the current input statement.MySQL [(none)]> select * from backends;+-------------+-------------------+---------+------+------+-------------------+| backend_ndx | address | state | type | uuid | connected_clients |+-------------+-------------------+---------+------+------+-------------------+|1|172.16.36.30:3309| up | rw | NULL |1||2|172.16.36.40:3306| unknown | ro | NULL |0|+-------------+-------------------+---------+------+------+-------------------+2 rows in set (0.00 sec)MySQL [(none)]>
5)、配置haproxy,给后端web服务器实现负载均衡,前端两主机配置文件一样,配置文件为/etc/haproxy/haproxy.cfg;
#---------------------------------------------------------------------# Example configuration for a possible web application. See the# full configuration options online.## http://haproxy.1wt.eu/download/1.4/doc/configuration.txt##---------------------------------------------------------------------#---------------------------------------------------------------------# Global settings#---------------------------------------------------------------------global# to have these messages end up in /var/log/haproxy.log you will# need to:## 1) configure syslog to accept network log events. This is done# by adding the '-r' option to the SYSLOGD_OPTIONS in# /etc/sysconfig/syslog## 2) configure local2 events to go to the /var/log/haproxy.log# file. A line like the following can be added to# /etc/sysconfig/syslog## local2.* /var/log/haproxy.log#log 127.0.0.1 local2chroot /var/lib/haproxypidfile /var/run/haproxy.pidmaxconn 4000user haproxygroup haproxydaemon# turn on stats unix socketstats socket /var/lib/haproxy/stats#---------------------------------------------------------------------# common defaults that all the 'listen' and 'backend' sections will# use if not designated in their block#---------------------------------------------------------------------defaultsmode httplog globaloption httplogoption dontlognulloption http-server-closeoption forwardfor except 127.0.0.0/8option redispatchretries 3timeout http-request 10stimeout queue 1mtimeout connect 10stimeout client 1mtimeout server 1mtimeout http-keep-alive 10stimeout check 10smaxconn 3000#---------------------------------------------------------------------# main frontend which proxys to the backends#---------------------------------------------------------------------frontend websrvs bind *:80acl url_img path_beg -i /img /imgs /p_w_picpath /p_w_picpathsacl url_img path_end -i .jpg .gif .png .jpegacl host_img hdr_beg(host) -i img. p_w_picpaths.acl url_txt path_beg -i /css /javascript /js /stylesheets acl url_txt path_end -i .css .js .html .htm .shtml .xmlacl host_static hdr_beg(host) -i img. video. download. ftp. imgs. p_w_picpaths. videos.acl url_php path_end -i .phpuse_backend imgserver if url_img or host_imguse_backend txtserver if url_txtuse_backend dynamicserver if url_phpdefault_backend dynamicserver#---------------------------------------------------------------------# static backend for serving up p_w_picpaths, stylesheets and such#---------------------------------------------------------------------backend imgserverbalance roundrobinserver img1.wubin.com 172.16.36.30:10010 checkserver img2.wubin.com 172.16.36.30:10020 checkbackend txtserverbalance roundrobinserver txt1.wubin.com 172.16.36.30:10030 checkserver txt2.wubin.com 172.16.36.30:10040 checkbackend dynamicservercookie node insert nocachebalance roundrobinserver dynamic1.wubin.com 172.16.36.30:10050 check rise 2 fall 5 cookie node1server dynamic2.wubin.com 172.16.36.40:10050 check rise 2 fall 5 cookie node2#---------------------------------------------------------------------# round robin balancing between the various backends#---------------------------------------------------------------------#backend dynamic# balance roundrobin# server node2.wubin.com 192.168.0.12:80 check maxconn 1000listen statisticsbind *:8009stats enablestats auth admin:adminstats uri /hastatusstats admin if TRUEstats hide-version
6)、配置keepalived,给前端haproxy作高可用集群,配置文件为/etc/keepalived/keepalived.conf;两VIP可以实现当haproxy服务下线时VIP自动转移至另一主机;
主机:172.16.36.10
! Configuration File for keepalivedglobal_defs { notification_email { root@localhost } notification_email_from admin@wubin.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL}vrrp_script chk_haproxy { script "killall -0 haproxy"interval 1weight 2} vrrp_instance VI_1 { state MASTERinterface eth0virtual_router_id 36priority 100advert_int 1authentication { auth_type PASSauth_pass 111136}virtual_ipaddress { 172.16.36.100}track_script { chk_haproxy}}vrrp_instance VI_2 { state BACKUPinterface eth0virtual_router_id 136priority 99advert_int 1authentication { auth_type PASSauth_pass 1111136}virtual_ipaddress { 172.16.36.200}track_script { chk_haproxy}}
主机:172.16.36.20
! Configuration File for keepalivedglobal_defs { notification_email { root@localhost } notification_email_from admin@wubin.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL}vrrp_script chk_haproxy { script "killall -0 haproxy"interval 1weight 2}vrrp_instance VI_1 { state BACKUPinterface eth0virtual_router_id 36priority 99advert_int 1authentication { auth_type PASSauth_pass 111136}virtual_ipaddress { 172.16.36.100}track_script { chk_haproxy}}vrrp_instance VI_2 { state MASTERinterface eth0virtual_router_id 136priority 100advert_int 1authentication { auth_type PASSauth_pass 1111136}virtual_ipaddress { 172.16.36.200}track_script { chk_haproxy}}
7)、配置DNS服务器,172.16.36.20;
配置文件/etc/named.conf:
//// named.conf//// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS// server as a caching only nameserver (as a localhost DNS resolver only).//// See /usr/share/doc/bind*/sample/for example named configuration files.//options { // listen-on port 53 { 127.0.0.1; };// listen-on-v6 port 53 { ::1; };directory "/var/named";dump-file "/var/named/data/cache_dump.db";statistics-file "/var/named/data/named_stats.txt";memstatistics-file "/var/named/data/named_mem_stats.txt";// allow-query { localhost; };// recursion yes;// dnssec-enable yes;// dnssec-validation yes;// dnssec-lookaside auto;/* Path to ISC DLV key */// bindkeys-file "/etc/named.iscdlv.key";// managed-keys-directory "/var/named/dynamic";};logging { channel default_debug { file "data/named.run";severity dynamic;};};zone "." IN { type hint;file "named.ca";};include "/etc/named.rfc1912.zones";#include "/etc/named.root.key";
配置文件/etc/named.rfc1912.zones
// named.rfc1912.zones://// Provided by Red Hat caching-nameserver package //// ISC BIND named zone configuration for zones recommended by// RFC 1912 section 4.1: localhost TLDs and address zones//and http://www.ietf.org/internet-drafts/draft-ietf-dnsop-default-local-zones-02.txt// (c)2007 R W Franks//// See /usr/share/doc/bind*/sample/for example named configuration files.//zone "localhost.localdomain" IN { type master;file "named.localhost";allow-update { none; };};zone "localhost" IN { type master;file "named.localhost";allow-update { none; };};zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN { type master;file "named.loopback";allow-update { none; };};zone "1.0.0.127.in-addr.arpa" IN { type master;file "named.loopback";allow-update { none; };};zone "0.in-addr.arpa" IN { type master;file "named.empty";allow-update { none; };};zone "wubin.com" IN { type master;file "wubin.com.zone";};
配置文件/var/named/wubin.com.zone
$TTL 600@ IN SOA dns.wubin.com dnsadmin.wubin.com (20140305042H4M1D2D )@ IN NS dns.wubin.com.@ IN MX 10 mail.wubin.com.dns IN A 172.16.36.20mail IN A 172.16.36.20www IN A 172.16.36.100www IN A 172.16.36.200
测试:
[root@ha2 ~]# dig -t A www.wubin.com @172.16.36.20; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6<<>>-t A www.wubin.com @172.16.36.20;; global options:+cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:7045;; flags: qr aa rd ra; QUERY:1, ANSWER:2, AUTHORITY:1, ADDITIONAL:1;; QUESTION SECTION:;www.wubin.com. IN A;; ANSWER SECTION:www.wubin.com. 600 IN A 172.16.36.200www.wubin.com. 600 IN A 172.16.36.100;; AUTHORITY SECTION:wubin.com. 600 IN NS dns.wubin.com.;; ADDITIONAL SECTION:dns.wubin.com. 600 IN A 172.16.36.20;; Query time:2 msec;; SERVER:172.16.36.20#53(172.16.36.20);; WHEN: Sun May 422:57:462014;; MSG SIZE rcvd:97
配置完成,打开一个windows xp系统,将DNS服务器的IP地址设成刚才配置的DNS服务器的IP地址;
打开浏览器安装程序,测试;
三、此集群不足之处
虽然各同类虚拟主机使用的是一个相同的目录实现的文件同步,但动静分离时,明显web文件需在所有虚拟主机目录存放一份;