Mysql Cluster: The definitive HOWTO （ndb）

gunguymadman

Mysql Cluster: The definitive HOWTOThis guide was written over a year ago and is an excellentintroduction however it has not been updated since! If you have anyquestions please email me or Buy my book (US).
IntroductionYou MUST have a third server as a managment node but this can beshut down after the cluster starts. Also note that I do not recommendshutting down the managment server (see the extra notes at the bottomof this document for more information). You can not run a mysql cluster with just two servers And have true redundancy.
You should also disable SELinux or use the following rules (many thanks to Robin Bowes for letting me know of these):

allow mysqld_t port_t:tcp_socket name_connect;
allow mysqld_t var_lib_t:file append;
allow mysqld_t var_lib_t:sock_file create;
allow mysqld_t var_lib_t:file read;
allow mysqld_t var_lib_t:sock_file unlink;
allow mysqld_t var_lib_t:file { getattr write };

Although it is possible to set the cluster up on two physicalservers you WILL NOT GET the ability to "kill" one server and for thecluster to continue as normal. For this you need a third server runningthe managment node.
I am going to talk about three servers,

mysql1.domain.com                 192.168.0.1
mysql2.domain.com                 192.168.0.2
mysql3.domain.com                 192.168.0.3

Servers 1 and 2 will be the two that end up "clustered". This wouldbe perfect for two servers behind a loadbalancer or using round robinDNS and is a good replacement for replication. Server 3 needs to haveonly minor changes made to it and does NOT require a mysql install. Itcan be a low-end machine and can be carrying out other tasks.

STAGE 1: Install mysql on the first two servers:Complete the following steps on both mysql1 and mysql2:

groupadd mysql
useradd -g mysql mysql
cd /usr/local/
wget http://dev.mysql.com/get/Downloa ... i686-glibc23.tar.gz
        /from/http://mirror.trouble-free.net/mysql_mirror/
tar -zxvf mysql-max-5.0.15-linux-i686-glibc23.tar.gz
rm mysql-max-5.0.15-linux-i686-glibc23.tar.gz
ln -s mysql-max-5.0.15-linux-i686-glibc23 mysql
cd mysql
scripts/mysql_install_db --user=mysql
chown -R root  .
chown -R mysql data
chgrp -R mysql .
cp support-files/mysql.server /etc/rc.d/init.d/
chmod +x /etc/rc.d/init.d/mysql.server
chkconfig --add mysql.server

Do not start mysql yet.
STAGE 2: Install and configure the managment serverYou need the following files from the bin/ of the mysql directory:ndb_mgm and ndb_mgmd. Download the whole mysql-max tarball and extractthem from the bin/ directory.

mkdir /usr/src/mysql-mgm
cd /usr/src/mysql-mgm
http://dev.mysql.com/get/Downloads/MySQL-4.1/mysql-max-5.0.15-pc-linux-gnu-i686.tar.gz/
        from/http://www.signal42.com/mirrors/mysql/
tar -zxvf mysql-max-5.0.15-pc-linux-gnu-i686.tar.gz
rm mysql-max-5.0.15-pc-linux-gnu-i686.tar.gz
cd mysql-max-5.0.15-pc-linux-gnu-i686
mv bin/ndb_mgm .
mv bin/ndb_mgmd .
chmod +x ndb_mg*
mv ndb_mg* /usr/bin/
cd
rm -rf /usr/src/mysql-mgm

You now need to set up the config file for this managment:

mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
vi [or emacs or any other editor] config.ini

Now, insert the following (changing the bits as indicated):

[NDBD DEFAULT]
NoOfReplicas=2
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Managment Server
[NDB_MGMD]
# the IP of THIS SERVER
HostName=192.168.0.3
# Storage Engines
[NDBD]
# the IP of the FIRST SERVER (STORAGE NODE)
HostName=192.168.0.1
DataDir= /var/lib/mysql-cluster
[NDBD]
# the IP of the SECOND SERVER (STORAGE NODE)
HostName=192.168.0.2
DataDir=/var/lib/mysql-cluster
# 2 MySQL Clients
# I personally leave this blank to allow rapid changes of the mysql clients;
# you can enter the hostnames of the above two servers here. I suggest you dont.
[MYSQLD]
[MYSQLD]

Now, start the managment server:

ndb_mgmd

This is the mysql managment server, not maganment console. Youshould therefore not expect any output (we will start the consolelater).
STAGE 3: Configure the storage/SQL servers and start mysqlOn each of the two storage/SQL servers (192.168.0.1 and 192.168.0.2) enter the following (changing the bits as appropriate):

vi /etc/my.cnf

Enter i to go to insert mode again and insert this on both servers(changing the IP address to the IP of the managment server that you setup in stage 2):

[mysqld]
ndbcluster
# the IP of the MANAGMENT (THIRD) SERVER
ndb-connectstring=192.168.0.3
[mysql_cluster]
# the IP of the MANAGMENT (THIRD) SERVER
ndb-connectstring=192.168.0.3

Be aware that anything in the [mysql_cluster] section will overridethe defaults in [mysql], so if you introduce a nodeid and then try torun multiple daemons on the same machines you should be aware of this!
Now, we make the data directory and start the storage engine:

mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
/usr/local/mysql/bin/ndbd --initial
/etc/rc.d/init.d/mysql.server start

If you have done one server now go back to the start of stage 3 and repeat exactly the same procedure on the second server.
NOTEthat you should ONLY use --initial if you are either starting fromscratch or have changed the config.ini file on the managment.
STAGE 4: Check its workingYou can now return to the managment server (mysql3) and enter the managment console:

/usr/local/mysql/bin/ndb_mgm

Enter the command SHOW to see what is going on. A sample output looks like this:

[root@mysql3 mysql-cluster]# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @192.168.0.1  (Version: 5.0.15, Nodegroup: 0, Master)
id=3    @192.168.0.2  (Version: 5.0.15, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.0.3  (Version: 5.0.15)

[mysqld(API)]   2 node(s)
id=4   (Version: 5.0.15)
id=5   (Version: 5.0.15)

ndb_mgm>

If you see
not connected, accepting connect from 192.168.0.[1/2/3]in the first or last two lines they you have a problem. Please emailme with as much detail as you can give and I can try to find out whereyou have gone wrong and change this HOWTO to fix it.
If you are OK to here it is time to test mysql. On either servermysql1 or mysql2 enter the following commands: Note that we have noroot password yet.

mysql
use test;
CREATE TABLE ctest (i INT) ENGINE=NDBCLUSTER;
INSERT INTO ctest () VALUES (1);
SELECT * FROM ctest;

You should see 1 row returned (with the value 1).
If this works, now go to the other server and run the same SELECTand see what you get. Insert from that host and go back to host 1 andsee if it works. If it works then congratulations.
The final test is to kill one server to see what happens. If youhave physical access to the machine simply unplug its network cable andsee if the other server keeps on going fine (try the SELECT query). Ifyou dont have physical access do the following:

ps aux | grep ndbd

You get an output like this:

root      5578  0.0  0.3  6220 1964 ?        S    03:14   0:00 ndbd
root      5579  0.0 20.4 492072 102828 ?     R    03:14   0:04 ndbd
root     23532  0.0  0.1  3680  684 pts/1    S    07:59   0:00 grep ndbd

In this case ignore the command "grep ndbd" (the last line) but killthe first two processes by issuing the command kill -9 pid pid:

kill -9 5578 5579

Then try the select on the other server. While you are at it run aSHOW command on the managment node to see that the server has died. Torestart it, just issue

ndbd

NOTE no
--inital!Further notes about setupI strongly recommend that you read all of this (and bookmark this page). It will almost certainly save you a lot of searching.
The Managment ServerI strongly recommend that you do not stop the managment server once it has started. This is for several resons:

• The server takes hardly any server resources
• If acluster falls over, you want to be able to just ssh in and type ndbd tostat it. You dont want to have to start messing around with anotherserver
• If you want to take backups then you need the managment server up
• Thecluster log is sent to the management server so to check what is goingon in the cluster or has happened since last this is an important tool
• All commands from the ndb_mgm client is sent to the management server and thus no management commands without management server.
• Themanagment server is required in case of cluster reconfiguration(crashed server or network split). In the case that it is not running,"split-brain" scenario will occure. The management server arbitrationrole is required for this type of setup to provide better faulttollerance.

However you are welcome to stop the server if you prefer.
Starting and stopping ndbd automatically on bootTo achieve this, do the following on both mysql1 and mysql2:

echo "ndbd" > /etc/rc.d/init.d/ndbd
chmod +x /etc/rc.d/init.d/ndbd
chkconfig --add ndbd

Note that this is a really quick script. You ought really to writeone that at least checks if ndbd is already started on the machine.
Use of hostnamesYou will note that I have used IP addresses exclusively throughtthis setup. This is because using hostnames simply increases the numberof things that can go wrong. Mikael Ronström of MySQL AB kindly explains:"Hostnames certainly work with MySQL Cluster. But using hostnamesintroduces quite a few error sources since a proper DNS lookup systemmust be set-up, sometimes /etc/hosts must be edited and their might besecurity blocks ensuring that communication between certain machines isnot possible other than on certain ports". I strongly suggest that while testing you use IP addresses if you can, then once it is all working change to hostnames.
RAMUse the following formula to work out the amount of RAM that you need on each storage node:

(Size of database * NumberofReplicas * 1.1) / Number of storage nodes

NumberofReplicas is set to two by default. You can change it inconfig.ini if you want. So for example to run a 4GB database withNoOfReplicas set to 2 you need just under 9GB of RAM in total (4 * 2 *1.1), so if you had two storage nodes you would need 4.5GB ram perstorage node. For the SQL nodes and managment nodes you dont need muchRAM at all.Bear in mind that if you have variable-width fields in MySQLCluster 4.0 or 5.0 you will find that you will need a LOT more RAM thanthis formula predicts.
Note: A lot of people have emailed me querying themaths above! Remember that the cluster is fault tollerant, and eachpiece of data is stored on at least 2 nodes. (2 by default, as set byNumberOfReplicas). So you need TWICE the space you would need just forone copy, multiplied by 1.1 for overhead.
Adding storage nodesIf you decide to add storage nodes, bear in mind that 3 is not anoptimal numbers. If you are going to move from two (above) then move to4.
Adding SQL nodesIf you want to add another SQL node (i.e. you have another serverthat you want to add to the cluster but you dont need it to act as astorage node), then just add the following to /etc/my.cnf on the server(it must be a mysql-max server):

[mysqld]
ndbcluster
# the IP of the MANAGMENT (THIRD) SERVER
ndb-connectstring=192.168.0.3
[mysql_cluster]
# the IP of the MANAGMENT (THIRD) SERVER
ndb-connectstring=192.168.0.3

Then you need to make sure that there is another [MYSQLD] line atthe end of config.ini on the managment server. Restart the cluster (seebelow for an important note) and restart mysql on the new API. Itshould be connected.
Important note on changing config.iniIf you ever change config.ini you must stop thewhole cluster and restart it to re-read the config file. Stop thecluster with a SHUTDOWN command to the ndb_mgm package on the managmentserver and then restart all the storage nodes.
Some useful configuration options that you will need if you have large tables:
DataMemory: defines the space available to storethe actual records in the database. The entire DataMemory will beallocated in memory so it is important that the machine contains enoughmemory to handle the DataMemory size. Note that  DataMemory is also used to store ordered indexes. Ordered indexes uses about 10 bytes per record. Default: 80MB
IndexMemory The IndexMemory is the parameter thatcontrols the amount of storage used for hash indexes in MySQL Cluster.Hash indexes are always used for primary key indexes, unique indexes,and unique constraints. Default: 18MB
MaxNoOfAttributes  This parameter defines the number of attributes that can be defined in the cluster. Default: 1000
MaxNoOfTables Obvious (bear in mind that each BLOB field creates another table for various reasons so take this into account). Default: 128
View this page at mysql.com for further information about the things you can put in the [NDBD] section of config.ini
A note about securityMySQL cluster is not secure. By default anyone can connect to your managment server and shut the whole thing down. I suggest the following precautions:

• Install APF and block all ports except those you use (do NOTinclude any MySQL cluster ports). Add the IPs of your cluster machinesto the /etc/apf/allow_hosts file.
• Run MySQL cluster over a second network card on a second, isolated, network.

Other resourcesI found the following resources very useful:

• The MySQL cluster documentation. This is gradually being reworked and is really quite good now.
  • On-line Backup of MySQL Cluster.
  • Defining MySQL Cluster Storage Nodes for information that you will need to allow for bigger database memory or a larger number of tables, indexes, unique indexes
         
• MySQL Cluster mailing list.
• Google.
• MySQL Forums
• The #mysql IRC chanel on freenode and EFNet. If you need a free (open source) IRC client I recomment Bersirc.