Showing posts with label MySQL Sandbox. Show all posts
Showing posts with label MySQL Sandbox. Show all posts

Sunday, March 15, 2020

MySQL & Dockers...a simple set up

MySQL & Dockers... are not new concepts,  people have been moving to Dockers for some time now.  For someone who is just moving to this for development, it can have a few hurdles.

While MySQL works just fine running locally, if you are testing code across different versions of MySQL it is nice to have several versions easily available.

One option for years has been of course https://mysqlsandbox.net/ by Giuseppe Maxia.  This is a very valid solution to be able to get several instances up and test replication and etc etc.

Dockers are now also another often used scenario when it comes to testing across different versions of MySQL. The following will just go over some of the steps to get several versions installed easily. I use OSX so these examples are for OSX.

You need Docker to start and of course and Docker Desktop is a handy tool for you to be able to get access easily.

Once I had Docker set up I can get my environment ready for MySQL. 

Here I created a Docker folder that contains the MySQL data directories, Config files as well as the mysql-files directory if I needed it. 

mkdir ~/Docker ;

mkdir ~/Docker/mysql_data;
mkdir ~/Docker/mysql-files;
mkdir ~/Docker/cnf;

Now inside mysql_data


cd  ~/Docker/mysql_data;
mkdir 8.0;
mkdir 5.7;
mkdir 5.6;
mkdir 5.5;


Now I set up simple cnf files for this example. The primary thing to note is the bind-address. This is set to ensure it is opened up for us to reach MySQL outside of the docker.  You can also notice that these files can be used to set up additional configuration information as you see fit per MySQL docker instance. 



cd  ~/Docker/cnf;

cat my.8.0.cnf
[mysqld]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= /var/lib/mysql-files
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
bind-address = 0.0.0.0
port=3306
server-id=80


# Custom config should go here
!includedir /etc/mysql/conf.d/

 cat my.5.7.cnf
[mysqld]
bind-address = 0.0.0.0
server-id=57
max_allowed_packet=32M

$ cat my.5.6.cnf
[mysqld]
bind-address = 0.0.0.0
server-id=56

$ cat my.5.5.cnf
[mysqld]
bind-address = 0.0.0.0
server-id=55


OK so now that we have configuration files set up, We need to build the dockers. A few things to note for the build commands. 

--name   We set a named reference for the docker. 

Here we are mapping the configuration files, data directory and mysql-files directories to the docker . This allows us to adjust the my.cnf file and etc easily. 
-v ~/Docker/cnf/my.8.0.cnf:/etc/mysql/my.cnf 
-v ~/Docker/mysql_data/8.0:/var/lib/mysql
-v ~/Docker/mysql-files:/var/lib/mysql-files

We want to be able to reach these MySQL instances outside of the docker so we need to publish and map the port accordingly. 
-p  3306:3306  This means 3306 local to 3306 inside docker
-p  3307:3306  This means 3307 local to 3306 inside docker
-p  3308:3306  This means 3308 local to 3306 inside docker
-p  3309:3306  This means 3309 local to 3306 inside docker

Then we also pass a couple of environment variables. 
-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=<set a password here>

So putting it all together...


docker run --restart always --name mysql8.0   -v ~/Docker/cnf/my.8.0.cnf:/etc/mysql/my.cnf -v ~/Docker/mysql_data/8.0:/var/lib/mysql -v ~/Docker/mysql-files:/var/lib/mysql-files -p  3306:3306 -d -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=<set a password here> mysql:8.0

docker run --restart always --name mysql5.7   -v ~/Docker/cnf/my.5.7.cnf:/etc/mysql/my.cnf -v ~/Docker/mysql_data/5.7:/var/lib/mysql -v ~/Docker/mysql-files:/var/lib/mysql-files -p  3307:3306 -d -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=<set a password here> mysql:5.7

docker run --restart always --name mysql5.6   -v ~/Docker/cnf/my.5.6.cnf:/etc/mysql/my.cnf -v ~/Docker/mysql_data/5.6:/var/lib/mysql -v ~/Docker/mysql-files:/var/lib/mysql-files -p  3308:3306 -d -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=<set a password here> mysql:5.6

docker run --restart always --name mysql5.5   -v ~/Docker/cnf/my.5.5.cnf:/etc/mysql/my.cnf -v ~/Docker/mysql_data/5.5:/var/lib/mysql -v ~/Docker/mysql-files:/var/lib/mysql-files -p  3309:3306 -d -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=<set a password here> mysql:5.5

After each execution of the above commands, you should get an id returned. 
example: 3cb07d7c21476fbf298648986208f3429ec664167d8eef7fed17bf9ee3ce6316

You can start/restart and access each docker terminal easily via the Docker Desktop or just keep note of the related IDs and you execute via the terminal.

The Docker Desktop also shows you all the variables you passed so you can validate. 
You can of course also access the CLI here, stop and start or destroy it easily. 



$ docker exec -it 3cb07d7c21476fbf298648986208f3429ec664167d8eef7fed17bf9ee3ce6316 /bin/sh; exit
# mysql -p 

If the Docker container is already running you can now access MySQL via your localhost terminal.

$ mysql --host=localhost  --protocol=tcp --port=3306 -p -u root 

Now if you are having any access issues remember to ensure that MySQL accounts are correct and that your ports and mapping correctly. 
  • Lost connection to MySQL server at 'reading initial communication packet'
  • ERROR 1045 (28000): Access denied for user 'root'@'192.168.0.5' (using password: YES)

Now you can see that all are up and available and the server Ids match what we set per cnf file eariler.

$ mysql --host=localhost --protocol=tcp --port=3306 -e "Select @@hostname, @@version, @@server_id "
+--------------+-----------+-------------+
| @@hostname | @@version | @@server_id |
+--------------+-----------+-------------+
| 58e9663afe8d | 8.0.19 | 80 |
+--------------+-----------+-------------+
$ mysql --host=localhost --protocol=tcp --port=3307 -e "Select @@hostname, @@version, @@server_id "
+--------------+-----------+-------------+
| @@hostname | @@version | @@server_id |
+--------------+-----------+-------------+
| b240917f051a | 5.7.29 | 57 |
+--------------+-----------+-------------+
$ mysql --host=localhost --protocol=tcp --port=3308 -e "Select @@hostname, @@version, @@server_id "
+--------------+-----------+-------------+
| @@hostname | @@version | @@server_id |
+--------------+-----------+-------------+
| b4653850cfe9 | 5.6.47 | 56 |
+--------------+-----------+-------------+
$ mysql --host=localhost --protocol=tcp --port=3309 -e "Select @@hostname, @@version, @@server_id "
+--------------+-----------+-------------+
| @@hostname | @@version | @@server_id |
+--------------+-----------+-------------+
| 22e169004583 | 5.5.62 | 55 |
+--------------+-----------+-------------+


Thursday, May 9, 2013

Comparing the Databases :: MySQL :: Percona :: MariaDB with the MySQL Sandbox

Do you often find yourself curious yet to busy to explore?

Often people stay with what they are comfortable with and started with. MySQL has a very loyal following of users. It is ok to be curious and explore the forks as well. The MySQL Sandbox makes it extremely easy for you to do just that.

MySQL Sandbox is a great tool and the documentation is available if you need help.
http://mysqlsandbox.net/docs.html. The MySQL Sandbox also allows you to check against forks and different versions of the MySQL server easily.

I already have a MySQL 5.6 version running on my server.


$ mysql -p
Enter password:
Server version: 5.6.10-log MySQL Community Server (GPL)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.




I used the MySQL Sandbox to compare three similar versions of MySQL:
  • mariadb-5.5.30
  • Percona-Server-5.5.30
  • mysql-5.5.31
I decided make my sandboxes this way:
$ make_multiple_custom_sandbox mariadb-5.5.30-linux-i686.tar.gz Percona-Server-5.5.30-rel30.2-500.Linux.i686.tar.gz mysql-5.5.31-linux2.6-i686.tar.gz


node1]$ ./use
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 5.5.30-MariaDB-log MariaDB Server

Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.
> select VERSION();
+--------------------+
| VERSION()          |
+--------------------+
| 5.5.30-MariaDB-log |
+--------------------+

  • Ok this is as expected.




node2]$ ./use
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4

Server version: 5.5.30-MariaDB-log MariaDB Server

Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others
> select VERSION();
+--------------------+
| VERSION()          |
+--------------------+
| 5.5.30-MariaDB-log |
+--------------------+
  • Ah? ok not as expected.


node3]$ ./use
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
 > select VERSION();
+------------+
| VERSION()  |
+------------+
| 5.5.31-log |
+------------+


I expected or at was at least hopeful it was going result with this.


I still think that MySQL Sandbox is a great tool and makes some comparisons and tests very easy to do. I just happen to run into a conflict when trying to run all 3 versions at once. It could completely be my error as well and I have just overlooked something. I think it would be a valid test for some people to want to do though. Regardless I moved and built from source instead. More on that is available here:





Saturday, May 4, 2013

Building from Source MySQL :: MariaDB :: Percona

It is possible to run more than one MySQL server on the same server. At times people might like to install another version of a database on the same hardware for testing purposes, as well as evaluations. 
Installing the databases from source and custom installations for each is easier than it might sound to some.  I would suggest to review MySQL Sandbox  first though, because it allows for evaluations and testing to be done very quickly and easily. However, installing from source worked out better for me when I did some comparisons.  Below is the process I used. I will be looking to build out future blog posts with these databases once I tune then configurations. 



This is the default information from mysql.com. I already had MySQL installed so I did not execute the following but I wanted this here for reference.  You can compare these steps to the MySQL, MariaDB and Percona source installations below too see how I updated the default steps in order to get all three versions of the database running on the same box. (No production value here was done just to test the process.)

# Preconfiguration setup
shell> groupadd mysql
shell> useradd -r -g mysql mysql

# Beginning of source-build specific instructions
shell> tar zxvf mysql-VERSION.tar.gz
shell> cd mysql-VERSION
shell> cmake .
shell> make
shell> make install
# End of source-build specific instructions

# Postinstallation setup
shell> cd /usr/local/mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> scripts/mysql_install_db --user=mysql
shell> chown -R root .
shell> chown -R mysql data

# Next command is optional
shell> cp support-files/my-medium.cnf /etc/my.cnf
shell> bin/mysqld_safe --user=mysql &

# Next command is optional
shell> cp support-files/mysql.server /etc/init.d/mysql.server



If you prefer to use the mysql.server script for start and stop make sure to review and edit accordingly.
# Preconfiguration setup
shell> groupadd oracle_mysql
shell> useradd -r -g 
oracle_mysql oracle_mysql

# Beginning of source-build specific instructions
shell> tar zxvf MariaDB
-VERSION.tar.gz
shell> cd MySQL-VERSION
shell> cmake .
shell> make
shell> make install DESTDIR="
/usr/local/
oracle_mysql-tmp"
# End of source-build specific instructions

I do not like the results
-- Installing: /usr/local/oracle_mysql-tmp/usr/local/mysql/
If DESTDIR is should install into that location not start with user under that location. This is a MySQL original issue as it does this with all versions of MySQL.

# Fix the odd/bug setup
shell> cd /usr/local/
oracle_mysql-tmp
shell> mv usr/local/mysql/ ../oracle_mysql ; 
shell> cd ../; # rm -Rf oracle_mysql-tmp

# Postinstallation setup
shell> cd /usr/local/oracle_mysql
shell> chown -R 
oracle_mysql .
shell> chgrp -R oracle_mysql .

# Next command is optional
shell> cp support-files/my-small.cnf /etc/
oracle_mysql.cnf
shell> vi /etc/oracle_mysql.cnf

       port            = 3309

       socket          = /tmp/oracle_mysql.sock

shell> scripts/mysql_install_db --defaults-file=/etc/oracle_mysql.cnf --basedir=/usr/local/oracle_mysql --skip-name-resolve --datadir=/var/lib/oracle_mysql --user=oracle_mysql
shell> chown -R oracle_mysql /var/lib/oracle_mysql/*

shell> 
# bin/mysqld_safe --defaults-file=/etc/oracle_mysql.cnf --user=oracle_mysql  --datadir=/var/lib/oracle_mysql/ --port=3309 & 


shell> # bin/mysql --port=3309 --socket=/tmp/oracle_mysql.sock 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.31 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.








# Preconfiguration setup
shell> groupadd mariadb
shell> useradd -r -g 
mariadb mariadb

# Beginning of source-build specific instructions
shell> tar zxvf MariaDB
-VERSION.tar.gz
shell> cd MariaDB-VERSION
shell> cmake .
shell> make
shell> make install DESTDIR="
/usr/local/mariadb-tmp"
# End of source-build specific instructions

I do not like the results
-- Installing: /usr/local/mariadb-tmp/usr/local/mysql/
If DESTDIR is should install into that location not start with user under that location. This is a MySQL original issue as it does this with all versions of MySQL.

# Fix the odd/bug setup
shell> cd /usr/local/
mariadb-tmp
shell> mv usr/local/mysql/ ../mariadb ; 
shell> cd ../; # rm -Rf mariadb-tmp

# Postinstallation setup
shell> cd /usr/local/mariadb
shell> chown -R 
mariadb .
shell> chgrp -R mariadb .


# Next command is optional
shell> cp support-files/my-small.cnf /etc/
mariadb.cnf

shell> vi /etc/mariadb.cnf

       port            = 3308

       socket          = /tmp/mariadb.sock

shell> scripts/mysql_install_db --defaults-file=/etc/mariadb.cnf --basedir=/usr/local/mariadb --skip-name-resolve --datadir=/var/lib/mariadb --user=mariadb
shell> chown -R mariadb /var/lib/mariadb/*

shell> 
# bin/mysqld_safe --defaults-file=/etc/mariadb.cnf --user=mariadb  --datadir=/var/lib/mariadb/ --port=3308 & 


shell> # bin/mysql --port=3308 --socket=/tmp/mariadb.sock 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1
Server version: 5.5.30-MariaDB Source distribution

Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.






# Preconfiguration setup
shell> groupadd percona
shell> useradd -r -g 
percona percona

# Beginning of source-build specific instructions
shell> tar zxvf Percona-Server
-VERSION.tar.gz
shell> cd Percona-Server-VERSION
shell> cmake .
shell> make
shell> make install DESTDIR="
/usr/local/percona-tmp"
# End of source-build specific instructions

I do not like the results

-- Installing: /usr/local/percona-tmp/usr/local/mysql/

If DESTDIR is should install into that location not start with user under that location. This is a MySQL original issue as it does this with all versions of MySQL.

# Fix the odd/bug setup
shell> cd /usr/local/percona-tmp

shell> mv usr/local/mysql/ ../percona ; 

shell> cd ../; # rm -Rf percona-tmp



# Next command is optional
shell> cp support-files/my-small.cnf /etc/
percona.cnf

shell> vi /etc/percona.cnf

       port            = 3307

       socket          = /tmp/percona.sock




# Postinstallation setup
shell> cd /usr/local/percona
shell> chown -R 
percona .
shell> chgrp -R percona .

shell> scripts/mysql_install_db --defaults-file=/etc/percona.cnf --basedir=/usr/local/percona --skip-name-resolve --datadir=/var/lib/percona --user=percona 
shell> chown -R percona /var/lib/percona/*
shell> # bin/mysqld_safe --defaults-file=/etc/percona.cnf --user=percona  --datadir=/var/lib/percona/ --port=3307 



shell> # bin/mysql --port=3307 --socket=/tmp/percona.sock 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.30 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.




Now I have access to all 3 flavors or MySQL. 
For easy client access I added this to my .bashrc file:

  • alias percona='/usr/local/percona/bin/mysql --port=3307 --socket=/tmp/percona.sock'
  • alias oracle_mysql='/usr/local/oracle_mysql/bin/mysql --port=3309 --socket=/tmp/oracle_mysql.sock'
  • alias maria='/usr/local/mariadb/bin/mysql --port=3308 --socket=/tmp/mariadb.sock'






Monday, April 29, 2013

"Tools" of the trade

I figured it might be worth creating a list of the top tools of the trade that we all use.
First and foremost this is to say thanks to everyone who helps create these tools.
Second it is to allow others who do not use these to see and learn how these can be used and why.
  • MySQL command line client. 
    • This is a given I know but it is the best access to MySQL. 
    • mysql -p --prompt="\u@\h [\d]>\_ Master >"
  • Xtrabackup
    • ./xtrabackup --defaults-file=/etc/my.cnf --backup --stats --target-dir=~/backups/  --prepare  --export --user=root --innodb_data_home_dir=/var/lib/mysql/ --innodb_data_file_path=/var/lib/mysql/ 
  • mysqlbinlog 
  • mysqltuner 
    • A good overall view into what you could have going on with your database. 
  • Percona Toolkit 
    • exampes:
      • pt-query-digest 
        • example: pt-query-digest --ask-pass /var/lib/mysql/mysql-slow.log
      • pt-table-checksum
        • example: pt-table-checksum --ask-pass
      • pt-table-sync
        • example: pt-table-sync --no-check-triggers --ask-pass --no-check-triggers --execute --print
      • pt-show-grants 
        • pt-show-grants  --ask-pass
  • MySQL Utilities 
    • exampes:
      • python mysqldiskusage --server=root:password@localhost
      • python mysqlindexcheck --server= root:password@localhost ps_helper.schema_index_statistics
      • python mysqlprocgrep --server= root:password@localhost --match-user=root
  • planet.mysql.com 
    • This might make some people question but... Knowledge is power. This is the best location to learn about MySQL. 
    • If you have to interview a MySQL DBA candidate, pick some common active authors and ask if they know who that person is. It will allow you to understand if they research the latest trends and information around MySQL or are just content with their experience.  
  • MySQL Sandbox 
  • mytop 
    • inspired by top but with a focus on MySQL
  • innotop
    • innotop is a 'top' clone for MySQL 
  • https://tools.percona.com/
    • This might also make some question. I add this because it is a quick and good way for people to get started and create a my.cnf that is better than the default installed version. If nothing else you can use it just to compare with what you think it should be. 
You might notice I am not a fan of the GUI. No offense to those GUI tools available but why GUI if you do not need it. But that is just my opinion and here is a list of GUI tools just to be fair. After all schema map is helpful and easily done with MySQL Workbench. 

Other tools for review: