Thursday, May 30, 2013

Size per Table information with MySQL

Knowing the size of your data is of course helpful.  The tools have become easier over the years and different versions of MySQL but it is something you should be checking regardless of your MySQL version.

If you are running an old version of MySQL (before information_schema) then you can still gather this data by using "Show table status and add the Data_length to the index_length." The information_schema makes this much easier but you are free to use them whenever you like.

Take advantage of the pager command to gather just the information you are after.
[world]> pager egrep -h "Data_length|Index_length"
PAGER set to 'egrep -h "Data_length|Index_length"'

Use the show table status command to gather the related information:
[world]> show table status like 'City'\G
Data_length: 409600
Index_length: 131072
1 row in set (0.00 sec)

Reset the pager:
[world]> pager
Default pager wasn't set, using stdout.
Table Size = Data_length + Index_length
[world]> select 409600 + 131072 as Table_Size;
+------------+
| Table_Size |
+------------+
| 540672 |
+------------+

The same information is available via the information_schema:

SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE,SUM(DATA_LENGTH+INDEX_LENGTH) AS size,SUM(INDEX_LENGTH) AS index_size
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA IN ('world') AND TABLE_NAME IN ('City') AND ENGINE IS NOT NULL
GROUP BY TABLE_SCHEMA, TABLE_NAME

TABLE_SCHEMA: world
TABLE_NAME: City
ENGINE: InnoDB
size: 540672
index_size: 131072
1 row in set (0.00 sec)


The point, pay attention and know your data. 

Wednesday, May 29, 2013

MySQL 4.1 -- Please Upgrade

A MySQL DBA is often asked to help with various versions of MySQL.

SELECT VERSION();
+----------------+
| VERSION() |
+----------------+
| 4.1.18-classic |
+----------------+
But I beg you all... Evaluate your options and upgrade. 

MySQL has made numerous SECURITY issues updates let alone performance updates. Check your version of MySQL. If it is anything below 5.5  or at a big stretch 5.1.69 PLEASE UPGRADE. 

While you might consider your database "working" and available for upgrade when it breaks.... Will it require some work... Yes. Will it save you in the long run.. Yes . Will you get more out of your system... Yes...  Could you take advantage of fixed bugs... Yes.   Would you rather be "broken" by a security vulnerability?  

Stop and think for a moment about what you will say to the CEO when the CEO asks why did you get hacked ?

Take a look at your system against the known security vulnerabilities:

4.1

Take advantage of all the new versions available:





Saturday, May 18, 2013

MySQL Users :: Grants :: mysql_config_editor :: Security

Secure access to the database is likely priority number one for any database administrator. If it is not then you need to seriously look into why it is not.

General guidelines via the manual are already available:

One of the primary issues with security in MySQL is of course the permissions that you give users.
These are a few simply guidelines.

First keep "super user" or "root" accounts to a minimum. A user with full access or "GRANT ALL" will still have access when you have reached your max connections. So the last thing you would want is a program to be executing commands with a user with full access.

Keep in mind what types of accounts you are creating. You can limit a user to MAX QUERIES, MAX UPDATES, MAX CONNECTIONS and MAX USER CONNECTIONS per HOUR.

Keep in mind of the network environment your users are connecting from. If users are going to be using DHCP network addresses within the same subnet you would just be creating more work for yourself if you limited them to a single IP. You can still limit them to the subnet though with a wildcard. For example '192.168.0.2' versus '192.168.0.%'

Stay away from entire wildcard access for host and users.



> CREATE USER ''@'192.168.0.56' ;
Query OK, 0 rows affected (0.02 sec)

> show grants for ''@'192.168.0.56';
+-----------------------------------------+
| Grants for @192.168.0.56                |
+-----------------------------------------+
| GRANT USAGE ON *.* TO ''@'192.168.0.56' |
+-----------------------------------------+



This would leave you wide open for anyone from 192.168.0.56 and is not a smart secure thing to do.
It could also violate other accounts from 192.168.0.56 because MySQL checks host first and username second.


> GRANT SELECT ON test.* TO 'exampleuser'@'192.168.0.%' IDENTIFIED BY 'somepassword';

> show grants for 'exampleuser'@'192.168.0.%';
+----------------------------------------------------------------------------------------------------------------------+
| Grants for exampleuser@192.168.0.%                                                                                   |
+----------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'exampleuser'@'192.168.0.%' IDENTIFIED BY PASSWORD '*DAABDB4081CCE333168409A6DB119E18D8EAA073' |
| GRANT SELECT ON `test`.* TO 'exampleuser'@'192.168.0.%'                                                              |
+----------------------------------------------------------------------------------------------------------------------+


This will allow selects only for exampleuser from '192.168.0.%'. You must also keep in mind that if exampleuser is connecting from LOCAL HOST that the system will likely user localhost first before the 192.168.0.% subnet address unless the user used the subnet address the host to connect to.

This means that you can create one user and password with different privileges per host.


> SHOW GRANTS FOR 'exampleuser'@'localhost';
+--------------------------------------------------------------------------------------------------------------------+
| Grants for exampleuser@localhost                                                                                   |
+--------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'exampleuser'@'localhost' IDENTIFIED BY PASSWORD '*DAABDB4081CCE333168409A6DB119E18D8EAA073' |
| GRANT SELECT, UPDATE, DELETE ON `test`.* TO 'exampleuser'@'localhost'                                              |
+--------------------------------------------------------------------------------------------------------------------+


Try your best to not use the --password=<password> option via the mysql client. You can use -p to prompt for a password.

You also have the option with MySQL 5.6 to use the MySQL Configuration Utility.


# mysql_config_editor set  --login-path=local --host=localhost --user=root --password
Enter password:




# mysql_config_editor print --all
[local]
user = root
password = *****
host = localhost


# mysql 
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

# mysql  --login-path=local
Welcome to the MySQL monitor.

# mysql  --login-path=local -e 'SELECT NOW()';


You do have options to name different paths like local or remote and etc as well. So you can encrypt more than one access user account in your ~/.mylogin.cnf file that is created once you use the set command.

If you have shell scripts that use the mysql client and likely then have passwords in the scripts updating them to use the "--login-path=" is a more secure way to go.


Of course when you no longer need a user... Drop the user.


> DROP USER 'exampleuser'@'localhost';



A Smaller IBDATA file


I have seen the desire for a smaller ibdata file come up lately on the forums.mysql.com

The innodb database uses the ibdata file(s) to store the database data to disk. Configuring your system correctly is key and you can learn more about such options here: http://dev.mysql.com/doc/refman/5.6/en/innodb-configuration.html

InnoDB provides an ACID compliant and transaction safe storage engine; it is very productive but if you are deleting and/or replacing data often, you will need to recover the lost space over time. How much time is dependent on your system and use. You cannot run a single command and recover space in an ibdata file. It will take a few steps and it is not a behind the scenes job, unless done on a slave server. If you have a slave it is best to do this work on the slaved database first then plan to rotate that database server to become the master database.

So two different situations; these are not the only solutions but some solutions :

  • You want to keep the ibdata file the same size but you want to just clear out the wasted space
The best way to recover lost space is to dump the data and reload it. Yes, not the first choice for a DBA I know. This is more troublesome the bigger your database is. I would hope you have a slave database and can do this off the slave then make it a master later.

  1. Backup the database 
    1. mysqldump --user=<username> --password=<> --add-drop-database   --master-data=2  --triggers --routines --events --databases (list database names and do not add mysql to this list) > /Just_AN_example/mysqldump_<DATEHERE>_.sql 
      1. This gives you an ASCII copy just in case of binary corruption. 
      2. It also has master data via a comment if needed.
      3. This will keep your mysql authentication in tact as well. 
        1. I would save the mysql database as a dump separately. 
    2. You  also can create a backup with MySQL Enterprise Backup or Percona XtraBackup, if the system was a bigger db and needed online backups these are good choices. Up to you which you use for various reasons. 
  2. Checksum your database. 
    1. Gather some numbers on what you have so you can compare it when you load it back. 
      1. This can be done with Percona Toolkit

        1. # ./pt-table-checksum --password=<Password>   > checksum_before_dump.txt
      2. A query you can write yourself.
        1. I have a blog post on this as well 
          1. http://anothermysqldba.blogspot.com/2013/05/mysql-checksum.html
  3. Stop/Start the database and take advantage of this downtime for any read only variables you would like to adjust 
  4. Load the database back 


  1. Follow steps 1 through 2 in the process listed above. 
  2. In the step 4 of the above process you will want to add the following to your my.cnf file.
    1. innodb_file_format=Barracuda
    2. innodb_file_per_table=1
  3. Remove the ibdata file and logs.
    1. No coming back from this point 
  4. Start the database 
  5. Confirm it is up and running
  6. Load the database from backup. 

This of course would be best to do on a non production/slave server so you can confirm all the steps and get yourself to a workable situation then rotate the slave to be the new master.

MySQL CHECKSUM

CHECKSUM TABLE is useful information when you are checking the status of a table. This is often used before and after a backup and restore to ensure data is intact.

Here is a simple way to use it via the MySQL command line and the tools already available to you.


mysql> CREATE USER 'checksumuser'@'localhost';
mysql>GRANT SELECT ON *.* TO 'checksumuser'@'localhost';


mysql>SELECT
CONCAT('mysql --user=checksumuser -e \'CHECKSUM TABLE ',TABLE_SCHEMA,'.',TABLE_NAME ,' EXTENDED\'; ') as cmd_line_query
INTO OUTFILE '/tmp/checksums.sh'
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA NOT IN ('mysql', 'INFORMATION_SCHEMA')
AND ENGINE IS NOT NULL
GROUP BY TABLE_SCHEMA, TABLE_NAME;

mysql> exit


# chmod +x /tmp/checksums.sh
# /tmp/checksums.sh > /tmp/checksums_b4dump.sql

Now you will have all your checksum data available to you in the file listed. Simple example of the data below


Table Checksum
world.City 2011482258
Table Checksum
world.Country 580721939
Table Checksum
world.CountryLanguage 1546017027


After the dump or process you are running you can then run the same script just change the output file then load it and compare results. It is not the cleanest method but it is an easy fast check for you to do.

# /tmp/checksums.sh > /tmp/checksums_after_dump.sql


mysql> use test
mysql> CREATE TABLE `checksums` (
`checksums_id` int(11) NOT NULL AUTO_INCREMENT,
table_name varchar(100) DEFAULT NULL,
size_a int(11) DEFAULT NULL,
size_b int(11) DEFAULT NULL,
PRIMARY KEY (`checksums_id`)
) ENGINE=InnoDB ;

LOAD DATA INFILE '/tmp/checksums_b4dump.sql'
IGNORE INTO TABLE checksums
(table_name, size_a);

LOAD DATA INFILE '/tmp/checksums_after_dump.sql'
IGNORE INTO TABLE checksums
(table_name, size_b);

DELETE FROM checksums WHERE table_name = 'Table';

SELECT a.table_name , a.size_a, b.size_b
FROM checksums a
INNER JOIN checksums b ON a.table_name = b.table_name and a.checksums_id != b.checksums_id
WHERE a.size_a IS NOT NULL AND b.size_b IS NOT NULL ;
+-----------------------------------------------------------------------+------------+------------+
| table_name | size_a | size_b |
+-----------------------------------------------------------------------+------------+------------+
| world.City | 2011482258 | 2011482258 |
| world.Country | 580721939 | 580721939 |
| world.CountryLanguage | 1546017027 | 1546017027 |
+-----------------------------------------------------------------------+------------+------------+


#mysql -p
mysql> DROP USER 'checksumuser'@'localhost';

DO NOT FORGET TO DROP THE USER when done.