Slicehost Articles - Home

Slicehost Maintenance and Upgrades FAQ

2012-03-20T21:20:00Z

As part of our ongoing effort to provide you with the best Slicehost service possible, we routinely perform maintenance and upgrades of our underlying systems. The majority of these are performed non-disruptively, however maintenances sometimes arise that impact customer slices.

If you have slices that are affected, you will receive notification of the specific slices impacted, steps required, and relevant dates. The purpose of this page is to provide more detailed information and answer common questions.

What does this mean for me in one sentence?

Some slices need to be migrated so the underlying hosts (the physical servers that slices run on) can be updated and those migrations require a slice reboot.

Do I need to do anything?

You only need to take action if you have slices that are impacted (you will receive an email maintenance notification if you do) and wish to control when certain slices are migrated. If you wish to control your migrations, please see below for details on how to perform an automated self-migration. If you do not self-migrate, Rackspace will automatically migrate slices on your behalf.

Why are migrations required?

Maintenance and upgrades are routinely performed on Slicehost systems, the majority of which are done transparently and with no impact to individual slices. In some cases however, the host itself needs to be updated which renders it unable to run slices for the duration of the update. Since the downtime associated with a slice migration is less than that of a host update, migrating slices is the least impactful way of deploying the update.

Can I postpone or opt out?

Managed migrations cannot be postponed nor can affected slices be opted out. Automated self-migrations are provided to allow you to control individual slice migrations at your convenience, within the provided self-migration window.

Will migrations change my IP addresses?

No, all IP addresses associated with your slices will remain the same.

How do I perform an automated self-migration?

Simply use the SliceManager or Slicehost API to perform a soft reboot of the affected slice. The first time this is done after you are notified of a need to migrate, your slice will be migrated, then rebooted, instead of just rebooted. Please note that there will be a delay while your slice data is copied to the new host before the reboot occurs. Once migrated, any subsequent reboots of that slice will result in a traditional reboot.

Please note that rebooting from within your slice (e.g. via SSH or the console) will NOT initiate an automated self-migration.

How is a migration performed?

While the slice is still running, a disk snapshot is transparently taken and the data copied to another host. During this time, the slice is still up and operational and any writes to disk are being tracked. After the base snapshot data has been fully copied to the new host, the slice is gracefully shut down, changes that have occurred since the beginning of the snapshot are synchronized, and the slice is booted. Albeit disruptive, the migration process minimizes downtime to a reboot plus the time it takes to copy over delta snapshot changes.

How long will my slice be down during a migration?

Actual slice downtime should not be much longer than a reboot, however, the length of time will vary based on the amount of data that changes during the snapshot process. If you are performing an automated self-migration, it is recommended that you minimize writes to disk (e.g. stop applications, shutdown databases, etc.) as much as possible before starting the migration.

Will a migration affect the configuration of my slice?

Migrated slices will remain in the same datacenter and will retain the same IP addresses, applications and data, slice IDs, etc.

How do I know when a slice has been successfully migrated?

If you are performing an automated self-migration, after initiating a reboot, the slice “status” field in the SliceManager and Slicehost API will transition from “Queued for move” to “Preparing for move” to “Moving” and back to “Active” in lieu of the traditional reboot status changes. This will allow you to monitor the migration progress. Once your slice has transitioned through these statuses and has returned to “Active”, the migration is complete and your slice should be back online. For both self and managed migrations, you will also receive email confirmation for each slice that is successfully migrated. Once you have confirmation that a migration is complete, please take a moment to verify proper functioning of your slice and applications. If you have any questions or problems during or after a migration, please contact our support team immediately so we can investigate.

What about my other slices not listed in the maintenance notification?

Existing slices not listed in your maintenance notification as well as any new slices you launch do not require a migration.

What happens if I don’t self-migrate?

You will be given a period of time to self-migrate which will be clearly indicated in your notification of the need to migrate affected slices. Once the automated self-migration window passes, managed migrations will begin being scheduled for any slices that have not been self-migrated. You will be notified at least 24 hours in advance of any managed migrations and will be provided a specific time window in which the migration will occur. Please note that managed migrations may be spread out over a number of weeks and may also result in migrations occurring at inconvenient times or multiple of your slices being migrated at the same time. As such, please be sure to perform automated self-migrations for any slices where this may be problematic.

Will migrations be required again in the future?

While we always seek to perform non-disruptive maintenance and upgrades of underlying systems, it may be necessary in the future for various slices to be migrated or rebooted.

What should I do if I still have questions?

If you have any further questions, please don’t hesitate to contact our support team directly via email at support@slicehost.com or visit us in chat at https://chat.slicehost.com.

STL to ORD Migration FAQ

2012-03-01T21:01:00Z

This forum post details how to migrate from the St. Louis (STL) datacenter to the Chicago (ORD) datacenter. This post outlines the following:

• Migration windows period

• Importance of self migration

• Migration process overview and step-by-step migration process

• Frequently-Asked Questions

MIGRATION WINDOWS PERIOD

As previously communicated, all Slices in the St. Louis (STL) datacenter will be migrated to the Rackspace Chicago (ORD) datacenter. Migration windows will open January 16, 2012 and will continue throughout the first quarter of 2012 until all customers are successfully migrated. Migration windows will vary for each customer. You will receive an email the day before your migration window opens.

IMPORTANCE OF SELF-MIGRATION

Migrating your own Slice(s) allows you to be in control of when your Slice migrations take place. Additionally, it will minimize the impact to your service or business. This allows you to prepare for changes to both your Slice's private and, if applicable, public IP addresses.

If you do not self-migrate your Slice(s) at the end of your migration window, your Slice(s) will automatically be migrated.

For all Slices, the migration will result in a private IP change. If both your public and private IP must change, however, it is in your best interest to self-migrate since you will need to update your DNS entries to your new IP address. Please note that only slices in STL-A with IP addresses in the range 208.75.84.0 - 208.75.87.255 will have to change their public IP addresses. All other slices in STL-A and STL-B will not have to change their public IP addresses.

MIGRATION PROCESS OVERVIEW

You will receive an email the day before your migration window opens. You will be given a two-week migration window for all your Slices. Additionally, you will be able to choose when each Slice is migrated within that window.

In preparation of migrating your Slice(s) take a few minutes to review this Migration Best Practices article.

After you receive your open migration window announcement you will need to log into SliceManager and click on the "DataCenter migration" link. This will take you to the Slice Migration page, where you can select the Slice(s) that you would like to have migrated. After you have migrated a Slice, you will receive an email requesting you to confirm that your new Slice is working properly. You will then verify it has been properly migrated in the SliceManager. Once verified any backups or snapshots associated with the Slice are copied to ORD.

Once your last Slice has been migrated, a final email will be sent to you indicating completion of your migration.

The datacenter migration process will work much like the Slice resize process. A new Slice will be created in ORD and your data will be synchronized to the new Slice from the old Slice. Your old Slice will be shut down and any differential data changes will be synchronized as a final step before finalizing your new Slice. The difference from the current Slice resize process is that instead of changing Slice flavors you will be changing datacenter locations.

MIGRATION STEP-BY-STEP PROCESS

Step 1. Locate the new "DataCenter Migration" link on the left hand side of SliceManager once your migration window opens.

Step 2. Once you click the "DataCenter Migration" link you will see a list of all your Slices that are eligible for migration. Both your previous IP address and your new IP address will be listed for each Slice.

Step 3. Select your Slice(s) by checking the check box for the Slices you would like to have migrated.

Step 4. Click the "Migrate" button to begin the migration process.

Step 5. You will see that your Slice(s) have moved from the "Slices to Migrate from St. Louis to Chicago"" to the "Slices Migrated to Chicago" section. Here, you will be able to watch the status of your Slice through the migration process real time (you will need to refresh your browser).

Step 6. After your migration has completed you will receive an email asking you to validate that your Slice(s) migrated successfully. Check to see if your Slice(s) is working properly.

Step 7. Click the "Verify" button under the "Action" header to confirm that your Slice(s) is working properly. If you encounter problems with your Slice(s) click the "Revert" button and your Slice(s) will rollback to the original Slice in STL.

PLEASE NOTE: If you do not verify that your Slice(s) has been successfully migrated within 24 hours, it will be auto-verified. If after auto-verification you need to revert your Slice you will need to contact Slicehost Support to assist with reverting your Slice back to STL.

Step 8. After clicking either the "Verify" or the "Revert" button you will see a confirmation box asking you to confirm your decision.

Verify Confirmation Box:

Revert Confirmation Box:

Step 9. After verifying your successful Slice(s) migration, the status of your Slice(s) will change.

Step 10. After your successful migration, your Slice(s) are Active and have the new IP address listed.

FAQs

Q: Will there be down time associated with this migration?

A: Yes, you will incur down time while the differential data updates and gets synchronized to your new Slice. The datacenter migration process will work much like the Slice resize process. A new Slice will be created in ORD and your data will be synchronized to the new Slice from the old Slice. Your old Slice will be shut down and any differential data changes will be synchronized as a final step before finalizing your new Slice. The difference from the current Slice resize process is that instead of changing Slice flavors you will be changing datacenter locations.

Q: What happens to IP addresses?

A: Your public IP address may not change, however, your private IP address will change. You will be able to view your current and new IP addresses in the SliceManager once the migration window opens. Once your Slice is migrated, the public IP address will automatically point to the new Slice. For customers currently leveraging private IP addresses for inter-slice communication, you may temporarily shift to using public IPs BEFORE the migration. This will enable individual Slices to be migrated and still be able to communicate with other Slices. Once all Slices have been migrated, you can shift communication back to using the new private IPs.

Please note that only slices in STL-A with IP addresses in the range 208.75.84.0 - 208.75.87.255 will have to change their public IP addresses. All other slices in STL-A and STL-B will not have to change their public IP addresses.

Q: What happens if something goes wrong?

A: If you notice errors or the Slice isn't working properly there will be a "Revert" button for you to select. Choosing to revert will reactivate your Slice in the STL datacenter. Also remember we are here to help, so don't hesitate to start a chat or submit a ticket if you need additional assistance.

Q: How will communication be handled prior to the migration windows?

A: The day before your migration window opens you will receive an email that will include the exact start and end date of your migration window. You will also see a new "DataCenter Migration" link in SliceManager. This page will list all Slices that will be eligible for migration. These Slices will list the current public and private IP addresses and the new public and private IP addresses that will be used during the migration. For most customers the public IP address will be the same for current and new.

Q: What happens if I do not migrate my Slice(s) during this two-week window?

A: If you do not migrate your Slice during your two-week window, it will be automatically migrated for you on the last day of your migration window. If the Slice is not working properly, you must submit a ticket to have the migration rolled back and your STL Slice reactivated.

Thanks again for your understanding and feedback during these changes. If you have any questions or concerns please contact us at support@slicehost.com.

STL to ORD Migration Best Practices

2012-01-13T22:10:00Z

Customers with slices in our St. Louis datacenters (STL-A and STL-B) will be migrated to our Chicago datacenter (ORD) as part of the transition from Slicehost to Rackspace Cloud. Customers will be notified in advance of the slice migration, giving them time to optimize their slices for the move.

The migration

At its most basic the migration is a copy operation in two phases.

Phase 1

The files on the original slice are copied to the new server. During this phase the slice is running and available.

Phase 2

The slice is brought down. The system then runs a sync operation so any files that changed during the first pass will be re-copied to the new server instance.

After these two phases complete the new server instance will boot and become available.

Mitigation

Review the contents of your slice to look for ways to reduce the length of each phase of the migration.

You can shorten phase 1 by eliminating unneeded files from the original slice, like archived logs and application cache files. You can also reduce the size of files to be copied by weeding old entries out of database tables.

You can reduce the length of phase 2, and thus your downtime, by keeping the number and size of files that will change during phase 1 to a minimum. With less to copy the new server will be ready that much sooner.

Advance preparation

Some mitigation tasks can be performed well in advance of the migration.

Establish a log rotation policy

Large log files can potentially increase the time for the initial copy and increase downtime during the final sync. Application and virtual host logs aren't generally rotated unless you specifically set up rotation for them.

You can use a utility like logrotate to make sure your logs stay at a reasonable size.

Prune and archive old data

Look for archived log files, application cache files, and old tables or entries in databases. The more you can remove the less there will be to copy to the new server.

Ruby on Rails applications in particular can create a large number of session files and never delete them. Before the migration make sure you look into where your apps might store those session and cache files.

For more advice on locating and pruning old session and cache files, see this article on preparing for slice resizes (the process behind the scenes is very similar to the migration process).

Check for large files

Removing other large files from the slice beforehand can also cut down on the overall copy time.

You can look for the largest files on your system using the find utility:

sudo find / -mount -type f -ls|sort -rnk7 |head -30|awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

The output will show the largest files on your slice so you can determine if any can be deleted or archived elsewhere.

Right before the migration

You can reduce downtime during the final sync operation by taking steps to reduce the number of files that will change during the first copy.

Use public IP addresses

Telling services that talk to other slices to use the public IP addresses or domain names of the target servers can help minimize the effects of a multiple-slice migration.

The migration will change the private IP addresses of your slices and switch them to a new private (backend) network. While the migration progresses, using the public network interfaces for slice communication will let those slices talk to each other even as they're brought up in the new datacenter.

We know customers often use private IPs for communications between slices to minimize bandwidth charges. With that in mind we will not charge for bandwidth overages for slices scheduled for migration.

Once the migration completes you can switch services to the new private IP addresses.

Force log rotations

Applications that are still running during the migration may generate new log entries. When the slice is brought down for the final data sync those changed log files will have to be re-synced. That can prolong your downtime.

If you force a log rotation beforehand you can ensure that any changed log files will be relatively small. Most log rotation utilities provide a means of forcing rotations manually.

For example, if you're using logrotate to manage your application logs you can force a rotation with the command:

sudo logrotate -f /etc/logrotate.conf

Lock databases

Databases that change during the first copy will have to be recopied in phase 2. Since database files tend to be on the large side this can noticeably extend the length of the final sync.

It's a lot safer to bring the database down entirely for the migration. If that isn't practical, however, the second-best approach is to make your tables read-only so they won't be written to during the copy.

To lock your tables in mysql run the command:

mysql -u root -p --execute="FLUSH TABLES WITH READ LOCK"

Flush application caches

Clearing out old, unneeded session and cache files can prevent them from dragging a migration to a crawl.

For each file on a server the sync process has to perform an operation to look at each file, to copy it, and to confirm its copy at the end. That means syncing a lot of very small cache or data files can take more time than syncing one file of the same size as all of them put together.

After the migration

When the migration completes and your new server boots you should test your web sites and applications. Make sure applications are responsive and that they can write information to their databases.

If you have any services that need to communicate to other servers, at this point you can switch them to use the private (backend) IP addresses for those servers instead of their public addresses.

Install Nagios3 on Debian 5.0 Lenny

2011-10-20T00:12:00Z

We show how to configure nginx on a 32-bit slice to serve dynamically-created content. Along the way, we show off a couple of troubleshooting techniques and how to work around the lack of a pre-compiled fastcgi package for Debian 5.0 Lenny.

This article demonstrates how to configure nginx on a 32-bit slice to serve dynamically-created content. Nginx first-timers will also find it shows how to link an nginx configuration to the static pages provided by applications like Nagios - a task usually automated during Nagios installs - but only for Apache web-servers.

We show how to test each newly-installed component of the installation is working properly as we install - a good practice that will save us from having to trace through the entire installation if we discover any problems at the end.

Finally, our choice of Debian 5.0 Lenny enables us to illustrate how to overcome the issue of there being no pre-compiled fcgi package in standard repositories for Debian 5.0 Lenny. You can export our work-around to any distribution that lacks an fcgi binary.

First, install nginx and set it to start on boot:

sudo aptitude install nginx
sudo update-rc.d nginx defaults

Test browse to the server's external IP and check you see nginx's default “It works!” message. If it doesn't, the problem is probably that the server's built-in firewall is blocking the incoming HTTP request.

You can confirm that by testing if nginx's listener is visibly running on the server. Do that by listing the TCP listeners now running on the slice and using grep to filter for the port 80 listener. Do that by issuing:

netstat -pantl | egrep ':80'

You should see a result something like:

nagios:~# netstat -pantl | egrep ':80'
tcp     0     0 0.0.0.0:80     0.0.0.0:*     LISTEN     16689/nginx

We know nginx is listening - ie responding - on port 80, so we can test if it is able to see our incoming requests. If we use netcat to make a test connection to nginx's listener. We should see an “open” response as below:

nagios:~# nc -v localhost 80
localhost [127.0.0.1] 80 (www) open

If we see nothing, we can be pretty sure a firewall - iptables - is blocking the connection attempt. Iptables configuration is explained elsewhere so let's assume that we do see nginx listening and we do see its default web page.

Install Nagios. The default Nagios3 package pulls in some Samba and WINS-related monitoring features that will ask you to configure a domain. If you have no WINS domain, just accept the defaults:

sudo aptitude install nagios3

Start Nagios:

sudo /etc/init.d/nginx start

And set Nagios to start on boot:

sudo update-rc.d nagios3 defaults

Uninstall apache2 if the Nagios install includes apache2 as a dependency:

sudo aptitude remove apache2

Configure Nagios access permissions. Not having Apache installed, we use Perl to generate htpasswd credentials. To generate passwords, run:

perl -le 'print crypt("<password>", "salt")'

and copy the encrypted password it generates. Using your favorite text editor, write the ID and password in the file:

/etc/nagios3/htpasswd.users

in this colon-separated format:

nagiosadmin:<encrypted>:'My comment here'

Configure nginx

First, optimise nginx for performance and security:

sudo vim /etc/nginx/nginx.conf

/etc/nginx/nginx.conf

    user www-data;
    worker_processes  4;

    error_log  /var/log/nginx/error.log;
    pid        /var/run/nginx.pid;

    events {
        worker_connections  1024;
    }

    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        access_log  /var/log/nginx/access.log;

        sendfile        on;
        tcp_nopush     on;

        keepalive_timeout  3;

        gzip  on;
        gzip_comp_level 2;
        gzip_proxied any;
        gzip_types      text/plain text/html text/css application/x-javascript text/xml
                    application/xml application/xml+rss text/javascript;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
    }

Second, let's assume this nginx installation will only serve Nagios so we need add no new vhosts. Instead, we replace the default server installation with a new server that is configured to point to Nagios's various static and dynamic components.

sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/orig.default
sudo vim /etc/nginx/sites-available/default

The Nagios install left its static pages at:

* /usr/share/nagios3/htdocs
* /usr/share/nagios3/stylesheets

and its cgi files at:

* /usr/lib/cgi-bin/nagios3

We use nginx's location directive to point nginx's root to the static pages, and include two sets of rewrite commands to correctly point URIs requests to the stylesheets and Nagios' cgi files. To do that, edit /etc/nginx/sites-available/default as below:

        server {
            server_name    nagios.demo.com
            listen 80;
            add_header      Cache-Control public;
            access_log      /var/log/nginx/nagios.demo.com_access.log;
            error_log       /var/log/nginx/nagios.demo.com_error.log info;
            expires         31d;

            location / {
                root /usr/share/nagios3/htdocs;
                index index.html;

                # A rewrite serves URI requests to the stylesheets
                rewrite ^/nagios3/stylesheets/(.*)$ /../stylesheets/$1 break;
                rewrite ^/nagios3/(.*)$ /$1 break;
            }

            location ~ \.cgi$ {
                root /usr/lib/cgi-bin/nagios3;
                include /etc/nginx/fastcgi_params;

                # Another rewrite serves URI requests to the cgi files
                rewrite ^/cgi-bin/nagios3/(.*)$ /$1;

                auth_basic "Restricted";
                auth_basic_user_file /etc/nagios3/htpasswd.users;

                fastcgi_pass 127.0.0.1:8998;
                fastcgi_param SCRIPT_FILENAME /usr/lib/cgi-bin/nagios3$fastcgi_script_name;
                fastcgi_param AUTH_USER     $remote_user;
                fastcgi_param REMOTE_USER   $remote_user;
            }
        } # end server

We can test that this nginx configuration is able to serve Nagios' static pages by browsing to the root URL of our nginx server. We should find that Nagios' static pages display correctly to present us with Nagios' lefthand menu bar. However, attempts to click on the links in the menu produce a “502 Bad Gateway” error. That's because nginx is not yet able to find a working cgi-handler to handle Nagios' dynamically-generated pages. We'll fix that in a moment.

If nginx is not able to serve Nagios' static pages at this point, check that the correct static page locations were provided correctly in the /etc/nginx/sites-available/default configuration.

Also, check the server-specific logs at:

* /var/log/nagios.demo.com_error.log
* /var/log/nagios.demo.com_access.log

and nginx's default logs at:

* /var/log/nginx/error.log
* /var/log/nginx/access.log

They should provide clues to what nginx is doing when errors occur. You can 'tail -f' each log while browsing the site to see in real-time what is logged as you test-click different links.

If Nagios' CGI processes are to work, we need to link nginx up to a cgi handler,which we did in the line in the above nginx configuration file that read:

fastcgi_pass 127.0.0.1:8998;

So we need to install a FastCGI handler that will listen on port 8998. We'll take the fcgi handler that is bundled with Lighttpd, wrap it with a script and start it on boot.

Step 1: Install spawn-fcgi:

sudo aptitude install lighttpd
cp /usr/bin/spawn-fcgi* /tmp/
sudo aptitude remove lighttpd
sudo mv /tmp/spawn-fcgi.lighttpd /usr/bin/spawn-fcgi

Now install fcgiwrap to run fcgi the way we configured nginx to use it.

Install the fcgiwrap tarball directly from its author's page at: http://nginx.localdomain.pl/wiki/FcgiWrap

Or follow the instructions at: http://wiki.nginx.org/Fcgiwrap

sudo aptitude install git-core build-essential libfcgi-dev autoconf libtool automake

cd /usr/local/src/
sudo git clone git://github.com/gnosek/fcgiwrap.git

cd /usr/local/src/fcgiwrap
sudo autoreconf -i
sudo ./configure
sudo make
sudo mv fcgiwrap /usr/local/bin/

Now we need a script that starts fcgi with the the parameters we configured nginx to expect. So, in /usr/bin/, create a script called fcgi.sh:

sudo vim /usr/bin/fcgi.sh
sudo chmod 755 /usr/bin/fcgi.sh

The /usr/bin/fcgi.sh script should contain:

    #!/bin/sh
    # /usr/bin/fcgi.sh
    /usr/bin/spawn-fcgi -f /usr/local/bin/fcgiwrap -a 127.0.0.1 -p 8998 -P /var/run/fastcgi-c.pid \
    -u www-data -g www-data

We can start fcgi with this fcgi.sh script to test it and we can set it to start or stop using a System V-style /etc/init.d/ start-up script at boot:

sudo vim /etc/init.d/fcgi

That start-up script should contain:

    #!/bin/bash
    FCGI_SCRIPT=/usr/bin/fcgi.sh
    RETVAL=0
    case "$1" in
        start)
            echo "Starting fastcgi"
            $FCGI_SCRIPT
            RETVAL=$?
      ;;
        stop)
            echo "Stopping fastcgi"
            killall -9 fcgiwrap
            RETVAL=$?
      ;;
        restart)
            echo "Restarting fastcgi"
            killall -9 fcgiwrap
            $FCGI_SCRIPT
            RETVAL=$?
      ;;
        *)
            echo "Usage: /etc/init.d/fcgi {start|stop|restart}"
            exit 1
      ;;
    esac
    exit $RETVAL

And set the permissions on our start-up script:

sudo chmod 755 /etc/init.d/fcgi

Now, attempting to start the fcgi wrapper with:

sudo /etc/init.d/fcgi start

should produce:

Starting fastcgi
spawn-fcgi.c.197: child spawned successfully: PID: 16510

Set the spawner to start on boot:

sudo update-rc.d fcgi defaults

Nginx should now find a listening daemon on port localhost's port 8998 so browsing to the root URL of your nginx server should now produce a fully-working Nagios website.

If not, check the nginx configuration again, check that fcgi is listening and consult the logs as described above.

If it is, we can clean up the remaining debris from our fcgi installation:

sudo aptitude remove git-core build-essential libfcgi-dev autoconf libtool automake

You should now have a clean Nagios installation ready for you to configure for the hosts and services you want to monitor!

Lee

Checking a server’s SSH host fingerprint with the web console

2011-04-18T23:11:00Z

Before you dismiss that error message about your server’s SSH host key changing, follow this simple procedure to make sure all is as it should be. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="Checking your SSH key with rescue mode.css"> <title> Explaining the host key</title></head> <body>

Explaining the host key

One of the fundamentals of SSH is that it uses a "fingerprint" generated using a server's unique "host key" to identify the server to a client. You may have seen a warning sometime related to the host fingerprint, either that it can't be verified or that it has changed.

The host key is randomly generated when the SSH server is set up and is used to identify the server you're connecting to. Warnings about it are more than just a devilish developer's effort to inconvenience users (though it's difficult to rule that motivation out entirely).

No, the host key is actually central to the security provided by SSH when you make a connection to your server. If someone malicious tries to set up a program to intercept your connection and steal your login credentials - a "man in the middle" attack - then the only warning you'll get is your SSH client complaining that the host key has changed.

Why the host key might change

The more innocent explanations for a changed host key include recompiling or upgrading SSH, rebuilding the server, or just using a different address to get to the same host. When your system stores the host key it records it by address, so even if "localhost" and "127.0.0.1" point to the same server an SSH client will treat them as entirely different entries.

Thus, sometimes that message is expected. But even an expected warning doesn't mean that there couldn't be a man-in-the-middle attack in progress. It sounds a little paranoid, but that's good security for you - anything can happen, at any time, and the more you do to rule out any variables the better.

So let's look at when and how to check the host fingerprint without using an SSH connection. We'll do it by going in through the server's web console.

A dire warning

First we’ll look at the error message that probably brought you to this article, a warning that the host’s identification has changed:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!      
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.           
The fingerprint for the RSA key sent by the remote host is                 
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.                           
Please contact your system administrator.
Add correct host key in /home/demo/.ssh/known_hosts to get rid of this message.
Offending key in /home/demo/.ssh/known_hosts:15                                
RSA host key for 1.2.3.4 has changed and you have requested strict checking.
Host key verification failed.

The warning can be summed up as: The fingerprint that identifies the SSH server is different from what it was the last time you connected to it. Expected or not, you'll want to check on that.

How to check

If you have your server's SSH fingerprint written down somewhere you can compare it to what SSH shows you to make sure you're connecting to the right machine. Most of us don't write that down, but it's a pretty good idea to do so if you connect from multiple machines or from unfamiliar computers (like from a consulting client's desktop or server).

If you don't have the host fingerprint handy you can use the control panel's web console to find it.

The web console

The web console lets you connect to your server as if you were, well, sitting at the console. If anything weird is going on with SSH it won't interfere with you connecting directly to the console through the SliceManager.

In the SliceManager

First connect to the SliceManager:

https://manage.slicehost.com/

In the list of slices click on the one you want to check. Look for the "Console" option at the top of the details screen:

Once you get into the web console hit "enter" a couple times and you should get a login prompt. Log in with an existing username and password.

If you don't have a username and password to use (if you've disabled passwords for all accounts, for example) you can use the SliceManager to reset your slice's root password. Then you can use the new credentials to get in.

In the console

Now that you're on the server it's time to get that host key fingerprint.

The best way

The official way to get that fingerprint is to run the "ssh-keygen" command against the server's public key, as in:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub

The "-l" option tells ssh-keygen you want to list the fingerprint, and the "-f /etc/ssh_host_rsa_key.pub" part tells ssh-keygen where it can find the host's public key file. That location is typical for Linux servers, but you may need to poke around a bit to find the file if it's not in that default location.

The output should be reminiscent of the fingerprint your SSH client showed you earlier:

2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx /etc/ssh/ssh_host_rsa_key.pub (RSA)

The first number indicates the strength of the key (in this case, 2048 bits). The fingerprint follows, along with the location of the key it analyzed and the type of key it's using (usually RSA).

The not-best way

This isn't a bad method to getting the fingerprint, it's just not as technical and fancy as the official way. It also only works when the SSH server is actively running on the machine. It's handy if you can't find the host's public key file easily.

To get the fingerprint this way, get into the web console and then ssh to localhost:

ssh localhost

If you've never completed an ssh connection to localhost before (you probably haven't) you'll see a warning:

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)?

And there's the fingerprint you wanted.

If you didn't get that message then you've completed an ssh connection to localhost before. To clear the stored key go into your account's ".ssh" directory:

cd ~/.ssh

And then edit the "known_hosts" file. Look for a line that starts with "localhost", delete it, save the file, and try again. You should get the fingerprint this time.

Write it down

Write the fingerprint down or put it in a note on your cell phone or something. You went to all this trouble to get the key, you might as make sure you have it handy in case you need to do this again.

Completing the connection - maybe

Now that we have the host key fingerprint in-hand we can see if the SSH connection is a good one.

First-time connection

If your client was connecting to the server for the first time and you were just confirming the host key before accepting it, you're set. Compare the fingerprint you dug up with what the client is showing you and if they match, accept the key.

If they don't match then don't complete the connection. If you're on a public wi-fi network (like at a coffee shop or a hotel) then disconnect from it and find someplace else to connect from - the interference may have been local, so moving may get the jerk out of your hair. Otherwise it might be a good time to get in touch with our support staff and they can help you figure out your options.

Host key has changed

If the warning you got was that the fingerprint didn't match what the client was expecting then you'll need to edit your client's list of known hosts before you can connect.

Linux and Mac OS X

On Linux, Mac OS X, and other Unix-based operating systems you usually use the "ssh" command to connect to a server via SSH. That should mean there's a directory named ".ssh" in your home directory. Inside that will be the "known_hosts" file that contains the known SSH host keys.

So basically, the file should be at:

~/.ssh/known_hosts

Sometimes you may see a "known_hosts2" file in place of or in addition to "known_hosts". If both are there then "known_hosts2" is usually the file being used when you make a connection.

Once you have the file identified, open it for editing and look for a line that starts with your server's address. It could be the IP address or the domain name, so look for whichever one you use when you're connecting via SSH.

If you were using the IP address to connect to the server at 1.2.3.4 it would look something like:

1.2.3.4 ssh-rsa AAAAB3NzaC1yc2EAAAABIwGAAQEA2Km5iIlopDndzSTbiaQZq8ynh8RPrvzBJ7dICnvAZWuH/YeNO+9DPnngzsOiYazwRD/CRSGEGRY6tS3GLclFO3Ae370aafbcq...

Once you find the line listing the key for your server you can delete the line and save the edited file. The next time you make a connection it will be as if you'd never connected to the server before. Remember to check the host fingerprint again before completing the connection!

Windows and PuTTY

There are a variety of SSH clients that can be used on Windows, but we'll talk about the free and widely-used PuTTY terminal program. If you use another program check your user documentation to find out where the client stores its known host keys.

PuTTY stores its host keys in the Windows registry. That means that before you continue with this you'll want to be sure you're comfortable editing your registry. If you don't have Administrator rights on your workstation then you might need to ask an admin to make this change for you.

To find the known host keys go to the Windows menu, then in the "search" or "run" box enter:

regedit

The registry is arranged as a hierarchy of a whole bunch of folders. The one you want to navigate to is:

HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys

And in there, you'll find one or more entries with names like:

rsa2@22:1.2.3.4

That key entry would be for an RSA version 2 encrypted key at port 22, address "1.2.3.4". With that format in mind, look for the one for the address you're connecting to.

Once you find the entry for your server right-click it and select "Delete" from the contextual menu. You'll get a warning that editing the registry can cause problems, are you sure you want to do this? Go ahead and do it. Experience the thrill of registry editing!

Well, maybe that's just me. Moving on.

Exit the registry editor and try your SSH connection again. You should get a warning that the server's host key is unknown and it will show the fingerprint again.

Double-check against the fingerprint you pulled up in the web console, and if it matches, accept the key.

Summary

It took a little poking around, but now you should have your server's host key fingerprint handy in case you need to check it again. At the least, you know how to bring it up some other time.

You might consider making a habit of recording the host key fingerprints of servers when you create them. That way you'll always have a reference handy if you need to check the fingerprint again. It's a little inconvenience at the outset in return for a pretty comforting security check you can run through easily when you connect from a new machine.

</body> </html>

Choosing a Linux Distribution

2011-03-15T20:34:00Z

If you’re new to Linux you’ll face a choice between some unfamiliar distributions. In this article we try to de-mystify those choices. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="Choosing a Linux Distribution.css"> <title> Choosing a Linux distribution</title></head> <body>

Choosing a Linux distribution

If you're new to Linux you'll face a choice between some unfamiliar distributions. In this article we try to de-mystify those choices.

So many options

When you're creating a server instance you have to choose what Linux distribution you want to run. Fortunately the distributions all share a lot of common functionality. They mostly differ in presentation and focus.

Primary considerations

When you look over this list of distributions bear in mind what kind of server you want to build. A production server needs to be stable and reliable, while a server you're just running for fun gives you more leeway to play with newer software and features.

Any of the distributions will run most software you need. They can all run web servers, database servers, and application servers - the standard "LAMP stack". They're all Linux and they all have access to software package repositories containing thousands of programs put together with that distribution in mind.

Finally, consider your level of Linux administration expertise. The distributions near the beginning of this list tend to be more friendly to new admins than those later in the list. Not coincidentally this mirrors the general popularity of each distribution as a server OS.

Quick rundown

First let's look a brief overview of each distribution we offer to help you narrow down the field.

Ubuntu focuses on being user-friendly and offering newer software versions.

CentOS emphasizes stability and enterprise software compatibility above cutting-edge features.

Debian is similarly conservative with a focus on tested and stable software, but with easier access to a repository of newer but potentially less stable packages.

Red Hat is the best choice when you absolutely need the maximum level of enterprise software compatibility but it costs an extra license fee.

Fedora is laid out similar to CentOS but offers a newer and broader variety of software packages.

Gentoo gives you obsessive control over every aspect of the system and how the software it runs is compiled, making it good for people learning to program for Linux.

Arch is targeted at people who are comfortable running a Linux server and want more control over the server's inner workings.

With that, let's look at each distribution in more detail.

The distributions

Some of these distributions are based off some of the others. Many share a package manager (CentOS, Red Hat, and Fedora use "RPM" packages while Ubuntu and Debian use "APT" or ".deb" packages). There are, in short, a lot of interrelations in the Linux world.

These similarities mean that you usually can't go wrong. At worst some tasks will take a little more work than others, so don't stress too much over your choice of distribution. Whatever you pick you should be fine.

Ubuntu

Ubuntu has a reputation for ease of use, which helps explain its popularity on desktops and servers. Ubuntu also helps users keep up with the latest software versions by releasing updates on a regular schedule.

The drawback of frequent updates is that it's harder to keep bugs from slipping into the mix. To this end Ubuntu releases an LTS version periodically, which stands for "Long-Term Support". The LTS version uses package versions that are considered more stable than cutting-edge, making it more suitable for use on a production server than the interim Ubuntu releases.

If you're completely lost as to which distribution to run Ubuntu LTS is a safe place to start. Its widespread adoption means there are several forums and sites on the Internet that provide help resources for Ubuntu users.

Ubuntu uses apt as its package manager.

CentOS

CentOS is a distribution that emphasizes reliability. It replicates Red Hat Enterprise Linux as much as possible, omitting only the non-free components of that distribution. That means CentOS is a very stable distribution and is well-suited to production environments. It also tends to be compatible with enterprise software, though it's not always officially supported by software vendors.

The price of stability is that the software versions included with CentOS are rarely the latest and greatest. The packages included with CentOS have been tuned over time to work out as many bugs and security flaws as possible.

CentOS uses rpm for its package manager.

Debian

Debian focuses on stability and security in its official releases. In that respect it can be similar to CentOS, using older packages with proven track records. The reliability of Debian is such that several other distributions (such as Ubuntu) build on top of Debian releases.

Debian provides an "unstable" repository for ambitious server admins looking to incorporate newer software releases into a Debian installation without sacrificing the stability of the rest of the system.

Debian uses apt as its package manager.

RHEL

Red Hat Enterprise Linux (RHEL) is aimed at enterprise-level servers. That means it's stable and handles heavy loads well.

The price for reliability is, in this case, a literal one. RHEL requires an additional license fee to Red Hat to access their non-free software components and updates.

The main reason to use RHEL would be if you're running a software package that has RHEL in its list of supported operating systems. This usually means enterprise software - heavy-duty stuff aimed at larger businesses. If you're spending that much on your software you'll want to make sure you run it on an OS that lets you get support from the software vendor.

If you aren't running software that requires RHEL but want to take advantage of its reliability you'll usually be fine running CentOS instead. RHEL is worth the extra cost when it gets you vendor support or if you want to be able to take advantage of support from Red Hat itself.

If you don't know if you use enterprise software then, well, you probably don't. Use another distribution for now and you can switch later if you decide to migrate to software that requires RHEL.

RHEL uses the rpm package manager.

Fedora

Fedora was originally the free version of Red Hat's Linux distribution. Red Hat still sponsors the distribution but while Red Hat's current distribution is very conservative in its package choices Fedora focuses on including cutting-edge software. The release cycle for Fedora is a short one as they continually update to newer software packages.

Fedora is a good choice if you want to have easy access to new software versions soon after release. Fedora is popular as a desktop distribution and for hobbyists learning Linux but it's still a strong server distribution.

Fedora uses the RPM package manager.

Gentoo

Gentoo is an unusual distribution in that its default behavior is to compile installed software itself instead of grabbing precompiled packages. This means that Gentoo can be intimidating for new system administrators and can take a while to set up (compiling takes time).

If you know what kind of compiler options are best for your environment then Gentoo can allow a level of system optimization that's difficult to achieve in other distributions. You can configure system default compiler options as well as set them up on a per-package basis so they'll be used when the package manager updates and recompiles software.

Gentoo is a great choice if you want an environment that forces you to learn more about Linux programming, or if you're a very knowledgeable system administrator who wants fine-grained control of every aspect of the system. Otherwise you're probably safer trying a different distribution.

Gentoo uses the emerge package manager.

Arch

Arch is a system administrator's distribution with its own design philosophy, the "Arch way". It's an approach that makes sense once you start learning how they've laid out the system but it can be a bit daunting if you're new to Linux administration.

If you're an experienced system administrator and want some good low-level control over how programs run on your server, but don't want to get into the level of detail and complexity Gentoo offers, then Arch can be worth trying. If you're new to system administration you may want to try another distribution for now. You can always take a look at Arch later when you're more comfortable.

Arch uses the pacman package manager.

Summary

That should give you a general idea of what distinguishes one Linux distribution from another. It's by no means a comprehensive examination of the distributions involved. You can get more information from their respective web sites.

Fortunately all these distributions are still Linux, so all have a baseline of performance. The distributions that don't emphasize stability aren't unstable, it just means there's a possibility they'll contain bugs that wouldn't be in a more conservative distribution. Similarly, if a distribution emphasizes stability it doesn't mean the software it runs is ancient - it just won't include the newest bells and whistles.

If you're having trouble deciding just try them out. It doesn't take long to rebuild a VPS with a new image so experiment a bit before settling on one and installing software in earnest. You might even have fun - tinkering with an instance you know you can rebuild at any time is surprisingly liberating.

Once you pick your distribution you can visit our articles repository for help setting it up or go to the distribution's web site to view their official documentation.

-- Jered

</body> </html>

Installing MySQL Server

2011-03-10T03:03:00Z

This article series discusses installing MySQL and getting it running with some basic configuration.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-install-portal.css"> <title> What is MySQL?</title></head> <body>

What is MySQL?

MySQL is a database server, software that stores and retrieves data for other applications. You might need a database that will work with an application you're developing, or you may just want a database so you can run web applications like WordPress or Drupal. MySQL should serve you well in either instance.

These articles are intended to help a MySQL novice get a server up and running, with just enough information for basic operation. If you're going to use MySQL in a busy production environment you'd do better to get a database administrator to help you run it - someone who specializes in running and tuning database software.

If you just want to get started with a minimum of fuss, the first article covers:

- Installing MySQL
- Launching MySQL
- Using the MySQL client
- Setting a root password
- Basic MySQL user concepts
- Creating databases
- Creating users
- Granting user permissions

The second article walks through the server's configuration options, including:

- Finding the config files
- Reading my.cnf
- Network settings
- Backups and mysqldump

And the third article is a list of common tasks you might need to perform on a MySQL server, primarily discussing the creation and manipulation of databases, users, and tables. It also mentions how to reset MySQL's root password.

Distribution links

The first two articles, covering installing and configuring MySQL, have separate versions for each of the distributions we support.

Ubuntu

CentOS

Debian

Fedora

RHEL

Gentoo

Arch

The third article is distribution-neutral:

Basic MySQL server tasks

Summary

These three articles should get you up and running with a minimum of fuss, and will hopefully put you in a good position to learn more about running and tuning your MySQL server.

--Jered

</body> </html>

Configuring MySQL server on Arch

2011-03-10T02:56:00Z

We continue our MySQL server setup for Arch by looking at configuration options to try and ensure the server doesn’t just run, but runs smoothly.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-config-arch.css"> <title> Beyond the defaults</title></head> <body>

Beyond the defaults

In the previous article we covered a basic MySQL server setup on Arch Linux. We set the root password, created a database, and created a user for the database. Now let's look at MySQL in a little more detail so we can tweak its configuration and be ready in case something goes wrong.

Finding the config files

By default you'll find MySQL's configuration files in:

/etc/mysql

If they're not there, however, you can ask mysqld where it looks for its config. Run the command:

/usr/sbin/mysqld --help --verbose

You'll get a flood of text back. The first part describes the options you can send to the server when you launch it. The second part is all the configuration stuff that was set when the server was compiled.

What we're looking for shows up near the start of the output. Find a couple lines that look like:

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf

And there we are. The server works down that list until it finds a configuration file.

my.cnf

With the location in hand, open the my.cnf file and have a look inside.

/etc/mysql/my.cnf

Any lines starting with "#" are comments, and they mostly document what the different settings are for. They're good to read through. You'll find details like the location of log files and where the database files are kept.

Config groups

There are lines in the config file that just contain a word in square brackets, like "[client]" or "[mysqld]". Those are "config groups" and they tell the programs that read the configuration file which parts they should pay attention to.

See, while we've been focusing on the server part of MySQL, it's technically a collection of tools. That includes the server (mysqld), the client (mysql), and some other tools we'll talk about in a bit. Those programs look in my.cnf to see how they should behave.

There's a bit more to it, but basically: The "client" config section controls the mysql client, and the "mysqld" section controls the server config.

Log files

If something does go wrong the best place to start troubleshooting any program is its logs.

/mysqlLook in the my.cnf file and look for a "log_error" line, as in:

log_error = /var/log/mysql/error.log

If you don't see a line like that, create one in the "mysqld" section so MySQL will use its own error log. We recommend using the location in the example, creating the "/var/log/mysql" directory if it doesn't already exist. Then restart MySQL to make the change.

Make sure the log directory you choose can be written to by the user controlling the MySQL process (usually "mysql"). The user running the process will be defined in the "user" config value for mysqld in my.cnf.

Network settings

There might be a "port" setting under both the client and server config sections. The port under the server section controls what port the server will listen to. By default that's 3306 but you can change it to anything you'd like.

The port in the client section tells the client what port to connect to by default. You'll generally want the two port settings to match up.

If you don't see the port entries in the config file that just means they're using the default. If you want to change the port you would add the lines in the appropriate categories:

[client]
port = 3306
 
[mysqld]
port = 3306

The other network setting to look for is the "bind-address" value. That usually gets set to the address for localhost, 127.0.0.1. By binding to localhost the server makes sure no one can connect to it from outside the local machine.

If you're running your MySQL server on a different machine from your application you'll want to bind to a remotely-accessible address instead of localhost. Change the bind-address setting to match your public IP address (or, better, a backend IP address on a network that fewer machines can access).

If you don't see a "bind-address" entry you should put one into the "mysqld" category to help control access to the server:

[mysqld]
bind-address = 127.0.0.1

Remember to account for the client's hostname when you set up your database users and to poke a hole in your firewall if you're running iptables.

mysqld and mysqld_safe

Behind the scenes there are actually two versions of the MySQL server, "mysqld" and "mysqld_safe". Both read the same config sections. The main difference is that mysqld_safe launches with a few more safety features enabled to make it easier to recover from a crash or other problem.

Both mysqld and mysqld_safe will read config entries in the "mysqld" section. If you include a "mysqld_safe" section, then only mysqld_safe will read those values in.

By default the mysql service launches "mysqld_safe". That's a good thing, and you should only look to change that if you really know what you're doing.

mysqladmin

The mysqladmin tool lets you perform some administrative functions from the command line. We won't talk much about it here because we're just trying to get you up and running with enough basics to get by. It's worth looking at the tool in more depth later to see what it can do, particularly if you need to build scripts that perform functions like checking the status of the server or creating and dropping databases.

Backups

You have a few options when it comes to making backups of your databases apart from the usual "back up the whole machine" approach. The main two are copying the database files and using mysqldump.

File copy

By default MySQL creates a directory for each database in its data directory:

/var/lib/mysql

Once you've found the data directory, hold off a moment before making a copy of it. When the database server is active it could be writing new values to tables at any time. That means if it writes to a table halfway through your copy some files will change and lead to a corrupt backup. Not a good thing if you're trying to plan for disaster recovery.

To make sure the database files are copied cleanly you can shut the MySQL server down entirely before the copy. That's safe but isn't always ideal.

Another approach you can take is to lock the database as read-only for the duration of the copy. Then when you're done, release the lock. That way your applications can still read data while you're backing up files.

Lock the databases to read-only by running, from the command line:

mysql -u root -p -e "FLUSH TABLES WITH READ LOCK;"

To unlock the database when you're done, run:

mysql -u root -p -e "UNLOCK TABLES;"

We're using a new option with the mysql client, "-e". That tells the client to run the query in quotes as if we'd entered it in the mysql shell proper.

Note that if you're setting these commands up in a script you can put the password in quotes right after "-p" with no space between the two, as in:

mysql -u root -p"password" -e "FLUSH TABLES WITH READ LOCK;"
mysql -u root -p"password" -e "UNLOCK TABLES;"

Just make sure you set the permissions on that file to restrict read access. We don't want just anyone to be able to see that password.

mysqldump

Another approach to backing up your database is to use the "mysqldump" tool. Rather than copying the database files directly, mysqldump generates a text file that represents the database. By default the text file contains a list of SQL statements you would use to recreate the database, but you can also export the database in another format like CSV or XML. You can read the man page for mysqldump to see all its options.

The statements generated by mysqldump go straight to standard output. You'll want to specify a file to redirect the output to when you run it. For example:

mysqldump -u root -p demodb > dbbackup.sql

That command will tell mysqldump to recreate the "demodb" database in SQL statements and to write them to the file "dbbackup.sql". Note that the username and password options function the same as the mysql client, so you can include the password directly after "-p" in a script.

Restore from mysqldump

Restoring a mysqldumped database looks similar to what was used to create it, but we use plain old "mysql" instead of "mysqldump":

mysql -u root -p demodb < dbbackup.sql

We also change from a greater-than to a less-than sign. That switches the command from redirecting its output to telling it to read its input from the existing file. That input is sent to the "mysql" command, causing the mysqldumped instructions to recreate the database.

Note that by default the SQL statements generated would just add to existing database tables, not overwrite them. If you're restoring a backup over an existing database you should drop the database's tables first, or drop and recreate the database itself. You can change that behavior by using the "--add-drop-table" option with the command that creates the mysqldump. That causes mysqldump to add a command to the backup files it writes that will drop tables before recreating them.

Database engine

The last concept we'll talk about here is that of the "database engine". The engine is the process that's churning away behind the scenes, writing to and reading data from files. You won't usually need to know anything other than that it's there, but sometimes you'll want to run an application that's been optimized for a particular database engine.

The engine type is set when a table is created. Tables are usually created by the application that's going to use them, which is why we aren't going to get into that syntax here.

To see the engine used by your database's tables you can run the following command in the MySQL shell:

SHOW TABLE STATUS FROM demodb;

Change "demodb" to the name of your database.

Choosing an engine

Ideally you won't need to choose an engine. If you're not very familiar with MySQL that's certainly the safest way to go - let the application do its thing, and if you're writing the application, use the default engine until you're more comfortable with your options.

If you have a database administrator, do whatever he or she says. They're smart people, they know what they're talking about.

The two database engines used most often with MySQL are "MyISAM" and "InnoDB". The default database engine for MySQL version 5.1 and earlier is MyISAM, while InnoDB is the default database engine starting with MySQL version 5.5.

MyISAM

Because MyISAM has been the default in MySQL for a while it's the most compatible choice of the two main engines. Certain types of searches perform better on MyISAM than InnoDB. Just because it's the older of the two doesn't mean it can't be the best for a given application type.

InnoDB

InnoDB is more fault-tolerant than MyISAM and handles crashes and recovery with a much smaller chance of database corruption. This is a good thing.

The main trouble with InnoDB is that for best performance it requires a lot of tweaking for your environment and access patterns. If you have a DBA that's no problem, but if you're a developer who just wants a database up and running for a test server you probably won't want to deal with tuning InnoDB.

It's possible you may be running an application that requires InnoDB, and if you're using MySQL 5.1 or earlier there might not be any settings already in the my.cnf config file. That can be a problem if you're running on a server that doesn't have an abundance of memory.

Some settings to get you started with InnoDB on a shared server with 256 megs of RAM are:

innodb_buffer_pool_size = 32M
innodb_log_file_size = 8M
innodb_thread_concurrency = 8
innodb_file_per_table

Add those to the [mysqld] section of the config file. Again, those are only rough guides - enough to get you running, but definitely not optimized. For that you'll probably want a DBA, or at least to experiment with incremental changes over time.

Summary

Now you should have MySQL configured for your environment, and might even have accounted for the database engine being used by your tables.

In the next article we'll look at some basic tasks you can run in the mysql shell to manipulate and observe your database at a high level.

--Jered

</body> </html>

Installing MySQL Server on Arch

2011-03-10T02:55:00Z

We look at installing MySQL on Arch and getting it running with a database and a user to access it.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-install-arch.css"> <title> Meet MySQL</title></head> <body>

Meet MySQL

MySQL is an open-source relational database. In a nutshell, for those unfamiliar with it: A database is where an application keeps its stuff.

To break it down a little further, "relational database" is a term that refers to how data is organized and accessed within the database. The "SQL" part refers to the language used by application queries to retrieve and store data ("Structured Query Language").

MySQL is free and widely used, meaning you can find a lot of application support, tools, and community help for it. MySQL is a safe choice if you know you need a database but don't know what to make of the options that are out there.

This article covers a basic installation of a MySQL server on Arch Linux - just enough to get you started. Remember that you might need to install other packages to let apps use MySQL, like extensions for PHP. Check your application documentation for details.

Installing MySQL

The easiest way to install the MySQL server is through the Arch package manager:

sudo pacman -Sy
sudo pacman -S mysql

We'll talk about how to manually change the root password and remove an anonymous user later, but to do it the easy way run this command now:

sudo /usr/bin/mysql_secure_installation

Hit "enter" to give no password for root when that program asks for it. To apply some reasonable security to your new MySQL server say "yes" to all the questions the program asks. In order, those will let you set the root password, remove anonymous users, disable remote root logins, delete the test database the installer included, and then reload the privileges so your changes will take effect.

iptables

If you have iptables enabled and want to connect to MySQL from another machine you'll need to open a port in your server's firewall (the default port is 3306). You don't need to do this if the application using MySQL is running on the same machine.

If you do need to open a port (again, only if you're accessing MySQL from a different machine from the one you're installing on), you can use the following rules in iptables to open port 3306:

-I INPUT -p tcp --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-I OUTPUT -p tcp --sport 3306 -m state --state ESTABLISHED -j ACCEPT

Launch MySQL

Now that MySQL is installed you can make sure it's running by trying to launch it:

sudo /etc/rc.d/mysqld start

If you see a message that it's already running that's okay (it means that, well, it's already running).

Launching at boot

Next you'll want to edit the rc.conf file at:

/etc/rc.conf

Go to the bottom of the file to find the "DAEMONS" line. Add "mysqld" to its list of services somewhere after "network" but before any services that might use mysql, as in:

DAEMONS=(syslog-ng network netfs mysqld crond sshd)

That makes sure your machine will launch the MySQL server when it reboots.

The MySQL shell

There is more than one way to manage a MySQL server, so we'll focus on the most basic and compatible approach: The mysql shell.

At the command prompt, run:

/usr/bin/mysql -u root -p

That will attempt to launch the mysql client and enter the shell as user "root". When you're prompted for a password enter the one you set at install time or, if you haven't set one, just hit enter to submit no password.

You should be greeted by the mysql shell prompt:

mysql>

Setting the root password

If you got in by entering a blank password, or want to change the root password you've set, you can do it by entering the following command in the mysql shell. Replace the "password" in quotes with your desired password:

UPDATE mysql.user SET Password = PASSWORD('password') WHERE User = 'root';

It's kind of a mouthful. The reason for this is that MySQL keeps user data in its own database, so to change the password we have to run an SQL command to update the database.

Next we'll reload the stored user information to make our change go into effect:

FLUSH PRIVILEGES;

Note that we're using all-caps for SQL commands. If you type those commands in lowercase they'll work too. By convention the commands are written in all-caps to make them stand out from field names and other data that's being manipulated.

Looking at users

As mentioned in the previous section, MySQL stores the user information in its own database. The name of the database is "mysql". Inside that database the user information is in a "table", a dataset, named "User".

If you want to see what users are set up in MySQL you need to run a query against the "user" table in the "mysql" database. Let's do that now:

SELECT User, Host, Password FROM mysql.user;

Breaking that down...

The "SELECT" command tells MySQL you're asking for data.

The "User, Host, Password" part tells MySQL what fields you want it to look in. Fields are categories for the data in a table. In this case we're looking for the username, the host associated with the username, and the encrypted password entry.

Finally, the "FROM mysql.user" part of the command tells MySQL to get the data from the "mysql" database and the "user" table.

And then the command ends with a semicolon

About that semicolon

All SQL queries end with a semicolon. MySQL will wait for you to keep entering additions to a query until it sees a semicolon.

That means that you can break lines up into smaller parts to make them easier to read. For example, the above command also works if you enter it in multiple lines in the mysql shell, as in:

mysql> SELECT User, Host, Password
    -> FROM mysql.user;

When you hit "enter" after the "Password" part you'll get a new line so you can keep typing. The "->" indicates that the shell thinks you're still in the middle of a statement. You can type a semicolon by itself to end the command if you simply forgot it on the first line.

User hosts

Okay, with that important aside about the semicolon done, let's look at the output of that query:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
|                  | %         |                                           |
+------------------+-----------+-------------------------------------------+

As you can see, users are associated with a host - specifically, the host they connect to. The "root" user in this case is defined for localhost, for the IP address of localhost, and the hostname of the server ("demohost" in this example). You'll usually only need to set a user for one host, the one you typically connect from.

If you're running your application on the same machine as the MySQL server the host it connects to by default will be "localhost". That means any new users you create will need to have "localhost" in its "host" field.

If your application connects remotely the "host" entry MySQL will look for is the IP address or DNS hostname of the remote machine (the one the client is coming from).

A special value for the host is "%", as you'll see for the blank user (more on that shortly). That's a wildcard that applies to any host value. You usually won't want to use that because it's more secure to limit access specifically to trusted hosts.

Anonymous users

In the example output above you'll notice there's one entry that has a host value but no username or password. That's an "anonymous user". When a client connects with no username specified it's trying to connect as an anonymous user.

You usually don't want one of those in there, but some MySQL installations include one by default. If you see one of those you should either delete the user (refer to the username with empty quotes, like '') or set a password for it (we cover both tasks later in this series).

Create a database

That covers some basic concepts surrounding users, so now let's look at creating a database.

It's worth noting at this point that there is a difference between a "database server" and an actual "database", even though you'll often see those terms used interchangeably. MySQL itself is a database server, meaning that it keeps track of databases and controls access to them. An actual database is where all the data goes. That's what applications are trying to get at when they talk to MySQL.

Some applications will create a database as part of their setup process, while others require you to create a database yourself and tell the application about it later. Fortunately it's an easy process.

To create a database, log into the mysql shell and run:

CREATE DATABASE demodb;

That's all there is to it. Replace "demodb" with the name of the database you want to create, of course. You can make sure it's there by running a query to list all databases:

SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| demodb             |
| mysql              |
+--------------------+
3 rows in set (0.00 sec)

Add a database user

It's not a good idea to have applications connecting to the database using the root user. That gives them more privileges than they need. We'll create a user named "demouser" that applications can use to connect to the new database.

To make the user run the following in the mysql shell:

INSERT INTO mysql.user (User,Host,Password) VALUES('demouser','localhost',PASSWORD('demopassword'));

When you make changes to the user table in the mysql database you need to tell MySQL to read the changes by flushing the privileges. To wit:

FLUSH PRIVILEGES;

You can make sure the user is there by running that "select" query again:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| demouser         | localhost | *0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6 |
+------------------+-----------+-------------------------------------------+

Grant database user permissions

Right now our new user has no privileges. It can be used to log on, but it can't be used to make any database changes. Let's give it full permissions for our new database by running:

GRANT ALL PRIVILEGES ON demodb.* to demouser@localhost;

And follow it up with the usual:

FLUSH PRIVILEGES;

To check those privileges were set, we'll run:

SHOW GRANTS FOR 'demouser'@'localhost';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for demouser@localhost                                                                                   |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'demouser'@'localhost' IDENTIFIED BY PASSWORD '*0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6' |
| GRANT ALL PRIVILEGES ON `demodb`.* TO 'demouser'@'localhost'                                                    |
+-----------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

What you get back are the commands needed to reproduce that user's permissions if you were to rebuild the server. The "USAGE on *.*" part basically means they get no privileges on anything by default. Then that gets overridden by the second command, which is the grant you ran for the new database.

Summary

And that's all you need if you're just creating a database and a user. We were a bit long on concepts there but that should give you a solid grounding from which to learn more. Good work.

In the next article we'll do some basic security and stability checks by looking at the MySQL server's configuration files and a few key tools.

--Jered

</body> </html>

Configuring MySQL server on Gentoo

2011-03-10T02:49:00Z

We continue our MySQL server setup for Gentoo by looking at configuration options to try and ensure the server doesn’t just run, but runs smoothly.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-config-gentoo.css"> <title> Beyond the defaults</title></head> <body>

Beyond the defaults

In the previous article we covered a basic MySQL server setup on Gentoo Linux. We set the root password, created a database, and created a user for the database. Now let's look at MySQL in a little more detail so we can tweak its configuration and be ready in case something goes wrong.

Finding the config files

By default you'll find MySQL's configuration files in:

/etc/mysql

If they're not there, however, you can ask mysqld where it looks for its config. Run the command:

/usr/sbin/mysqld --help --verbose

What we're looking for shows up near the start of the output. Find a couple lines that look like:

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf

And there we are. The server works down that list until it finds a configuration file.

my.cnf

With the location in hand, open the my.cnf file and have a look inside.

/etc/mysql/my.cnf

Config groups

There's a bit more to it, but basically: The "client" config section controls the mysql client, and the "mysqld" section controls the server config.

Log files

If something does go wrong the best place to start troubleshooting any program is its logs. By default MySQL stores its log files in the directory:

/var/log/mysql

You may need to use sudo to get a listing of the files in that directory.

If you don't find the MySQL logs in the default directory you'll need to check MySQL's config. Look in the my.cnf file and look for a "log_error" line, as in:

log_error = /var/log/mysql/error.log

Network settings

The port in the client section tells the client what port to connect to by default. You'll generally want the two port settings to match up.

If you don't see the port entries in the config file that just means they're using the default. If you want to change the port you would add the lines in the appropriate categories:

[client]
port = 3306
 
[mysqld]
port = 3306

If you don't see a "bind-address" entry you should put one into the "mysqld" category to help control access to the server:

[mysqld]
bind-address = 127.0.0.1

Remember to account for the client's hostname when you set up your database users and to poke a hole in your firewall if you're running iptables.

mysqld and mysqld_safe

Both mysqld and mysqld_safe will read config entries in the "mysqld" section. If you include a "mysqld_safe" section, then only mysqld_safe will read those values in.

By default the mysql service launches "mysqld_safe". That's a good thing, and you should only look to change that if you really know what you're doing.

mysqladmin

Backups

You have a few options when it comes to making backups of your databases apart from the usual "back up the whole machine" approach. The main two are copying the database files and using mysqldump.

File copy

By default MySQL creates a directory for each database in its data directory:

/var/lib/mysql

To make sure the database files are copied cleanly you can shut the MySQL server down entirely before the copy. That's safe but isn't always ideal.

Lock the databases to read-only by running, from the command line:

mysql -u root -p -e "FLUSH TABLES WITH READ LOCK;"

To unlock the database when you're done, run:

mysql -u root -p -e "UNLOCK TABLES;"

We're using a new option with the mysql client, "-e". That tells the client to run the query in quotes as if we'd entered it in the mysql shell proper.

Note that if you're setting these commands up in a script you can put the password in quotes right after "-p" with no space between the two, as in:

mysql -u root -p"password" -e "FLUSH TABLES WITH READ LOCK;"
mysql -u root -p"password" -e "UNLOCK TABLES;"

Just make sure you set the permissions on that file to restrict read access. We don't want just anyone to be able to see that password.

mysqldump

The statements generated by mysqldump go straight to standard output. You'll want to specify a file to redirect the output to when you run it. For example:

mysqldump -u root -p demodb > dbbackup.sql

Restore from mysqldump

Restoring a mysqldumped database looks similar to what was used to create it, but we use plain old "mysql" instead of "mysqldump":

mysql -u root -p demodb < dbbackup.sql

Database engine

The engine type is set when a table is created. Tables are usually created by the application that's going to use them, which is why we aren't going to get into that syntax here.

To see the engine used by your database's tables you can run the following command in the MySQL shell:

SHOW TABLE STATUS FROM demodb;

Change "demodb" to the name of your database.

Choosing an engine

If you have a database administrator, do whatever he or she says. They're smart people, they know what they're talking about.

MyISAM

InnoDB

InnoDB is more fault-tolerant than MyISAM and handles crashes and recovery with a much smaller chance of database corruption. This is a good thing.

Some settings to get you started with InnoDB on a shared server with 256 megs of RAM are:

innodb_buffer_pool_size = 32M
innodb_log_file_size = 8M
innodb_thread_concurrency = 8
innodb_file_per_table

Summary

Now you should have MySQL configured for your environment, and might even have accounted for the database engine being used by your tables.

In the next article we'll look at some basic tasks you can run in the mysql shell to manipulate and observe your database at a high level.

--Jered

</body> </html>

Installing MySQL Server on Gentoo

2011-03-10T02:48:00Z

We look at installing MySQL on Gentoo and getting it running with a database and a user to access it.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-install-gentoo.css"> <title> Meet MySQL</title></head> <body>

Meet MySQL

MySQL is an open-source relational database. In a nutshell, for those unfamiliar with it: A database is where an application keeps its stuff.

This article covers a basic installation of a MySQL server on Gentoo Linux - just enough to get you started. Remember that you might need to install other packages to let apps use MySQL, like extensions for PHP. Check your application documentation for details.

Installing MySQL

The easiest way to install the MySQL server is through the Gentoo package manager:

sudo emerge --sync
sudo emerge mysql
sudo /usr/bin/mysql_install_db

We'll talk about how to manually change the root password and remove an anonymous user later, but to do it the easy way run this command now:

sudo /usr/bin/mysql_secure_installation

iptables

If you do need to open a port (again, only if you're accessing MySQL from a different machine from the one you're installing on), you can use the following rules in iptables to open port 3306:

-I INPUT -p tcp --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-I OUTPUT -p tcp --sport 3306 -m state --state ESTABLISHED -j ACCEPT

Launch MySQL

Now that MySQL is installed you can make sure it's running by trying to launch it:

sudo /etc/init.d/mysql start

If you see a message that it's already running that's okay (it means that, well, it's already running).

Launching at boot

Next we'll run the command:

sudo /sbin/rc-update add mysql default

That makes sure your machine will launch the MySQL server when it reboots.

The MySQL shell

There is more than one way to manage a MySQL server, so we'll focus on the most basic and compatible approach: The mysql shell.

At the command prompt, run:

/usr/bin/mysql -u root -p

You should be greeted by the mysql shell prompt:

mysql>

Setting the root password

UPDATE mysql.user SET Password = PASSWORD('password') WHERE User = 'root';

It's kind of a mouthful. The reason for this is that MySQL keeps user data in its own database, so to change the password we have to run an SQL command to update the database.

Next we'll reload the stored user information to make our change go into effect:

FLUSH PRIVILEGES;

Looking at users

If you want to see what users are set up in MySQL you need to run a query against the "user" table in the "mysql" database. Let's do that now:

SELECT User, Host, Password FROM mysql.user;

Breaking that down...

The "SELECT" command tells MySQL you're asking for data.

Finally, the "FROM mysql.user" part of the command tells MySQL to get the data from the "mysql" database and the "user" table.

And then the command ends with a semicolon

About that semicolon

All SQL queries end with a semicolon. MySQL will wait for you to keep entering additions to a query until it sees a semicolon.

That means that you can break lines up into smaller parts to make them easier to read. For example, the above command also works if you enter it in multiple lines in the mysql shell, as in:

mysql> SELECT User, Host, Password
    -> FROM mysql.user;

User hosts

Okay, with that important aside about the semicolon done, let's look at the output of that query:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
|                  | %         |                                           |
+------------------+-----------+-------------------------------------------+

If your application connects remotely the "host" entry MySQL will look for is the IP address or DNS hostname of the remote machine (the one the client is coming from).

Anonymous users

Create a database

That covers some basic concepts surrounding users, so now let's look at creating a database.

To create a database, log into the mysql shell and run:

CREATE DATABASE demodb;

That's all there is to it. Replace "demodb" with the name of the database you want to create, of course. You can make sure it's there by running a query to list all databases:

SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| demodb             |
| mysql              |
+--------------------+
3 rows in set (0.00 sec)

Add a database user

To make the user run the following in the mysql shell:

INSERT INTO mysql.user (User,Host,Password) VALUES('demouser','localhost',PASSWORD('demopassword'));

When you make changes to the user table in the mysql database you need to tell MySQL to read the changes by flushing the privileges. To wit:

FLUSH PRIVILEGES;

You can make sure the user is there by running that "select" query again:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| demouser         | localhost | *0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6 |
+------------------+-----------+-------------------------------------------+

Grant database user permissions

Right now our new user has no privileges. It can be used to log on, but it can't be used to make any database changes. Let's give it full permissions for our new database by running:

GRANT ALL PRIVILEGES ON demodb.* to demouser@localhost;

And follow it up with the usual:

FLUSH PRIVILEGES;

To check those privileges were set, we'll run:

SHOW GRANTS FOR 'demouser'@'localhost';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for demouser@localhost                                                                                   |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'demouser'@'localhost' IDENTIFIED BY PASSWORD '*0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6' |
| GRANT ALL PRIVILEGES ON `demodb`.* TO 'demouser'@'localhost'                                                    |
+-----------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Summary

And that's all you need if you're just creating a database and a user. We were a bit long on concepts there but that should give you a solid grounding from which to learn more. Good work.

In the next article we'll do some basic security and stability checks by looking at the MySQL server's configuration files and a few key tools.

--Jered

</body> </html>

Configuring MySQL server on RHEL

2011-03-10T02:44:00Z

We continue our MySQL server setup for CentOS by looking at configuration options to try and ensure the server doesn’t just run, but runs smoothly.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-config-rhel.css"> <title> Beyond the defaults</title></head> <body>

Beyond the defaults

In the previous article we covered a basic MySQL server setup on Red Hat Linux. We set the root password, created a database, and created a user for the database. Now let's look at MySQL in a little more detail so we can tweak its configuration and be ready in case something goes wrong.

Finding the config files

By default you'll find MySQL's configuration file at:

/etc/my.cnf

If it's not there, however, you can ask mysqld where it looks for its config. Run the command:

/usr/libexec/mysqld --help --verbose

What we're looking for shows up near the start of the output. Find a couple lines that look like:

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf

And there we are. The server works down that list until it finds a configuration file.

my.cnf

With the location in hand, open the my.cnf file and have a look inside.

/etc/my.cnf

Config groups

There's a bit more to it, but basically: The "client" config section controls the mysql client, and the "mysqld" section controls the server config.

Log files

If something does go wrong the best place to start troubleshooting any program is its logs. By default MySQL stores its log files in the directory:

/var/log

You may need to use sudo to get a listing of the files in that directory.

If you don't find the MySQL logs in the default directory you'll need to check MySQL's config. Look in the my.cnf file and look for a "log_error" line, as in:

log_error = /var/log/mysql/error.log

Network settings

The port in the client section tells the client what port to connect to by default. You'll generally want the two port settings to match up.

If you don't see the port entries in the config file that just means they're using the default. If you want to change the port you would add the lines in the appropriate categories:

[client]
port = 3306
 
[mysqld]
port = 3306

If you don't see a "bind-address" entry you should put one into the "mysqld" category to help control access to the server:

[mysqld]
bind-address = 127.0.0.1

Remember to account for the client's hostname when you set up your database users and to poke a hole in your firewall if you're running iptables.

mysqld and mysqld_safe

Both mysqld and mysqld_safe will read config entries in the "mysqld" section. If you include a "mysqld_safe" section, then only mysqld_safe will read those values in.

By default the mysql service launches "mysqld_safe". That's a good thing, and you should only look to change that if you really know what you're doing.

mysqladmin

Backups

You have a few options when it comes to making backups of your databases apart from the usual "back up the whole machine" approach. The main two are copying the database files and using mysqldump.

File copy

By default MySQL creates a directory for each database in its data directory:

/var/lib/mysql

To make sure the database files are copied cleanly you can shut the MySQL server down entirely before the copy. That's safe but isn't always ideal.

Lock the databases to read-only by running, from the command line:

mysql -u root -p -e "FLUSH TABLES WITH READ LOCK;"

To unlock the database when you're done, run:

mysql -u root -p -e "UNLOCK TABLES;"

We're using a new option with the mysql client, "-e". That tells the client to run the query in quotes as if we'd entered it in the mysql shell proper.

Note that if you're setting these commands up in a script you can put the password in quotes right after "-p" with no space between the two, as in:

mysql -u root -p"password" -e "FLUSH TABLES WITH READ LOCK;"
mysql -u root -p"password" -e "UNLOCK TABLES;"

Just make sure you set the permissions on that file to restrict read access. We don't want just anyone to be able to see that password.

mysqldump

The statements generated by mysqldump go straight to standard output. You'll want to specify a file to redirect the output to when you run it. For example:

mysqldump -u root -p demodb > dbbackup.sql

Restore from mysqldump

Restoring a mysqldumped database looks similar to what was used to create it, but we use plain old "mysql" instead of "mysqldump":

mysql -u root -p demodb < dbbackup.sql

Database engine

The engine type is set when a table is created. Tables are usually created by the application that's going to use them, which is why we aren't going to get into that syntax here.

To see the engine used by your database's tables you can run the following command in the MySQL shell:

SHOW TABLE STATUS FROM demodb;

Change "demodb" to the name of your database.

Choosing an engine

If you have a database administrator, do whatever he or she says. They're smart people, they know what they're talking about.

MyISAM

InnoDB

InnoDB is more fault-tolerant than MyISAM and handles crashes and recovery with a much smaller chance of database corruption. This is a good thing.

Some settings to get you started with InnoDB on a shared server with 256 megs of RAM are:

innodb_buffer_pool_size = 32M
innodb_log_file_size = 8M
innodb_thread_concurrency = 8
innodb_file_per_table

Summary

Now you should have MySQL configured for your environment, and might even have accounted for the database engine being used by your tables.

In the next article we'll look at some basic tasks you can run in the mysql shell to manipulate and observe your database at a high level.

--Jered

</body> </html>

Installing MySQL Server on RHEL

2011-03-10T02:43:00Z

We look at installing MySQL on RHEL and getting it running with a database and a user to access it.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-install-rhel.css"> <title> Meet MySQL</title></head> <body>

Meet MySQL

MySQL is an open-source relational database. In a nutshell, for those unfamiliar with it: A database is where an application keeps its stuff.

This article covers a basic installation of a MySQL server on Red Hat Linux - just enough to get you started. Remember that you might need to install other packages to let apps use MySQL, like extensions for PHP. Check your application documentation for details.

Installing MySQL

The easiest way to install the MySQL server is through the Red Hat package manager:

sudo yum install mysql-server
service mysqld start

We'll talk about how to manually change the root password and remove an anonymous user later, but to do it the easy way run this command now:

sudo /usr/bin/mysql_secure_installation

iptables

If you do need to open a port (again, only if you're accessing MySQL from a different machine from the one you're installing on), you can use the following rules in iptables to open port 3306:

-I INPUT -p tcp --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-I OUTPUT -p tcp --sport 3306 -m state --state ESTABLISHED -j ACCEPT

Launch MySQL

Now that MySQL is installed you can make sure it's running by trying to launch it:

sudo service mysqld start

If you see a message that it's already running that's okay (it means that, well, it's already running).

Launching at boot

Next we'll run the command:

sudo chkconfig mysqld on

That makes sure your machine will launch the MySQL server when it reboots.

The MySQL shell

There is more than one way to manage a MySQL server, so we'll focus on the most basic and compatible approach: The mysql shell.

At the command prompt, run:

/usr/bin/mysql -u root -p

You should be greeted by the mysql shell prompt:

mysql>

Setting the root password

UPDATE mysql.user SET Password = PASSWORD('password') WHERE User = 'root';

It's kind of a mouthful. The reason for this is that MySQL keeps user data in its own database, so to change the password we have to run an SQL command to update the database.

Next we'll reload the stored user information to make our change go into effect:

FLUSH PRIVILEGES;

Looking at users

If you want to see what users are set up in MySQL you need to run a query against the "user" table in the "mysql" database. Let's do that now:

SELECT User, Host, Password FROM mysql.user;

Breaking that down...

The "SELECT" command tells MySQL you're asking for data.

Finally, the "FROM mysql.user" part of the command tells MySQL to get the data from the "mysql" database and the "user" table.

And then the command ends with a semicolon

About that semicolon

All SQL queries end with a semicolon. MySQL will wait for you to keep entering additions to a query until it sees a semicolon.

That means that you can break lines up into smaller parts to make them easier to read. For example, the above command also works if you enter it in multiple lines in the mysql shell, as in:

mysql> SELECT User, Host, Password
    -> FROM mysql.user;

User hosts

Okay, with that important aside about the semicolon done, let's look at the output of that query:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
|                  | %         |                                           |
+------------------+-----------+-------------------------------------------+

If your application connects remotely the "host" entry MySQL will look for is the IP address or DNS hostname of the remote machine (the one the client is coming from).

Anonymous users

Create a database

That covers some basic concepts surrounding users, so now let's look at creating a database.

To create a database, log into the mysql shell and run:

CREATE DATABASE demodb;

That's all there is to it. Replace "demodb" with the name of the database you want to create, of course. You can make sure it's there by running a query to list all databases:

SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| demodb             |
| mysql              |
+--------------------+
3 rows in set (0.00 sec)

Add a database user

To make the user run the following in the mysql shell:

INSERT INTO mysql.user (User,Host,Password) VALUES('demouser','localhost',PASSWORD('demopassword'));

When you make changes to the user table in the mysql database you need to tell MySQL to read the changes by flushing the privileges. To wit:

FLUSH PRIVILEGES;

You can make sure the user is there by running that "select" query again:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| demouser         | localhost | *0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6 |
+------------------+-----------+-------------------------------------------+

Grant database user permissions

Right now our new user has no privileges. It can be used to log on, but it can't be used to make any database changes. Let's give it full permissions for our new database by running:

GRANT ALL PRIVILEGES ON demodb.* to demouser@localhost;

And follow it up with the usual:

FLUSH PRIVILEGES;

To check those privileges were set, we'll run:

SHOW GRANTS FOR 'demouser'@'localhost';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for demouser@localhost                                                                                   |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'demouser'@'localhost' IDENTIFIED BY PASSWORD '*0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6' |
| GRANT ALL PRIVILEGES ON `demodb`.* TO 'demouser'@'localhost'                                                    |
+-----------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Summary

And that's all you need if you're just creating a database and a user. We were a bit long on concepts there but that should give you a solid grounding from which to learn more. Good work.

In the next article we'll do some basic security and stability checks by looking at the MySQL server's configuration files and a few key tools.

--Jered

</body> </html>

Configuring MySQL server on Fedora

2011-03-10T02:39:00Z

We continue our MySQL server setup for Fedora by looking at configuration options to try and ensure the server doesn’t just run, but runs smoothly.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-config-fedora.css"> <title> Beyond the defaults</title></head> <body>

Beyond the defaults

In the previous article we covered a basic MySQL server setup on Fedora Linux. We set the root password, created a database, and created a user for the database. Now let's look at MySQL in a little more detail so we can tweak its configuration and be ready in case something goes wrong.

Finding the config files

By default you'll find MySQL's configuration file at:

/etc/my.cnf

If it's not there, however, you can ask mysqld where it looks for its config. Run the command:

/usr/libexec/mysqld --help --verbose

What we're looking for shows up near the start of the output. Find a couple lines that look like:

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf

And there we are. The server works down that list until it finds a configuration file.

my.cnf

With the location in hand, open the my.cnf file and have a look inside.

/etc/my.cnf

Config groups

There's a bit more to it, but basically: The "client" config section controls the mysql client, and the "mysqld" section controls the server config.

Log files

If something does go wrong the best place to start troubleshooting any program is its logs. By default MySQL stores its log files in the directory:

/var/log

You may need to use sudo to get a listing of the files in that directory.

If you don't find the MySQL logs in the default directory you'll need to check MySQL's config. Look in the my.cnf file and look for a "log_error" line, as in:

log_error = /var/log/mysql/error.log

Network settings

The port in the client section tells the client what port to connect to by default. You'll generally want the two port settings to match up.

If you don't see the port entries in the config file that just means they're using the default. If you want to change the port you would add the lines in the appropriate categories:

[client]
port = 3306
 
[mysqld]
port = 3306

If you don't see a "bind-address" entry you should put one into the "mysqld" category to help control access to the server:

[mysqld]
bind-address = 127.0.0.1

Remember to account for the client's hostname when you set up your database users and to poke a hole in your firewall if you're running iptables.

mysqld and mysqld_safe

Both mysqld and mysqld_safe will read config entries in the "mysqld" section. If you include a "mysqld_safe" section, then only mysqld_safe will read those values in.

By default the mysql service launches "mysqld_safe". That's a good thing, and you should only look to change that if you really know what you're doing.

mysqladmin

Backups

You have a few options when it comes to making backups of your databases apart from the usual "back up the whole machine" approach. The main two are copying the database files and using mysqldump.

File copy

By default MySQL creates a directory for each database in its data directory:

/var/lib/mysql

To make sure the database files are copied cleanly you can shut the MySQL server down entirely before the copy. That's safe but isn't always ideal.

Lock the databases to read-only by running, from the command line:

mysql -u root -p -e "FLUSH TABLES WITH READ LOCK;"

To unlock the database when you're done, run:

mysql -u root -p -e "UNLOCK TABLES;"

We're using a new option with the mysql client, "-e". That tells the client to run the query in quotes as if we'd entered it in the mysql shell proper.

Note that if you're setting these commands up in a script you can put the password in quotes right after "-p" with no space between the two, as in:

mysql -u root -p"password" -e "FLUSH TABLES WITH READ LOCK;"
mysql -u root -p"password" -e "UNLOCK TABLES;"

Just make sure you set the permissions on that file to restrict read access. We don't want just anyone to be able to see that password.

mysqldump

The statements generated by mysqldump go straight to standard output. You'll want to specify a file to redirect the output to when you run it. For example:

mysqldump -u root -p demodb > dbbackup.sql

Restore from mysqldump

Restoring a mysqldumped database looks similar to what was used to create it, but we use plain old "mysql" instead of "mysqldump":

mysql -u root -p demodb < dbbackup.sql

Database engine

The engine type is set when a table is created. Tables are usually created by the application that's going to use them, which is why we aren't going to get into that syntax here.

To see the engine used by your database's tables you can run the following command in the MySQL shell:

SHOW TABLE STATUS FROM demodb;

Change "demodb" to the name of your database.

Choosing an engine

If you have a database administrator, do whatever he or she says. They're smart people, they know what they're talking about.

MyISAM

InnoDB

InnoDB is more fault-tolerant than MyISAM and handles crashes and recovery with a much smaller chance of database corruption. This is a good thing.

Some settings to get you started with InnoDB on a shared server with 256 megs of RAM are:

innodb_buffer_pool_size = 32M
innodb_log_file_size = 8M
innodb_thread_concurrency = 8
innodb_file_per_table

Summary

Now you should have MySQL configured for your environment, and might even have accounted for the database engine being used by your tables.

In the next article we'll look at some basic tasks you can run in the mysql shell to manipulate and observe your database at a high level.

--Jered

</body> </html>

Installing MySQL Server on Fedora

2011-03-10T02:38:00Z

We look at installing MySQL on Fedora and getting it running with a database and a user to access it.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"><html> <head> <meta> <meta> <meta> <link href="mysql-install-fedora.css"> <title> Meet MySQL</title></head> <body>

Meet MySQL

MySQL is an open-source relational database. In a nutshell, for those unfamiliar with it: A database is where an application keeps its stuff.

This article covers a basic installation of a MySQL server on Fedora Linux - just enough to get you started. Remember that you might need to install other packages to let apps use MySQL, like extensions for PHP. Check your application documentation for details.

Installing MySQL

The easiest way to install the MySQL server is through the Fedora package manager:

sudo yum install mysql-server
service mysqld start

We'll talk about how to manually change the root password and remove an anonymous user later, but to do it the easy way run this command now:

sudo /usr/bin/mysql_secure_installation

iptables

If you do need to open a port (again, only if you're accessing MySQL from a different machine from the one you're installing on), you can use the following rules in iptables to open port 3306:

-I INPUT -p tcp --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-I OUTPUT -p tcp --sport 3306 -m state --state ESTABLISHED -j ACCEPT

Launch MySQL

Now that MySQL is installed you can make sure it's running by trying to launch it:

sudo service mysqld start

If you see a message that it's already running that's okay (it means that, well, it's already running).

Launching at boot

Next we'll run the command:

sudo chkconfig mysqld on

That makes sure your machine will launch the MySQL server when it reboots.

The MySQL shell

There is more than one way to manage a MySQL server, so we'll focus on the most basic and compatible approach: The mysql shell.

At the command prompt, run:

/usr/bin/mysql -u root -p

You should be greeted by the mysql shell prompt:

mysql>

Setting the root password

UPDATE mysql.user SET Password = PASSWORD('password') WHERE User = 'root';

It's kind of a mouthful. The reason for this is that MySQL keeps user data in its own database, so to change the password we have to run an SQL command to update the database.

Next we'll reload the stored user information to make our change go into effect:

FLUSH PRIVILEGES;

Looking at users

If you want to see what users are set up in MySQL you need to run a query against the "user" table in the "mysql" database. Let's do that now:

SELECT User, Host, Password FROM mysql.user;

Breaking that down...

The "SELECT" command tells MySQL you're asking for data.

Finally, the "FROM mysql.user" part of the command tells MySQL to get the data from the "mysql" database and the "user" table.

And then the command ends with a semicolon

About that semicolon

All SQL queries end with a semicolon. MySQL will wait for you to keep entering additions to a query until it sees a semicolon.

That means that you can break lines up into smaller parts to make them easier to read. For example, the above command also works if you enter it in multiple lines in the mysql shell, as in:

mysql> SELECT User, Host, Password
    -> FROM mysql.user;

User hosts

Okay, with that important aside about the semicolon done, let's look at the output of that query:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
|                  | %         |                                           |
+------------------+-----------+-------------------------------------------+

If your application connects remotely the "host" entry MySQL will look for is the IP address or DNS hostname of the remote machine (the one the client is coming from).

Anonymous users

Create a database

That covers some basic concepts surrounding users, so now let's look at creating a database.

To create a database, log into the mysql shell and run:

CREATE DATABASE demodb;

That's all there is to it. Replace "demodb" with the name of the database you want to create, of course. You can make sure it's there by running a query to list all databases:

SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| demodb             |
| mysql              |
+--------------------+
3 rows in set (0.00 sec)

Add a database user

To make the user run the following in the mysql shell:

INSERT INTO mysql.user (User,Host,Password) VALUES('demouser','localhost',PASSWORD('demopassword'));

When you make changes to the user table in the mysql database you need to tell MySQL to read the changes by flushing the privileges. To wit:

FLUSH PRIVILEGES;

You can make sure the user is there by running that "select" query again:

SELECT User, Host, Password FROM mysql.user;
+------------------+-----------+-------------------------------------------+
| User             | Host      | Password                                  |
+------------------+-----------+-------------------------------------------+
| root             | localhost | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | demohost  | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| root             | 127.0.0.1 | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |
| demouser         | localhost | *0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6 |
+------------------+-----------+-------------------------------------------+

Grant database user permissions

Right now our new user has no privileges. It can be used to log on, but it can't be used to make any database changes. Let's give it full permissions for our new database by running:

GRANT ALL PRIVILEGES ON demodb.* to demouser@localhost;

And follow it up with the usual:

FLUSH PRIVILEGES;

To check those privileges were set, we'll run:

SHOW GRANTS FOR 'demouser'@'localhost';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for demouser@localhost                                                                                   |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'demouser'@'localhost' IDENTIFIED BY PASSWORD '*0756A562377EDF6ED3AC45A00B356AAE6D3C6BB6' |
| GRANT ALL PRIVILEGES ON `demodb`.* TO 'demouser'@'localhost'                                                    |
+-----------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Summary

And that's all you need if you're just creating a database and a user. We were a bit long on concepts there but that should give you a solid grounding from which to learn more. Good work.

In the next article we'll do some basic security and stability checks by looking at the MySQL server's configuration files and a few key tools.

--Jered

</body> </html>

Slicehost Articles - Home

Slicehost Maintenance and Upgrades FAQ

What does this mean for me in one sentence?

Do I need to do anything?

Why are migrations required?

Can I postpone or opt out?

Will migrations change my IP addresses?

How do I perform an automated self-migration?

How is a migration performed?

How long will my slice be down during a migration?

Will a migration affect the configuration of my slice?

How do I know when a slice has been successfully migrated?

What about my other slices not listed in the maintenance notification?

What happens if I don’t self-migrate?

Will migrations be required again in the future?

What should I do if I still have questions?

STL to ORD Migration FAQ

MIGRATION WINDOWS PERIOD

IMPORTANCE OF SELF-MIGRATION

MIGRATION PROCESS OVERVIEW

MIGRATION STEP-BY-STEP PROCESS

FAQs

STL to ORD Migration Best Practices

The migration

Phase 1

Phase 2

Mitigation

Advance preparation

Establish a log rotation policy

Prune and archive old data

Check for large files

Right before the migration

Use public IP addresses

Force log rotations

Lock databases

Flush application caches

After the migration

Further reading

Install Nagios3 on Debian 5.0 Lenny

Configure nginx

Checking a server’s SSH host fingerprint with the web console

Explaining the host key

Why the host key might change

A dire warning

How to check

The web console

In the SliceManager

In the console

The best way

The not-best way

Write it down

Completing the connection - maybe

First-time connection

Host key has changed

Linux and Mac OS X

Windows and PuTTY

Summary

Choosing a Linux Distribution

Choosing a Linux distribution

So many options

Primary considerations

Quick rundown

The distributions

Ubuntu

CentOS

Debian

RHEL

Fedora

Gentoo

Arch

Summary

Installing MySQL Server

What is MySQL?

Contents

Distribution links

Ubuntu

CentOS

Debian

Fedora

RHEL