Avatar

Michael's Blog

Home | RSS | Gallery | Stats | About | Comics | Downloads | Links | Scripts | Fuel Stats | Test Site | Wiki |
Quote:
I would rather you fall off the ladder then break a sprinkler head.
-- Matthew Hill     Add quote.

Selecting random records in postgresql

2015-08-18 09:15:00 by Michael 0 Comments
Tags: postgresql sql code

In an attempt to make my site more responsive I have been working on optimizing the SQL code used on the backend, this includes the random quote generator that I have set up on the main page.

The old code used a query similar to below.

SELECT quote, name FROM quotes ORDER BY RANDOM() LIMIT 1

This works fine if you have small tables and fast disks but consider the issue when there is a table with millions of rows. To find *one* record the server must read through the table, sort the records, and then discard every result but one. This operation is slow and inefficient.

To improve performance you can reduce the number of rows read by using a primary key on the table. Each row has a unique ID number which can then be used as the limit for the random() function. For example, the following query will select a random record based on the last index number created:

SELECT quote_text, name FROM quotes WHERE quote_id = (SELECT floor(random() * (SELECT last_value from quotes_quote_id_seq)+1)) ;

This query is not perfect and may result in empty results which your code will need to accommodate for but it is still more efficient than reading the entire table every time the page is loaded.

Prevent iptables from spamming your console

2015-08-14 13:41:00 by Michael 2 Comments
Tags: linux kernel iptables sysadmin netfilter

How to disable firewall "spam" on your console.

I worked on a ticket recently for a customer concerned about firewall messages being sent to every user's console by the kernel. After doing a bit of research I discovered that the nf_ct_ftp module logs messages to syslog as *emergency* level by default which results in every console being spammed by firewall messages. To prevent this you can make a few simple changes as follows.

First, set up a custom rsyslog conf file to send iptables messages to a different file.

cat << EOF > /etc/rsyslog.d/iptables.conf 
:msg, contains, "nf_ct_ftp:" -/var/log/iptables.log
& ~
EOF

The first line means send all messages that contain the “nf_ct_ftp:” string to /var/log/iptables.log. The second line causes rsyslog to discard messages that were matched on the previous line. Adjust this rule according to your needs.

Second, update sysctl.conf with the following lines and then run "sysctl -p".

kernel.printk = 4 4 1 7

sysctl -p

See https://www.kernel.org/doc/Documentation/sysctl/kernel.txt for a description of these values.

Now restart rsyslog and test your changes using the "logger" command.

service rsyslog restart
logger -p kern.emerg -t kernel "nf_ct_ftp: dropping packet test"

You should not see anything on the console. cat /var/log/iptables.log to confirm that the entry was logged properly. After you have confirmed that the messages are being logged properly you can set up logrotate to manage the logs. Create a config file to do this similar to below.

cat << EOF > /etc/logrotate.d/iptables 
/var/log/iptables.log
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d rsyslog rotate > /dev/null
	endscript
}
EOF

There is nothing else to do at this point.

How to Manually Change Domain in Magento

2014-10-10 10:20:00 by Michael 0 Comments
Tags: magento linux mysql sysadmin

Changing the domain name on a Magento install requires a few steps to update the site URL in mysql. The procedure should be similar to below.

Update your core_config_data table to edit the two records for web/unsecure/base_url and web/secure/base_url

mysql 

mysql> update core_config_data set value = 'http://dev.example.com/' where path = 'web/unsecure/base_url';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
 
mysql> update core_config_data set value = 'http://dev.example.com/' where path = 'web/secure/base_url';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

After this is done delete the contents of WEBROOT/var/cache. The location of the WEBROOT varies depending on how your server is set up.

cd /home/username/public_html/var/
rm -rf ./cache/*

Update any .htaccess redirects you may have added.

That's it, you're done! Open the site in a new browser tab to make sure that everything loads properly.

How to fix "Your profile could not be loaded" error in Google Chrome

2014-03-20 21:04:00 by Michael 0 Comments
Tags: chrome linux sqlite

If you get an error from chrome stating that your profile could not be loaded properly here is the PROPER way to fix the issue. Unfortunately googling for this error leads to a lot of false information and speculation.

First, go to your profile's data directory. In Linux this would be ~/.config/google-chrome/Default.

Now check for any processes that have the Web Data file open.

lsof Web\ Data

Kill those processes.

Next run an integrity check on the database.

sqlite3 Web\ Data "pragma integrity_check"

This should repair any errors in the file. After that is done start up chrome.

Canola Soap Recipe

2013-09-15 09:47:00 by Michael 0 Comments
Tags: soap chemistry homesteading crafts

Here is a simple canola/soybean oil soap recipe. Yield is approximately 8 lbs of soap. Lye amount is based on a saponification value of 0.130. See http://www.millersoap.com/soapdesign.html#SAP Tables for more details.

Canola/Soybean Soap

Ingredients: Preparation:
  1. Dissolve lye into water. Prepare this mixture in advance, the water will take time to cool.
  2. Pour oil into large pot, heat to 130 degrees.
  3. Stir lye mixture into the oil.
  4. Blend with a stick blender until you see signs of tracing.
  5. Pour into molds and let harden.

Raw soap will take 24-48 hours to harden, after that the soap can be removed from the mold to cure. Allow 30 days cure time.

Manually create a Wordpress admin user from the mysql command line

2013-05-01 16:06:00 by Michael 0 Comments
Tags: linux mysql wordpress

If you need admin access to a wordpress install you can easily create a new admin user by running a few SQL commands on the database. This has been tested and verified to work on Wordpress 3.5.

To do this you will first need to identify what database the site is actually using. Check wp-config.php for the database name and mysql host info. Once you have that connect to mysql and run the following statements.

INSERT INTO wp_users (user_login,user_pass,user_email,user_registered,user_status) VALUES("user_name",md5('password'),"username@example.com",NOW(),0);

Find user ID from wp_users table:

SET @user_id = (SELECT ID FROM wp_users where user_login = 'user_name');

INSERT INTO wp_usermeta (user_id,meta_key,meta_value) VALUES (@user_id,"wp_user_level","10");

INSERT INTO wp_usermeta (user_id,meta_key,meta_value) VALUES (@user_id,"wp_capabilities",'a:1:{s:13:"administrator";s:1:"1";}');

Storm VPS Lustre Benchmarks

2013-04-11 13:56:00 by Michael 0 Comments
Tags: linux sysadmin lustre storage

After reading about various cluster file systems I decided to set up a small cluster running Lustre using Storm VPS instances. All nodes have the same hardware configuration and use a 50 GB SAN volume connected through iSCSI as the lustre block device. Specs are as follows.

Node configuration:

OS: CentOS 6.3 x86_64
Kernel: 2.6.32-279.19.1.el6_lustre.x86_64
RAM: 3556 MB (Storm 4 GB)
Primary Disk: 300 GB virtual disk
Secondary Disk (iscsi): 50 GB SAN volume
CPU: Two Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz cores

Lustre configuration: 1 management server, 1 metadata server, 1 object storage server.  LNET was configured to use a private network interface.

Disk performance was tested with the sgpdd_survey script from the Lustre IOkit. Write speed appears to average around 35-40 MB/s.

Wed Apr 10 10:29:39 EDT 2013 sgpdd-survey on /dev/sda from oss1.watters.ws
total_size  8388608K rsz 1024 crg     1 thr     1 write   49.32 MB/s     1 x  49.32 =   49.32 MB/s read   68.15 MB/s     1 x  68.15 =   68.15 MB/s
total_size  8388608K rsz 1024 crg     1 thr     2 write   77.15 MB/s     1 x  77.15 =   77.15 MB/s read   92.85 MB/s     1 x  92.85 =   92.85 MB/s
total_size  8388608K rsz 1024 crg     1 thr     8 write   36.15 MB/s     1 x  36.14 =   36.14 MB/s read   94.08 MB/s     1 x  94.09 =   94.09 MB/s
total_size  8388608K rsz 1024 crg     1 thr    16 write   35.84 MB/s     1 x  35.85 =   35.85 MB/s read  101.59 MB/s     1 x 101.59 =  101.59 MB/s
total_size  8388608K rsz 1024 crg     2 thr     2 write   35.34 MB/s     2 x  17.67 =   35.34 MB/s read   67.38 MB/s     2 x  33.69 =   67.39 MB/s
total_size  8388608K rsz 1024 crg     2 thr     4 write   39.09 MB/s     2 x  19.55 =   39.10 MB/s read   79.20 MB/s     2 x  39.60 =   79.19 MB/s
total_size  8388608K rsz 1024 crg     2 thr     8 write   40.40 MB/s     2 x  20.20 =   40.40 MB/s read   98.16 MB/s     2 x  49.09 =   98.17 MB/s
total_size  8388608K rsz 1024 crg     2 thr    16 write   37.73 MB/s     2 x  18.86 =   37.73 MB/s read   99.31 MB/s     2 x  49.66 =   99.32 MB/s
total_size  8388608K rsz 1024 crg     2 thr    32 write   38.08 MB/s     2 x  19.04 =   38.07 MB/s read   97.30 MB/s     2 x  48.66 =   97.31 MB/s
total_size  8388608K rsz 1024 crg     4 thr     4 write   38.38 MB/s     4 x   9.59 =   38.38 MB/s read   98.17 MB/s     4 x  24.55 =   98.19 MB/s
total_size  8388608K rsz 1024 crg     4 thr     8 write   38.25 MB/s     4 x   9.57 =   38.26 MB/s read  100.06 MB/s     4 x  25.01 =  100.06 MB/s
total_size  8388608K rsz 1024 crg     4 thr    16 write   39.42 MB/s     4 x   9.85 =   39.41 MB/s read   99.96 MB/s     4 x  25.00 =   99.98 MB/s
total_size  8388608K rsz 1024 crg     4 thr    32 write   39.43 MB/s     4 x   9.86 =   39.44 MB/s read   99.93 MB/s     4 x  24.99 =   99.95 MB/s
total_size  8388608K rsz 1024 crg     4 thr    64 write   38.22 MB/s     4 x   9.56 =   38.22 MB/s read   97.80 MB/s     4 x  24.45 =   97.81 MB/s
total_size  8388608K rsz 1024 crg     8 thr     8 write   38.73 MB/s     8 x   4.84 =   38.76 MB/s read   87.71 MB/s     8 x  10.97 =   87.74 MB/s
total_size  8388608K rsz 1024 crg     8 thr    16 write   39.70 MB/s     8 x   4.96 =   39.67 MB/s read   81.09 MB/s     8 x  10.14 =   81.10 MB/s
total_size  8388608K rsz 1024 crg     8 thr    32 write   43.40 MB/s     8 x   5.43 =   43.41 MB/s read   81.21 MB/s     8 x  10.16 =   81.25 MB/s
total_size  8388608K rsz 1024 crg     8 thr    64 write   38.88 MB/s     8 x   4.86 =   38.91 MB/s read   67.10 MB/s     8 x   8.39 =   67.14 MB/s
total_size  8388608K rsz 1024 crg     8 thr   128 write   42.19 MB/s     8 x   5.27 =   42.19 MB/s read   65.92 MB/s     8 x   8.24 =   65.92 MB/s

IOPS performance was tested using iozone, here are the results.

	OPS Mode. Output is in operations per second.
	Include fsync in write timing
	No retest option selected
        Record Size 4 KB
        File size set to 4194304 KB
        Command line used: iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 32
        Max process = 32
        Throughput test with 32 processes
        Each process writes a 4194304 Kbyte file in 4 Kbyte records

        Children see throughput for 32 initial writers  =   27764.87 ops/sec
        Parent sees throughput for 32 initial writers   =   26692.16 ops/sec
        Min throughput per process                      =     840.07 ops/sec
        Max throughput per process                      =     903.35 ops/sec
        Avg throughput per process                      =     867.65 ops/sec
        Min xfer                                        =  975918.00 ops

        Children see throughput for 32 readers          =   26758.37 ops/sec
        Parent sees throughput for 32 readers           =   26755.12 ops/sec
        Min throughput per process                      =     448.79 ops/sec
        Max throughput per process                      =    1372.74 ops/sec
        Avg throughput per process                      =     836.20 ops/sec
        Min xfer                                        =  342845.00 ops

As you can see lustre is a relatively high performance file system and is easily scalable to store petabytes of data. Adding more space is as simple as building a new object server and running mkfs.lustre.

create an rpm mirror using wget

2013-04-10 13:35:00 by Michael 0 Comments
Tags: wget linux sysadmin scripting

If you want to set up a yum repo you can easily mirror an existing site using wget. To do this you will need to run this command.

wget --mirror -np --no-host-directories -A rpm,srpm http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/

In this case we are mirroring the lustre rpm repo.

After the files are downloaded you can run the createrepo command to create yum metadata.

quickly remove old ssh keys with sed

2012-04-24 10:13:00 by Michael 0 Comments
Tags: sed linux ssh sysadmin

If you work on a lot of servers and do a lot of reinstalls you will see the following error often.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
a8:6a:60:5a:48:64:ac:90:33:b9:f2:7c:be:56:92:81.
Please contact your system administrator.
Add correct host key in /var/root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:9948
RSA host key for host.example.com has changed and you have requested strict checking.
Host key verification failed.

To save some time you can quickly remove the old host key with a single sed command:

sed -i '9948d' .ssh/known_hosts

Building the wl module on linux 3.2

2012-04-02 21:57:00 by Michael 0 Comments
Tags: linux kernel hardware hacking

After upgrading my netbook kernel to the latest stable version available on backports.org I soon discovered that my wireless interface no longer worked. Trying to rebuild the module resulted in the following error:

/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c: In function ‘_wl_set_multicast_list’:
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1435: error: ‘struct net_device’ has no member named ‘mc_list’
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1435: error: ‘struct net_device’ has no member named ‘mc_count’
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1436: error: dereferencing pointer to incomplete type
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1442: error: dereferencing pointer to incomplete type
make[4]: *** [/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.o] Error 1
make[3]: *** [_module_/usr/src/modules/broadcom-sta/amd64] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [all] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.2.0-0.bpo.2-amd64'
make: *** [all] Error 2
root@netbook:/usr/src/modules/broadcom-sta/amd64#  run "make API=WEXT"
bash: run: command not found
root@netbook:/usr/src/modules/broadcom-sta/amd64# "make API=WEXT"
bash: make API=WEXT: command not found
root@netbook:/usr/src/modules/broadcom-sta/amd64# make API=WEXT
KBUILD_NOPEDANTIC=1 make -C /lib/modules/`uname -r`/build M=`pwd`
make[1]: Entering directory `/usr/src/linux-headers-3.2.0-0.bpo.2-amd64'
  CC [M]  /usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.o
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c: In function ‘_wl_set_multicast_list’:
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1435: error: ‘struct net_device’ has no member named ‘mc_list’
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1435: error: ‘struct net_device’ has no member named ‘mc_count’
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1436: error: dereferencing pointer to incomplete type
/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.c:1442: error: dereferencing pointer to incomplete type
make[4]: *** [/usr/src/modules/broadcom-sta/amd64/src/wl/sys/wl_linux.o] Error 1
make[3]: *** [_module_/usr/src/modules/broadcom-sta/amd64] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [all] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.2.0-0.bpo.2-amd64'
make: *** [all] Error 2

A bit of googling lead me to a few patches that helped solve the issue. Here is a unified diff of my changes which should allow you to cleanly build and install the wl module using module-assistant.

http://www.watters.ws/broadcom_bcm4313_linux3.2.patch

One thing to note is that the source code needs to be patched BEFORE you run m-a, i.e. cd to /usr/src/modules/broadcom_sta/amd64/src/wl/sys and run patch the patch from there.

I hope that somebody will find this useful.

Happy Thanksgiving!

2009-11-26 13:16:12.582753 by Michael 0 Comments
Tags: thanksgiving ascii shell

Hard to believe it's been a year since I posted this.

                              ,+*^^*+___+++_
                         ,*^^^^              )
                      _+*                     ^**+_
                    +^       _ _++*+_+++_,         )
        _+^^*+_    (     ,+*^ ^          \+_        )
       {       )  (    ,(    ,_+--+--,      ^)      ^
      { (@)    } f   ,(  ,+-^ __*_*_  ^^\_   ^\       )
     {:;-/    (_+*-+^^^^^+*+*<_ _++_)_    )    )      /
    ( /  (    (        ,___    ^*+_+* )   <    <
     U _/     )    *--<  ) ^\-----++__)   )    )       )
      (      )  _(^)^^))  )  )\^^^^^))^*+/    /       /
    (      /  (_))_^)) )  )  ))^^^^^))^^^)__/     +^^
   (     ,/    (^))^))  )  ) ))^^^^^^^))^^)       _)
    *+__+*       (_))^)  ) ) ))^^^^^^))^^^^^)____*^
    \             \_)^)_)) ))^^^^^^^^^^))^^^^)
     (_             ^\__^^^^^^^^^^^^))^^^^^^^)
       ^\___            ^\__^^^^^^))^^^^^^^^)
            ^^^^^\uuu/^^\uuu/^^^^\^\^\^\^\^\^\^
               ___) >____) >___   ^\_\_\_\_\_\_\)
              ^^^//\_^^//\_^       ^(\_\_\_\)
                ^^^ ^^ ^^^ ^^


 #    #    ##    #####   #####    #   #
 #    #   #  #   #    #  #    #    # #
 ######  #    #  #    #  #    #     #
 #    #  ######  #####   #####      #
 #    #  #    #  #       #          #
 #    #  #    #  #       #          #


  #####  #    #    ##    #    #  #    #   ####
    #    #    #   #  #   ##   #  #   #   #
    #    ######  #    #  # #  #  ####     ####
    #    #    #  ######  #  # #  #  #         #
    #    #    #  #    #  #   ##  #   #   #    #
    #    #    #  #    #  #    #  #    #   ####


  ####      #    #    #     #    #    #   ####
 #    #     #    #    #     #    ##   #  #    #
 #          #    #    #     #    # #  #  #
 #  ###     #    #    #     #    #  # #  #  ###
 #    #     #     #  #      #    #   ##  #    #
  ####      #      ##       #    #    #   ####

Download the script here: http://linuxshellaccount.blogspot.com/2008/11/happy-thanksgiving-unixlinux-ascii-art.html

Ubuntu Automatic Updates

2009-09-14 00:38:05.296489 by Michael 0 Comments
Tags: ubuntu linux

If you're sick of the update notifier bugging you on your Ubuntu desktop you can easily set up a cron job to automatically take care of things.

sudo crontab -e
0 5 * * * apt-get -y upgrade

Change the time to whenever you want.