Shard-Query blog

The only open source MPP database engine for MySQL

TokuDB vs Percona XtraDB using Tokutek’s MariaDB distribution

Following are benchmark results comparing Tokutek TokuDB and Percona XtraDB at scale factor 10 on the Star Schema benchmark. I’m posting this on the Shard-Query blog because I am going to compare the performance of Shard-Query on the benchmark on these two engines. First, however, I think it is important to see how they perform in isolation without concurrency.

Because I am going to be testing Shard-Query, I have chosen to partition the “fact” table (lineorder) by month. I’ve attached the full DDL at the end of the post as well as the queries again for reference.

I want to note a few things about the results:
First and foremost, TokuDB was configured to use quicklz compression (the default) and InnoDB compression was not used. No tuning of TokuDB was performed, which means it will use up to 50% of memory by default. Various InnoDB tuning options were set (see the end of the post) but the most important is that the innodb_buffer_pool_size which was set to 20G (all data fits in buffer pool for hot test).

Size of data after loading (note this is the size ON DISK):

  • TokuDB: 3.6GB
  • InnoDB: 9.2GB

I collected statistics on the tables with ANALYZE TABLE. I verified that the query plans were the same for both storage engines.

Here are the results of the tests. Click a graph to zoom in:


image001

image009

image003

image011

image005

image013

image007

image015

The tests were done on the following benchmark machine:
Intel i7-970 (12 virtual cores)
24GB DDR3-1600MHz
LSI 9211-8i SATA III HBA
Two Intel 520 120MB SSD and one OCZ Vertex 3 120GB SSD in Software RAID 0.

Software:
I used the MariaDB 5.5.30 distribution from Tokutek: version 5.5.30-tokudb-7.0.1-MariaDB.

my.cnf:

[mysqld]
datadir=/data/datadirs/mariadb
basedir=/data/mariadb
socket=/var/lib/mysql/mysql.sock
user=mysql

innodb_buffer_pool_size=20G
innodb_log_file_size=4G
innodb_file_per_table
innodb_file_format=barracuda
innodb_buffer_pool_instances=6
innodb_write_io_threads=12
innodb_read_io_threads=12
innodb_flush_method=O_DIRECT
innodb_io_capacity=10000

innodb_flush_log_at_trx_commit=2
innodb_log_buffer_size=32M

read_buffer_size=2M
key_buffer_size=32M

innodb_fast_checksum

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

DDL:

CREATE TABLE `lineorder` (
  `LO_OrderKey` int(10) unsigned NOT NULL,
  `LO_LineNumber` tinyint(4) NOT NULL,
  `LO_CustKey` mediumint(9) NOT NULL,
  `LO_PartKey` mediumint(9) NOT NULL,
  `LO_SuppKey` mediumint(9) NOT NULL,
  `LO_OrderDateKey` int(11) NOT NULL,
  `LO_OrderPriority` varchar(15) DEFAULT NULL,
  `LO_ShipPriority` char(1) DEFAULT NULL,
  `LO_Quantity` tinyint(4) DEFAULT NULL,
  `LO_ExtendedPrice` decimal(10,0) DEFAULT NULL,
  `LO_OrdTotalPrice` decimal(10,0) DEFAULT NULL,
  `LO_Discount` decimal(10,0) DEFAULT NULL,
  `LO_Revenue` decimal(10,0) DEFAULT NULL,
  `LO_SupplyCost` decimal(10,0) DEFAULT NULL,
  `LO_Tax` tinyint(4) DEFAULT NULL,
  `LO_CommitDateKey` int(11) NOT NULL,
  `LO_ShipMode` varchar(10) DEFAULT NULL,
  KEY `LO_CustKey` (`LO_CustKey`),
  KEY `LO_OrderDateKey` (`LO_OrderDateKey`),
  KEY `LO_SuppKey` (`LO_SuppKey`),
  KEY `LO_PartKey` (`LO_PartKey`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (LO_OrderDateKey)
(PARTITION p1992_01 VALUES LESS THAN (19920131) ENGINE = TokuDB,
 PARTITION p1992_02 VALUES LESS THAN (19920231) ENGINE = TokuDB,
 PARTITION p1992_03 VALUES LESS THAN (19920331) ENGINE = TokuDB,
 PARTITION p1992_04 VALUES LESS THAN (19920431) ENGINE = TokuDB,
 PARTITION p1992_05 VALUES LESS THAN (19920531) ENGINE = TokuDB,
 PARTITION p1992_06 VALUES LESS THAN (19920631) ENGINE = TokuDB,
 PARTITION p1992_07 VALUES LESS THAN (19920731) ENGINE = TokuDB,
 PARTITION p1992_08 VALUES LESS THAN (19920831) ENGINE = TokuDB,
 PARTITION p1992_09 VALUES LESS THAN (19920931) ENGINE = TokuDB,
 PARTITION p1992_10 VALUES LESS THAN (19921031) ENGINE = TokuDB,
 PARTITION p1992_11 VALUES LESS THAN (19921131) ENGINE = TokuDB,
 PARTITION p1992_12 VALUES LESS THAN (19921231) ENGINE = TokuDB,
 PARTITION p1993_01 VALUES LESS THAN (19930131) ENGINE = TokuDB,
 PARTITION p1993_02 VALUES LESS THAN (19930231) ENGINE = TokuDB,
 PARTITION p1993_03 VALUES LESS THAN (19930331) ENGINE = TokuDB,
 PARTITION p1993_04 VALUES LESS THAN (19930431) ENGINE = TokuDB,
 PARTITION p1993_05 VALUES LESS THAN (19930531) ENGINE = TokuDB,
 PARTITION p1993_06 VALUES LESS THAN (19930631) ENGINE = TokuDB,
 PARTITION p1993_07 VALUES LESS THAN (19930731) ENGINE = TokuDB,
 PARTITION p1993_08 VALUES LESS THAN (19930831) ENGINE = TokuDB,
 PARTITION p1993_09 VALUES LESS THAN (19930931) ENGINE = TokuDB,
 PARTITION p1993_10 VALUES LESS THAN (19931031) ENGINE = TokuDB,
 PARTITION p1993_11 VALUES LESS THAN (19931131) ENGINE = TokuDB,
 PARTITION p1993_12 VALUES LESS THAN (19931231) ENGINE = TokuDB,
 PARTITION p1994_01 VALUES LESS THAN (19940131) ENGINE = TokuDB,
 PARTITION p1994_02 VALUES LESS THAN (19940231) ENGINE = TokuDB,
 PARTITION p1994_03 VALUES LESS THAN (19940331) ENGINE = TokuDB,
 PARTITION p1994_04 VALUES LESS THAN (19940431) ENGINE = TokuDB,
 PARTITION p1994_05 VALUES LESS THAN (19940531) ENGINE = TokuDB,
 PARTITION p1994_06 VALUES LESS THAN (19940631) ENGINE = TokuDB,
 PARTITION p1994_07 VALUES LESS THAN (19940731) ENGINE = TokuDB,
 PARTITION p1994_08 VALUES LESS THAN (19940831) ENGINE = TokuDB,
 PARTITION p1994_09 VALUES LESS THAN (19940931) ENGINE = TokuDB,
 PARTITION p1994_10 VALUES LESS THAN (19941031) ENGINE = TokuDB,
 PARTITION p1994_11 VALUES LESS THAN (19941131) ENGINE = TokuDB,
 PARTITION p1994_12 VALUES LESS THAN (19941231) ENGINE = TokuDB,
 PARTITION p1995_01 VALUES LESS THAN (19950131) ENGINE = TokuDB,
 PARTITION p1995_02 VALUES LESS THAN (19950231) ENGINE = TokuDB,
 PARTITION p1995_03 VALUES LESS THAN (19950331) ENGINE = TokuDB,
 PARTITION p1995_04 VALUES LESS THAN (19950431) ENGINE = TokuDB,
 PARTITION p1995_05 VALUES LESS THAN (19950531) ENGINE = TokuDB,
 PARTITION p1995_06 VALUES LESS THAN (19950631) ENGINE = TokuDB,
 PARTITION p1995_07 VALUES LESS THAN (19950731) ENGINE = TokuDB,
 PARTITION p1995_08 VALUES LESS THAN (19950831) ENGINE = TokuDB,
 PARTITION p1995_09 VALUES LESS THAN (19950931) ENGINE = TokuDB,
 PARTITION p1995_10 VALUES LESS THAN (19951031) ENGINE = TokuDB,
 PARTITION p1995_11 VALUES LESS THAN (19951131) ENGINE = TokuDB,
 PARTITION p1995_12 VALUES LESS THAN (19951231) ENGINE = TokuDB,
 PARTITION p1996_01 VALUES LESS THAN (19960131) ENGINE = TokuDB,
 PARTITION p1996_02 VALUES LESS THAN (19960231) ENGINE = TokuDB,
 PARTITION p1996_03 VALUES LESS THAN (19960331) ENGINE = TokuDB,
 PARTITION p1996_04 VALUES LESS THAN (19960431) ENGINE = TokuDB,
 PARTITION p1996_05 VALUES LESS THAN (19960531) ENGINE = TokuDB,
 PARTITION p1996_06 VALUES LESS THAN (19960631) ENGINE = TokuDB,
 PARTITION p1996_07 VALUES LESS THAN (19960731) ENGINE = TokuDB,
 PARTITION p1996_08 VALUES LESS THAN (19960831) ENGINE = TokuDB,
 PARTITION p1996_09 VALUES LESS THAN (19960931) ENGINE = TokuDB,
 PARTITION p1996_10 VALUES LESS THAN (19961031) ENGINE = TokuDB,
 PARTITION p1996_11 VALUES LESS THAN (19961131) ENGINE = TokuDB,
 PARTITION p1996_12 VALUES LESS THAN (19961231) ENGINE = TokuDB,
 PARTITION p1997_01 VALUES LESS THAN (19970131) ENGINE = TokuDB,
 PARTITION p1997_02 VALUES LESS THAN (19970231) ENGINE = TokuDB,
 PARTITION p1997_03 VALUES LESS THAN (19970331) ENGINE = TokuDB,
 PARTITION p1997_04 VALUES LESS THAN (19970431) ENGINE = TokuDB,
 PARTITION p1997_05 VALUES LESS THAN (19970531) ENGINE = TokuDB,
 PARTITION p1997_06 VALUES LESS THAN (19970631) ENGINE = TokuDB,
 PARTITION p1997_07 VALUES LESS THAN (19970731) ENGINE = TokuDB,
 PARTITION p1997_08 VALUES LESS THAN (19970831) ENGINE = TokuDB,
 PARTITION p1997_09 VALUES LESS THAN (19970931) ENGINE = TokuDB,
 PARTITION p1997_10 VALUES LESS THAN (19971031) ENGINE = TokuDB,
 PARTITION p1997_11 VALUES LESS THAN (19971131) ENGINE = TokuDB,
 PARTITION p1997_12 VALUES LESS THAN (19971231) ENGINE = TokuDB,
 PARTITION p1998_01 VALUES LESS THAN (19980131) ENGINE = TokuDB,
 PARTITION p1998_02 VALUES LESS THAN (19980231) ENGINE = TokuDB,
 PARTITION p1998_03 VALUES LESS THAN (19980331) ENGINE = TokuDB,
 PARTITION p1998_04 VALUES LESS THAN (19980431) ENGINE = TokuDB,
 PARTITION p1998_05 VALUES LESS THAN (19980531) ENGINE = TokuDB,
 PARTITION p1998_06 VALUES LESS THAN (19980631) ENGINE = TokuDB,
 PARTITION p1998_07 VALUES LESS THAN (19980731) ENGINE = TokuDB,
 PARTITION p1998_08 VALUES LESS THAN (19980831) ENGINE = TokuDB,
 PARTITION p1998_09 VALUES LESS THAN (19980931) ENGINE = TokuDB,
 PARTITION p1998_10 VALUES LESS THAN (19981031) ENGINE = TokuDB,
 PARTITION p1998_11 VALUES LESS THAN (19981131) ENGINE = TokuDB,
 PARTITION p1998_12 VALUES LESS THAN (19981231) ENGINE = TokuDB) */;

CREATE TABLE `customer` (
  `C_CustomerKey` mediumint(9) NOT NULL,
  `C_Name` varchar(25) DEFAULT NULL,
  `C_Address` varchar(25) DEFAULT NULL,
  `C_City` varchar(10) DEFAULT NULL,
  `C_Nation` varchar(15) DEFAULT NULL,
  `C_Region` varchar(12) DEFAULT NULL,
  `C_Phone` varchar(15) DEFAULT NULL,
  `C_MktSegment` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`C_CustomerKey`),
  KEY `C_Name` (`C_Name`),
  KEY `C_City` (`C_City`),
  KEY `C_Region` (`C_Region`),
  KEY `C_Phone` (`C_Phone`),
  KEY `C_MktSegment` (`C_MktSegment`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;

CREATE TABLE `part` (
  `P_PartKey` mediumint(9) NOT NULL,
  `P_Name` varchar(25) DEFAULT NULL,
  `P_MFGR` varchar(10) DEFAULT NULL,
  `P_Category` varchar(10) DEFAULT NULL,
  `P_Brand` varchar(15) DEFAULT NULL,
  `P_Colour` varchar(15) DEFAULT NULL,
  `P_Type` varchar(25) DEFAULT NULL,
  `P_Size` tinyint(4) DEFAULT NULL,
  `P_Container` char(10) DEFAULT NULL,
  PRIMARY KEY (`P_PartKey`),
  KEY `P_Name` (`P_Name`),
  KEY `P_MFGR` (`P_MFGR`),
  KEY `P_Category` (`P_Category`),
  KEY `P_Brand` (`P_Brand`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;

CREATE TABLE `supplier` (
  `S_SuppKey` smallint(6) NOT NULL,
  `S_Name` char(25) DEFAULT NULL,
  `S_Address` varchar(25) DEFAULT NULL,
  `S_City` char(10) DEFAULT NULL,
  `S_Nation` char(15) DEFAULT NULL,
  `S_Region` char(12) DEFAULT NULL,
  `S_Phone` char(15) DEFAULT NULL,
  PRIMARY KEY (`S_SuppKey`),
  KEY `S_City` (`S_City`),
  KEY `S_Name` (`S_Name`),
  KEY `S_Phone` (`S_Phone`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;

CREATE TABLE `dim_date` (
  `D_DateKey` int(11) NOT NULL,
  `D_Date` char(18) DEFAULT NULL,
  `D_DayOfWeek` char(9) DEFAULT NULL,
  `D_Month` char(9) DEFAULT NULL,
  `D_Year` smallint(6) DEFAULT NULL,
  `D_YearMonthNum` int(11) DEFAULT NULL,
  `D_YearMonth` char(7) DEFAULT NULL,
  `D_DayNumInWeek` tinyint(4) DEFAULT NULL,
  `D_DayNumInMonth` tinyint(4) DEFAULT NULL,
  `D_DayNumInYear` smallint(6) DEFAULT NULL,
  `D_MonthNumInYear` tinyint(4) DEFAULT NULL,
  `D_WeekNumInYear` tinyint(4) DEFAULT NULL,
  `D_SellingSeason` char(12) DEFAULT NULL,
  `D_LastDayInWeekFl` tinyint(4) DEFAULT NULL,
  `D_LastDayInMonthFl` tinyint(4) DEFAULT NULL,
  `D_HolidayFl` tinyint(4) DEFAULT NULL,
  `D_WeekDayFl` tinyint(4) DEFAULT NULL,
  PRIMARY KEY (`D_DateKey`),
  KEY `D_Year` (`D_Year`,`D_DayNumInYear`),
  KEY `D_YearMonth` (`D_YearMonth`),
  KEY `D_Year_2` (`D_Year`,`D_WeekNumInYear`),
  KEY `D_Year_3` (`D_Year`,`D_WeekNumInYear`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;
Queries:
-- Q1.1
 select sum(lo_extendedprice*lo_discount) as
 revenue
 from lineorder join dim_date on lo_orderdatekey = d_datekey
 where
 d_year = 1993
 and lo_discount between 1 and 3
 and lo_quantity < 25;
-- Q1.2
 select sum(lo_extendedprice*lo_discount) as revenue
 from lineorder
 join dim_date on lo_orderdatekey = d_datekey
 where d_yearmonth = 'Jan1994' and lo_discount
 between 4 and 6 and lo_quantity between 26 and 35;
-- Q1.3
 select sum(lo_extendedprice*lo_discount) as revenue
 from lineorder
 join dim_date on lo_orderdatekey = d_datekey
 where d_weeknuminyear = 6
 and d_year = 1994
 and lo_discount between 5 and 7 and lo_quantity between 26 and 35;
-- Q2.1
 select sum(lo_revenue), d_year, p_brand
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join part
 on lo_partkey = p_partkey join supplier
 on lo_suppkey = s_suppkey
 where p_category = 'MFGR#12'
 and s_region = 'AMERICA'
 group by d_year, p_brand
 order by d_year, p_brand;
-- Q2.2
 select sum(lo_revenue), d_year, p_brand
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join part
 on lo_partkey = p_partkey
 join supplier
 on lo_suppkey = s_suppkey
 where p_brand between 'MFGR#2221' and 'MFGR#2228'
 and s_region = 'ASIA'
 group by d_year, p_brand
 order by d_year, p_brand;
-- Q2.3
 select sum(lo_revenue), d_year, p_brand
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join part
 on lo_partkey = p_partkey
 join supplier
 on lo_suppkey = s_suppkey
 where p_brand= 'MFGR#2239'
 and s_region = 'EUROPE'
 group by d_year, p_brand
 order by d_year, p_brand;
-- Q3.1
 select c_nation, s_nation, d_year, sum(lo_revenue) as revenue
 from customer
 join lineorder
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join dim_date on lo_orderdatekey = d_datekey
 where c_region = 'ASIA'
 and s_region = 'ASIA'
 and d_year >= 1992 and d_year <= 1997
 group by c_nation, s_nation, d_year
 order by d_year asc, revenue desc;
-- Q3.2
 select c_city, s_city, d_year, sum(lo_revenue) as revenue
 from customer
 join lineorder
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join dim_date
 on lo_orderdatekey = d_datekey
 where c_nation = 'UNITED STATES'
 and s_nation = 'UNITED STATES'
 and d_year >= 1992
 and d_year <= 1997
 group by c_city, s_city, d_year
 order by d_year asc, revenue desc;
-- Q3.3
 select c_city, s_city, d_year, sum(lo_revenue) as revenue
 from customer
 join lineorder
 on lo_custkey = c_customerkey
 join supplier on lo_suppkey = s_suppkey
 join dim_date on lo_orderdatekey = d_datekey
 where (c_city='UNITED KI1' or c_city='UNITED KI5')
 and (s_city='UNITED KI1' or s_city='UNITED KI5')
 and d_year >= 1992
 and d_year <= 1997
 group by c_city, s_city, d_year
 order by d_year asc, revenue desc;
-- Q3.4
 select c_city, s_city, d_year, sum(lo_revenue)
 as revenue
 from customer
 join lineorder
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join dim_date
 on lo_orderdatekey = d_datekey
 where
 (c_city='UNITED KI1' or c_city='UNITED KI5')
 and (s_city='UNITED KI1' or s_city='UNITED KI5')
 and d_yearmonth = 'Dec1997'
 group by c_city, s_city, d_year
 order by d_year asc, revenue desc;
-- Q4.1
 select d_year, c_nation,
 sum(lo_revenue - lo_supplycost) as profit
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join customer
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join part
 on lo_partkey = p_partkey
 where
 c_region = 'AMERICA'
 and s_region = 'AMERICA'
 and (p_mfgr = 'MFGR#1'
 or p_mfgr = 'MFGR#2')
 group by d_year, c_nation
 order by d_year, c_nation;
-- Q4.2
 select d_year, s_nation, p_category,
 sum(lo_revenue - lo_supplycost) as profit
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join customer
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join part
 on lo_partkey = p_partkey
 where
 c_region = 'AMERICA'
 and s_region = 'AMERICA'
 and (d_year = 1997 or d_year = 1998)
 and (p_mfgr = 'MFGR#1'
 or p_mfgr = 'MFGR#2')
 group by d_year, s_nation, p_category
 order by d_year, s_nation, p_category;
-- Q4.3
 select d_year, s_city, p_brand,
 sum(lo_revenue - lo_supplycost) as profit
 from lineorder
 join dim_date
 on lo_orderdatekey = d_datekey
 join customer
 on lo_custkey = c_customerkey
 join supplier
 on lo_suppkey = s_suppkey
 join part
 on lo_partkey = p_partkey
 where
 s_nation = 'UNITED STATES'
 and (d_year = 1997 or d_year = 1998)
 and p_category = 'MFGR#14'
 group by d_year, s_city, p_brand
 order by d_year, s_city, p_brand;

5 responses to “TokuDB vs Percona XtraDB using Tokutek’s MariaDB distribution

  1. Tim Callaghan (@tmcallaghan) May 26, 2013 at 4:32 AM

    Justin,

    There are a few configuration parameter defaults worth changing for this type of benchmark:
    tokudb_read_block_size=64K
    – default is 128K
    – this is the smallest readable unit of a leaf node in a fractal tree index
    – I suspect this will have an impact on the cold-buffers tests
    tokudb_row_format=zlib
    – default is quicklz
    – zlib will usually produce far greather compression than zlib
    – again, this should help with cold-buffers tests
    tokudb_cache_size=20G
    – default is 50% RAM (so 12G for your test)
    – same as your InnoDB setting
    tokudb_directio=1
    – use direct IO instead of buffered IO

    -Tim

    • Justin May 26, 2013 at 12:32 PM

      Tim,

      I understand that zlib may give better compression, thus better cold results, but most people run their databases hot, not cold. Will zlib hurt hot performance? Why was quicklz chosen as the default? I think benchmarks should reflect the most common configuration. I guess I’ll just give zlib a test run, but it takes a few hours to run this benchmark, collect the results, graph them etc and I’m not getting paid to do this stuff.

      My benchmarking rule is “only change one thing at a time”. I guess I will pick one query flight (probably 3) and test it with different parameters because it will take to long to test all these different variables in isolation over all four flights otherwise.

      Does TokuDB store compressed or uncomressed data in the cache? Either way, the default TokuDB buffer size (12GB) should be sufficient, as the InnoDB size is 10GB with indexes. I didn’t change the cache size because the manual advises not to change it from default unless you want to decrease it because you are using more than one engine. I only used one engine in isolation, so they were not competing for RAM.

      The manual also suggests not using direct-io. FWIW, sysbench shows no difference between buffered and directio access to my device because I am not using hardware RAID. I will test direct-io and see if it makes a difference.

      • Tim Callaghan (@tmcallaghan) May 26, 2013 at 4:14 PM

        Good points, given that the tables are fully cached there won’t be much difference between the scenarios. One thing you could try is changing the secondary indexes used in the queries to clustering indexes, that would dramatically improve the query performance.

      • Justin May 26, 2013 at 4:57 PM

        The problem is that this is a star schema, so all four indexes will have to be clustering for the four flights to work which will probably make the data size larger than InnoDB.

        I think TokuDB will be very interesting when I test Shard-Query star-schema optimization. A wide PK or clustering indexes will make a lot more sense then, because Shard-Query will push down filters into the WHERE clause instead of having to filter via joins.

  2. Pingback: Shard-Query 2.0 performance on the SSB with InnoDB | Shard-Query blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 162 other followers

%d bloggers like this: