Thursday, October 13, 2016

Bemoaning DB2

About a year ago I was presented with an addition to my role. As part of the company acquiring Infosphere I acquired DB2 under its bonnet. This has proved somewhat of a learning curve and at this time DB2 still has a couple of mysteries up its sleeve. I still have answers to find about DB2's space management and table sizing and storage. I had previously had some experience with installing and some basic application integration steps but never learned much about the day to day management.

A definite win for the DB2 community is Data Server Manager. DSM is a great tool and is rapidly maturing. If you are new to DB2 or are using it definitely keep up with the upgrades right now as there is a lot happening. The current release has some great new features over its earlier counterparts and whilst there are a lot of graphical tools which could be added to the dashboard it's very handy as is.

Easy SQL tool, and the simple dashboard are great. Being able to check any part of the DB2 system in a couple of clicks are all great to have.

I am sure there are those who will still frown upon people using graphical tools but using this as a way to better understand rather than just a mindless point and click will learn a lot very quickly from being assisted by the tool.

So I am still no fan of DB2, it is the 5th product I have worked on seriously in the database realm so I have some considerable experience across them, DB2 will be easier to manage and easier to teach someone in due course  with DSM. Back to war tomorrow and see what else I learn.

See ya round


Friday, June 12, 2015

QLD Government Innovation Portal - Cool idea

QLD Government has recently opened up a site to garner outside thoughts on challenges which the government faces. These might be unusual problems about data, security, and just about anything which requires a solution  that some off the shelf software won't cut out of the box.

There is presently a discussion going on with regards to preserving calendars which are currently in Outlook PST files. Now as to the problems they are trying to solve with that preservation, such as it needs to have the ability to be readable and useful 100 years hence and what formats are open to preserve such things and retain the integrity of the original calendar entry  and be in a format which shall be readable by the fact the file specification will be openly preserved allowing anyone to go and read the file with readily available software knowing the file format.

There is learning going on with both sides of the conversation, with a response of the request specification needing to be addressed as the ideas whilst perhaps good have  a second set of issues. The asked and are altering the question in response to responses as they had learned their initial request may have had some gaps in it.

Well done to the guys at QLD government who set up and run the site and those that are willing to put their ideas out there and canvas a wider pool of thought.

This is a great idea and good to see some people getting into the spirit to try and resolve the question/problem raised.

I hope a lot more departments are able to bring problems to the open to get solutions which may never have been thought of using only oe set of views.

The site is

See ya round


Friday, May 01, 2015

Modelio 3.3 on Xubuntu 3.13

I recently started working with Modelio running on Xubuntu. f you are not familiar with Xubuntu it is a lightweight desktop Ubuntu build. Lighter UI than KDE or Gnome, you will likely end up with parts of KDE or Gnome in your system due to other applications you want to need either KDE or gnome libraries. It is still lightweight and the only reason I use Windows at home is for one purpose and that is accessing the Tax Office (ATO)
Now I am probably not as thin as originally installed as there are some things that you just try and work with.

Modelio has a few issues that you need to resolve to get it working.

Firstly download the software, mine was the open source version 3.3
Unpack it and move it to a directory.
Then you need to make sure your Java is correct. Modelio 3.3 requires java 8 you will need to make sure it is your default Java version, there are a few notes around on Ubuntu you require the alternatives command sudo update-alternatives --config java

I have then set the -vm parameter in modelio.ini to your Java 8 location

Without Java 8 you may find it loading with blank screen and no menus, or it wont start with a framework error.

I had a problem that the TOGAF module wouldn't enable in a project. It was a problem from a previous version of Modelio, actually a 32bit version. I deleted the project. So warning if changing Modelio export your projects before doing anything and import them to the new version. In this case, it was nothing important. You may not be so lucky

Wednesday, April 15, 2015

Setting up mariadb with non standard directories

Today I was configuring a mariadb 10.0.16 server with non default directories. There is a lot of things to overcome in the process.

Firstly make sure your directories are owned by mysql where you want data logs and other elements created.

The following had to be set in server.cnf and set in separate sections in some cases

socket in client section

The rest could be safely accommodated in the mysqld section due to running only a single flavour of mysql/mariadb on the server using this cnf file.


#PID file







#INNODB logs on separate location


We are running multiple log files to allow better log handling.
Hopefully these examples will be of some use to someone contemplating the change to not using the default of /var/lib/mysql and not using /var/log for the logging and not using /var/lib/mysql as a place of other logs such as slow query log

We hit another random error related to MySQL bug #45379. We had added user= and group= to the [mysqld] section and had the server randomly setting the group_concat_max_len to 4 breaking the use of group_concat. Moving those settings to mysqld_safe resolved that. Thanks Pythian for writing about this.




Now looking forward to getting the new servers commissioned and into production.

If all the above is not done then the database will not create files in all the expected locations but use defaults and try using /var/lib/mysql, one day this will hopefully gone, instead replaced by an intelligent installer. which will ask questions and build a config file and create directories.

One final note make sure you run mysql_install_db as the mysql user or you will have issues with starting your database once again due to permissions of files created when running the install to create the database. If you have all your config correct with directories in teh config matching actual directories then when you run mysql_install_db you wont require any parameters with it. Simple.

See ya Round


Sunday, April 12, 2015

Learning about Cassandra and Coursera

Watched the presentation produced recently by Datastax and Coursera on Cassandra and how and some of why they chose Cassandra over Mongo and Hbase to replace some of their MySQL for the website.

Some key takeaways I saw was that Cassandra better fitted their data model over HBase or MongoDB. That Mysql Sharding was difficult for some of their needs.

Coursera has a very distributed customer base and MySQL wouldn't handle multi-site information capture and distribution.

Cassandra could do rolling upgrades without downtime,this is important for Coursera, if you have ever filled out a set of responses to an assignment and then not be able to save then because the application was offline due to a database maintenance window. This was a key tenet for what replaced MySQL.
Now if your geographic requirements, ie your customer base live local to Australia or Europe, then a single site of MySQL with Galera, or using AWS RDS, may well produce suitable results. You can also use a message bus within your application to provide some resilience to the database being offline.
Rolling updates are difficult for most RDBMS and Cassandra is strong in this area.

Like so much, it's not about being cool, it's about knowing the problem you are solving to choose the right product. There is no bad database only bad implementations using the wrong product in a given situation. MongoDB seems to cop a little flak, but this is probably more about a headlong rush to get away from a perceived problem with MySQL. Often early-stage web applications ar  built with some part as Key pair store or as document store mentality. This is often simple to handle from the web development position, however, it ultimately won't scale in MySQL, or other RDBMS for that matter. So then people start looking for alternate storage ie other databases. This often creates a solution to that part of the database which is a document but causes problems with relational parts of the application. Ultimately this ends up with MongoDB being a bad choice for solving the problem.

Cassandra will be a fantastic solution as will MongoDB to the right problems. It won't solve poor design or a bad understanding of application requirements or business needs to access the data for reporting purposes. For this reason think carefully about any decision to jump off an original choice of an RDBMS such as MySQL and into the arms of a NoSQL database. It might leave a very sour taste in your mouth in the NoSQL database you choose.

Disclaimer: I have taken courses from Coursera

See ya round


Tuesday, November 11, 2014

Mariadb 10 Slave wont set up after upgrading from mariadb 5.5

We are in teh process of upgrading mariadb 5.5 databases to maridb 10. The actual upgrade process of the database has been flawless for the stand alone databases. I have started on our first replication set a pre-production system and after upgrading a slave to 10.1.14  have hit a wall

The maria10 database slave not starting up as a slave after upgrading from 5.5 The master is still running as 5.5

This is the error log

141111 10:57:10 [Note] Event Scheduler: Purging the queue. 0 events
141111 10:57:10 [Note] InnoDB: FTS optimize thread exiting.
141111 10:57:10 [Note] InnoDB: Starting shutdown...
141111 10:57:12 [Note] InnoDB: Shutdown completed; log sequence number 36086013855
141111 10:57:12 [Note] /usr/sbin/mysqld: Shutdown complete

141111 10:57:12 mysqld_safe mysqld from pid file /var/run/mysqld/ ended
141111 10:57:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/nxq
141111 10:57:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
141111 10:57:13 [Note] InnoDB: The InnoDB memory heap is disabled
141111 10:57:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
141111 10:57:13 [Note] InnoDB: Memory barrier is not used
141111 10:57:13 [Note] InnoDB: Compressed tables use zlib 1.2.3
141111 10:57:13 [Note] InnoDB: Using Linux native AIO
141111 10:57:13 [Note] InnoDB: Using CPU crc32 instructions
141111 10:57:13 [Note] InnoDB: Initializing buffer pool, size = 128.0M
141111 10:57:13 [Note] InnoDB: Completed initialization of buffer pool
141111 10:57:14 [Note] InnoDB: Highest supported file format is Barracuda.
141111 10:57:14 [Note] InnoDB: 128 rollback segment(s) are active.
141111 10:57:14 [Note] InnoDB: Waiting for purge to start
141111 10:57:14 [Note] InnoDB:  Percona XtraDB ( 5.6.20-68.0 started; log sequence number 36086013855
141111 10:57:14 [Note] Plugin 'FEEDBACK' is disabled.
141111 10:57:14 [ERROR] Failed to create a socket for IPv6 '::': errno: 97.
141111 10:57:14 [Note] Server socket created on IP: ''.
141111 10:57:14 [Note] Event Scheduler: Loaded 0 events
141111 10:57:14 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-nxq-relay-bin' to avoid this problem.
141111 10:57:14 [ERROR] Server id not set, will not start slave
141111 10:57:14 [ERROR] Failed to create slave threads
141111 10:57:14 [ERROR] Failed to allocate memory for the Master Info structure
141111 10:57:14 [Note] /usr/sbin/mysqld: ready for connections.

This is the server.cnf it isn't as clean and tidy as it should be but it is what it is


# this is only for the mysqld standalone daemon

binlog_format = mixed

# MASTER config
# log-bin=mysql-bin.log

# Autostart the event scheduler

# SLAVE config

# Performance Tuning

innodb_file_per_table = 1
innodb_flush_method = O_DSYNC


# this is only for embedded server

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here

# This group is only read by MariaDB-10.0 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand


I moved the server_ID value to the [mariadb-10.0] section from the [mysqld] section to try and find a solution

Update: Turns out the reason for this behaviour was due to a typo in the init.d startup script in this line defaultfile="/etc/my.cnf" It had to be changed from another location and it had an error. Funnily enough though the strace output was not helpful as it showed what appeared to be it reading the files in the /etc/my.cnf.d directory.

How did I find it? By using the startup process of executing mysqld --help. It displays the default location parameters and i could see the server-id set correctly

See ya round


Wednesday, November 05, 2014

OSDC 2014 Day One

OSDC 2014 kicked off today and I was attending, my OSDC virginity has gone. Opens Source Developers Conference has convened on the Gold Coast

It was very interesting, I have seen some great talks getting an insight to a number of technologies
There are some people there who are madly hacking all sorts of things, Graphing technology, Arduino devices and the sustainable home, Gov hack and databases.
Its both old and young along side with only a passion about open source software
Talks about protecting yourself regarding the footprint you leave on the web from browsing and how that leaves you vulnerable to snooping about your surfung habits
Using to remove some of the inevitable jet trail you leave in metadata around the web
What about Postgres and JSON and the ability of Postgres to provide more ways to handle JSON objects? Including indexing elements of the JSON. More toi come in Postgres 9.4

Have you heard about Bluehackers, perhaps you should hopefully you never need it but one day you might just save a fellow worker with knowing about it.

Mari10DB is a leap forward, multi threaded Replication, Galera for clustering, new performance improvements with exists and in queries. GTID for replication consistency a long standing problem for MySQL and its derivatives such as mariadb. Improved performance on inserts.

Also a talk on stress and stress management with a tilt to Bluehackers

Some others are there for their first and some have been around since their inception

Tomorrow more stuff and conference dinner

It has been an interesting day with some people doing some interesting things

See ya round