Facebook exit stage left

Comments Off

In an older article about social network trends, I discussed how Facebook in particular was far past its peak as measured by Google Trends. And that trend has not turned. There is a smaller and smaller fraction of web searches for the term “facebook”. It is now around a similar level as in 2009. Some of this can be explained by the fact that more people use the Facebook mobile app to access the social network, and therefore do not need to search Google to access the site. However, as pointed out by this Guardian article, despite claiming 1.79 billion users, 2016 was the year Facebook “became the bad guy”. In the Western world, interest was saturated several years ago, and new users come from other regions and possibly where Facebook have been blocked. They now plan to adapt their site and network to China’s censor and surveillance requirements.

Yet, if you are an investor in the social network, you might want to hold on a bit longer before you sell. In recent years, Facebook has snapped up both of the new popular social apps Whatsapp and Instagram (in 2014 and 2012 respectively). They continue their upwards trend, and will do so for many years to come. Compared to Facebook they both pale on Google Trends, meaning there’s a lot of potential.

Comments Off

12 TB helium and 14TB helium + SMR announced

Comments Off

HGST has just announced a new helium filled drive of 12 TB. The increase in capacity comes from an impressive and unexpected 8 platter design. Meanwhile, Western Digital forecasts that they will take the Ultrastar He12 based disk to 14 TB by combining it with Shingled Magnetic Recording (SMR). No prices are indicated yet.

Comments Off

Review: “The Internet Is Not the Answer”, Andrew Keen

Comments Off

Andrew Keen is a bitter man. He longs for the time when his family ran a tailor business in London and middle class people could afford to buy clothes from their store. He resents the fact that his own music Internet site failed, while seemingly similar sites like MySpace, Spotify came to be valued in billions. For some reason he mourns the demise of Kodak and its film roll processing centre in Rochester, NY. And most of all, he despises rich folk, but not any billionaire, just those who happen to have made their money through the Internet. Keen’s book “The Internet Is Not the Answer” reads a bit like a rant towards all these things, while blaming it all on Silicon Valley and The Internet. The solutions he favour are mostly based on government regulation: six strikes laws for copyright infringement; antitrust and monopoly busting; mixed with labor unions.

Even though Keen’s book has a bitter tone throughout, he does touch on important points regarding increasing wealth disparity, middle class jobs being replaced by automation and far fewer specialized jobs, monopolistic mega-cooperations, centralized services. He takes on Amazon, Google, Facebook, Instagram, Twitter, WhatsApp. He has done thorough research, and his book includes a substantial reference section. It is just that his conclusions doesn’t always align with actual causes and effects: Take the downfall of Kodak, where he spends a full chapter lamenting Instagram for the killing of film processing. Companies like Canon and Nikon which developed and sell high quality DSLR cameras and thus more directly caused the replacement of film are not mentioned.

Similarly, Keen reviews the history of the early days of the Internet, and its inventors and pioneers like Paul Baran, Bob Taylor, Bob Kahn, Vint Cerf, Tim Berners-Lee and points out how altruistic and anti-commercial they were. He contrasts this to “winner-take-all” companies in today’s economy. However, he does not discuss the seemingly obvious conclusion that what we’re lacking from Facebook, Instagram, Twitter are common open standards and protocols, which is what made early technology successful and long-lasting. Free and Open Source Software (FOSS) does not enter Keen’s field of view at all.

Maybe not worth it

If you already work in tech, and have good insight into these topics, Keen does not bring much new to the table. In fact, he disappoints in that regard. However, if you are interested in gentrification in San Francisco, Kodak in Rochester, or just want to hear a different point of view, give the book a try.

What Keen does have going for him, is that he is a very good writer. He writes almost poetically, albeit with great sarcasm, about topics like Internet economy, government regulation, and pretentious billionaires. Sound bites like the one below at least make the book entertaining.
While talking about the cult and praise of “failure” in tech-companies:

“Instagram actually represents the reverse side of Silicon Valley’s cult of failure. In the Valley, the rich and famous claim to be failures; on social networks like Instagram, millions of failures claim to be rich and famous”.

Finally, although it’s no way to be sure, it sometimes feels like the writes of Silicon Valley, the TV comedy series, have studied the book thoroughly and lifted several ideas from Keen onto the screen. What Keen scorns, like the double-speak; the feel-good big-company efforts and speeches; and the general Valley culture has made the TV series a hit. Keen’s book makes the series even more fun to watch.

Comments Off

Upgrading Debian Wheezy 7 to Jessie 8

Comments Off

Upgrading from Debian 7 to 8 is reasonably straight forward, following the official instructions. These shorter summaries are also useful references.

Very briefly then, make sure you have backup.
dpkg --get-selections "*" > dpkg_selections.txt
tar zcvf upgrade_backup.tar.gz /etc /var/lib/dpkg /var/lib/apt/extended_states /etc/mysql/my.cnf /etc/fuse.conf /etc/ssh/ssh_config

Update /etc/apt/sources.list, and replace all occurrences of wheezy with jessie.
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list

If VirtualBox is installed, update to the new key:
wget -q -O - http://download.virtualbox.org/virtualbox/debian/oracle_vbox_2016.asc | sudo apt-key add -

Then comes the upgrade dance, with a few prompts, warnings, questions.

apt-get update
apt-get upgrade
apt-get dist-upgrade

After the upgrade, it is recommended to purge unused packages
apt-get purge $(dpkg -l | awk '/^rc/ { print $2 }')
apt-get autoremove

It is also recommended to install the linux-image-* metapackage, e.g. for AMD CPUs
apt-get install linux-image-amd64

Finally, cross your fingers and reboot.

Comments Off

Android Hacking

Comments Off

In a recent post on his blog, Thanassis Tsiodras has an impressive tale of his Android hacking adventures. Wanting to run Debian from chroot, he ends up customizing the boot image; attaching a serial logger to the headphone jack; and intercepting the over-the-air update image to control the boot process.

At last, it seems he achieved what wanted, but of course at an extremely high price. Most mobile devices are now very hostile towards any other use than was is dictated by the manufacturer and OS vendor. Thanassis ends his article on a sober note, saying that once even this hack is secured against “Android might as well be called iOS”.

Comments Off

Storage prices – end of 2016

Comments Off

The last half-year have seen few movements in spinning disk prices, and actually some have gotten a few coins more expensive. It is still the 8 TB SMR Seagate drive which give most storage for money, only with the exception of a special offer on an external 8 TB WD disk. In fact, several of the external offerings are now cheaper than their internal counterparts.

Also worth noting, is that Western Digital has deprecated their long time cheapest Green line, in favour of the Blue color. However, take care: The Blue line contains both 5400 and 7200 RPM drives. The ones listed here are all 5400 RPM.

On the SSD side, things have also not changed much. There were some promises, but not much in terms of concrete offerings this year.

Finally, many of the flash cards are getting more competitive. Indeed, the the Sandisk Ultra 256 GB key, as well as the 200 GB micro SD card are now pretty great offering for on-the-go storage. They’d fit very decent sized music collections, picture albums, etc.

Media Type Product Capacity Price CHF Price Euros Euros / GB GBs / Euro
External 3.5 Western Digital My Book 8TB, USB3 8000 GB 250.00 233.64 0.03 34.24
HDD-SMR Seagate ARCHIVE HDD 8TB 8000 GB 266.00 248.60 0.03 32.18
HDD Seagate Desktop 4TB 4000 GB 134.00 125.23 0.03 31.94
External 3.5 Western Digital My Book 6TB, USB3 6000 GB 202.00 188.79 0.03 31.78
External 3.5 Western Digital Elements Desktop 4TB, USB3 4000 GB 135.00 126.17 0.03 31.70
HDD Western Digital Blue, 5400 RPM 3TB 3000 GB 107.00 100.00 0.03 30.00
HDD Western Digital Blue, 5400 RPM 4TB 4000 GB 145.00 135.51 0.03 29.52
External 3.5 Western Digital My Book 4TB, USB3 4000 GB 149.00 139.25 0.03 28.72
SMR External 3.5 Seagate Backup Plus Desktop 8TB 8000 GB 300.00 280.37 0.04 28.53
External 3.5 Western Digital My Book 3TB, USB3 3000 GB 115.00 107.48 0.04 27.91
External 3.5 Western Digital Elements Desktop 3TB, USB3 3000 GB 118.00 110.28 0.04 27.20
HDD Western Digital Blue, 5400 RPM 6TB 6000 GB 244.00 228.04 0.04 26.31
HDD Western Digital Red 4TB 4000 GB 164.00 153.27 0.04 26.10
HDD Western Digital Blue, 5400 RPM 5TB 5000 GB 208.00 194.39 0.04 25.72
External 2.5 Western Digital My Passport Ultra 3TB, USB3 3000 GB 125.00 116.82 0.04 25.68
HDD Western Digital Red 3TB 3000 GB 126.00 117.76 0.04 25.48
HDD Western Digital Red 6TB 6000 GB 252.00 235.51 0.04 25.48
HDD Western Digital Blue, 5400 RPM 2TB 2000 GB 84.90 79.35 0.04 25.21
HDD-He Western Digital Red 8TB 8000 GB 345.00 322.43 0.04 24.81
External 2.5 Western Digital Elements Portable 2TB, USB3 2000 GB 88.00 82.24 0.04 24.32
HDD Western Digital Red 2TB 2000 GB 98.00 91.59 0.05 21.84
External 2.5 Western Digital My Passport Ultra 2TB, USB3 2000 GB 103.00 96.26 0.05 20.78
HDD-He Hitachi Ultrastar He6 6TB 6000 GB 361.00 337.38 0.06 17.78
HDD Western Digital Blue, 5400 RPM 1TB 1000 GB 61.70 57.66 0.06 17.34
External 2.5 Western Digital Elements Portable 1TB, USB3 1000 GB 62.00 57.94 0.06 17.26
HDD-He Hitachi Ultrastar He8 8TB 8000 GB 517.00 483.18 0.06 16.56
External 2.5 Western Digital My Passport Ultra 1TB, USB3 1000 GB 72.00 67.29 0.07 14.86
Blu-ray Verbatim BD-R SL 10 @ 25GB 250 GB 23.90 22.34 0.09 11.19
DVD-R Verbatim 16x DVD-R 100 @ 4,7GB 470 GB 46.00 42.99 0.09 10.93
Blu-ray Verbatim BD-R DL 10 @ 50GB 500 GB 49.00 45.79 0.09 10.92
DVD+R DL Verbatim 8x DVD+R DL 25 @ 8,5GB 213 GB 39.00 36.45 0.17 5.83
DVD+R DL Verbatim 8x DVD+R DL 50 @ 8,5GB 425 GB 95.40 89.16 0.21 4.77
SSD Crucial BX200 SSD, MLC, 480GB 480 GB 139.00 129.91 0.27 3.69
USB Flash SanDisk Ultra, USB 3.0, 256GB 256 GB 78.00 72.90 0.28 3.51
SSD Samsung SSD 850 EVO Basic, TLC, 500GB 500 GB 165.00 154.21 0.31 3.24
SSD Crucial BX200 SSD, MLC, 240GB 240 GB 81.00 75.70 0.32 3.17
SSD Samsung SSD 850 EVO Basic, TLC, 1TB 1000 GB 339.00 316.82 0.32 3.16
SSD Crucial MX200 SSD, MLC, 500GB 500 GB 175.00 163.55 0.33 3.06
SSD Crucial MX200 SSD, MLC, 1000GB 1000 GB 374.00 349.53 0.35 2.86
SSD Crucial MX200 SSD, MLC, 250GB 250 GB 95.30 89.07 0.36 2.81
SSD Crucial BX100 SSD, MLC, 500GB 500 GB 193.00 180.37 0.36 2.77
SSD Samsung SSD 850 EVO Basic, TLC, 250GB 250 GB 100.00 93.46 0.37 2.68
microSDXC SanDisk Ultra Premium microSDXC 90MB/s, 200GB 200 GB 85.00 79.44 0.40 2.52
USB Flash SanDisk Cruzer Edge Flash Drive 64GB 64 GB 27.80 25.98 0.41 2.46
SSD Samsung SSD 850 Pro, MLC, 1024GB 1024 GB 454.00 424.30 0.41 2.41
USB Flash SanDisk Ultra, USB 3.0, 64GB 64 GB 28.80 26.92 0.42 2.38
CD-R Verbatim CD-R 100 @ 700MB 70 GB 32.60 30.47 0.44 2.30
SSD Samsung SSD 850 Pro, MLC, 512GB 512 GB 239.00 223.36 0.44 2.29
SSD Samsung SSD 850 Pro, MLC, 256GB 256 GB 139.00 129.91 0.51 1.97
USB Flash SanDisk Cruzer Edge Flash Drive 32GB 32 GB 18.80 17.57 0.55 1.82
USB Flash SanDisk Ultra, USB 3.0, 32B 32 GB 19.50 18.22 0.57 1.76
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 3, 95/90MB/s, 128GB 128 GB 82.80 77.38 0.60 1.65
SSD-NVM-M.2 Samsung SSD 950 Pro, M.2 2280, MLC, 2500/1500MB/s, 512GB 512 GB 361.00 337.38 0.66 1.52
SSD-NVM-M.2 Samsung SSD 950 Pro, M.2 2280, MLC, 2200/900MB/s, 256GB 256 GB 187.00 174.77 0.68 1.46
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 3, 95/90MB/s, 512GB 512 GB 389.00 363.55 0.71 1.41
SSD Samsung SSD 850 Pro, MLC, 128GB 128 GB 99.70 93.18 0.73 1.37
SSD Samsung SSD 850 EVO Basic, TLC, 120GB 120 GB 95.70 89.44 0.75 1.34
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 1, 95/90MB/s, 64GB 64 GB 55.00 51.40 0.80 1.25
USB Flash SanDisk Cruzer Edge Flash Drive 16GB 16 GB 14.90 13.93 0.87 1.15
SDHC SanDisk Extreme Pro SDHC UHS-I, Class 10/UHS 1, 95/90MB/s, 32GB 32 GB 33.00 30.84 0.96 1.04
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 64GB 64 GB 75.00 70.09 1.10 0.91
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 256GB 256 GB 347.00 324.30 1.27 0.79
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 128GB 128 GB 175.00 163.55 1.28 0.78
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 64GB 64 GB 98.50 92.06 1.44 0.70
USB Flash SanDisk Cruzer Edge Flash Drive 8GB 8 GB 12.90 12.06 1.51 0.66
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 32GB 32 GB 52.70 49.25 1.54 0.65
SDHC SanDisk Extreme Pro SDHC UHS-I, Class 10/UHS 1, 95/90MB/s, 16GB 16 GB 27.00 25.23 1.58 0.63
SDXC SanDisk Extreme Pro SDXC UHS-II, UHS 3, 280/250MB/s, 64GB 64 GB 117.00 109.35 1.71 0.59
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 32GB 32 GB 60.10 56.17 1.76 0.57
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 16GB 16 GB 36.40 34.02 2.13 0.47
SDHC SanDisk Extreme Pro SDHC UHS-II, UHS 3, 280/250MB/s, 16GB 16 GB 52.70 49.25 3.08 0.32

Exchange rate: 1 Euro = 1.070000 CHF.

Comments Off

Review: “Flashpoints – The Emerging Crisis in Europe”, George Friedman

Comments Off

In his latest book, “Flashpoints – The Emerging Crisis in Europe”, George Friedman gives a summary of various events through European history, focusing on the 20th century wars, inter-war years, and cold war eras. Although the book tries to sell itself as being about future predictions, he spends more than half of it looking at the past. In the second half, he tries to foresee various possibilities and outcomes for the regions of Russia / Eastern Europe; Germany; France; the Mediterranean region; Turkey and Britain. However, most of the attempt at analysis remains limited, shallow, and biased.

The problem with Friedman’s book, is that it seems he didn’t quite decide what it was supposed to be about: his own and his family’s escape from war in Europe; his travels across Europe in later years; a history of Europe; or as the title suggests, about future emerging crisis. It turns in to a bit of everything, and becomes colloquial and incoherent. Between the endless anecdotes, it seems what Friedman is missing is an editor who can cut away the cruft. Totally irrelevant stories covering taking a leak behind a border office in Ukraine or that some hotelier in Sarajevo reminds him of his aunt, are just some of what could have been left out.

Skip it

Frankly, this book is a waste of time. Although the history of Europe is important in order to understand the present and the future, this book is not a good summary. If any section of it had been part of Wikipedia, it would have been littered with “citation needed”. In fact, Friedman hasn’t included any references at all, and sometimes it seems he has not done his homework very well, with plenty of inaccurate facts.

This makes the future predictions in the book rather worthless. They are not based on any hard facts or research, but rather what Friedman could imagine. Furthermore, the book shows some age, despite being republished this year: In the chapter on Britain, he completely missed the risk of Britain voting to exit the EU, while it’s mentioned briefly in the after-word.

Overall, this is not a book worth spending time on.

Comments Off

Jekyll: Include partial snippets of code

Comments Off

The Jekyll include tag is useful when including files or templates on a page. Combined with the highlight tag, it makes including code snippets easy. However, it will include the complete file, and often it is desirable to include only a few lines, or maybe a method. That could of course be done by simply copy/pasting the code in question into the article, but then the code gets out of sync if the example file is changed.

The basic usecase is something like this:

{% highlight java %}
    {% include src/HelloWorld.java %}
{% endhighlight %}

Ruby based plugin

A Jekyll tag to include only a section of a file would be great. As far as I can tell, that does not exist yet, so I started writing one. Unfortunately, Github Pages does not allow custom plugin for security reasons. There are work-arounds for that, but it also makes the deployment more complex, and loses the convenience of being able to edit the articles and code directly on github.com.

Sans error handling or caching, a simple implementation could look like this. It works outside Github Pages, so it’s a start.

module Jekyll
  class IncludeLines < Liquid::Tag
     Syntax = /(#{Liquid::QuotedFragment}+)\s(\d+)\s(\d+)\s\z/o
     def initialize(tag_name, markup, options)
       if markup =~ Syntax
         @file = $1
         @startline = $2.to_i
         @endline = $3.to_i
         raise "Syntax error in includelines: " + markup
     def render(context)
       lines = IO.readlines(context.evaluate(@file))
       part = lines.drop(@startline)
       part.take(@endline - @startline)
 Liquid::Template.register_tag('includelines', Jekyll::IncludeLines)

Liquid slice and split

Using the Liquid capture block, it's possible to read a file and store it as a string variable. It can then be processed by Liquid instead of the plugin, and works fine with Github pages. The Liquid syntax is certainly verbose, but it gets the job done.

An initial implementation cutting the file content as a single string looked like this. However, it is far from ideal, since the character index and count will shift with any source code modifications on the included file.

{% capture filecontent %}
    {% include src/HelloWorld.java %}
{% endcapture %}

{% highlight java %}
    {{ filecontent | slice: 132, 57 }}
{% endhighlight %}

A slightly better solution uses the same idea, but operates on line numbers instead. It is almost as fragile when it comes to changes, but at least usable.

{% capture filecontent %}
    {% include src/HelloWorld.java %}
{% endcapture %}

{% assign lines = filecontent | newline_to_br | split: '<br />' %}
{% highlight java %}
    {% for line in lines offset:10 limit:5 %}{{ line }}{% endfor %}
{% endhighlight %}

A helper include file implementing this idea can be found here. It can be used like this:

{% include includelines filename='src/HelloWorld.java' start=10 count=5 %}

Include method

Ideally, it would be possible to mark the start of a line to include, and then indicate how much should be included. Improving on the line based iterator above, this helper file does that. Usage goes like this:

{% highlight java %}
    {% include includemethod filename='src/HelloWorld.java' method='test()' before=5  after=1 %}
{% endhighlight %}

It also adds options to include lines before and after the specified method, for example for comment blocks or further methods below the first. There are of course some extensions which could be made, e.g. to include multiple split sections; support other non-C like languages, etc. The linked code is under the GPL 3 license, so feel free to improve.

Comments Off

Review: “ISIS: Inside the Army of Terror”, Michael Weiss and Hassan Hassan

Comments Off

In their book “ISIS: Inside the Army of Terror” from early 2016, Michael Weiss and Hassan Hassan give detailed insight into the Islamic State, its origin, key members, alliances, critical battles, and strategy of terror. The story begins with the early ties between Ayman Mohammed Rabie al-Zawahiri and Osama bin Laden in the 1990s, and the later split between them and the more radical and extreme Abu Musab al-Zarqawi. Their disagreement on who are their enemies, crucially whether it includes Muslims or not, has underlined the split between al-Qaeda and ISIS / Islamic State. In later chapters, the rise of the current leader, theology professor Abu Bakr al-Baghdad, is investigated. The book goes into great detail about several key battles in Iraq and Syria, and analyses positions and outcome. Finally, some of the terror attacks on civilians in Europe and the US is put in context.

Sunni vs. Shia

There are a few important take-aways from the book: The divide between Sunni and Shia Muslims is usually at the core of most of the conflicts. Crucially, the numbers goes a long way to explain the various positions: Word-wide, Sunnis are in majority at around 85–90% while 10–15% are Shia. However, in Iraq and Iran this is reversed, where 80% and 95% are Shia respectively. An important point is the fact that Saddam Hussein was Sunni, and his mostly Sunni minority Baath party ruled over the Shia majority. When US invaded and ended their rule and tried to create democracy, the stage was set for bitter conflict. Furthermore, Paul Bremer (presidential envoy to Iraq) fired the mostly Sunni Iraqi army, along with most other official positions. So around 2003 a large part of the previous Iraq elite was suddenly jobless, but with plenty of military experience and even weapons on their hands. al-Zarqawi exploited these fault lines to his fullest, and ignited the ensuing civil war.

A similar setup, but again reversed has been the background for the civil war in Syria. There, Bashar Hafez al-Assad and his party are Alawites, a branch of Shia, but in a minority at around 13%. When the spring revolutions in 2011 swept other Muslim countries, al-Assad pitted themselves as under attack by the Sunni (74%) majority. al-Assad’s regime has support from Iran and Hezbollah who are also Shia. The opposition in Syria has many factions, and ISIS has time and again proved that they are experts at driving a wedge between opposing forces to divide and conquer.

Enemy of my Enemy

From small tribes, to national organizations and rebel groups and all the way to international alliances, the relationships network is extremely complex. A graph like this hardly scratches the surface. Furthermore, alliances shift frequently, and often the short term strategy is “the enemy of mine enemy is my friend”. This can be seen going far back, and characterizes much of US and Russian involvement in the various conflicts: During the Cold War; the US backing of Iraq against Iran in 1979; later US attacks against Iraq; US backing of the Kurds. Iran and Russia have tended to back the opposite groups, and Iran in particular has now infiltrated much of the Shia resistance and politics in Iraq.

The book goes into detail on several of these fluid alliances, and looks at the decisive battles and opposing personalities. The point is made many times over that in order to understand the conflicts, one has to understand the tribal politics. At a higher level, the relationships are often more pragmatic: Although ISIS is waging war across Syria, and also against al-Assad forces, they have a business relationship in the oil trade, where ISIS is selling back oil to al-Assad’s regime from the oilfields they have captured. al-Assad benefits since ISIS also fights the rebels in Syria. The enemy of mine enemy is my friend.

Read it

Weiss and Hassan have done plenty of research and interviews for this book, and it shows through all the details revealed. They have also done a good job of explaining the background history, religious underpinnings, and political motivations for the parties involved in the conflict. However, it can get somewhat tedious to go through all the nitty-gritty, and the writing style can be trite with the occasional odd analogies.

Overall, the book is definitively worth a read if you are interested in the current conflict, want to understand the terror attacks. Regional and international politics become more clear with the information provided by this book.

Comments Off

Backblaze: SMART stats to predict hard drive failure

Comments Off

Over the years, Backblaze has published several interesting reports and statistics on their harddrive performance and failures. In their latest post, they look at which SMART metrics they monitor and react to. They say, “when the RAW value for one of these five attributes is greater than zero, we have a reason to investigate”.

SMART 5 Reallocated Sectors Count
SMART 187 Reported Uncorrectable Errors
SMART 188 Command Timeout
SMART 197 Current Pending Sector Count
SMART 198 Uncorrectable Sector Count

They go on to point out, that although a single one of those metrics might only indicate 30% to 40% probability of a failed drive, when combined the probability increases to 76%. Furthermore, looking at multiple failed values, and their rate of increase might also help determine if a drive is about to fail. A higher rate is usually worse. Finally, they note that the the SMART 189 – High Fly Writes is another good candidate to consider, and again when looking at the rate.

On a related note, here’s a note on Google’s data on their SSD failures.

Comments Off

Getting started with GitHub Pages and Jekyll

Comments Off

In the beginning, there were static HTML pages, TABLE-tags, and FTP; later came dynamic sites, WordPress and other Content Management Systems; but now we are back to static pages again, albeit templated. So goes Quinn Supplee’s narrative of the move to Jekyll, Markdown and Liquid based static sites. Add in GitHub for free hosting, and it makes a very compelling offering for a small site run by tech savvy people. It’s not your web sites for dummies solution.

Enabling web page hosting from a Github repository is quick, as explained here. And setting a custom domain is a single setting on Github, and additional setting up your domain DNS. The Github Settings panel have a default example site template generator, so with that you’ll have some pages to look at in few clicks. Now you can download the generated files, modify and upload with standard git commands. (Replace username/repository with your own names).

git clone http://github.com/username/repository
git commit
git push

Github uses Jekell to statically generate the site, and this requires a special file and directory structure. It is of course possible to experiment with the files directly on Github, however it might be easier to edit and compile locally. For that, a local install of Jekyll and relevant dependencies and tools is required.

sudo apt-get install ruby ruby-dev rubygems-integration nodejs gcc make
sudo gem install jekyll jekyll-docs jekyll-feed jekyll-paginate bundler minima iconv

On older Debian (7 – Wheezy) or Ubuntu (14.04) distributions, the ruby packages where versioned incorrectly, so the 2.0 version is required for the ruby package and gem binary.

sudo apt-get install ruby2.0 ruby2.0-dev rubygems-integration nodejs ruby-mkrf gcc make
sudo gem2.0 install jekyll jekyll-docs jekyll-feed jekyll-paginate bundler minima iconv

With that in place, you can generate a new blank site, and make Jekyll serve it locally on . Of course, the final destination would be the Github repository.

jekyll new test --skip-bundle
cd test
jekyll serve

For more advanced options and functions, the Jekyll documentation is good. There are some Jekeyll based example sites here. In particular, Patrick Mckinley’s pagination example with source looks interesting.

Comments Off

Samsung boasts about its SSDs

Comments Off

In a few recent articles at Tom’s Hardware, Paul Alcorn has summarized the latest roadmap and product visions from Samsung, including an upcoming new form factor to replace the Intel M.2 connector and QLC (Quad Layer Cell) SSD that could reach 100 TB drives. Finally, Chris Ramseyer takes a look at the new Samsung 960 EVO and 960 Pro NVMe drives.

Some developments are clear: More data will be stacked in a smaller space, through 3D NAND, 16 bits QLC (Quad Layer Cell), and more dies stacked on top of each other. This will all result in lower prices per byte of storage. Obviously, it makes no sense to include a spinning disk in a laptop anymore, but the holy grail is the data center. There things are not so clear-cut. According to Alcorn’s article, Facebook is experimenting with QLC drives which could reach 100 TB. But there is of course no mention of price.

What’s interesting, is that the newer drives, which use the new NVMe based M.2 controller has reset the downwards price trend. In one of the slides, Samsung points out that SSD storage is now at 36 US cents per GB. However, the latest 960 series are almost double that, at 64 cents. (512 GB for $329; 1 TB for $629; 2 TB for $1299). In comparison, the Seagate Archive 8 TB drives cells for $249 now, is 1/20 of that, at 3 cents per GB.

In other words, Samsung has traded the 4x to 6x increase in read / write speed on the NVMe controllers for a doubling in SSD storage price. Now, Alcorn points out that there might be some margin to shave off there. Samsung has almost half the world market on both SSD drive and total SSD capacity shipments right now, but the competition is ramping up. That we will see lower prices per byte on SSD in the next year is a given.

Comments Off

Add-on development for Kodi

Comments Off

On the heels of the QNAP NAS setup notes, here’s a fun integration with my home automation system for living room lights.

The idea is to send the same commands from the Kodi app as the custom Android app does to the Arduino controlled relays. Before the movie starts, the lights go off. I’ll skip the details of that code, but point to a few useful pages to get started. It’s simple.

The Kodi Add-on documentation is good. To get started, you need at least two files: The addon.xml configuration, and your Python script, e.g. myaddon.py. These have to be in a directory on the format script.name (more in the linked documentation) and zip-ed in a ZIP file which does not use use compression, as seen below. This zip-file can now be copied to the NAS, and installed from Kodi.
zip -0 -r myaddon.zip script.myaddon

One gottcha is that the addon.xml file cannot contain a final new-line. At least some people have reported that causing a install error.

For an easy way to get started, look at the Hello World add-on example, as well as its source code. It doesn’t get easier than that.

Comments Off

Securing a Postfix mail server – TLS transport encryption

Comments Off

I previously discussed SPF and DKIM setup for the Postfix mail server. Here’s some notes on TLS transport encryption. (Although, maybe those articles should have come in opposite order).

Using a self-signed certificate (which should be fine for small scale usage), setup is rather easy and straight forward. Creating the keys and certificats boils down to these instructions, copied from here. (Similar instructions here).

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -out rootCA.pem
openssl genrsa -out device.key 2048
openssl req -new -key device.key -out device.csr
openssl x509 -req -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out device.crt -days 500

Modifying /etc/postfix/main.cf, you might end up with something like this, assuming you’ve copied the keys as indicated by the linked article.
smtp_use_tls = yes
smtpd_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtpd_tls_security_level = may
smtpd_tls_ask_ccert = yes
smtpd_tls_security_level = may
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom
smtpd_tls_key_file = /usr/share/ssl/certs/postfix/device.key
smtpd_tls_cert_file = /usr/share/ssl/certs/postfix/device.crt
smtpd_tls_CAfile = /usr/share/ssl/certs/postfix/rootCA.pem

Once all the changes are made, restart postfix:
service postfix restart

Now you can verify the setup with telnet:
telnet mail.example.com 25
EHLO example.com

This should yield:
220 Ready to start TLS

Another way to confirm the setup is to send an email to a gmail.com account, and observe the lock status icon on the header field drop-down, explained in detail here.

Finally, the official Postfix documentation and notes on authentication (older doc) might come in handy.

Comments Off

QNAP NAS and autofs auto mount

Comments Off

After considering multiple options to cover a HTPC and a NAS, I finally went with the combined “living room” QNAP HS-251+ NAS earlier this year. I’ll leave the reviews to other sites, and just summarize the main features:

  • 2 bay 3.5″ or 2.5″ HDD or SSD
  • Intel Celeron 2GHz Quad core; 2 GB DDR3 RAM
  • 2x 1Gb RJ-45 ports; 2x USB 2.0; 2x USB 3.0
  • 1x HDMI
  • Fan-less
  • Simple remote control
  • Multiple in-house and external apps
  • Good support for Kodi (aka. XBMC)
  • Linux based 32-bit OS, with most common tools and network services available, including SSHD, NFS, SMB, FTPS, rsync.


Setting up NFS shares on the NAS side is straight forward through the web based UI under “Control Panel”. You probably want to create one or more users which match your own client (e.g. laptop) user, and possibly also related group. All this can be achieved through the UI, however, for setting specific user IDs, SSH into the NAS (using the admin account) and edit /etc/passwd and /etc/group. If the IDs are changed, you’ll also have to update /mnt/HDA_ROOT/.config/nfssetting.

david:x:1001:8008:Linux User,,david,:/share/homes/david:/bin/sh
john:x:1000:8008:Linux User,,john,:/share/homes/john:/bin/sh


The reason for changing the user or group IDs manually might be to match existing IDs on the client machines. In that case, you might also have to provide this option, to make those IDs are actually used by the NAS. This setting is not permanent, so if the NAS is restarted frequently, you might consider a start-up script solution.
echo N > /sys/module/nfs/parameters/nfs4_disable_idmapping

The two relevant configuration files for the NFS setup on the NAS are /etc/exports and /mnt/HDA_ROOT/.config/nfssetting. They will be automatically configured by the UI, however some manual tweaking might be needed. I ended up with something like this, for two machines (with DNS names”laptop”, “desktop” – you can also use IP address) and two shares (“pictures”, “video”). The user (UID) and group (GID) ids will match what’s seen in the /etc/passwd and /etc/group files above.


"/share/CACHEDEV1_DATA/pictures" laptop(rw,async,no_subtree_check,insecure,no_root_squash) desktop(rw,async,no_subtree_check,insecure,no_root_squash)
"/share/CACHEDEV1_DATA/video" laptop(rw,async,no_subtree_check,insecure,no_root_squash) desktop(rw,async,no_subtree_check,insecure,no_root_squash)

"/share/CACHEDEV1_DATA/Public" *(rw,async,no_root_squash,insecure)
Version = 4.2.0
/share/CACHEDEV1_DATA/Public = FALSE
/share/CACHEDEV1_DATA/pictures = TRUE
/share/CACHEDEV1_DATA/video = TRUE
/share/CACHEDEV1_DATA/Public = *
/share/CACHEDEV1_DATA/pictures = laptop,desktop
/share/CACHEDEV1_DATA/video = laptop,desktop
/share/CACHEDEV1_DATA/Public = rw
/share/CACHEDEV1_DATA/pictures = rw,rw
/share/CACHEDEV1_DATA/video = rw,rw
/share/CACHEDEV1_DATA/Public = no_root_squash
/share/CACHEDEV1_DATA/pictures = no_root_squash,no_root_squash
/share/CACHEDEV1_DATA/video = no_root_squash,no_root_squash
/share/CACHEDEV1_DATA/Public = 65534
/share/CACHEDEV1_DATA/pictures = 1001,1000
/share/CACHEDEV1_DATA/video = 1001,1000
/share/CACHEDEV1_DATA/Public = 65534
/share/CACHEDEV1_DATA/pictures = 8008,8008
/share/CACHEDEV1_DATA/video = 8008,8008

After making any changes to the NFS config, restart the service:
/etc/init.d/nfs restart

Client side and autofs

On the client, e.g. laptop or desktop, you’d want to point your NFS mount configuration to the shares created above. However, since either NAS or more likely personal machine will be rebooted, it is useful to configure this through autofs instead of the traditional /etc/fstab config. That way, the shares will be mounted and re-mounted on demand. It will also avoid long waits at boot and shutdown of the client machines.

First, make sure the NFS and autofs packages are installed:
apt-get install portmap nfs-common autofs cifs-utils

Edit /etc/auto.master and add the following line, which specify local mount point, and specific configuration files. Note that that has to match with your setup, so you might want to change the names here. As long as the /mnt directory and config file match, you can use whatever names you like.

/mnt/qnap /etc/auto.qnap

The share specific configuration is then added in the file referenced above. It assumes you’ve named the shares on the NAS “pictures” and “video”. It also assumes the DNS name of the NAS is “qnap” (or you can use an IP here). Finally, it assumes that the shared group is called “foobar”, which should match the GID 8008 above. That GID should also be present on the client machine.

pictures -fstype=nfs,rw,soft,tcp,nolock,gid=foobar qnap:/pictures
video -fstype=nfs,rw,soft,tcp,nolock,gid=foobar qnap:/video

Finally, after making changes to the NFS / autofs confg, restart the service:
/etc/init.d/autofs restart

Comments Off