Archive for the ·

Tech

· Category...

Samsung 30 TB Enterprise SSD

no comments

Samsung recently announced mass production of a 30.72TB SAS SSD, the PM1643. The drive combines “32 of the new 1TB NAND flash packages, each comprised of 16 stacked layers of 512Gb V-NAND chips”.

There’s no word on price yet, but if we go by the street price of the older PM1633a 15.36 TB model, at around 8000 USD, then expect doubling in size to mean at last doubling in price as well. At 16000 USD, thats 53 cents per GB. Currently, spinning HDD is about 3 cents per GB, and the cheapest SSD at around 30 cents. A 75% premium for an enterprise SSD over the consumer options is perhaps not too unreasonable. However, 17 times more expensive than a HDD will not scale in the data center. In comparison, four 8 GB WD Red drives will give higher capacity and cost about 1075 USD.

Backblaze Hard Drive Stats for 2017

Comments Off

Backblaze recently published their yearly stats for hard drive failures for 2017. It covers the latest rate of replacement of spinning drives in the last quarter of 2017, plus their overall failure stats since 2013.

Besides the fact that most drives fail, it’s interesting to see their praise for Toshiba and HGST/Hitachi, where certain modules did not fail at all. Although, they readily admit for some the sampling size is low, “The HGST/Hitachi 4 TB models delivered sub 1.0% failure rates for each of the three years”, while some Seagate and WD modules have a failure rate of more than 3%.

For more stats from Backblaze see here. Also note their older article and stats on SMART data metrics used predict HDD failure.

Comments Off

Desktop build – an expensive attempt

Comments Off

Planning my next desktop build, here’s an attempt at putting together a modern machine that will last. Only problem is, at more then €2000, it’s a bit too expensive. Maybe the SSD and RAM prices will go back down to reasonable levels towards the end of the day. And maybe some of the wishes will have be be compromised on.

Details

At €1783 in new parts, and about €2085 in total, it’s a bit too expensive for a “normal” desktop machine, and can probably not be justified by what it will be used for. Some thoughts on the parts.

CPU: AMD’s new Ryzen CPU is interesting, however, the top of the line is expensive. Going for a slower clocked Ryzen 7 would save a bit, while going down to Ryzen 5 would be about half the price.

GPU: It would be fun to play with some machine learning and Tensorflow, but at the same time have a quiet passively cooled card. At €160, the Palit GeForce 1050 is already cheap, however, 4 GB RAM is maybe on the low side.

RAM: More is better, and 32 GB seems about right these days for a high-end developer machine. However, with current RAM prices, it’s too much. If they go down to about half within the year, it’s fine.

Motherboard: Asus has always been a personal favorite, and in the AM4 socket and top X370 chipset range there’s only so much to choose from. However, there might be other cheaper options besides the Crosshair VI Hero, which is an arbitrary choice at the moment. Besides the obvious component compatibility, requirements include 8 SATA ports; NVMe M.2; excellent audio I/O.

SSD: At 3500 read and 2100 write MB/s speeds, a NVMe based drive is very tempting. However, this Tom’s Hardware review points out that to get the full benefit, you’ll have to look upwards in size, and price. Again, SSD might come down towards the end of the year, but speed is always going to come at a premium.
Using my existing, but old, 2.5″ SSD is also an option, and would reduce the price further.

HDD: These will be transfered from my existing machine, so do not incur extra cost. When time comes, and the oldest 3 TB version fails, replacing with a 8+ TB drive is probably reasonable.

Case: I’ve done one previous build with the Fractal Define case, and it really gave a taste for more. At about €100, the R5 a fairly standard price for a midi-tower.

PSU: Again going fanless for no-noise, but I hope 520W is enough for the parts listed above.

CPU cooler: Even though the Ryzen packages come with a cooler, I assume it will not be cool / quiet enough. Noctua CPU coolers are the best around, and with two 14 cm fans, they only have to run at around 300 – 400 RPM to keep the CPU cool under normal load. That is almost not audible, and will then be the only fans strictly needed. (The case also comes with two fans, but through bios tuning, they can be set to trigger only when it gets to hot).

Parts

Main

Storage

The rest

Comments Off

Historical Cost of Computer Memory and Storage

Comments Off

Following up on the last post, here is the long view of cost of computer storage. In the categories magnetic disks; RAM; and flash / SSD, it plots the cheapest USD-per-byte option over time. It confirms the stagnation of price improvements previously discussed. RAM is going through its cyclic volatility, and is now at a local peak (due to supply shortage; see below), while prices for magnetic drives are flatting out. Finally, SSD have gotten more expensive over the last year, also due to supply issue. Crossing the HDD trajectory is far of, and according to Western Digital’s projections in the chart below, it will not happen in the next ten years.

The first three charts below (in white) zoom in on these developments in recent years, linear scale, at USD / GB. That way, it becomes a bit easier to examine current trends, without having to deal with the prices from 60 years ago which were twelve magnitudes higher. The following two charts use logarithmic USB / MB scale to visualize the full history, which makes the very consistent longterm trends clear.

Detailed analysis continues below the charts.

See here for the updated data and charts, and detailed information.

 
 

HDD: Recent history 2005 – present; Linear scale; USD / GB

 

 
 

SSD: Recent history 2005 – present; Linear scale; USD / GB

 

 
 

RAM: Recent history 2005 – present; Linear scale; USD / GB


 
 

Recent history 2005 – present; Logarithmic scale; USD / MB

 

 
 

Full history 1957 – present; Logarithmic scale; USD / MB

 

(Click image for larger version)
 
 

HDD

Despite sobering movements in price, there are interesting news on the horizon: Heat Assisted Magnetic Recording (HAMR) is now not too far away. Seagate talked about their plans for 2018 already at the beginning of this year, mentioning a 16 TB HAMR option. And this autumn, Western Digital demonstrated their first helium Microwave-Assisted Magnetic Recording (MAMR) drive. The principle of energized assistance in dealing with smaller magnetized areas is the same as HAMR, but MAMR uses a microwave while HAMR uses a laser. Western Digital talked about MAMR technology enabling growth to 40 TB by 2025.

Meanwhile, Toshiba announced a 9 platter PMR drive at 14 TB. This is probably closer to market than MAMR technology, and should see shelves in 2018. Even though nine platters is a very impressive feat, that direction will eventually stagnate, and so will PMR. That’s way WD’s chart below shows PMR flattening out, while in their opinion, MAMR will produce a drive double the size by 2025. Most likely, this will be realized combining all these technologies: helium filled drives; nine platters and MAMR.

It’s interesting to note that with these developments, WD believes the 10x margin from SSD will continue for at least another ten years. However, with the sizes of the HDD options now available, we might also be at a point where it has saturated the “average” consumer demand. Except for special interests like photo, video or hoarding, a 1 or 2 TB SSD is still plenty of space in a laptop, everything else is “in the cloud” anyway. Therefore, the demand for large HDD will come from data centers and other corporate systems. They can afford higher prices, so the large drives might not see the same exponential decrease in price as the smaller consumer options.

SSD

The prices of SSDs still suffer from a supply shortage of NAND chips since autumn 2016. This has caused prices to increase during 2017. Furthermore, transition from 2D planer to 3D based technology has caused problems and delays for some manufacturers.

Future developments in SSD is now more focused on IO speed, and size, than reduction in price. Samsung’s massive 15 TB PM1633a became available last year, but at $10.000, that’s for special interest only. Micron is following, and recently announced a 11 TB NVMe based drive, and interestingly designing its own rack solution for SSDs in “shared storage” configuration for the data centre.

Even for the home user, it looks like SATA based SSD storage will soon be over, with newer solutions connecting to the PCIe bus over NVMe using the M.2 or U.2 connectors, or directly through PCIe HHHL (half-height half-length) cards. This gives extreme bandwidth potential, and Kingston’s DCP1000 1.6TB has reached 7 GB/s (in RAID configuration). It’s amusing to see that the PCIe the card merely a container for four M.2 drives.

It seems SATA will be left behind in the SSD market, as most high-end products now connect to the faster PCIe bus, either over M.2 or U.2 connectors, or directly with a HHHL (half-length half-height) PCIe slot card.

RAM

Finally, memory prices have also been increasing over the last year, again due to supply shortage. Some blame Apple’s latest iPhone, and expect the supply shortage to last into 2018, when “Samsung and SK Hynix [are] expected to begin production at additional factories”.

Technology wise, DDR4 is now well established, and Corsair just released a 32 GB kit at 4 GHz, while G.Skill announced a 16 GB 4.6 GHz kit. Meanwhile, Rambus is planning the next generation: DDR5 and HBM3 (High Bandwidth Memory for GPUs). They plan to move to a 7 nm process (down from 28 nm in DDR4), and expect 4.8 – 6.4 Gb/s bandwidth. However, products will not show up before 2019, the earliest.

Comments Off

Storage prices – end of 2017

Comments Off

HDD

Since last year’s update, there are a few interesting newcomers in the very large HDD section, including 10 TB and 12 TB offerings from Western Digital and Seagate. WD has also just released a 14TB helium SMR (shingled magnetic recording) drive, and we can expect it to be available soon. However, these drives are still expensive, and the most byte-for-buck is still in the 8 TB SMR Seagate, even though it has almost kept its price-point since a year ago.

In documenting the newer drives, it has become increasingly difficult to find information on what technology they use. Product listing often don’t mention numbers of platters, helium (or not) or SMR vs. PMR (perpendicular magnetic recording), and not even WD’s own technical specification sheets go into detail. Thus, some of it has become guess-work. I’m assuming that all drives at 8 TB and above have to use helium to be able to include more platters. Furthermore, it seems anything beyond 14 TB will need to combine helium and SMR. This is more or less in line with this prediction from 2014, where the next technology stage for the next fives years is Heat Assisted Magnetic Recording (HAMR).

SSD

SSDs continue to grow in size, however, have moved only a little in price. The 1 TB Samsung EVO basic was €316 last year, and is now €238, or a 25% reduction. Most offerings are at least 10x more expensive per byte than HDD, which has always been the case.

More interestingly, NVM-M.2 based SSDs are becoming more common, and faster. The Samsung EVO 960 is now at 3500 MB/s read and 2100 MB/s write, and the 512 GB version costs €281, or a 2x premium from 2.5″ SSDs. Whether they will make a noticeable difference in day-to-day applications is up for discussion.

Flash

Finally, flash cards and drives of all types are also increasing in size, and decreasing somewhat in price. Furthermore, several modules from SanDisk have significantly increased IO speeds, with the Extreme Pro micro SDXC range at 275 MB/s read and 100 MB/s write. And the Extreme Pro CFast series, using the newer CFast standard reaches 525 MB/s read and 450 MB/s write. (However, it is not compatible with the old CF standard).

Media Type Product Capacity Price CHF Price Euros Euros / GB GBs / Euro
HDD-SMR Seagate ARCHIVE HDD 8TB 8000 GB 233.00 199.15 0.02 40.17
HDD Western Digital Blue, 5400 RPM 4TB 4000 GB 120.00 102.56 0.03 39.00
External 3.5 Western Digital My Book 8TB, USB3 8000 GB 249.00 212.82 0.03 37.59
HDD Western Digital Blue, 5400 RPM 6TB 6000 GB 199.00 170.09 0.03 35.28
External 3.5 Western Digital My Book 6TB, USB3 6000 GB 205.00 175.21 0.03 34.24
External 3.5 Western Digital My Book 4TB, USB3 4000 GB 139.00 118.80 0.03 33.67
HDD-He Western Digital Red 8TB 8000 GB 279.00 238.46 0.03 33.55
HDD Western Digital Blue, 5400 RPM 3TB 3000 GB 105.00 89.74 0.03 33.43
HDD Western Digital Red 6TB 6000 GB 215.00 183.76 0.03 32.65
HDD Western Digital Red 4TB 4000 GB 145.00 123.93 0.03 32.28
HDD-He Seagate Enterprise Capacity 12TB 12000 GB 448.00 382.91 0.03 31.34
HDD Western Digital Blue, 5400 RPM 2TB 2000 GB 75.00 64.10 0.03 31.20
HDD Western Digital Red 3TB 3000 GB 115.00 98.29 0.03 30.52
HDD-He Western Digital Red 10TB 10000 GB 385.00 329.06 0.03 30.39
HDD-He Seagate Enterprise Capacity 8TB 8000 GB 320.00 273.50 0.03 29.25
HDD Western Digital Blue, 5400 RPM 5TB 5000 GB 204.00 174.36 0.03 28.68
HDD-He Seagate Barracuda Pro 10TB 10000 GB 410.00 350.43 0.04 28.54
HDD-He Seagate Enterprise Capacity 10TB 10000 GB 411.00 351.28 0.04 28.47
External 2.5 Western Digital Elements Portable 3TB, USB3 3000 GB 125.00 106.84 0.04 28.08
External 2.5 Western Digital Elements Portable 2TB, USB3 2000 GB 85.00 72.65 0.04 27.53
HDD-He Seagate IronWolf Pro 10TB 10000 GB 425.00 363.25 0.04 27.53
External 2.5 Western Digital My Passport Ultra 4TB, USB3 4000 GB 175.00 149.57 0.04 26.74
Blu-ray Verbatim BD-R SL 50 @ 25GB 1250 GB 55.00 47.01 0.04 26.59
HDD-He Seagate IronWolf Pro 12TB 12000 GB 529.00 452.14 0.04 26.54
HDD-He Hitachi Ultrastar He8 8TB 8000 GB 362.00 309.40 0.04 25.86
HDD-He Seagate Barracuda Pro 12TB 12000 GB 557.00 476.07 0.04 25.21
HDD Western Digital Red 2TB 2000 GB 94.00 80.34 0.04 24.89
External 2.5 Western Digital My Passport Ultra 3TB, USB3 3000 GB 155.00 132.48 0.04 22.65
HDD Western Digital Blue, 5400 RPM 1TB 1000 GB 55.00 47.01 0.05 21.27
External 2.5 Western Digital Elements Portable 1TB, USB3 1000 GB 65.00 55.56 0.06 18.00
External 2.5 Western Digital My Passport Ultra 1TB, USB3 1000 GB 89.00 76.07 0.08 13.15
Blu-ray Verbatim BD-R SL 10 @ 25GB 250 GB 23.10 19.74 0.08 12.66
Blu-ray Verbatim BD-R DL 10 @ 50GB 500 GB 47.70 40.77 0.08 12.26
DVD-R Verbatim 16x DVD-R 100 @ 4,7GB 470 GB 46.00 39.32 0.08 11.95
DVD+R DL Verbatim 8x DVD+R DL 50 @ 8,5GB 425 GB 81.50 69.66 0.16 6.10
DVD+R DL Verbatim 8x DVD+R DL 25 @ 8,5GB 213 GB 41.00 35.04 0.16 6.06
SSD Samsung SSD 850 EVO Basic, TLC, 1TB 1000 GB 279.00 238.46 0.24 4.19
SSD Crucial MX300 SSD, 3D, 2000GB 2000 GB 579.00 494.87 0.25 4.04
SSD Crucial MX300 SSD, 3D, 1050GB 1050 GB 309.00 264.10 0.25 3.98
SSD Crucial MX300 SSD, 3D, 525GB 525 GB 165.00 141.03 0.27 3.72
USB Flash SanDisk Ultra, USB 3.0, 256GB 256 GB 84.90 72.56 0.28 3.53
SSD Samsung SSD 850 EVO Basic, TLC, 2TB 2000 GB 699.00 597.44 0.30 3.35
SSD Samsung SSD 850 EVO Basic, TLC, 500GB 500 GB 175.00 149.57 0.30 3.34
SSD Crucial MX300 SSD, 3D, 275GB 275 GB 99.00 84.62 0.31 3.25
SSD Samsung SSD 850 EVO Basic, TLC, 4TB 4000 GB 1475.00 1260.68 0.32 3.17
USB Flash SanDisk Ultra, USB 3.0, 64GB 64 GB 24.40 20.85 0.33 3.07
SSD Samsung SSD 850 EVO Basic, TLC, 250GB 250 GB 103.00 88.03 0.35 2.84
USB Flash SanDisk Ultra, USB 3.0, 32B 32 GB 14.60 12.48 0.39 2.56
SSD Samsung SSD 850 Pro, MLC, 1024GB 1024 GB 479.00 409.40 0.40 2.50
SSD Samsung SSD 850 Pro, MLC, 2048GB 2048 GB 969.00 828.21 0.40 2.47
SSD Samsung SSD 850 Pro, MLC, 512GB 512 GB 255.00 217.95 0.43 2.35
CD-R Verbatim CD-R 100 @ 700MB 70 GB 35.70 30.51 0.44 2.29
SSD Samsung SSD 850 Pro, MLC, 256GB 256 GB 145.00 123.93 0.48 2.07
SDXC SanDisk Extreme Pro SDXC U3, Class 10/UHS 3 95/90MB/s, 128GB 128 GB 79.60 68.03 0.53 1.88
SDXC SanDisk Extreme Pro SDXC U3, Class 10/UHS 3 95/90MB/s, 256GB 256 GB 160.00 136.75 0.53 1.87
SSD-NVM-M.2 Samsung SSD 960 Pro, M.2 2280, MLC, 3500/2100MB/s, 1000GB 1000 GB 629.00 537.61 0.54 1.86
SSD-NVM-M.2 Samsung SSD 960 Pro, M.2 2280, MLC, 3500/2100MB/s, 2000GB 2000 GB 1279.00 1093.16 0.55 1.83
SSD-NVM-M.2 Samsung SSD 960 Pro, M.2 2280, MLC, 3500/2100MB/s, 512GB 512 GB 329.00 281.20 0.55 1.82
SDXC SanDisk Extreme Pro SDXC U3, Class 10/UHS 3 95/90MB/s, 64GB 64 GB 47.90 40.94 0.64 1.56
SDXC SanDisk Extreme Pro SDXC U3, Class 10/UHS 3 95/90MB/s, 512GB 512 GB 389.00 332.48 0.65 1.54
microSDXC SanDisk Extreme Pro microSDXC, Class 10, 100/90MB/s, 64GB 64 GB 50.10 42.82 0.67 1.49
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 64GB 64 GB 75.00 64.10 1.00 1.00
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 256GB 256 GB 326.00 278.63 1.09 0.92
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 128GB 128 GB 168.00 143.59 1.12 0.89
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 32GB 32 GB 44.00 37.61 1.18 0.85
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 64GB 64 GB 92.70 79.23 1.24 0.81
microSDXC SanDisk Extreme Pro microSDXC, Class 10, 100/275MB/s, 128GB 128 GB 220.00 188.03 1.47 0.68
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 32GB 32 GB 57.20 48.89 1.53 0.65
microSDXC SanDisk Extreme Pro microSDXC, Class 10, 100/275MB/s, 64GB 64 GB 125.00 106.84 1.67 0.60
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 16GB 16 GB 37.00 31.62 1.98 0.51
Compact Flash SanDisk Extreme Pro 450MB/s, CFast 2.0, 128GB 128 GB 389.00 332.48 2.60 0.38
Compact Flash SanDisk Extreme Pro 450MB/s, CFast 2.0, 64GB 64 GB 236.00 201.71 3.15 0.32

Exchange rate: 1 Euro = 1.170000 CHF.

Comments Off

Wired: Mother Earth Mother Board

Comments Off

In a blast from the past, this 1996 Wired article about the intrigues of the underwater optical fiber cable business resurfaced. It’s a real long-read about the construction of the 28000 km long FLAG cable, connecting UK to Japan, and various countries and obstacles along the way. Still, it’s well worth the journey. Enjoy!

Comments Off

Asus Thinker Board

Comments Off

Asus recently released their single board computer “Thinker Board”, modeled on the same form-factor and pin layout as the Raspberry Pi 2 and 3, and at the same price point. It comes with some interesting upgrades over the RPi 3: 2 GB RAM (over 1 GB); 1.8 GHz quad core (over 1.2 GHz quad) dedicated Realtek 1 Gb/s Ethernet; Realtek audio codec; and support for 4k HDMI out. It could make both a good media player platform, as well as a usable desktop box.

In their early review, Hackaday laments the lack of website and community. The former has since been addressed, and Asus’ official site is actually rather slick fun and informative, and includes a Debian based OS ISO and other downloads. They have also put up a Facebook page, but it’s mostly linked product reviews and blog post for now.

Comments Off

QNAP TS-431P NAS

Comments Off

Tasked with setting up another NAS solution, I went for the simple 4-bay QNAP TS-431P, since the previous QNAP gave a good impression. This one does not have HDMI; in fact the only external ports are three USB 3 ports and two RJ-45 Gigabit Ethernet – no eSATA. Compared to its previous version, TS-431P has double the amount of RAM (for a total of 1 GB), and a slightly faster CPU. Software is as expected from QNAP.

The following describes the standard disk layout when using a single / stand alone disk, which still gets formatted as RAID + LVM, and optionally an encrypted partition.

Windows shares setup is covered at the end.

 

RAID and LVM

The QNAP NAS OS supports encryption, and I wanted to evaluate how secure this is in terms of failure. That is, if a disk fails, or the NAS itself fails, can you recover the data from the remaining disks. You can, but there are a few steps to watch out for.

First of all, even if each disk in the NAS is set up as “Single Disk / Stand Alone”, using no RAID, the NAS will still configure each partition on the separate disks as RAID partitions and in a LVM2 single volume group. That means you’ll need the Linux RAID and LVM tools and commands to mount. (Some useful discussion here).

General install, scan and list commands:

apt-get install mdadm lvm2

mdadm --assemble --scan
cat /proc/mdstat
lsblk

vgscan
lvs
lvscan
lvmdiskscan
lvdisplay

And to mount, use the example commands below.

Note: The device names and volume names will most certainly be different. Use the commands above to understand the layout of the disk you’re working with.

Also note: if the mdadm scan command does not make all the RAID partitions available, it could be due to an existing /etc/mdadm/mdadm.conf file. You could try to rename it to mdadm.conf.old, or append the RAID details with mdadm –detail –scan >> /etc/mdadm/mdadm.conf. See here for more.

mdadm --assemble --scan
lsbkl

vgscan
vgchange -ay vg1
lsblk

mount /dev/vg1/lv1 /mnt/tmp

That should mount the drive, however, if you are working with an encrypted drive, you’ll need one more step before the mount command works, so ignore the last line and continue reading.

 

Encryption

If you have followed the steps above, and type lsblk, part of the output will look something like this. It shows the layers so far: from the physical partition (sdb3) to the raid1 partition (md126), which contains two LVM logical volumes. In this case, the second is the LUKS encrypted main partition.

├─sdb3              8:19   0   3.6T  0 part  
│ └─md126           9:126  0   3.6T  0 raid1 
│   ├─vg288-lv545 254:1    0  37.2G  0 lvm   
│   └─vg288-lv2   254:2    0   3.6T  0 lvm   

So, we continue to decrypt, and mount it. Using cryptsetup luksDump, you can confirm that there is only one keyslot on the encrypted volume, which uses the paraphrase you typed in when installing the drive. However, the password is salted and MD5 hashed, so you have to generate a key-file with the new key. The salt is YCCaQNAP when using the mkpasswd tool, but encoded as $1$YCCaQNAP$ when calling the crypt library. Also make sure the key-file does not contain a newline.

cryptsetup luksDump /dev/vg288/lv2

mkpasswd --hash=md5 --salt='YCCaQNAP' | tr -d '\n' > /tmp/key-file
cryptsetup luksOpen /dev/vg288/lv2 unenc_lv2 --key-file /tmp/key-file

mkdir /mnt/tmp
mount /dev/mapper/unenc_lv2 /mnt/tmp
lsblk

You now have access to the data files on the drive.

Coming back to the original question: Is this a resilient way of storing files? There are certainly a lot of layers, and although they each are well established technologies, they add complexity. Especially in the scenario when you would need to do the recovery it adds additional stress. Ideally, a single partition, no RAID, no LVM could be used. However, it seems that is not possible with the stock QNAP OS, since it will format any drive which is added to the NAS in its own way, including the RAID + LVM stack. In fact, this warning from the user manual is worthing taking careful note of:

Caution: Note that if you install a hard drive (new or used) which has never been installed on the NAS before, the hard drive will be formatted and partitioned automatically and all the disk data will be cleared.

 

Windows shared folders

The Windows sharing is easier to set up, but not without hurdles. On the local network, it typically will work out of the box when you point Windows Explorer to \\NAS_DOMAIN. If you need to connect across a firewall, you’ll have to open or forward at the minimum TCP 139,445, but possibly more ports on TCP and UDP.

The problem is that when sharing these ports cross the Internet, you will very likely run into other firewalls. ISP might block the default 139 or 445 ports. Although it is possible to port-forward to non-default ports, and this will work on Mac and Android, Windows will not accept it. A work-round if all else fails is therefore to set up a VPN or tunnel. Using SSH, this can easily be done with:

ssh -L 0.0.0.0:139:qnap:139 -L 0.0.0.0:445:qnap:445 admin@remotehost

Here it is assumed the NAS has DNS “qnap” on its local network, otherwise, replace with it’s IP. You might also want to forward 8080, forward SSH on a different port (e.g. 2222), as well as keep it running with autossh:

autossh -M 12340 -f -N -p 2222 -L 0.0.0.0:139:qnap:139 -L 0.0.0.0:445:qnap:445 -L 0.0.0.0:8080:qnap:8080 admin@remotehost

Finally, if using only Windows machines to connect to the shares, there is the option of combining multiple shares into one. However, if other OSes also connect, you probably want to skip that.

Comments Off

Facebook exit stage left

Comments Off

In an older article about social network trends, I discussed how Facebook in particular was far past its peak as measured by Google Trends. And that trend has not turned. There is a smaller and smaller fraction of web searches for the term “facebook”. It is now around a similar level as in 2009. Some of this can be explained by the fact that more people use the Facebook mobile app to access the social network, and therefore do not need to search Google to access the site. However, as pointed out by this Guardian article, despite claiming 1.79 billion users, 2016 was the year Facebook “became the bad guy”. In the Western world, interest was saturated several years ago, and new users come from other regions and possibly where Facebook have been blocked. They now plan to adapt their site and network to China’s censor and surveillance requirements.

Yet, if you are an investor in the social network, you might want to hold on a bit longer before you sell. In recent years, Facebook has snapped up both of the new popular social apps Whatsapp and Instagram (in 2014 and 2012 respectively). They continue their upwards trend, and will do so for many years to come. Compared to Facebook they both pale on Google Trends, meaning there’s a lot of potential.


Comments Off

12 TB helium and 14TB helium + SMR announced

Comments Off

HGST has just announced a new helium filled drive of 12 TB. The increase in capacity comes from an impressive and unexpected 8 platter design. Meanwhile, Western Digital forecasts that they will take the Ultrastar He12 based disk to 14 TB by combining it with Shingled Magnetic Recording (SMR). No prices are indicated yet.

Comments Off

Review: “The Internet Is Not the Answer”, Andrew Keen

Comments Off

Andrew Keen is a bitter man. He longs for the time when his family ran a tailor business in London and middle class people could afford to buy clothes from their store. He resents the fact that his own music Internet site failed, while seemingly similar sites like MySpace, Spotify came to be valued in billions. For some reason he mourns the demise of Kodak and its film roll processing centre in Rochester, NY. And most of all, he despises rich folk, but not any billionaire, just those who happen to have made their money through the Internet. Keen’s book “The Internet Is Not the Answer” reads a bit like a rant towards all these things, while blaming it all on Silicon Valley and The Internet. The solutions he favour are mostly based on government regulation: six strikes laws for copyright infringement; antitrust and monopoly busting; mixed with labor unions.

Even though Keen’s book has a bitter tone throughout, he does touch on important points regarding increasing wealth disparity, middle class jobs being replaced by automation and far fewer specialized jobs, monopolistic mega-cooperations, centralized services. He takes on Amazon, Google, Facebook, Instagram, Twitter, WhatsApp. He has done thorough research, and his book includes a substantial reference section. It is just that his conclusions doesn’t always align with actual causes and effects: Take the downfall of Kodak, where he spends a full chapter lamenting Instagram for the killing of film processing. Companies like Canon and Nikon which developed and sell high quality DSLR cameras and thus more directly caused the replacement of film are not mentioned.

Similarly, Keen reviews the history of the early days of the Internet, and its inventors and pioneers like Paul Baran, Bob Taylor, Bob Kahn, Vint Cerf, Tim Berners-Lee and points out how altruistic and anti-commercial they were. He contrasts this to “winner-take-all” companies in today’s economy. However, he does not discuss the seemingly obvious conclusion that what we’re lacking from Facebook, Instagram, Twitter are common open standards and protocols, which is what made early technology successful and long-lasting. Free and Open Source Software (FOSS) does not enter Keen’s field of view at all.

Maybe not worth it

If you already work in tech, and have good insight into these topics, Keen does not bring much new to the table. In fact, he disappoints in that regard. However, if you are interested in gentrification in San Francisco, Kodak in Rochester, or just want to hear a different point of view, give the book a try.

What Keen does have going for him, is that he is a very good writer. He writes almost poetically, albeit with great sarcasm, about topics like Internet economy, government regulation, and pretentious billionaires. Sound bites like the one below at least make the book entertaining.
While talking about the cult and praise of “failure” in tech-companies:

“Instagram actually represents the reverse side of Silicon Valley’s cult of failure. In the Valley, the rich and famous claim to be failures; on social networks like Instagram, millions of failures claim to be rich and famous”.

Finally, although it’s no way to be sure, it sometimes feels like the writes of Silicon Valley, the TV comedy series, have studied the book thoroughly and lifted several ideas from Keen onto the screen. What Keen scorns, like the double-speak; the feel-good big-company efforts and speeches; and the general Valley culture has made the TV series a hit. Keen’s book makes the series even more fun to watch.

Comments Off

Storage prices – end of 2016

Comments Off

The last half-year have seen few movements in spinning disk prices, and actually some have gotten a few coins more expensive. It is still the 8 TB SMR Seagate drive which give most storage for money, only with the exception of a special offer on an external 8 TB WD disk. In fact, several of the external offerings are now cheaper than their internal counterparts.

Also worth noting, is that Western Digital has deprecated their long time cheapest Green line, in favour of the Blue color. However, take care: The Blue line contains both 5400 and 7200 RPM drives. The ones listed here are all 5400 RPM.

On the SSD side, things have also not changed much. There were some promises, but not much in terms of concrete offerings this year.

Finally, many of the flash cards are getting more competitive. Indeed, the the Sandisk Ultra 256 GB key, as well as the 200 GB micro SD card are now pretty great offering for on-the-go storage. They’d fit very decent sized music collections, picture albums, etc.

Media Type Product Capacity Price CHF Price Euros Euros / GB GBs / Euro
External 3.5 Western Digital My Book 8TB, USB3 8000 GB 250.00 233.64 0.03 34.24
HDD-SMR Seagate ARCHIVE HDD 8TB 8000 GB 266.00 248.60 0.03 32.18
HDD Seagate Desktop 4TB 4000 GB 134.00 125.23 0.03 31.94
External 3.5 Western Digital My Book 6TB, USB3 6000 GB 202.00 188.79 0.03 31.78
External 3.5 Western Digital Elements Desktop 4TB, USB3 4000 GB 135.00 126.17 0.03 31.70
HDD Western Digital Blue, 5400 RPM 3TB 3000 GB 107.00 100.00 0.03 30.00
HDD Western Digital Blue, 5400 RPM 4TB 4000 GB 145.00 135.51 0.03 29.52
External 3.5 Western Digital My Book 4TB, USB3 4000 GB 149.00 139.25 0.03 28.72
SMR External 3.5 Seagate Backup Plus Desktop 8TB 8000 GB 300.00 280.37 0.04 28.53
External 3.5 Western Digital My Book 3TB, USB3 3000 GB 115.00 107.48 0.04 27.91
External 3.5 Western Digital Elements Desktop 3TB, USB3 3000 GB 118.00 110.28 0.04 27.20
HDD Western Digital Blue, 5400 RPM 6TB 6000 GB 244.00 228.04 0.04 26.31
HDD Western Digital Red 4TB 4000 GB 164.00 153.27 0.04 26.10
HDD Western Digital Blue, 5400 RPM 5TB 5000 GB 208.00 194.39 0.04 25.72
External 2.5 Western Digital My Passport Ultra 3TB, USB3 3000 GB 125.00 116.82 0.04 25.68
HDD Western Digital Red 3TB 3000 GB 126.00 117.76 0.04 25.48
HDD Western Digital Red 6TB 6000 GB 252.00 235.51 0.04 25.48
HDD Western Digital Blue, 5400 RPM 2TB 2000 GB 84.90 79.35 0.04 25.21
HDD-He Western Digital Red 8TB 8000 GB 345.00 322.43 0.04 24.81
External 2.5 Western Digital Elements Portable 2TB, USB3 2000 GB 88.00 82.24 0.04 24.32
HDD Western Digital Red 2TB 2000 GB 98.00 91.59 0.05 21.84
External 2.5 Western Digital My Passport Ultra 2TB, USB3 2000 GB 103.00 96.26 0.05 20.78
HDD-He Hitachi Ultrastar He6 6TB 6000 GB 361.00 337.38 0.06 17.78
HDD Western Digital Blue, 5400 RPM 1TB 1000 GB 61.70 57.66 0.06 17.34
External 2.5 Western Digital Elements Portable 1TB, USB3 1000 GB 62.00 57.94 0.06 17.26
HDD-He Hitachi Ultrastar He8 8TB 8000 GB 517.00 483.18 0.06 16.56
External 2.5 Western Digital My Passport Ultra 1TB, USB3 1000 GB 72.00 67.29 0.07 14.86
Blu-ray Verbatim BD-R SL 10 @ 25GB 250 GB 23.90 22.34 0.09 11.19
DVD-R Verbatim 16x DVD-R 100 @ 4,7GB 470 GB 46.00 42.99 0.09 10.93
Blu-ray Verbatim BD-R DL 10 @ 50GB 500 GB 49.00 45.79 0.09 10.92
DVD+R DL Verbatim 8x DVD+R DL 25 @ 8,5GB 213 GB 39.00 36.45 0.17 5.83
DVD+R DL Verbatim 8x DVD+R DL 50 @ 8,5GB 425 GB 95.40 89.16 0.21 4.77
SSD Crucial BX200 SSD, MLC, 480GB 480 GB 139.00 129.91 0.27 3.69
USB Flash SanDisk Ultra, USB 3.0, 256GB 256 GB 78.00 72.90 0.28 3.51
SSD Samsung SSD 850 EVO Basic, TLC, 500GB 500 GB 165.00 154.21 0.31 3.24
SSD Crucial BX200 SSD, MLC, 240GB 240 GB 81.00 75.70 0.32 3.17
SSD Samsung SSD 850 EVO Basic, TLC, 1TB 1000 GB 339.00 316.82 0.32 3.16
SSD Crucial MX200 SSD, MLC, 500GB 500 GB 175.00 163.55 0.33 3.06
SSD Crucial MX200 SSD, MLC, 1000GB 1000 GB 374.00 349.53 0.35 2.86
SSD Crucial MX200 SSD, MLC, 250GB 250 GB 95.30 89.07 0.36 2.81
SSD Crucial BX100 SSD, MLC, 500GB 500 GB 193.00 180.37 0.36 2.77
SSD Samsung SSD 850 EVO Basic, TLC, 250GB 250 GB 100.00 93.46 0.37 2.68
microSDXC SanDisk Ultra Premium microSDXC 90MB/s, 200GB 200 GB 85.00 79.44 0.40 2.52
USB Flash SanDisk Cruzer Edge Flash Drive 64GB 64 GB 27.80 25.98 0.41 2.46
SSD Samsung SSD 850 Pro, MLC, 1024GB 1024 GB 454.00 424.30 0.41 2.41
USB Flash SanDisk Ultra, USB 3.0, 64GB 64 GB 28.80 26.92 0.42 2.38
CD-R Verbatim CD-R 100 @ 700MB 70 GB 32.60 30.47 0.44 2.30
SSD Samsung SSD 850 Pro, MLC, 512GB 512 GB 239.00 223.36 0.44 2.29
SSD Samsung SSD 850 Pro, MLC, 256GB 256 GB 139.00 129.91 0.51 1.97
USB Flash SanDisk Cruzer Edge Flash Drive 32GB 32 GB 18.80 17.57 0.55 1.82
USB Flash SanDisk Ultra, USB 3.0, 32B 32 GB 19.50 18.22 0.57 1.76
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 3, 95/90MB/s, 128GB 128 GB 82.80 77.38 0.60 1.65
SSD-NVM-M.2 Samsung SSD 950 Pro, M.2 2280, MLC, 2500/1500MB/s, 512GB 512 GB 361.00 337.38 0.66 1.52
SSD-NVM-M.2 Samsung SSD 950 Pro, M.2 2280, MLC, 2200/900MB/s, 256GB 256 GB 187.00 174.77 0.68 1.46
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 3, 95/90MB/s, 512GB 512 GB 389.00 363.55 0.71 1.41
SSD Samsung SSD 850 Pro, MLC, 128GB 128 GB 99.70 93.18 0.73 1.37
SSD Samsung SSD 850 EVO Basic, TLC, 120GB 120 GB 95.70 89.44 0.75 1.34
SDXC SanDisk Extreme Pro SDXC UHS-I, Class 10/UHS 1, 95/90MB/s, 64GB 64 GB 55.00 51.40 0.80 1.25
USB Flash SanDisk Cruzer Edge Flash Drive 16GB 16 GB 14.90 13.93 0.87 1.15
SDHC SanDisk Extreme Pro SDHC UHS-I, Class 10/UHS 1, 95/90MB/s, 32GB 32 GB 33.00 30.84 0.96 1.04
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 64GB 64 GB 75.00 70.09 1.10 0.91
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 256GB 256 GB 347.00 324.30 1.27 0.79
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 128GB 128 GB 175.00 163.55 1.28 0.78
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 64GB 64 GB 98.50 92.06 1.44 0.70
USB Flash SanDisk Cruzer Edge Flash Drive 8GB 8 GB 12.90 12.06 1.51 0.66
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 32GB 32 GB 52.70 49.25 1.54 0.65
SDHC SanDisk Extreme Pro SDHC UHS-I, Class 10/UHS 1, 95/90MB/s, 16GB 16 GB 27.00 25.23 1.58 0.63
SDXC SanDisk Extreme Pro SDXC UHS-II, UHS 3, 280/250MB/s, 64GB 64 GB 117.00 109.35 1.71 0.59
Compact Flash SanDisk Extreme Pro 160MB/s, UDMA 7, 32GB 32 GB 60.10 56.17 1.76 0.57
Compact Flash SanDisk Extreme 120MB/s, UDMA 7, 16GB 16 GB 36.40 34.02 2.13 0.47
SDHC SanDisk Extreme Pro SDHC UHS-II, UHS 3, 280/250MB/s, 16GB 16 GB 52.70 49.25 3.08 0.32

Exchange rate: 1 Euro = 1.070000 CHF.

Comments Off

Jekyll: Include partial snippets of code

Comments Off

The Jekyll include tag is useful when including files or templates on a page. Combined with the highlight tag, it makes including code snippets easy. However, it will include the complete file, and often it is desirable to include only a few lines, or maybe a method. That could of course be done by simply copy/pasting the code in question into the article, but then the code gets out of sync if the example file is changed.

The basic usecase is something like this:

{% highlight java %}
    {% include src/HelloWorld.java %}
{% endhighlight %}

Ruby based plugin

A Jekyll tag to include only a section of a file would be great. As far as I can tell, that does not exist yet, so I started writing one. Unfortunately, Github Pages does not allow custom plugin for security reasons. There are work-arounds for that, but it also makes the deployment more complex, and loses the convenience of being able to edit the articles and code directly on github.com.

Sans error handling or caching, a simple implementation could look like this. It works outside Github Pages, so it’s a start.

module Jekyll
  class IncludeLines < Liquid::Tag
     Syntax = /(#{Liquid::QuotedFragment}+)\s(\d+)\s(\d+)\s\z/o
     
     def initialize(tag_name, markup, options)
       super
       if markup =~ Syntax
         @file = $1
         @startline = $2.to_i
         @endline = $3.to_i
       else
         raise "Syntax error in includelines: " + markup
       end
     end
 
     def render(context)
       lines = IO.readlines(context.evaluate(@file))
       part = lines.drop(@startline)
       part.take(@endline - @startline)
     end
 
   end
 end
 
 Liquid::Template.register_tag('includelines', Jekyll::IncludeLines)
 

Liquid slice and split

Using the Liquid capture block, it's possible to read a file and store it as a string variable. It can then be processed by Liquid instead of the plugin, and works fine with Github pages. The Liquid syntax is certainly verbose, but it gets the job done.

An initial implementation cutting the file content as a single string looked like this. However, it is far from ideal, since the character index and count will shift with any source code modifications on the included file.

{% capture filecontent %}
    {% include src/HelloWorld.java %}
{% endcapture %}

{% highlight java %}
    {{ filecontent | slice: 132, 57 }}
{% endhighlight %}

A slightly better solution uses the same idea, but operates on line numbers instead. It is almost as fragile when it comes to changes, but at least usable.

{% capture filecontent %}
    {% include src/HelloWorld.java %}
{% endcapture %}

{% assign lines = filecontent | newline_to_br | split: '<br />' %}
{% highlight java %}
    {% for line in lines offset:10 limit:5 %}{{ line }}{% endfor %}
{% endhighlight %}

A helper include file implementing this idea can be found here. It can be used like this:

{% include includelines filename='src/HelloWorld.java' start=10 count=5 %}

Include method

Ideally, it would be possible to mark the start of a line to include, and then indicate how much should be included. Improving on the line based iterator above, this helper file does that. Usage goes like this:

{% highlight java %}
    {% include includemethod filename='src/HelloWorld.java' method='test()' before=5  after=1 %}
{% endhighlight %}

It also adds options to include lines before and after the specified method, for example for comment blocks or further methods below the first. There are of course some extensions which could be made, e.g. to include multiple split sections; support other non-C like languages, etc. The linked code is under the GPL 3 license, so feel free to improve.

Comments Off

Backblaze: SMART stats to predict hard drive failure

Comments Off

Over the years, Backblaze has published several interesting reports and statistics on their harddrive performance and failures. In their latest post, they look at which SMART metrics they monitor and react to. They say, “when the RAW value for one of these five attributes is greater than zero, we have a reason to investigate”.

SMART 5 Reallocated Sectors Count
SMART 187 Reported Uncorrectable Errors
SMART 188 Command Timeout
SMART 197 Current Pending Sector Count
SMART 198 Uncorrectable Sector Count

They go on to point out, that although a single one of those metrics might only indicate 30% to 40% probability of a failed drive, when combined the probability increases to 76%. Furthermore, looking at multiple failed values, and their rate of increase might also help determine if a drive is about to fail. A higher rate is usually worse. Finally, they note that the the SMART 189 – High Fly Writes is another good candidate to consider, and again when looking at the rate.

On a related note, here’s a note on Google’s data on their SSD failures.

Comments Off

Getting started with GitHub Pages and Jekyll

Comments Off

In the beginning, there were static HTML pages, TABLE-tags, and FTP; later came dynamic sites, WordPress and other Content Management Systems; but now we are back to static pages again, albeit templated. So goes Quinn Supplee’s narrative of the move to Jekyll, Markdown and Liquid based static sites. Add in GitHub for free hosting, and it makes a very compelling offering for a small site run by tech savvy people. It’s not your web sites for dummies solution.

Enabling web page hosting from a Github repository is quick, as explained here. And setting a custom domain is a single setting on Github, and additional setting up your domain DNS. The Github Settings panel have a default example site template generator, so with that you’ll have some pages to look at in few clicks. Now you can download the generated files, modify and upload with standard git commands. (Replace username/repository with your own names).

git clone http://github.com/username/repository
git commit
git push

Github uses Jekell to statically generate the site, and this requires a special file and directory structure. It is of course possible to experiment with the files directly on Github, however it might be easier to edit and compile locally. For that, a local install of Jekyll and relevant dependencies and tools is required.

sudo apt-get install ruby ruby-dev rubygems-integration nodejs gcc make
sudo gem install jekyll jekyll-docs jekyll-feed jekyll-paginate bundler minima iconv

On older Debian (7 – Wheezy) or Ubuntu (14.04) distributions, the ruby packages where versioned incorrectly, so the 2.0 version is required for the ruby package and gem binary.

sudo apt-get install ruby2.0 ruby2.0-dev rubygems-integration nodejs ruby-mkrf gcc make
sudo gem2.0 install jekyll jekyll-docs jekyll-feed jekyll-paginate bundler minima iconv

With that in place, you can generate a new blank site, and make Jekyll serve it locally on http://127.0.0.1:4000/ . Of course, the final destination would be the Github repository.

jekyll new test --skip-bundle
cd test
jekyll serve

For more advanced options and functions, the Jekyll documentation is good. There are some Jekeyll based example sites here. In particular, Patrick Mckinley’s pagination example with source looks interesting.

Comments Off