Category: General

Posted on

Repurposing netapp disk trays with FreeBSD and ZFS

I recently received a couple of DS14-MKII Netapp disk arrays and a pile of spare parts for free. I have no use (and no desire to pay for the excessive power) for a full blown Netapp but I thought that perhaps I could re-use the disks for something. After a little searching on the web it seemed possible to re-purpose these for use as generic JBOD disk trays, Ben Rockwood has an old article on his site describing using older DS8 and DS9 shelves with Linux and LVM. Among the pile of Netapp parts I was able to rescue was the Netapp’s fibre-channel adapter and Ethernet cards, these appeared to be generic PCI cards.

Once home I looked over the pile of disks and hardware I had and realized I had about twenty 300GB disks and at least 30 144GB disks. I picked one of the trays and filled it with the same model 300GB Netapp disks and set the tray ID number to 0. Once powered on it beeped and roared to life. As soon as it had spun up all the disks and initialized the tray the fans went down to a nice quiet level so it actually wasn’t too loud. For my own information I took a power reading of the disk tray and it draws around 260 watts. Not great since the whole tray is only about 3.8TB of space but for experimentation it will do.

Since my server at home is a Foxconn low profile machine based on the Intel Atom D510 I would need a low profile fibre-channel card. After looking at the online classifieds I realized that a 2Gb low-profile fibre card wasn’t going to come cheap. I examined the card I received from the Netapp filer head which turned out to be a Qlogic 2312 dual port 2Gbit adapter. It appeared that it was a low profile capable card, I was just missing the low profile adapter plate. I removed the existing full height PCI plate and slid the card into the slot with no back plate. It fit perfectly and the optical connectors seemed to keep the card from wiggling too much from side to side. The card is a 64bit PCI card but my research showed that it would work in a 32bit slot, I would just have to be careful not to jostle the card too much.

I now focused my attention on getting the Qlogic card working in FreeBSD. Looking at the HCL it looked like the card was on the supported list for the isp driver. I booted the system and it appeared to auto-load the isp driver properly but I didn’t see any attached disk. After examining the logs I noticed the following error.

Jan 20 22:57:45 helix kernel: isp0: port 0xe800-0xe8ff mem 0xfebff000-0xfebfffff irq 21 at device 0.0 on pci3
Jan 20 22:57:45 helix kernel: isp0: [ITHREAD]
Jan 20 22:57:45 helix kernel: isp0: Polled Mailbox Command (0x2) Timeout (1000000us) (started @ isp_reset:953)
Jan 20 22:57:45 helix kernel: isp0: Polled Mailbox Command (0x8) Timeout (100000us) (started @ isp_reset:1017)
Jan 20 22:57:45 helix kernel: isp0: Mailbox Command 'ABOUT FIRMWARE' failed (TIMEOUT)
Jan 20 22:57:45 helix kernel: device_attach: isp0 attach returned 6
Jan 20 22:57:45 helix kernel: isp1: port 0xe400-0xe4ff mem 0xfebfe000-0xfebfefff irq 22 at device 0.1 on pci3
Jan 20 22:57:45 helix kernel: isp1: [ITHREAD]
Jan 20 22:57:45 helix kernel: isp1: Polled Mailbox Command (0x2) Timeout (1000000us) (started @ isp_reset:953)
Jan 20 22:57:45 helix kernel: isp1: Polled Mailbox Command (0x8) Timeout (100000us) (started @ isp_reset:1017)
Jan 20 22:57:45 helix kernel: isp1: Mailbox Command 'ABOUT FIRMWARE' failed (TIMEOUT)
Jan 20 22:57:45 helix kernel: device_attach: isp1 attach returned 6

After a quick search it appeared that I had to load the isp-fw module which loads the compatible Qlogic firmware into the card before it’s initialized by the kernel driver. I added the following two lines to /boot/loader.conf and rebooted the system.

ispfw_load="YES"
isp_load="YES"

Success!

Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
Jan 20 23:20:54 helix kernel: ispfw: registered firmware
...
Jan 20 23:20:54 helix kernel: isp0: port 0xe800-0xe8ff mem 0xfebff000-0xfebfffff irq 21 at device 0.0 on pci3
Jan 20 23:20:54 helix kernel: isp0: [ITHREAD]
Jan 20 23:20:54 helix kernel: isp1: port 0xe400-0xe4ff mem 0xfebfe000-0xfebfefff irq 22 at device 0.1 on pci3
Jan 20 23:20:54 helix kernel: isp1: [ITHREAD]
...
da18: Fixed Direct Access SCSI-3 device
da18: 200.000MB/s transfers WWNN 0x2000001862e6b359 WWPN 0x2200001862e6b359 PortID 0x1b
da18: Command Queueing enabled
da18: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da19 at isp0 bus 0 scbus0 target 6 lun 0
da19: Fixed Direct Access SCSI-3 device
da19: 200.000MB/s transfers WWNN 0x2000001862eebf72 WWPN 0x2200001862eebf72 PortID 0x1e
da19: Command Queueing enabled
da19: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da21 at isp0 bus 0 scbus0 target 9 lun 0
da21: Fixed Direct Access SCSI-3 device
da21: 200.000MB/s transfers WWNN 0x2000001862d03be1 WWPN 0x2200001862d03be1 PortID 0x18
da21: Command Queueing enabled
da21: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da20 at isp0 bus 0 scbus0 target 7 lun 0
da20: Fixed Direct Access SCSI-3 device
da20: 200.000MB/s transfers WWNN 0x2000001862cdf7f2 WWPN 0x2200001862cdf7f2 PortID 0x1d
da20: Command Queueing enabled
da20: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da22 at isp0 bus 0 scbus0 target 13 lun 0
da22: Fixed Direct Access SCSI-3 device
da22: 200.000MB/s transfers WWNN 0x20000018627f3052 WWPN 0x22000018627f3052 PortID 0x1
da22: Command Queueing enabled
da22: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da26 at isp0 bus 0 scbus0 target 5 lun 0
da26: Fixed Direct Access SCSI-3 device
da26: 200.000MB/s transfers WWNN 0x2000001862d03fc2 WWPN 0x2200001862d03fc2 PortID 0x1f
da26: Command Queueing enabled
da26: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da23 at isp0 bus 0 scbus0 target 12 lun 0
da23: Fixed Direct Access SCSI-3 device
da23: 200.000MB/s transfers WWNN 0x20000018627f2f42 WWPN 0x22000018627f2f42 PortID 0x4
da23: Command Queueing enabled
da23: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da24 at isp0 bus 0 scbus0 target 11 lun 0
da24: Fixed Direct Access SCSI-3 device
da24: 200.000MB/s transfers WWNN 0x200000186285c3ec WWPN 0x220000186285c3ec PortID 0x8
da24: Command Queueing enabled
da24: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da25 at isp0 bus 0 scbus0 target 10 lun 0
da25: Fixed Direct Access SCSI-3 device
da25: 200.000MB/s transfers WWNN 0x200000186285c982 WWPN 0x220000186285c982 PortID 0x10
da25: Command Queueing enabled
da25: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da27 at isp0 bus 0 scbus0 target 3 lun 0
da27: Fixed Direct Access SCSI-3 device
da27: 200.000MB/s transfers WWNN 0x2000001862d04077 WWPN 0x2200001862d04077 PortID 0x25
da27: Command Queueing enabled
da27: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da4 at isp1 bus 0 scbus1 target 5 lun 0
da4: Fixed Direct Access SCSI-3 device
da4: 200.000MB/s transfers WWNN 0x2000001862e6b359 WWPN 0x2100001862e6b359 PortID 0x1b
da4: Command Queueing enabled
da4: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da6 at isp1 bus 0 scbus1 target 3 lun 0
da6: Fixed Direct Access SCSI-3 device
da6: 200.000MB/s transfers WWNN 0x2000001862eebf72 WWPN 0x2100001862eebf72 PortID 0x1e
da6: Command Queueing enabled
da6: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da28 at isp0 bus 0 scbus0 target 4 lun 0
da28: Fixed Direct Access SCSI-3 device
da28: 200.000MB/s transfers WWNN 0x2000001862d0349d WWPN 0x2200001862d0349d PortID 0x23
da28: Command Queueing enabled
da28: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da5 at isp1 bus 0 scbus1 target 4 lun 0
da5: Fixed Direct Access SCSI-3 device
da5: 200.000MB/s transfers WWNN 0x2000001862cdf7f2 WWPN 0x2100001862cdf7f2 PortID 0x1d
da5: Command Queueing enabled
da5: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da16 at isp0 bus 0 scbus0 target 1 lun 0
da16: Fixed Direct Access SCSI-3 device
da16: 200.000MB/s transfers WWNN 0x20000018627f3241 WWPN 0x22000018627f3241 PortID 0xf
da16: Command Queueing enabled
da16: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da12 at isp1 bus 0 scbus1 target 9 lun 0
da12: Fixed Direct Access SCSI-3 device
da12: 200.000MB/s transfers WWNN 0x20000018627f3241 WWPN 0x21000018627f3241 PortID 0xf
da12: Command Queueing enabled
da12: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da13 at isp1 bus 0 scbus1 target 7 lun 0
da13: Fixed Direct Access SCSI-3 device
da13: 200.000MB/s transfers WWNN 0x20000018627bec13 WWPN 0x21000018627bec13 PortID 0x17
da13: Command Queueing enabled
da13: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da3 at isp1 bus 0 scbus1 target 6 lun 0
da3: Fixed Direct Access SCSI-3 device
da3: 200.000MB/s transfers WWNN 0x2000001862d03be1 WWPN 0x2100001862d03be1 PortID 0x18
da3: Command Queueing enabled
da3: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da15 at isp0 bus 0 scbus0 target 2 lun 0
da15: Fixed Direct Access SCSI-3 device
da15: 200.000MB/s transfers WWNN 0x20000018627f2b3b WWPN 0x22000018627f2b3b PortID 0x2
da15: Command Queueing enabled
da15: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da1 at isp1 bus 0 scbus1 target 8 lun 0
da1: Fixed Direct Access SCSI-3 device
da1: 200.000MB/s transfers WWNN 0x200000186285c982 WWPN 0x210000186285c982 PortID 0x10
da1: Command Queueing enabled
da1: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da9 at isp1 bus 0 scbus1 target 13 lun 0
da9: Fixed Direct Access SCSI-3 device
da9: 200.000MB/s transfers WWNN 0x20000018627f3052 WWPN 0x21000018627f3052 PortID 0x1
da9: Command Queueing enabled
da9: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da10 at isp1 bus 0 scbus1 target 12 lun 0
da10: Fixed Direct Access SCSI-3 device
da10: 200.000MB/s transfers WWNN 0x20000018627f2b3b WWPN 0x21000018627f2b3b PortID 0x2
da10: Command Queueing enabled
da10: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da17 at isp0 bus 0 scbus0 target 0 lun 0
da17: Fixed Direct Access SCSI-3 device
da17: 200.000MB/s transfers WWNN 0x20000018627bec13 WWPN 0x22000018627bec13 PortID 0x17
da17: Command Queueing enabled
da17: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da11 at isp1 bus 0 scbus1 target 11 lun 0
da11: Fixed Direct Access SCSI-3 device
da11: 200.000MB/s transfers WWNN 0x20000018627f2f42 WWPN 0x21000018627f2f42 PortID 0x4
da11: Command Queueing enabled
da11: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da7 at isp1 bus 0 scbus1 target 2 lun 0
da7: Fixed Direct Access SCSI-3 device
da7: 200.000MB/s transfers WWNN 0x2000001862d03fc2 WWPN 0x2100001862d03fc2 PortID 0x1f
da7: Command Queueing enabled
da7: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da2 at isp1 bus 0 scbus1 target 10 lun 0
da2: Fixed Direct Access SCSI-3 device
da2: 200.000MB/s transfers WWNN 0x200000186285c3ec WWPN 0x210000186285c3ec PortID 0x8
da2: Command Queueing enabled
da2: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da8 at isp1 bus 0 scbus1 target 0 lun 0
da8: Fixed Direct Access SCSI-3 device
da8: 200.000MB/s transfers WWNN 0x2000001862d04077 WWPN 0x2100001862d04077 PortID 0x25
da8: Command Queueing enabled
da8: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)
da14 at isp1 bus 0 scbus1 target 1 lun 0
da14: Fixed Direct Access SCSI-3 device
da14: 200.000MB/s transfers WWNN 0x2000001862d0349d WWPN 0x2100001862d0349d PortID 0x23
da14: Command Queueing enabled
da14: 284481MB (573653847 520 byte sectors: 255H 63S/T 35708C)

Huh? 14 disks appearing as 28 devices? I hooked both ports of the FC adapter to the shelf, one to each controller. I guess the controllers are running in an active-active scenario. For now we’ll concern ourselves with the first 14 disks, later we’ll look at multipath. If you look at the entries you’ll notice something else that’s odd, 520 byte sectors. Netapp stores ECC information in each block of the disk for data protection purposes, I guess that’s what the additional bytes in the sectors are for. Unfortunately FreeBSD 8-stable doesn’t appear to have native support for 520 byte sectors. Any time I attempted to work with the disks I received a permission denied error or an I/O error however I was able to see the disks.

# camcontrol devlist
at scbus0 target 0 lun 0 (pass15,da15)
 at scbus0 target 1 lun 0 (pass16,da16)
 at scbus0 target 2 lun 0 (pass20,da20)
 at scbus0 target 3 lun 0 (pass19,da19)
 at scbus0 target 4 lun 0 (pass18,da18)
 at scbus0 target 5 lun 0 (pass17,da17)
 at scbus0 target 6 lun 0 (pass27,da27)
 at scbus0 target 7 lun 0 (pass26,da26)
 at scbus0 target 8 lun 0 (pass28,da28)
 at scbus0 target 9 lun 0 (pass24,da24)
 at scbus0 target 10 lun 0 (pass23,da23)
 at scbus0 target 11 lun 0 (pass25,da25)
 at scbus0 target 12 lun 0 (pass22,da22)
 at scbus0 target 13 lun 0 (pass21,da21)
 at scbus1 target 0 lun 0 (pass4,da4)
 at scbus1 target 1 lun 0 (pass9,da9)
 at scbus1 target 2 lun 0 (pass8,da8)
 at scbus1 target 3 lun 0 (pass3,da3)
 at scbus1 target 4 lun 0 (pass7,da7)
 at scbus1 target 5 lun 0 (pass13,da13)
 at scbus1 target 6 lun 0 (pass2,da2)
 at scbus1 target 7 lun 0 (pass12,da12)
 at scbus1 target 8 lun 0 (pass14,da14)
 at scbus1 target 9 lun 0 (pass11,da11)
 at scbus1 target 10 lun 0 (pass1,da1)
 at scbus1 target 11 lun 0 (pass10,da10)
 at scbus1 target 12 lun 0 (pass6,da6)
 at scbus1 target 13 lun 0 (pass5,da5)

After a little more searching it appeared that it was possible to reformat the disks with 512 byte sectors using a low-level format utility. Unfortunately in the beginning all the information I found was related to Windows or Linux. After some more searching I was able to locate this thread which helped me greatly. Essentially you have to set the sector size using the first camcontrol line. Then issue a low-level format command to the disk, the disk then handles formatting it self and reports back when done. Warning, this will destroy ALL data on the disks, so please make sure you’re formatting the right disk!

# camcontrol cmd da1 -v -c "15 10 0 0 v:i1 0" 12 -o 12 "0 0 0 8 0 0:i3 0 v:i3" 512
# camcontrol format da1 -q -y

I did this for all of the disks in the shelf and went to bed for the night. When I awoke in the morning all the low-level formats had finished and the disks appeared to now be formatted with 512 byte blocks!

# diskinfo -v da1
da1
512 # sectorsize
300351537152 # mediasize in bytes (280G)
586624096 # mediasize in sectors
0 # stripesize
0 # stripeoffset
36515 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
3KR22JF400009731E2K6 # Disk ident.

Awesome! Now all the disks appeared to be formatted properly. I created a partition which was 200 MB smaller than the size of the disk to account for any size difference between the 300GB NA07 disks and the older 300GB NA04 disks I had for spares.

[root@helix ~]# for i in da{1,2,3,4,5,6,7,8,9,10,11,12,13,14} ; do gpart create -s GPT $i; done
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk00 da1
[root@helix ~]# gpart show da1
=> 34 586624029 da1 GPT (280G)
34 2014 - free - (1.0M)
2048 586214429 1 freebsd-zfs (280G)
586216477 407586 - free - (199M)
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk01 da2
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk02 da3
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk03 da4
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk04 da5
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk05 da6
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk06 da7
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk07 da8
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk08 da9
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk09 da10
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk10 da11
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk11 da12
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk12 da13
[root@helix ~]# gpart add -b 2048 -s 586214429 -t freebsd-zfs -l disk13 da14
[root@helix ~]# zpool create array0 raidz2 gpt/disk00 gpt/disk01 gpt/disk02 gpt/disk03 gpt/disk04 gpt/disk05 gpt/disk06 gpt/disk07 gpt/disk08 gpt/disk09 gpt/disk10 gpt/disk11 gpt/disk12 gpt/disk13
[root@helix ~]# zpool status
pool: array0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
array0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gpt/disk00 ONLINE 0 0 0
gpt/disk01 ONLINE 0 0 0
gpt/disk02 ONLINE 0 0 0
gpt/disk03 ONLINE 0 0 0
gpt/disk04 ONLINE 0 0 0
gpt/disk05 ONLINE 0 0 0
gpt/disk06 ONLINE 0 0 0
gpt/disk07 ONLINE 0 0 0
gpt/disk08 ONLINE 0 0 0
gpt/disk09 ONLINE 0 0 0
gpt/disk10 ONLINE 0 0 0
gpt/disk11 ONLINE 0 0 0
gpt/disk12 ONLINE 0 0 0
gpt/disk13 ONLINE 0 0 0

Now I have a raidz2 array assembled with ZFS on my little intel Atom server!

Posted on

Giving up control…

So I’ve been avoiding iTunes for at least 5 years, simply because I was unwilling to relinquish control of my music. After recent issues with winamp’s support of my old iPod nano I finally decided to download the latest itunes and give it another whirl. I know I should have simply bought a non-apple mp3 player but this one was a gift and until it dies I’m going to keep using it. So far so good, my iPod has been behaving well and iTunes hasn’t been too annoying. As a nice bonus our new house has a central speaker system that I have the airport jacked into, so we can play our music from iTunes wirelessly to the house.

One side of me hates supporting the man and giving up control of my music but the other side is tired of constantly organizing and dealing with ripping CDs. iTunes is definitely a one stop shop for importing and managing my music. Damn you Apple!

Posted on

Elance Fail

I signed up for elance a few years ago when I was looking for work and thought of doing a little consulting. I never really saw any decent paying work on their site and really kinda forgot about them until I received this e-mail today.

Dear xxxx,

We recently learned that certain Elance user information was accessed without authorization, including potentially yours. The data accessed was contact information — specifically name, email address, telephone number, city location and Elance login information (passwords were protected with encryption). This incident did NOT involve any credit card, bank account, social security or tax ID numbers.

We have remedied the cause of the breach and are working with appropriate authorities. We have also implemented additional security measures and have strengthened password requirements to protect all of our users.

We sincerely regret any inconvenience or disruption this may cause.

If you have any unanswered questions and for ongoing information about this matter, please visit this page in our Trust & Safety center: http://www.elance.com/p/trust/account_security.html

For information on re-setting your password, visit: http://help.elance.com/forums/30969/entries/47262

Thank you for your understanding,

Michael Culver
Vice President
Elance

I commend them for going public with this rather embarrassing story, a lot of companies hide these events. However, I’m annoyed that all these companies that retain this kind of data really only care about security after this kind of thing happens.

Posted on

Sun Creator3D UPA

So I recently decided to reload my ultra10 machine with a recent copy of Solaris 9. Once the system was reloaded I had what appeared to be 8 bit color on my Creator 3D framebuffer. Issuing ffbconfig -prconf only listed resolutions, not color depths I could use. I found the answer on google, by issuing the following command I set my card to 1280×1024 @ 60hz and 24bit color depth:

ffbconfig -dev /dev/fbs/ffb0 -deflinear true -g 1.1 -res 1280x1024x60

If you don’t set the -g (gamma) option your color will look washed out. Another bit of strangeness I noted was that the part number (525-1746-06) on my creator3D UPA framebuffer can’t be found in Sunsolve which made it awfully difficult to identify the card and get it’s specs. Just thought I’d post this info in case someone else out there is looking.

Posted on

A sad day

I don’t normally post personal items on this site but I just found out through a post on his blog that Mark Hoekstra has passed away at the age of 34. I never had the privilege of meeting Mark but I have been reading his blog for years. I share his love of technology, old and new. I will truly miss his posts. My heart goes out to his family and friends.