Deep storage: How to save your data for a billion years

Bible and Magna Carta stored in glass by University of Southhampton
Adrian Pennington
Post Production
May 19th 2016 at 12:42PM : By

Fresh breakthroughs in tape, disc, film and glass herald a new era of eternal data archiving. If employed intelligently, there’s no reason we won’t be able to preserve Keeping Up With Kardashians for our great-great grandchildren. Adrian Pennington reports.

Many in the industry are concerned about how to store their data over the next year or two. But how do we preserve our data for the next decade? Or the next century? Or beyond?

In Egypt, around 196 BC, someone carved an honours list in three languages onto a slab of granodiorite. The mundane text was rediscovered in 1799 and finally decrypted to provide the essential key to modern understanding of ancient Egyptian civilization. The Rosetta Stone is the perfect database. It has physically lasted for centuries and its information can be read without any new technology. If only the quest to find an archive solution for digital media were as simple.

The world is overflowing with digital data, and the digital universe is doubling every two years according to IDC. A share of this digital universe has value in the long term so what are the options?

In the digital age, tape has proved surprisingly durable. Anyone who has seen the film The Big Easy will know how easy it is to put a magnet next to tape and erase its contents. Tape is subject to degradation and bit drop out over time, and while industry standard LTO gets around this by recording data without the revolving head drum used on video tape, the system needs manual intervention every few years in order to migrate the data stored on it to the latest generation.

The new generation of LTO-7 tape, manufactured by Fujifjlm, is composed of Barium Ferrite, a medium with magnetic properties which means the tape does not deteriorate, and it gives tape headends a longer lifespan. Plus the capacity has jumped from 2.5TB to 6TB.

“It's like a whole new format,” says Fuji's commercial manager Richard Alderson. “Nothing has been done like this in the past and we are the only manufacturer who can provide gen-7 tape.” Which is increasingly important given the move to UHD.

“A single movie at 4K can need over a petabyte and as the data sets get bigger, customers are realising that tape is far safer and more reliable than disc as a storage medium,” explains David McKenzie, storage and archive specialist, Oracle.

Oracle's StorageTek division is readying a new enterprise version of its tape drive called T10K for release early 2017. This will have capacity for 10-15TB. In addition Oracle is working with the team and the Diva technology from Front Porch, the firm it acquired in September 2014. Meanwhile LTO-8 with a projected 12.8TB capacity and 427MBps speed is expected in three years.

“Tape is far from dead. In fact it is a lot cheaper than disc. It is more environmentally friendly and most important it is far less corruptible. It's the reason why broadcasters like the BBC and Sky choose to archive their programme catalogues on it.”

30 year optical disc

The main alternative to LTO is optical disc, which, as McKenzie alludes to, can drain power in order to keep the mechanism cool. Earlier this year, Sony and Panasonic launched new optical disc-based storage systems for data centres. Sony's Everspan can store 181 Petabytes for 100 years. Four systems can be ganged together to offer 724PB of total storage. To grasp that, if you were to envision one bit of data as the equivalent to one second, then 1PB would equal 285 million years.

Sony optical disk drive

Sony says Everspan is able to transfer 18GB of data per second, “outpacing the best performance of tape libraries and archival drive platforms. Because of the durability of optical discs, unlike other storage media, users are expected to never need to migrate data.”

The initiative is led by Frank Frankovsky whose start-up company Optical Archive was acquired by Sony last year. Previously, Frankovsky led a project for Facebook to store the social network's burgeoning data and helped Panasonic develop something along similar lines called freeze-ray. It seems that Facebook is hedging its bets by deploying both Sony and Panasonic variants of Frankovsky's system.

Frankovsky says the goal is to make it possible for customers to store everything for as long as they wish in a low-touch, low-cost optical library. “We’re finally bringing a product to market that will make tape obsolete technology,” he says.

The Everspan media developed by Panasonic and Sony is the same as used in Sony's next version of its Optical Disc Archive (ODA) unveiled at NAB 2016. A single cartridge has doubled in capacity to 3.3 TB. ODA is designed for use in near-line applications, deep archive storage or disaster recovery systems. Hardware configurations range from stand-alone to large, scalable robotic archive systems. The main components of ODA Generation 2 include: a stand-alone USB drive unit (ODS-D280U), an 8 GB fiber channel library drive unit (ODS-D280F), for use in robotic systems, and the Optical Disc Archive media cartridge (ODC3300R).

100 year metal alloy tape

While LTO tape has a lifespan of 30 years, DOTS (Digital Optical Technology System) stores digital data onto metal alloy tape and is claimed to be archival for 100 years. Originated by Kodak and developed since 2008 by Group 47, the technology's software converts a digital file into a visual representation of the data. With sufficient magnification, one can actually see the digital information.

Its specification – the 'Rosetta Leader' - calls for microfiche-scale human readable text at the beginning of each tape with instructions on how the data is encoded and instructions on how to actually construct a reader (it even resembles the Rosetta Stone – see image). Because the information is visible, as long as cameras and imaging devices are available, the information will always be recoverable, the company says.

Declaration of Independence

Group 47 has software that converts files into a visual representation of the data. This example contains the American Declaration of Independence. This is how the media appears under polarised light.

500 year film

However, the only technology which has proven it can last a century is film. What's more it has the valuable benefit of easy reading simply by shining a light through the negative. Yet celluloid is fragile, some types are notoriously flammable, and it’s expensive despite the fact that the bulk of film stock made by Kodak and 35mm scans made from the material are now for the archive market.

With Fraunhofer and Norner, Norway's Piql has devised a way to use the preservation qualities of photosensitive film combined with the accessibility of being part of a standard IT infrastructure. Its turnkey solution includes all equipment and processes needed for writing, storing and retrieving files and is claimed to last 500 years. A high-precision piqlWriter records digital files and related metadata onto photosensitive film. Checksums are applied to verify the integrity of the data. Forward Error Correction is used for controlling errors, making it possible to fully retrieve even damaged or corrupted data.

Piql box and film

The PiqlBox and film storage package is rated to last 500 years. 

“Both digital and visual storage of data is possible,” according to the company. “This means users can select between storing data in computer readable digital format (binary codes), or as text or images. It can even combine the two, allowing users to get visual previews of the data.

It provides a self-documenting preservation master containing all information needed for decoding and understanding the preserved data. The source code of the decoding software is open and written in text format on the reel.

Piql writer and reader

The Piql Reader and Writer unit 

A billion years and more

Scientists at the University of Southampton have gone way further. Using glass, scientists from the university’s Optoelectronics Research Centre have developed the recording and retrieval processes of five dimensional (5D) data, which is calculated to survive for billions of years.

The glass isn't the common or garden double glazed variety. The data is recorded via an unbelievably fast laser, with pulses of light fired at 280 quadrillionths of a second onto self-assembled nanostructures created in discs of fused quartz.

Pulses of light fired at 280 quadrillionths of a second onto self-assembled nanostructures created in discs of fused quartz

A file is written in three layers of nanostructured dots separated by five micrometres (one millionth of a metre) and in five dimensions: the size and orientation in addition to the three dimensional position of these nanostructures.

It sounds like science fiction and has already been christened as the ‘Superman memory crystal’, yet Hitachi also announced a similar etched glass data storage solution in 2012.

Cultural heritage documents like the Bible and Magna Carta have already been fused in 5D (see image, top) and the team are looking for partners to commercialise the technology.

The medium permits thermal stability up to 1000°C and virtually unlimited lifetime at room temperature and can be read by combination of optical microscope and a polariser, similar to that found in Polaroid sunglasses. Just don't drop it.