Title: Class 17: Flash!
Date: 2014-04-01
Category: Classes
Tags: operating systems, kernel, kernel programming, storage, delay lines
Author: David Evans

   <div class="todo"> 
Because of insufficient "MU" answers on Exam 2, there is an [Extra
Exam](http://www.cs.virginia.edu/~evans/april1/grads-exam.pdf) required for all students 
and due at **11:69pm today, 1 April**.  (Note: if you have not yet
received a [theory textbook](http://www.dori-mic.org), you can skip the
theory questions 1 and 2, but since cs3330 is a prerequisite for this
course, everyone is expected to answer the architecture questions 3 and
4.)  Note: please make sure you follow [RFC 3514](http://www.ietf.org/rfc/rfc3514.txt) when downloading the exam.
   </div>

<center>
<iframe src="http://www.slideshare.net/slideshow/embed_code/32998203" width="476" height="400" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe>
</center>


## Readings

Part III of the [OSTEP Book](http://pages.cs.wisc.edu/~remzi/OSTEP/) is
all about persistent storage.  None of these readings are "required",
but some of these chapters may be helpful to you in designing and
implementing your file system for PS4 (and the others are interesting
also).

[Dialogue on Persistence](http://pages.cs.wisc.edu/~remzi/OSTEP/dialogue-persistence.pdf)  
[I/O Devices](http://pages.cs.wisc.edu/~remzi/OSTEP/file-devices.pdf)  
   <div class="indented">
How the OS communicates with devices.  
   </div>
[Hard Disk Drives](http://pages.cs.wisc.edu/~remzi/OSTEP/file-disks.pdf)  
   <div class="indented">
How to store data on hard drives to minimize seek times.
   </div>
[Redundant Arrays of Inexpensive Disks](http://pages.cs.wisc.edu/~remzi/OSTEP/file-raid.pdf)  
   <div class="indented">
We cover this some in class tody, but the chapter has lots of details we don't get into.  
   </div>
[Files and Directories](http://pages.cs.wisc.edu/~remzi/OSTEP/file-intro.pdf)  
[File System Implementation](http://pages.cs.wisc.edu/~remzi/OSTEP/file-implementation.pdf)  
   <div class="indented">
This is the section you will find most directly relevant to PS4
(although your filesystem can be even simpler than the _very simple file
system_ described in the section.
   </div>
[Locality and the Fast File System](http://pages.cs.wisc.edu/~remzi/OSTEP/file-ffs.pdf)  
[Crash Consistency: FSCK and Journaling](http://pages.cs.wisc.edu/~remzi/OSTEP/file-journaling.pdf)  
[Log-structured File Systems](http://pages.cs.wisc.edu/~remzi/OSTEP/file-lfs.pdf)  
[Data Integrity and Protection](http://pages.cs.wisc.edu/~remzi/OSTEP/file-integrity.pdf)  
[Summary Dialogue](http://pages.cs.wisc.edu/~remzi/OSTEP/file-dialogue.pdf)

# Making a File

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/QRF7VzpPqXA?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

# Requirements for Modern File Systems

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/p-d55Yn5ROY?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>


# Improving Reliability

[OpenZFS](http://open-zfs.org/)  
[maczfs](https://code.google.com/p/maczfs/)

> _It's for people who have Mac OS, who have any data, and who really like their data. Whether on a single-drive laptop or on a massive server, it'll store your petabytes with ragingly redundant RAID reliability, and it'll keep the bit-rotted bleeps and bloops out of your iTunes library._

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/Js535LqapFE?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

## Merkle Trees

When one block changes, how many hashes need to be updated?
<div class="gap">

</div>

## RAID

David Patterson, Garth Gibson, and Randy Katz. [_A Case of Redundant Arrays of Inexpensive Disks (RAID)_](|filename|./Patterson88.pdf) (SIGMOD 1988)

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/Ugbua8pTS6c?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

With RAID 3, if Disk 2 fails, how can it be recovered?
<div class="gap">

</div>

# Adaptive Replacement Cache

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/_XDHPhdQHMQ?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

What should cause the recently used cache size to increase?
<div class="gap">
</div>

What should cause the recently used cache size to shrink?
<div class="gap">
</div>

# Modern Storage Systems

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/qXjuVxlVyfc?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

Fujio Masuoka, Masaki Momodomi, Yoshihisa Iwata, and Riichiro Shirota.
[_New Ultra High Density EPROM and Flash EEPROM with NAND Structure
Cell_](|filename|./nandflash.pdf). IEEE Electron Devices Meeting, 1987.

What does it mean for storage to be _solid state_?
<div class="gap">

</div>

What does it mean for storage to be _non-volatile_?
<div class="gap">

</div>

How should file systems be re-designed to account for SSDs and networked storage?
<div class="gap">

</div>

## Log-Structured File Systems

<center>
<iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/KTCkW_6zz2k?list=PLvpsxlEF9cP0ZrgWIm5RPoCSuwoUAaT9f" frameborder="2" allowfullscreen></iframe>
</center>

Mendel Rosenblum and John K. Ousterhout, [_The Design and Implementation of a Log-Structured File System_](|filename|./lsfs.pdf). 13<textsuperscript>th</textsuperscript> Symposium on Operating System Principles (SOSP) 1991.

What are the advantages of doing all writes sequentially on the disk?
<div class="gap">

</div>

How is garbage collection different for a LSFS compared to in-memory garbage collection?
<div class="gap">

</div>

## Flash Memory File Systems

Why is a Log-Structured File System more appropriate for flash memory than a traditional Unix-style file system?
<div class="gap">

</div>

Bunny Huang, [_On Hacking MicroSD Cards_](http://www.bunniestudios.com/blog/?p=3554), December 2013.

For more interesting hardware hacks, see Karsten Nohl's talk on [SIM
Card Hacking](http://rust-class.org/0/class-18-sim-card-exploitation.html)
from last year's course, and his public talk:
<center>
<iframe width="480" height="360" src="//www.youtube-nocookie.com/embed/fCwY27EnOF8?rel=0" frameborder="0" allowfullscreen></iframe>
</center>

<div id="disqus_thread"></div>

<script type="text/javascript">
        /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
        var disqus_shortname = 'rust-class'; // required: replace example with your forum shortname
	var disqus_url = 'http://www.rust-class.org/class-17-flash.html';

        /* * * DON'T EDIT BELOW THIS LINE * * */
        (function() {
            var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
            dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
            (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
        })();
</script>
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>