What is the best storage device for my project?

General Discussions
1

One question that I see show up periodically is “Which storage device is best, fastest, smallest, etc.”

A gentleman named James Chambers has done extensive research on this topic, and has a blog posting, Raspberry Pi Storage Benchmarks, where he discusses the pro’s and cons of various storage types used in various project configurations.

He also provides a script that, when run as root, will evaluate the drive based on a number of parameters and then earns a “score” which is a measure of how well it will do running as storage on a Raspberry Pi. Since it is designed to emphasize short file read/write performance, this will show how well a device will function as a mounted root filesystem instead of just a repository for pictures and videos.

Though provided as a “curl” script, if you download it directly and save it somewhere, you can run repeated tests, and can specify a device to test - via it’s mount point - on the command line.

Storage_Benchmark.sh.txt (32.1 KB)

Rename by removing the “.txt”, allow execution, and run as root from the command-line which will benchmark the root filesystem by default. You can specify an optional path leading to a mounted device to test if you want to test something else.

Though designed specifically for Raspbian on the Pi, it should run under any other Debian/Ubuntu based distribution, on whatever desktop/laptop/Raspberry Pi hardware supports it.

Takeaways:

  1. If you have the room and power budget, SSD’s are the way to go.
    (Note that the new Seagate One Touch SSD and Seagate Expansion SSD drives are about the size of two flash-drives side-by-side, and appear to require little current, so for a more portable situation that want’s an SSD, this is an option.)

  2. The next best choice are the SanDisk Extreme Plus series of micro SD cards. Since they are “application” (A1 or A2) rated, they’'re much faster than normal SD cards and are almost like SD cards.

  3. Micro Center SD cards are also very good, but not as good as the SanDisk Extreme Plus series. For what you pay, they’re truly worthy cards.

  4. Running from any kind of flash-drive storage is an exercise in frustration.

  5. You can use the attached script to benchmark your own devices and see how they compare.

1 Like
2

Cool!

It appeared to hang at

40% [Connecting to mirror.pit.teraswitch.com (2607:fdc0:1::50)]

but eventually completed the downloads/installation stuff.

It ran a few tests, then hung Carl “incommunicado”

Clock speeds: CPU: 1200 - Core: 400 - RAM: 450
System rootfs drive (/) has been detected as /dev/mmcblk0p2 (mmcblk0p2)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  725k  100  725k    0     0  1375k      0 --:--:-- --:--:-- --:--:-- 1374k
System:    Host: Carl Kernel: 4.14.98-v7+ armv7l bits: 32 compiler: gcc v: 4.9.3 Console: tty 0 
           Distro: Raspbian GNU/Linux 9 (stretch) 
Machine:   Type: ARM Device System: Raspberry Pi 3 Model B Rev 1.2 details: BCM2835 rev: a22082 serial: 00000000395aab86 
CPU:       Info: Quad Core model: ARMv7 v7l variant: cortex-a53 bits: 32 type: MCP arch: v7l rev: 4 
           features: Use -f option to see features bogomips: 307 
           Speed: 1200 MHz min/max: 600/1200 MHz Core speeds (MHz): 1: 1200 2: 1200 3: 1200 4: 1200 
Graphics:  Device-1: bcm2708-fb driver: bcm2708_fb v: kernel bus ID: N/A 
           Device-2: bcm2835-hdmi driver: N/A bus ID: N/A 
           Display: server: X.org 1.19.2 driver: fbturbo tty: 152x55 
           Message: Advanced graphics data unavailable in console for root. 
Network:   Device-1: Standard Microsystems SMSC9512/9514 Fast Ethernet Adapter type: USB driver: smsc95xx bus ID: 1-1.1:3 
           IF: eth0 state: down mac: xxx 
           Device-2: Belkin F7D2102 802.11n N300 Micro Wireless Adapter v3000 [Realtek RTL8192CU] type: USB driver: rtl8192cu 
           bus ID: 1-1.5:5 
           IF: wlan0 state: up mac: xxxx 
           IF-ID-1: wlan1 state: up mac: xxxx 
Drives:    Local Storage: total: 29.72 GiB used: 9.62 GiB (32.4%) 
           ID-1: /dev/mmcblk0 model: SC32G size: 29.72 GiB 
           Message: No Optical or Floppy data was found. 
Partition: ID-1: / size: 29.09 GiB used: 9.59 GiB (33.0%) fs: ext4 dev: /dev/mmcblk0p2 
           ID-2: /boot size: 41.7 MiB used: 22.1 MiB (53.1%) fs: vfat dev: /dev/mmcblk0p1 
Info:      Processes: 172 Uptime: 53d 22h 16m Memory: 1003.7 MiB used: 739.5 MiB (73.7%) gpu: 128.0 MiB Init: systemd 
           runlevel: 5 Compilers: gcc: 6.3.0 Packages: 1749 Shell: Sudo v: 1.8.19p1 inxi: 3.1.08 
Card CSD status register: MID: 3 OID: SD PNM: SC32G PRV: 8.0 MDATE: 5/2020
Card SCR status register: SD Physical Version Specification: 5
MicroSD information: Clock Speed: 50.0 - Manufacturer: SanDisk - Model: SC32G - Vendor: SD - Product: SD - HW Version: 0x8 - FW Version: 0x0 - Date Manufactured: 05/2020
Class: A1 Class 10 U1
Running HDParm tests ...
/dev/mmcblk0p2:
 Timing O_DIRECT cached reads:    44 MB in  2.03 seconds =  21.62 MB/sec
 Timing O_DIRECT disk reads:  68 MB in  3.09 seconds =  22.04 MB/sec
HDParm: 22.04 MB/s - HDParmCached: 21.62 MB/s
Running dd tests ...
81920+0 records in
81920+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 19.166 s, 17.5 MB/s
DD Write Speed: 17.5 MB/s
Running fio write test ...
fio: job startup hung? exiting.
fio: 1 job failed to start
^C

I can’t get into him to clean up or reboot. And the disk access LED is green constant…Ouch!

BTW, this is SanDisk Ultra 32GB.

…

UPDATE: Patience is a virtue I have little of, but Carl just “woke up” and started complaining “Safety Shutdown Is Imminent, Please Put Me On My Dock!”

1 Like
3

Yup, that’s what I use. The Sandisk Extreme Plus! I also like the EVO select from Samsung, when it’s on sale.

2 Likes
4

Finally got it to run to completion:

Description: SanDisk Ultra 32GB microSD on Pi 3B GoPiGo3 robot running Raspbian For Robots Stretch             



     Category                  Test                      Result     
HDParm                    Disk Read                 21.99 MB/s               
HDParm                    Cached Disk Read          21.95 MB/s               
DD                        Disk Write                17.8 MB/s                
FIO                       4k random read            2364 IOPS (9458 KB/s)    
FIO                       4k random write           705 IOPS (2820 KB/s)     
IOZone                    4k read                   7961 KB/s                
IOZone                    4k write                  3370 KB/s                
IOZone                    4k random read            8244 KB/s                
IOZone                    4k random write           2379 KB/s                

                          Score: 1028                                        

Compare with previous benchmark results at:
https://storage.jamesachambers.com/
1 Like
5

How’d ya get it to work?

6

I pulled out Carl’s battery plug. That shut him up good!

After a short nap on his dock, I woke him up and ran the test after he settled down from booting.

1 Like
7

One thought I was going to mention before, but forgot, is that THE VERY FIRST THING this script does is “trim” the device you’re testing.

Assuming that you, like all of us, diligently TRIM their solid state devices, (like SD cards), hourly via a cleverly written CRON job. . . .

What’s that? You’ve never TRIMmed your SD cards?  (Nope, me neither.)

That being true, (was true for me too), and on a bot like Carl that’s been around a few times, the initial TRIM that the script does could take. . . . hours.

There’s actually an article I should write about solid state memory devices and TRIM.

It’s kind-of like defragging that Windows '95 drive that hadn’t been touched since DOS 1.0. The first full defrag takes forever!

The magic word here is “fstrim”
(Viz.: https://www.man7.org/linux/man-pages/man8/fstrim.8.html)

Short answer waiting for an article:
TRIM on a solid-state device is analogous to “defrag” on hard, spinning, media, except that solid-state type memory devices don’t have fixed-in-place, immutable, sectors and solid state devices cannot intrinsically tell what data can be nuked to make room for new data.

TRIM is what tells the solid-state device’s media controller on-chip exactly which of the zillions of clusters are fair-game. Once it knows what’s nukeable and what’s not, it can actively gather valid data and clear space that isn’t needed to make those blocks ready for use.

If it hasn’t been run in a while, (or, maybe, at all?), that initial TRIM cycle can take non-trivial amounts of time because it is now, and only now, that the SD card’s (or SSD’s) internal electronics know for sure what it can throw away and how to best optimize the data that’s real.

That being said. . . . .

Running fstrim, (I’m assuming it needs root access), will automatically work on the Pi since the support is enabled, but the fstrim.timer service isn’t started. There’s a bunch of stuff you can read about enabling the fstrim service timer and changing it’s frequency.

You can, (after cloning the card, just in case!), execute:

fstrim -v /

(Trim solid-state memory, verbose, on the partition mounted on “/”)

Since /boot is it’s own partition, you can trim it too.

You can experiment with this by creating a cron-job that runs fstrim -a periodically - daily, weekly, hourly, it depends on how much data manipulation you do on your device.

In any event, please try it - it would be interesting to see how it goes. Since this is, (to me, at least), somewhat experimental, make backups first!

1 Like
8

The Extreme Plus series cards blow everything else so far outta the water it’s not funny. Benchmark score wise, by a factor of 2 or 3 times.

The EVO series are good, but not so good at being run in a Pi.

Micro Center cards are at least as good, score wise, as the Samsung EVO, but for a LOT less money. If you have Micro Center up in Canada, stock up. Otherwise, you’re due a visit to the 'States anyway! (laughing)